Born in Oxford, MS, in 1938, the son of a white sharecropper, Ivy grew up on country music. His interest in R&B was fueled by the music of Ivory Joe Hunter and LaVern Baker and the WLAC-AM Nashville DJs, John R and Hoss Allen.
After starting a radio DJ career in his native Oxford, Ivy moved on to WMPS in Memphis, TN, and WLAY in Muscle Shoals, AL, before settling in Sheffield, AL. Opening a record store called Tune Town, Ivy began writing songs with producer Rick Hall of Fame Recording Studios in Muscle Shoals. Singer Jimmy Hughes recorded the duo's "I'm Qualified" and "Lollipops, Lace and Lipstick" was the flip side of Hughes' "Steal Away," number 17 R&B, summer 1964, the debut release of Hall's Fame label. In 1965, Ivy opened his Quinvy recording studio across the street from Tune Town, hiring guitarist/engineer Marlin Greene and giving him a percentage of the business. Wanting to expand into record production, Ivy approached weekend singer Percy Sledge about auditioning. He liked one song Sledge's band members Andrew Wright and Calvin Lewis had written but wanted a rewrite.
The revised song now titled "When a Man Loves a Woman" was recorded at Ivy's studio in late 1965 using some of the musicians who had played on hits produced by Hall: drummer Roger Hawkins, organist Spooner Oldham, bassist Junior Lowe, and Ivy's co-producer/guitarist Marlin Greene. Through Rick Hall, Ivy sent a the tape to Jerry Wexler of Atlantic Records.
In spring 1966, "When a Man Loves a Woman" became a million seller, holding the number one R&B spot for four weeks and the number one pop spot for two weeks on Billboard's charts. The When a Man Loves a Woman LP peaked at number 37 pop in summer 1966. The single was used in a 1987 Levi jeans ad, reissued in England, and became a number two U.K. hit that same year. A standard, "When a Man Loves a Woman" has been covered by countless artists (Shirley Scott, Barbara Mandrell, Wes Montgomery, Yusef Lateef) hitting number one pop again, this time for Michael Bolton in fall 1991.
Ivy started his own labels, Quinvy and South Camp, distributed by Atco/Atlantic. Some of the releases are favorites among Northern soul devotees: Tony Varner's "Tearstained Face"; Bill Brandon's "Self-Preservation," "One Minute Woman," and "Rainbow Road"; and Tony Borders' "Cheaters Never Win" and "What Kind of Spell."
In the '70s, Ivy sold his studio, earned an M.A. degree from Mississippi State University, and began teaching accounting courses.