Greene took guitar lessons when he was eight years old, then added drumming to his abilities. Moving to Atlanta in the late 40s, he became part of the Cherokee Trio with Lem Bryant and Speedy Price. He then became a member of the Rhythm Ranch Boys and was a popular radio entertainer on Georgia Jublilee on WTJH. Greene's career was interrupted for military service in Korea, but he returned to Atlanta and joined the Peachtree Cowboys, also working as a salesperson and construction worker. In 1962, he joined Ernest Tubb & the Texas Troubadours as a drummer and occasional vocalist. He was featured on Presents the Texas Troubadours, and his performance on "The Last Letter" led to solo recordings. Starting in 1965 with "Ever Since My Baby Went Away'" Greene had a succession of country hits, including number ones with "There Goes My Everything," "All The Time," "You Are My Treasure," "Until My Dreams Come True," and "Statue of a Fool." He did not leave Tubb’s band until 1967, and only then because Tubb tired of hearing calls for the drummer to sing.
In 1969, Greene had another hit with Hank Cochran's song "I Wish I Didn’t Have to Miss You,"’ on which he was partnered with Cochran’s wife, Jeannie Seely. She became part of his road show and they continued to record together. Capitalizing on the popularity of outlaw country, they changed the name of their band from the Jolly Greene Giants to the Renegades, but they stayed with middle-of-the-road country music. Although his last chart entry was with "If It’s Love (Then Bet It All)" in 1984, Greene, who joined the Grand Ole Opry in 1967, remained a regular performer.