Although he played with a many first-rate leaders over the course of his career, Woodyard is most closely associated with Duke Ellington. And rightfully so; one has only to listen to the Ellington Orchestra's legendary performance of "Diminuendo and Crescendo in Blue" from the 1956 Ellington at Newport album to appreciate how important Woodyard became to the legendary composer's later work. Woodyard wasn't a flashy player, but few swung harder; he drove the Ellington band like no other drummer in the band's storied history.
Musically, Woodyard was largely self-taught. In the '40s, he played with local groups in and around his hometown of Newark. He played in Paul Gayten's R&B group around 1950. In the early '50s, prior to joining Ellington, Woodyard played with tenor saxophonist Joe Holiday, trumpeter Roy Eldridge, and pianist Milt Buckner. His tenure with Ellington began in 1955; it lasted (with a few interruptions) until 1966. He then joined Ella Fitzgerald's band and moved to Los Angeles. His health failed during the '70s and his playing activities declined, although he did perform, most notably on congas with the Buddy Rich Big Band on such albums as Roar of '74 and Ease on Down the Road. In the late '70s, Woodyard toured with pianist Claude Bolling. In 1983, he recorded with an all-star band that included pianist Teddy Wilson, tenor saxophonist Buddy Tate, and bassist Slam Stewart. In July 1988, then living in Paris, Woodyard played on Steve Lacy's album The Door. He died just over a month later.