A blues-based pianist who played with trumpeter Henry "Red" Allen, trumpeter Buck Clayton, and saxophonist Johnny Hodges, among others, Williams lived in Chicago from an early age. He studied piano as a child and began playing professionally in Chicago around 1935. He formed a 12-piece dance band around 1936, and a trio in 1942. During the '40s he played with such musicians as Allen, clarinetist Jimmy Noone, and bandleader Erskine Tate. In the late '40s Williams played in a duo with his wife, singer Aubrey Hobbs; the pair was known as Alfred and Aubrey. Williams also arranged for Chicago-area bands around this time. In the '50s he played around New York, including gigs at the Metropole and Savoy Ballroom. He toured Europe with Clayton in 1959 and Hodges in 1961.