Born Mitchell William Miller to a Jewish family in Rochester, New York, he and his choir were famed for their campy Sing Along With Mitch television show of the 1960s, where the choir sang a popular or well-known tune while the lyrics scrolled across the television screen, the current position of the choir's singing marked by an animated ball that bounced from word to word. During the second season of "Sing Along with Mitch," Mitch Miller himself coined the catch phrase "All Smiles." These were preceded by the instructions to "sing along; just follow the bouncing ball." Miller's biggest hits with the choir were "The Yellow Rose of Texas," a drum-propelled version of the old march, and "Colonel Bogey March", then enjoying new popularity from its use in the movie The Bridge on the River Kwai (1957). In addition to the television show, Miller and his choir recorded a series of best-selling albums.

One of the singers in Miller's choir, Bob McGrath, went on to a long career as one of the hosts of the PBS children's television show, Sesame Street.

As head of artists & repertory (A&R) at Mercury Records in the late forties, and Columbia Records in the early fifties and early sixties, Miller gained a reputation for creating records that created what music historian Will Friedwald calls an "aural texture":

"Miller also conceived of the idea of the pop record "sound" per se: not so much an arrangement or a tune, but an aural texture (usually replete with extramusical gimmicks) that could be created in the studio and then replicated in live performance, instead of the other way around. Miller was hardly a rock 'n' roller, yet without these ideas there could never have been rock 'n' roll. 'Mule Train,' Miller's first major hit (for Frankie Laine) and the foundation of his career, set the pattern for virtually the entire first decade of rock. The similarities between it and, say, 'Leader of the Pack,' need hardly be outlined here." -- Will Friedwald noted in SINATRA: THE SONG IS YOU, Da Capo Press, 1997.

While this did not always sit well with Columbia's more established performers (Frank Sinatra, for example), Columbia had an amazing hit-to-release ratio during that time.

Miller signed and produced many important popular/standards artists for Columbia, including Frankie Laine, Johnnie Ray, Ray Conniff, Percy Faith, Johnny Mathis, Tony Bennett, Guy Mitchell (whose pseudonym actually was based on Miller's first name), and helped direct the careers of artists who were already signed to the label, like Doris Day, Dinah Shore and Jo Stafford to just name a few. He therefore became one of the most successful pop music producers ever.

Miller is frequently (and probably unfairly) referred to by rock music historians as an "enemy" of early rock and roll. He did back John Hammond's signing of Bob Dylan to capitalize on the folk music craze. While he did ultimately lose his job as Columbia head for not signing the types of acts teenagers were buying, Miller did originally attempt to sign Elvis Presley, but balked at the amount Presley's manager, Colonel Tom Parker, was asking.

Mitch Miller is a graduate of the Eastman School of Music. He supported himself in his early career in the 1930s and 1940s as a session oboist; among his more celebrated studio dates in the non-classical field were for The Voice of Frank Sinatra and bebop pioneer Charlie Parker's famous Bird With Strings albums.