Jimmy Dorsey was born in Shenandoah, Pennsylvania, the son of a music teacher and older brother of Tommy Dorsey who also became a prominent musician. He played trumpet in his youth, appearing on stage in a Vaudeville act as early as 1913. He switched to saxophone in 1915, and then learned to double on clarinet.

He formed a band with his brother Tommy, then in 1924 joined the California Ramblers (who were based in New York City). He did much free lance radio and recording work throughout the 1920s. The brothers also appeared as session musicians on many jazz recordings. He joined Ted Lewis's band in 1930, with whom he toured Europe.

After returning to the USA he worked briefly with Rudy Vallee and several other bandleaders, in addition to the Dorsey Brothers Orchestra with Tommy. Tommy broke off to form his own band in 1935. Jimmy continued leading his own band until the start of the 1950s. In 1953 he joined Tommy's Orchestra, renamed "The Fabulous Dorseys"; he took over leadership of the orchestra after Tommy's death.

Jimmy Dorsey appeared in a number of Hollywood motion pictures, including That Girl From Paris, Shall We Dance?, The Fleet's In, Lost in Harlem, I Dood It, and the bio-pic with his brother Tommy, The Fabulous Dorseys.

Dorsey is considered one of the most prominent alto saxophone players of the pre-bebop era.