Born in Budapest (his original name was Jenö Blau), he studied at the conservatoire there. In 1921 he moved to the United States of America (taking his name from the ship on which he travelled, the Normandie), where he worked first as a violinist and later conductor of the Capitol Theater Orchestra in New York City, a group which accompanied silent movies.

Ormandy was appointed conductor of the Minneapolis Symphony Orchestra from 1931, where he served until 1936. During the darkest days of the Great Depression, RCA Victor contracted Ormandy and the Minneapolis Symphony for many recordings including famous readings of Bruckner's Symphony No. 7 and Mahler's Symphony No. 2. The excellence of these records contributed to Ormandy's reputation as an exceptional musician. His career was also aided enormously by Arthur Judson, the most powerful manager on the American classical music scene in the 1930s.

Ormandy's 44-year tenure with the Philadelphia Orchestra is the source of much of his current reputation and fame. Appointed as the Philadelphia Orchestra associate conductor (alongside Leopold Stokowski) from 1936, he became two years later the sole music director and conductor until his retirement in 1980, when he was made the band's Conductor Laureate.

To Ormandy's direction the Philadelphia Orchestra owed much of its famous lush, legato style, particularly in terms of string bowing and tone. His style attracted praise for its opulent sound but also was criticized for a purported lack of any real individual touch. In recent years, revisionist music critics have tended to downplay Ormandy's life achievements and impact, even though Ormandy enormously raised the bar for musical performance standards. He demonstrated exceptional musical and personal integrity, and will be forever remembered for his exceptional leadership skills, and his podium manner was formal and reserved, in the style of his idol, Toscanini. One orchestra musician complimented him by saying: "He doesn't try to conduct every note as some conductors do."

He always conducted the orchestra from memory. Interestingly, many web sites feature stories about Ormandy's humor and occasional awkward lapses in English-language usage while preparing musicians in rehearsal at Philadelphia's Academy of Music auditorium.

Ormandy, particularly noted for late Romantic and early 20th century works, was a champion of Sergei Rachmaninoff's music, conducting the premiere of his Symphonic Dances. He also directed the US premiere of several symphonies by Dmitri Shostakovich, as well as the premiere performances of several works by American composers. He made the first recording of Deryck Cooke's first performing edition of the complete Mahler Tenth Symphony, which many critics praised.

Ormandy led his orchestra through many performances in New York and other cities around the United States. He also led his orchestra through many foreign tours. Among the most famous foreign were a 1955 tour of Finland where many of the orchestra's members visited the elderly composer Jean Sibelius at his obscure country estate, and a 1973 tour of the People's Republic of China where the orchestra performed to enthusiastic audiences who had been isolated from Western classical music for many decades.

He also appeared as a guest conductor with other orchestras, and in the late 1960s recorded a highly memorable and idiomatic rendition of Dvorak's New World Symphony with the London Symphony Orchestra. In 1950 he directed New York's Metropolitan Opera in a fondly-remembered production of Johann Strauss' Die Fledermaus, which also was recorded. Nonetheless, his overriding loyalty for 48 years had been to Philadelphia.

From 1936 until his death, Ormandy made literally hundreds of recordings with the Philadelphia Orchestra spanning almost every classical-music genre. Curiously, few of the later stereo recordings were undertaken in the orchestra's performing venue at Philadelphia's Academy of Music, because record producers believed the dry acoustic environment didn't produce optimum sound quality for records. Many Philadelphia Orchestra recordings were taped at Philadelphia's Broadwood Hotel or Town Hall, which furnished a more resonant and bright acoustic environment for impressive-sounding "high fidelity" recording techniques. Recordings were produced for the following record labels: RCA Red Seal (1936 to 1942), Columbia Masterworks (1944 to 1968), RCA Red Seal (1968 to 1977) and EMI/Angel (1977-on). A few of the very late albums were also recorded for Delos and Telarc.