Born in Brooklyn, New York, Mostel was a member of Civilian Conservation Corps and served briefly in the U. S. Army during World War II. Due to his leftist political views, Mostel was called to testify before the House Un-American Activities Committee on October 14, 1955. He denied that he was a member of the Communist Party, but refused to cooperate with the committee's investigation. Afterward, he was essentially banned from film roles for several years. During this period, his longtime friend (and fellow HUAC target) Burgess Meredith helped him land some Off-Broadway theater work to keep his career afloat. He later drew upon his experience with HUAC and the blacklist for his role as Hecky Brown in the film The Front.

In 1962, his career recovered with his Tony Award-winning performance as Pseudolus in the musical A Funny Thing Happened on the Way to the Forum, a role he reprised in the 1966 film version. In 1964, he originated the role of Tevye in the Broadway musical Fiddler on the Roof, for which he won a Tony.

Mostel won an Obie Award for his 1958 role as Leopold Bloom in Ulysses in Nighttown, a stage adaptation by Marjorie Barkentin of James Joyce's novel Ulysses. He reprised his role in 1974. In 1961, Mostel starred as Jean in Eugene Ionesco's absurdist play Rhinoceros. His most famous film role was as Max Bialystock in Mel Brooks' The Producers (1968) (coincidentally, the co-protagonist of The Producers is named Leo Bloom).

He was nominated for a BAFTA Award for Best Supporting Actor for his performance in The Front, a 1976 film costarring Woody Allen that presented one perspective of the blacklisting period in Hollywood.

In 1977, Zero was cast as Shylock in Arnold Wesker's The Merchant, a pro-Jewish reimagining of William Shakespeare's The Merchant of Venice. He suddenly fell ill after a tryout performance in Philadelphia, PA, and died of an aortic aneurysm on September 8. Mostel was 62 years of age.

His last film appearance was as the voice of Kehaar in the animated adaptation of Watership Down. Sadly, Mostel has the distinction of being the only guest on The Muppet Show to die before his episode aired.

He married Kathryn Harkin on July 2, 1944, and they had two sons: Josh Mostel, also a film actor, born 1946, and Tobias, born 1949.