Born to a Jewish family in the Bronx, New York, he studied dramatics at The New School in New York City and then served in the Army Air Corps during World War II. In 1947 he was selected by Elia Kazan and Lee Strasberg to be a player in the Actors' Studio television program and went on to appear in a number of television plays in the 1950s and returned frequently to television as a guest star on numerous dramas.

Balsam appeared in such films as On the Waterfront, 12 Angry Men (as Juror #1), Psycho, Cape Fear (1962) as the police chief, Breakfast at Tiffany's, Seven Days in May, Catch-22, Tora! Tora! Tora!, and The Delta Force.

In 1965 he won an Academy Award for best supporting actor for his role as Arnold Burns, the agent brother of the oddball non-conformist and unemployed kiddie-show writer Jason Robards, Jr. in A Thousand Clowns and starred as Murry Klein on the All in the Family spin-off Archie Bunker's Place for two seasons.

In 1967 he won a Tony Award for his appearance in the 1967 Broadway production of You Know I Can't Hear You When the Water's Running.

He married and divorced actress Joyce Van Patten. Their daughter, Talia Balsam, was born January 1, 1960.