Born to Italian immigrants, Antonio Gazzara and Angela Consumano, Gazzara grew up on New York's tough Lower East Side. He found relief from his bleak surroundings by joining a theater company at a very young age. Years later, he said that the discovery of his love for acting saved him from a life of crime during his teenage years. Despite his obvious talent, he went to City College of New York to study electrical engineering. After two years, he gave it up, and after a short intermission joined the Actor's Studio.

In the 1950s, he starred in various Broadway productions, most notably Tennessee Williams' Cat On A Hot Tin Roof, directed by Elia Kazan. However, he lost out on the film role to Paul Newman. As a young actor, Gazzara joined other Actors Studio members in the 1957 film, The Strange One.

He has had a long and varied acting career, with spells as an accomplished director too (TV mostly). His most popular acting roles include Anatomy of a Murder (1959), A Rage to Live (1965), The Bridge at Remagen (1969), Capone (1975), Voyage of the Damned (1976), and High Velocity (1977). He also starred in a successful television series Run for Your Life from 1965 to 1968.

His most formidable appearances, however, were characters he created for his friend John Cassavetes in the 1970s. They collaborated for the first time on Cassavetes' film Husbands (1970) where he appeared alongside Peter Falk and Cassavetes himself. The collaboration of the two men achieved its peak in The Killing of a Chinese Bookie in which Gazzara took the leading role of the hapless strip joint owner, Cosmo Vitelli. In order to pay off a gambling debt to the mob, Vitelli agrees to kill a Chinese unknown to him. Against all odds, he succeeds in killing the man, but he gets severely wounded during his flight. But the gangsters turn against him, as they had not expected him to survive the assassination and Vitelli is forced to kill these men too. The plot itself hardly describes the true meaning of the movie, as John Cassavetes did everything to keep it from turning into an ordinary genre flick. Gazzara delivered a life-like portrayal of a simple man who found his happiness in running a third-rate strip bar, and who gets caught in something that is much too big for him. Sometimes he does not even seem to understand the whole meaning of it. The little emotional involvement that Gazzara's character shows during the events is played with stunning accuracy, with Gazzara's performance and Cassavetes' direction complementing each other. A year later Gazzara starred in yet another Cassavetes-directed movie, Opening Night, playing the role of stage director Manny Victor who struggles with the mentally unstable star of his show, played by Cassavetes' wife Gena Rowlands.

In the 1980s, he could be seen in a variety of different movies, such as They All Laughed (directed by Peter Bogdanovich) or Quicker Than the Eye (1989). He also appeared in the critically acclaimed AIDS-themed TV movie An Early Frost (1985), which also starred Gena Rowlands.

In the 1990's, he appeared in 38 films, among these many TV productions. In Hollywood movies he mostly appeared as a supporting actor, but worked with several renowned directors, such as the Coen Brothers (The Big Lebowski), Spike Lee (Summer of Sam), and John McTiernan (The Thomas Crown Affair).

Now in his seventies, Gazzara is still acting. In 2003, he appeared in the film Dogville, directed by Danish enfant terrible Lars von Trier, alongside Nicole Kidman.