Born John Lebzelter in Newark, New Jersey, and was raised in Louisville, Kentucky, his early jobs included lifeguard and nightclub bouncer. He also fought as a professional boxer under the name Johnny Costello. He served in the 101st Airborne Division during World War II.

Warden decided to pursue an acting career after leaving the military, and moved to New York City. He joined the company of the Dallas Alley Theater and performed on stage for five years. He made his television debut in 1948 in The Philco Television Playhouse and Studio One. He had an uncredited film debut in 1951 in You're in the Navy Now, a movie which also featured the debuts of Lee Marvin and Charles Bronson.

Warden had his first credited film role in The Man with My Face in 1951. In 1952, he began a three year role in the television series Mr. Peepers. Warden's breakthrough role was his appearance in 12 Angry Men in 1957.

Warden has appeared in over one hundred movies during a career which is now in its sixth decade. Warden received an Emmy Award for his performance as George Halas in Brian's Song (1971), and was nominated for Academy Awards as Best Supporting Actor for his performances in Shampoo (1975) and Heaven Can Wait (1978). He also appeared in Problem Child (1990) and its sequel (1991).