Born in White Plains, New York, Whitmore graduated from Yale University, where he was a member of Skull and Bones, and served in the United States Marine Corps during World War II.

Following the war, Whitmore appeared on Broadway, where he won a special Tony Award in 1948. His first major movie was Battleground, for which he was nominated for the Academy Award for Best Supporting Actor. Other major films included The Asphalt Jungle, The Next Voice You Hear, Kiss Me, Kate, Them!, Black Like Me and Give 'em Hell, Harry!, for which he was nominated for the Academy Award for Best Actor for his portrayal of former President of the United States Harry S. Truman.

James Whitmore also appeared as General Oliver O. Howard in the 1975 TV movie I Will Fight No More Forever, based on the 1877 conflict between the United States and the Nez Percé tribe, led by Chief Joseph.

Whitmore's last major role was that of librarian Brooks Hatlen in the critically-acclaimed and Academy award-nominated 1994 film The Shawshank Redemption.

He is the father of actor James Whitmore Jr. and the grandfather of actor James Whitmore III.

To a younger generation, he is probably best known, in addition to his role in Shawshank, as the commercial spokesman for Miracle-Gro plant food for many years.

Whitmore has a star on the Hollywood Walk of Fame at 6611 Hollywood Blvd.

In 2002, Whitmore got the role of the Grandfather in the Disney Channel original movie A Ring of Endless Light.