Agee was born at 15th and Highland Streets in Knoxville, Tennessee. When Agee was six, his father died in an automobile accident, and from the age of seven he was sent boarding schools, where he felt isolated from, and abandoned by his mother. He attended St. Andrews-Sewanee School (then an Episcopal monastery run by the monks), and Phillips Exeter Academy, (class of 1928), where he edited the Monthly and was president of The Lantern Club, (though barely passing many of his courses), before going on Harvard University (class of 1932) where he was president of the Harvard Advocate and delivered the class ode at commencement.

After graduation, he wrote for Fortune and Time magazines. (He is better known, however, for his later film criticism in The Nation.) He married Via Saunders on January 28, 1933; they divorced in 1938 and that same year he married Alma Mailman. In 1934, he published his first volume of poetry, Permit Me Voyage, with a foreword by Archibald MacLeish.

In the summer of 1936, Agee spent eight weeks on assignment with photographer Walker Evans living among sharecroppers in Alabama. While Fortune didn't publish his article (he left the magazine in 1939), he turned the material into a book entitled, "Let Us Now Praise Famous Men" (1941). It sold only 600 copies before being remaindered. That same year, Alma moved to Mexico with their year-old son, Joel, to live with Communist writer Bodo Uhse, while Agee began living with Mia Fritsch in Greenwich Village. (They were married in 1944, and had two daughters, Teresa, and Andrea, and a son John, who was only eight months old when Agee died).

In 1942, Agee became the film critic for Time, while also writing occasional book reviews, and subsequently becoming the film critic for The Nation. In 1948, however, he quit both magazines to become a freelance writer. As a freelance in the 1950's, he continued to write magazine articles while working on movie scripts (often with photographer Helen Levitt).

In 1951 in Santa Barbara, Agee suffered the first two in a series of heart attacks, which ultimately claimed his life four years later at the age of 45. He died on May 16, 1955 (while in a taxi cab enroute to a doctor's appoinment) -- ironically on the same month and day on which his father had died.

His considerable, if erratic, career as a movie scriptwriter was curtailed by alcoholism, and his contribution to The Night of the Hunter (1955) remains unclear. What is certain, however, is that Agee is one of the credited screenwriters on two of the great films of the 1950s.

During his lifetime, Agee enjoyed only modest public recognition, but since his death his literary reputation has grown enormously. In 1957 Agee's novel, "A Death in the Family" (which was based on the events surrounding his father's death), was published posthumously. It won a Pultizer Prize in 1958. In addition, Let Us Now Praise Famous Men (1941) has since been placed among the greatest literary works of the 20th Century by the New York School of Journalism and the New York Public Library.