Talmadge was born in Telfair County, Georgia, the only son of Eugene Talmadge, who served as Governor of Georgia during much of the 1930s and '40s. He earned a law degree from the University of Georgia in 1936, where he had been a member of the Demosthenian Literary Society and Sigma Nu fraternity.

The younger Talmadge saw combat in the United States Navy during World War II. On his return from the South Pacific as a lieutenant commander, Herman ran his father's successful campaign for governor in 1946. Supporters of Eugene Talmadge were unsure of Eugene's chances of surviving until he was sworn in, so they did some research into the state constitution and found that if Eugene died, the Georgia state legislature would choose between the second and third place finishers. Talmadge was running unopposed, so they got write-in votes for Herman as insurance. And in December 1946, Eugene Talmadge died.

The Talmadge supporters stopped a challenge from the new Lieutenant Governor, Melvin E. Thompson. Thompson claimed that he should succeed Talmadge. The Georgia legislature elected Herman to become Governor. Thompson appealed to the State supreme court. Meanwhile, Governor Ellis Arnall refused to turn over power due to the uncertainty of who the next Governor would be. So on January 15th, 1947, both men served in the state Capitol as Governor. The next day, Talmadge took control of the Governor's office and had the locks changed. Arnall soon gave up his claim and supported Thompson's claim.

But soon afterwards, the Georgia Supreme Court ruled that the legislature had violated the State Constitution by electing Herman governor and that Thompson was the legitimate Governor of Georgia. Talmadge soon gave in to the court decision and prepared for the special election in 1948.

Herman defeated Governor Thompson in 1948 and he was elected to a full term in 1950. Talmadge encouraged industry to move into Georgia and he was a staunch supporter of segregation.

Talmadge was unable to seek another full term as Governor in 1954. Instead, he chose to run for the United States Senate in 1956. Talmadge remained a foe of civil rights legislation as a Senator. He also sponsored bills to help farmers, an important constituency. He served on the Senate Watergate Committee.

On 11 October 1979, he was censured by the Senate by a vote of 81-15 for "improper financial conduct" between 1973 and 1978, for accepting reimbursements of $43,435.83 for official expenses not incurred, and for improper reporting of campaign expenditures.

He also went though a divorce and a tough primary challenge from Zell Miller in 1980. Talmadge defeated Miller but lost to Mack Mattingly in the general election, making Mattingly the first Republican to represent Georgia in the Senate since Reconstruction.

After his defeat, Talmadge retired to his home and died there at the age of 88.