Although born in Poulan, Georgia, Heflin was the nephew of prominent Alabama politician James Thomas Heflin, and grew up in Heflin, Alabama. Following graduation from high school, Heflin attended Birmingham-Southern College (graduated 1942) and the University of Alabama Law School (graduated 1948). During this time he also served as a U.S. Marine officer in World War II and was awarded the Silver Star for combat, seeing action on Bougainville and Guam.

He became a law professor, and then became the Chief Justice of the Alabama Supreme Court from 1971 to 1977.

In 1978, Heflin was elected as a Democrat to the United States Senate to succeed John Sparkman. He remained in the Senate, where he rose to become Chairman of the Select Committee on Ethics, until January 3, 1997. While on the Ethics Committee, he led the prosecution against fellow Senator Howard Cannon (D-NV) for violations of Senate rules.

Heflin was one of the last conservative Southern Democrats to serve in the U.S. Senate. His stances on cultural issues most often reflected the region he was from. He strongly opposed legal abortion and all gun control laws. Heflin supported prayer in public schools and opposed extending federal laws against discrimination to homosexuals. He voted in favor of the Gulf War and against limiting spending on defense. He was one of only two Democrats in the Senate to vote against the Family and Medical Leave Act. He occasionally voted with Republicans on taxes. On other economic issues he was more in sync with the populist wing of his party. He voted against the North American Free Trade Agreement (NAFTA), General Agreement on Tariffs and Trade (GATT), and lawsuit reform measures. He strongly supported affirmative action laws.

During his tenure, Heflin was considered to have bi-partisan support if he was nominated for a vacancy on the United States Supreme Court by President Ronald Reagan. Nevertheless, Heflin did not wish to serve on the highest court in the United States.