In 1908, while a member of the U.S. House of Representatives, he shot and seriously wounded a black man who confronted him on a Washington streetcar. Although indicted, Heflin had the charges dismissed. In subsequent campaigns, he bragged of the shooting as one of his major career accomplishments.

He continued to serve in the House until 1920, when he was elected to the Senate to fill the vacancy caused by the death of John Hollis Bankhead. In 1928, Heflin supported Herbert Hoover for President. Because of this apostasy, Heflin was denied renomination as a Democrat in 1930. This forced him to run for the Senate as an Independent candidate and he lost decisively to John Hollis Bankhead II. Returning to Washington to serve out his term, he initiated a Senate investigation of voting fraud in hopes of overturning Bankhead's election. The inquiry lasted fifteen months and cost $100,000.

In April 1932, with Heflin's term expired and Bankhead seated, the Senate prepared to vote on a committee recommendation against Heflin. Heflin, face crimson, delivered a five-hour oration, punctuating his remarks with vehement gestures and racist jokes. As he thundered to a conclusion, the gallery audience, packed with his supporters, jumped to its feet with a roar of approval and was immediately ordered out of the chamber. Two days later, the Senate voted by a wide margin to dismiss Heflin's claim.

After his defeat, Heflin was an unsuccessful candidate for election to the House and Senate on several occasions, and later would serve as an appointed special representative of the Federal Housing Administration under Franklin Roosevelt. He died in 1951 in Lafayette.

His nephew, Howell Heflin, would also become a Senator from Alabama and was Chief Justice of the Alabama Supreme Court; and known by all for his reasoned fairness to persons of all races (Chairman of Senate Ethics committee in late 1990's with no issue of racism ever during his long career). See his page.