Gordon was born in Murfreesboro, Tennessee, where he has lived all of his life. He graduated from Middle Tennessee State University in 1971, earning a law degree from the University of Tennessee in 1973. He then entered private practice in Murfreesboro.

Active in Democratic politics early on, he was briefly executive director of the state Democratic Party in 1979 and state party chairman in from 1981 to 1983. When Sixth District Congressman Al Gore announced in 1983 that he would run for the United States Senate in 1984, Gordon stepped down as state party chairman to run for the seat. He initially faced a hard-fought race against the brother of the publisher of Nashville's former conservative newspaper, the Nashville Banner. However, he won handily in November 1984, riding Gore's coattails in the midst of Ronald Reagan's landslide victory in that year's presidential election. Gordon is regarded as a moderate. He has favored the repeal of the inheritance tax and the "marriage tax penalty".

Gordon was reelected by huge margins until 1994, when his Republican opponent was attorney Steve Gill, a former basketball player at Tennessee who is now a radio talk show host. Gordon only won by one percentage point, but managed to defeat Gill more handily in 1996. Gordon was re-elected in 1998 and 2000 by margins similar to those he scored in the 80s and early 90s. He faced no significant opposition in 2002 and 2004, in large part because the 2002 reapportionment by the Democratic-controlled Tennessee General Assembly removed Williamson County, a wealthy and heavily Republican suburban area south of Nashville, from the Sixth Congressional District and added it to the already heavily Republican Seventh District. Ironically, that district is now represented by Marsha Blackburn, a Williamson County resident who was Gordon's opponent in 1992--the only time besides Gill's two bids that Gordon has faced a well-financed Republican opponent.

Gordon will likely hold onto the seat for as long as he wants it, but the 6th has become increasingly friendly to Republicans in the last decade as Nashville's suburbs have spread further into this once-rural district. It has voted for the Republican candidate in the last three presidential elections, even when Gore ran as vice president in 1996 and as the Democratic presidential candidate in 2000.