He was born in Roane County, Tennessee, and graduated from Mayo College (now Texas A&M University-Commerce), at Commerce, Texas. After a year teaching school, he won election to the Texas State Legislature. During his third two-year term in the Legislature, he was elected Speaker of the House at the age of 29. The next year, he won election to the United States House of Representatives in District 4. He went to Congress on 4 March 1913 at the beginning of Woodrow Wilson's administration and served as a Democrat without interruption for over 48 years. During this entire time, he never faced a Republican or other serious challenger for reelection. On 16 September 1940, at the age of 58, he became Speaker of the United States House of Representatives. When his career as Speaker was interrupted during the sessions of 1947-1948 and 1953-1954 during Republican control of the House, Rayburn served as minority leader. He usually worked quietly in the background in the shaping of legislation. As Speaker, he won a reputation for being fair in his rulings and for forgetting politics when he handled the gavel. He had served as Speaker more than twice as long as any predecessor when he died of cancer in Bonham, Texas. His record for total service as Speaker has not yet been broken although Tip O'Neill surpassed Rayburn's record for continuous amount of time as Speaker having served six uninterrupted terms to Rayburn's five. Rayburn was legendary for his honesty and integrity. He despised lobbyists and wouldn't accept anything from them. He would only say "I am not for sale" and walk away. In his years in Congress, Rayburn always insisted on paying his own expenses. When he died, his personal savings only totalled $10,000 and most of his holdings were in his family ranch.

Rayburn grew up in abject poverty, and would champion the interests of the poor once in office. He was a close friend of Lyndon B. Johnson, and was instrumental in LBJ's ascent to power, particularly his unusually rapid rise to the position of House Minority Leader after only four years in that body, and his subsequent elevation to Majority Leader once the Democrats regained control of Congress. In fact, Rayburn knew Johnson's father, Sam Ealy Johnson, from their days in the Texas State Legislature.

Rayburn, though a menacing and powerful presence on the House floor, was incredibly shy outside of work. He had married once, to Metze Jones, sister of Texas Congressman Marvin Jones and Rayburn's colleague, but the marriage ended quickly and no one really ever knew why. Biographer D.B. Hardeman guessed that Rayburn's work schedule and long bachelorhood, combined with the couple's differing views on alcohol contributed to the rift. The court's divorce file in Bonham, Texas, could never be located, and Rayburn avoided speaking of his brief marriage. One of his greatest, most painful regrets was that he did not have a son, or as he put it in Robert Caro's biography of Lyndon B. Johnson, "a towheaded boy to take fishing."

Rayburn coined the term "Sun Belt" while strongly supporting the construction of Route 66. It originally ran south from Chicago, through Oklahoma, and then turned westward from Texas to New Mexico, Arizona and California before ending at the beach in Santa Monica, California. During one of his many arguments for the project he stated famously that America absolutely must connect the "Frost Belt with the Sun Belt."

The phrase, "a jackass can kick a barn down, but it takes a carpenter to build one," is attributed to Rayburn.

Rayburn died of cancer in 1961 at the age of 79, and shortly afterwards was given the Congressional Gold Medal.

The ballistic missile submarine USS Sam Rayburn (SSBN-635) was named in his honor, as was the Rayburn House Office Building, a building of offices for Representatives next to the United States Capitol. The Sam Rayburn Reservoir in East Texas was named after him in 1963. Sam Rayburn High School in Pasadena, TX also bears his name and houses the desk he used as Speaker of the House.