Rahall was born in Beckley, West Virginia. His father was the wealthy owner of many businesses, including many radio stations throughout the state. Rahall attended Duke University and received several draft deferments during the Vietnam War. Following his graduation, he attended graduate school at the George Washington University. He then went to work for Senator Robert Byrd (who is from nearby Sophia) as a staff member.

After 4th District Congressman Ken Hechler stepped down to run for governor in 1976, Rahall narrowly won a 10-way Democratic primary, which was tantamount to election in the heavily Democratic district. At the age of 27, he was one of the youngest members of Congress. He has been reelected 14 times with virtually no Republican opposition aside from 1988 and 1990. Rahall defeated Hechler with 57 percent of the vote. He has served in the House longer than anyone in West Virginia history except Harley Staggers, who represented the now-defunct 2nd District from 1949 to 1981. The district was renumbered the 3rd after the 1990 census, when West Virginia's declining population cost it a congressional seat.

He is ranking Democrat on the Resources Committee, and also serves on the Transportation and Infrastructure Committee.

In addition to having worked on every federal highway bill since coming to Congress, Rahall was a key architect in the formulation of the Transportation Equity Act for the 21st Century (known as TEA 21). The bill established the Rahall Transportation Institute (RTI), a consortium of five Southern West Virginia colleges, housed at Marshall University. Recently, Rahall helped RTI win designation as a National Maritime Enhancement Institute to enable the school to compete for federal grants related to research regarding inland maritime activities. This is one of only seven so-named universities in the nation, further advancing RTI’s mission of “Building Jobs through Transportation” for West Virginia.

He also took on the cause of greater protections for citizens of West Virginia's coalfields by twice spearheading legislation to extend the Abandoned Mine Reclamation program, which provides funds to combat health, safety and environmental threats that are connected to old mine sites. In the course of this effort, provisions were included to allow these funds to be used for public water supply systems in the region, as well as to help finance health care for retired coal miners. Rahall additionally gained enactment of provisions to provide land owners with protections from subsidence caused by underground coal mining.

In other areas related to mining and minerals, Rahall is the sponsor of legislation to promote the re-mining of lands previously mined for coal with the Coalbed Methane Development Program, the reauthorization of the Mining and Mineral Resources Research Institute program, the Rahall/Byrd amendment to the Export-Import Bank Reauthorization prohibiting the financing of foreign mining ventures, and the National Geologic Mapping Act of 1992.

Rahall is the author of the 1978 legislation which established the New River Gorge National River as a unit of the National Park System. Ten years later, he gained enactment of legislation to designate the Gauley River National Recreation Area and the Bluestone National Scenic River, in the process creating the largest network of federally protected rivers in the eastern United States. Rahall legislation also established the National Coal Heritage Area in 11 southern West Virginia counties.

Numerous State and national organizations have recognized Rahall for his work on environmental issues, he is the recipient of the 1997 Citizen’s Coal Council award, the 1996 "Keeper of the Flame Award" presented in 1996 by Interior Secretary Bruce Babbitt, the 1990 "Friend of the Earth Award" for his work on protecting the coalfield environment, the Sierra Club’s 1988 "Seneca Award" for Outstanding Environmental Stewardship, American River's 1988 "River Conservation Award,” and the Ansel Adams Award from the Wilderness Society in 2004.

Rahall has faced several primary challenges from Hechler, and in the 1980s several challenges in general elections funded in part by some Jewish-American groups that expressed concern on Rahall's somewhat pro-Palestinian leanings on Middle Eastern issues. However, he holds one of the safest Democratic districts in the country (Democrats frequently win by margins of 90-10). Speculation on his political future centers on a possible run for the Senate whenever Byrd's seat opens up and on the continued population loss of his district, which will likely result in some degree of redistricting in the state after the 2010 census.