He was born in Pontotoc, Mississippi to William Holmes Cochran and Emma Grace Cochran, a school principal and a teacher, respectively. His family settled in Hinds County, Mississippi, home of the state capital, Jackson, in 1946 after a few moves around the northern part of the state. Cochran still lives in Jackson today.

Cochran earned Eagle Scout as a youth. He graduated from Byram High School near Jackson and received a B.A. degree from the University of Mississippi with a major in psychology and a minor in political science in 1959. There he joined the Pi Kappa Alpha Fraternity. After a time in the United States Navy (1959-1961), he attended the University of Mississippi School of Law and graduated in 1965. He then practiced law for seven years.

Cochran had always been active in politics, working on several state Democratic campaigns. Cochran became a Republican in the late 1960s and served as Mississippi executive director of Richard Nixon's 1968 campaign.

In 1972, 3rd District Congressman Charles Griffin decided not to run for a third full term. Cochran won the Republican nomination for the Jackson-based district, which was renumbered the 4th District after redistricting. He defeated Democratic state senator Ellis Bodron by just under four points. A major factor in Cochran's victory was the gigantic Republican landslide in that year's presidential election, in which Nixon won 49 of 50 states and 78 percent of Mississippi's popular vote. Cochran and Trent Lott (who later served alongside him in the Senate) became the second and third Republicans to represent Mississippi in the House since Reconstruction. He was handily reelected in 1974, one of the few bright spots in a year when anger over Watergate caused several Republicans to lose their seats. He was reelected by an even larger margin in 1976.

In 1978, Senate Judiciary Committee chairman and longtime segregationist firebrand James Eastland retired after six terms in the Senate. Cochran defeated former state senator and future circuit court judge Charles Pickering, Sr. for the Republican nomination. The independent candidacy of Charles Evers, mayor of Fayette and brother of slain civil rights worker Medgar Evers, siphoned off enough black votes from Democrat Maurice Dantin to allow Cochran to become the first Republican to win a statewide election in Mississippi since Reconstruction. When Eastland resigned two days after Christmas, Cochran was appointed to the seat by governor Cliff Finch and started his Senate career a week early. He handily defeated Governor William Winter in 1984, was unopposed in 1990, reelected with over 70 percent of the vote in 1996 and faced no major-party opposition in 2002.

Cochran's voting record has been fairly moderate by Southern Republican standards, and in many ways is more moderate than that of Eastland, despite Cochran being a Republican. He has maintained a very low profile, at least compared to Eastland and his colleague Lott, who was elected to the Senate in 1988 when longtime incumbent John C. Stennis retired.

However, Cochran has considerable influence behind the scenes, especially in Mississippi. This is not surprising given his status as the first Republican to hold statewide office since Reconstruction. In April 2006, Cochran was selected by Time as one of "America's 10 Best Senators"; the magazine quoted an unnamed "senior G.O.P. Senator" who said "He doesn't get a whole lot of play in terms of coverage, but he is effectively stubborn doing what needs to be done."

Cochran served as chairman of the Senate Republican Conference (caucus) from 1991 to 1996 and chaired the Senate Agriculture Committee from 2003 to 2005. In 2005, he was appointed as chairman of the powerful Senate Appropriations Committee, making him the first Republican from a former Confederate state to chair the committee.

It appears that recognition from his colleagues was quick in coming: In 2005, an agricultural appropriations bill proposed by the Committee Cochran chaired contained a provision (sec. 782) that said: