Chafee was born in Providence, Rhode Island to a politically active family. His great-grandfather, Henry Lippitt, was a Rhode Island governor and among his great-uncles were a Rhode Island governor, Charles Warren Lippitt, and United States senator Henry Frederick Lippitt. His uncle, Zechariah Chafee, was a Harvard law professor, and a notable civil libertarian.

In 1940, he graduated from Deerfield Academy in Massachusetts. Chafee served in the Marines during World War II, spending his 20th birthday on Guadalcanal. He received degrees from Yale University in 1947 and Harvard University law school in 1950. In 1951, he was recalled to active service to be a Marine rifle company commander during the Korean War.

Chafee became active in behind-the-scenes Rhode Island politics by helping elect a mayor of Providence in the early 1950s. He successfully ran for a seat in the Rhode Island House of Representatives in 1956 and later became the minority leader. He was reelected in 1958 and 1960, the latter a year when many Republicans were swept from office in his state.

Chafee was elected governor in 1962, helping create the state's public transportation administration as well as what was known as the Green Acres program, a conservation effort. Chafee was head of the Republican Governors Association in the late 1960s. In 1968 he serived as chair of the Republican Governors Association. He served as governor until 1969.

He was appointed Secretary of the Navy in 1969. His tenure as secretary was marked by a willingness to make bold decisions and stand by them. Emblematic of this was his decision to elevate Adm. Elmo Zumwalt as Chief of Naval Operations over 33 more senior officers, and his judicious handling of the USS Pueblo situation. He served as Secretary of the Navy until 1972.

After an unsuccessful candidacy for the Senate in 1972, Chafee was elected to that body in 1976, serving as chairman of the Senate Committee on Environment and Public Works. He first joined the committee in 1977 and made environmental matters a chief concern, often breaking with his party to the delight of conservation groups.

Among the bills Chafee fostered while in the minority was the Clean Water Act of 1986, and the 1990 amendments to the Clean Air Act. He also was an architect of the Superfund program in 1980 to clean up hazardous waste sites as well as the Oil Pollution Act of 1990.

Frequently following a moderate path, Chafee was pro-choice on abortion and supported the North American Free Trade Agreement. He took a moderate stance on taxes and government assistance to the needy. On social issues, Chafee was among the most liberal members of the Senate. He opposed the death penalty, school prayer, and the ban on gays serving in the military. Chafee was one of the few Republicans to support strict gun control laws. He sponsored a bill that, if passed, would have prohibited the "manufacture, importation, exportation, sale, purchase, transfer, receipt, possession, or transportation of handguns and hand ammunition."

"John Chafee proved that politics can be an honorable profession," President Bill Clinton said in a statement to the Associated Press, shortly after Chafee died. "He embodied the decent center which has carried America from triumph to triumph for over 200 years."

Chafee sat on the U.S. Senate Select Committee on Intelligence and was chairman of the Senate Finance Committee's Subcommittee on Health Care, but his biggest imprint was on environmental concerns.

His last major act was authoring and sponsoring the Transportation Equity Act for the 21st Century, which authorized funding for transportation programs for the next six years.

A few months after declaring that he would not seek reelection in 2000, he died suddenly from congestive heart failure in October 1999 at the National Naval Medical Center in Bethesda, Maryland. His son, Lincoln Chafee was appointed to serve the remainder of his term.

The USS Chafee (DDG-90) and the John H. Chafee Blackstone River Valley National Heritage Corridor was named in his honor.