Rhodes was born in Council Grove, Kansas. He met Calvin Coolidge when he was eleven years old, and after shaking hands with the President, he reportedly refused to wash his hand for a week. He attended public schools, and in 1938 graduated from Kansas State University, in Manhattan, Kansas. In 1941, he graduated from Harvard Law School, in Cambridge, Massachusetts, and became a lawyer with a private practice. From 1941 to 1946 he was in the United States Army Air Corps, and serving at Williams Field, Arizona, he chose to relocate there with his new wife, Elizabeth Harvey Rhodes in 1946. From 1947 to 1952 he was the staff advocate of the Arizona National Guard, and from 1951 to 1952 he was the vice chairman of the Arizona Board of Public Welfare.

Despite not "wanting" the post, John ran for Attorney General of Arizona in the 1950 election as a Republican. True to his mentor and friend, Barry Goldwater's prediction, he lost; Arizona was over 75 emocrat at the time. In 1952, 1954, and 1968 he was a delegate to the Republican National Conventions. In 1952 John ran again, this time for the U.S. House of Representatives. He won, despite a shoe-string budget, by 8f the vote, and was elected to the Eighty-third United States Congress.

John would stay in office for the next 30 years, from January 3, 1953 to January 3, 1983, serving in the 83rd to 96th Congresses. He spent seven years as House Minority Leader, from the 93rd to 96th Congresses.

During his stay in office, John played a significant role in countless initiatives. He will most likely be remembered for two things, only the first of which was he genuinely proud of. He became the driving force behind the Central Arizona Project, a project which brought desperately needed water to the cities and towns of Arizona. His name will also be remembered for when he met with President Richard M. Nixon in August, 1974 along with two other men, and informed the President that the Republican Party would not support him through an impeachment. Days later, President Nixon resigned.

After his time on Capitol Hill, John spent most of his remaining days in Mesa, Arizona. On August 14, 2003, Speaker Dennis Hastert made the trip to Arizona to personally award John with the Congressional Distinguished Service Medal, one of only a handful awarded. Rhodes remarked to his friend Hastert that he had the only job Rhodes had ever really wanted. He died only days later after a long fight with cancer, with his wife and most of his children with him. He had been visited by most of his numerous grand and great-grandchildren in the week leading to his death.

Following his death, over 100 different newspapers carried his obituary, and President George W. Bush also delivered a statement at the White House's website.