Rankin was born near Bolanda in Itawamba County, Mississippi and he graduated from the University of Mississippi law school in 1910. Rankin served in the United States Army during World War I. In 1920, he was elected to the House as a Democrat in 1920. He served sixteen consecutive terms (March 4, 1921 - January 3, 1953) as Mississippi's First District Representative. As a Congressman, Rankin co-authored the bill to create the Tennessee Valley Authority and was a supporter of the Rural Electrification Administration. He was a sponser with Edith Nourse Rogers of Massachusetts, the House version of the Servicemen’s Readjustment Act of 1944 (also known as the G. I. Bill of Rights). Seen as an economic liberal, he was a strong supporter of President Franklin D. Roosevelt's New Deal.

He was considered a racial demagogue who was a supporter of segregation and white supremacy. Rankin's racism and anti-Semitism was apparent in his dealings with African Americans and Jewish members of the House. When Adam Clayton Powell Jr. was elected to Congress in 1945, Rankin vowed to never sit next to him.

John Rankin was a prominent part of the House Un-American Activities Committee (HUAC). He has been criticized for his failure to fully investigate the Ku Klux Klan. When HUAC's chief counsel Ernest Adamson announced that: "The committee has decided that it lacks sufficient data on which to base a probe," Rankin added: "After all, the KKK is an old American institution." HUAC then concentrated on investigating the possibility that the American Communist Party had infiltrated the Federal Writers' Project. He displayed his opinions about Jewish influence in America with this prominent quote without making an explicit reference to Jews, he stated that:

"They whine about discrimination. Do you know who is being discriminated against? The white Christian people of America, the ones who created this nation.... I am talking about the white Christian people of the North as well as the South.... Communism is racial. A racial minority seized control in Russia and in all her satellite countries, such as Poland, Czechoslovakia, and many other countries I could name. They have been run out of practically every country in Europe in the years gone by, and if they keep stirring race trouble in this country and trying to force their communistic program on the Christian people of America, there is no telling what will happen to them here." (Cong. Rec., April 23, 1952, p. 4320). He was also a vocal opponent of the United Nations:

"The United Nations is the greatest fraud in all History. Its purpose is to destroy the United States." Rankin ran for the United States Senate on the death of Theodore G. Bilbo, finishing last among the five major candidates with over 24,000 votes and 13f the vote.

Rankin was defeated for re-election in 1952 by Congressman Thomas G. Abernethy after their districts were joined together through Redistricting.

John Rankin died in at his home in Tupelo on November 26, 1960. He is interred in Greenwood Cemetery in West Point, Mississippi.