Abernathy was born in Linden, Alabama, on March 11, 1926. The son of a farmer, he served in the army during World War II and afterward enrolled at Alabama State University, in Montgomery, Alabama, graduating with a degree in mathematics in 1950. His involvement in political activism began in college while he was a member of Kappa Alpha Psi Fraternity, when he led demonstrations protesting the lack of heat and hot water in his dormitory and the dreadful food served in the cafeteria. In 1951 he earned an M.A. in sociology from Atlanta University (later Clark Atlanta University) and then became pastor of the First Baptist Church in Montgomery, Alabama. While living in Montgomery he formed a close and enduring partnership with Dr. Martin Luther King.

In 1955, when Rosa Parks refused to move to the back of the bus, Abernathy and King organized the bus boycott in Montgomery. After a year of the boycott, it finally ended when the United States Supreme Court affirmed the U.S. District Court's ruling that segregation on buses was unconstitutional.

Abernathy was Martin Luther King's Number Two in the Southern Christian Leadership Conference, holding the official title of Secretary-Treasurer. Abernathy was with Martin Luther King in Memphis when King was assassinated; in fact, they shared Room 307 at the Lorraine Motel the night before. Abernathy assumed the presidency of the SCLC after King's death. Less than a week after the assassination, Abernathy led a march to support striking sanitation workers in Memphis, Tennessee. In May 1968 he organized the Poor People's Campaign March on Washington, D.C. Hoping to bring attention to the plight of the nation's impoverished, he constructed huts in the nation's capital, precipitating a showdown with the police. Ignoring orders to remove the huts, he was jailed for nearly three weeks.

Abernathy resigned in 1977 amid accusations of financial mismanagement. That same year he ran for a Georgia congressional seat and lost.

Although some people believe the American Freedom Coalition, which Abernathy later founded with Robert Grant, was controlled by Rev. Sun Myung Moon, the Coalition has denied those allegations. American Freedom Coalition acknowledges that it receives major funding from Moon's Unification Church and that it works in partnership with The Washington Times, CAUSA, the American Constitution Committee, and other Moon controlled organizations. Abernathy served as Vice President of American Freedom Coalition until his death in 1990.

His youngest daughter is actress and writer Donzaleigh Abernathy.

In 1989, his autobiography, And the Walls Came Tumbling Down, was published. But before the publication, Abernathy faced strong condemnation from civil rights leaders for revealing Martin Luther King Jr.'s private infidelity problems. As a result, he had to spend the rest of his life, convincing his colleagues that he had not 'sold out' his long lost friend's legacy. Abernathy died on April 17, 1990.

Abernathy received many awards, most notably honorary degrees from Long Island University in New York, Morehouse College in Atlanta, Kalamazoo College in Michigan, and his alma mater, Alabama State University. Interstate 20, in Atlanta, is named in his honor.