King was born in Scarborough, which was then a part of Massachusetts but is now in the state of Maine. Rufus attended Harvard, but his studies were interrupted by the Revolutionary War. He fought in the Battle of Lexington and Concord and then participated in the Siege of Boston. He returned to Harvard after the British withdrew and completed his studies in 1777. He was admitted to the bar, and began a legal practice in Newburyport, Massachusetts. King was first elected to the Massachusetts state assembly in 1783, and returned there each year until 1785. Massachusetts sent him to the Continental Congress under the Articles of Confederation from 1784 to 1787. King opposed the expansion of slavery into the Northwest Territory and proposed that slavery be prohibited in that area. This prohibition became law in the Northwest Ordinance of 1787.

Also in 1787 King was sent to the Constitutional Convention, where he worked closely with Alexander Hamilton on the Committee of Style and Arrangement to prepare the final draft. He returned home and went to work to get the Constitution ratified and to position himself to be named to the U.S. Senate. He was only partially successful. Massachusetts ratified the Constitution, but his efforts to influence the state legislature and get to the Senate were rebuffed.

At Hamilton's urging he moved to New York City and was elected to the New York state legislature in 1788. When the U.S. Constitution took effect, the state senate and house agreed to send Philip Schuyler to the U. S. Senate, but they disagreed on who should serve in the second seat. After several shifts, the upstate and New York City groups were still deadlocked. Governor George Clinton proposed Rufus King as a compromise candidate, and he was elected. He represented New York in the Senate from 1789 to 1796 and from 1813 to 1825.

King also served as the United States ambassador to the United Kingdom from 1796 to 1803 and in 1825 to 1826. He was the Federalist Party candidate for Vice President of the United States in 1804 and 1808 and for President of the United States in 1816.

Many of King's family were also involved in politics and he had a number of prominent descendants. His brother Cyrus King was a U. S. Congressman. In 1786 he had married Mary Alsop, the daughter of Congressman John Alsop, and their sons John Alsop King and James Gore King also went on to serve in the Congress. Another son, Charles King, was also President of Columbia College. From his son John Alsop King descended the actress Jane Wyatt, cinematographer Floyd Crosby, and his son, musician David Crosby. From his son Charles King descended Rufus King, who was a delegate to the Wisconsin constitutional convention, served as a Union Brigadier General during the Civil War and was the American Minister to the Papal States, and Admiral William Halsey, Jr. From his son James Gore King descended the wives of the communications executive and philanthropist Clarence Hungerford Mackay, the composer Irving Berlin, and the lawyer and diplomat John Chandler Bancroft Davis.

King died in 1827 at his farm in Jamaica, Queens and is buried in the Grace Church Cemetery in Jamaica, Queens, New York. The home that King built in 1806 and some of his farm make up King Park in Queens. The home, called King Manor, is now a museum and is open to the public.

The Rufus King School, also known as P.S. 26, in Fresh Meadows, New York, was named after Rufus King.