Spencer was born in Hudson, New York, the son of Ambrose Spencer. He became secretary to Governor Daniel D. Tompkins in 1807, studied law in Albany, New York and was admitted to the bar in 1809; in the same year, he married Elizabeth Scott Smith. He moved to Canandaigua, New York, where he entered the practice of law and became a master of chancery in 1811.

He was postmaster at Canandaigua, 1814; became assistant attorney general and district attorney for the five western counties of New York, 1815; served in the United States House of Representatives, 1817-1819; was a member of the committee that reported unfavorably on the affairs of the National Bank; was nominated but defeated for the U.S. Senate; served in the New York Assembly 1820-1822; served in the New York State Senate, 1825-1828; became special prosecutor to investigate the disappearance of William Morgan, author of a manuscript on Masonic rituals, 1829; and again served in the state legislature, 1831-1833. He moved to Albany in 1837.

He edited an English edition of Alexis de Tocqueville's Democracy in America, 1838; was secretary of state of New York, 1839; served as Secretary of War, October 12, 1841-March 3, 1843; proposed a chain of posts extending from Council Bluffs, Iowa, to the Columbia River; urged that the government adhere to field commander arrangements by compensating the Creek Indians who were removed; lost his son Philip Spencer, who was executed for attempted mutiny aboard the brig USS Somers, 1842; was nominated to the Supreme Court but rejected by the U.S. Senate, 1844.

He died in Albany, New York.