He studied under private tutors, at private schools, the College of New Jersey, and Columbia College, New York City. He studied law in Philadelphia, Pennsylvania, but never practiced. At an unusually young age Randolph was elected to the Sixth and to the six succeeding Congresses (1799 to 1813). Federalist William Plumer wrote in 1803 of his striking presence:

Mr. Randolph goes to the House booted and spurred, with his whip in hand, in imitation, it is said, of members of the British Parliament. He is a very slight man but of the common stature. At a little distance, he does not appear older than you are; but, upon a nearer approach, you perceive his wrinkles and grey hairs. He is, I believe, about thirty. He is a descendant in the right line from the celebrated Indian Princess, Pochahontas. The Federalists ridicule and affect to despise him; but a despised foe often proves a dangerous enemy. His talents are certainly far above mediocrity. As a popular speaker, he is not inferior to any man in the House. I admire his ingenuity and address; but I dislike his politics.

Randolph was chairman of the Committee on Ways and Means in the Seventh through the Ninth Congresses, acting as the Republican party leader. After breaking with President Thomas Jefferson in 1806, he founded the faction Tertium quids that called on Republicans to return to the principles of 1798 and renounce what they saw as creeping Federalism and nationalism. Although he greatly admired the political ideals of the previous Revolutionary generation, Randolph, influenced by Southern anti-Federalism, propounded a version of republicanism that relied on the traditional patriarchal society of Virginia's elite gentry to preserve social stability with minimal government interference. Randolph was one of the Congressional managers who conducted the successful impeachment proceedings against John Pickering, judge of the United States District Court for New Hampshire, in January 1804. But critics complained that he mismanaged the failed effort in December of the same year against Samuel Chase, Associate Justice of the Supreme Court of the United States. He was defeated for reelection in 1812, but elected in 1814 and 1816, skipped a term, and served from 1819 until his resignation in 1825. Randolph was appointed to the Senate in December 1825, to fill a vacancy and served until 1827. Randolph was elected to the Congress in 1826, chairing the Committee on Ways and Means.

Randolph was a member of the Virginia constitutional convention at Richmond in 1829. He was appointed United States Minister to Russia by President Andrew Jackson and served from May to September, 1830, when he resigned for health reasons. Elected again in 1832 he served until his death in Philadelphia on May 24, 1833. He is buried at Hollywood Cemetery in Richmond, Virginia.

John Greenleaf Whittier's poem "Randolph of Roanoke," although written after the Virginian had become a symbol of the "slave power," capture his strange brilliance:

Mirth, sparkling like a diamond shower, From lips of life-long sadness; Clear picturings of majestic thought Upon a ground of madness While others hailed in distant skies Our eagle's dusky pinion, He only saw the mountain bird Stoop o'er his Old Dominion! All parties feared him; each in turn Beheld its schemes disjointed, At right or left his fatal glance And spectral finger pointed A modern conservative political group, the John Randolph Club, is named after Randolph.