John Rutherfurd was born in New York City. His parents were Walter and Catherine Rutherfurd. Walter was a veteran of the British Army, and was a hostage of Patriots during the Revolutionary War while John was a teenager. Walter died in 1804.

John attended the College of New Jersey (now Princeton University) and studied law. He practiced law in New York City for several years, and then moved to a farm in Warren County, New Jersey in 1787. He entered politics, serving in the New Jersey State Assembly from 1788 to 1790. He was then elected as a Federalist to the United States Senate from New Jersey and served in the Senate from 1791 to 1798. He was reelected in 1796 but resigned in December 1798, for reasons that are unclear.

Rutherfurd then retired from politics, but he was to undertake several important projects during the rest of his life. He laid out the plans for a new section of New York City north of 14th street from 1807 to 1811. Around 1816 he investigated the building of a possible canal connecting the Delaware, Raritan and Hudson rivers. Finally, from 1827 to 1833, he helped settle New Jersey's boundaries with New York and Pennsylvania.

In 1808, Rutherfurd moved with his family to a farm on the banks of the Passaic River near what is now Rutherford, New Jersey. He lived at this place for the rest of his life, naming it "Edgerston", and died there. Rutherfurd had a large family. He was married to Helena Morris Rutherfurd from 1782 until his death in 1840, and Helena died shortly after him. They had eight children.

One of John Rutherfurd's sons, Robert Walter Rutherfurd, was a member of the New Jersey State Legislature. One of Robert's sons, and John's grandson, was the astronomer Lewis Morris Rutherfurd.

The town of Rutherford, New Jersey was named at least in part after John Rutherfurd, who had owned much of the land during his life. However, the spelling was changed due to the fame of President Rutherford B. Hayes who was President of the United States during the 1870s when the town was created.