Born in Newburyport, Massachusetts, he attended the common schools and then was apprenticed to a cabinetmaker. In his youth he worked as a printer in Philadelphia and then, in 1847, moved to Washington, D.C., where he was a secretary to a Congressman.

He moved to California in 1849 and settled in Nevada City in 1850. There he was on the staff of the Nevada Daily Journal, eventually becoming that newspaper's owner. He was admitted to the California bar in 1854 and began practicing in Nevada City, becoming district attorney for Nevada County in 1856. He was served in the California Senate in 1856.

Sargent was elected as a Republican to the 37th Congress; skipped several terms and was reelected to the 41st and 42nd Congresses. In 1861 he was the author of the first Pacific Railroad Act that was passed in Congress.

He was elected to the United States Senate and served 1873 to 1879. During his time in the Senate he was chairman of the U.S. Senate Committee on Mines and Mining during the 44th Congress and chairman of the U.S. Senate Committee on Naval Affairs during the 45th Congress.

After leaving the Senate he practiced law in San Francisco for three years, leaving to become ambassador to Germany for two years, and held office until the action of the German authorities in excluding American pork from the empire made his incumbency personally distasteful.

He died in San Francisco in 1887 and is currently buried in the Pioneer Cemetery in Nevada City, California.

He was sometimes called the Senator for the Southern Pacific Railroad.