Belknap was born in Massena, New York and attended the common schools there. He moved with his family to Grand Rapids, Michigan in 1855. During the American Civil War, he enlisted as a private August 12, 1862 in Company H of the Twenty-first Regiment, Michigan Volunteer Infantry. He was rapidly promoted through several ranks, including

Fourth Sergeant, September 1, 1862 First Sergeant, January 1, 1863 Sergeant Major of the Regiment, February 1, 1863 Second Lieutenant, April 1, 1863 First Lieutenant, September 22, 1863, for gallant service at the Battle of Chickamauga, by special order of General Sheridan Captain, January 8, 1864, in recognition of services rendered at and near Chattanooga, Tennessee. He served in the Army of the Cumberland during the Atlanta Campaign, and with General Sherman's March to the Sea and through the Carolinas. He was mustered out of service on June 8, 1865. At the battles of Stones River and Chickamauga he received seven wounds, none of them very serious.

After the war, from the fall of 1865 to 1871, Belknap lived on a farm in Sparta, Michigan, before returning to Grand Rapids, where he organized the Belknap Wagon and Sleigh Company, a very successful business that manufactured wagons and sleighs.

Beginning in 1872, Belknap served in the city's volunteer fire service for many years, as both foreman of Company No. 3 and as Assistant Chief. He was instrumental in the transition from a volunteer to a paid fire service. He was a member of the Grand Rapids board of education 1878-1885, served on the board of aldermen from the Seventh Ward 1880-1882, and was mayor in 1884. In 1885, he was appointed by Governor Russell A. Alger as a Trustee of the Institution for the Deaf and Dumb in Flint, Michigan for a term ending in 1891.

Belknap defeated Melbourne H. Ford in 1888 to be elected as a Republican from Michigan's 5th District to the United States House of Representatives for the Fifty-first Congress. He served from March 4, 1889 to March 3, 1891. He was not a candidate for re-nomination to the Fifty-second Congress in 1890, but was subsequently elected to fill the vacancy caused by the death of Ford on April 20, 1891. Belknap served from November 3, 1891 to March 3, 1893. He unsuccessfully contested the election of George F. Richardson to the Fifty-third Congress. He performed staff duty at Fort Oglethorpe during the Spanish-American War.

He died in Grand Rapids and is interred in the Greenwood Cemetery there.