Conger was born in Cooperstown, New York and moved with his father, the Rev. E. Conger, to Huron County, Ohio in 1824. He pursued academic studies at the Huron Institute in Milan, Ohio, and graduated from Western Reserve College (now Case Western Reserve University) in Hudson, Ohio in 1841. He engaged in mineral explorations of the Lake Superior copper and iron regions in connection with the Michigan State Geological Survey 1845-1847. He engaged in the practice of law in Port Huron, Michigan in 1848 and was elected judge of the St. Clair County Court in 1850. He was a member of the Michigan State Senate 1855-1859, and served as President pro tempore in 1859. He was a member of the State military board during the Civil War, holding the rank of colonel. He was a Presidential Elector for Michigan in 1864 and a member of the State constitutional convention in 1866.

He was elected as a Republican to the United States House of Representatives for the Forty-first Congress and to the five succeeding Congresses, serving from March 4, 1869, until March 3, 1881, when he resigned to become U.S. Senator. He served as chairman, Committee on Expenditures in the Department of State in the Forty-second Congress, and the Committee on Patents in the Forty-third Congress. Conger was elected in 1881 as a Republican to the United States Senate and served from March 4, 1881 to March 3, 1887. He was an unsuccessful candidate for renomination. He served as chairman, Committee on Manufactures in the Forty-seventh Congress, the Committee on Revision of the Laws in the Forty-eighth Congress, and Committee on Post Office and Post Roads in the Forty-ninth Congress.

After leaving Congress, he engaged in the practice of law in Washington, D.C. and died in Ocean City, Maryland. He is interred in Lakeside Cemetery, Port Huron, Michigan.

This article incorporates facts obtained from the public domain Biographical Directory of the United States Congress.