Bate had served as a volunteer in the Mexican War. He served in the Tennessee House of Representatives from 1849 to 1851. He graduated from law school in Lebanon, Tennessee in 1852 and was admitted to the bar in that year, establishing his practice in Gallatin, Tennessee. He became district attorney general for the Nashville district in 1854.

Bate served in the Confederate forces in the Civil War, attaining the rank of major general. After the defeat of the Confederacy he returned to the practice of law; as was the case of many prominent ex-Confederates, full civil rights were eventually restored to him.

Bate was elected governor as a Democrat in 1882 over the incumbent Republican, Alvin Hawkins, and re-elected in 1884 and is credited with having found a satisfactory solution to the debt problems of the state. His subsequent four elections to the U.S. Senate were by the Tennessee General Assembly, the method of choosing U.S. Senators prior to the ratification of the Seventeenth Amendment to the United States Constitution. He was elected to the U.S. Senate in 1887, 1893, 1899, and 1905, one of only three Tennessee senators to be elected to more than three terms and one of two prior to the adoption of popular election to the office. As a Senator, he served as the chairman of the Committee on the Improvement of the Mississippi River and Its Tributaries in the 53rd Congress and the chairman of the Committee on Public Health and the National Quarantine in two later Congresses. He died only five days into his fourth term, in Washington, D.C.. His funeral was held in the Senate chamber of the United States Capitol, and he is buried in Nashville's Mount Olivet Cemetery.