Smith was born in Newton, North Carolina and moved to Georgia in 1872 with his parents. He was primarily educated by his father, a professor at the University of North Carolina at Chapel Hill. Smith became a lawyer in Atlanta, Georgia, passing the bar examination in 1873. He served as chairman of the Fulton County and State Democratic Conventions and was president of the Atlanta Board of Education. In 1887, Smith bought the Atlanta Journal. His strong support in the Journal for Grover Cleveland during the 1892 Presidential election gained him the attention of Cleveland.

Smith was appointed as Secretary of the Interior by Cleveland in 1893. He worked against the railroad monopolies and for the economic development of the South. He resigned in 1896 over his disagreement with Cleveland about William Jennings Bryan's candidacy for President. Smith supported Bryan, the eventual Democratic candidate, while Cleveland did not.

Smith then allied himself with Bryan's Vice Presidential candidate, Populist Tom Watson, one of the most influential politicians in Georgia at the time. With Watson's support, Smith was elected governor in 1907. He established several Jim Crow laws requiring literacy tests and property ownership for voting. Smith also supported railroad reform and election reform. After losing the support of Watson, he was defeated in the next election by Joseph M. Brown. Smith was re-elected as governor in 1911.

In 1911, while still governor, he was chosen by the Georgia General Assembly to fill out the term of United States Senator Alexander S. Clay. Smith won re-election in 1914, but was defeated by Tom Watson in 1920. Afterwards, Smith practiced law in Washington, D.C. and Atlanta. He is buried in Oakland Cemetery, Atlanta, Georgia.