Smith was born in Trenton, New Jersey. He obtained a bachelor's degree from Lafayette College in 1965 and served in the United States Navy from 1965 to 1967, including a year of duty in Vietnam. Smith then taught history and English and got into the real estate business.

He unsuccessfully ran for a seat in the U.S. House of Representatives in 1982. He represented New Hampshire in the House of Representatives from 1985 to December 1990. He was elected to the Senate in 1990 to succeed the retiring Republican Senator Gordon J. Humphrey. He began to serve in December of 1990 because he was appointed to the position early following Humphrey's resignation. Smith was reelected in 1996 and served until 2003. In his 1996 reelection campaign he defeated Democrat Dick Swett with 49f the vote. Smith had established himself as the most conservative Senator from the Northeast, and Bill Clinton's coattails nearly caused his defeat. The campaign's most famous incident occurred on election night when several networks claimed that Swett had won, and were then forced to admit their mistake later. The coverage foreshadowed the controversies over coverage of the 2000 presidential election.

In January 1999, Smith announced that he was a candidate for the Republican nomination for president (at the time the front-runner was Texas Gov. George W. Bush). In July, after failing to gain any ground in the presidential race, Smith announced he was leaving the Republican Party and would seek the nomination of the U.S. Taxpayers Party for president. One month later, Smith swore off the Taxpayers Party and announced as an independent. He withdrew completely from the race in October and endorsed Bush.

In the meantime, Sen. John Chafee (R-R.I.) had died and thus the chairmanship of the Senate Committee on Environment and Public Works had reopened. Smith recanted his repudiation of the Republican party, claiming it had been "a mistake" and claiming that since he had never officially changed his voting registration that he had never left the party. Smith then was appointed as Chafee's successor to the chairmanship.

In 2002, Smith was denied re-nomination and was defeated in the Republican primary by John E. Sununu, after polls showed him losing badly to Democratic Governor Jeanne Shaheen in November. He moved to Sarasota, Florida after his defeat to sell real estate.

In 2004, Smith endorsed John Kerry for president, citing the growing federal deficit and Kerry's plan to balance the federal budget. He considered running for the Florida Senate seat now held by Mel Martinez, but dropped out after raising little money and receiving less than 1upport in Republican polls.

Less than a month before the election, Smith wrote an op-ed for the Concord Monitor in which he denounced the lack of Republican outrage over phone jamming on Election Day 2002. Republican operatives jammed phone banks used by the Democrats to contact Democratic voters and get them to the polls. He implied that this action may have made the difference in Shaheen's narrow loss to Sununu.