He was born in Washington County, Pennsylvania with the name John Mitchell Hipple. He moved with his parents to Butler County, Pennsylvania at the age of 2. He attended public schools during much of his childhood, but also attended some private schools including the Witherspoon Institute. As a young man he was a schoolteacher. He seduced a 15-year-old female student, and, due to the resulting scandal, was forced to marry her.

In 1857, Mitchell stopped teaching and decided to become a lawyer. He built a successful law practice in Pennsylvania. However, in 1860, he decided to leave his community and family, and moved to California with a local schoolteacher with whom he was having an affair. After arriving in California, he abandoned her and moved to Portland, Oregon. It was then that he decided to change his name to John Hipple Mitchell, using his middle name as his last name, and attempted to start a completely new life in Oregon. Almost immediately, he started to become a successful lawyer and build political connections. Mitchell was not an intellectual man, but he was very ambitious and knew how to develop business and political friendships with important people.

Just two years after arriving in Oregon, in 1862, he was elected to the Oregon State Senate. In 1864 he became President of the state senate and served in that position until 1866. Because United States Senators were elected by the state legislatures during his lifetime, and that was the only office that Mitchell was to seek, this early position in the state senate was the only popularly elected office that he would ever run for or win.

Mitchell was an unsuccessful candidate for the United States Senate from Oregon in 1866. he tried again in 1872 and this time won, taking office in 1873. By this time, he had married again, but had not divorced the women he had married in Pennsylvania. His opponents tried to block him from becoming a senator by asking a senate committee to expel him for what he had done in the past, charging him with bigamy, desertion and living under an assumed name. Though these charges were certainly true, the Senate Committee decided they were not relevant. Mitchell served in the Senate from 1873 to 1879, and was defeated for reelection. He ran for reelection to the Senate in 1882 but lost. In 1885, however, he was elected again to the Senate, and reelected in 1890. He was defeated for reelection in 1896, but elected again for a final time in 1900. While not in the Senate, Mitchell practiced law. His last term in the Senate began in 1901 and was to last until 1907, but Mitchell died before it expired.

In 1905, Mitchell was indicted for a scandal involving his use of political influence in the federal government to help clients with their land claims. An appeal was under way and the Senate was beginning proceedings to expel him when Mitchell died of an illness in Portland, Oregon.

Mitchell was devoted to business interests and was against the populists and their political reforms. In the Senate, he was interested in transportation issues. He was chairman of the committee on railroads from 1877 to 1879 and from 1889 to 1893, and chairman of several committees related to coastlines and the ocean during his terms in the Senate. He was also chairman of the committee on claims from 1891 to 1893 and chairman of the committee of elections and privileges from 1895 to 1897.