Phelan was born in San Francisco, California in 1861, the son of an Irish immigrant who became wealthy during the California Gold Rush as a trader, merchant and banker. Phelan graduated from St. Ignatius College in that city in 1881. He studied law at the University of California, Berkeley and then became a banker. He was elected Mayor of San Francisco and served from 1897 until 1902.

Phelan was president of Relief and Red Cross Funds after the 1906 San Francisco earthquake. He was then elected as a Democrat to the United States Senate and served from March 4, 1915 to March 3, 1921. He was an unsuccessful candidate for reelection in 1920. During his time in the Senate he was chairman of the U.S. Senate Committee on Railroads during the 64th Congress and of the U.S. Senate Committee on Irrigation and Reclamation of Arid Lands during the 65th Congress.

After his time in the Senate, Phelan returned to banking, and collected art. He died at his country estate Villa Montalvo in Saratoga, California in 1930. After his death, the estate was gifted to the people of California. Some of his momentos and correspondence are on display in the library at Villa Montalvo.

Phelan is buried in the family mausoleum in Holy Cross Cemetery in Colma, San Mateo County, California.