Weller was born in Reisterstown, Maryland and attended the public schools. He entered the United States Naval Academy in Annapolis, Maryland in 1877 and graduated in 1881. After two years of service in the United States Navy, he was honorably discharged in 1883.

Following his service in the Navy, Weller enrolled at the National Law School in 1883. While attending school, he was employed as a clerk in the Post Office Department in Washington, D.C. from 1883 to 1887. He graduated in 1887 and was admitted to the Bar in 1888.

After practicing law for three years, Weller engaged in banking and manufacturing, and was a member of a stock brokerage firm until retiring in 1901. He traveled extensively following retirement.

Weller became chairman of the State Roads Commission of Maryland in 1912. He ran as Republican party candidate in the 1915 Maryland gubernatorial election, but was defeated by Democratic candidate Emerson C. Harrington.

Weller was treasurer of the Republican National Senatorial Committee from 1918 to 1920 until he was elected to the United States Senate in 1920, defeating incumbent John Walter Smith. Weller unsuccessfully tried for re-election in 1926, losing his position to Millard Tydings.

While senator, Weller was chairman of the U.S. Senate Committee on Manufactures.

Weller resumed the practice of law in Baltimore, Maryland until his death. He is buried in Arlington National Cemetery in Arlington, Virginia.