Goldberg earned a degree in engineering from the University of California, Berkeley in 1904. Goldberg was hired by the city of San Francisco as an engineer out of college. However his fondness for drawing cartoons prevailed, and after just a few months he left the city to employ for a job with the San Francisco Chronicle as a sports cartoonist. The following year he took a job with the San Francisco Bulletin where he remained until 1907, when he relocated to New York City.

He drew cartoons for several newspapers, including the New York Evening Journal and the New York Evening Mail. His work entered syndication in 1915, beginning his nationwide popularity. A prolific artist, Goldberg produced several cartoon series simultaneously; titles included Mike and Ike, Bob McNutt, Foolish Questions, Lala Palooza, and The Weekly Meeting of the Tuesday Women's Club.

While all these series were quite popular, the one which led to his lasting fame involved a character named Professor Lucifer Gorgonzola Butts. In this series, Goldberg would draw labeled schematics of comical "inventions" which would later bear his name. In 1995, "Rube Goldberg's Inventions", depicting Professor Butts' Self-Operating Napkin, was one of 20 strips included in the Comic Strip Classics series of commemorative U.S. postage stamps.

He was awarded the Pulitzer Prize for his political cartooning in 1948.

Later in his career Goldberg was employed by the New York Journal American, remaining there until his retirement in 1964. During his retirement he occupied himself with making bronze sculptures. Several one-man shows of his work were organized, the last one of his lifetime being in 1970 at the National Museum of American History in Washington, D.C.. Shortly afterward, he died at the age of 87; he is buried at Mount Pleasant Cemetery in Hawthorne, New York.