Drafted by the Montreal Expos in 1991 after playing in the Cape Cod League, he quickly came to the majors with them by 1993. His years with the Expos were uneven, with a reasonable 1994 performance followed by a solid 1995 and a mediocre 1996.

That year, the San Francisco Giants traded Mark Leiter, then the organization's most prominent starting pitcher, to the Montreal Expos. In return, the Giants received reliever Tim Scott and Rueter. Scott was a disappointment, posting an 8.24 ERA with the Giants, but Rueter blossomed into one of the Giants' most dependable starters and was with the team for nine seasons. For many fans, Rueter's defining moment as a Giant was his gutsy bullpen performance in Game 2 of the 2000 NLDS, where he relieved starter Shawn Estes after Estes sprained his ankle on a horrible baserunning play and proved far more effective in relief than Estes did starting. However, he began to struggle badly in 2004 with a 9-12 record and a 4.73 ERA and in 2005 after posting a 2-7 record and 5.95 ERA the Giants finally gave up and designated him for assignment.

In 2000 Rueter was the first pitcher to start a major league game at Pacific Bell Park in San Francisco. He is the 5th most successful lefty all time for the Giants organziation with an overall win-loss percentage of .601.

Rueter's trademarks are his fast-paced pitching style and his large ears. He is also known to wear Tommy Bahama dress shirts when not pitching. During the offseason he resides in Nashville, IL with his wife and two daughters and his home is famous for its "Shed", a large recreational facility filled with games and sports memorabilia.

On March 6th, 2006, Rueter announced his retirement from the game after 13 seasons. He retired as the all-time most successful left-handed pitcher in San Francisco Giants history, with 105 of his 138 wins in a Giants uniform.