Born in Lonaconing, Maryland, Grove was a sandlot star in the Baltimore area during the 1910s. His performance naturally caught the eye of Jack Dunn, the owner of the minor league Baltimore Orioles, who also discovered Babe Ruth.

Grove joined the Orioles in 1920 and embarked on an epic minor league career which saw him regarded by some as one of the best pitchers in the game before he ever threw a pitch in the majors. Breaking into the team's pitching rotation at midseason, he posted a 12-2 record. Over the next four seasons, he posted marks of 25-10, 18-8, 27-10 and 27-6, leading the International League in strikeouts every season.

Grove remained in the minor leagues through 1924 because Dunn, who ran an independent operation with no major-league affiliation, refused several offers from the majors to acquire him. Finally, early in 1925, Dunn agreed to sell Grove's rights to the Philadelphia Athletics for $106,000, the highest amount ever paid for a player at the time.

He battled injuries as a rookie and posted only a 10-13 record despite leading the league in strikeouts, then settled down in 1926 and won the first of a record nine earned run average (ERA) titles with a mark of 2.51. In 1927, he won 20 games for the first time, and a year later he led the league in wins, with 24.

From 1929 to 1931, the Athletics won the pennant all three seasons, and the World Series for the first two, and Grove led the way as the league's top pitcher. He posted records of 20-6, 28-5 and 31-4 in those years, the last of which was his greatest season. He led the league in wins, ERA (2.06), strikeouts (175), winning percentage, complete games and shutouts. He was chosen as league MVP in 1931, making him one of the few pitchers to achieve this. His MVP Award is the only one not housed in Coopertown as it is housed at the Georges Creek Library in Lonaconing, Maryland.

The Athletics continued to contend for the next two seasons, but finished second to the New York Yankees both years. Following the 1933 season, team owner Connie Mack suffered severe financial problems and was forced to sell Grove to the rival Boston Red Sox.

At the time, the Red Sox were a bad team, and Grove didn't help much his first year, when an arm injury held him to an 8-8 record. But in 1935, he returned to form with a 20-12 record and a league-leading 2.70 ERA. He won his eighth ERA title a year later, and also led the league in that category and winning percentage in 1938. He didn't win as many games in Boston, as managers protected his arm as he aged, but he continued to post outstanding records, such as 14-4 in 1938 and 15-4 a year later.

Grove retired in 1941 with a career record of 300-141. His .680 lifetime winning percentage is eighth all-time, but none of the seven men ahead of him won more than 218 games. His lifetime ERA of 3.06, when adjusted for the hitters' parks he played his entire career in and the era in which he played, is the best of any pitcher in history (except the still-active Pedro Martínez) at 48 percent above average.

Grove was elected to the Baseball Hall of Fame in 1947. He died in Norwalk, Ohio and was interred in the Frostburg Memorial Cemetery, in Frostburg, Maryland.

In 1999, he ranked number 23 on The Sporting News' list of the 100 Greatest Baseball Players, the highest-ranking left-handed pitcher. That same year, he was elected to the Major League Baseball All-Century Team.