Caray made his debut in 1945 with the Cardinals, but was fired in 1969 amid rumors of a personal relationship that caused problems with the Busch family, who owned both the Cardinals and the Anheuser-Busch breweries. Caray was well known to his Cardinal radio audience for drinking and advertising a competetive brand of beer before the Busch family acquisition of the team. Caray attributed his firing to a business-related grudge. After a season with the Athletics, Caray broadcast for the White Sox from 1971 to 1981, and then for the Cubs from 1982 to 1997.

He was extremely popular among the citizens of Chicago, and was known as much for his public carousing and jovial spirit as for his sportscasting; it was not for nothing that he was proclaimed "The Mayor of Rush Street", referencing Chicago's famous bar-hopping neighborhood. In the years before his death, his skills as a broadcaster gradually declined due to illness and the effects of age, a remarkable recovery from a 1987 stroke notwithstanding. This led some people to say that he should retire, and in fact he was retained well beyond the normal mandatory retirement age of WGN-TV announcers. But his popularity was such that the normal rules were suspended. His tendency to mispronounce players' names (often humorously, such as trying to say a complicated name backwards), was widely parodied.

His style became fodder for pop culture parody as well, including a memorable Saturday Night Live recurring sketch featuring Caray (played by Will Ferrell) as a host of a space and astronomy TV talk show, in which his questions to scientists and professors included whether or not they would eat the moon if it was made of ribs. The sketch continued after Caray's death. When asked by Joan Allen about his death, Will Ferrell as Caray replied, "What's your point?" He was recently parodied by WWE Raw's Todd Grisham, who dressed up and impersonated him during his interview on Halloween, introducing Harry Caray to a whole new audience, even though none of the people he interviewed that night figured out who he was dressed up as. The Bob and Tom Show also had a Harry Caray parody show called "After Hours Sports" which eventually became "Afterlife Sports" after Caray's death. In 2005, the cartoon Codename: Kids Next Door had two annoucers reporting a baseball game. One was a parody of Caray, the other, Howard Cosell. The most widely-accepted impersonation of Caray in Chicago was done by Jim Volkman, heard most often on the Loop and AM1000.

His over-the-top extroversion, and attendant ego, occasionally led to rifts with his broadcasting partners. But in Chicago, with the eccentric and knowledgeable ex-outfielder Jimmy Piersall on the South Side, and the brilliant ex-pitcher Steve Stone on the North Side, he found sidekicks who could stand up to his style, and the partnerships thrived.

He was well-known for his frequent exclamation of "Holy Cow!" (which he said he trained himself to say, to avoid any chance of cursing on the air); his cautious-but-hopeful "It might be... It could be... It is!" call when a batter hit a deep fly ball that had a chance to be a home run; and for publicly leading crowds - especially at Comiskey Park and then Wrigley Field - in singing "Take Me Out to the Ball Game" during the seventh-inning stretch. He was not an especially good singer, often a bit off-key and with a strained voice, especially in his later years with the Cubs; but that was part of the charm of having him sing every game. Since his death, a "guest conductor" has sung the song at each Cubs home game. He had sung the song during the seventh inning for years in the broadcast booth when White Sox owner Bill Veeck secretly amplified his voice for all to hear during one game. The song then became a tradition and possibly his best-remembered trademark.

The other best-remembered aspect of his broadcasts was his exclamation, "Cubs win! Cubs win!" after every Cub triumph, which always seemed to represent genuine, unfeigned enthusiasm, carrying on the upbeat tradition of his predecessor Jack Brickhouse. While with the White Sox, of course, he had exclaimed "Sox win! Sox win!" And old-timers who heard his "Cardinals win!" over a quarter-century would never have dreamed he would become a Chicago icon. If anything, he was an even more shameless "homer" in St. Louis, but as he reported later, his firing changed his outlook and made him realize that his passion was for the game itself, and the fans, more than anything else.

Although he was known in later years for his deteriorating skills, early in his career Caray was known for his mastery of all aspects of broadcasting - writing his own copy, conducting news interviews, writing and presenting editorials, covering other sports such as University of Missouri football and hosting a sports talk program.

Following his death, during the entire 1998 season the Cubs wore a patch on the sleeves of their uniforms depicting a caricature of Caray. Cubs slugger Sammy Sosa dedicated each of his 66 home runs that season to Caray.

His son Skip Caray and his grandson Chip Caray have followed him into the booth as baseball broadcasters. The younger Caray was actually hired by WGN-TV to work with his grandfather, but Harry passed away before that pairing could take place. Chip Caray replaced Harry as the Cubs' play-by-play announcer from 1998 to 2004, then left the Cubs to work with his father Skip on Atlanta Braves broadcasts.

In 1989 the Baseball Hall of Fame presented Caray with the Ford C. Frick Award for "major contributions to baseball." He also has his own star on the St. Louis Walk of Fame.

On October 23, 1987 Harry Caray's Restaurant opened on Kinzie Street in Chicago, and there are now three restaurants, a 10-pin bowling alley, and an off-premises catering division which bear the Harry Caray name. The original restaurant has received numerous awards for its food and service, and features many items of memorabilia, even a "Holey Cow" (complete with holes !) wearing the trademark Harry Caray eyeglasses.

Harry Caray is interred in All Saints Cemetery in Des Plaines, Illinois.