The amiable, wavy-haired leading man is equally at home in drama and comedy, although he is more often seen in the latter. Originally a stage actor and musician, Segal appeared in several nondescript films in the early 1960s before raising eyebrows in 1965 as a distraught newlywed in Ship of Fools and as a P.O.W. in King Rat. He followed with top performances as Nick in Who's Afraid of Virginia Woolf? (for which he was nominated for an Oscar), a Cagneyesque gangster in The St. Valentine's Day Massacre, perplexed police detective Mo Brummel in No Way to Treat a Lady, a bookworm in The Owl and the Pussycat, a man laying waste to his marriage in Loving, and a hairdresser turned junkie in Born to Win. Segal starred with Ruth Gordon in Carl Reiner's 1970 outrageous dark comedy Where's Poppa?.

He played an inept burglar in the 1972 comedy The Hot Rock with Robert Redford, a comically unfaithful husband in A Touch of Class and a midlife crisis victim in Blume in Love. He co-starred with Jane Fonda as suburbanites-turned-bank-robbers in Fun with Dick and Jane (1977 film), and starred as a faux gourmet in Who Is Killing the Great Chefs of Europe?.

Segal was so appealing that too often he was asked to carry a film on his charm alone, especially in the 1970s. He was relatively inactive in the 1980s, but bounced back as the sleazy father of Kirstie Alley's baby in Look Who's Talking, and in the 1993 sequel Look Who's Talking Now, and as a left-wing comedy writer in For the Boys (1991).

He has since starred in the long-running NBC television sitcom Just Shoot Me! (1997-2003) as the head of the wacky fashion and style magazine Blush.

He is also an accomplished banjo player; in 1974 he played in "A Touch of Ragtime" {Stereophonic LP}, an album with his band, the Imperial Jazzband.