Born in New York City to a Jewish family, Arkin is one of only five actors to receive an Academy Award nomination for Best Actor for his first screen appearance (for The Russians Are Coming, the Russians Are Coming) in 1966. Two years later, he was again nominated, for The Heart Is a Lonely Hunter. In a piece he wrote for the Second City book, Arkin wanted to be an actor since he was five. It was in a performance in St. Louis that Arkin caught a casting director's eye, who later met with the actor to tell him about a comedy troupe he was assembling in Chicago and if he wanted a job, it was open. Arkin politely agreed, before heading back to New York City with the impression that he wasn't going to lose out on a career by moving to Chicago. But after another year as a struggling actor at 29 years old, Arkin called the director and asked if the offer was still open. With the offer still on the table, he packed his bags and headed for the midwest, thinking it was a mistake. But Arkin later said that it turned out to be the best thing in his life since it turned around when he joined what was The Second City comedy troupe.

Arkin is equally comfortable in comedy and dramatic roles, in such diverse films as Inspector Clouseau, Catch-22, Freebie and The Bean, Hearts of the West, The Seven-Per-Cent Solution (where he played Sigmund Freud), Edward Scissorhands, The Rocketeer, the 1979 The In-Laws, Glengarry Glen Ross, Four Days in September, So I Married an Axe Murderer, Jerky Boys: The Movie and Raising Flagg.

His son is actor Adam Arkin.