He was born in Weedon, Buckinghamshire to a wealthy Catholic family, who named him after the reigning pope Leo XIII. Carroll made his stage debut in 1912, and played in London and Broadway until he moved to Hollywood in 1934 to start a career in film. Once there he soon made his film debut in Sadie McKee (1934). More parts followed, often playing doctors or butlers. He made notable appearances as Marley's ghost in A Christmas Carol (1938) and as Joseph in Wuthering Heights (1939).

Carroll is perhaps most well-known for his roles in six of Alfred Hitchcock's films. As with earlier roles he was often cast as doctors or other figure of authority, such as the spymaster The Professor in North by Northwest. He was also popular on television as the befuddled banker Topper (1953-56) and later as spymaster Alexander Waverly on The Man from U.N.C.L.E. (1964-68), echoing his earlier work for Hitchcock. Several U.N.C.L.E. films followed, and a spin-off The Girl from U.N.C.L.E. (1966). He was one of the first actors to appear in two different television series as the same character. He is also remembered for his role as the frustrated banker haunted by the ghosts of George and Marion Kirby, in the 1950's television series Topper which also starred Anne Jeffries, Robert Sterling and Lee Patrick. In 1972 he died in Hollywood of pneumonia brought on by cancer and was interred in the Grand View Memorial Park Cemetery in Glendale, California.