He was born in Detroit, Michigan. He moved with his divorced mother, mystery author Eleanor Atkinson, and a younger sister to Evanston, Illinois, when he was about 10, where he met and became close friends with another neighborhood child, Marlon Brando. During the war years he and his family moved to New York, where Cox attended CCNY and New York University. He supported his invalid mother and sister by making and selling jewelry at parties, and started doing comedy monologues for the guests, which were well-received enough to lead to regular performances at nightclubs such as the Village Vanguard, beginning in December of 1948. At one point, he became the roommate of his boyhood friend Marlon Brando, with whom he is often rumoured to have been romantically involved, although Cox was eventually married four times. Brando encouraged him to study acting with Stella Adler. Cox and Brando remained very close friends for the rest of Cox's life, and Brando is reported to have kept Cox's ashes in his bedroom and conversed with them nightly, until his own death.

Cox appeared in Broadway musical reviews, night clubs, and early TV comedy-variety programs in the period 1949 - 1951, creating a huge impact with a starring role as a well-meaning but ineffective policeman on Philco Television Playhouse in 1951. Producer Fred Coe approached Cox about a starring role in a proposed live TV sitcom, Mr. Peepers, and the rest is history. Peepers ran on NBC for three years and made Cox a household name in the US.

His best-known roles, apart from mild-mannered teacher Robinson Peepers of the television sitcom Mr. Peepers (1952-1955), were as the eponymous hero of The Adventures of Hiram Holliday (1956-1957); regular panelist on the television game show Hollywood Squares (1966-1973); and voice of the animated cartoon character Underdog (1964- ). He played character roles in over twenty motion pictures, and worked frequently in guest-star roles in a large number of TV drama, comedy and variety series in the 1960s and early 1970s. His television and screen persona was that of a shy, timid but kind man who wore thick eyeglasses and spoke in a tentative, high-pitched voice. Cox protested in vain to reporters over the years that he was nothing like Peepers; he was physically quite strong, hiked and rode a motorcycle, and especially in his later years sometimes displayed an unpleasantly sarcastic and peevish personality.

Cox died of a heart attack apparently brought on by a sleeping pill overdose, in Los Angeles at the age of 48. Eventually his ashes were mingled with those of Brando and another friend and scattered in Death Valley, California.