Born in Gilmer, Texas, and raised in San Francisco, California, Mathis began singing publicly at school and church events shortly after his father discovered his talents. His father found him a voice teacher when he was about thirteen years old.

He remains one of the very few popular singers who received years of professional voice training that included opera. He also was an athletic talent, earning four athletic letters in high school and a subsequent athletic scholarship to San Francisco State College. Mathis also remains a large part of San Francisco State College's sports history—in 1954 he broke future basketball great Bill Russell's high jump record by jumping six feet five inches (1.96 meters). At the time only four Olympic athletes had managed to clear this height. He was talent-spotted by club owner Helen Noga and upon his introduction to jazz producer George Avakian was subsequently signed by Columbia Records.

His most difficult decision was deciding whether to go to the Olympic tryouts, to which he had been invited, or to keep an appointment to make his first recordings which were released in 1956. While Mathis opted for a recording career, he has never completely abandoned his enthusiasm for sports. An avid golfer who has completed a minimum of five holes-in-one, and has hosted several tournaments in his name in the USA and the United Kingdom.

Although frequently described as a romantic singer, his vast discography includes jazz, traditional pop, Brazilian and Spanish music, Soul/R&B, soft rock, and Broadway/Tin Pan Alley standards. He enjoyed some early cinematic visibility when he sang one of his first hits, "It's Not for Me to Say", in the film, Lizzie, in which he also had a small acting role. Mathis also remains highly synonymous with holiday music, having recorded six Christmas albums. He has recorded over 110 albums and sold more than 100 million albums and singles worldwide. He has the distinction of having the longest stay of any recording artist on the Columbia Record label, having been with the label from 1956 to 1963 and from 1968 to the present. He is one of only a few recording artist whose career has spanned six calendar decades.

Some of his hit songs include "Chances Are", "It's Not for Me to Say," "Wonderful! Wonderful!," "The Twelfth of Never," "Wild Is the Wind," "Heavenly," "Misty," "Small World" "A Certain Smile," "Gina," "I'm Coming Home," "When a Child Is Born," "Gone, Gone, Gone", "Too Much, Too Little, Too Late" (with Deniece Williams), and "Friends in Love" (with Dionne Warwick).

Mathis has had much more success as an albums artist. His albums achieved success in part due to their reputation as an accompaniment to lovemaking. Some of his celebrated early albums include "Heavenly", "Faithfully", "Open Fire, Two Guitars", "Warm", "Swing Softly", "Johnny's Mood", and "I'll Buy You a Star".

Mathis continues to perform and record regularly and his latest album, 2005's Isn't It Romantic: the Standards Album, has been enthusiastically received by critics.

A 1982 article in Us Magazine quoted Mathis as saying "Homosexuality is a way of life that I've grown accustomed to" and admitting to having had a sexual relationship with a male saxophone player. In 2006, Mathis revealed in a new interview that he didn't discuss his sexuality any further after the 1982 interview because he had received death threats.