Davis was born Raiford Chatman Davis in Waycross, Georgia. Following his parents' wishes, he attended Howard University, graduating in 1938. His acting career, which spanned seven decades, began in 1939 with the Rose McClendon Players in Harlem. He made his film debut in 1950 in the Sidney Poitier film No Way Out.

Davis experienced many of the same struggles that most African-American actors of his generation underwent; he wanted to act but he did not want to play stereotypical subservient roles, such as butler, that were the standard for black actors of his generation. Instead, he tried to follow the example of Sidney Poitier and play more distinguished characters. When he found it necessary to play a Pullman porter or a butler, he tried to inject the role with a certain degree of dignity.

In addition to acting, Davis, along with Melvin Van Peebles, was one of the first African-American directors. Along with Bill Cosby and Poitier, Davis was one of a handful of African-American actors able to find commercial success while avoiding stereotypical roles prior to 1970. However, it should be noted that Davis never had the tremendous commercial or critical success that Cosby and Poitier enjoyed.

Davis found recognition late in his life by working in several of director Spike Lee's films, including Do The Right Thing, Jungle Fever, She Hate Me and Get on the Bus. He also found work as a commercial voice-over artist and served as the narrator of the early-1990s CBS sitcom Evening Shade, starring Burt Reynolds.

Ossie Davis and his wife, actress Ruby Dee (they married in 1948), were well-known civil rights activists, being personal friends of Malcolm X, Jesse Jackson, Martin Luther King Jr. and others. Davis and Dee helped organize (and served as MCs for) the 1963 civil rights March on Washington for Jobs and Freedom. Davis, alongside Ahmed Osman, delivered the eulogy at the funeral of Malcolm X; he re-read part of this eulogy at the end of Spike Lee's film Malcolm X. He also delivered the eulogy for Martin Luther King, Jr.

Davis and wife Ruby Dee were recipients of the Kennedy Center Honors in 2004. They were also named to the NAACP Image Awards Hall of Fame in 1989. Their son Guy Davis is a blues musician.

Davis was found dead on February 4th 2005, in a hotel room in Miami, Florida, of natural causes. He was in the first stages of working on a film called Retirement.

His last role was a several episode guest arc on the groundbreaking Showtime drama series The L Word as a father struggling with the acceptance of his daughter Bette (Jennifer Beals) parenting a child with her lesbian partner.

In his final episodes, his character was taken ill and died. His wife Ruby Dee was present during the filming of his own death scene. That episode, which aired shortly after Davis's own death, aired with a dedication to the actor.