Davis was born in Clarksburg, West Virginia. His father was John James Davis, a West Virginia legislator who had supported slavery and opposed ratification of the Fifteenth Amendment to the United States Constitution. Davis acquired much of his father's conservative politics, opposing women's suffrage, child-labor laws, anti-lynching legislation and Harry S. Truman's civil rights program while privately defending the poll tax and questioning whether African-Americans should be allowed to vote. He also maintained his father's staunch allegiance to the Democratic Party, even as he later represented the interests of conservative business interests opposed to the New Deal.

Davis graduated with a law degree from Washington and Lee University, where he was a member of Phi Kappa Psi Fraternity. He represented West Virginia in the U.S. House of Representatives from 1911 to 1913, where he was one of the authors of the Clayton Act. He served as U.S. Solicitor General from 1913 to 1918 and as an ambassador to the United Kingdom from 1918 to 1921. As Solicitor General he successfully argued for the illegality of Oklahoma's "grandfather law", which effectively disenfranchised most black citizens of Oklahoma by exempting white residents descended from a voter who had been registered in 1866 from the literacy requirements of its electoral law, in Guinn v. United States. Davis's personal posture differed from his position as an advocate. Throughout his career he could separate his personal views and professional advocacy.

Davis was a dark horse candidate for the Democratic nomination for President in 1924. He won the nomination in 1924 as a compromise candidate on the one hundred and third ballot. His denunciation of the Ku Klux Klan and his prior defense of black voting rights as Solicitor General under Wilson cost him votes in the South and among conservative Democrats elsewhere. He lost in a landslide to Coolidge, who did not leave his house to campaign.

He was a member of the National Advisory Council of the Crusaders, an influential organization that promoted the repeal of prohibition.

Davis was one of the most prominent and successful lawyers of the first half of the twentieth century, arguing 140 cases before the U.S. Supreme Court, more than anyone had argued to that time. His firm, Davis, Polk, Wardlaw, Sunderland & Kiendl (now Davis Polk & Wardwell), represented many of the largest companies in the United States in the 1920s and following decades.

The last twenty years of Davis's practice included representing large corporations in the United States Supreme Court challenging the constitutionality and application of New Deal legislation. Davis lost many of these battles, though eloquent in his advocacy. His legal career is most remembered for his final, losing appearance before the Supreme Court, in which he unsuccessfully defended the "separate but equal" doctrine in Briggs v. Elliot, a companion case to Brown v. Board of Education. Davis not only brought his great talents as an advocate to the defense of racial segregation but, uncharacteristically, displayed his emotions in arguing that South Carolina had shown good faith in attempting to eliminate any inequality between black and white schools and should be allowed to continue to do so without judicial intervention. He expected to win, most likely through a divided Supreme Court, even after the matter was reargued after the death of Chief Justice Fred M. Vinson. He declined the fee that South Carolina offered him after the Court ruled against it unanimously.