He attended the University of Tennessee, graduating at the age of 19 with two degrees in engineering. An early fascination in automobiles led Brown to a job with the Stevens Duryea Company, then to his own Brown Motor Car Company in Alabama. He later abandoned the car dealership after developing an in motion pictures around 1913. He was hired by the Peerless Studio at Fort Lee, New Jersey, and became an assistant to the great French-born director Maurice Tourneur.

After serving in World War I, Brown was given his first co-directing credit (with Tourneur) for 1920's The Great Redeemer. Later that year, he directed a major portion of The Last of the Mohicans when Tourneur was injured in a fall.

Brown moved to Universal in 1924, and then to MGM, where he stayed until the mid-1950s. At MGM he was one of the main directors of their female stars - he directed both Joan Crawford and Greta Garbo five times. Garbo referred to Brown as her favorite director.

He worked across the introduction of sound and continued to use the silent films visual techniques throughout his career; he did not work particularly well with dialogue. His works have been regarded as considerate and atmospheric, but often were perceived as conventional, placid and slow. Nevertheless, he was nominated seven times for an Academy Award (six times as a director, once as a producer) but never received an Oscar. However, he did win Best Foreign Film for Anna Karenina at the 1935 Venice International Film Festival.

Brown retired a wealthy man due to his real estate investments, but refused to watch new movies, as he feared it might cause him to restart his career. In the 1970s, Brown became a much-sought-after guest lecturer on the film-festival circuit, thanks in part to his connection with Garbo.

The Clarence Brown Theater, on the campus of the University of Tennessee, is named in his honor.