Root was born in Clinton, New York where he attended Hamilton College; there he joined the Sigma Phi Society, later becoming its national mentor and spiritual leader. After graduation, Root taught for one year at the Rome Academy. In 1867, Root graduated from the Law School of New York University. He went into private practice as a lawyer. While mainly practicing corporate law, Root was a junior defense counsel during the corruption trial of William "Boss" Tweed. Root also had private clients including Jay Gould, Chester A. Arthur, Charles Anderson Dana, William C. Whitney, Thomas Fortune Ryan, and E. H. Harriman.

Root was appointed United States Attorney for the Southern District of New York by President Chester A. Arthur.

Root married Clara Frances Wales (died in 1928), who was the daughter of Salem Wales, the managing editor of Scientific American, in 1878. They had three children: Edith (married Ulysses S. Grant III), Elihu, Jr. (who became a lawyer), and Edward.

He served as the United States Secretary of War 1899-1904 under William McKinley and Theodore Roosevelt. He reformed the organization of the United States Army. He was responsible for enlarging West Point and establishing the U.S. Army War College as well as the General Staff. He changed the procedures for promotions and organized schools for the special branches of the service. He also devised the principle of rotating officers from staff to line. Root was concerned about the new territories acquired after the Spanish-American War and worked out the methods of how Cuba would be turned over to the Cubans, wrote the charter of government for the Philippines, and eliminated tariffs on goods imported to the United States from Puerto Rico.

In 1904, Root left the cabinet and returned to private practice as a lawyer.

In 1905, President Roosevelt named Root to be the United States Secretary of State. As secretary, Root placed the consular service under the Civil Service. He maintained the Open Door Policy in the Far East. On a tour to Latin America in 1906, Root persuaded those governments to participate in the Hague Peace Conference. He worked with Japan in emigration to the United States and in dealings with China. He worked with Great Britain in resolving border disputes between the United States (Alaska) and Canada and also in the North Atlantic fisheries. He supported arbitration in resolving international disputes.

He served in the United States Senate (Republican - New York) from 1909 to 1915, and did not seek reelection. He served as President of the Carnegie Endowment for International Peace from 1910 to 1925. In that capacity, he helped created the Hague Academy of International Law in the Netherlands.

In 1912, as a result of his work to bring nations together through arbitration and cooperation, he received the Nobel Peace Prize.

At the outbreak of World War I, Root opposed President Woodrow Wilson's policy of neutrality. He did support Wilson once the United States entered the war.

In June 1916, Root sought the Republican presidential nomination. However, Root reached his peak strength of 103 votes on the first ballot. The Republican presidential nomination went to Charles Evans Hughes, who lost the election to Democrat Woodrow Wilson.

In June 1917, at age 72, he was sent to Russia by President Wilson to arrange American co-operation with the new revolutionary government. A large party of well-known people accompanied the Senator, and they traveled from Vladivostok across Siberia in the Czar's former train. Root remained in Petrograd for close to a month, and was not much impressed by what he saw. The Russians, he said, "are sincerely, kindly, good people but confused and dazed." He summed up his attitude to the Provisional Government very trenchantly: "No fight, no loans." Which referred to the current conflict with Germany in World War I.

After World War I, Root supported the League of Nations and served on the commission of jurists, which created the Permanent Court of International Justice. In 1922, President Warren G. Harding appointed him as a delegate to the International Conference on the Limitation of Armaments.

Root worked with Andrew Carnegie in programs for international peace and the advancement of science. He was the first president of the Carnegie Endowment for International Peace.

He was among the founders of the American Law Institute in 1923.

Root died in 1937 in Clinton, New York, with his family by his side.

He is buried at the Hamilton College Cemetery .

In addition to the Nobel Prize, Root was awarded the Grand Cordon of the Order of the Crown (Belgium) and the Grand Commander of the Order of George I (Greece).