Whyte was born in Baltimore, Maryland, the son of Joseph and Isabella White (he later changed his surname to Whyte following a family disagreement). His grandfather, William Pinkney, had been a famous United States politician, administrator, and diplomat.

Whyte's early education involved instruction by a private tutor, who had been personal secretary to Napoleon Bonaparte. From 1842-1844, Whyte was unable to attend college as a result of his family's poor financial situation, and began work at the banking firm of Peabody, Riggs and Co. in Baltimore. He began to study law in Baltimore at the law office of Brown and Brune for one year before being admitted to the law school of Harvard University in 1844. Whyte returned to Baltimore in 1845 for further study, and was admitted to the Bar soon after in 1846.

From 1847-1849, Whyte served one term as a member of the Maryland House of Delegates. In 1850, Whyte was unsuccessful in a bid for election to the 32nd Congress. From 1853-1855, he served one term as Comptroller of the State Treasury of Maryland, for which he was credited for introducing a more simplified financial system to the State.

In 1857, Whyte was again nominated to serve in Congress. He was defeated, but brought forth evidence before the House of Representatives of fraud and corruption regarding the election. The House did not concur on whether or not he should have been appointed, however.

After nearly a decade out of the political arena, Whyte was asked by then-governor Thomas Swann to fill the remainder of resigning senator Reverdy Johnson's term from July 13, 1868 to March 3, 1869. During his short tenure as senator, Whyte steadfastly supported the embattled President Andrew Johnson, and also supported easing the tension on the Southern states during Reconstruction. He chose not to be a candidate for re-election in 1868, however.

In 1872, Whyte was elected Governor of Maryland, defeating Republican challenger Jacob Tome. In the election of 1874, Whyte was elected by the legislature as a Democrat to the United States Senate and accordingly resigned from the position of Governor. In 1874, in between his terms as governor and senator, he served as counsel for Maryland before the arbitration board in the boundary dispute between Virginia and Maryland. During this tenure as senator, Whyte opposed paying the nation's debt with silver and gold instead of solely gold, and served as the chairman of the U.S. Senate Committee on Printing (46th Congress). In the election of 1880, Whyte chose not to run for re-election, due to family illness and strife between him and his counterpart senator, Arthur P. Gorman.

Whyte was elected unopposed to be mayor of Baltimore in 1881. At the conclusion of his term in 1883, Whyte chose to go back to practicing law. From 1887-1891, Whyte was Attorney General of Maryland, and from 1900-1903, the Baltimore City Solicitor. In 1906, Whyte was appointed by Maryland Governor Edwin Warfield to fill the Senate seat vacancy caused by the death of Arthur P. Gorman.

Whyte served as senator until his unexpected death in Baltimore, and is buried in Greenmount Cemetery.