He was born in Lenoir, North Carolina, the youngest son of North Carolina furniture magnate J. E. Broyhill. The senior Broyhill was a member of the Republican National Committee for 28 years. However, for most of that time the party was almost nonexistent in the former Confederacy, including North Carolina. Jim Broyhill joined his father's company in 1945 and served in various capacities there until 1962. He was also active in several state industry associations, as well as a civic leader in Lenoir.

He made his first run for elected office in 1962 for North Carolina's 9th Congressional District. Broyhill was the unexpected beneficiary of redistricting after the 1960 census, which cost North Carolina a congressional district. The Democratic-controlled General Assembly saw a chance to get rid of the then lone Republican in the congressional delegation, Charles Raper Jonas of the Charlotte area by drawing his district from under him. In the process, they shifted some strongly Republican areas into the 9th, a district where growing Republican influence had kept five-term Democrat Hugh Quincy Alexander from establishing a foothold. At the same time, they shifted some strongly Democratic areas of the 9th into the new 8th District designed to defeat Jonas. However, the plan backfired disastrously when Jonas handily defeated 8th District incumbent A. Paul Kitchin and Broyhill defeated Alexander by just under one percentage point.

Broyhill never faced another contest nearly that close again. Due to his very conservative stances on nearly all issues and an emphasis on taking good care of his constituents (most of whom had never been represented by a Republican before), he became very popular in his district. He won reelection by 11 points in the midst of the gigantic Democratic landslide of 1964, in which Lyndon Johnson carried 87 of North Carolina's 100 counties. Broyhill won reelection 10 times thereafter, never receiving less than 54 percent of the vote and only winning by less than 10 points twice in what became the most Republican district in North Carolina. His district was renumbered the 10th in 1969.

In July 1986, Senator John P. East, who was not seeking reelection, committed suicide. Broyhill had already won the Republican nomination for the race to succeed East, but Governor Jim Martin appointed Broyhill to the seat until at least the November elections. The plan was to give Broyhill a leg up on the Democratic nominee, former governor Terry Sanford. However, the gambit failed when Sanford narrowly defeated Broyhill in November. Sanford took office immediately because there had been two elections that night--the regular Senate election and a special election to fill the remainder of East's term.

Broyhill later served as Chairman of the North Carolina Economic Development Commission and then as the state's Secretary of Commerce under Martin.

Broyhill remains active in North Carolina Republican politics as an "elder statesman." He was inducted into the North Carolina Republican Party Hall of Fame and the Lenoir, North Carolina Post Office was renamed in his honor. He now lives in Winston-Salem. His son Ed was a candidate for the Republican nomination in the 5th District in 2004.