Pickle was born in Roscoe, Texas. He acquired his nickname Jake from a mischievous character he portrayed in a family play when he was four years old. Pickle was an Eagle Scout and recipient of the Distinguished Eagle Scout Award from the Boy Scouts of America.

Pickle attended the public schools in Big Spring and received his Bachelor of Arts from the University of Texas at Austin where he was a member of the 1934 Southwest Conference championship swimming team and the student body president as a senior in 1937. He served as an officer in the United States Navy for three and a half years (1942 - 1945) and was stationed on the USS St. Louis and the USS Miami.

Pickle was elected as a Democrat to the Eighty-eighth Congress, by special election, to fill the vacancy caused by the resignation of United States Representative Homer Thornberry. Pickle was reelected to the fifteen succeeding Congresses (December 21, 1963-January 3, 1995) before retiring. His campaign trademark was a "squeaky pickle" rubber toy he handed out to those he met and tossed from floats in area parades.

While in the House, Pickle rose through the ranks to become the third ranking Democrat on the House Ways and Means Committee. He played a key role in passing major Social Security reform legislation in 1983 to save the system from insolvency. The reforms increased the payroll tax rate, slowly increased the full benefit retirement age to 67 and taxed some of the benefits. He considered this legislation his greatest accomplishment.

Pickle was able to steer research money to the University of Texas, and today the University's J.J. Pickle Research Campus is named in his honor. He was influential in the city of Austin, as well, most notably for relocating Austin's main airport from Robert Mueller Municipal Airport to Austin-Bergstrom International Airport. He was also instrumental in bringing the SEMATECH and the MCC consortiums to Austin.

Before Pickle left for World War II in 1942, he married Ella Nora Critz. They had one daughter together. Critz died in 1952 and Pickle remarried in 1960 to Beryl Bolton McCarroll. Pickle struggled with prostate cancer for 14 years and with lymphoma for four. He died at his home in Austin and was buried in the Texas State Cemetery, also in Austin.