He was born in East Los Angeles, California to a Southern Baptist Mexican immigrant with 1/4 Hungarian Jewish ancestry (the family name was originally Olmosh) and a Mexican-American Catholic mother. Olmos grew up wanting to be a professional baseball player. In his teen years, he turned to rock and roll, and for several years played various clubs in and around Los Angeles. He branched out into acting, appearing in many small productions, until he finally got his big break by portraying the narrator, called "El Pachuco", in the play Zoot Suit, which dramatized the World War II-era rioting in Southern California brought about by the tensions between Mexican-Americans and local police. (See Zoot Suit Riots.) He took the role to the filmed version in 1981. Other film appearances followed, including Wolfen, Blade Runner and The Ballad of Gregorio Cortez.

In 1984, Olmos starred in his biggest role up to that date as the authoritative police Lieutenant Martin Castillo in the television series Miami Vice, for which he was awarded an Emmy. Returning to film, he received an Academy Award nomination for Best Actor for the 1988 movie Stand and Deliver, portraying a real-life math teacher, Jaime Escalante, who turned his students into math whizzes, despite their disadvantaged backgrounds. He directed American Me in 1992, and starred in the multigenerational story of a Chicano family in My Family (aka Mi Familia) in 1995.

Olmos has often become involved in social issues, especially those affecting the Hispanic-American community in the United States. In 1998 Olmos founded Latino Public Broadcasting and currently serves as its Chairman. Latino Public Broadcasting funds programming for public television which focuses on issues effecting Hispanic-Americans and advocates for diverse prespectives in public television. Olmos also makes frequent appearances at juvenile halls and detention centers to speak to teenagers at risk. He has also been an international ambassador for UNICEF. In 2001, he was arrested and spent 20 days in prison for taking part in the Navy-Vieques protests against United States Navy target practice bombings of the island of Vieques, Puerto Rico.

Olmos played Rafael Leonidas Trujillo in the 2001 movie In the Time of the Butterflies. He also appeared as a recurring character, Justice Mendoza, in the TV series The West Wing.

Olmos was married to actress Lorraine Bracco, but she filed for divorce in January 2002. He has a star on the Hollywood Walk of Fame at 7021 Hollywood Boulevard. In 2003, he starred as Commander Adama in the Sci Fi Channel mini-series remake of the series Battlestar Galactica and has since appeared in every season of the new Battlestar Galactica TV series.

Olmos is also the former son-in-law of Howard Keel and the father of Bodie Olmos (who also appears on Battlestar Galactica) and Mico Olmos; he also has three adopted children: Michael D. Olmos, Brandon Olmos, and Tamiko.