Utah State University assistant athletic media relations director Zach Fisher says Olsen died Wednesday night.
Olsen was an All-American at Utah State and a first-round draft pick of the Los Angles Rams in 1962.
He was part of the Rams' "Fearsome Foursome" defensive line, which set an NFL record for the fewest yards allowed during a 14-game season in 1968.
Olsen is still the Rams' all-time leader in career tackles with 915.
Olsen moved on to acting with roles in Little House on the Prairie and later his own series, Father Murphy, which ran from 1981 to 1983.
He was diagnosed with mesothelioma last year.