Jeff was born in Birmingham, Alabama and attended Banks High School. He signed with Ole Miss after being ignored by college scouts. While at Ole Miss, Herrod set the school record for tackles with 528 total tackles, led the Rebels in tackles for three straight seasons, and was the 2nd all-time leading tackler in the SEC.

In 1988, Herrod was drafted in the 9th round by the Indianapolis Colts. He spent 10 seasons with the Colts and one with the Philadelphia Eagles. Over his NFL career, Herrod started in 123 of 143 regular season games, had 1,347 tackles with 6 forced fumbles, 4 fumble recoveries, and 14.5 sacks. He broke the single season, single game, and career tackle records with the Indianapolis Colts.

In addition to being inducted into the Mississippi Sports Hall of Fame, Jeff Herrod has also been selected as a member of the Alabama Sports Hall of Fame, the Ole Miss Athletic Hall of Fame, SEC Football Legend in 2013, and Indianapolis Colts Football Legend.