Gene Mills thinks it’s probably for the best that he never ended up on the mat with the now deceased former president Jimmy Carter. Mills, at one time an Olympic-caliber wrestler, was stung by Carter’s decision to boycott the 1980 Olympics. “At the time, I would’ve given anything in the world to be locked in a wrestling room with him,” he told the Washington Post. “I’d probably be in prison today.”

Carter’s decision, a reaction to the Soviet invasion of Afghanistan, was popular at the time. But Mills, who was personally affected by it, says Carter “stole” and is guilty of “ruining” his life. Mills instead became a coach, and injuries kept him from the 1984 Olympics. And he wasn’t alone. The Post talks to other athletes, such as rower Anita DeFrantz, who earned a bronze in 1976 and temporarily left her full-time job to go for gold in 1980.

The end of the Carter era has been a cause for reflection about more than just his presidency. And rightly so, in the case of these athletes. They never had the chance to realize their Olympic potential, a demonstration of human performance that entrances the world every four years. How best to recognize the athletes caught in this strange twilight zone is uncertain. But the least we can do is not forget them.