Despite having acted for over 60 years across film and theater, Brian Cox shot to a new level of fame after starring as the vicious Logan Roy in Succession.

Speaking to The Guardian, Cox revealed that his newfound fame cost him something very valuable: his privacy.

“I’ve lost my anonymity, and I’ve realized that was what was important to me,” he said. “I haven’t ever experienced anything like this. I mean, you ask for success in your work, and you get it, and then you have to deal with the consequences.”

He continued, “I’ve always valued my privacy, but that’s gone. I’ve been very lucky that I’ve had it for so long. You know, I’ve been doing this for over 60 years. And finally it’s over.”

In his memoir, the seasoned actor previously revealed that he had been given the chance to appear in Game of Thrones and Pirates of the Caribbean, though he turned down both opportunities.

“I’m often asked if I was offered a role in Game of Thrones – reason being that every other bugger was – and the answer is, yes, I was supposed to be a king called Robert Baratheon,” he wrote, per The Guardian. He went on to write that American productions tended to use British actors “to get them cheap.”

He also wrote that he turned down a well-paying role in Pirates that later went to Jonathan Pryce, though he had a rather blunt explanation as to why: “It would have been a moneyspinner but, of all the parts in the film, it was the most thankless.”

In the end, it was Succession that put him on the mainstream map. And with that came unexpected fame for the entire cast. He’s previously griped about being recognized more often in public by fans – especially those who want him to tell them his famous catch phrase from Succession.

“I’ve lost my anonymity, you know, I’ve lost all of that. That’s all gone,” he told Barley earlier this year. “I used to be the guy you’d say ‘Oh, you were in such and such or no, or no you weren’t’….but now I’m Logan Roy and everybody just wants me to tell them to fuck off!”

But, he added, “There’s no question it’s changed my life in so many ways.”