Prime Minister Justin Trudeau’s Liberal Party should be a shoo-in for a Parliamentary seat at the southern point of the island of Montreal.

The district has been a stronghold for his party for more than half a century. It was home to another Liberal prime minister a generation ago. The base for a former Liberal justice minister. An easy drive to Mr. Trudeau’s own redoubt in the city.

And yet, days before a special election on Monday to choose the district’s member of Canada’s Parliament, polls show a tight three-way contest. For many lifelong Liberals, the problem is clear: It is Mr. Trudeau himself.

“I am a Liberal supporter, but it’s almost like enough is enough,” Michael Altimas, 79, a retired city bus driver, said during a walk on a sunny day along the district’s long pedestrian commercial street. “For the most part, he’s been a good prime minister.”

“But he’s had nine years,” Mr. Altimas added, “and people are hearing often enough that he messed up and they don’t want to support him anymore.”

The election to fill a vacancy in the district has become a referendum on Mr. Trudeau, the once golden boy of Western leaders who is now fighting for his political survival. His own Liberal Party members are increasingly calling for him to step aside, worried that the party risks a drubbing in the next general election under the deeply unpopular leader.