Georg Wenzel was standing outside a bank in the town of Pirna on a recent morning, lamenting how some German lawmakers want to ban the only political party he trusts.

A young woman walked by just as I asked Mr. Wenzel, a 67-year-old retiree, what he thought the biggest issues were for his vote. She was wearing a hijab.

Mr. Wenzel pointed at her. “That,” he said.

More than a million refugees from Africa and the Middle East have legally settled in Germany over the last decade, many of them Muslim. In towns like Pirna, in the country’s east near the Czech border, anger over immigration runs high. It has turned much of eastern Germany into a stronghold for the hard-right Alternative for Germany party, or AfD, which finished second in national elections in February, in part on the strength of its promises to seal borders and deport migrants.

The AfD has been classified as right-wing extremist by German intelligence, over its denigration of immigrants and what the government called an unconstitutional campaign to treat German residents differently depending on where they were born.

That designation has strengthened other parties’ resolve not to invite the AfD into government. It has also fueled a push by some lawmakers, including center-left Social Democrats who are partners in the current government, to ban the AfD entirely. Germany has outlawed political parties twice before, both times in the 1950s, banning a neo-Nazi party, which dissolved, and the Communist Party. America and its allies banned the party of Hitler soon after defeating Germany in 1945.

The push to eliminate the AfD is in its early stages. There is no guarantee it will succeed. Recent polls show about half of Germans support outlawing the party. But many lawmakers, including AfD opponents in the government, worry that the evidence is too weak for the nation’s constitutional court — which would make the ultimate decision — to agree.