DES MOINES, Iowa — Four voters and a Hispanic civil rights group sued Iowa’s top election official, alleging he infringed upon their rights when he directed election workers to challenge some voters’ ballots in an attempt to keep ineligible noncitizens from illegally voting.

The American Civil Liberties Union of Iowa filed a legal challenge in federal court late Wednesday on behalf of four individuals flagged by Iowa Secretary of State Paul Pate as registered voters who might not be citizens. They are naturalized citizens, according to the complaint. The suit also names some of Iowa’s county auditors, who are tasked with facilitating elections.

Concern about elections being undermined by noncitizen voting has been a focus of political messaging this year from Republican presidential nominee and former President Donald Trump and other Republicans, even though such voting is rare in American elections.

Pate’s office said last week that it provided county auditors with a list of 2,022 people who told the state’s Department of Transportation that they are not citizens but subsequently registered to vote or voted. Since those individuals may have become naturalized citizens in the lapsed time, Pate’s office told county elections officials to challenge their ballots and have them cast a provisional ballot instead.

They would have seven days — one more than usual because of a federal holiday — to provide proof of their citizenship status so that their ballot is counted.

Orcun Selcuk is one new voter who registered last year, a day after he became a U.S. citizen, and has since voted in elections without issue. He voted early this month but recently received a letter stating that his ballot was being challenged.

Now, the native of Turkey who settled in Decorah fears other new citizens will be intimidated.

“As a political science professor, I understand and feel deeply about the importance of voting in a democracy,” he said.

The ACLU is also representing the League of United Latin American Citizens, or LULAC, of Iowa.

It is illegal for non-U.S. citizens to vote in federal elections, but there is no evidence that it is occurring in significant numbers, though Iowa and some other states have identified dozens of such cases.

Before the lawsuit was filed, Pate told reporters at a news conference Wednesday that the DOT list is the “only list that we have available to us” without access to federal immigration records.

“We’re balancing this process. We want everyone to be able to vote. That’s why none of them have been taken off the voter rolls,” he said. But “we do owe an obligation to make sure that they are citizens now.”

With early voting well underway and just days to go before the Nov. 5 election, the lawsuit asks for the list to be rescinded and voters on it to not be challenged on this basis. It alleges Iowa’s election officials are burdening the right to vote and discriminating against naturalized citizens, treating these voters differently from others in violation of their constitutional right to equal protection.

Rita Bettis Austen, ACLU of Iowa’s legal director, said Thursday that Pate is “spearheading an effort to disenfranchise Iowa U.S. citizens” when he should be protecting voting rights.

“He is fueling a false narrative about voter fraud by noncitizens and laying the groundwork to undermine confidence in the election,” she said.