Syrian and Muslim community leaders have issued a warning to the Prime Minister – stop stigmatising their communities with racist and heavy-handed actions.
They say linking New Zealanders sympathetic with rebels in Syria to al-Qaeda is unfounded.
A small number of New Zealanders have joined the fight in Syria – and Prime Minister John Key admits some could be radicalised by al-Qaeda.
"Well, that is the concern, which is why we either stop people from going or continue to monitor them," he says.
But claims New Zealanders are aligning with terrorists have been rubbished by Muslim leaders.
"We have no knowledge of any individual going and joining terrorist groups like al-Qaeda," says Syrian community leader Ali Akil. "Singling out a community puts them and risk and makes them vulnerable."
And the Government's been labelled racist and discriminatory for cancelling the passports of New Zealanders who've fought or planned to fight in Syria.
"Stripping people of their rights is injustice and that's the cause of radicalisation," says Mr Akil.
Syrian opposition is divided with rebels splitting, some joining al-Qaeda.
Recent suicide attacks by an Australian and a Briton were both in the name of al-Qaeda.
Mr Akil says two of the New Zealanders whose passports were revoked had no links to the terrorist group.
"Two brothers booked tickets and then their parents found out and asked the authorities to not allow them to travel."
Both New Zealand's spy agencies - the GCSB and SIS - have been involved with monitoring the Syrian rebels.
But charging them under terrorism laws could prove impossible.
"It's quite difficult to prove they've associated with those groups and not other groups," says Mr Key.
Mr Key's still leaving it open that some New Zealanders could be fighting for al-Qaeda, but equally admits it may not be so sinister - the Government talked to the families of those who planned to fight.
Mr Key says everyone eventually agreed it was simply "a silly move".