Polish people are said to be "very scared" to report being victims of crime according to a report in the Guardian.
Speaking ahead of Tuesday's home affairs select committee, Joanna Mludzinska, head of the Polish Social and Cultural Association, told the newspaper: "Very few people are reporting hate crimes at the moment. People are very scared, because often this occurs in the workplace and they are scared they might lose their job. Or it's from a neighbour and they don't want to cause more problems."
Mludzinska added: "They can't turn around and say: 'No, I don't have to go home because your government has said I am allowed to stay.' That puts you in a weaker position, doesn't it? It makes people vulnerable. It makes them scared to stand up for themselves, to properly report things."
She said the government's current position was "immoral" for failing to support the 2.9 million EU citizens in the UK.
Shortly after the EU referendum, there was a 50% rise in hate crime reports.
A Polish man was beaten to death, allegedly by a large group of teenagers in Harlow, Essex.
Professor Matthew Goodwin, professor of politics at the University of Kent, told the Guardian it was difficult to quantify the issue, despite the rise in hate crimes.
"The police measures for recording hate crime and how it is categorised have only recently become more sophisticated, so we're not really able to say whether things are actually getting worse or better," he said. "Historically, it's been an area where we haven't really invested much."
Polish workers living in Plymouth said they were even scared to speak their own language after the Brexit vote.
Shop assistant Karina Janek, 30, told the Plymouth Herald: "After English people voted to leave, we noticed a change. English people have not been so nice to us.
"They have been mean to Polish people. We do get lots of English customers, who are really nice, but when I go to the playground with my daughter, other mums aren't very nice.
"I don't understand because I work, my husband works, my daughter goes to school here, and we pay our taxes.
"There is no work for us in Poland, which is why we came to the UK."
In the lead up to the referendum, immigration was the most searched issue in the city, according to Google data. Some 80,000 people in the city voted Leave, compared to 53,458 for Remain.