Like Kobal2 says, it’s a pretty complicated, dug in situation, and certainly there’s issues on both sides.
If people here knew the word Romany at all they’d immediately think ‘travellers’, and there’s a significant proportion of travellers who are, frankly, causing a lot of problems. It’s not a lifestyle that can be followed while also following the rules of the rest of society.
As an example; only last week one of the local hospitals had to cancel their annual fundraiser because a group of travellers moved in to the fair site the day before and refused to leave, even though there’s actually a dedicated traveller site less than 20 miles away and there was a huge banner announcing the event on the site entrance. A legal eviction will take weeks if not months. Bear in mind that this is England, it’s a non-profit community hospital that would be treating the travellers free if they needed it, so actions like that are not a good way to win friends. And yes, there’s no escaping it, when one of these groups move in, the local petty crime rate often shoots up.
It’s also a very enclosed society, and puts effort into keeping it that way. The traveller community frequently takes their kids (daughters especially) out of school at a very young age, then they’re typically married in their teens and encouraged to have kids asap. They’ll marry within the community, have almost no contact with those outside it, and are not given the opportunity to get skills enabling them to do anything else.
A friend who did some council sponsored outreach a few years ago with a local Romany group said it seemed like a deliberate attempt to keep the kids in the community by making sure they weren’t equipped to leave. She was working with some kids who’d enjoyed school and learning, but had only been allowed to go when there wasn’t much going on at home, then were taken out for good at 15 (compulsory education is supposed to last until 18 here), because their parents didn’t want them hanging round with outsiders and besides, they were going to be married in a few years and it’s the man’s role to bring in money. They’re not entirely being marginalised by society, quite a bit of it is deliberate separation, they seem to want it to be an ‘Us’ and ‘Them’. I personally think there’s a bit of the same logic as the Westboro trolls for some of the groups; make people dislike you and the kids will carry on the traditions, because they’ll be unwelcome elsewhere.
Are all Romany dodgy travelling uneducated thieves? No, of course not. But there is a significant chunk of people, including the most visible section of the Romany, who ‘other’ the hell out of outsiders, so it’s not surprising that’s often mutual. And yes, some non-Romany are just straight up racist and would refuse to employ/house/befriend someone who’s got some heritage from the group but doesn’t join in with any of the problematic aspects of the culture, which also means there’s pressure for Romany not following the traveller lifestyle to not mention their background, heightening the perception that they’re all travellers. It’s been going on for centuries, time for a lot of issues to build up all round.