While you may not prefer to hang around people from various groups, certain groups are more toxic than others, in the sense that for any fixed number of them, the more toxic groups pollute the space more than the less toxic ones. Perhaps the easiest way to see this is to ask whether a public space feels contaminated with only one or two members of the group present, as opposed to there needing to be quite a few of them before their disruptive influence is felt.
Ethnic and racial groups are of the less harmful type. If half of the people in attendance are black, you might be in for a bumpy ride. But just one or two black guys aren't going to act up. Same for Mexicans, Chinese, Jews, etc. The lesson is that entire races can't be so dysfunctional down to more or less the last member, or else they wouldn't be here still.
As for those who can ruin a space acting alone, bums come to mind. It just takes one of them aggressively panhandling, shouting crazy shit, looking filthy, pushing or carrying or wearing weird stuff, or storming around half-possessed. That puts most people right away into a defensive mindset and posture, preventing the openness and trust that are necessary for a public space to feel communal. This is why bums, a subset of the homeless, are so seriously targeted by businesses -- it just takes one to drive away most or all of the normals. Even liberal businesses like Starbucks often put locks or keypads on their bathroom doors so that only paying customers can use them.
Faggots are the other obvious case. It doesn't matter if there's only one around -- that buzzing, breathy voice with its sing-song, kiddie inflection broadcasts the sound of a hypersexualized child throughout the space. People may try to ignore it, but there are few things that sound more creepy than a 7 year-old boy trying to seduce all the other males within earshot.
Their juvenile Peter Pan tendencies also mean they are totally unaware of what level of voice is appropriate, what topics are and aren't appropriate, and so on. It's not uncommon to hear them ejaculating loudly about their fucked up sex lives, sexual innuendo, and the rest of their many neuroses. Everybody has to hear their problems and rush over to hug them, and like, is it seriously to much to ask to blare "Beautiful" by Christina Aguilera over the PA system? They're living through a continual emotional trainwreck, so it's not like the normals will catch much of a break from hearing about their airheaded anxieties.
And then there's the visual aspect, just as with bums. Black people don't dress in such a deviant way, as though triggering your disgust response. Queers not only dress in a campy and childishly provocative way, but they just look plain freakish to begin with, from their boyishly straight hips to their sissy mincing gait and alien-looking gay face. And yet we're just supposed to sit there and not notice how bizarre it is to see a 40 year-old wearing cut-off jean shorts and an Abercrombie t-shirt, while shaving his bald head in a lame attempt to hide his age, and staring out of dead eyes sunken into his mummified face.
Outright criminals are another clear-cut case -- even if there's just one of them who bust in to rob the store, to shoot up their workplace, or whatever, that's enough.
The common denominator here seems to be level of socialization. Bums and criminals choose to live outside of normal society, so they certainly aren't socialized in its ways. Homos are an interesting case because they don't choose to be unsocialized. Rather, it stems from their fundamental state of being stunted in childhood. You have to physically and psychologically mature before you can become sensitive to the social dynamics of normal adolescents and adults, and to gradually fit your way into their world. Being always held back before that transitional stage, queers are mostly incapable of becoming fully socialized.
The best that they can hope for is to stay in the closet and just keep quiet, kind of like well-behaved children when they're out in public -- incapable of taking part in the interactions around them, but still not disrupting things for everyone else. If we expect that of our children, then why not of our man-children?
Members of this or that racial / ethnic group are on average fully socialized, albeit to different norms and customs than for other racial / ethnic groups. But they don't come off as characteristically deviant. Certainly those different norms can lead to tension between groups, and even outright conflict. But that's not the same as everyday pollution of the in-group's public spaces.
Conservatives for the past 20 years have focused too much on between-group conflict stemming from differences in customs, like moving away from blacks and their culture. Meanwhile they've abandoned public spaces within their own community to the liberals, preferring instead to hunker down like a pussy in their home with the nuclear family unit only.
Facing no opposition from conservatives, who tend to react more strongly to notions about sanctity and taboo, the libs have encouraged all sorts of repulsive and deviant types to infest public spaces -- libraries, parks, book stores, coffee shops, fast food dining rooms, and so on. And the typical conservative could care less, hiding away from all that mess and browsing / purchasing only from Amazon, going to the fast food drive-thru, etc. Let somebody else worry about it, not me since I don't use that space anymore.
One good thing about a rising crime rate is that it brings the conservative weenies out of their bunkers and back into regular face-to-face relationships with their fellow neighbors and citizens, to provide and receive mutual aid and support.