It's just that other people judge and label you a racist when you truly are not.
Example...
I have nothing against people from other countries, religions, and so on. But I do not agree that they all should come here and drain our system.
That is NOT racist, yet many see it that way. But I have nothing against these people.

I don't want a relationship with someone from another race/culture/religion. Not because I have something against any of those, but because I know it doesn't work with me as it'd bring too many changes that I'm not willing to implement.
It's knowing who I am, what I need, what I stand for. Not racism. Yet many will see it that way when mentioned.

That's the problem with so called racism. It's gone too far. You cannot say anything anymore or you're a racist.