“Yes, we have white folks out there, about one in every ten thousand, and they don't think anything of it and neither do we. You can't hide from the universe. You're going to be tramped under with color – all the colors of the rainbow!” And he understood then that that was exactly what they feared.
In 1957's short story “All The Colors of the Rainbow”, Leigh Brackett has Earth join something like the Federation soon after entering the Space Age. Many aliens have been coming down to Earth to help us eventually reach the organization's technological level and to clean up our environment. That's how an alien couple finds itself driving thru a small American town of the South which is proud of never having had any issue with racial integration because, well, black folks had all decided they wanted to go live elsewhere so it never was an issue. But... The couple looks like white humans, except that their skin is green, and that is enough for the town to feel threatened – and inferior – to these two 'colored' people. Things do not go well for anybody involved.
It is but one of the stories in Brackett's story collection “Shannach the Last”, published by Haffner Press in 2011. It makes you remember that Brackett's storytelling ranged far beyond her tales of Eric John Stark's Mars. And even one of the last stories she wrote of that Mars in 1963 puts the lie to this year's assertion by certain people regarding this year's Hugo Award that political statements in SF are a recent thing. I highly recommend that you read this book.
I wish I had met Brackett.