As recent events show, it does happen. I haven't personally run across any that devolved into the intended victim shooting the assailant, but that of course doesn't mean that it's never happened.
Some big city cops will warn you about things like not wearing blue and red in places where Crips and Bloods take the wearing of signature colors seriously.
In most places, it ain't like you're parading through Harlem wearing Klan robes. Hard to see where it would compromise an innocent victim's defensibility if there was no demonstrable intent by the wearer to provoke confrontation.
Basically, don't go to a Gators game at the home stadium and yell "Gay Goaters!" while everyone around you is yelling "Go Gators!" and you'll probably be OK.