Part of why cats like you is probably because you don’t like them and seek them out. Most cats are wary of people directly trying to approach them without invitation or greeting. In general cats will find the person ignoring them the least threatening and most approachable.
It never fails that cats tend to flock toward the person that is allergic to them, because that person is typically entirely disinterested in them lol. Not being interested in the cats is likely why you make friends with them so quickly!
Many cats die every year from them actually, just like children. I am super vigilant about hiding mine out of the way so ours can’t see them to play with because I’m terrified of it happening. I really just need to replace them, but they’re the nice heavy wooden white ones and throwing them out seems like such a waste.