This argument has always struck me as odd as in virtually every other discussion we would accept that the exception ‘proves the rule’.

This is category theory, the existence of exceptions means that the model is incomplete because it cannot categorize everyone. In this case, the exceptions prove that the rule cannot be binary, but must instead be bimodal to allow for the variation seen in the population.

Humans have two hands, except when they don’t due to something impacting fetal development.

Are you defining people without two hands as non-human, or are you admitting that defining humanity as exclusively two-handed will necessarily fail to account for all the exceptions to the rule?

Or just let the exceptions be exceptions with no social stigma rather than refusing to recognise that the vast majority of humans, and mammals, can be accurately identified as one of two distinct sexes.

Again, this is category theory. Exceptions mean you have forgotten to account for someone. Admitting that some people don’t fit neatly into the only two boxes you’ll recognize as legitimate is itself a form of social stigma that you perpetuate with your desire to “let exceptions be exceptions”.

All you have to do is recognize the obverse, that regardless of how vast the majority of allosexual folks and critters might be, it is not the totality.