Subtractive masculinity is when things are only defined as "male" or "manly" or "masculine", and the people doing them as successful men, if there are few or no, women and girls doing them. Subtractive ideas of masculinity can lead to active sexism towards women and girls trying to enter a field, because their presence becomes a threat to the masculinity of the male participants.
The term may have been coined in a 2007 blog post by Doctor Science:
One of the best illustrations of how subtractive masculinity works is in a old, not all that good science fiction story, "The Last Man" (written by Wallace West in 1929, anthologized in The Pocket Book of Science Fiction). My copy seems to have disappeared (or crumbled into dust), but IIRC the narrator talks about how women's energy and ambition couldn't be suppressed forever, and they moved into one field after another until all that was left for men to consider important were sports and war. And then war became unthinkable, and women got into sports, and then all the men just died out, useless (our hero is a throwback in a zoo).