It's quite simple really (if indeed there are a significant proportion of men 'dropping out' of society), if you make an environment hostile to a particular organism, don't be surprised if the number of organisms in that environment dwindle.
If you subject a baby to exposure, then don't be surprised if it dies.
If you build a town on a mosquito infested marsh, then don't be surprised if your population ends up with malaria and becomes sick.
If you increase the level of acidity in a lake by filling it with toxic pollutants, then don't be surprised if the number of fish living in it plummets, and the remainder are left with an abnormal physiology.
If you fill a society with misandric propaganda, and gynophillic propaganda, then don't be surprised if the number of men and male men dwindle.
If you create an environment that is hostile to men, then you will increase the rate of failure of the majority to such an extent that you will create a dichotomy in the group: compliant female men, and rebellious super-male men. Those are the only two groups that will survive the harsh anti-male conditions of society by either complying (the female men), or becoming stronger (the super male men).
As for those who think that having a population of female-men is a good thing, then look at what happens when you decrease the level of testosterone in a person, they end up becoming riddled with neuroses, anxiety, depression etc (1), all of which are anathemas to virile erections. Thus if you end up with a feminised population, you ultimately end up with no population, as it is incapable of breeding, and thus sustaining itself.
(1) And, oddly (or perhaps not), there is a negative correlation between testosterone and psychopathy. i.e. the more womanly a person is, the more psychopathic they are likely to become.