My impression is that you're trying to understand. There are likely many reasons why women you've known and worked with did not assert that khaki pants were sexist:

-It's risky to push back against male-dominated decisions, such as forced dress code that favours male-shaped bodies.

-Women who push back or refuse may harm their career prospects, or be labelled as 'difficult' (a common misconception held against women who assert themselves in the workplace).

-Women may not have felt comfortable enough with you personally to assert that something at work is sexist.

Try this thought experiment (that the author has raised) - how comfortable would you feel by being forced (or strongly encouraged) to wear a dress at work? I imagine you would feel quite uncomfortable, both physically and mentally. Now switch this around to imagine a woman wearing typical male clothes such as khaki pants and boxy t-shirts. Put yourself in the mind and viewpoint of women colleagues and you will find your answers.

Finally, just because no woman has ever asserted something is sexist to you, does not mean the thing is not sexist. You may already be aware of this, but worth highlighting. I believe this last point is the trigger that has caused other commenters here to push back on you.