Alfred Hitchcock's "Psycho!"?? Ring a bell? (I didn't mean to give away such a plot twist here, sorry.)
Maybe the easy gender identification through long hair, dresses makes guys focus in on these things and gets their wheels turning one of which might be saying something like "No physical threat coming from this person". then the lights go up, they turn around and the male brain goes into alert status because it just missed the potential threat ?
Sort of like the camoflauge falling away, then the brain wonders:"Why does this newly identified male-person need camoflauge? Are they hunting me?"
Nobody likes to be hunted, at least not without permission.
That's called "Tresspassing!"