Finally had an electrician out today. Long story short it was due to low voltage on that circuit. So hat's off to everyone who called that. Turns out the tool I used to diagnose the problem is very low tech and rudimentary. It checks for something in the 110volt range... or in the 12v range, not anything specific. Electric-dude said it was right around 100volts. So while my meter read it as "correct", it wasn't "correct enough." LOL
Another light in the bathroom is actually on a different circuit, so instead of spending another chunk of money for him to diagnose the low voltage problem, etc etc. He just tapped the other (good) circuit and installed the fan and all is good.
I asked about dangers/issues/causes. He said there is no danger, if it was grounding out or a mouse chewed on it or something the circuit breaker would trip. Cause could be something like brittle/old wires (which is possible I guess, this house is old), or something like a loose ground wire. Looking back I should have probably noticed the low voltage thing because that switch is one of the ones that glows orange when off (so you can see it in the dark) and it was very dim. But now it's bright as it should be.
Thanks to everyone who offered help and suggestions.