The system isn't that complicated, really. In short:
Running for extended periods of time make you tired slowly
Fighting for an extended period of time makes you tired slowly.
Running fight to fight, not taking any time to rest in between, makes you tired moderately quickly
Carrying more than twice your max encumbrance makes you tired quickly. I'm not sure if this scales based on how much over.
Walking while massively overloaded, then fighting while still massively overloaded, makes you tired insanely quickly.
I've never seen the exhaustion system affect anyone not doing one of these five things, and they all seem reasonable to me. If there are things outside of those that are adding to fatigue, then yes, I would agree it needs more tweaking. This is based on the times I've seen it kick in with my characters.