Doesn't being in Stealth mode represent actively hiding, and attempting to remain unseen? If passive stealth were a thing, involuntarily, you would at minimum always roll Hide when idle. It isn't just an invisibility button, after-all. The roleplay is that one is actively concealing their presence, not wearing a camouflage cape.