The temple is really exceptionally bugged about this. Mummies are constantly ambushing me while invisible. I've tried all the fixes people have suggested, like opening and closing the doors before going in, but they still do it constantly. And since they have fear auras and disease and disarm, it's a constant terrible nuisance to do anything in there.