The thing that drives me up the wall are the Ghostly Armors and Shadow Demons. These two enemies seem to swing the difficulty way up for the payoff. Everything else I've encountered, seems alright.Bittersweet feeling knowing I'm not just copping out on this. But right, exactly as you mentioned. Everything else seems almost too easy (the servants and animators, at least) in hopes to counterbalance the difficulty of the two enemies you mentioned. Doesn't do much but leave them as cannon fodder to annoy you, while the real problems knock you down a few inches.
The hidden areas are also interesting, and seem to match the surrounding dungeon areas naturally.
The area used to be locked off and you had to break in if you want to explore.