Points that I do not necessarily agree with are underlined.
i use the Spell Fear often in my horror scenes, and i can reflect that the lose of control is necessary. to often i run into players "who just dont grasp" the concept of horror, and fear more or less gets that point across. from my personal experience, horror can only be achieved when the victim player has no control over the situation/encounter, because all those who can challenge the situation for control, will do so (I.E. High level characters). fear is a loss of control and there for is a wonderful tool in my encounters. as with my domination ability, ive had people ignore its effects after the 30 second timer runs out. according to the P&P D&D description of the spell, the domination would last a number of days equal to the casters level (obviously unrealistic amount of time). 30 seconds isnt long enough to muster any amount of RP, and i have to rely on the players to RP out the effects longer. you would be amazed how many avoid/ignore it.
From my experience, you cannot really begin to try to force RP, even horror roleplay on anyone...it can be frustrating for the other player. There are many factors that might influence why that certain player "Just doesn't grasp" the concept of horror. They could be a younger player, or they might not really be here for the horror aspect of our server...that is alright.
Loss of control is not necessary to induce horror. Horror is being placed in a situation that will induce fear and panic, most often because there is opportunity for loss-of-life. However, (And not singling anyone out when I say this)
no player has the right to force any amount of roleplay onto another player, so therefore it would be better that the CHARACTER loses control, not the player.
I admit that a certain degree of loss of control can generate fear. But when a player completely loses control of what can be done with THEIR character, that creates FRUSTRATION and not horror or fear. I guarantee you even if they can still move around 90% of the characters under the effects are going to experience fear, and their players as well.
Horror can often be achieved when the other player has control of the situation. Take Destiny's werewolf for example. Everyone who encounters it has control. They can either run, or stand their ground...most who do the latter tend to become a spot on the ground. Going up against something large and nearly unstoppable that wants to rip you apart is horror...and there is no loss of control.
When I played Vicho Cain, I would attack the outskirts a few times. Usually I would give fair warning that he was there, and those that didn't want to participate would run inside. Those who stayed faced PKs, fireballs and magic missles, and after casting a few spells hasted I would go invisible and hide, then move to avoid being shredded. Going up against a methodical villian who is clever and cunning is horror...and there is no loss of control.
When I played out my torture scenes with Vicho Cain, my victim always got a tell beforehand and in most cases I even asked them ahead of them if they'd be for this type of RP. Most were excited. He would proceed to strap them down on his machine and wake them up (They could break free with a high strength roll). He would proceed to torture them, cut them open, play with their insides, and fill them with worms...etc. In this case, there was a loss of control...but for the PC...and NOT for the player.
A high-level, as you put it isn't necessarily attempting to control a situation if they move to defy your PC, whom is a high level sorcerer. There could be any number of reasons why they would confront you. You have many tools at your disposal being a vampire, and as so horror would be somehow making those high-level characters feel powerless to stop you...not attempting to detract them from the situation entirely based on their level.
As for 30 seconds not being enough time to generate RP, I find that statement to be false. When Vicho would dominate someone, I would have an emote ready to fire off. Something along the line of a soothing voice, and having them unsheath their blade and turn it on themselves. Or, if there was a friend present I would have them draw the sword and stalk menacingly towards them. All this could be done in 30 seconds, and generates roleplay which will unfold even AFTER the spell duration subsides.
The point of my post here is that despite anything that is going on screen, it is up to the player to feel fear based on what is going on. Some people will feel it, some won't, but you can't ever attempt to force any degree of RP on another player. You can only be suggestive about it. That being said, replacing the horrible mechanic for a player's control over their character will still initiate fear into those players who want to feel afraid. But in all cases the factor of frustration will be gone, as it should be.
Cheers.