Time_Stomped: It sounds like you're describing something that happened to me after only a very short time here. A couple of other newbies and I decided we wanted to explore the great big world and went in search of the Village of Barovia and then the Port. Of course we found we were over-matched and had a terrible struggle getting there. We also had a similar hard time getting back and were so relieved to be in Vallaki again that we immediately sighed and logged out.
When we logged back in, our "Faction Base" lo and behold had been changed to the Mist Camp, rather than the Vistani Camp. (This may have since been amended or it may not always work that way.) We had to make our way back a second time because we didn't realize our faction base had been changed until we were already in.
Not fun. Once we made it the second time I prayed at that Ezra Shrine because I wanted 100 percent to be sure it wouldn't do it to me again at the next login.
(I don't think, BTW, that the challenges are exactly the same going both directions because getting past the mage ogres IMO is more difficult coming from Barovia than coming from Vallaki because of how they are positioned. Sometimes they are virtually in the transition when you come from Barovia.)
Be that as it may, really, what worked for us was simply to travel in daylight and run as hard as we could, especially if you're a fighter. The ogres at the narrow squeeze or whatever it is, I've stealthed by, but I don't know if a fighter can. You'll probably get hit, but it is possible to keep running and be okay. Then rest up at one of the safe places along the way like Midway Haven.
It's not a fun journey for a low level, though, and I still run most of the time when I'm going through there. If you have access to a potion of cowardice, it's a lifesaver.