Not sure how I made this happen. But I just realised that as I jumped from level 4 to 5 on my Hexblade Hagen Hardrade, that I lost 1 spell known per day. When he rests he only can cast 1 spell per day, but he was able to cast two per day before. I did change one of the spells he could cast at the relevel, so maybe it's that.
I might even need to do a remake because of that, but I mostly wanted to point that issue out.