You do that and also assign a resolution that is lower than your current one. If the resolution matches your max resolution, it'll just go fullscreen. For example, my monitor is 1680x1050, and I run NWN at 1600x900. It doesn't matter that my native aspect ratio is 16:10 and I have it set to a 16:9 resolution, it just wants the next legitimate resolution down from what I've got.
What resolution are you running? Probably 1680x1050 or 1920x1080? Try 1600x900 if either of these.