The Walrider segments, and to a lesser extent the crazed Doctor.
Digging deeper, the reason I think these are much scarier than the conventional segments is that they are unconventional based on both immersion, and game-wise.
What this means is that the way the enemies are designed, particularly the Walrider is so unknown (Not just visually) as opposed to just a bunch of crazed humans. I'm having a hard time spelling out why exactly I found him much more disturbing than the other enemies from a critical stand-point, and that's really the best I can do.
Spoiler below!
That's funny, because I found the Walrider segment to be probably the least scary in the game, although it was a lot of fun. I guess it's the lighting and open level design, plus the fact that the supernatural doesn't scare me that much, but in this segment I definitely agree the game is more thrilling than scary - which isn't a bad thing, this segment was quite effective and very fun in my opinion.
The doctor level plus the two brothers near the end, as well as a few of Chris Walker's appearances are definitely highlights, but I disagree with you that the normal encounters are not scary. I mean, I completely agree that the character design and just the idea of these deformed lunatics is not that scary in and of itself, but merely the fact that you are totally helpless. It taps into the most primal aspect of horror, but when you combine this with the excellent lighting and amazing sound design it really becomes something much more than the sum of its parts to me.
Consider the scariness of a well-dressed and well-groomed man running up to you with a knife in a crowded street in broad daylight. If you take all of the aspects apart one by one then you could conclude quite easily that that scenario is not scary, and it probably would not be in a video game. There are plenty of people around, plenty of avenues for escape, bright lighting, the assailant himself is not scary at all, in fact he just looks ordinary. But he's still trying to kill you, and your body's fight-or-flight kicks in. It doesn't care about a well developed story or fleshed out characters with horrifying backstories. It doesn't even care if there was any build up to this encounter or if the encounter was "well designed" so to speak. If you don't run - you die. That's the reason why I find it scary. That's not to say I find Outlast to be badly designed although the story and character development is clearly horrible.
But as I said, I understand if you didn't find it scary because you're desensitized. After all, an army of mooks running after you with guns trying to kill you in most other games is not scary at all. Hell, even games with some horrifying imagery and creepy sound design like Half-Life 2 end up not being scary simply because there really is nothing to fear from the enemies. But for some reason I just felt Outlast struck an amazing balance in this regard and I ended up being legitimately terrified throughout the entire game.