Let me quote Zeropunctuation on this:
"You see, there are three kinds of horror games.
First, there’s the kind where you’re in a dark room and a guy in a spooky mask jumps out of a cupboard going “abloogy woogy woo!” - that would be your Doom 3.
Then there’s the kind where the guy in the spooky mask isn’t in a cupboard but standing right behind you and you just know he’s going to go “abloogy woogy woo” at some point but he doesn’t and you’re getting more and more tense but you don’t want to turn around because he might stick his <claw> in your eye - that would be your Silent Hill 2.
And then there are horror games where the guy in the spooky mask goes “abloogy woogy woo” while standing on the far side of a brightly lit room before walking slowly over to you plucking a violin and then slapping you in the face with a t-bone steak - that would be your Dead Space."
Zero-punctuation then goes onto say that the second one is the best. The best type of horror game is a horror experience that the user creates for himself and the game to mediate. Sending the player through a "spook ride" of jump-scares, in the sense of Dead Space or even Doom 3, is not how to do a horror game. Even then I would say there are a few good horror games that still don't achieve a certain effect, ie Amnesia.
In the case of amnesia, the reason I think it works is a sense of helplessness, the idea that you can't attack back and all you can do is hide, if you had a shotgun or even a rock, the game would be considerably less scary. Games like slender are scary primarily because it's the idea that there's something you can't see "following" you, the slender character itself wouldn't be that scary by itself to most gamers, but because of the free-roam style of the gameplay and the idea of the slenderman following you, it increases the scary experience. Still even with these games, I still am not sufficiently scared (Amnesia did scare me a bit but didn't scare me as much as it could have, Slenderman didn't scare me at all because I found a glitch in the game (FOUND) which made it not scary).
My idea for a perfect horror game is one in which has a few elements.
Non-linear play: The player needs to create his or her own horror experience, the game can't do it for the player. Making the game non-linear and a bit more open ended increases that idea of the player being the sole controller of the game. A game which simply tells you to go to the next room to be scared is not going to scare anyone.
Force the player to be scared: This may seem like an apparent conflicting idea with the prior statement, but it actually makes sense. If you can allow the player to be open ended (an example is to have many paths to an objective) but have EACH path scaring a player, then it will ultimately cause fear. For example, if you are told that you can either go to the dark room, that's almost pitch black, where monsters are, or told that you can go in the other hallway that's lighted, but monsters can come out of the walls, both of those alternatives are scary and the player feels he can choose to go to each one, feeling that he isn't being forced to be scared, but he is infact scaring himself (even if he IS being forced).
To be unseen: A particularly scary game moment was in no other than Fallout: New Vegas. In a certain quest you are tasked to clear out an area full of strong monsters called NightKin, who are essentially big guys with big hammers. What I didn't know is that they were all INVISIBLE, and knowing I had to clear these things which could literally just pop up at any moment and attack me, scared me a WHOLE lot, especially when walking in the hallways and seeing small blurs walk across the hallway. Having an enemy that is unseen (not completely invisible, but hard to spot) is EXTREMELY important for all horror games, almost all horror games utilize this.
Good comparison: Games can't scare you non-stop, otherwise you get used to being scared. You need to have points of calm and give the player a false sense of security, when the player finally feels safe and calm and lets his guard down, that's where you scare him. For example, have "safe camps" and the first two safe camps are actually safe and the player can't get harmed, then the third safe camp appears safe but there are actually monsters hiding in it, or something like that.
Don't be direct: Try not to just fling monsters at a player, only show 'parts" of threats, in some cases don't even have the threats hurt the player, but just scare them (something Amnesia does several times). Make the player wonder what's harmful and what isn't, what is real and what isn't.
Hindsight: It's nice to have some hindsight in a game. For example, I was playing an online horror game, I heard some random knocking on the wall but didn't really pay attention to it. It was only later in the game that I heard the knocking meant "inevitable death" and that a ghost was trying to kill me, having little hindsight things like that can really send shivers down your spine.
Those are just some tips I have for making a good horror game. A rather unused tactic is to have everything seem ok for the first few levels and then just ever-so-gradually have things turn evil, in a non-forced way where it appears that the "horror" aspect of the game isn't even the main focus of the game. Such as a shooter where on the second or third playthrough, things seem different and more horrifying.