"Catspaw", TOS (02x07)
Although I don't find "Catspaw" particularly scary, it's gotta be here considering that it was deliberately filmed for and aired on Halloween. After losing contact with a landing party, Kirk and his senior staff beam down to find a fog-covered arena of mystery, apparitions of witches warning them away in a threatening chant, and a gloomy gothic castle shrouded in the mist. Inside the castle, their lost landing party waits for them in a dungeon, under the control of a malicious "wizard".
"Macrocosm", ST-VOY (03x12)
Captain Janeway and Neelix return to a Voyager which is strangely empty, except for occasional noises deep in the interior, and see evidence that the crew left their stations abruptly. But there is something else on the ship that's alive...
"Night Terrors", ST-TNG (04x17)
The Enterprise-D is caught in a spatial anomaly that denies the crew the ability to really sleep. As they sink into hallucinations and violence, Deanna Troi -- who keeps hearing the voice within her head intone "EYES IN THE DARK, ONE MOON CIRCLING" -- tries to find an answer to what is happening. A nearby ship adrift, filled with the bodies of a crew that murdered itself, is an ominous reminder of what will happen if she doesn't.
"Whispers", ST-DS9 (02x14)
"Whispers opens with Miles O'Brien escaping from...Deep Space Nine, where his friends and coworkers have inexplicably begun treating him like an enemy in disguise. One of the many "abuse O'Brien" episodes of DS9, viewers witness poor Miles suffering cold distrust from first both his wife and his command, and then everyone -- even children. After using his engineering know-how and knowledge of the station's innards to escape, he looks to Starfleet Command for reprieve. In a related episode, "The Assignment", Mile's wife is possessed by a malevolent alien who wishes to attack Bajor...and if the chief doesn't assist the creature, it will kill his wife and daughter.
"Distant Voices", ST-DS9 (03x18)
Shortly after being physically attacked by an alien in his lab, Dr. Julian Bashir wakes up to a seemingly abandoned and crippled station. What's more, he's aging -- rapidly, and hears faint voices all around him. When he finally finds a few scattered members of the crew, they're acting uncharacteristically. When Bashir's failing faculties seem to align with the crew being killed by a monstrous assassin, the doctor realizes he is fighting for his sanity within his own head.
"One"/"Doctor's Orders" ST-VOY/ST-ENT (04x25 | 03x16)
An effective enough story that it was recycled between shows, "One" features Voyager entering an area of space dangerous to most of the life on the ship. Seven of Nine and the Emergency Medical Hologram are immune to the effect, but everyone else must be put into medical stasis. At first, matters go smoothly...but then the EMH is compromised, and Seven is left alone to battle both technical problems and the creeping terror of being alone for weeks on end. ENT re-used the story, but Seven's status as someone still establishing her own identity apart from the Borg collective made the original far more compelling, with Borg hallucinations driving Seven's panic. The filming of "One" used a lot of perspective shots that made it look like Seven was being followed or stalked.
"The Haunting of Deck Twelve", ST-VOY (06x25)
Voyager, for reasons undeclared to the viewer, is shutting down all engines and drifting through a nebula more mysterious than normal. While Starfleet's finest will be at their stations during the darkness, monitoring something Very Important, Neelix is assigned to take care of four children whom Voyager rescued. To entertain them, he tells them a "ghost story" about why it's important that the ship is powered down and at full alert, which mixes fact and fantasy and keeps the kids and viewers alike spellbound. There are comedic elements as well, because one of the children is older and keeps asking about the plot holes.
"Schisms", ST-TNG (06x05)
The Enterprise crew is overtaken by creeping paranoia, flashes of memory from a terrible place, and feelings of being out-of-time. When Crusher and Troi begin comparing notes, they realize there are common points of reference, and begin to suspect that the crew are being abducted in their sleep.
"Empok Nor", ST-DS9 (05x24)
A sudden crisis aboard Deep Space Nine forces a small team to raid an abandoned Cardassian outpost for supplies. Because the outpost is booby-trapped, mysterious Cardassian exile Garak comes along to watch for and disable any traps. But the station isn't quite abandoned, and as members of the team begin to be murdered one by one, a psychotropic toxin turns friends against one another. The experience is harrowing enough that a season later, a survivor's behavior is influenced by it while in another tense situation.
1. "Frame of Mind", ST-TNG (06x21). Commander Riker wakes up in an asylum, accused of having murdered a man. He has no memory of the event, and everyone treats him like he is insane. What's more....he is. What Riker experiences and the reality around him constantly conflict, and even when members of the Enterprise crew show up to check on him, they prove only to be part of the delusion. The episode is a complete mindscrew, keeping the viewer and Riker completely unsettled. "Frame of Mind" is the reason I made this list to begin with, and I went ahead and made it number one before anything else.
Back in the early 2000s, a guy named DarkMateria did three remix songs, using TNG clips and music -- one for Picard, one for Worf, and one for "Frame of Mind". I'm including a fan-made vid above using the music.