One major difference that I've noticed with psychological fiction, that I don't see in books labeled just thriller or horror, is the feeling of mental uncertainty. Either the narrator is unreliable or events are taking place that makes the main character feel they are going crazy or out of control. The Girl on the Train and Gone Girl are great examples of the unreliable narrator. Also, some novels have the main character as the serial killer, bad guy, etc., and the reader can see into their heads. They may also be unreliable as well. It's really a very fascinating genre.
I agree with Novelbound that there is a certain feeling of uncertainty that comes with Psychological Fiction. I feel you can't really trust any of the events or people in the book because for you know, the narrator is lying to you or is completely insane and hallucinates. For horrors and thrillers, there are concrete ways to run away or defeat the scary stuff but what happens when they're in your own head? The narrator could go to the moon and back but the thoughts and hallucinations would still follow them and personally, I find that more horrifying than any gory horror.
For me, I notice some differences being that psychological fiction seems to focus more on the character's psyche than necessarily affecting the reader's psyche. Inevitably, both play into it, but psychological fiction plays more to the former.
"Books are unique portable magic."-Stephen King
Latest Review: "A Twist of Fate" by Mark W. Johnson