We Have Always Lived In The Castle by Shirley Jackson
It's a small book that takes very little time to read, but I don't recommend going through it too quickly because you'll miss the slow build-up of bizarreness (I know it's not a word, but it describes this book so well). Don't be in a hurry; read it in a few sittings.
What attracted me was the title. I am a pushover for books about houses so "We Have Always Lived In The Castle" was like a drug to an addict. The house does play a large role in the story, even after.....well, I'll let you read the book. Do. Read the book. I don't usually enjoy creepiness, but this was really good.