I know there’s a fuller reason, I don’t know.

It’s not just because he wanted to keep Harry a slave.

I know the Dursley’s hated magic but they hated Harry equally, why weren’t they pleased at the offer of Harry going to stay at Hogwarts?

So they don’t have to take responsibility for him?