I think there is probably a little more to It.

Aunt Petunia wanted to be a witch herself and her hatred is more out of jealousy than anything else.

I do think that without the letter they may have just taken him to an orphanage right away and not intervened. Simply because they don’t seem like the type of people to care about anyone else but themselves. The letter basically made them not want to have blood on their hands if he would die since they did not protect him.

There’s also a big difference between basically condemning an infant to death and treating him like crap when he grows up. They were mean people, but i don’t think they were THAT stone cold cruel

Does the UK still have orphanages? I would think a foster home would be more likely. Either way, Petunia didn’t actually want Harry to die, and Dumbledore explained that he would die if they didn’t keep him.