Just a theory.

Personally, I think so. We already know that Horcrux’s have an influence on people, such as the Locket increasing Ron’s paranoia and jealousy, the Ring’s curse, the journal’s grooming, etc.

What if the little bit of Horcrux in Harry amplified the Dursley Family’s hatred and negativity towards Harry? Particularly Petunia?

Even though Petunia outwardly displayed hatred towards Lily and Harry, I don’t think it was necessarily towards them, more so magic as it is, because we know she still loved Lily, so to some degree also Harry. While she was jealous of Lily, she admits that she was still her sister and died in-part because she was a witch. So she had general distain towards magic as a whole to begin with, even more so when Lily died. So the Horcrux within Harry could’ve just increased this ten-fold. I don’t think it affected Dudley as much, if at all, and I think he was just rude.

And eventually, after Harry was no longer a Horcrux, they were all able to set aside their differences, albeit to an extremely low degree; granted, that bar was never particularly high anyway, but still.

What do you think?