Marge is close, but I'm pretty sure Petunia hated him the most because she always insulted Lily (his mom) to his face.
Marge did too, yes, but Lily is Petunia's sister, not Marge's.
Vernon and Marge both took joy in hurting Harry, but Petunia definitely hated him the most because she's the only one who Lily is blood related to and she had a strong hatred for her sister, so her nephew is no question.