After Harry's parents were killed by Lord Voldemort, I wish his mum's parents were still around to raise him for a few years before dying and that leaves the Dursleys to raise Harry.

So, the grandparents have moved to Privet Drive before the Dursleys do because one of the grandparents had a new job or the couple wants to live in a better place. And them Vernon and Petunia move into Number 4 across the street from them.

While Harry is still a baby, the grandparents take him in (which I think is a safer idea) and then by the time Harry is 6 or 7, the grandparents have passed away and so it only leaves the Dursleys to raise Harry. Mrs. Figg is also made to be a friend of the grandparents.

The grandparents have left several stuff in their will to the Dursleys, but most things are left to Harry, though they are kept by Mrs. Figg until he reaches adulthood, she also keeps Harry's toys and other belongings so the Dursleys can't do anything to them and so Harry could use them in the times he's left with Mrs. Figg.

And with Harry living under the Dursleys, he then begins having his character shaped up.