How do you think the Malfoys and Dursleys would act if they met?
I think the Dursleys would be torn between viewing them as fancy and upper class while also viewing them as freaks.
As for the Malfoys, they'd probably view the Dursleys as a prime example of how they view most Muggles.