Wouldn't Slughorn want to ''collect him'', considering that James's father Fleamont was an accomplished potion maker? Slughorn was a potioneer himself, so he would value this. James was also a very gifted student. Did Slughorn not notice this?
Perhaps Slughorn disliked James's rulebreaking and bullying tendencies, and therefore didn't invite him. What are your thoughts?