I kinda saw it more as, yes he would be willing to work with shady types, but he wouldn't 'buy' mags constant holier than thou, and woe is me routine that Xavier and the rest always do. And as such wouldn't trust him as much either, hence wouldn't want to work with him (also he's got more 'human' friends and family than most of the other characters )