Got to remember, X2 was a big success. Twice or thrice the movie X1 was. Fox wanted another hit and didn't want to wait while their director/producer went off and made a possible block buster for a competing studio (it was panned).
Not to mention, to keep control of the franchise they needed to spit out a X-Men movie within two years. I don't think waiting for Singer to make and push Superman would have been prudent in maintaining that timeline, on the other hand, Disney hadn't bought Marvel yet, so there was no real threat that they'd try and repossess it. Perhaps renegotiate the terms.