You can still get it off, but you'll need a tiny flathead or pocket knife. You'll want to stick either in there along the side to try and pry up on the tab that is still sticking in there. If you cant get that to work, you might try prying the whole connector off with a flathead, if the connector is as brittle as it seems, the retaining tab might just snap off and allow the connector to slide off.
Another option is to just cut the wires and replace the connector, its already broken, since you're replacing the fuel pump you don't really need to get the old connector off.