Was my boyfriend's joke funny or hurtful?

It does seem quite badly placed, but I wouldn't dwell upon it too much, it was only a joke after all. If you feel that hurt by it, just tell him you don't find things like that funny, and he will probably knock it off.