I think many people are lazy. Some have no clue what RSVP means and think that it means call if you are coming. There are probably a few people who want to attend, but are embarassed to say that they will not attend. Who knows why you are getting no response? I would be a little embarrassed if my mom was throwing me birthday parties every year up to 35, but it sounds as if your son likes it. I don't mind a party or two on my birthday, but I would enjoy solitude on my birthday, just as well. It can be a great time to reflect on life, where one has been and where one is going.
As for dress, I often try to fit the scene, but I tire of dressing. I think people place too much emphasis on the social aspects of dress. I am always amazed at how one gets stared at if not dressed for church, as if God cares more for those who dress up nicely, or as if the well-dressed folks get a front pew in Heaven. "If you don't know me, look inside. Don't look at the outside." -- Wally Meyer's Grandfather
Of all of the advice on this board, I like Unplugged's the best for this situation. Party planning should not be a stressful thing.