If it's in your divorce decree that your ex-husband has visitation, then he has visitation.
So far as I know, 5 year olds don't make up the rules.
I'm assuming your 5 year old isn't running the show entirely at your house? If so, you've got some issues you need to resolve in my opinion.
If you allow it, your 5 year old probably would prefer not to go to school, eat icecream for dinner, stay up late, etc etc. You really should NOT be allowing your 5 year old to make whatever decisions he/she wants in order to "be happy". That's just crazy!
By the way, you ex has the right to visitation, not the obligation. Child support is a completely separate thing from visitation by the way. In fact, the child support is the childs right, not your right to decide you don't want to take it. Were that the case, should you lose your job, next thing you know, the taxpayer (me and the other posters here) are supporting your child.
Instead of "fighting this", I suggest you tell the 5 year old what he is going to do, not request what he wants, and bend your life around his every desire.