Ok, you sound exactly like I did when I was going through this and honestly, I don't think your situation will turn out any different because your ex sounds just like mine.
He won't be agreeable at mediation because it is a twisted way for him to keep controlling you and this situation. You have to pay for mediation, you have to pay for your attorneys fees, so whether you are paying those fees to sit in mediation and bang your head against the wall or go to court, you are still paying.
If mediation fails, you are not likely to get different results if you go back a second and third time and it isn't saving you money. I don't mean this about you, but I heard a saying ..."the definition of insanity is doing the same thing over and over and expecting a different outcome." Think about it.
When you go to mediation, your attorney needs to fight for you and what you want. YOU have say. You want a painfully detailed mediation agreement that outlines EVERYTHING you could possibly think of regarding custody, visitation, holidays, exchange times, exchange locations, who drives to drop child off, who drives to pick child up... summer vacation, etc... The more detailed it is the less likely you will have to go back to court in the future. Two hours for mediation really is not very long. I sat in mediation a full day. If you feel you are getting somewhere with it, by all means ask to have another session, but if your ex is fighting every single thing, don't waste your time or money and let a judge decide.
What has been the custody/visitaiton schedule during these two years? Was he court ordered to pay CS? If he was, time for your attorney to file for enfocement/contempt. You playing nice is great, but clearly it isn't getting you anywhere. Financial situation doesn't dictate primary custody. The courts like to maintain status qou so if the child has been with you this whole time and Dad every other weekend, that is what would very likely be ordered in court. Through mediation my ex was given EOW plus two nights for dinner a week. He was fighting for primary custody because he too did not want to pay child support. As soon as it was court ordered, he stopped the dinners, stopped the Friday night of the EOW and eventually stopped seeing them altogether. It wasn't about the children, it was about control and I am pretty sure that is what your ex is trying to do to you.