Before I say anything else, I would strongly encourage you not to listen to your ex regarding legal issues.
Generally speaking, whether custodial or non-custodial, a parent who moves away typically is responsible for paying 100% of the cost of visitation travel and expense. It does not matter whether your child's other parent feels he was forced to move, had no choice, etc. You are not to blame, and therefore should not have to bear the burden of decisions he makes in that regard. They are hornswaggling you!
It is extremely generous of you to offer to pay for half of the travel expense for now. I would strongly suggest you ask for nothing less than 100% payment on their part. You may still agree to drive half way, but your ex should be responsible for reimbursing you for the expense, and you can carry that as far as you want. If you have to take time off of work to do this, he should pay for that, too.
My suggestion would be for you to work on a new visitation schedule, which allows him more time at once and fewer visits. For example, maybe 2 months on the summer and one weekend a month, as opposed to the every other weekend you have now. Depending on how he decides to get the children - wether he comes to get them, flies them to him, or pays you to bring them, the kids would probably be happier to go for longer periods.
It was not your decision to have him move - and no matter what he says, we all have choices. He could have found another line of work, he could have done several different things to remain in the area. You are being penalized for something you had nothing to do with and I would suggest you talk to an attorney about your rights.