How far is he or she traveling for the visitation.
I myself, travel 1000 miles, and I can tell you, there have been times, that other than flapping my arms and trying to fly there myself, there was absolutely nothing I could do to be on time. If an airport closes, etc, I can tell you from first hand experience, the judge will NOT look upon the custodial parent fondly if they have refused the visitation for being late.
That said, your circumstances are likely the deciding factor. If your ex is simply not showing up on time for no great reason, and never calls, that's another story. If that's the case, go to court, show your evidence, and explain have the visitation changed.