It's not clear why you'd want to put yourself on your employer's "ess" list arguing about this. Bottom line is that even if the employer wanted to cede to your wish that you take the leave unpaid, it's free to refuse to approve any vacation time in future even if you've earned it.
You're free to file a complaint with the federal DOL about it, but I wouldn't hold your breath that they'd consider this punitive in any way. You also risk the employer deciding to change its policies (say, telling folks if they haven't actually earned the leave despite the beginning-of-year banking that makes it easier for accounting purposes, that they won't be allowed to use it until they've earned it). You also risk them using a pretext to can your behind down the road, having put yourself on the aforementioned "ess" list.
"... it seems pretty unjust that if I chose to quit the day after I return I would have to pay them several thousand dollars to compensate them for time I never wanted to use in the first place."Might be "unjust" but it wouldn't be unlawful.