I'm not sure I follow.
Brother has full legal custody of his child--isn't that enough? Not clear in your post if you have court ordered role or not.
Child does not get to decide who has visitation --child is the subject of an order.
If Mom has visitation rights she is entitled to the exercise of same .
If Mom wants more visitation she seek same via order
If mom contends that dad is unable or unfit to be custodian and mom would be better for the child then mom raises that point in court.
I'm not sure what your role is in all this except to meddle and take sides?