I am not a lawyer.
My cousin was in a similar situation. His wife left him and the kids, one day. She conducted a lifestyle with drugs, booze and multiple sex partners.
The father (my cousin) had sole custody of the kids. He was in the military and had a will designating custody to his brother should something happen to him.
Sadly, my cousin died earlier this year. The mother went to court and slam-dunk, got custody of the kids. She also gets the considerable insurance benefits my cousin had in place to take care of his kids should something happen to him.
My cousin was not married, so there was no possibility of a new wife adopting the children.
My best suggestion to you is to talk to a family practice lawyer about your husband formally adopting your son. You could still continue the informal visitation with his biological father, but the bio-dad would have no legal rights to the boy, nor any obligation for support. From what you wrote, it seems to me that might just suit everyone just fine.
Finally, I am an adult adoptee. I am pleased to hear you are permitting some relationship between your son and his bio-dad. I think that's really great.