As someone who also has an intherited IRA, the 10% early withdrawal penalty does not apply to withdrawals from such an account. In fact, you generally have to take a minimum withdrawal from such an account on an annual basis, even if under age 59.5. If you are being charged a penalty it is is because you did not report the withdrawal on your tax return for the year in question and did not pay the taxes when due.
You almost ALWAYS have to pay ordinary income taxes on any IRA withdrawal, unless it's a Roth IRA. Doesn't matter whether you are disabled or live on Mars. (OK, you might be out of reach of the IRS on Mars.) About the only exception that I can think of is if the income from the IRA account was the only income you had in that tax year and it was little enough that you would not have otherwise been liable for any income taxes. Given that you mention that you have a full-time job, I doubt your income was low enough. If the financial institution sent you a 1099-R, then it was not a Roth IRA.
I don't know why you think that being disabled exempts you from the payment of income taxes. I know people on Social Security Disability Income (SSDI) who have to pay taxes on at least some of that income. I do know that the 10% penalty for an early withdrawal (before age 59.5 years) from a traditional IRA (not inherited) can be waived or not applied when someone is deemed disabled. But the disabled person still owes ordinary income taxes on the withdawal even if they don't owe the 10% early withdrawal penalty. If a person making such a withdrawal failed to report it on their tax return for the year in question, then I could see that peson potentially being liable for some significant negligence penalties and interest.
I suggest that you find a tax pro or attorney to help you with your dealings with the IRS if you are unable to pay all of what the IRS wants. But, I don't think you are going to get out of this one without some significant financial pain. Moreover, the next time someone tells you that the disabled don't pay income taxes on certain kinds of income, I suggest that you ask them for the source of their information before submitting a tax return excluding that income, particarly when you received a 1099 reporting it.