There's no feature to turn off VAT requirement if VAT is supplied (e.g. you can't supply VAT and it ignore validation; VAT field it self is not mandatory). Invoice address validation is done in the below file, which you'd have to modify to remove that validation.

components/com_comprofiler/plugin/user/plug_cbpaidsubscriptions/plugin/cbsubstax/validations/tax_eu/validation.php

Have retested with and without mandatory invoice address and wasn't able to confirm your issue. With Company Name only supplied while leaving VAT empty it saves the invoice and allows the purchase to go through perfectly fine. It should only validate the VAT if supplied. Are you sure they're not just typing in VAT by mistake? You should have at least 2 tax rules as I understand it. 1 for EU consumer and 1 for EU business and should apply the tax rate accordingly.

If it is still having issues please provide the exact steps your taking including any sample data for invoice address (feel free to PM this information) so we can attempt to reproduce your issue.

EDIT: Switching customer invoicing off will switch entirely invoice addresses. Not what you wish, I guess.

The VAT field is never mandatory to fill-up in CBSubs invoice edits.

Not sure what exact issue you are reporting ?

In your tax rates, what setting do you have for "Methods to validate that the customer is a business (B2) for this tax" ?