It checks if the email exists first then tries to check if the email is valid. No, there's no way to isolate the 2 features for them to be separate.

I suppose so. You could try using a Code action with PHP to do all the comparisons and checks then perform a direct if necessary using the on before registration trigger. No idea how well it'll work though as it's not really designed for such usage. Maybe easier to just make a new trigger plugin and perform all your validation in it. You can also try the incubator project DB Lookup Field, but have little experience with it.

Please see the below tutorial regarding trigger based plugins. A installable example is provided for download at bottom of tutorial.

www.allmysocials.com/directory/tutorials/item/228-event-triggers

You'd basically use the before registration trigger and check if the email was already registered then trigger a $_PLUGIN error if it is to fail registration.