Yes, SSL can be slow if it's not setup correctly. Contact your host regarding such large performance losses. Your site should also be enforcing SSL from the domain level. If you're enforcing SSL from Joomla ensure you've configured it to force SSL for frontend in Joomla global configuration and that live_site is empty in configuration.php to avoid http to https redirects.