Most likely your caching issue is due to Joomlas page caching. Page caching is meant primarily for static content sites. It does not work well in dynamic environments as it just caches the Joomla output buffer. The problem with this is it ends up displaying old form data and then posts it, which contains an antispam token that has already expired. Do not use page caching unless your site is primarily static. Instead look to optimize headers and images using something like JCH Optimizer. Module caching and more specific extension caching is also fine, but if using a 3rd party caching extension be sure it's set to ignore CB or at least ignore any CB page that has an HTML form.

The other strange thing is that I thought I would try using reCaptcha instead of the internal code I used in the past. Now I seem stuck on reCapthcha showing in the backend and internal code at the frontend. See attachments.

You're probably overriding the global setting from within the Captcha fields parameters in CB > Field Management.