They shouldn't. They should always be sending to the registration page. This is working fine on our demo site and in my test installs fine.
My guess is CB Invites is not accessible to public users due to the Itemid in the invite link. Create a Joomla menu item with a menu type of Community Builder > Plugins then for Plugin select CB Invites and leave that menu item as public access. Its menu id should then be used for the Itemid in the URL and for SEF construction of the URL allowing access.