Speed should not be a major issue even with current releases. The current release of CB Activity has been performance tested against over 1 million database entries for all CB Activity tables (activity, comments, likes, and hidden entries). How slow is it loading for you? Additionally as Beat provided CB Activity 6.0.0 is bringing even more performance improvements.
Currently the main thing that can slow CB Activity is CB Privacy on the Recent Activity page. When used it checks privacy of each activity entry before display as there isn't an optimized way to do this from SQL yet (likely still won't be in 6.0.0 until CB Privacy is redesigned). So for example a user with 50 posts set to private is output to Recent Activity this results in CB Activity removing all 50 posts and trying to fetch more, but what happens if that happens again? It just keeps going until it can find at least 1 page of posts. It can be a problem. In cases where you need strict privacy I HIGHLY recommend NOT using Recent Activity page.