That's not how Recent Activity works. It shows all activity the viewing user has access to. Since they've access to the content of a open group then they've access to that activity. Recommend using My Activity if you only want to see activity relevant to you. You can also use the following feature and a custom Asset for an Activity stream to better control what displays. For example you could enable the system action for outputting a follow button on groups. The user can then follow individual groups. Next you could output a stream with the "following" or "followingonly" asset to show activity of things they've followed.

BUT what about making it so that when a user joins a group, then he automatically follows that group?

Yes, that's doable. He doesn't need to join the group to follow it though. To follow on group join you'd need to use the below.

Global
Triggers: gj_onAfterJoinGroup
Type: CB Activity
User: Automatic
Access: Everybody
Action
Mode: Follow
Method: Create
Asset: groupjive.group.[var2_id]
Owner: [user_id]
Load By: By Asset and Owner

You'd then use "following" or "followingonly" asset in your stream. The following asset shows your activity in addition to activity you're following. The followingonly asset only shows activity you are following. You can of course combine multiple assets to output more or less activity as needed.