A module would be the only solution. Create a Joomla menu item with menutype Community Builder > Plugin then for Plugin select CB GroupJive and select whatever action type you like (e.g. if you want the module on all groups page). Next assign your module to it as you would any Joomla menu item.

As module assignment is just based of Itemid (menu item) there isn't much you can do to stop that beyond making more menu items. If you create a menu item to each category those categories should use their respective menu items and won't display the module assigned to the all groups menu item.

I'm not using categories, you mean creating a link for each group?

Yes.

Then I couldn't display them thoghether...

They don't have to be on an accessible menu module. You can still use your current all groups menu item. Your other menu items would just be for their Itemid, but would not actually be displayed.

Sorry but I'm a bit disappointed, to me the ability to integrate things with joomla articles or modules looks quite a basic feature

That is just how Joomla works. Modules are assigned to pages using Itemid, which is the menu id. So to assign modules to specific pages you need menu items to those specific pages.