Can it be possible to add users to CB Lists WITHOUT adding them to a Joomla! User Group?

Because I don't really need so many permissions, and too many User Groups slow down Joomla! in general. However, I need them members displayed in lists. What are my options?