I have fixed this issue but wanted to share this so that anyone else having the same issue as me might have it fixed.

If you've set up CB Connect and the avatar is not being pulled through, check your CB Configuration->Images and make sure that you have an image library (eg. GD2) enabled and working on your server.

This feature is required for using avatars even without CB Connect but somehow I missed this. There are other fixes for this issue, but no one has mentioned this specific fix before.

Hope this helps someone.