There's only so much space available in the modal window, which is typically squared or landscape since it's just 90% the size of the browser window. If the actual image it self has become squared then check your settings within CB Gallery > Parameters > Photos > Validation as I'm guessing you may have disabled maintain aspect ratio. As for the images on the page and not in the modal window they will always be squared as it's a preview container.