It's only going to show the paginated ones when scrolling left or right. It can't show images that it doesn't know even exists. The query stops it from grabbing more rows for performance reasons than the paging limit. You can turn off pagination for media within the galleries parameters under its Media > Paging tab. There's no separate paging parameters for album media.
Have added a feature ticket to add parameters to control paging for album media so non-album media and album media can have different paging settings.
forge.joomlapolis.com/issues/6219