Yes, this is very possible. The CB Ajax File Fieldtype plugin available to advanced subscriber will assist with the file aspect as it is specific to users profile just like Image fields expect with files. You can set this field to Read Only so they can't edit it. You can then set the field to not show on profile. The user could then download it from Profile Edit, but it'll never show on their actual profile.

You could also display the field using a Delimiter field after setting the field on a not shown on profile tab position. You'd use complex IF substitutions so only the profile owner can see the delimiter fields contents (the file field).