Emailing time is affected both by your own website host and the recipient's email server. Both can be almost instantaneous or only check for email once every XX minutes, hours, days or months... although I doubt any actually wait months or days.
A little testing should tell where the problem lies, whether it's with your server's sending time or their receiving time or both.
Another thing that can affect it is any outages inbetween that are rerouted, such as a disaster that brings down servers on your normal routed path.