Site taking multiple attempts and a long time to load, even though it's optimized

I’ve optimized my site for load time. http://tools.pingdom.com says it loads at ~1 second.

However, some of my visitors have been telling me that my site has been taking multiple requests to load, and taking a long time to load. I’m not sure why that is, and I’m hoping someone can tell me how to diagnose this issue.

Here’s my site:

http://michaelscollins.com

Only thing I can think of off the top of my head is the SSL cert, but I had people telling me it was taking a while to load before I had that on.

Thanks