I wonder if someone could answer my questions about TTFB:

What can be done on the server to speed up TTFB for a first time visitor? The main issue is TTFB for the very first visitor. (I don't use CDN but I tested locally)

when I PING I get in average 0.045ms for the IP, so that is awesome! There is no domain connected to the IP.
But when I go to the IP address with Chrome TTFB is going up and down from 1.5s to 3.5s from time to time.

The file is 123kb and content download is around 0,5-1ms. Initial connection is 1,7ms and stalled is 1,6ms

When I turn on simple gzip cache in nginx.conf the file that was 123kb is now only 18kb. DNS lookup, connecting and sending are 0ms. TTFB is between 1.1 to 1,6s and content download is 0-1ms.

I'm not sure where to start searching for the issue?
Redis frontend cache and session is installed and working, haven't tried to see what happens with the TTFB if Redis is turned on yet because the main issue is not TTFB when the site is cached, the main issue is TTFB for the very first visitor.

Maybe not much I can do about it?

i run a droplet with 4gb ram, centos 7, nginx 1.6.2, php 5.6 with opcache and mariadb 10.
I run magento on the droplet.

Any suggestions for what and where I should look are grateful !!