No, not at all.
As you optimized PageSpeed Insight to 100/100, you done what most would say:
- Enable (gzip) compression.
- Enable and configure caching.
- Optimize images.
- Minify HTML.
- Minify CSS.
- Minify JS.
I myself done PageInsight optimization this weekend, and they're definitely very strict to improve your page load time.
We need to know we can't really make everything perfectly. For some type of performance you will 'sacrifice' other one.
First of all, I guess you know what it is but for future readers. TTFB is 'Time-To First Byte'. So basically how much time you need from making HTTP request to receiving first byte (to start to receive content).
What is the first thing that can make this effect? Well, Compression. The reason is it takes time to compress content before transferring it. If you really want to test things, disable it - then do the PageSpeed Insight. You'll get lower score definitely. Measure TTFB. It could be that it's much lower.
I guess that compression could be biggest bottleneck for TTFB. Other things does nothing to slow it down.
The second part we need to take in consideration is your Internet. Especially if you're not close to the server.
@newbie gave idea of nginx and CloudFlare.
For nginx - I'll not agree. It's basically the same thing that you use Apache or Nginx (especially for static content). You'll not get any performance boost from one or other.
Yet for CloudFlare - I'll agree. It offers you CDN possibilty, that can improve both page load and probably TTFB. I recommend you to set up it and try it. They have a free plan that will finish job for you.
In very basic lines - why does it improves it. Let's say that CDN caches static content on their servers and use that to answer your request. It's much faster because serves content from many servers around the world. You can get content from server much closer than your one.
Will upgrading server help... You have to test it. For example, maybe one more CPU can reduce compression time if that's the problem. Maybe no, I'm not sure and can't say yes or no.
If you got any more questions - feel free to ask. :)