Great question. The honest truth is that there is no right answer. Let's say, for example, that you and your neighbor have two locally competing ISPs. Those two ISPs may get their bandwidth from different upstream providers. This means that you and your neighbor may follow different paths to get to your destination. One datacenter may be faster for your neighbor, another may be faster for you.
The other reality is that TTFB is generally less about latency and more about web application optimization. While latency absolutely does impact it, it is very commonly exaggerated as being the most relevant factor. Take, for example, a Wordpress site. A Wordpress site, on request, has to do all of this:
- Load MySQL database
- Load plugins
- Load theme
- Use database to interpret how to organize plugin/theme data
- Organize content based on plugin/theme instructions
- Return finished product to web server
When loading a Wordpress site, this is where your TTFB is mostly managed. That time spent waiting for the application to return it's finished value. The bulkier your website, the longer it takes. Streamlining that process by minimizing the amount of things that must be processed, as well as utilizing things like static page caching, can dramatically improve TTFB without worrying about physical latency.
I hope that helps :)