TTFB can be affected by most areas of your stack, ranging from DNS to the Web Server, PHP to the database server, and everything the falls between.
Reducing TTFB can generally be achieved by implementing some form of caching and using a CDN. I would recommend either WP Super Cache or W3 Total Cache (the more complex of the two).
If you want super-simple, use WP Super Cache.
As far as a CDN, you have numerous options, though I'd start with CloudFlare as it's free and rather easy to setup.
The physical stack can also come in to play, so making such your web server, whether it's Apache or NGINX -- the database server, whether MySQL, MariaDB, or Percona -- and PHP (where possible) is optimized and tuned is definitely critical. WordPress relies on PHP and MySQL heavily, so beyond any sort of caching, making sure the database server is tuned for the best possible performance is going to be an absolute must.
I would start by setting up CloudFlare and getting one (and only one) of those caching plugins setup. Once you have, re-test and see where you're at and we can troubleshoot from there.
Feel free to tag me in your reply by using @jtittle.