Usually, the Server Response time is influenced by the optimization of the website as well as server optimization.
Now, let’s begin with the website. Since you are using a CMS - WordPress, you’ll be able to install some plugins to help you optimize your website. One of the first plugins you’ll need is a caching plugin. If you are on a WordPress website, I’ll recommend using W3-Total-Cache, it provides good caching capabilities and additionally, you can minify your CSS and js files. The second plugin you’ll need would be for optimizing images, I can’t remember the name of the plugin I’ve used in the past however if searched you should be able to easily find it.
Having said that, I’ll recommend using Google Page Insights, it’s a tool which scans your website and gives you suggestions on how to fix/optimize it. You can check out this free online tool here.
Now, onto the server optimization. First, I’ll recommend installing the latest PHP version at the moment - 7.4 and using FPM with it. You’ll need to install a server-side caching services as well, Opcache is a good choice in that.
You can additionally, install Redis and tie it up with your website, it should tremendously decrease the loading time of the website. There are plugins in WordPress to actually tie Redis between your server and website.