A few things to check:

Try loading an image or other static file from your site directly. Does it load quickly or do you still see a slow response? If this loads quickly than the problem is likely not with your main web server configuration but is related to your PHP configuration or your WordPress plugins.

Try disabling all your WordPress plugins and then re-enabling them one at a time. If after enabling one you see the slowdown return, you've likely isolated the problem.

While using a control panel like VestaCP makes it easier to do normal administration of your server, most control panels use very specific and non-standard configurations for the services used to support a secure multi-user environment. These differences can often make troubleshooting when you do encounter a problem more difficult as a lot of documentation out there related to the core services being used are written based on the default configurations provided by the Linux distribution and not based on the modified ones provided when you install a control panel suite.