A few things to check:

Try loading an image or other static file from your site directly. Does it load quickly or do you still see a slow response? If this loads quickly than the problem is likely not with your main web server configuration but is related to your PHP configuration or your WordPress plugins.

Try disabling all your WordPress plugins and then re-enabling them one at a time. If after enabling one you see the slowdown return, you’ve likely isolated the problem.

While using a control panel like VestaCP makes it easier to do normal administration of your server, most control panels use very specific and non-standard configurations for the services used to support a secure multi-user environment. These differences can often make troubleshooting when you do encounter a problem more difficult as a lot of documentation out there related to the core services being used are written based on the default configurations provided by the Linux distribution and not based on the modified ones provided when you install a control panel suite.

You said you are using Cloudflare services right?
In Cloudflare they share One single IP with many sites. If any of the sites with which they are sharing your IP is spammed then you can face this issue.
I am also using CloudFlare services on my website Top10league, but I am not using shared IP. Previously on one of my domains, I got penalties only because of this reason because my site was put in spammed Packet.
Get singular IP from Cloudflare & use premium Wordpress Plugin like WP- Rocket. It is costly but works very nicely. Apart from this, you can use SMUSH plugins to reduce the size of Images for higher speed. Just cancel the higher Ping rate on Server. Clear your Cache on a weekly basis.

You will see the results.