Hello,

What I could suggest here is to SSH to your server whenever you notice poor performance and check the running processes and the resource utilization, maybe you get more users during this period of time and the load on the server increases.

You can take a look at this answer here on how to find the processes that are consuming the most server resources:

https://www.digitalocean.com/community/questions/how-to-find-the-processes-that-are-consuming-the-most-server-resources

Hope that this helps!
Regards,
Bobby