I have been getting "Reduce server response time" issues with the Pagespeed Insights tool.
To test whether it is due to DNS, PHP, Wordpress or hosting issue, I duplicated my website at Digitalocean and pasted it in Google Cloud - Google Compute Engine instance.
Problem solved! In Google Cloud I'm able to get faster server response times and pass the Pagespeed test with flying colors even with the smallest VPS in gcloud.
So it's not due to PHP or Wordpress. Digitalocean server has server response time issues. The difference was between 300ms and 1s.