Hi
We are having issues with our website running on a droplet and the googlebot index reaching the website.
When running a mobile-friendly and rich results tests using the googlebot smartphone agent we get the following result: https://search.google.com/test/rich-results?id=Q_z1x7NiYJDJI-CJSpGRuw
If we run the test with the googlebot desktop agent it at least finds the site: https://search.google.com/test/rich-results?id=ha7gO-Yxm_VWahAFk3qf5w
Could someone please assist into where I can start looking where the bot could be being blocked.
I checked our apache log and I cannot see any connection come through when the smartphone bot test is done.
But for the desktop one the following logs appeared
66.249.71.120 - - [06/Feb/2020:08:32:54 +1000] "GET / HTTP/1.1" 200 20616 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.120 - - [06/Feb/2020:08:32:55 +1000] "GET /robots.txt HTTP/1.1" 200 860 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.119 - - [06/Feb/2020:08:32:56 +1000] "GET /js/jquery-2.2.3.min.js HTTP/1.1" 200 34147 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
66.249.71.120 - - [06/Feb/2020:08:32:56 +1000] "GET /js/bootstrap.min.js HTTP/1.1" 200 10598 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
66.249.65.155 - - [06/Feb/2020:08:32:57 +1000] "GET /css/trainingcamps.css HTTP/1.1" 200 4513 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
66.249.71.121 - - [06/Feb/2020:08:32:57 +1000] "GET /css/customize.css HTTP/1.1" 200 6096 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
66.249.71.120 - - [06/Feb/2020:08:32:58 +1000] "GET /public/vendor/slick/slick.css HTTP/1.1" 200 1328 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
66.249.71.120 - - [06/Feb/2020:08:32:58 +1000] "GET /css/achievement.css HTTP/1.1" 200 1051 "https://dailyfantasyrankings.com.au/" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/79.0.3945.120 Safari/537.36"
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hello,
I’ve just checked this now and it seems to be working as normal.
Would you mind sharing your solution here with the community? This would be highly appreciated!
Regards, Bobby
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.