Question
How i can setup robots.txt for my blog?
- Posted December 11, 2013
Hello! I have successfully created my blog instance with Ghost droplet and already do some customization. Now i need to create robots.txt file. I googled, that i need to create robots.txt under my theme folder. But when i try to access it i get 404 error and i see in my adress line hostname/robots.txt/ (with slash). I read that i need to configure nginx to serve this file correctly. But i have no experience in this, so if somebody can help me with that - it will be very good :)
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Hello there,
The robots.txt needs to be inside the public_html directory of your site and then you can request reindexing from Google’s Developer Tools portal
https://developers.google.com/search/docs/advanced/crawling/ask-google-to-recrawl
Hope that this helps!
hi how to create sitemap
This comment has been deleted
Yes. There are file /etc/nginx/conf.d/default.conf <br>I’m used this solution http://stackoverflow.com/questions/1090823/how-do-i-configure-nginx-to-redirect-to-a-url-for-robots-txt-sitemap-xml <br>Now it’s work fine.
Odd. Are there any files in the conf.d directory? <br><pre>ls /etc/nginx/conf.d</pre>
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3124/nginx
Are you sure this is the correct file? There doesn’t seem to be anything that’s telling nginx to proxy requests to Ghost. <br> <br>What’s listening on port 80? <br><pre>sudo netstat -plutn | grep 80</pre>
+1 <br>I have same issue. <br> <br>I place my robots.txt in the root folder of my current theme. <br> <br>The nginx config is in default state. <br> <br>Here is the default.conf file: <br>server { <br> listen 80; <br> server_name martsen.me ; <br> <br> client_max_body_size 10M; <br> <br> location = /robots.txt { <br> allow all; <br> log_not_found off; <br> access_log off; <br> } <br> } <br>} <br> <br> <br>Maybe i need to add some lines? <br>
Where did you store robots.txt? Please pastebin your nginx config as well.
No, it’s adding slash when i try to access robots.txt (without slash)
This comment has been deleted