Question

How i can setup robots.txt for my blog?

  • Posted December 11, 2013

Hello! I have successfully created my blog instance with Ghost droplet and already do some customization. Now i need to create robots.txt file. I googled, that i need to create robots.txt under my theme folder. But when i try to access it i get 404 error and i see in my adress line hostname/robots.txt/ (with slash). I read that i need to configure nginx to serve this file correctly. But i have no experience in this, so if somebody can help me with that - it will be very good :)

Subscribe
Share

This comment has been deleted


Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hello there,

The robots.txt needs to be inside the public_html directory of your site and then you can request reindexing from Google’s Developer Tools portal

https://developers.google.com/search/docs/advanced/crawling/ask-google-to-recrawl

Hope that this helps!

hi how to create sitemap

This comment has been deleted

Yes. There are file /etc/nginx/conf.d/default.conf <br>I’m used this solution http://stackoverflow.com/questions/1090823/how-do-i-configure-nginx-to-redirect-to-a-url-for-robots-txt-sitemap-xml <br>Now it’s work fine.

Odd. Are there any files in the conf.d directory? <br><pre>ls /etc/nginx/conf.d</pre>

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3124/nginx

Are you sure this is the correct file? There doesn’t seem to be anything that’s telling nginx to proxy requests to Ghost. <br> <br>What’s listening on port 80? <br><pre>sudo netstat -plutn | grep 80</pre>

+1 <br>I have same issue. <br> <br>I place my robots.txt in the root folder of my current theme. <br> <br>The nginx config is in default state. <br> <br>Here is the default.conf file: <br>server { <br> listen 80; <br> server_name martsen.me ; <br> <br> client_max_body_size 10M; <br> <br> location = /robots.txt { <br> allow all; <br> log_not_found off; <br> access_log off; <br> } <br> } <br>} <br> <br> <br>Maybe i need to add some lines? <br>

Where did you store robots.txt? Please pastebin your nginx config as well.

No, it’s adding slash when i try to access robots.txt (without slash)