How i can setup robots.txt for my blog?

December 11, 2013 8k views
i.terhin
By:
i.terhin
Hello! I have successfully created my blog instance with Ghost droplet and already do some customization. Now i need to create robots.txt file. I googled, that i need to create robots.txt under my theme folder. But when i try to access it i get 404 error and i see in my adress line hostname/robots.txt/ (with slash). I read that i need to configure nginx to serve this file correctly. But i have no experience in this, so if somebody can help me with that - it will be very good :)
Log In to Comment
10 Answers
kamaln7 MOD December 12, 2013
Does it work without the slash?
Reply
i.terhin December 17, 2013
No, it's adding slash when i try to access robots.txt (without slash)
Reply
kamaln7 MOD December 17, 2013
Where did you store robots.txt? Please pastebin your nginx config as well.
Reply
martsen.anton December 18, 2013
+1
I have same issue.

I place my robots.txt in the root folder of my current theme.

The nginx config is in default state.

Here is the default.conf file:
server {
listen 80;
server_name martsen.me ;

client_max_body_size 10M;

location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
}
}


Maybe i need to add some lines?
Reply
kamaln7 MOD December 20, 2013
Are you sure this is the correct file? There doesn't seem to be anything that's telling nginx to proxy requests to Ghost.

What's listening on port 80? 
sudo netstat -plutn | grep 80
Reply
martsen.anton January 1, 2014
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3124/nginx
Reply
kamaln7 MOD January 4, 2014
Odd. Are there any files in the conf.d directory? 
ls /etc/nginx/conf.d
Reply
martsen.anton January 10, 2014
Yes. There are file /etc/nginx/conf.d/default.conf
I'm used this solution http://stackoverflow.com/questions/1090823/how-do-i-configure-nginx-to-redirect-to-a-url-for-robots-txt-sitemap-xml
Now it's work fine.
Reply
amit1990gupta January 24, 2016
[deleted]
Reply
kiran605in November 29, 2017

hi
how to create sitemap

Reply
Have another answer? Share your knowledge.