How i can setup robots.txt for my blog?

December 11, 2013 3.7k views
Hello! I have successfully created my blog instance with Ghost droplet and already do some customization. Now i need to create robots.txt file. I googled, that i need to create robots.txt under my theme folder. But when i try to access it i get 404 error and i see in my adress line hostname/robots.txt/ (with slash). I read that i need to configure nginx to serve this file correctly. But i have no experience in this, so if somebody can help me with that - it will be very good :)
9 Answers
Does it work without the slash?
No, it's adding slash when i try to access robots.txt (without slash)
Where did you store robots.txt? Please pastebin your nginx config as well.
I have same issue.

I place my robots.txt in the root folder of my current theme.

The nginx config is in default state.

Here is the default.conf file:
server {
listen 80;
server_name martsen.me ;

client_max_body_size 10M;

location = /robots.txt {
allow all;
log_not_found off;
access_log off;

Maybe i need to add some lines?
Are you sure this is the correct file? There doesn't seem to be anything that's telling nginx to proxy requests to Ghost.

What's listening on port 80?
sudo netstat -plutn | grep 80
tcp 0 0* LISTEN 3124/nginx
Odd. Are there any files in the conf.d directory?
ls /etc/nginx/conf.d
Yes. There are file /etc/nginx/conf.d/default.conf
I'm used this solution http://stackoverflow.com/questions/1090823/how-do-i-configure-nginx-to-redirect-to-a-url-for-robots-txt-sitemap-xml
Now it's work fine.
Have another answer? Share your knowledge.