Question

How i can setup robots.txt for my blog?

Posted December 11, 2013 8.3k views
Hello! I have successfully created my blog instance with Ghost droplet and already do some customization. Now i need to create robots.txt file. I googled, that i need to create robots.txt under my theme folder. But when i try to access it i get 404 error and i see in my adress line hostname/robots.txt/ (with slash). I read that i need to configure nginx to serve this file correctly. But i have no experience in this, so if somebody can help me with that - it will be very good :)
Add a comment
i.terhin
By:
i.terhin
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

10 answers
kamaln7 December 12, 2013
Does it work without the slash?
Reply Report
i.terhin December 17, 2013
No, it's adding slash when i try to access robots.txt (without slash)
Reply Report
kamaln7 December 17, 2013
Where did you store robots.txt? Please pastebin your nginx config as well.
Reply Report
martsen.anton December 18, 2013
+1
I have same issue.

I place my robots.txt in the root folder of my current theme.

The nginx config is in default state.

Here is the default.conf file:
server {
listen 80;
server_name martsen.me ;

client_max_body_size 10M;

location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
}
}


Maybe i need to add some lines?
Reply Report
kamaln7 December 20, 2013
Are you sure this is the correct file? There doesn't seem to be anything that's telling nginx to proxy requests to Ghost.

What's listening on port 80? 
sudo netstat -plutn | grep 80
Reply Report
martsen.anton January 1, 2014
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3124/nginx
Reply Report
kamaln7 January 4, 2014
Odd. Are there any files in the conf.d directory? 
ls /etc/nginx/conf.d
Reply Report
martsen.anton January 10, 2014
Yes. There are file /etc/nginx/conf.d/default.conf
I'm used this solution http://stackoverflow.com/questions/1090823/how-do-i-configure-nginx-to-redirect-to-a-url-for-robots-txt-sitemap-xml
Now it's work fine.
Reply Report
amit1990gupta January 24, 2016
[deleted]
Reply Report
kiran605in November 29, 2017

hi
how to create sitemap

Reply Report
Submit an Answer

Looking for something else?

Ask a new question Search for more help