Question

How to add robot.txt when using nginx, flask and uwsgi?

Posted May 8, 2017 5k views
Nginx

After getting flask running with nginx and uwsgi with the help of this eminent community, and with the guide found here https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-16-04.

I am confused since I can not understand how to add robot.txt to a site running flask.
Robot.txt is a static file, but how to configure to be able to run flask with robots.txt?

If I am not misstaken shouldn’t nginx handle static files and robot.txt in production?
How? Any guide toward correct answer is more than appreciated.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
2 answers

@broodforge

NGINX will serve pretty much anything you configure it to – whether static or dynamic.

Generally, you want to place the robots.txt file in the web root where you’re serving your app, but flask seems to work a little differently, so you’d most likely need to place in the root that you’ve set for your static file path.

You may want to check the following to see if it can help you to get things setup:

https://coderwall.com/p/kc-ewg/flask-handler-for-robots-txt

https://www.pythonanywhere.com/forums/topic/2899/

http://stackoverflow.com/questions/4239825/static-files-in-flask-robot-txt-sitemap-xml-mod-wsgi

I ran into this issue and was trying to make it work with static directories that were loading from a blueprint.

This solution works well for me because it shows how to modify headers and control the response more granularly for a variety of scenarios.

from flask import Flask, Response
app = Flask(__name__)

@app.route('/robots.txt')
def noindex():
    r = Response(response="User-Agent: *\nDisallow: /\n", status=200, mimetype="text/plain")
    r.headers["Content-Type"] = "text/plain; charset=utf-8"
    return r

Cheers.

Submit an Answer