Question

NGINX - Send request to backend if the request is coming from a crawler/bot

Posted September 11, 2021 15 views
Nginx

I have a Single Page Application without server-side rendering. So whenever someone shares a URL from the site on Facebook or Twitter it does not show the dynamic title of the page.

I would like to send crawler/bot requests to a special backend that serves a page with OpenGraph meta tags.

Front-end is hosted on NGINX and I have this configuration so far:

server {
    listen       80;
    listen  [::]:80;

    location / {

        if ($http_user_agent ~* "linkedinbot|googlebot|yahoo|bingbot|baiduspider|yandex|yeti|yodaobot|gigabot|ia_archiver|facebookexternalhit|twitterbot|developers\.google\.com") {
            return 307 https://renderer_service_domain$request_uri;
    }

    root   /usr/share/nginx/html;
    index  index.html;
}

This works, but Angular’s URL has the # before the route url, I need to remove it before sending the request to the special backend.

So, how can I remove the hash from the $request_uri?

I tried couple of rewrite examples, but it sends the request to the same domain, I need to send the request to another server/domain name.

Submit an answer

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!