Related

Does DigitalOcean preserve snapshots of deleted droplets? Question Question
Is there a DigitalOcean Free Trial Available? Question Question

Question

Why is crawling user profiles disallowed?

Posted February 21, 2017 750 views
DigitalOcean

I was wondering why was my profile not indexed by Google. So i looked into DO’s robots.txt and there was a line:

# Only allow crawling of the canonical user profile
Disallow: /community/users/*?primary_filter=*

So can you explain why is profile crawling disallowed and what is a canonical profile?

Add a comment
Sheogorath
By:
Sheogorath
Add a comment

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer
hansen February 21, 2017

It is indexed. And by canonical URL means DO want to index /community/users/someuser but not /community/users/someuser?primary_filter=something
Canonical URLs are used by many sites to avoid having search engines indexing multiple versions of the same data, just with a slightly different URL.

Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help