Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
Does DigitalOcean preserve snapshots of deleted droplets? Question Question
Is there a DigitalOcean Free Trial Available? Question Question

Question

Why is crawling user profiles disallowed?

Posted February 21, 2017 993 views
DigitalOcean

I was wondering why was my profile not indexed by Google. So i looked into DO’s robots.txt and there was a line:

# Only allow crawling of the canonical user profile
Disallow: /community/users/*?primary_filter=*

So can you explain why is profile crawling disallowed and what is a canonical profile?

Add a comment
Sheogorath
By:
Sheogorath

Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
Does DigitalOcean preserve snapshots of deleted droplets? Question Question
Is there a DigitalOcean Free Trial Available? Question Question

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
1 answer
hansen February 21, 2017

It is indexed. And by canonical URL means DO want to index /community/users/someuser but not /community/users/someuser?primary_filter=something
Canonical URLs are used by many sites to avoid having search engines indexing multiple versions of the same data, just with a slightly different URL.

Reply Report

Related

Looking for something else?

Ask a new question Search for more help