Join 1M+ other developers and:
- Get help and share knowledge in Q&A
- Subscribe to topics of interest
- Get courses & tools that help you grow as a developer or small business owner
Why is crawling user profiles disallowed?
I was wondering why was my profile not indexed by Google. So i looked into DO’s robots.txt and there was a line:
# Only allow crawling of the canonical user profile Disallow: /community/users/*?primary_filter=*
So can you explain why is profile crawling disallowed and what is a canonical profile?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.×