I was wondering why was my profile not indexed by Google. So i looked into DO’s robots.txt and there was a line:
# Only allow crawling of the canonical user profile
Disallow: /community/users/*?primary_filter=*
So can you explain why is profile crawling disallowed and what is a canonical profile?
Subscribe
Share
Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign in to AnswerThese answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
It is indexed. And by canonical URL means DO want to index /community/users/someuser but not /community/users/someuser?primary_filter=something Canonical URLs are used by many sites to avoid having search engines indexing multiple versions of the same data, just with a slightly different URL.