I was wondering why was my profile not indexed by Google. So i looked into DO’s robots.txt and there was a line:
# Only allow crawling of the canonical user profile
Disallow: /community/users/*?primary_filter=*
So can you explain why is profile crawling disallowed and what is a canonical profile?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
It is indexed. And by canonical URL means DO want to index /community/users/someuser but not /community/users/someuser?primary_filter=something Canonical URLs are used by many sites to avoid having search engines indexing multiple versions of the same data, just with a slightly different URL.
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and AI-native businesses
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.