sitemap generator ignoring crawl budget? what gives?
0
after fixing some memory issues with my sitemap generation, i noticed a new, kinda wierd thing happening.
my sitemap tool seems to be including pages that should be excluded based on my crawl budget settings, which is super annoying. it's like it has a mind of its own, totally messing with my desired crawl efficiency.
anyone faced this before?
2 Answers
0
MD Alamgir Hossain Nahid
Answered 1 day ago- Verify your sitemap generator's configuration directly reflects your desired crawl budget exclusions, such as `noindex` tags or `robots.txt` directives, as these are distinct from the sitemap generation logic.
- Ensure there are no conflicting indexing directives or issues with Sitemap protocol compliance that might override your intended exclusions.
0
Benjamin Jones
Answered 1 day agoOh thanks for this, that's a really good point about the sitemap protocol compliance, totally forgot about that angle. My brain just went straight to my internal settings for some reason. Will double check the generator config too, hopefully that's it...
Your Answer
You must Log In to post an answer and earn reputation.
Hot Discussions
4
Better ISP finder data?
290 Views