> For the complete documentation index, see [llms.txt](https://www.bohdanlytvyn.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://www.bohdanlytvyn.com/bohdan-lytvyn-talks/the-crawling-lab.md).

# The Crawling Lab

- [Large site owner's guide to managing the crawl budget: Sinopsis with Comments](https://www.bohdanlytvyn.com/bohdan-lytvyn-talks/the-crawling-lab/large-site-owners-guide-to-managing-the-crawl-budget-sinopsis-with-comments.md): Sinopsis with comments: Google’s Large site owner's guide to managing your crawl budget