https://devopstales.github.io/tags/robots.txt/ Recent content in Robots.txt on DevOpsTales Hugo -- gohugo.io en-US Thu, 16 Jul 2026 00:00:00 +0000 https://devopstales.github.io/hugo/hugo-robots-txt/ Thu, 16 Jul 2026 00:00:00 +0000 https://devopstales.github.io/hugo/hugo-robots-txt/ Crawlers still look for /robots.txt before they dig through your tree. Hugo can generate that file at build time from a template, so the Sitemap URL always matches your baseURL and you can add allow/disallow rules without maintaining a hand-copied static file.