Skip to content
internal/robots
Updated
On this page

The crawler policy, declared once for every robots.txt this repository writes, so a crawler the assembled site allows cannot be one its projects disallow.

#internal/robots

#internal/robots

Package robots holds the crawler policy, declared once for every robots.txt this repository writes.

Two generators emit a robots.txt: a project's own build writes one at its output root, and the assembly's shared-element generator writes the site-wide one that is actually served. They read the policy from here, so a crawler the site allows cannot be one its projects disallow -- and a disallow written into one of them cannot leave the other allowing it.

#Agents

Go go
var Agents = []string{

Agents names every crawler the generated robots.txt names, in the order it names them.

The wildcard already allows all of them; naming each one is what keeps a future disallow from being written once and applying to everybody by accident.

The slice is package state a caller must not write to: mutating an element would change what every generated robots.txt says. Read it, never assign into it.

#RenderRobotsTxt

Go go
func RenderRobotsTxt(sitemapURL string) string

RenderRobotsTxt returns robots.txt text allowing every agent in [Agents] and naming a sitemap.

sitemapURL is the absolute URL of the sitemap this robots.txt points at. Crawlers find a sitemap by being told where it is, and the "Sitemap:" directive takes an absolute URL.

More tools from this site

  • claudestream Drive Claude Code from Python: run it as a subprocess and read its output as typed events, with async and sync sessions, sandbox policies, and tools you define in Python
  • claudewheel A TUI Claude Code Launcher that lets you have more than one profile, manage sessions lifecycle, pick the exact CC version, model to use (even older unlisted ones), pick which GitHub account to use, etc.
  • dirstat Fast, single-binary directory statistics CLI: every file under a tree grouped by format, with counts, sizes, and lines of code, as a colored terminal table or as JSON
  • fastware A batteries-included ASGI framework: msgspec JSON, a managed Granian server, dependency injection, SSE, WebSockets, auth, and a test client
  • go-toml-edit Zero-dep TOML editing library for Go with comment preservation
  • howmuchleft The fastest Claude Code statusline: context window, 5-hour, and weekly limit usage as three customizable gradient bars, rendering in about 6 ms
  • orxtra
  • pgdesign
  • predraw Declarative rendering pipeline: describe a scene in JSON and get SVG, PNG and WebP out, with light and dark style tokens, reusable components and text converted to path outlines
  • reposummary Turn a git repository's history into a Markdown journal: pick a time window or revision range and get a readable digest of what changed, optionally narrated by an LLM
  • rlsbl Release orchestration and project scaffolding CLI that bumps versions, validates a structured JSONL changelog, tags only the commit CI verified, and publishes to npm, PyPI, Go and more
  • safegit git wrapper CLI that gives each commit its own temporary index and retries ref updates on conflict, so concurrent agents share one repository
  • saferm Command-line replacement for rm that archives every deletion with a mandatory reason and the context it ran in, so deleted files can be listed, inspected and restored
  • strictcli
  • stricttest An always-on test-isolation floor: a pytest plugin and a Go env-hygiene module that make a test suite structurally unable to reach real credentials, the real HOME, the network, or the development repository.
  • wesktop A Python framework that turns an ASGI web app into a desktop application, serving it from a local Granian server and displaying it in a native OS window via pywebview
Search