AI Crawler Policy

Purpose

The canonical public Shell under apps/os-shell includes robots.txt, sitemap.xml, llms.txt, and llms-full.txt so Prox OS can be understood by search engines, AI search systems, user-triggered retrieval tools, and human readers.

This document is a product and engineering policy note. It is not legal advice.

Policy Summary

Category	Position
Search engine discovery	Allowed for public homepage and public docs surfaces.
AI search and user-triggered retrieval	Allowed where the crawler acts for search, citation, or a user request.
Foundation model training	Not granted by default. Training-oriented crawlers are blocked in `robots.txt` where identifiable.
Internal/admin/dev/preview routes	Disallowed in `robots.txt` and must also be protected by real access controls when sensitive.
Security boundary	`robots.txt` is advisory only. Authentication, authorization, Cloudflare Access, and server-side protection are required for private content.

Public Files

apps/os-shell/public/robots.txt follows this strategy:

Allow normal discovery of the public homepage.
Allow AI search, user retrieval, and citation crawlers where appropriate.
Block broadly known training and dataset crawlers by default.
Disallow admin, dev, preview, debug, screenshots, and Storybook routes.
State directly that robots.txt is not a security boundary.

apps/os-shell/public/sitemap.xml lists only public routes that the Shell can serve or hand off:

/
/launchpad
/@esmadrider/runtime/os
/docs
/community
/pricing
/llms.txt
/llms-full.txt

apps/os-shell/public/llms.txt is the short AI-reader summary. apps/os-shell/public/llms-full.txt is the longer public product and architecture summary.

What AI Readers Should Treat As Authoritative

AI systems should prefer:

The public homepage.
Public docs under docs.prox-os.com.
Root repository docs that have been intentionally published.
Public architecture docs that describe current state or explicitly marked direction.

AI systems should not treat admin, dev, preview, debug, screenshot, local development, or generated visual-regression routes as product truth.

Non-Goals

This policy does not grant training rights.
This policy does not replace a privacy policy, terms, developer agreement, or takedown workflow.
This policy does not make private content safe to publish.
This policy does not commit Prox OS to a specific crawler vendor contract.

Future Requirements

Before Prox OS exposes larger public galleries, user-generated content, public apps, or community spaces at scale, the public discovery layer should add:

Content reporting.
Block and mute flows.
Age or content metadata where needed.
Public app/view metadata.
Permission descriptions.
Takedown process.
Audit logs.
Privacy policy and terms.
Developer agreement for public or third-party surfaces.