Robots, noindex, and crawl directives in API-driven audits

A page can look healthy and still be effectively removed from search discovery because of robots directives, noindex instructions, or conflicting control signals. Audit outputs need to surface these states immediately.

The consumer should not have to parse raw HTML to understand crawl directives. The payload should already normalize the important pieces into explicit fields and readable statuses.

When a page that should rank is blocked from indexing, almost every other optimization becomes secondary. That is why crawl-control signals deserve high visibility in summaries and alerting.