Loading content...
Processing sitemap indexes and compressed sitemaps safely at scale
Sitemap support gets more complex once you encounter indexes, compressed files, and edge-case payloads from the public web.
sitemap indexgzip sitemapdomain monitoringsafe parsing
Real sitemaps are messy
Public websites do not all ship a neat `/sitemap.xml` file. Some provide sitemap indexes, some use compressed `.xml.gz` payloads, and some mix useful URLs with things you should not store at all.
Safe extraction rules
Processing sitemap content should be conservative. The system should enforce size limits, decompress carefully, ignore off-domain URLs, and reject malformed or obviously abusive payloads.
- Support sitemap indexes recursively within limits
- Validate decompressed size before processing
- Store only same-domain paths
- Drop malformed, duplicate, or unsupported entries
Why this improves product trust
When sitemap import works reliably across different site setups, domain monitoring feels much more mature. Users notice very quickly whether import works on simple sites only or on real-world infrastructure too.