Persistent Memory Exhaustion in Custom XML Sitemap Generator for Large Sites and Sitemap Protocol Compliance

I'm facing a significant technical hurdle with our custom-built Free XML Sitemap Generator, specifically regarding memory management when processing extremely large websites. The generator consistently hits PHP's memory limits (even at 4GB+) when attempting to process sites exceeding 5 million URLs, leading to incomplete sitemaps or script termination. We urgently need to handle 10-20 million URLs efficiently while strictly adhering to the sitemap protocol.

Our current architecture is built in PHP, primarily utilizing DOMDocument for XML generation, though we've experimented with alternatives. Crawling is performed via Guzzle HTTP client with asynchronous requests, and URL discovery relies on internal link parsing (HTML <a> tags) and canonical tag analysis. Temporary storage for discovered URLs currently uses an in-memory SplFixedArray or array_map for deduplication before final XML serialization. The tool already supports sitemapindex generation and splitting into multiple sitemap.xml files per the sitemap protocol's specifications.

We've attempted several solutions, each with its own limitations. Initially, we progressively increased PHP's memory limit up to 4GB, but this only postpones the inevitable for larger sites and isn't a scalable solution. We then implemented XML writing in chunks using XMLWriter instead of DOMDocument for final output; this helped with final serialization memory, but the core issue of memory exhaustion during URL collection and deduplication persisted. We also employed PHP generator functions to yield URLs one by one during crawling, which reduced memory consumption for the crawl phase itself. However, the aggregated list required for deduplication still consumes too much memory. Experimenting with externalizing URL storage to Redis and MySQL for temporary URL storage and deduplication proved effective in terms of memory, but the I/O overhead significantly increased processing time, making the tool unacceptably slow for users expecting quick results. Furthermore, we explored using highly optimized C-extension based hash tables (e.g., ds extension) or implementing Bloom filters for deduplication. These showed some promise but integrating them seamlessly while maintaining performance and reliability has proven complex. Aggressively unsetting variables and forcing gc_collect_cycles() after processing batches also provided minimal relief, as the underlying data structures for storing unique URLs still accumulate.

The deep technical block we're encountering appears to be the cumulative memory footprint of storing millions of unique URL strings (and associated metadata like last modification date, priority, change frequency) in a PHP-native structure *before* they are written to disk. Even with external storage, the process of reading, deduplicating, and preparing these URLs for XMLWriter in batches still causes significant memory spikes. The critical question is: how can we manage this vast dataset of unique URLs entirely in a memory-efficient, streaming fashion that avoids loading the full set into RAM at any point, while still ensuring high performance and strict adherence to the sitemap protocol's formatting and size constraints?

I'm specifically seeking guidance on alternative architectural patterns for managing massive URL datasets in PHP, highly optimized, low-level data structures or libraries (PHP or C-extension based) for unique string storage that are truly memory-light, strategies for "real-time" deduplication and sorting of URLs without requiring full in-memory sets, and any experiences with specific cloud-based solutions or services designed for this kind of data processing at scale. Thanks in advance for any insights or shared experiences!