XML Sitemap Generation Memory Issue

Hey everyone,

I'm wrestling with a persistent memory consumption issue in our 'Free XML Sitemap Generator' tool, especially when dealing with client sites that have a very large number of URLs (1M+). While sitemap generation generally works for smaller sites, anything substantial pushes the limits, leading to timeouts or outright memory exhaustion.

The core of the problem seems to be how we're building the sitemap XML structure in PHP. We're currently fetching URLs from a database, processing them, and then adding them to an in-memory DOMDocument or SimpleXMLElement object before writing to disk. This approach, while straightforward, is proving highly inefficient for scale.

Here's what I've attempted so far:

• Increased memory_limit: We've pushed it to 2GB, sometimes 4GB, but this is clearly not a sustainable solution and just delays the inevitable for extremely large sites.
• Explicit Garbage Collection: Tried using gc_collect_cycles() and unset() on variables after processing batches of URLs, but the impact has been minimal, suggesting the issue isn't just about cyclic references.
• Batch Processing & File Appending: Instead of building one giant XML object, I've experimented with generating smaller sitemap files (e.g., sitemap1.xml, sitemap2.xml) and then linking them via a sitemap index. This helps distribute the load but doesn't solve the peak memory usage during the generation of each individual large file.
• Stream Writing: I've looked into using XMLWriter in PHP for direct stream writing to avoid loading the entire XML into memory. While promising, integrating it with our existing URL processing logic (which involves some URL canonicalization and priority calculations) has been trickier than anticipated without introducing new bottlenecks.

The typical error we're seeing in the logs looks something like this:

[2023-10-27 14:35:12] production.ERROR: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 123456789 bytes) in /var/www/html/app/Services/SitemapGenerator.php on line 145 {"exception":"[object] (Symfony\Component\ErrorHandler\Error\FatalError(code: 0): Allowed memory size of 2147483648 bytes exhausted (tried to allocate 123456789 bytes) in /var/www/html/app/Services/SitemapGenerator.php on line 145 at /var/www/html/app/Services/SitemapGenerator.php:145)"}

Line 145 is usually where a new DOMElement or similar object is being added to the main sitemap structure. My gut tells me we're either holding onto too many URL objects, or the XML DOM itself is consuming far more memory than expected for its size, possibly due to internal PHP XML parser overhead.

Has anyone successfully tackled high memory consumption during large-scale XML sitemap generation in PHP? Are there specific libraries, design patterns, or even server-side optimizations (beyond memory_limit) that are particularly effective here? I'm open to completely rethinking the approach if necessary.

Thanks in advance!