Troubleshooting large-scale sitemap generation for dynamic websites?

Our 'Free XML Sitemap Generator' has been quite popular, helping users create sitemaps for their websites. While it performs admirably for small to medium-sized static or moderately dynamic sites, we're encountering significant scaling challenges when attempting to generate sitemaps for very large, highly dynamic websites, especially those with frequently updated content and hundreds of thousands to millions of URLs.

The primary issue manifests as persistent memory exhaustion and unacceptably slow processing times. For sites exceeding 500,000 URLs, and particularly those with content that changes multiple times a day, the generation process often either times out, crashes due to out-of-memory errors, or takes an impractical amount of time (many hours) to complete a full website crawling and sitemap build.

Our current sitemap generation logic is built around a recursive website crawling mechanism. It starts from a root URL, fetches the HTML, parses it for internal links, and then adds discovered URLs to a queue. URLs are stored in a MySQL database, with checks for existing URLs to prevent duplicates and cycles. We use standard PHP DOMDocument for parsing and cURL for fetching.

We've already attempted several optimizations. We've increased PHP memory limits significantly (up to 4GB for the CLI process), implemented URL chunking for database writes, and even experimented with a basic asynchronous processing queue using Gearman to offload link parsing and URL discovery. For URL storage, we've tried both in-memory arrays (for smaller crawls) and direct database insertions. None of these have fully resolved the scalability bottleneck. For instance, even with increased memory, deep crawling still eventually hits limits, as illustrated by this common error:

PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 123456789 bytes) in /path/to/crawler.php on line 123

Our analysis points to a few specific bottlenecks: 1) The recursive nature of the crawler, even with depth limits, can lead to excessive call stack usage for highly interconnected sites. 2) DOM parsing with DOMDocument for millions of pages, especially complex ones, appears to be a major CPU and memory hog. 3) Database lock contention during high-volume concurrent URL writes, even with transactions, seems to be a significant factor. 4) I/O limitations from constantly reading/writing discovered URLs to disk (for persistence between runs or large datasets) are also evident.

We're looking for expert advice on alternative, highly scalable architectural patterns for website crawling and sitemap generation. Are there specific, battle-tested libraries or frameworks (beyond standard PHP DOM/cURL) that are designed for efficient, high-volume link extraction and URL management? What proven strategies exist for handling frequently updated content efficiently without re-crawling entire sites, perhaps differential crawling or change detection mechanisms at scale?