Anyone else having issues with large XML sitemaps?

"it seems to just hang or timeout during the site map generation proces when processing really big websites (10k+ pages)."

1. Server Resource Limits: The error `Max execution time exceeded` is a dead giveaway. Your server (or more specifically, PHP) is set to stop a script after a certain duration to prevent runaway processes from consuming all resources. Generating a sitemap for 10,000+ pages involves a lot of URL fetching, parsing, and data manipulation, which can easily hit typical default limits (e.g., 30-60 seconds).
   • Action: If you have control over your server configuration (e.g., a VPS or dedicated server), you can increase max_execution_time and memory_limit in your php.ini file. For example, setting max_execution_time = 300 (5 minutes) or even higher for very large sites, and memory_limit = 512M or 1024M. Be cautious not to set them excessively high, as this can affect overall server stability.
2. Sitemap Splitting & Index Files: This is the most common and recommended best practice for large sites. Google and other search engines actually prefer sitemaps to be under 50,000 URLs and 50MB in size.
   • Action: Modify your tool to generate multiple smaller sitemap files (e.g., sitemap1.xml, sitemap2.xml, etc.), each containing a subset of your URLs. Then, create a sitemap index file (e.g., sitemap_index.xml) that lists all these individual sitemaps. This significantly reduces the processing load for any single generation run. Your tool would then generate one small index file, which points to many smaller sitemap files.
3. Incremental Generation & Caching: Instead of regenerating the entire sitemap from scratch every time, especially for pages that don't change often.
   • Action: Implement a system where your tool only processes new or updated pages, appending them to existing sitemaps or regenerating only the affected sitemap segments. You could also cache the list of URLs and only re-crawl/re-process parts that are explicitly flagged for update.
4. Asynchronous Processing: Running the sitemap generation as a background task.
   • Action: Instead of having the web request directly trigger the generation, which is subject to web server timeouts, queue the generation process as a background job. Tools like cron jobs (for PHP scripts) or message queues (like RabbitMQ, Redis Queue) can handle this. This allows the script to run for as long as needed without directly impacting the user's browser session or hitting web server `max_execution_time` limits.
5. Optimize Crawling Logic: How your generator discovers URLs can impact performance.
   • Action: Ensure your internal crawler is efficient. Avoid re-crawling external links, optimize database queries if URLs are pulled from a database, and consider depth limits for very deep sites to manage your crawl budget and processing time more effectively.

MD Alamgir Hossain Nahid, that breakdown on server resource limits and sitemap splitting really gives me a lot of new context on why this whole thing is such a headache for big sites.