Debugging Persistent "Too Many Redirects" Errors with Large XML Sitemap Protocol Files in Our Generator

hey folks,

we've been running our 'Free XML Sitemap Generator' for a while now, and it's been pretty solid for most users. but lately, with some really large sites, we're hitting a wall, and it's impacting their technical SEO efforts.

the main issue:

• users trying to generate sitemaps for sites with hundreds of thousands, sometimes millions, of URLs are frequently getting "too many redirects" or sometimes just a 500 internal server error when trying to fetch the generated sitemap file.
• this isn't during the generation process itself (that part is handled asynchronously and seems to complete), but specifically when the user tries to download or view the final sitemap.xml.

what we've already done/checked:

• our server is NGINX + PHP-FPM. we've bumped up php_memory_limit, max_execution_time, and NGINX's proxy_read_timeout significantly.
• we're generating sitemaps in chunks (sitemap index files) to adhere to the XML sitemap protocol limits (50k URLs per file, 50MB per file).
• we're using GZIP compression for the generated files directly from NGINX.
• checked for any hidden NGINX rewrite rules or PHP redirects that might be looping – nothing obvious found.
• verified file permissions and ownership for the sitemap output directory.
• we've tried direct file serving vs. serving through a PHP script – both exhibit similar issues under high load/large files.

the puzzling part is smaller sitemaps (under ~200k URLs total, across multiple index files) work flawlessly. it's only when the total number of linked sitemap files in the sitemap index gets really high that these redirect loops or 500s start appearing inconsistently. we're wondering if there's some obscure NGINX or PHP-FPM configuration limit we're missing related to serving a high number of static files referenced by a single index, or perhaps a subtle aspect of the XML sitemap protocol implementation we're overlooking.

any thoughts on how to debug this further for large-scale XML sitemap protocol delivery? or common pitfalls with NGINX serving a huge number of static files referenced by an index file?

thanks in advance!