Dealing with Dynamic Sitemaps: Any Tips for Crawl Efficiency?

hey everyone, following up on some of the earlier chats about sitemap crawl budget. my main challenge now is with dynamic content.

my SaaS generates a ton of user-generated content daily, and keeping our XML sitemap truly up-to-date for all these new pages is proving to be a real pain. we want to ensure good crawl efficiency, and this whole sitemap management thing is getting tricky.

when i try to regenerate or even just update specific sections of the sitemap that change frequently, i sometimes run into performance bottlenecks or even hit API rate limits when fetching data for new URLs. it's kinda frustrating, you know?

[2023-10-27 03:15:22] sitemap.generator.ERROR: Sitemap update for /dynamic-pages/ timed out after 300s. 
[2023-10-27 03:15:22] sitemap.generator.INFO: Attempting partial update for last 24 hours.
[2023-10-27 03:16:05] api.provider.WARNING: Rate limit exceeded for content fetch API. Delaying...

what are some of your best practices or tooling recommendations for effectively managing dynamic sitemaps, especially when it comes to improving crawl efficiency and preventing server strain?

specific points i'm wondering about:

• strategies for handling lastmod dates for constantly changing content without making Google re-crawl everything.
• is breaking them into smaller, more granular sitemap index files the way to go?
• any tips on incremental sitemap updates vs. full regeneration for large dynamic sections?

seriously, any advice would be super helpful. help a brother out please...