Optimizing Laravel dynamic sitemaps
Right now, we're using a pretty standard approach, fetching all relevant records in one go. It looks something like this (simplified, of course):
Route::get('/sitemap.xml', function () {
$sitemap = App::make("sitemap");
$posts = Post::all(); // This is the bottleneck!
foreach ($posts as $post) {
$sitemap->add(URL::to('posts/' . $post->slug), $post->updated_at, '0.8', 'daily');
}
return $sitemap->render('xml');
});The symptoms are pretty clear: increased generation time (sometimes 30+ seconds), occasional server timeouts, and most frequently, memory exhaustion. I'm seeing stuff like this in our logs:
[2023-10-27 14:30:05] local.ERROR: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Collection.php on line 123We've tried bumping up PHP's memory limits, but that's just kicking the can down the road and not a real solution. I've been looking into sitemap chunking and generating sitemap index files, but I'm not entirely sure how to implement this effectively, especially with existing packages or if a custom approach is better. Caching is also on my mind, particularly for data that doesn't change hourly, but again, the implementation details for large-scale sitemaps are fuzzy.
So, my specific questions are:
- What are the absolute best practices for generating dynamic sitemaps in Laravel for applications with millions of records?
- How do you effectively implement sitemap chunking and sitemap index files, ideally with a package like
spatie/laravel-sitemap, or is a custom solution more robust for extreme scale? - Are there specific database query optimizations or packages you'd recommend for fetching such large datasets efficiently without loading everything into memory?
- And finally, how do you handle frequent content updates efficiently without regenerating the entire sitemap from scratch every time it needs a tiny tweak?
Any insights, code examples, or architectural advice on improving our sitemap performance would be massively appreciated. Help a brother out please...
2 Answers
Leonardo Rodriguez
Answered 3 days ago- For millions of records, you absolutely must implement sitemap chunking and index files. Instead of `Post::all()`, use `chunkById()` or `cursor()` on your Eloquent queries to process records in smaller batches, preventing memory exhaustion.
- With `spatie/laravel-sitemap`, you can generate multiple sitemap files and an index. Configure `SitemapGenerator::create()->hasSitemapIndex()` and use `addLink()` within a loop that processes chunks. Each chunk becomes a separate sitemap file (e.g., `sitemap-1.xml`, `sitemap-2.xml`), and the main `sitemap.xml` will be an index file referencing these.
- Database query optimizations involve using `->chunk(1000)` or `->cursor()` for fetching. Ensure you have database indexes on columns like `id`, `updated_at`, and `slug` to speed up these operations. Consider using a read replica if the database is also under heavy write load.
- To handle frequent content updates efficiently without full regeneration, implement a caching layer (e.g., Redis or file-based cache) for your sitemap chunks. Set a reasonable cache TTL. For real-time updates, consider a queued job that regenerates only the affected sitemap chunk(s) when a `Post` (or other content type) is created, updated, or deleted, rather than rebuilding the entire sitemap. This is crucial for maintaining `sitemap performance` in a high-growth environment.
Youssef Syed
Answered 3 days agoAh, perfect, Leonardo Rodriguez, this totally lines up with what I was hoping to hear about best practices, good to know I was on the right track...