Struggling with Horizon worker management after Laravel deployments
Hey everyone, I was here a couple of weeks ago asking for help with some persistent Laravel queue failures after a recent deployment. Thanks to some great advice, we managed to iron out most of those initial bugs related to job serialization and environment caching. Things were looking good for a bit, but now we're hitting a new wall, and it's specifically around Horizon and how it manages our workers post-deployment.
The issue now is that after almost every deployment, our Horizon worker management becomes incredibly inconsistent. It's like the workers just don't want to play nice. We're seeing situations where they don't scale correctly, jobs get stuck in a pending state even when there's capacity, or conversely, we get unexpected resource spikes with workers seemingly doing nothing useful. It's really impacting our background processing and user experience because critical tasks are either delayed or fail silently.
We've tried a bunch of things to get a handle on this. We've gone through our Horizon configuration multiple times, experimenting with different balance strategies, adjusting max_processes and max_time settings, and even trying tries and timeout values more aggressively. We're constantly checking server metrics โ CPU, memory, I/O โ to see if there's an underlying infrastructure bottleneck, but usually, the server itself seems fine; it's just Horizon misbehaving. We've also resorted to manual restarts of the Horizon supervisor after deployments, which sometimes helps, but it's not a reliable or automated solution, and definitely not ideal for a CI/CD pipeline.
The symptoms are pretty clear: in the Horizon dashboard, we'll see jobs accumulating rapidly in the 'Pending' queue, while the worker list shows many workers as 'inactive' or 'paused' despite having plenty of jobs to process. Other times, we'll see high CPU usage from the Horizon process itself, but the throughput of actual job completion is minimal. It's like the workers are busy doing nothing, or they're just not picking up new tasks efficiently. Sometimes, a full server reboot or a complete re-provisioning of the supervisor process is the only thing that kicks it back into gear, which is obviously a last resort.
So, my main question is, what are the best practices for robust Laravel worker management with Horizon, especially concerning deployment strategies? How do you guys ensure your Horizon workers come up gracefully and consistently after a new code push without these kinds of hiccups? Are there specific deployment scripts or supervisor configurations that are more resilient?
I'm really looking for any insights or proven methodologies here. Anyone faced this before and found a solid, repeatable solution?
2 Answers
Hiroshi Park
Answered 1 month agoHello Isabella Miller,
Ah, the classic post-deployment Horizon worker tango โ I feel your pain on this one! It's one of those issues that can really make you pull your hair out, especially when you think you've nailed the initial queue stability, only to find the workers have decided to go on strike after a code push. We've definitely wrestled with similar inconsistencies in our own Laravel queue management setups, and it's incredibly frustrating when critical background tasks get delayed or become unreliable.
The core of the problem you're describing often boils down to how Horizon workers are gracefully restarted and reloaded with new code after a deployment. Simply pushing new code doesn't automatically tell existing, long-running Horizon processes to pick up the changes. Here are some proven methodologies and best practices we've adopted to ensure robust Laravel worker management with Horizon post-deployment:
- Implement
php artisan horizon:terminatein Your Deployment Script: This is arguably the most crucial step. When you deploy new code, you need to signal Horizon to gracefully terminate its existing workers so they can be restarted by the supervisor with the fresh code. Thehorizon:terminatecommand does exactly this โ it tells workers to finish their current job and then exit.- How to integrate: Place this command *after* your new code is deployed and your caches are cleared, but *before* the deployment script finishes. For example, in a CI/CD pipeline integration, it would typically be a step after `php artisan migrate --force` and `php artisan config:cache`.
- Example (simplified):
git pull origin master
composer install --no-dev --prefer-dist
php artisan migrate --force
php artisan config:cache
php artisan view:cache
php artisan horizon:terminate
- Verify Supervisor/Systemd Configuration: Ensure the process manager (like Supervisor or Systemd) that's running your `php artisan horizon` command is correctly configured to automatically restart the Horizon master process when it terminates.
- For Supervisor, check your `stopwaitsecs` parameter. Setting it to a reasonable value (e.g., 30-60 seconds) gives workers time to finish current jobs before the supervisor forcefully kills them. Also, ensure `autorestart=true`.
- For Systemd, ensure `Restart=always` is set in your service unit file.
- The goal is: `horizon:terminate` -> master process exits -> supervisor/systemd detects exit -> supervisor/systemd restarts master process -> new workers spun up with new code.
- Review Horizon Configuration (`config/horizon.php`): While you've experimented, it's worth re-emphasizing some points:
balanceStrategy: For initial stability, `balance: simple` is often the most predictable. `auto` can sometimes be too aggressive or slow to react in dynamic environments, leading to the pending jobs you're seeing. Only switch to `auto` once `simple` is stable and you truly need its dynamic scaling.max_processesand `min_processes`: Ensure these are set logically based on your server resources and expected load. If `min_processes` is too low, it might not scale up fast enough.max_timeand `max_jobs`: These are crucial for memory leaks and code freshness. Setting `max_time` (e.g., 3600 seconds or 1 hour) ensures workers don't run indefinitely with old code or accumulating memory. `max_jobs` serves a similar purpose. When a worker reaches these limits, it gracefully exits and is restarted by the supervisor, picking up the latest code.
- Aggressive Caching Strategy: Make sure your deployment process includes a robust cache clearing and re-caching strategy. Stale configuration or opcode caches can lead to workers running old code or having issues deserializing jobs. Always run `php artisan config:cache`, `php artisan route:cache`, and `php artisan view:cache` after a deployment, and clear any other relevant application caches.
- Consider Zero-Downtime Deployment Tools: For more complex setups, tools like Laravel Forge, Envoyer, or custom scripts using `rsync` with atomic deployments (e.g., deploying to a new directory and symlinking) can help. These tools often handle the `horizon:terminate` and graceful restarts as part of their deployment flow, minimizing the window where workers might be out of sync.
The high CPU usage with low throughput you're observing can sometimes indicate workers are stuck in a loop, processing a malformed job, or struggling with a specific resource call that isn't failing outright but is taking an inordinate amount of time. Combining the `horizon:terminate` step with proper supervisor configuration and sensible `max_time`/`max_jobs` limits should significantly improve your post-deployment stability.
Hope this helps your conversions!
Isabella Miller
Answered 1 month agoRight, this is gold, Hiroshi Park. Really appreciate all these detailed steps, gonna bookmark this immediately!