cPanel Backup Script Failing Randomly After Recent Server Upkeep, Getting 'Disk Quota Exceeded' Errors

hey everyone,

we run 'Website Maintenance & cPanel Management Services' for our clients, and as part of our managed services, things usually run super smooth. but we've hit a real snag with our automated backups recently.

so, the problem is our cPanel full backup scripts are failing randomly. it's not every time, which makes it even more frustrating to debug, but enough to be a serious headache, especially after some recent server upkeep we did. we keep getting these 'Disk Quota Exceeded' errors, even when there's clearly plenty of space on the server.

i've tried a bunch of stuff:

• checked actual disk space on the server (df -h consistently shows ample free space across all partitions).
• verified cron job syntax and execution times, everything looks correct and permissions are fine.
• looked at cPanel error logs and Apache logs, but they're not always clear or specific about this particular error.
• tried reducing backup frequency and size temporarily for some accounts, but the error still pops up.
• checked user disk quotas within WHM, they seem perfectly fine for all the accounts encountering this issue.

the weirdest part is definitely the 'Disk Quota Exceeded' message. it just doesn't make sense given our checks. here's a snippet from a recent cPanel backup log when it failed:

[2023-10-27 03:00:01 -0500] info [backup] Initializing backup for user 'clientdomain'\n[2023-10-27 03:00:02 -0500] info [backup] Backup directory created: /backup/2023-10-27/accounts/clientdomain\n[2023-10-27 03:00:05 -0500] error [backup] Disk Quota Exceeded for user 'clientdomain' while writing data.\n[2023-10-27 03:00:05 -0500] error [backup] Backup failed for user 'clientdomain' with error: Disk Quota Exceeded.

this happens even when the account's actual disk usage is well under its set quota, like 5GB used out of a 50GB limit. could it be a temporary file system issue, maybe some inode limits getting hit, or something related to recent kernel updates during the server upkeep? we had a few patches applied last week.

has anyone experienced this specific 'Disk Quota Exceeded' error on cPanel backups when disk space and quotas appear fine? any ideas what else to check? anyone faced this before?