Move from: bobsql.com
I was troubleshooting a master-0 pod eviction due to disk pressure.
Right after the restart I saw the disk space usage drop to 38% but found no events in the journal or other logs indicating how the additional space was acquired. Looking at the disk metrics I was able to map the time of the disk space increase to the startup of SQL Server, specifically TEMPDB.
The TEMPDB had grown large, prior to the restart (auto grow no limit), consuming disk space on the master-0 persistent volume. Once the 80% disk space usage threshold was crossed the pod was evicted followed by the cleanup and restart attempt. Luckily, enough space was reclaimed by deleting the unused images allowing the master-0 pod to restart. The master-0 pod restarted (SQL Server restarted) and TEMPDB was recreated using the default file sizes. The TEMPDB recreation caused disk space usage to change from 73% to 38%.
If enough space could not be reclaimed as part of the eviction resolution, the pod generally goes into CrashLoopBackOff and/or remains in evicted state. When disk space used remains above the eviction threshold (80%) Kubernetes is unable to restart the pod and space reclamation requires intervention to correct. You commonly must log into the node and free disk space or add additional disk space resources for the persistent volume to allow the pod to restart.
Recommendation
Be sure to set file size limits on your TEMPDB to avoid unwanted disk space consumption leading to pod eviction.
