A large log has been deleted, yet df -h shows the same free space as before. Where did its blocks go?
A common cause is an open unlinked file. rm removes the name from the directory, but while some process keeps the file open, its blocks stay allocated. du no longer finds the name and does not count those blocks, hence the mismatch between df and du.
lsof shows such files: sudo lsof +L1 lists open files that have no directory name, along with the processes holding them. The space returns when the process closes the file.
Restarting every service in turn to reclaim space creates a new problem: downtime. Formatting, resizing the volume or deleting whole log collections is not where the fix starts either: first separate block exhaustion from inode exhaustion, find the source of growth and confirm what may be removed.
Once you are done, write down the path and filesystem, which resource ran out, what was removed and why that was safe, and how the new write was confirmed. The same sequence helps in the "Root partition is full" lab, where the blocks run out, as in the lesson's second exercise.
You deleted a 2 GiB service log, but df -h still shows the same free space. What do you do next?
-
A reboot frees the space because it closes every process, but it takes the whole server down. One process holds the deleted file; find it with
sudo lsof +L1. -
dfshows the current state, so there is nothing to wait for: while the service holds the deleted log open, its blocks stay in use. The space frees when the process closes the file. -
fsckon a mounted volume can damage it, and the counters here are correct: the blocks belong to an open file. The space returns when the process closes the file.
While a process keeps the file open, its blocks stay allocated even though the name is gone from the directory. sudo lsof +L1 lists such files and their processes; then use the supported way to reopen the log, such as restarting that one service.
Key points
-
A deleted file keeps its blocks while a process holds it open;
ducannot see those blocks. -
sudo lsof +L1shows deleted open files and the processes holding them.