寫java代碼，文件資源的釋放需要特別謹(jǐn)慎的對待.通常文件資源使用后必須close，然后再刪除。

如果先刪除但沒有close掉，會造成文件句柄未被釋放，這會造成實際使用磁盤空間較大，刪除文件不釋放磁盤空間。

此時文件關(guān)閉了，但是out還持有文件，out未關(guān)閉則文件句柄未被釋放，會造成實際可使用空間小于可使用空間。

文件句柄的調(diào)試可用lsof 命令進(jìn)行查看：

lsof  -s | grep  java
lsof  -s |grep deleted

系統(tǒng)告警磁盤空間不足，因為某個服務(wù)一直在刷錯誤日志，磁盤爆了，把容器刪除重新起了一個。

df -h 后磁盤空間沒有釋放

du -sh 統(tǒng)計沒有占用那么多空間

通過指令：lsof | grep deleted 指令，查看當(dāng)前系統(tǒng)句柄未釋放情況

因為都是容器空間，所以只查看容器進(jìn)程未釋放的文件句柄。

lsof | grep deleted
lsof -p 3495 | grep deleted
lsof -p $(ps aux |grep dockerd |grep -v grep  |awk '{print$2}') | grep deleted

發(fā)現(xiàn)有很多已經(jīng)不存在的容器空間文件句柄未釋放。

問題找到后怎么解決，有兩種方法。

1、將當(dāng)前線程進(jìn)行重啟，關(guān)閉線程，從而讓句柄釋放，釋放空間。
2、找到指定的文件句柄，將當(dāng)前文件句柄的大小設(shè)置為空。

第一種方法頻繁重啟不適合當(dāng)前業(yè)務(wù)場景在生產(chǎn)環(huán)境不適用，采用第二種方法。

通過lsof | grep deleted拿到 PID（進(jìn)程標(biāo)識符）和 FD（文件描述符，應(yīng)用程序通過文件描述符識別該文件。）

image.png

置空文件內(nèi)容，然后查看磁盤使用發(fā)現(xiàn)空間恢復(fù)了：

echo > /proc/${pid}/fd/${fd}
echo > /proc/3840/fd/124
或者
truncate -s 0 /proc/${pid}/fd/${fd}
truncate -s 0 /proc/3840/fd/124

文件刪除空間不釋放，必須重啟解決？
DBA日常運維過程中經(jīng)常會遇到服務(wù)器磁盤空間不足的問題，容易一頓操作猛如虎，直接刪除服務(wù)器不常用的日志和文件，然而空間并沒有釋放，給后來者留下隱患。

##/usr/local空間不足
# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  9.0G  9.6G  49% /
/dev/sda3        20G  8.2G   11G  79% /usr/local
/dev/sda4       401G  297G   84G  78% /data

##查看只有8.2G，實際占用15G
# du -sh /usr/local/
8.2G    /usr/local/

檢查/usr/local目錄下刪除的文件，發(fā)現(xiàn)有日志被刪除，但是不少進(jìn)程占用文件句柄，空間并未釋放

# lsof /usr/local/|grep -i delete

mysqld_sa 21589 user_00    2w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
router_up 27504 user_00    1w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
router_up 27504 user_00    2w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 28895 user_00    1w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 28895 user_00    2w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 28897 user_00    1w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 28897 user_00    2w   REG    8,3 6858136961 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
router_up 70763 user_00    1w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
router_up 70763 user_00    2w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 71398 user_00    1w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 71398 user_00    2w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 71403 user_00    1w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
dcagent_t 71403 user_00    2w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
java      72021 user_00    1w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
java      72021 user_00    2w   REG    8,3 6858137684 1053546 /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)

方案一：重啟相關(guān)進(jìn)程
這是最常見的方案，只有重啟相關(guān)進(jìn)程后，占用的文件句柄才會釋放，磁盤空間也會釋放

方案二：置空未釋放文件句柄的文件

##從相關(guān)進(jìn)程中隨機(jī)找1個，查看文件句柄
# ls -l /proc/21589/fd
total 0
lr-x------ 1 user_00 users 64 Jun 22 15:05 0 -> /dev/nulll
-wx------ 1 user_00 users 64 Jun 22 15:05 1 -> /data/log/dblogs/nohup.out
l-wx------ 1 user_00 users 64 Jun 22 15:05 2 -> /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)
lrwx------ 1 user_00 users 64 Jun 22 15:05 3 -> socket:[3094866717]

##發(fā)現(xiàn)文件句柄2占用了刪除文件
l-wx------ 1 user_00 users 64 Jun 22 15:05 2 -> /usr/local/soft/shell_pkginstall_2021-07-07.log (deleted)

##清空文件句柄2占用的文件
#  >/proc/21589/fd/2

##查看磁盤，發(fā)現(xiàn)空間已釋放
# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G  9.0G  9.6G  49% /
/dev/sda3        20G  8.2G   11G  79% /usr/local
/dev/sda4       401G  297G   84G  78% /data

#優(yōu)化文件打開數(shù)
##CentOS6.x版本，是先讀/etc/security/limits.conf，如果/etc/security/limits.d/目錄下還有配置文件的話
##CentOS7.x會遍歷讀取里面文件，所以/etc/security/limits.d/里面的文件里面的配置會覆蓋/etc/security/limits.conf的配置

##注釋原有的nofile行
sed -i "/nofile/s/^/#/g" /etc/security/limits.conf

##注釋原有的nproc行
sed -i "/nproc/s/^/#/g" /etc/security/limits.conf

echo "* soft nofile 1048576" >>/etc/security/limits.conf
echo "* hard nofile 1048576" >>/etc/security/limits.conf
echo "root soft nofile 1048576" >>/etc/security/limits.conf
echo "root hard nofile 1048576" >>/etc/security/limits.conf
echo "* soft nproc 65535" >>/etc/security/limits.conf
echo "* hard nproc 65535" >>/etc/security/limits.conf
echo "root soft nproc unlimited" >>/etc/security/limits.conf
echo "root hard nproc unlimited" >>/etc/security/limits.conf

##注釋原有的nproc行
sed -i "/nproc/s/^/#/g" /etc/security/limits.d/90-nproc.conf

##注釋原有的nofile行
sed -i "/nofile/s/^/#/g" /etc/security/limits.d/90-nproc.conf

echo "* soft nofile 1048576" >>/etc/security/limits.d/90-nproc.conf
echo "* hard nofile 1048576" >>/etc/security/limits.d/90-nproc.conf
echo "root soft nofile 1048576" >>/etc/security/limits.d/90-nproc.conf
echo "root hard nofile 1048576" >>/etc/security/limits.d/90-nproc.conf
echo "* soft nproc 65535" >>/etc/security/limits.d/90-nproc.conf
echo "* hard nproc 65535" >>/etc/security/limits.d/90-nproc.conf
echo "root soft nproc unlimited" >>/etc/security/limits.d/90-nproc.conf
echo "root hard nproc unlimited" >>/etc/security/limits.d/90-nproc.conf
echo "* soft memlock unlimited" >>/etc/security/limits.d/90-nproc.conf
echo "* hard memlock unlimited" >>/etc/security/limits.d/90-nproc.conf