Im deploying to my server, using ansible, part of which runs a bunch of "update" commands before I start installing/updating packages. I recently got an error about disk space, but appear to be going in circles trying to fix it, below is as much informationas I can think to provide as to what has happened.
starting error:
sudo apt-get --fix-broken install -y
Gave the following:
sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
linux-headers-3.13.0-160
The following NEW packages will be installed:
linux-headers-3.13.0-160
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
Need to get 0 B/8,911 kB of archives.
After this operation, 63.6 MB of additional disk space will be used.
(Reading database ... 505365 files and directories currently installed.)
Preparing to unpack .../linux-headers-3.13.0-160_3.13.0-160.210_all.deb ...
Unpacking linux-headers-3.13.0-160 (3.13.0-160.210) ...
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb (--unpack):
error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
For me, the important part was:
error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device
No apport report written because the error message indicates a disk full error
So, after some googling, I ended up at these places: /boot and /usr/src/, and checking innodes
df /boot | df /usr/src:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8115168 5474876 2205016 72% /
df -i:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 480531 391 480140 1% /dev
tmpfs 481833 312 481521 1% /run
/dev/xvda1 524288 521345 2943 100% /
none 481833 2 481831 1% /sys/fs/cgroup
none 481833 1 481832 1% /run/lock
none 481833 1 481832 1% /run/shm
none 481833 2 481831 1% /run/user
/dev/xvdb 262144 11 262133 1% /mnt
uname -r:
3.13.0-158-generic
I'm not great at this stuff, hence trying to provide as much information as possible. I apologise if any of this is useless.
In the /usr/src I have a lot of linux-headers:
drwxr-xr-x 24 root root 4096 Nov 24 2016 linux-headers-3.13.0-101
drwxr-xr-x 7 root root 4096 Nov 24 2016 linux-headers-3.13.0-101-generic
drwxr-xr-x 24 root root 4096 Jan 20 2017 linux-headers-3.13.0-107
drwxr-xr-x 7 root root 4096 Jan 20 2017 linux-headers-3.13.0-107-generic
drwxr-xr-x 24 root root 4096 Mar 9 2017 linux-headers-3.13.0-112
drwxr-xr-x 7 root root 4096 Mar 9 2017 linux-headers-3.13.0-112-generic
drwxr-xr-x 24 root root 4096 Apr 7 2017 linux-headers-3.13.0-116
drwxr-xr-x 7 root root 4096 Apr 7 2017 linux-headers-3.13.0-116-generic
drwxr-xr-x 24 root root 4096 May 4 2017 linux-headers-3.13.0-117
drwxr-xr-x 7 root root 4096 May 4 2017 linux-headers-3.13.0-117-generic
drwxr-xr-x 24 root root 4096 May 18 2017 linux-headers-3.13.0-119
drwxr-xr-x 7 root root 4096 May 18 2017 linux-headers-3.13.0-119-generic
drwxr-xr-x 24 root root 4096 Jun 22 2017 linux-headers-3.13.0-121
drwxr-xr-x 7 root root 4096 Jun 22 2017 linux-headers-3.13.0-121-generic
drwxr-xr-x 24 root root 4096 Jul 7 2017 linux-headers-3.13.0-123
drwxr-xr-x 7 root root 4096 Jul 7 2017 linux-headers-3.13.0-123-generic
drwxr-xr-x 24 root root 4096 Jul 18 2017 linux-headers-3.13.0-125
drwxr-xr-x 7 root root 4096 Jul 18 2017 linux-headers-3.13.0-125-generic
drwxr-xr-x 24 root root 4096 Aug 31 2017 linux-headers-3.13.0-129
drwxr-xr-x 7 root root 4096 Aug 31 2017 linux-headers-3.13.0-129-generic
drwxr-xr-x 24 root root 4096 Jan 25 2018 linux-headers-3.13.0-141
drwxr-xr-x 7 root root 4096 Jan 25 2018 linux-headers-3.13.0-141-generic
drwxr-xr-x 24 root root 4096 Feb 22 2018 linux-headers-3.13.0-142
drwxr-xr-x 7 root root 4096 Feb 22 2018 linux-headers-3.13.0-142-generic
drwxr-xr-x 24 root root 4096 Mar 27 2018 linux-headers-3.13.0-143
drwxr-xr-x 7 root root 4096 Mar 27 2018 linux-headers-3.13.0-143-generic
drwxr-xr-x 24 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145
drwxr-xr-x 7 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145-generic
drwxr-xr-x 24 root root 4096 May 14 11:11 linux-headers-3.13.0-147
drwxr-xr-x 7 root root 4096 May 14 11:11 linux-headers-3.13.0-147-generic
drwxr-xr-x 24 root root 4096 May 23 11:04 linux-headers-3.13.0-149
drwxr-xr-x 7 root root 4096 May 23 11:04 linux-headers-3.13.0-149-generic
drwxr-xr-x 24 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158
drwxr-xr-x 7 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158-generic
drwxr-xr-x 7 root root 4096 Oct 12 11:15 linux-headers-3.13.0-160-generic
drwxr-xr-x 24 root root 4096 Mar 25 2015 linux-headers-3.13.0-48
drwxr-xr-x 7 root root 4096 Mar 25 2015 linux-headers-3.13.0-48-generic
I assume I don't need many of these, but when I try to to uninstall the 2015 kernels, I get the same error as before, which is complaining about the most recent and the fact it can not install its dependency:
sudo apt-get remove linux-headers-3.13.0-48
sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-headers-3.13.0-48-generic : Depends: linux-headers-3.13.0-48 but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
If I try "apt-get -f install", I get same message as before about no space.
/boot ls
abi-3.13.0-101-generic config-3.13.0-149-generic System.map-3.13.0-117-generic
abi-3.13.0-107-generic config-3.13.0-158-generic System.map-3.13.0-119-generic
abi-3.13.0-112-generic config-3.13.0-160-generic System.map-3.13.0-121-generic
abi-3.13.0-116-generic config-3.13.0-48-generic System.map-3.13.0-123-generic
abi-3.13.0-117-generic grub System.map-3.13.0-125-generic
abi-3.13.0-119-generic initrd.img-3.13.0-101-generic System.map-3.13.0-129-generic
abi-3.13.0-121-generic initrd.img-3.13.0-107-generic System.map-3.13.0-141-generic
abi-3.13.0-123-generic initrd.img-3.13.0-112-generic System.map-3.13.0-142-generic
abi-3.13.0-125-generic initrd.img-3.13.0-116-generic System.map-3.13.0-143-generic
abi-3.13.0-129-generic initrd.img-3.13.0-117-generic System.map-3.13.0-145-generic
abi-3.13.0-141-generic initrd.img-3.13.0-119-generic System.map-3.13.0-147-generic
abi-3.13.0-142-generic initrd.img-3.13.0-121-generic System.map-3.13.0-149-generic
abi-3.13.0-143-generic initrd.img-3.13.0-123-generic System.map-3.13.0-158-generic
abi-3.13.0-145-generic initrd.img-3.13.0-125-generic System.map-3.13.0-160-generic
abi-3.13.0-147-generic initrd.img-3.13.0-129-generic System.map-3.13.0-48-generic
abi-3.13.0-149-generic initrd.img-3.13.0-141-generic vmlinuz-3.13.0-101-generic
abi-3.13.0-158-generic initrd.img-3.13.0-142-generic vmlinuz-3.13.0-107-generic
abi-3.13.0-160-generic initrd.img-3.13.0-143-generic vmlinuz-3.13.0-112-generic
abi-3.13.0-48-generic initrd.img-3.13.0-145-generic vmlinuz-3.13.0-116-generic
config-3.13.0-101-generic initrd.img-3.13.0-147-generic vmlinuz-3.13.0-117-generic
config-3.13.0-107-generic initrd.img-3.13.0-149-generic vmlinuz-3.13.0-119-generic
config-3.13.0-112-generic initrd.img-3.13.0-158-generic vmlinuz-3.13.0-121-generic
config-3.13.0-116-generic initrd.img-3.13.0-160-generic vmlinuz-3.13.0-123-generic
config-3.13.0-117-generic initrd.img-3.13.0-48-generic vmlinuz-3.13.0-125-generic
config-3.13.0-119-generic retpoline-3.13.0-143-generic vmlinuz-3.13.0-129-generic
config-3.13.0-121-generic retpoline-3.13.0-145-generic vmlinuz-3.13.0-141-generic
config-3.13.0-123-generic retpoline-3.13.0-147-generic vmlinuz-3.13.0-142-generic
config-3.13.0-125-generic retpoline-3.13.0-149-generic vmlinuz-3.13.0-143-generic
config-3.13.0-129-generic retpoline-3.13.0-158-generic vmlinuz-3.13.0-145-generic
config-3.13.0-141-generic retpoline-3.13.0-160-generic vmlinuz-3.13.0-147-generic
config-3.13.0-142-generic System.map-3.13.0-101-generic vmlinuz-3.13.0-149-generic
config-3.13.0-143-generic System.map-3.13.0-107-generic vmlinuz-3.13.0-158-generic
config-3.13.0-145-generic System.map-3.13.0-112-generic vmlinuz-3.13.0-160-generic
config-3.13.0-147-generic System.map-3.13.0-116-generic vmlinuz-3.13.0-48-generic
I tried removing the latest kernel from here - that seems to have been partially installed:
sudo apt-get purge linux-image-3.13.0-160
Gave me this:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'linux-image-3.13.0-160-lowlatency' for regex 'linux-image-3.13.0-160'
Note, selecting 'linux-image-3.13.0-160-generic' for regex 'linux-image-3.13.0-160'
Package 'linux-image-3.13.0-160-lowlatency' is not installed, so not removed
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-image-virtual : Depends: linux-image-3.13.0-160-generic but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
I'm really confused as to what to do. If anyone can point me in the right direction, that would be great! Am I go to just use rm to remove some of the old files in either /boot or /usr/src ? Cos most posts suggested not to do this, but to use the apt-get remove etc.. commands
Cheers
You are out indeed of inodes (you knew that already).
The condition is indeed caused by all those kernel header packages (you knew that, too).
Apt cannot do the uninstalls because completing the in-process installs are earlier in the queue...and those in-process installs abort due to out-of-inodes. Apt always tries to complete in-process actions before new actions. (you figured that out, too)
The correct easy trick here is to uninstall the first kernel header package using dpkg instead of apt. This will free enough inodes to unclog apt. Subsequent uninstalls can be done using apt.
sudo dpkg --remove linux-headers-3.13.0-48 linux-headers-3.13.0-48-generic
sudo apt-get remove linux-headers-3.13.0-101 linux-headers-3.13.0-101-generic
There's a bit more cleaning up to be done - lots more kernel headers, plus lots of obsolete kernel images hanging out in /boot.
Pro Tip: NEVER use rm to remove files placed by the package manager. You will simply replace today's out-of-space error with next week's missing-file error...which can be much harder to fix. ALWAYS use the package manager to remove files that it placed.
No comments:
Post a Comment