Hi all,
I restarted to install updates after putting lots of new photos on my disk and then PureOS would not boot anymore (after disk decryption I just have a blinking ‘_’ in the upeer left corner). In recovery mode, df -h showed /dev/dm-0 to be 100% full despite 2 GB left. So I freed more than another 30GB but it still doesnt boot and still tells me its a 100% full.
How can I change boot entries to not be quiet, so I can see more information?I tried deleting ‘quiet splash’ in /etc/default/grub but it didnt change anything.
I would appreciate any help on how to get more information about the error (or on the problem itself of course).
Thanks a lot,
Algernon
1 Like
You probably want to use df to check, rather than du e.g.
df -h /
df -h /boot
Have you tried fsck to see whether you can repair the percentage full?
1 Like
Im sorry, I did df, stupid spelling mistake. fsck gives me an error: /dev/mapper/luks… is mounted. cannot continue, aborting. Can I safely unmount that disk?
1 Like
OK, I missed “recovery mode”. Do a live boot from a USB flash drive. (You will have to do the LUKS open manually then, I suppose.)
1 Like
Could you hint me to a good source for how do that? I can boot with a usb, but Im not sure how to open LUKS. If I do df -h in the system from usb I cant see the troubled disk at all.
1 Like
sudo cryptsetup luksOpen /dev/xyz foo
where you replace xyz with the device name of specific usual main disk and applicable partition
(it will prompt for the LUKS passphrase)
(if you don’t know the value for xyz then please say how many disks there are in the computer, and what make and model of disk(s), and what hardware interface type(s) e.g. NVMe v. SATA/USB, which would usually result in respectively nvme0n1pN and sdXN where N is the partition number and you have to work out X, and you can use e.g. fdisk -l /dev/sdX to sanity check your choice of disk device and verify that your choice of partition is plausible)
then
sudo fsck /dev/mapper/foo
(you may need to specify and vary options to fsck depending on what it says)
and when finished
sudo cryptsetup luksClose /dev/mapper/foo
1 Like
Thanks a lot! I have used these commands, before seeing your answer and fsck said clean…
The only thing Im not sure is, I openend with
sudo cryptsetup luksOpen /dev/xyz /dev/mapper/foo
instead of how you wrote just foo, is that a problem?
1 Like
I don’t think it’s a problem. It may be that the command accepts it either way.
But did you make any progress on the actual problem?
1 Like
Unfortunately not, I have to admit I dont know what to do to find out whats the problem. The only other thing I tried is looking at journalctl, there are a few orange/red warnings/errors, which a friend of mine looked at since I have no experience in interpreting them, but he says thats nothing which should make the laptop not boot anymore.
1 Like
If there’s confusion over whether the root file system is or is not 100% full then it wouldn’t surprise me that some components fail to start properly during boot. I think that’s what you need to resolve.
Note that freeing up disk space won’t do any good if you have more than one file system and you are freeing up space on the file system that is not full.
1 Like
I am sure, that I freed the space on the right disk, since I could see the used/availible space change, just the 100% stayed there. Do you have any other ideas what I could check to see where this discrepancy comes from?
Could it have todo something with a backup or sync or file transform that did not finish correctly? After I put new media on my laptop and transformed to other formats (or maybe still during the transformation I cannot say for sure), I got the warning that space was low and it said I should empty the trash, which I did but I guess that wasnt enough. Then I started rsync and borg backup to an external disk and went to bed and the next day the rsync had not managed to finish, things went slow and strange so I restarted, which in hindsight was not so smart, since then I cannot boot anymore.
1 Like
Can you confirm that the root file system type is ext4?
It is possible that the partition is not the right size i.e. there is unallocated space after the root file system instead of space in that partition. How many partitions are there on the disk that contains the root file system? How is the space divided between the partitions?
A typical set up would be
- a very small amount of unallocated space at the beginning of the disk
- a boot partition, maybe 512MB
- the root partition using the remainder of the disk, and, in the event that LUKS is in use, within the root partition, a LUKS header followed by the actual encrypted root file system occupying almost all of the partition
At a certain point your options become
- restore from backup, or
- rescue files and settings, and then reinstall from scratch
and if you don’t have a backup then the second option might be the only option.
To understand the possibility of the second option, from a Live Boot, after you do the luksOpen, you can attempt to mount the encrypted file system and see whether you get any problems doing that / see whether you can copy files off.
1 Like
I think it is ext4, after some googling I did
mount -v | grep " on / "
and it gave me /dev/mapper/luks… type ext4.
df -T shows the 100% use fs and the one mounted on boot also to be ext4.
About the partitions, (I hope I get that right), lsblk -ll says there is
a disk nvme01 with 931,5G,
with parts nvme01p1n 1.1G mountpoint boot and nvme01p2 930,4G.
and
luks-… also 930,4G type crypt mountpoint / .
(and an 8G disk zram with mountpoint [swap]
Does it mean I have only 2 partitions?
So the copying in life mode works without problems, I already got the data I wanted to back up (not the ~900G, they are already on 3 backup harddrives, just the newest things that I dont have there because the last try of backup failed). So then I have borgbackup from a few days earlier and all the data including /etc in sync. Problem is I have never used the borgbackup, Im wondering what is easier, probably just fresh install and copy everything there. Not very keen do do so, keep thinking there is an easy solution I miss because I just dont have the skills.
1 Like
Yes, and that would be very normal. I prefer the output of the partition table from fdisk -l on the actual underlying disk.
Can you post the actual output from df / but you can censor the actual UUID, if applicable, replacing it with UUID?
Can you post the actual output from df /boot and no need for censorship?
1 Like
/dev/mapper/luks-UUID 959137080 926316436 0 100%
/dev/nvme0n1p1 1125476 121808 928304 12% /boot
Thanks for the help!
1 Like
OK, two answers.
The underlying problem seems to be that Linux reserves 5% (default) of the file system space to root in case the disk fills up. System (root) processes can continue to operate above 95% full (thus ensuring that the system doesn’t die completely) while user processes get an error about the disk being full. You would then use the console (or a live boot) to clean up before the system dies completely.
So your choices are:
- Free up a little more disk space. By my calculation you are currently at 3.4% free. You need to get to 5% free or even a little beyond that so that we aren’t having this conversation again in a month’s time
. OR
- Change the threshold from 5% to e.g. 2%.
sudo tune2fs -m 2 /dev/mapper/luks-UUID
If the truth be told (well, if my opinion be told), the default of 5% is inappropriate for such a large root disk. That default no doubt makes sense on much smaller disks.
As a general system management observation … I assume that to fill a 1 TB disk, you must have a lot of user content. There would be some merit in using an extra partition for user content i.e. three partitions - boot, root and user content. You decide what you need in the boot+root partition for an ongoing viable computer (e.g. let’s say 100 GB for boot+root but I didn’t research that and it depends on your specifics and you could certainly screw a vanilla Linux install down smaller than that; and then 900 GB in the user content partition).
That way, no amount of user content will ever cause the root file system to be full (either full to the 95%/98% threshold, or actually full). So you will always be able to boot and log in regardless of the amount of user content.
However it is usually quite difficult to achieve that retrospectively. So let’s forget about that for the moment. An alternative of course is just to put user content on a completely separate disk.
2 Likes
Thanks so much, I just managed to boot again without recovery mode 
I did not know about this 5%, what a stupid problem. For the next time I set up a new system I will make 3 partitions! For now I will just shift stuff to another disk, so I can at least come up with new problems next time
2 Likes
For clarity, only do that on a larger root disk (such as 1 TB) - and only if you expect to use most of the disk space. With, say, a 256 GB root disk, you would waste too much space having 3 partitions.
At the end of the day, 3 partitions is a trade-off. You gain: reliability and availability. You lose: some disk space because you will inevitably use the total space less efficiently.
If configuring for only 2 partitions, there can be merit in having a cron job that warns you as you approach the 95% limit (or whatever other limit you set), rather than waiting until you get the failure that started this topic.
Anyway, glad to hear that you are back up and running again!
1 Like