As time and blocks pass, the disk space required to store the blockchain data grows.
As of the writing of this post, approximately 82GB is needed …
$ du -d 1 -h /opt/xxnetwork | grep db
82G /opt/xxnetwork/db
In recent days, I have on a few occasions been contacted by node operators whose xxnetwork-chain process crashes and the chain.log contains messages with No space left on device
The log message may vary …
Error: Service(Client(Backend("Io error: No space left on device (os error 28)")))
GRANDPA voter error: Client(DatabaseError(DatabaseError(Custom { kind: Other, error: Error { message: "IO error: No space left on device: While appending to file: /opt/xxnetwork/db/chains/xxnetwork/db/full/10105292.log: No space left on device" } })))
DatabaseError(DatabaseError(Custom { kind: Other, error: Error { message: "IO error: No space left on device: While appending to file: /opt/xxnetwork/db/chains/xxnetwork/db/full/6431858.log: No space left on device" } }))
… but the one thing in common is “No space left on device.”
Assuming the size of the computer’s storage device meets the Hardware Requirements :
- Node – 1TB
- Gateway – 500GB
… if xxnetwork-chain repeatedly crashes with one of the previously mentioned No space left on device messages, then it likely means the full available capacity of the storage device was not partitioned and formatted during the installation of the OS.
For example, nvme0n1, a 1TB (931.5G available) disk only had 100GB allocated to the Logical Volume named ubuntu--vg-ubuntu--lv and mounted on /
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1.1G 0 part /boot/efi
├─nvme0n1p2 259:2 0 1.5G 0 part /boot
└─nvme0n1p3 259:3 0 929G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 100G 0 lvm /
As a result, you can see 98% of ubuntu--vg-ubuntu--lv mounted on / is used.
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 98G 91G 2.8G 98% /
/dev/nvme0n1p2 1.5G 367M 1016M 27% /boot
/dev/nvme0n1p1 1.1G 5.3M 1.1G 1% /boot/efi
To rectify the situation, the Logical Volume can be extended to use the full available capacity of the partition. There are multiple ways to accomplish this so it is best you refer to general Ubuntu support via the internet which will provide you plenty of resources …
Search for keywords such as ubuntu extend lvm partition using DuckDuckGo or Google.
Once the issue of the disk being full is rectified, I would suggest you have a look at ERROR or OFFLINE Nodes to ensure all xx network related processes are functioning properly.
Additional notes:
- A full disk can cause other dependent services like PostgreSQL to crash. You might need to restart the postgresql service or just reboot the computer to make sure everything is running correctly.
- A full disk may also result in having to re-sync the chain due to database corruption. But, you will want to double check before you take steps required to fix DB has been previously marked as corrupted, attempting repair
And as always, if you’re still having issues please feel free to contact support at [email protected] or via the xx network Discord channel
Thanks for running a node!
Keith