Unattended Updates on Ubuntu Disconnecting the GPU

To determine if you are affected, please see if you node is listed as OFFLINE on https://dashboard.xx.network/

You can take two courses of action to bring the node back online if your node is OFFLINE …
See Option 4a - restart the computer so the unattended update completes.
See Option 4b - you can also take this opportunity to update the OS packages.

I suggest the following procedure, in this order …

Step 1. On the node computer, stop the cmix service:
$ sudo systemctl stop xxnetwork-cmix

Step 2. On the gateway computer, stop the gateway and chain services:
$ sudo systemctl stop xxnetwork-gateway xxnetwork-chain

Step 3. On the node computer, stop chain service:
$ sudo systemctl stop xxnetwork-chain

Option 4a. Complete the unattended update without updating all OS packages.
Restart the node and gateway computers.
$ sudo shutdown -r now

Option 4b. Optional: Update additional OS packages.
Perform update/upgrade on the node and gateway computers and restart the computers.
$ sudo apt update && sudo apt upgrade
$ sudo shutdown -r now

Step 5. After the computers reboot, stop cmix and gateway services.
Node:
$ sudo systemctl stop xxnetwork-cmix
Gateway:
$ sudo systemctl stop xxnetwork-gateway

Step 6. Check chain.log on both computers to be sure it’s printing Imported and Idle messages which indicates it is synced.
$ tail -F /opt/xxnetwork/log/chain.log
You will want to re-issue the command until you see Idle and Imported messages similar to the following:

2022-02-08 15:35:48 ✨ Imported #1186571 (0x0cbf…2b7c)    
2022-02-08 15:35:52 💤 Idle (50 peers), best: #1186571 (0x0cbf…2b7c), finalized #1186569 (0x2d6c…f6e6), ⬇ 385.9kiB/s ⬆ 697.6kiB/s    
2022-02-08 15:35:54 ✨ Imported #1186572 (0x6c96…801f)    
2022-02-08 15:35:57 💤 Idle (50 peers), best: #1186572 (0x6c96…801f), finalized #1186570 (0x169d…3875), ⬇ 492.2kiB/s ⬆ 650.2kiB/s    
2022-02-08 15:36:00 ✨ Imported #1186573 (0xf6f2…2db2)    
2022-02-08 15:36:02 💤 Idle (50 peers), best: #1186573 (0xf6f2…2db2), finalized #1186571 (0x0cbf…2b7c), ⬇ 658.1kiB/s ⬆ 688.8kiB/s

Step 7. After confirming the chain is synced on each computer, start the gateway and cmix services again.
Gateway:
$ sudo systemctl start xxnetwork-gateway
Node:
$ sudo systemctl start xxnetwork-cmix

You can follow the cmix and gateway logs by running the following on the respective computer …
On the node computer …
$ tail -F /opt/xxnetwork/log/cmix.log
On the gateway computer …
$ tail -F /opt/xxnetwork/log/gateway.log

After 5-10 minutes, you should be running rounds.
If you are having trouble and need additional assistance, please visit the xx network Discord server.

3 Likes

This is my nvidia-smi output.

Tue Feb 8 08:05:28 2022
±----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:09:00.0 Off | N/A |
| 38% 37C P2 37W / 215W | 716MiB / 7981MiB | 5% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2086 C …etwork/bin/xxnetwork-cmix 713MiB |
±----------------------------------------------------------------------------+

Should i be updating the nvidia driver?
I do update with sudo apt update and sudo apt upgrade , weekly

You do not need to do the driver update. You should only need to allow the update to complete by restarting the node computer.
If you want to update to the latest driver at this time, 510, you can run …
$ sudo apt update
$ sudo apt install nvidia-driver-510
$ sudo shutdown -r now

The 510 driver should not be expected to provide any advantage over previous versions.

As time passes 510 will not be the latest so to find the latest version one can run …

$ sudo apt update
$ apt-cache pkgnames nvidia-driver
2 Likes

I did the update, worked like a charm

±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+

2 Likes