Feb. 7/8 - Unattended Updated on Ubuntu Disconnecting the GPU

MainNet
#1

In the past 48 hours Ubuntu pushed an update related to the newest Nvidia driver, Version 510.

To determine if you are affected, please see if you node is listed as OFFLINE on https://dashboard.xx.network/

You can take two courses of action to bring the node back online if your node is OFFLINE …
See 4a - restart the computer so the unattended update completes.
See 4b - you can also take this opportunity to update the OS packages.

The driver package nvidia-driver-510 has been tested and is compatible with nodes participating on MainNet.

We suggest the following procedure, in this order …

  1. On the node computer, stop the cmix service:
    $ sudo systemctl stop xxnetwork-cmix
  2. On the gateway computer, stop the gateway and chain services:
    $ sudo systemctl stop xxnetwork-gateway xxnetwork-chain
  3. On the node computer, stop chain service:
    $ sudo systemctl stop xxnetwork-chain

4a. Complete the unattended update without updating all OS packages.
Restart the node and gateway computers.
$ sudo shutdown -r now

4b. Optional: Update additional OS packages.
Perform update/upgrade on the node and gateway computers and restart the computers.
$ sudo apt update && sudo apt upgrade
$ sudo shutdown -r now

  1. After the computers reboot, stop cmix and gateway services.
    Node:
    $ sudo systemctl stop xxnetwork-cmix
    Gateway:
    $ sudo systemctl stop xxnetwork-gateway
  2. Check chain.log on both computers to be sure it’s printing Imported and Idle messages which indicates it is synced.
    $ tail -n10 /opt/xxnetwork/log/chain.log
    You will want to re-issue the command until you see Idle and Imported messages similar to the following:
2022-02-08 15:35:48 ✨ Imported #1186571 (0x0cbf…2b7c)    
2022-02-08 15:35:52 💤 Idle (50 peers), best: #1186571 (0x0cbf…2b7c), finalized #1186569 (0x2d6c…f6e6), ⬇ 385.9kiB/s ⬆ 697.6kiB/s    
2022-02-08 15:35:54 ✨ Imported #1186572 (0x6c96…801f)    
2022-02-08 15:35:57 💤 Idle (50 peers), best: #1186572 (0x6c96…801f), finalized #1186570 (0x169d…3875), ⬇ 492.2kiB/s ⬆ 650.2kiB/s    
2022-02-08 15:36:00 ✨ Imported #1186573 (0xf6f2…2db2)    
2022-02-08 15:36:02 💤 Idle (50 peers), best: #1186573 (0xf6f2…2db2), finalized #1186571 (0x0cbf…2b7c), ⬇ 658.1kiB/s ⬆ 688.8kiB/s
  1. After confirming the chain is synced on each computer, start the cmix and gateway services again.
    Node:
    $ sudo systemctl start xxnetwork-cmix
    Gateway:
    $ sudo systemctl start xxnetwork-gateway

After a few minutes, you should be back online and running rounds.
If you are having trouble, please visit the xx network Discord server, email [email protected] or ask for assistance in this post.

2 Likes
#2

This is my nvidia-smi output.

Tue Feb 8 08:05:28 2022
±----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:09:00.0 Off | N/A |
| 38% 37C P2 37W / 215W | 716MiB / 7981MiB | 5% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2086 C …etwork/bin/xxnetwork-cmix 713MiB |
±----------------------------------------------------------------------------+

Should i be updating the nvidia driver?
I do update with sudo apt update and sudo apt upgrade , weekly

#3

You do not need to do the driver update. You should only need to allow the update to complete by restarting the node computer.
If you want to update to the newest driver, 510, you can run …
$ sudo apt update
$ sudo apt install nvidia-driver-510
$ sudo shutdown -r now

The 510 driver should not be expected to provide any advantage over previous versions.

2 Likes
#4

I did the update, worked like a charm

±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+

2 Likes