ERROR or OFFLINE Nodes

If you find the node status is ERROR or OFFLINE on the xx network MainNet Dashboard, the first thing is to determine if any processes have stopped running and determine the cause of the crash. To do this run the following command on both the node and gateway computer:

$ ps -A | grep xxn

If you find any of the processes are <defunct>, you should check the logs, which can be found in /opt/xxnetwork/log/, before restarting the services. Some crashes can be prevented from happening repeatedly by making minor configuration changes. With a log message you can ask others for help.

You may be able to restart the service that has crashed to get the node back ONLINE. However, it is advised you take the following step to ensure it will recover properly.

  1. Visit the Account Actions tab of xx wallet

  2. If the only action is Stop, DO NOT PRESS STOP, just proceed to the Step 3. If you see the Validate button beside the stash and controller accounts. Press the Validate button and follow the prompts.

  3. Restart the chain services on the node and/or gateway computers:

$ sudo systemctl restart xxnetwork-chain
  1. The cmix and gateway processes require that chain be up to date with the current block. To check, tail the chain logs and wait until you see multiple Imported messages scrolling by in the logs on both the node and gateway computers.
$ tail -F /opt/xxnetwork/log/chain.log

The messages will look like so…

2022-07-21 14:32:28 💤 Idle (41 peers), best: #3512831 (0xb26c…8c8e), finalized #3512829 (0xd3b7…bcea), ⬇ 421.8kiB/s ⬆ 552.7kiB/s    
2022-07-21 14:32:30 ✨ Imported #3512832 (0xab2a…9f71)    
2022-07-21 14:32:33 💤 Idle (41 peers), best: #3512832 (0xab2a…9f71), finalized #3512830 (0xd6e8…3751), ⬇ 552.1kiB/s ⬆ 758.9kiB/s    
2022-07-21 14:32:36 ✨ Imported #3512833 (0x4ccc…21a2)
  1. Once multiple Imported messages are regularly appearing in both the chain.log files, restart the gateway and cmix services.

On the gateway computer run:

$ sudo systemctl restart xxnetwork-gateway

On the node computer run:

$ sudo systemctl restart xxnetwork-cmix
  1. If you had to press Validate in Step 2, the validator may have been moved to the Waiting tab of xx wallet

You can check how long it must wait before joining the list of active validators by checking the amount of time remaining in the current era, on the Overview tab of xx wallet. It might be multiple hours or even days if not elected.

  1. Once the validator has been elected and is an active validator, the cmix and gateway processes will start. It might take a few minutes for the scheduling server to add the node to the NDF.

If you find the node status is ERROR or OFFLINE often, please contact support at [email protected] or on the xx network Discord channel.

21 Likes

This is helpful and so rightfully on time.

1 Like

Great post about some practical problems, so good you guys helped out at the start yesterday Really appreciate this!

2 Likes

Very helpful