ERROR or OFFLINE Nodes

If you find the node status is ERROR or OFFLINE on the xx network ProtoNet Dashboard, the first thing is to determine if any processes have stopped running and determine the cause of the crash. To do this run the following command on both the node and gateway computer:

$ ps -A | grep xxn

If you find any of the processes are <defunct>, you should check the logs, which can be found in /opt/xxnetwork/log/

You may be able to restart the service that has crashed to get the node back ONLINE. However, it is advised you take the following step to ensure it will recover properly.

  1. Visit the Account Actions tab of the ProtoNet Explorer

  2. If the only action is Stop, you can proceed to the next step. Otherwise, depending on how long it has been offline, you may see the Validate button beside the stash and controller accounts. Press the Validate button and use the hexNodeID from the output of:

$ cat /opt/xxnetwork/cred/cmix-IDF.json
  1. Restart the chain services on the node and gateway computers:
$ sudo systemctl restart xxnetwork-chain
  1. The cmix and gateway processes require that chain be up to date with the current block. To check, tail the chain logs and wait until you see multiple Imported messages scrolling by in the logs on both the node and gateway computers.
$ tail -F /opt/xxnetwork/log/chain.log

The messages will look like so…

2021-09-20 09:16:48 ✨ Imported #589139 `(0x87bc…53b6)` 2021-09-20 09:16:51 💤 Idle (48 peers), best: #589139 `(0x87bc…53b6), finalized ` `#589137` (0x452e…b48e), ⬇ 109.4kiB/s ⬆ 261.2kiB/s `2021-09-20 09:16:54 ✨ Imported ` `#589140` (0xd87d…ab1d) `2021-09-20 09:16:56 💤 Idle (47 peers), best: ` `#589140` (0xd87d…ab1d), finalized #589138 `(0x4169…43b5), ⬇ 115.9kiB/s ⬆ 155.9kiB/s` 2021-09-20 09:17:00 ✨ Imported #589141 `` (0xf56d…cae6)

  1. Once multiple Imported messages are regularly appearing in both the chain.log files, restart the cmix and gateway services.

On the node computer run:

$ sudo systemctl restart xxnetwork-cmix

On the gateway computer run:

$ sudo systemctl restart xxnetwork-gateway
  1. If you had to press Validate in Step 2, the validator may have been moved to the Waiting tab of the ProtoNet Explorer

You can check how long it must wait before joining the list of active validators by checking the amount of time remaining in the current era, on the Overview tab of the ProtoNet Explorer. It might be multiple hours.

  1. Once the validator is no longer listed on the Waiting tab, the cmix and gateway processes will start. It might take a few minutes for the scheduling server to add the node to the NDF.

If you find the node status is ERROR or OFFLINE often, please contact support at [email protected] or on the xx network Discord channel.

19 Likes

This is helpful and so rightfully on time.

1 Like

Great post about some practical problems, so good you guys helped out at the start yesterday Really appreciate this!

2 Likes