If you find the node status is ERROR or OFFLINE on the xx network MainNet Dashboard, the first thing is to determine if any processes have stopped running and determine the cause of the crash. To do this run the following command on both the node and gateway computer:
$ ps -A | grep xxn
If you find any of the processes are <defunct>, you should check the logs, which can be found in /opt/xxnetwork/log/, before restarting the services. Some crashes can be prevented from happening repeatedly by making minor configuration changes. With a log message you can ask others for help.
You may be able to restart the service that has crashed to get the node back ONLINE. However, it is advised you take the following step to ensure it will recover properly.
-
Visit the Account Actions tab of xx wallet
-
If the only action is
Stop, DO NOT PRESS STOP, just proceed to the Step 3. If you see theValidatebutton beside the stash and controller accounts. Press theValidatebutton and follow the prompts. -
Restart the chain services on the node and/or gateway computers:
$ sudo systemctl restart xxnetwork-chain
- The cmix and gateway processes require that chain be up to date with the current block. To check,
tailthe chain logs and wait until you see multipleImportedmessages scrolling by in the logs on both the node and gateway computers.
$ tail -F /opt/xxnetwork/log/chain.log
The messages will look like so…
2022-07-21 14:32:28 💤 Idle (41 peers), best: #3512831 (0xb26c…8c8e), finalized #3512829 (0xd3b7…bcea), ⬇ 421.8kiB/s ⬆ 552.7kiB/s
2022-07-21 14:32:30 ✨ Imported #3512832 (0xab2a…9f71)
2022-07-21 14:32:33 💤 Idle (41 peers), best: #3512832 (0xab2a…9f71), finalized #3512830 (0xd6e8…3751), ⬇ 552.1kiB/s ⬆ 758.9kiB/s
2022-07-21 14:32:36 ✨ Imported #3512833 (0x4ccc…21a2)
- Once multiple
Importedmessages are regularly appearing in both the chain.log files, restart the gateway and cmix services.
On the gateway computer check the gateway.log file which is located in /opt/xxnetwork/log/ and see if there are any notable log messages, if not, run:
$ sudo systemctl restart xxnetwork-gateway
On the node computer check the cmix.log file which is located in /opt/xxnetwork/log/ and see if there are any notable log messages, if not, run:
$ sudo systemctl restart xxnetwork-cmix
- If you had to press
Validatein Step 2, the validator may have been moved to the Waiting tab of xx wallet
You can check how long it must wait before joining the list of active validators by checking the amount of time remaining in the current era, on the Overview tab of xx wallet. It might be multiple hours or even days if not elected.
- Once the validator has been elected and is an active validator, the cmix and gateway processes will start. It might take a few minutes for the scheduling server to add the node to the NDF.
You can follow the cmix and gateway logs by running the following on the respective computer …
On the node computer …
$ tail -F /opt/xxnetwork/log/cmix.log
On the gateway computer …
$ tail -F /opt/xxnetwork/log/gateway.log
You can also get the last relevant messages [FATAL, ERROR, INFO, etc] from the cmix and gateway logs by running the following on the respective computer …
On the node computer …
$ tail -n 200 /opt/xxnetwork/log/cmix.log
On the gateway computer …
$ tail -n 200 /opt/xxnetwork/log/gateway.log
If you find the node status is ERROR or OFFLINE often, please ask for assistance on the xx network Discord channel.