The xxnetwork Logs and Useful Tools

In this post I am going to write about:

  • The xxnetwork Logs
  • Using less to view a log
  • Using grep to search a log
  • Using Tail to follow a log (Tip: tail | grep)
  • Using MultiTail to follow multiple logs

The xxnetwork Logs

By default the logging of node and gateway are set at logLevel: 0 in their respective yaml configuration files. There are higher log levels but for each level, more information is printed and the log files grow in size, often very quickly. For all intents and purposes you should leave it at 0 unless you plan to help debug code. Here are a few example log messages just to show the format.

INFO 2020/08/01 09:37:12 Updating to STANDBY
WARN 2020/08/01 09:33:43 Could not get CPU usage info: no CPU time progression since last call
ERROR 2020/08/01 09:31:12 Attempt number 3 to connect to 0.0.0.0:22222 failed: context deadline exceeded
FATAL 2020/08/01 09:22:33 Failed to change state to ERROR state: Resource queue kill timed out: Signal denoting kill of resource queue kill was not received
  • Each starts with the Log Level indicator. You can see INFO, ERROR, WARN, and FATAL. There are others such as DEBUG and TRACE but that is for another post.
  • Next are the date and time the event occurred. This will be displayed in your local time.
  • Finally, the message itself. In a perfect world these messages would make sense to anyone with a basic grasp of the English language. In reality, developers do not use the same English most humans do.

There are three logs a node operator should be able to access and have a reasonable understanding of what they are looking at. The files are:

  • node.log - a history of the xxnetwork-node process
  • gateway.log - a history of the xxnetwork-gateway process
  • xxnetwork-wrapper.log - a history of either the node or gateway service

Where are the logs?

If you are running the node and gateway software on the same PC then all the logs are located on the one PC. If you are running the node and gateway software on separate PCs, then node logs will be on the node PC, and gateway logs will be on the gateway PC.

Either way the default paths are:

  • node logs:
/opt/xxnetwork/node-logs/
├── node.log
└── xxnetwork-wrapper.log
  • gateway logs:
/opt/xxnetwork/gateway-logs/
├── gateway.log
└── xxnetwork-wrapper.log

Now that you know where to look, let’s talk about how to look.


Useful Tools and Usage

As a node operator, most times you’re going to be glancing, searching and copying, or watching logs for short periods of time. Here are some of the ways you can interact with the logs when you just need to gather timely information from the logs.

less - less Manual Page

What is less?

less is a program which allows backward movement in the file as well as forward movement.

less Usage:

  • View the node.log file

$ less /opt/xxnetwork/node-logs/node.log

You are at the start of the log.
You can move through the log with Page Up and Page Down keys.
To quickly move to the end of the log, press the key combination, Shift+g.
To move back to the start of the log press the g key.
To exit less press the q key.


g/re/p - grep Manual Page

What is grep?

grep searches for and prints regular expressions in a file.

grep Usage:

  • Search for the single word “took” to find all the rounds your node participated in.

$ grep took /opt/xxnetwork/node-logs/node.log

The above command likely returned many lines similar to …

INFO 2020/08/01 16:34:32 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3587089 Round took 18.139590725s seconds

The sample output above shows the node participated in Round ID 3587089. Your node log will list other Round ID numbers so choose one and search for the phrase ‘RID [RID NUMBER]’. Phrases must be surrounded by single or double quotation marks.

  • Search for a phrase

$ grep 'RID 3587089' /opt/xxnetwork/node-logs/node.log

INFO 2020/08/01 10:19:03 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3587089 CreateNewRound COMPLETE
INFO 2020/08/01 10:19:04 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3587089 PostPhase FROM "PrecompGeneration" FOR "PrecompGeneration" RECIEVE/START
--truncated--

Tail - Tail Manual Page

What is tail?

tail prints the last 10 lines of a file to standard output.

Tail Usage:

  • By default tail shows the last 10 lines of the log.

$ tail /opt/xxnetwork/node-logs/node.log

INFO 2020/08/01 10:45:04 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3589424 Round took 13.739183607s seconds
INFO 2020/08/01 10:45:04 Updating to COMPLETED
INFO 2020/08/01 10:45:04 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3589424 ReceiveFinishRealtime CLEARING CMIX BUFFERS
INFO 2020/08/01 10:45:04 Host et7QqM1YuTpFnmN7ruyzB9uUYN9o3gpRQm7Uh4q2VXkC not connected, attempting to connect...
INFO 2020/08/01 10:45:04 Connecting to 45.33.58.107:11420. Attempt number 0 of 100
INFO 2020/08/01 10:45:05 Updating to WAITING
INFO 2020/08/01 10:45:05 Successfully connected to 45.33.58.107:11420
INFO 2020/08/01 10:45:05 Attempting to establish authentication with host et7QqM1YuTpFnmN7ruyzB9uUYN9o3gpRQm7Uh4q2VXkC
INFO 2020/08/01 10:45:05 Updating to PRECOMPUTING
INFO 2020/08/01 10:45:05 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3589442 CreateNewRound COMPLETE
  • You can show more or fewer lines with the -n flag

ubuntu@node02:~$ tail -n 2 /opt/xxnetwork/node-logs/node.log

INFO 2020/08/01 10:46:51 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC:22222]: RID 3589544 Beginning execution of Phase "RealPermute"
INFO 2020/08/01 10:46:52 [nz++X5QGYMdEIRY+8+49ICKuTe5OCev3YKbP80Sn6bIC]: RID 3589544 Finishing execution of Phase "RealPermute" -- Adapt: 801.405µs, outMod: 3.622334ms
  • You can “follow” a log to keep the log open and it will print new lines as they are added to the log.

$ tail -F /opt/xxnetwork/node-logs/node.log

To end the tail process, press the key combination Ctrl+c.

  • Combine tail and grep to follow and print specific events as they happen.

$ tail -F /opt/xxnetwork/node-logs/node.log | grep took
$ tail -F /opt/xxnetwork/node-logs/node.log | grep Adapt
$ tail -F /opt/xxnetwork/node-logs/node.log | grep 'Updating to'
$ tail -F /opt/xxnetwork/node-logs/node.log | grep 'with host'
$ tail -F /opt/xxnetwork/node-logs/node.log | grep 'Round trip'

To end the tail process, press the key combination Ctrl+c.


MultiTail - MultiTail Website

What is MultiTail?

MultiTail allows you to monitor logfiles and command output in multiple windows in a terminal, colorize, filter and merge.

How to install MultiTail

$ sudo apt update
$ sudo apt install multitail

MultiTail Usage:

  • The most basic usage would be to follow two logs at once in a split screen. You might want to follow the node.log and node wrapper log.

$ multitail -f /opt/xxnetwork/node-logs/node.log /opt/xxnetwork/node-logs/xxnetwork-wrapper.log

  • You can merge the two logs into one window.

$ multitail -f --mergeall /opt/xxnetwork/node-logs/node.log /opt/xxnetwork/node-logs/xxnetwork-wrapper.log

  • You might want to merge them into one window and print each in different colors so it’s easier to read.

$ multitail -f --mergeall -ci green /opt/xxnetwork/node-logs/node.log -ci red /opt/xxnetwork/node-logs/xxnetwork-wrapper.log

  • If you are running node and gateway on the same PC you can merge all four logs and have them print in different colors.

$ multitail -f --mergeall -ci blue /opt/xxnetwork/node-logs/node.log -ci green /opt/xxnetwork/gateway-logs/gateway.log -ci red /opt/xxnetwork/node-logs/xxnetwork-wrapper.log -ci yellow /opt/xxnetwork/gateway-logs/xxnetwork-wrapper.log

MultiTail has too many options to cover in this post. You can view multiple logs in multiple screens, split screens, merge logs together, colorize, sort, etc.

  • For a full list of options run:

$ multitail -h


ProTip: ssh + MultiTail + Tail = Monitor Multiple Remote Files Using MultiTail

That is it for this post. I hope it is helpful!

16 Likes