FATAL Database call timed out: context deadline exceeded

The PostgreSQL database on the gateway computer is by far one of the most important components of an “xx network node.” The gateway database stores the message data of xx network dApps such as xx Messenger and codename: Speakeasy.

For each round a node participates in the message data sent during that round, and later retrieved, is stored in the gateway databases of the 5 nodes that participated in the round. The data being stored on 5 gateways provides redundancy but when a gateway computer crashes due to a database error, all of the message data on that gateway is not available to be retrieved by dApps.

If you find your gateway crashing regularly with the following …

FATAL yyyy/mm/dd hh:mm:ss Database call timed out: context deadline exceeded panic: Database call timed out: context deadline exceeded

… it is imperative you investigate the cause ASAP.

Usually, this crash is caused by poor IOPS of the storage device of the gateway computer. You should test the IOPS of the storage device. A benchmark can be done by running YABS (Yet-Another-Bench-Script) via the terminal …

$ curl -sL yabs.sh | bash -s -- -ig

To give you an idea of whether or not IOPS is the cause, I have provided some results from YABS of storage devices that meet or exceed the Gateway Hardware Requirements, followed by benchmark results from operators that have reported experiencing database timeout crashes regularly.

SATA SSD :white_check_mark: (All results greater than 150 MB/s)

Acceptable Benchmark Results
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 151.23 MB/s  (37.8k) | 178.33 MB/s   (2.7k)
Write      | 151.62 MB/s  (37.9k) | 179.26 MB/s   (2.8k)
Total      | 302.85 MB/s  (75.7k) | 357.60 MB/s   (5.5k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 188.80 MB/s    (368) | 193.34 MB/s    (188)
Write      | 198.83 MB/s    (388) | 206.22 MB/s    (201)
Total      | 387.64 MB/s    (756) | 399.57 MB/s    (389)

NVMe SSD :white_check_mark: (All results greater then 750 MB/s)

Recommened Benchmark Results
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 805.53 MB/s (201.3k) | 1.10 GB/s    (17.2k)
Write      | 807.66 MB/s (201.9k) | 1.11 GB/s    (17.3k)
Total      | 1.61 GB/s   (403.2k) | 2.21 GB/s    (34.6k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 2.28 GB/s     (4.4k) | 2.20 GB/s     (2.1k)
Write      | 2.40 GB/s     (4.6k) | 2.35 GB/s     (2.2k)
Total      | 4.68 GB/s     (9.1k) | 4.55 GB/s     (4.4k)

VPS - Operator Submitted Results :x: (Any result less than 100 MB/s)

Poor Benchmark Results
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 5.26 MB/s     (1.3k) | 63.69 MB/s     (995)
Write      | 5.28 MB/s     (1.3k) | 64.10 MB/s    (1.0k)
Total      | 10.55 MB/s    (2.6k) | 127.80 MB/s   (1.9k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 127.51 MB/s    (249) | 98.42 MB/s      (96)
Write      | 134.28 MB/s    (262) | 104.98 MB/s    (102)
Total      | 261.79 MB/s    (511) | 203.40 MB/s    (198)

VPS - Operator Submitted Results :x: (Any result less than 100 MB/s)

Poor Benchmark Results
fio Disk Speed Tests (Mixed R/W 50/50):
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 6.01 MB/s     (1.5k) | 96.17 MB/s    (1.5k)
Write      | 6.00 MB/s     (1.5k) | 96.68 MB/s    (1.5k)
Total      | 12.02 MB/s    (3.0k) | 192.85 MB/s   (3.0k)
           |                      |                     
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ---- 
Read       | 537.31 MB/s   (1.0k) | 611.49 MB/s    (597)
Write      | 565.86 MB/s   (1.1k) | 652.22 MB/s    (636)
Total      | 1.10 GB/s     (2.1k) | 1.26 GB/s     (1.2k)

If you determine poor IOPS is the cause of the database timeout and you are self-hosting the gateway, please upgrade the storage device with a better performing storage device.

If your gateway is hosted, contact your host and see if they can increase the IOPS of the device. If they cannot, you may be forced to find a host that can provide a better performing server.

Thanks for running a node!
Keith