Optimizing the BetaNet

As of the writing of this post, the mean 24 hour real time latency of the network is 3.44 seconds with a teamsize of 3. Per our measurements, the average message sending latency is 5 seconds once message delivery is included.

Overall, we believe the roughly 1 and a half seconds message delivery latency can be brought down by roughly 0.5 to 0.75 seconds fairly rapidly with continued engineering, but do not see a path though software improvements to improve the realtime latency further.

The following graph compares real time latency to how long it takes, on average, other nodes to receive the streamed data during the realtime permutation (A time which should be highly correlated with upload bandwidth but not correlated with latency).

As can be seen from this data, the correlation is extremely strong. Interestingly, individual lines can be made out for regions. We believe this primarily correlates with different latencies.

Given how cMix is a co-operative algorithm, the values here are not unique to a node, but a property of their relationship to the entire network. As a result, a single slow node will in fact impact the realtime latency for every other node in the network. As a result, we need to begin a process of discovery to determine how pervasive performance issues are.

To begin with, one’s upload speed is primarily slowed by another’s download speed. The equivalent download speed graph is as follows:

By comparing these graphs, it can be seen that download speed is in general not nearly as correlated with performance as upload is. We believe this is because far fewer nodes are download limited. But it also points us to the best culprit for improving performance - dealing with nodes with limited upload.

Through a rough (and potentially inaccurate) analysis of the upload data, it looks like the average node is supplying 19 megabits of upload, far shy of the 100 megabits. Starting today, we will begin a series of tests where we disable nodes based upon different criteria to determine how much poor performing nodes impact each other.

If you are disabled, DO NOT CHANGE ANYTHING. Your node, whether enabled or disabled, is helping the development process by allowing us to analyze the impact that ALL nodes have on the system.

Starting today, we would like to take a network wide baseline and disable a large portion of nodes to see the network performance in order to determine an overall baseline for maximum performance. The goal will be to compare the results to the disabling of other subsets - allowing us to determine which faces are more influential than others.

As a result, We are disabling roughly 15% of the network for the next 24 hours. These are nodes which, over the last 24 hours, have a realtime average time above 5.5 seconds or a realtime permute transmission average in excess of 850 milliseconds.

The disabled nodes are:

+mTYsByq2UDKEERbMGM9bGvVTxSDqnD0rlbNdfCuvz4C
0bRQkL1dK6g8ZDnvfOJhW8kXjQsJ9jHZthqF2/qZCm8C
EpuR+EffEwvrtwFPatBo+Ae6jjD2/X9f06WEFO8YVBUC
8zzTqAG8+VMR6TVi/u2oStH4BZyVu9PIFyivwwuSNaAC
eQTt1q/QSGkgOJOJ4vBeIWEBeS55A4bd4w5AhjIpQFYC
IqiZ4ca/e8yMKInFzNT5ta0XplEr92JVBXN9dzEBTmgC
0PbYARB7pRq2tAJwEN0Q1lnC4ip3ZThYjuwRrhR6cnwC
1VMmgjJMDXDcPb2a2tkgi/YUJJATlgADGWUCq5+fBEUC
2qeUKWWQSv8cNB8XT4RW+4OROVlz6vQSAi0S/5CELWgC
3110HQ7Jl/CQjK1JDLVv9J+hwh+Rrva92AYYcYY9k6UC
3miUvizbq8dJr7eFIJGj8Y03QL8SczF5e2J+q/bJ6TMC
3vaKmaJnEC9cES32aleF7am5/M2rguGT0NSfnJU09BYC
3WiO+pruuFTijVUj8S/ckAWxwatRDQIYYhLJZ7YsdBIC
5PL1LZvlkqOyVFv1Ly4qSSGjoBV545s1Ucf/Z0cxa+IC
5StSHICOHLMNZrPTJuQrgfI8Ha2wFYAzaWXltY2ZvPcC
8/hKY3NZXI3bCvqWgvk8O2HFbmkKxAsyqqPIWgGlBMoC
97LbEHPuoyj1bZ4hY9po6KNDMFhiTuLbdgsi514w81wC
Aa9RGTn2SZrSfH4cevidnYBHVhouNjq79pvpmZOywb0C
AIQqFxnb4i8438QaL9JgeIHQbBNyyCJhj3KqNS9mZ+8C
ani05Iqa3i+XDzPjMKtuT1skvn261drIpNUX1KqtZS0C
aUrBY+vOWRiy+1dlIF4qe1mXd2Pm7KxpwVfKhrYOOucC
d9flItpkadOst86zEzb8NSuLe0jMWp4ZmbqgAmKDTVoC
dhhq0ItlFsQ5wZaBt2/QN1M86hMBcI8qeDYDbPjnXqEC
dlhMfTenHXUyRWfGPgznwojezJcRmAcIipSxnTHGmk0C
dpy+8RlemtS/4h4ebl65kOUyoWQsN+N0qvx23loIgqEC
dxKoYF6eC/EITryAdA9X9KK+wR5ixJMeSUHpnGoZY+kC
Eft7eN16/R55yg+sR0CX3Oz/WvAikC6iwA8Gi6KnfHYC
f95Z/cbhHvwv+SXpPoFCZ0+ui3sQoP30vMnHV6YpEwkC
fr0EwplwSgpEbFKW5dt0Cqw9G01VfKOKxrDzdaXPreQC
FSwdjmiWaZeVCEX+D8F8eWKNoo0pCZYJHV8JfXuVHdgC
FYRd1nLZBR465LYvG4oqbM3sLFJ8DH0w7AhgoMvTo3sC
hDeL1NW1mjaoWlNYNLktIqlzdSl+YsIBoVP0aBVF/QEC
hpCH1SRpWmPzv58JQETksQI+q3rd6LzsdDxehZLDU70C
hPIW+MTsYJMhD1DL0WNuH590Plw8p7j0jnFlpWqNPmgC
hQJBvh0d15YzzCJmniAtPVCZ5P/6bBmKteUz/6+pi3wC
hWpKnjfZ2z5iMYA7JKrtwLGoD9GbPvR/88M6o8COyhcC
Jd1UAUyTBLlEyGZLQ85nTmdgB/6udEkaijVO2VhjKYMC
JZ5LdYHi7yhKgtuWFaj4mQfTycQMoqa0bGmtEalDt7kC
McWd4Z22GtL56B0Cc/csbv7NDpm4mhR2KLb8p/3Ujo4C
MgbMvS0A3TtQiQDjFnezvfL2nZbTs9PyWoEv9FkPmaQC
Mm8mNYTQvu/4CezrzYHTFv+PHW5aIWKpoK6kzy+yd94C
N96uWg07cE9Jnhc2/yxYSH1FvFSUhCqYzrTlSmFsQ8IC
nR+7PFIEycXifnlUAef6ZH3Jxfe1OqkCy2/VIhMgJdAC
o6OCo4SZmiDtA7IaYuxjUYkPfteX7BeMZk8JVn4pjJkC
OaG8nOjd5fVN7SzOk/szW83FHEJHVxaBYmRBSRdvZcYC
ocYuwivXkw6qq73oV3FblUi7MVrKHif4yhNhbkE0yMYC
pAQVHK4vw43Zwe0txQVe2/899waNxToJntREoZycIHcC
Pts3X7zimgIH9IlnNHEZYWFsLyf4jZsrdMKJGbL3WnUC
pUzl01rs6h5rdvfYabW9NHEObxk79ohg3AmfJ6jEsQ0C
qEiXYfGPQ9/5xadVQG/xIWZ7dICilVeKjCakm/lqQmwC
qgyw4JpuBsu2ZJc7H4rO3GT3dA5n0vezcgUGZITpyhsC
QJ2wzWPX1i5pi6ma2RtoHQLIZ1dWFzK8ou6m4ibIkcMC
QkgkiCjuRxoeg4SBsruDgTQxWmJc+kmAjdSjR19Ubm0C
Qs/iQniQ9mvlNg4OS5E32wKd/qOLg468nuF1BIhzVxIC
R2ZdiPeAToYyAQUGxGsL7X2aNH4I+kJ+/pHAwvNpUYIC
RSDyCR4abj386Cl2mN17AIO353sv4l6P//qdCbpzwkgC
RyPIFyn/jflA/8YQWf2BaQHPH5WnvcYyam6QYvWRUiYC
SA/C8y1ymb5uKmItxsvFUSUl9+WpqmNoM4jEVxqTar8C
sYz3qh7EzwXMoCXlEdBOsFXanYD/AAZS0CZgwZ+2Im0C
tMxKfRRr/kHUaQCm8b1rwScPJfc1bFqw5v+Gaa2EusAC
Tn3Shjhv1k/vNg9l5XGEsFcknGxzxY6hwR2JxPDxs0EC
UDvzzkO7nCpe/hjh9LkCmv8eN887xwVbpG15DmatO64C
wUrbq8aEZCDUNeQ9kmWHOD2IBvFdcw9jVswsyyYnJ8YC
xo7eywMmbdTJHv/cDu71bisCu9yELB+qaD8UFwTEjhEC
xQOllAPBlE9VDKHLDms724LWHPHN5qylHZBLw+pspVEC
xyepIg1Lg1LzG6l3I9BMkjzqmD6f+zvHGzyL8SamKvUC
yE9WSjaE7y4iQFxnT4u4b3MlOKamvv244TM1VCYypAoC
yIVYjVgQJyZuaGFRvQQZhsU9J6OuGt+zDB62YUGu0gEC
z3o28UKEgZjCGg4F75zYSfMtfjEWZoGh5XA640mmesAC
Zj9Q8ASv3/7Zyc/Jjp5j82jVlbn4laTQpHtR4291tcUC
zpNDnPdO/Ck6T/1eqAJ1SBz5FrVWG2yHb3ibMkFEDiEC

3 Likes

While a node is disabled, it is expected that the dashboard show the status of ERROR.

Additionally, if you tail the node.log with …
$ tail -f /opt/xxnetwork/node-logs/node.log

you will see the following message …
Poll of permissioning failed, will try again in 1s: The NDF was not returned, 'permissioning is likely in the process of vetting the node

This is the intended behavior. You can leave the node in this state and you will continue to receive UPTIME towards your monthly statistics. Your other statistics will not be impacted in a negative way. While the node is disabled, it is not scheduled to participate in rounds, so it does not fail rounds.

You can’t restart it away. Just consider the node paused.

Thanks for running a node!
Keith

1 Like

How can I find out the “realtime permute transmission average” of my node?

2 Likes

Suggestion, maybe it is worth including in the analysis the comparison of node and gateway sitting on the same machine vs node and gateway sitting on separate machines. Also in the case of node and gateway on separate machines, the correlated data(e.g. upload) refers to the gateway, the node or both? Thank you.
PS it will be really useful to know upfront if the node operator need to “deal with ISPs” as it can take several weeks to make adjustments.

1 Like

By disabling nodes which have a realtime average time above 5.5 seconds or a realtime permute transmission average in excess of 850 milliseconds the mean 24 hour realtime latency of the network is down to 2.76 second from 3.44 seconds

For the next 24 hours, the following nodes will be disabled. These are only the nodes that have a realtime average time above 5.5 seconds.

+mTYsByq2UDKEERbMGM9bGvVTxSDqnD0rlbNdfCuvz4C
0bRQkL1dK6g8ZDnvfOJhW8kXjQsJ9jHZthqF2/qZCm8C
3WiO+pruuFTijVUj8S/ckAWxwatRDQIYYhLJZ7YsdBIC
8/hKY3NZXI3bCvqWgvk8O2HFbmkKxAsyqqPIWgGlBMoC
8zzTqAG8+VMR6TVi/u2oStH4BZyVu9PIFyivwwuSNaAC
97LbEHPuoyj1bZ4hY9po6KNDMFhiTuLbdgsi514w81wC
ani05Iqa3i+XDzPjMKtuT1skvn261drIpNUX1KqtZS0C
d9flItpkadOst86zEzb8NSuLe0jMWp4ZmbqgAmKDTVoC
dhhq0ItlFsQ5wZaBt2/QN1M86hMBcI8qeDYDbPjnXqEC
dxKoYF6eC/EITryAdA9X9KK+wR5ixJMeSUHpnGoZY+kC
EpuR+EffEwvrtwFPatBo+Ae6jjD2/X9f06WEFO8YVBUC
fr0EwplwSgpEbFKW5dt0Cqw9G01VfKOKxrDzdaXPreQC
FSwdjmiWaZeVCEX+D8F8eWKNoo0pCZYJHV8JfXuVHdgC
FYRd1nLZBR465LYvG4oqbM3sLFJ8DH0w7AhgoMvTo3sC
hpCH1SRpWmPzv58JQETksQI+q3rd6LzsdDxehZLDU70C
IqiZ4ca/e8yMKInFzNT5ta0XplEr92JVBXN9dzEBTmgC
Mm8mNYTQvu/4CezrzYHTFv+PHW5aIWKpoK6kzy+yd94C
N96uWg07cE9Jnhc2/yxYSH1FvFSUhCqYzrTlSmFsQ8IC
nR+7PFIEycXifnlUAef6ZH3Jxfe1OqkCy2/VIhMgJdAC
o6OCo4SZmiDtA7IaYuxjUYkPfteX7BeMZk8JVn4pjJkC
OaG8nOjd5fVN7SzOk/szW83FHEJHVxaBYmRBSRdvZcYC
qEiXYfGPQ9/5xadVQG/xIWZ7dICilVeKjCakm/lqQmwC
RSDyCR4abj386Cl2mN17AIO353sv4l6P//qdCbpzwkgC
SA/C8y1ymb5uKmItxsvFUSUl9+WpqmNoM4jEVxqTar8C
tMxKfRRr/kHUaQCm8b1rwScPJfc1bFqw5v+Gaa2EusAC
Tn3Shjhv1k/vNg9l5XGEsFcknGxzxY6hwR2JxPDxs0EC
vCxVkFPCb67QWkgB1ITfbz2gjSIZiYY+UiCXiVWKvIYC
wUrbq8aEZCDUNeQ9kmWHOD2IBvFdcw9jVswsyyYnJ8YC
Zj9Q8ASv3/7Zyc/Jjp5j82jVlbn4laTQpHtR4291tcUC
zpNDnPdO/Ck6T/1eqAJ1SBz5FrVWG2yHb3ibMkFEDiEC