Disabling low performance nodes

Last week we released a planned update to increase network security which had a secondary effect of significantly increasing the performance requirements of the network. Those nodes which met specs have had no trouble with the increased load, but those which did not have found themselves unable to keep up with the network.

Today, we disabled those nodes which were not performing well enough to keep up with the network. As of now, there are 167 enabled nodes and 39 disabled nodes.

The process for disabling nodes started with analysis of performance changes. We compared pre-update precomputation performance to post-update precomputation performance as well as the post update failure rate. Many nodes were completely unable to perform following the update, and some had extremely poor performance.

This graph shows the change in average precomputation time since the last update. Nodes with 60s precomps mean they failed and hit the timeout. As you can see, the graph is linear for the majority, which shows the expected outcome, that GPU nodes are able to take the extra workload with little issue. The spike and plateau at the end are nodes without GPUs but with large beefy CPUs, generally 24+ physical cores (48+ threads). These nodes have been allowed to remain, although they are expected to be unable to handle the loads in the future.

If we look at the failure rate graph, it shows a very similar relationship.

As can be seen with failure rates, the graph is very similar. As can be further seen on the following scatter graph:

The clean line on the scatter plot shows an almost perfect relationship between the two. We combined these two metrics to produce a list of nodes which are not performing, initially finding 44 nodes were problematic.

We then reviewed each node individually and we removed 5 nodes from the list who were listed erroneously due to downtime or other issues.

These nodes have been emailed and told that due to their failure rates they are unlikely to receive compensation for the month, and have been given an opportunity to correct their configurations. They may be removed from the network permanently if they are unable to improve their node’s performance or do not respond to us in a timely manner.

7 Likes

does this mean that more applications will be considered in the near future?

Can you clarify your question? I don’t understand.

i meant more new nodes would be needed

Yes, new applicants are welcome. The application process is ongoing.

I apologize for my impudence, but can I find out when my application appears on the forum, plans have changed and I would like to supplement it a little. It was filled about 2020-11-05

Please email [email protected] with your application ID.

1 Like

ok i will try to find it =)