Helium network team resolves consensus error after 4-hour outage

The Internet of Things (IoT) blockchain Helium shut down for about 4 hours on July 11 due to validator outages from a software update, causing delayed transaction finality.

During the outage, devices transferring data over the network were not affected, but miner rewards and token transfers were left pending. The team resolved the issue by skipping the blockchain forward by one block and resuming normal functions.

At 10:20 am EDT, the Consensus Group stopped producing blocks at block height 1435692 on the Helium (HNT) blockchain, according to a status update. Lacking network consensus, token transfers could not be completed, and new blocks were not being produced.

We're back folks. The chain just processed block 1,435,693. We'll continue to monitor the chain throughout the day.

— Helium (@helium) July 11, 2022

Helium is an IoT network that uses physical radio hotspots to allow users to connect to their devices from anywhere there is radio coverage. On the Helium network, a Consensus Group consists of 43 validator nodes randomly chosen in fixed intervals to provide network consensus.

In the postmortem for the event, Helium engineers cited two reasons validators stopped creating consensus on the network.

First, an issue with a July 8 software update for validators contributed to the problem. The v1.12.3 update was designed to provide support for the 5G Mobile subnetwork and its MOBILE token.

Additionally, a “local network outage” was also to blame. In the project’s Discord channel, Helium moderator “Digerati” explained that a high concentration of validators randomly chosen as the Consensus Group at the time of the outage was running on the same Amazon Web Service (AWS) network, which experienced technical difficulties.

AWS is a global cloud computing and data storage service that can be used to enhance computer networks like Helium.

Compounding the problem was the failure of an auto-skip feature that should have chosen a new Consensus Group automatically. The team stated that “a known issue prevented the auto-skip feature from working as expected.” However, it is not clear what the “known issue” is that the team referred to.

Although the network could not choose a different Consensus Group, it was designed with the ability to skip the blockchain a block forward “to mitigate situations such as these,” according to a 10:56 am ET announcement from the team.

Related: Cloudflare outage affects multiple crypto exchanges

By 1:45 pm ET, new block production started with block number 1435693. The team said it worked closely with validator operators to ensure they were synced correctly and released a new software update.

HNT is down 4.1% over the past 24 hours, trading at $8.76 according to CoinGecko. It is down 84% from its November 2021 all-time high.