English
Chiliz Chain
Block Explorer
Community
Socios

Details on the May 21st outage

Summary The initial Dragon8 hard fork, planned on May 20th 2024, ran into an issue. We were able to roll it back and recover any missing block.

The issue is now fixed, and the new ETA for the Dragon8 hard fork is on June 17th 2024.

Context

As planned, we started applying hard forks on Chiliz Chain the week of Monday 20th of May. Unfortunately, one of the hard forks that were planned on Mainnet failed.

In more details:

  • On Monday 20th of May, at 7 PM UTC, the Chiliz tech team successfully applied 10 hard forks.

  • However on Tuesday 21st of May, at 2 PM UTC, one of the 3 planned hard forks failed. That failed hard fork, Dragon8, is needed to introduce our new tokenomics model.

Consequently, Chiliz Chain stopped at around 2.02 PM UTC (37 blocks after the application of the hard fork).

Mitigation plan

To remediate the situation, we put in place the following mitigation plan:

  1. Create a new Chiliz Chain Client version, reverting Tuesday’s Dragon8 hard fork (v2.0.8).

  2. Roll Back the 37 blocks generated after Tuesday’s hard fork that occurred on block #13226483.

  3. Restart the Chain - with Validators updated to version 2.0.8 and ensure it runs as expected.

  4. Update RPCs to version 2.0.8.

  5. Update any other nodes to 2.0.8. If you run a Chiliz Chain Mainnet node, make sure to update to 2.0.8, otherwise your node will not sync.

What went wrong

The formula for inflation we were using originally was coded into the client using floating numbers.

What happened was that due to the fact that each validator has somewhat unique node architecture, the floating numbers were causing non-deterministic results, which caused the the nodes to stop synchronising with each other.

The chain then stopped as consensus could no longer be achieved.

The Solution

The solution was to remove floating numbers and work only with integers in the code. We also decided to simplify the formula while still retaining the same outcome.

In the end we now have a determined inflation rate decaying every year instead of a slowly decaying inflation every block but the inflation percentage is the same.

Timeline

Here is a timeline of events :

  • 2024-05-21 13:59:57 UTC Block #13226482: Chiliz Chain is not yet on the Dragon8 hard fork.

  • 2024-05-21 14:00:00 UTC Block #13226483: The Dragon8 hard fork kicks in.

  • 2024-05-21 14:02:00 UTC Block #13226529: Chiliz Chain stops after 37 blocks mined, following the Dragon8 hard fork.

  • 2024-05-21 20:17:00 UTC We release version 2.0.8 of the Chiliz Chain client. It's aim is to rollback Dragon8 only (including the 37 blocks). All other forks successfully applied on Monday 20th and today 21st are documented here.

  • 2024-05-22 08:10:00 AM UTC At this point, enough validators upgraded to the latest version v2.0.8 to start getting consensus on the blocks being created, and blocks began to seal. From this point on the chain was in a stable and recovered state. Transactions that were in the 37 blocks that we rolled back, were re-submitted and included automatically when the chain recovered. In short, all 37 blocks were properly reprocessed in the end.

The new ETA for the Dragon8 new hard fork is: Monday 17th June at 8:00 AM UTC.

Impacted transactions

Here are the 37 blocks that were lost during the outage.

Last updated