Details on the May 21st outage

Summary The initial Dragon8 hard fork, planned on May 20th 2024, ran into an issue. We were able to roll it back and recover any missing block.

The issue is now fixed, and the new ETA for the Dragon8 hard fork is on June 17th 2024.

Context

As planned, we started applying hard forks on Chiliz Chain the week of Monday 20th of May. Unfortunately, one of the hard forks that were planned on Mainnet failed.

In more details:

• On Monday 20th of May, at 7 PM UTC, the Chiliz tech team successfully applied 10 hard forks.

• However on Tuesday 21st of May, at 2 PM UTC, one of the 3 planned hard forks failed. That failed hard fork, Dragon8, is needed to introduce our new tokenomics model.

Consequently, Chiliz Chain stopped at around 2.02 PM UTC (37 blocks after the application of the hard fork).

Mitigation plan

To remediate the situation, we put in place the following mitigation plan:

1. Create a new Chiliz Chain Client version, reverting Tuesday’s Dragon8 hard fork (v2.0.8).

2. Roll Back the 37 blocks generated after Tuesday’s hard fork that occurred on block #13226483.

3. Restart the Chain - with Validators updated to version 2.0.8 and ensure it runs as expected.

4. Update RPCs to version 2.0.8.

5. Update any other nodes to 2.0.8. If you run a Chiliz Chain Mainnet node, make sure to update to 2.0.8, otherwise your node will not sync.

What went wrong

The formula for inflation we were using originally was coded into the client using floating numbers.

What happened was that due to the fact that each validator has somewhat unique node architecture, the floating numbers were causing non-deterministic results, which caused the the nodes to stop synchronising with each other.

The chain then stopped as consensus could no longer be achieved.

The Solution

The solution was to remove floating numbers and work only with integers in the code. We also decided to simplify the formula while still retaining the same outcome.

In the end we now have a determined inflation rate decaying every year instead of a slowly decaying inflation every block but the inflation percentage is the same.

Timeline

Here is a timeline of events :

• 2024-05-21 13:59:57 UTC Block #13226482: Chiliz Chain is not yet on the Dragon8 hard fork.

• 2024-05-21 14:00:00 UTC Block #13226483: The Dragon8 hard fork kicks in.

• 2024-05-21 14:02:00 UTC Block #13226529: Chiliz Chain stops after 37 blocks mined, following the Dragon8 hard fork.

• 2024-05-21 20:17:00 UTC We release version 2.0.8 of the Chiliz Chain client. It's aim is to rollback Dragon8 only (including the 37 blocks). All other forks successfully applied on Monday 20th and today 21st are documented here.

• 2024-05-22 08:10:00 AM UTC At this point, enough validators upgraded to the latest version v2.0.8 to start getting consensus on the blocks being created, and blocks began to seal. From this point on the chain was in a stable and recovered state. Transactions that were in the 37 blocks that we rolled back, were re-submitted and included automatically when the chain recovered. In short, all 37 blocks were properly reprocessed in the end.

The new ETA for the Dragon8 new hard fork is: Monday 17th June at 8:00 AM UTC.

Impacted transactions

Here are the 37 blocks that were lost during the outage.

Tx Hash	Block number	Block hash	Tx Type	Source	Status
0xb7e54fdddbc6ffff5a5a23ec83181bf10438853b9107ed1c41129a8e593cf0b0	13226484	0x65289089b94545c274f247e0af14474c532c5de724e017ac435496546a40653a	transfer	Unclear	GOOD
0xc7beda33eb07011c6ddfbf1b0a715ea0788195d7f104cdc660a474e9ada72dcb	13226484	0x65289089b94545c274f247e0af14474c532c5de724e017ac435496546a40653a	transfer	Unclear	GOOD
0x0d94a58a178bf0c7d4a3074a763e4010f1f2f7cc43b8e33f1ca4ee30b2995434	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0xbd36012ee38fb00cb342d1360304c4e17b7efe16a8ae3f6e7fef01faa25c44c5	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0xfef3dd7af0efad9d33f24d3e7a10cb5bad2d4c22494981cf25aae3707b83913d	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0x18723a9d1c76f083608536412ae13c8b8b051a4d206dd798aad376c410b0aa71	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0x3715189b87f3521bbd61d5c568babc3065d0974a0f38cc19121ac90d683492bd	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0x24b56e88fcff686f7bc3d587783b8eddbad8a03eed7bec2d97a46c01a89df9cf	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0x68ed2e862ba0331e56140b24099ac91d6aec0b74e42b8c4bdb2f58bbb8334cee	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0xa475b3035b080139f7f92f815593353dd33b11ca55341341523ffb04f4e77fa0	13226487	0xd797206a31ff5bbbefb1411f87982ed034e3732a333fdfd2518b9388b49832e1	ContractCall	Socios.com	GOOD
0xe8e9a3b4cd03f0e993e8650501e00235183c1e5403d66ce71069ad7ccb4eaf88	13226488	0x5752a0d30661cc1ea6ecdb788e42314426c6726c2644b090a634643ce789cad4	ContractCall	Socios.com	GOOD
0x8a491bf1a19a1662e9e82766f50537bc7b540afd15d9158d124c3c5a7f4bd430	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0xfbce9dc84fcbe008d00f3d8fd5dbc3afd3ac497c75af23c28fa899a03f2029a9	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0xa43cb56785ee620b95fcfd3d7e3451cc2ef59c46c3af98936edaa7553afdcdf7	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0x5007c393dcd59ae8b1000d89d0ccc6d4a271eda565e6f26c6f4cdb12810a1acb	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0x873ef58a1fa2861f9cf981b989b2030dd057eafed98c2fd3d9ed1f5ba4050f71	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0x5a218a7384e1eea3d97c6eafb889cb99245150ec6165ae7cacef7974a110aeed	13226493	0x0c5dab18fc323f171af8c72fdb0e8c99193cf95e37578074d23c46682e547769	ContractCall	Socios.com	GOOD
0x5bcd74743e999befa6bf8d43119255d33cda3e450f78453f28f718bcb241f179	13226494	0xcf6bf63299908c05624c1900cf19ed45819fa1212e773187f7991fba65c1cc9a	transfer	Socios.com	GOOD
0xe77269af259f4bb6d402b7cff221afab4ab01f2a020d581069a14feaf6dad157	13226494	0xcf6bf63299908c05624c1900cf19ed45819fa1212e773187f7991fba65c1cc9a	ContractCall	Socios.com	GOOD
0x1e2a6516ad05542342b4dafbb766b2e5f58834af6597430539bbe768b18ca85d	13226494	0xcf6bf63299908c05624c1900cf19ed45819fa1212e773187f7991fba65c1cc9a	approve	Socios.com	GOOD
0x2bfe7cd55d334fc58f09af4f5bd7c8f74a562942ed1ca8a1f78882314a298333	13226494	0xcf6bf63299908c05624c1900cf19ed45819fa1212e773187f7991fba65c1cc9a	approve	Socios.com	GOOD
0x946f06da557265619783a01214939151c93649ff3751310c3ec0b53891661211	13226496	0x4cda4d21482dea6dfd6ba7caac04f860d8ee91eea60d18394003c6e9632cc68f	ContractCall	Socios.com	GOOD
0x7f0e8ac9564baf4481c0e3fc6b8ae9b564d9144058a8e3a8b7b894ee74df5547	13226496	0x4cda4d21482dea6dfd6ba7caac04f860d8ee91eea60d18394003c6e9632cc68f	ContractCall	Socios.com	GOOD
0x5d085189907c9fda2d4829714751c3fd7a7effa879a69bc431301531464dce84	13226497	0x37229d1218ea21987dc4c33ff628b65d90ce6828b5b546f3604f848a36d072d4	ContractCall	Socios.com	GOOD
0xd2c0bc6cd5a0cba082f00f22cf8806ce1b0954c1562f9297d350787ab98634fd	13226497	0x37229d1218ea21987dc4c33ff628b65d90ce6828b5b546f3604f848a36d072d4	ContractCall	Socios.com	GOOD
0xfeaa0e1fd91c05acb82b5ee07c3001841c23a180dcf99414f7e84e92cdceba8f	13226497	0x37229d1218ea21987dc4c33ff628b65d90ce6828b5b546f3604f848a36d072d4	ContractCall	Socios.com	GOOD
0xd068775ab2c6b508d40bec2cdbdae5c2aee52bf8f2374d82e6829499d3f23e87	13226497	0x37229d1218ea21987dc4c33ff628b65d90ce6828b5b546f3604f848a36d072d4	ContractCall	Socios.com	GOOD
0x88952858515054a2d9199f0a1372d3b099742887ec64df2e00980dc0b4ad36db	13226500	0xb3625e10a538ce13c09528fbf1a2514cafb45360ee1c6d938efcaaa5838098e1	ContractCall	Socios.com	GOOD
0x8aa577a6c18441e778dc942e49893b2bccc2267511c0fc3196284db7ec9d000c	13226500	0xb3625e10a538ce13c09528fbf1a2514cafb45360ee1c6d938efcaaa5838098e1	ContractCall	Socios.com	GOOD
0x640caca07c918148cee80a7f8a9dea0cf66b8b81e877067f516d8219107ca516	13226500	0xb3625e10a538ce13c09528fbf1a2514cafb45360ee1c6d938efcaaa5838098e1	ContractCall	Socios.com	GOOD
0xb33a2f6be9605ad0878b3cb8244645a593d3e3020edd0208bfdbdf68310218e6	13226500	0xb3625e10a538ce13c09528fbf1a2514cafb45360ee1c6d938efcaaa5838098e1	ContractCall	Socios.com	GOOD
0xc23434878c1f5603bd47c7eb9288bb26155ab9da47a4c936e00200cbf2a44f40	13226500	0xb3625e10a538ce13c09528fbf1a2514cafb45360ee1c6d938efcaaa5838098e1	ContractCall	Socios.com	GOOD
0x5b16ae425081ff5c1db9a9feaa8812214375cb3f90f5d6a96ab8a876a0627e4a	13226503	0xd58e65a3eb887866ff92e8da12b3f01b1f32b2a2311933673aa53a46d4ca0d31	ContractCall	Socios.com	GOOD
0x08c918ee38503b17b141674e3d3d9cded2d315e66421af901b490fef23dec9bf	13226503	0xd58e65a3eb887866ff92e8da12b3f01b1f32b2a2311933673aa53a46d4ca0d31	ContractCall	Socios.com	GOOD
0xda29e32d41d59376b882cafb6f1bfa4efdc6f0ac65a1405f47e105de10ce199a	13226503	0xd58e65a3eb887866ff92e8da12b3f01b1f32b2a2311933673aa53a46d4ca0d31	ContractCall	Socios.com	GOOD
0xd4cf541fc86f06b17ea65903294b7654339af9477bd546e1a6f0b017fa0e2254	13226503	0xd58e65a3eb887866ff92e8da12b3f01b1f32b2a2311933673aa53a46d4ca0d31	ContractCall	Socios.com	GOOD
0x5a5a97b77f764048ba919f796a8e937493f7f8e63dba7892efff75d40f983833	13226506	0x2bbd82d4aacc94536b03ca4514c2b3b6e839429732e2e401bdf1f05a2aa5eb9f	ContractCall	Socios.com	GOOD
0x81337d3332ea7d53187e3745941fc937b5800e160787398fcad8948200f0d127	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x7d217555e2381e2bafdcc18a7b5692a0e2e8f91c895ff8a2df8fde7d6338a128	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x547aa8e8a2e793d2d37f5dd9c7f692b9a0c49186a6573c429d797aff448f1941	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	transfer	Socios.com	GOOD
0x993c1e6dd05a4cddc6a8e3043c8869ffe2f4785fc3663126b7918c6ddd715744	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x9e2da9ab37a5c628d945face1cb89305b0384928c18cdee75c923540b91a74b6	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x48b5fd9c24044ef449b370c1d53a741ad2467f1068a5465ffde5020568517628	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x313cc7b24f0384238d08fc5ed3eedc1f461d9d41fbed0fc2fd7b9fae2220c217	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	transfer	Socios.com	GOOD
0xd5ed5f2df685b72fb1573710418769f9a9a052cb31b735476a8b1c4a5af4bf81	13226510	0x1d79ff0903c454b67ebbd4e8f34a676a3a6f06ddca0490abc1bfcff722186d1b	transfer	Socios.com	GOOD
0xd65757c9b09c1325c2a8cae5ec6feef56b70da6265bcf6253100a0e495322710	13226512	0x991cde5e73d8fadbd813a69373008b3f1092fb88b900bb82a0de8a7733b877c3	transfer	Socios.com	GOOD
0xaf91bcfb41b78c9ae584ff0fd37e6ea3bcf5c7076317ea29283e2e5c3c11dd71	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x8eb57a0e34d605f674c6a6bbd1935ed4f4c39a746f30d1ccff842cb6e29d0dc5	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0xfee9ed83faf926f833c4adab54071955bc33687f27fdcefd0bf94324990d02b0	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x4e2ff87fbbe0f334def4d8c8ba2cb1d01c57e6b084bdec4ae4d15fb3d16e5cff	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0xda4af3bacd688a1cbaa77c46fed6a8a3d2de8c875965ca4383136ca61111a84b	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	transfer	Socios.com	GOOD
0x4b684bd976e0fdd05c8cf39a28d4dcbce1ee74f4aa6b37f4b77619daced8c5ac	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD
0x159aa1c6c2ac5afe7ba50b6f12dba378131bccdb09d030172d5265101cf0d720	13226548	0x45a672df4b085f6be73732392165245f99bb5d55d6f6cadbfc113f2d550b27ce	ContractCall	Socios.com	GOOD