All nodes have finished their restart cycle and synced up with the network again. All xrplcluster.com services have been restored. We will expand cluster capacity the upcoming days and keep spare nodes in hot-standby, so we can hopefully replace failed nodes instantly.
We will work with developers to see if XRP Ledger node (`rippled`) source code can be updated to allow faster node recovery after a restart.
Posted Dec 02, 2021 - 20:48 CET
Monitoring
XRP Ledger nodes still have not made it back into sync with the XRP Ledger. There's nothing we can do but wait.
Posted Dec 02, 2021 - 15:55 CET
Identified
Due to existing instability on the XRP Ledger, it takes a long time for the nodes that restarted themselves to get back in sync with the network (XRP Ledger). We will report back when they are back in sync with the XRP Ledger.
Posted Dec 02, 2021 - 12:11 CET
Investigating
All 10 XRP Ledger Full History nodes (maintained by different operators at different geographical locations) restarted themselves.
We will now have to wait for the Full History nodes to start, sync up with the network and start replying to requests again. This will take approximately 20 minutes. In the meantime, your transaction history may not show.
The cause of all Full History nodes to restart themselves (on different hardware, different server versions) is to be investigated: this is most likely caused by a bug in 'rippled', the source code powering the XRP Ledger (all nodes). This bug could be triggered by a specific transaction (unintentionally crafted in a way that is) causing a bug to be triggered.
We have informed RippleX technical staff over a month ago about these issues, and they the issues are being investigaged.
Posted Dec 02, 2021 - 11:15 CET
This incident affected: XRP Ledger - Public nodes (xrplcluster.com (XRPL Mainnet)).