This incident has been resolved. As previously reported, xrplcluster.com has recently been expanded (capacity) significantly thanks to generous donations (https://opencollective.com/xrpl-adopt-a-node) of the XRP Ledger community & by significant hardware investments by the XRP Ledger Foundation and XRPL Labs.
The new server capacity is still fully running from XRPL Labs HQ in Amersfoort, The Netherlands. The servers have been delivered and installed there, and are currently operational, awaiting shipments to multiple datacenter locations around the world. The future server locations will be redundant and at multiple datacenters around the world, so users can be served with low latency.
However, with a large part of xrplcluster.com capacity operated from the same location, a power cut in the zipcode where the XRPL Labs office resides last night impacted xrplcluster.com performance. The XPRL Labs office still had power, but somewhere upstream the fiber infrastructure of our provider was affected, causing the internet connection to the servers to be down. This caused over half of the cluster capacity to be down, resulting in significantly higher load on the remaining servers.
With power restored and nodes back in sync with the XRP Ledger, all problems are resolved and the situation has stabilized. We will now ship the servers to their definitive locations around the world as soon as possible.
Posted Jan 07, 2022 - 11:30 CET
Identified
A large part of the public nodes behind xrplcluster.com are unreachable due to a fiber outage. This would not have been a problem during normal production, but as these are the newly purchased nodes, they are not yet running at redundant datacenter locations.
Engineers are working on resolving this issue ASAP.
Posted Jan 07, 2022 - 09:29 CET
This incident affected: XRP Ledger - Public nodes (xrplcluster.com (XRPL Mainnet)).