For about one hour on the night of 11 November, creating sign requests from apps and websites was slow or failed, and many xApps did not open. Your accounts and funds were never at risk, not for a second. Xaman is self-custodial: your keys stay on your phone, our servers never have them, and nothing moves without your own approval in Xaman. The worst this outage could do was make you wait.

Xaman is the largest wallet on the XRP Ledger. That means that at any given time, someone, somewhere, is always scanning and poking at our services. That is normal, it does not scare us, and dealing with it is part of the job. We have been doing this for many years, and our setup is built for it.

One of these probing actors had been scanning more and more over the last days, acting like a regular user. They found one read only call to our services that made our database do a lot of extra work: a lookup that normally takes a few milliseconds took minutes. Our servers have a (high but) limited number of connections to the database. Each of these slow lookups held on to one of them, and they kept coming. At some point all connections were busy, and normal requests had to wait in line.

The database itself was fine and pretty much idle the whole time. The problem was the line in front of it.

Timeline (UTC)

• 22:30: the first xApps open slowly or not at all
• 00:45: most sign requests from apps and websites, and most xApps, get stuck
• 00:51: first monitoring trigger
• 01:40: internally escalated by team, decided to prioritise health recovery over digging for cause
• 01:46: we reload the services, and everything is back to normal
• 02:10: cause found and fixed

Some requests that were waiting in line were still processed later. That is why a few people got a sign request late or twice. You can safely reject those.

Our monitoring checks look at many of our services, but a lot of that is cached or does not need the active specific database pools who were saturated right now. So everything looked green while the line kept growing. That is fixed now: see below.

Fixes

• The path itself. The lookup is now fast and limited, so it cannot slow the database down any more.
• Servers give up on stuck lookups. If a lookup hangs, our servers stop waiting after a few seconds, replace the connection and carry on. They now fix themselves instead of waiting for a restart.
• More room, separate lanes. More database connections per server, and our background jobs now have their own lane, so they can never crowd out your requests.
• Timeouts on outside services. Calls to outside services, such as price data, now have a time limit too.
• Better monitoring. We now check the database connections of every single server process, plus how long it takes for a new sign request to be stored. If something looks off, we know within seconds. A deploy that fails these checks rolls back automatically.

We don’t waste a good (mini) crisis: with every incident we make Xaman a bit better, and this one did too. Sorry for the trouble, and thanks for your patience.