October security update: what happened

Between October 16 and 22, Bidali was attacked. I want to tell you what happened in plain terms, what it cost us, what we changed, and why new gift card orders are paused right now.

The short version: the attackers used AI and complex chained attacks to gain access to an administrator account on one of our internal dashboards, they took nine $5 gift cards without paying for them, and they stole about 987 XLM (roughly US$180) of the network reserve we fund for new wallets. We have found no evidence that they accessed customer data or customer funds, and everything below has been patched. Our team responded within minutes, isolated systems and paused orders. I believe that is the reason this stayed small.

Ordering is paused for a second reason too, which is an industry-wide event I explain further down. I'm sorry for the abrupt stops and starts, for how long it has lasted, and for the lack of direct communication until now other than through our Bidali Support channel on X, which is often where we post first. We were swamped and focused on mitigation first. We also got very little sleep this week. I alone put in a 30-hour straight shift earlier in the week.

What happened

Probing around the edges. They began by testing our APIs for weaknesses, including a flaw that let one API read data across organizations and an earlier gap that let organization owner details be changed without authorization. We found and closed both. We built our stablecoin payments system that powers parts of our gift card infrastructure to be multi-tenant, but we are currently the only ones actively using it.

Internal dashboard access. They used AI and complex chained attacks to gain access to an administrator account on one of our internal dashboards. This is the part I most want you to understand, because it was clever and not immediately obvious. They exploited a few things in how our system worked because we had disabled the ability to reset the admin account password externally. Even then, you would have needed to have access to the email account tied to the admin account which is protected by a strong password and 2FA, or so we thought.

One of our password reset endpoints expected a single email address as a string. The standard. We had tests around this and tests excluding bad values like null, undefined, true, "true", "object", etc., as well as SQL injection guards and the usual things you'd do to sanitize inputs. The attackers tried all this stuff. We could see it in the logs. Eventually the attackers sent a list of addresses instead including our valid email, this is key.

Because we use Node.js, our APIs are written in JavaScript. In JavaScript, a list never equals a string, so the strict equality check we relied on was falsy but because it's JS it simply did not apply to it.

The request fell through to a database query that matches any address in a list (an $in query), this found the actual admin user entry even though the attacker's email was also in the query. Because it found a real user, even though the attacker's email didn't exist, this fell through to triggering an email send through our email provider. The email provider and mailing service supported arrays for multiple recipients because it is used in other parts of the system for other email types. The result was a password reset email sent to both our administrator address and the attackers. That allowed them to go through the password reset flow and get into the account.

Looking back, two-factor authentication on this dashboard would have added a second layer of protection, and I want to own that. It is an internal dashboard, it is not public-facing, and it was protected by a strong password, which is a common setup for internal tools. In a small team, 2FA stayed on our list behind other priorities, and we had discussed it a few times. This attack showed me that internal tools deserve the same hardening as public ones. We are rolling out passkeys, and I take responsibility for how we prioritized it.

The free gift cards. From there they tried to get gift cards without paying. Two more flaws let them buy nine $5 gift cards (seven Razer Gold and two Xbox, US$45 face value): in one, they exploited a bug in our processing fee logic that allowed them to make the fee negative, and put through a payment worth a fraction of a cent for a $5 gift card. Another clever one. We obviously patched that right when we saw it live.

In the other, they used our HMAC secret for our account they had obtained via our admin dashboard for our org, to trigger orders on our gift card API that were never paid for.

We didn't think they had access to this given how quickly we shut them out as it's only exposed in the UI under the expectation that our system is multi-tenant so would be visible like you would see in any infrastructure service dashboard or API key. We even hide the value in the UI so you don't see the full key. They likely scraped the browser localStorage or web request.

We use the HMAC to verify webhooks from one system to another and they are normally internal calls for our services. This exploit took advantage of both: gaining access to the key in the first place, and a misconfiguration in our proxy where we were not rejecting external calls to our particular organization missing the corresponding client ID, because we never expected those calls to come from an external source with a valid HMAC, let alone someone outside the org to even have access to the HMAC secret.

One way to mitigate this is to only have the HMAC be visible one time so if you need to view or rotate it you are provisioning a new one. Something we'll be rolling out even though we're the only active tenant.

Stolen XLM. When someone creates a Bidali wallet, we put a small amount of Stellar network reserve into it so the account can exist on the network and have, in Stellar's nomenclature, "trust lines". Starting October 17, the attackers signed up in bulk, collected that reserve from each new account, and moved it out within minutes. Over the week they created hundreds of accounts, and they kept coming back, mostly in the early morning hours. After each wave we moved more of the XLM in the funding wallet into cold storage, until there was almost nothing left to take.

They also used some of the stolen XLM to buy a $20 ChatGPT gift card: they swapped it into USDC on Stellar and paid for it at checkout like any customer.

We've had very high test coverage in our codebases since inception, in many cases 100%. We also have a bunch of experience building web systems and with Node.js as well as in crypto and with private keys and key security. Clearly this doesn't guarantee that you are safe, and I hope that's a lesson people reading this take here.

Additionally, in the new AI era it's clear to me that you NEED to have defensive AI in your workflow and active monitoring and circuit breakers. If you don't you will not be able to keep up with attacks that use or come from AI. They're just too fast. Some quick thinking, fortuitous timing, and AI assistance really saved us here. We know we're near the frontier compared to most companies as we fully embraced AI internally years ago. I'm concerned for other companies so I hope this is a lesson and guide for others out there.

I also hope you can see that none of these hacks, without being able to see clearly the code, are trivial. And while there's always a feeling of shame or guilt when you get hacked, and of wondering what you could have done differently or why you didn't prioritize something higher, which I and the team definitely feel now, we have learned a ton through this experience. We also rapidly levelled up our skills through the assistance of AI, even as a team with decades of experience.

So what's the damage?

Overall, we feel pretty lucky and we've come out largely unscathed here, other than way more grey hairs and not enough sleep.

Cost: about 987 XLM of our network reserve stolen (roughly US$180), which includes the funds used to buy the $20 ChatGPT card, plus nine $5 gift cards. This was Bidali's own money and inventory.
Service disruption: some of our fixes briefly disrupted service on October 20, including about 45 minutes when gift card checkout was unavailable. These are resolved.
Not affected: we have found no evidence that customer funds were affected or that customer data was accessed. Customer balances live on a cryptographically secure distributed network (Stellar), completely separate from the systems that were attacked. No Bidali wallet private keys were exposed. No customer keys were exposed either: we cannot access them ourselves, because they are encrypted by both the customer and us using a multi-party computation (MPC) method, with our keys isolated and encrypted. Our cloud and cluster audit logs for the period show only our own engineers and our deployment automation. Existing gift cards and balances are unaffected.

If we had not isolated systems and paused orders as quickly as we did, I believe the attackers could have taken tens of thousands of dollars of gift cards, or worse.

It felt like a game of ping pong or speed chess

I have not seen anything like this, and I want to describe what happened and how it felt in more detail.

They were not using VPNs, which our system normally detects. They came from data centres in Europe and the US. They worked nightly, roughly 05:00 to 17:00 UTC (10 pm to 10 am MST), and they kept provisioning new infrastructure: new IP addresses and new, valid email addresses, through dotted Gmail and googlemail.com aliases and through other domains. We blocked these among others:

googlemail.com and gmail.com (dotted and plus-tagged aliases of the same inbox)
inflowmint.com
ascentmint.com
plainsailing1.com
disposable-mail services such as sharklasers.com, guerrillamail, yopmail and mail.tm

Every time we closed a door, they adapted. When we rolled out aggressive rate limiting just as they were ramping up the XLM siphoning, they pushed harder, making multiple sign-up attempts from new IP addresses every second.

During this period I was awake for more than 30 hours straight, actively mitigating the attack myself with the help of Anthropic's Claude Fable and OpenAI's GPT Astra, with multiple sessions running on one screen and live system logs and alerting on the other. I have been developing software for decades. If you picture what hackers look like in the movies, for the first time in my life that is what it felt like. It was both exhilarating and terrifying at the same time.

We do not know who was behind this, and I am not going to guess. But I believe an AI agent was in the loop. We could see the attackers pause, change tactics and come back with something different in response to exactly what we had just done, as if they were adjusting a prompt and trying again. I can't prove that, but it's worth mentioning because my gut feel is this was very much an AI assisted attack. It also matters for what comes next: if attacks can be steered this way, the controls around gift cards and commerce need to be better.

Timeline

Times are in UTC, with Mountain Standard Time (MST) in brackets. Where we only have the local time of day, we give it in MST.

October 16 (evening MST): elevated probing of our API endpoints from many different IP addresses.
October 17 (early morning MST): we detect more password reset attempts than usual for one of our internal dashboards, and an email showing that one had worked, which should not have been possible. Our CTO starts investigating within minutes. We take systems offline immediately, lock the attackers out in under 30 minutes, mitigate, and bring everything back up. Over this day and the next we rotate secrets, send password reset emails only to the account's own address, and shorten how long a login stays valid.
October 17 (evening MST): we find the stolen gift cards. We take systems offline again, mitigate, and bring them back up. The first bulk wallet sign-ups for the network reserve also begin on October 17.
October 18: the bulk wallet sign-ups peak, with close to 200 accounts in a day, most within a single hour.
October 19–20: we move all but about 300 XLM out of the account funding wallet into cold storage, add harsher rate limiting, improve our email and IP blocking, cap how many wallets can be funded per hour, and lower the amount funded to each wallet from 3 XLM to 2.1 XLM.
October 20, 06:00–09:00 UTC (11 pm on October 19 to 2 am MST): the attackers resume with several waves of sign-ups. At 08:43 UTC (1:43 am MST) our limits are live and the bulk sign-ups stop.
October 20, 12:00–12:45 UTC (5:00–5:45 am MST): some of our fixes briefly disrupt service, including about 45 minutes when gift card checkout is unavailable.
October 20–21: we upgrade our cluster components and move every service to the patched Node.js CVE release and deal with the eMudhra CA cert fiasco published over this window.
October 21–22: more probing from the attackers, and our own penetration tests validating that all of our mitigations are working. We complete structured, thorough security reviews of every codebase and roll out high-priority fixes.
October 22, 01:30–01:40 UTC (6:30–6:40 pm MST on October 21): the attackers return briefly. The hourly cap holds, and they take 2.1 XLM from each of 10 accounts, about 21 XLM in total. At 02:24 UTC (7:24 pm MST) we move all but about 3 XLM of the funding wallet into cold storage. We put our systems into maintenance mode to roll out some high-priority fixes identified in internal security reviews and because upstream providers notified us of an overnight cybersecurity event. We are unsure if they are connected.
October 22–23: gift card issuers we partner with tell us they are under cyber attack. We keep our systems offline out of caution, pause new orders and put the site in maintenance mode.

That's where we are today.

When we took systems offline, and why

We took systems offline on October 17 twice: first, immediately after we detected the dashboard intrusion, so the attackers could not do more while we locked them out; and again that evening, when we found the stolen gift cards. Each time we mitigated first and brought systems back up once the problem was closed. Since October 22 we have kept ordering offline because of the industry-wide event described below, and we are using the time for planned upgrades.

What we changed

Rate limiting and access controls across our APIs. We have put aggressive, hardened rate limiting in place across every service, with top-level WAF rules.
Every flaw in the chain is patched. Password reset emails now go only to the account's own address, and lists are rejected where a single value is expected. Our system now ensures transactions can't be driven to zero or below. The affected secret was rotated, along with other keys and secrets.
We performed a comprehensive internal AI-driven security audit. While our system was offline, we used multiple models and Cloudflare's security audit skill to adversarially review our systems. This surfaced more than 40 issues, which we have patched and already rolled out or are rolling out as part of our planned downtime window. The high-severity ones are already deployed, and we are reviewing and working through the last few lower-severity ones. We will also be rolling out a new mobile app update to change the account funding flow.
Wallet funding. We knew this was a risk when we built the system the way we did, which is why we only kept a limited amount of XLM in the dedicated funding wallet at a time, and adjusted that amount as the price of XLM rose and fell. Before AI, we felt it was an acceptable risk: it would take a lot of effort, with real emails and real devices, to siphon off 3 XLM at a time. Not really worth an attacker's time. Based on what we have seen, that calculus has changed. We are completely overhauling how account funding works so that this type of attack is not possible.
Key rotation and secret storage. We rotated keys quickly. Our secrets are encrypted and pulled from a key management service when we deploy, which I recommend to anyone running production systems.
Updates for this week's public security releases. We upgraded our cluster components, moved our services to the patched Node.js release, and have already patched and rolled out the web framework releases across our websites.

Why orders are still paused

Starting Thursday, gift card issuers we partner with told us they were experiencing cyber attacks. Out of an abundance of caution we kept our systems offline. Since then, to our knowledge, this has turned into an industry-wide event. Most if not all of the major gift card platforms are seeing partial or complete outages.

Our systems are secure, but because we work with many of these issuers we can't provision gift cards right now, and certainly not with confidence. We don't want to take on that risk or pass it on to our customers, who would be out money if they received a gift card that may have been compromised. This is a huge problem with gift cards at the best of times, and certainly is now! We are using the time for planned upgrades that need downtime. We are working with our partners to bring everything back as securely as possible, and we will tell you the moment we can reopen, on X and here.

To our customers: I'm personally sorry for the abrupt stoppage and for the length of this outage. I know this is hard on you.

What we are doing next

We will contact OpenAI about the $20 gift card (if anyone from OpenAI is reading this, please contact me), and we may involve other industry partners and law enforcement if we can gather enough evidence for them to pursue. We will keep sharing what we can.

What you should do

As a user, nothing is required. If you want to take a few minutes this week, please make sure to update your operating system and browser on any internet connected devices. The broader global security updates published this week remove a compromised certificate authority from your device's trust list. If you run your own infrastructure, this is also a good week to update Node.js, PHP, Python, Go, your other runtimes and your certificate bundles.

If you're dealing with this too

If your team is facing something similar, or you want to compare notes on what we saw, reach out. We will help however we can.

One last thing. A gift card today is a code that is real money. That code works for whoever holds it, and it unlocks the value stored behind it. Coming from a background in cryptography and distributed systems, I think the industry's security standard of a PIN code is woefully inadequate.

That matters more as software agents start buying on our behalf and as attackers use AI to probe us, and it is why we built Bidali Issuing: digital gift cards issued to one customer and cryptographically verified when they are claimed, so stolen or copied codes stop working.

If you issue or sell gift cards, or you are building agentic commerce, and you would like to learn more, write to sales@bidali.com. If you are having an issue because of this incident, please contact our support team at support@bidali.com.

Questions

Write to support@bidali.com or visit our support center. If you find a security issue, write to security@bidali.com.

Thank you for your patience, and for trusting us.

Eric Kryski Co-founder and CEO, Bidali