Rescuing Schrodinger’s Cat in DeFi Dark Forest
A Real-World Million Dollars DeFi Incident Response
Victor Fang, AnChain.AI, 2020/10
Disclaimer: Based on a recent real world incident response conducted by AnChain.AI team in San Francisco. Due to the client’s confidentiality agreement, this article has removed all PII data.
1 — The million dollar cat lost in quantum state
“Victor, there? ”
A text message from my investor. It was 8:27pm in San Francisco.
I frowned. When a VC calls at night, it is either something extremely good, or extremely bad.
I had a feeling it was going to be bad:
“My friend’s Metamask got hacked. DeFi. Millions. 😢 ”
I texted back: “Ask them to call me.”
Minutes later, a “650-” phone number called. Catherine, in a trembling anxious voice, explained what happened. She’s an investor in a San Francisco based VC firm, and her Metamask wallet was hacked.
9:00 PM — I summoned the crew for an emergency IR meeting, Tomo, and Ralph joined.
- 1 day ago Catherine’s Metamask Ethereum wallet got hacked via social engineering. The private key was leaked.
- Zero wallet balance. 4 ETH has been wired out to the hacker’s wallet.
- The bigger deal: This hacked wallet has staked $1.2 Million ERC20 USDC stable coins in a DeFi smart contract, for yield farming!!!
These staked DeFi assets are intact, but that 1.2 million dollars could be gone any second.
These DeFi assets are like Schrodinger’s cat: lost in the quantum state, locked in the cold hard concealed box as in the Copenhagen quantum mechanics experiment.
We don’t know if the hacker knows this staked DeFi fund exists, or is monitoring the wallet:
- Case 1: Hackers know. They may be waiting for a better DeFi yield?
- Case 2: Hackers don’t know. But they will realize it if we interact w/ the Ethereum dark forest.
If we start probing, we will have a definitive answer one way or another. There’s just one nagging problem with that approach. Our curiosity might kill the cat.
Most clients come to AnChain.AI for help after their crypto has been wired out, and seek out a blockchain forensics service as a last-ditch effort. By this point, it’s often too late, and despite our best efforts, only some retrieve their funds.
But this DeFi incident presents a unique opportunity for us. If we play our cards right, we can rescue all $1.2 Million.
I typed up the email: “Your funds can be gone in any second. You need to hire IR professionals. Our advice to you, whether or not you engage with us:
1. Disconnect the Internet and power off your computer.
2. Use a brand new computer to communicate.
3. DO NOT share the private keys with anyone, including security firms.
4. DO NOT connect your wallet to that DeFi. ”
When typing the last advice, “Dark Forest” flashes into my mind.
7:00 AM — I woke up to the signed engagement letter email, timestamped just past 4:00 AM. It must have been a long night for Catherine. After she proved her identity and ownership of the assets, the IR engagement began in earnest.
“Dark forest theory”, as depicted in the Three Body sci-fi trilogy, holds that civilizations fear one another so much that they don’t dare to reveal themselves lest they immediately be considered a potential threat and destroyed.
Ethereum can be considered a dark forest, as described in the previous two episodes Dan Robinson [2] and Sam Sun [3] at Paradigm.
The AnChain.AI team can sense the daunting green eyes staring in the dark forest as we set to work, as if any move might attract the worst kind of attention. No one said it aloud, but the question hung over us: Can we rescue this million dollar Schrodinger’s cat?
2 — Profiling the dark forest demon: hacker attribution
The world’s most prestigious Incident Response (IR) team, FireEye Mandiant, lives by a work of cybersecurity gospel, a combination of bible and playbook authored by CEO Kevin Mandia’s crew [1].
When I left FireEye Mandiant and founded AnChain.AI, I brought these cybersecurity best practices and created AnChain.AI’s blockchain incident response program.
Versus network and cloud security, blockchain security manifests its unique challenges in the decentralized wild west.
Incident Response is all about 2 key questions: Investigate, and Respond.
To Investigate is to answer the question: “Who is the hacker?”
Hacker attribution is to infer a hacker’s TTP (Tactics, Techniques and Procedures).
What tools are used by the hacker?
- Computer forensics: Besides that social engineering phishing website tactics, is that laptop compromised by malware or backdoors that can tap into her Chrome Metamask plugin?
- Blockchain forensics: Skill set by investigating hackers’ related blockchain transactions and flows. CISO investigation tool visualized the hacker’s on-chain activities. We don’t see much trait of smart contract interactions, other than token transfers.
I opened the AnChain.AI CISO investigation tool, and started digging into the victim and the hacker’s addresses. Looks like the hacker’s phishing scam has compromised six victims so far. Similar tactics like our 2020 Twitter hack investigation: the hacker has already laundered the stolen funds into other wallets.
Thank god, our client’s DeFi staked funds are still there.
- Where is the hacker located? Unlike web servers that can track IP and user agent strings, the Ethereum blockchain ledger only records anonymous wallet addresses and smart contract states. A quick Python script computes the statistics of related wallets, and shows the probability density function (Gaussian kernel smoothed) on its active hours. This hacker is probably in East Asia.
This probability distribution helps us pick the prime time in the strike zone. I have heard incident response stories of hitting too early (prematurely understanding hacker attribution) or too late (hacker realizes the IR). If the hacker spots our activity at any point, the jig is up. To catch our criminal unawares, it looks like our best window is 10am to 8pm PT.
The dark forest demon we profiled, is a teenage computer geek located in East Asia, good at computer hacking but (probably and hopefully) lacking knowledge in DeFi and smart contracts.
3 — Planning the rescue
Sun Tzu’s Art of War, “know yourself and your opponents” is actually hacker attribution, which lays down the assumptions for our response plan.
To Respond is to answer the question “How to fix it?”
Our remediation goal is to surgically move the $1.2 million in stable coins to Catherine’s brand new hardware wallet, the safe.
The biggest hope lies in that DeFi smart contract, esp. The IR team sketched these plans:
Plan A: Can it withdraw to a different wallet, without landing on the victim wallet?
Tomo reviewed the DeFi smart contract code:
Unfortunately, withdraw() has no receiver: it can only withdraw by sending the asset back to the original wallet. (Worth mentioning, in contrast, Uniswap has implemented such a feature that probably considered such an incident response scenario.)
Plan B: Can we freeze the fund, so the hacker cannot transfer it?
Freeze() is a critical function in DeFi governance.
But as seen, _transfer() does not check for “lock” conditions, there is no such “lock” concept for the stable coin tokens.
Plan C: Most well designed DeFi has a “pause” admin button, for emergency scenario.
However, setPause() only works for pausing transactions on certain token contracts, but not individual wallets, which is irrelevant for this case.
Clearly, this DeFi has not considered extreme scenarios. Bummer.
Meanwhile, I also reached out to this DeFi team’s telegram, email, Twitter, Linkedin, as well as their investors (perhaps a bit of a long shot, but they do have a reputation to protect).
No response. Not ideal, but not terribly surprising either.
Well, poor customer service is a common problem for DeFi, isn’t it?
We turned to Catherine: “Unfortunately, we just ruled out 3 plans that could directly save your funds from DeFi. The only option left is Plan Z. It will be a risky and offensive plan. Are you ready? ”
As the big fan of Japanese anime Dragon Ball Z, I think Kamehameha, the deadly finishing move (必殺技), is the best illustration.
4 — Plan-Z: the surgical precision remediation
My coworker in a research hospital, Dr. Anderson, a surgeon, never drinks coffee a day prior to his next surgery because caffeine impacts his fingers. He once grinned at me, waving his hands: ”I need surgical precision! ”
Incident response needs surgical precision too. Quickly jitters can ruin an operation.
Plan Z, our last resort, is to go on the offensive, as simple as:
1, Transfer ETH to the victim’s wallet as gas fee.
2, Unstake the funds from the DeFi pool back to the victim wallet.
3, Transfer them to the safe.
I sketched out the steps to Catherine, increasingly aware of the fact that the plan looked even more reckless in writing than in theory. As expected, this sounded like a horrifying plan: “What if … what if the hacker sees it and moves it before you? ”
“Plan Z is all about speed, and our automated offensive tool will increase our odds, let me explain. ”
I sketched a game theory strategic form analysis that enumerates all possible strategy combinatorics. It’s my favorite tool when facing complex and uncertain adversaries.
The $1.2 million Schrodinger’s cat is in a quantum state: the hacker can be idle (unaware of), or manually transfer our funds to another wallet, or use an automated tool.
Even if the hacker has automated tools, our engineers have promised me the 80% chance that we can win. Overall, game theory shows we have 93% likelihood to win. Good odds, but even a 7% chance of failure feels unnerving when millions of dollars are on the line.
Plan Z is all about speed. Specifically, our goal is to minimize the delta T between unstake and transfer. The 2 key factors:
1. Unstake speed. Optimal gas fee for miners is the key. Ethereum has been notorious for its skyrocketing gas price in 2020 due to the DeFi explosion. In June it peaked 700 Gwei !
200 Gwei would rank us top 1 in the txpool, within 33 seconds confirmation time. Forking over a $12.8 gas express lane DeFi unstaking fee would normally make me wince, but it’s peanuts compared to the $1.2 million. Every fraction of a second counts.
2, Offensive tool: A Python script based on web3 that will frontrun the txpool for adversarial transactions, so that our ERC20 stable coin transaction can be top priority in the mining pool, to transfer to our designated safe. Some related art of frontrun can be found here.
The “Kamehameha” tool is ready. We named the file:
>> FrontrunDarkForest.py
We did a few practice runs on Rinkeby testnet, and Plan Z was ready.
Side note: Due to the offensive nature of this tool, we decide to exclude the technical details. But we may host an Ethereum frontrun theme contest. Stay tuned @AnChainAI.
5 — Rescuing Schrodinger’s cat
2:00 PM, the million dollar DeFi Schrodinger’s cat rescue mission began after two rehearsals.
Catherine took a deep breath, opened the DeFi unstake webpage, signed in to the metamask’s orange fox plugin, then connected the compromised metamask wallet to DeFi.
Immediately, Catherine screamed hysterically, bursting into tears.
“Zero balance!? What!!? ”
The DeFi web page says this wallet has 0 balance in the LP staking pool! The stunned silence lasts just a moment, but it’s just another reminder of what’s at stake.Rationality prevails, and Tomo re-examines the victim’s wallet. Negative, we don’t see any smart contract transaction since the initial, there is NO WAY that the hacker could steal the funds.
I took another look at the cartoonish DeFi website, hmm… on the top right corner, wasn’t it supposed to display the connected Ethereum address?
“Can you try to connect Metamask one more time?”
Catherine held her breath, closed the browser tab, and reopened the DeFi webpage.
Reconnect Metamask.
$1.2 Million USDC staked in the pool.
As the relief permeated the tense atmosphere, all of us had to suppress a laugh at the absurdity of it all. Well, no time to complain about such silly UX design issues for a $200 Million TVL DeFi.
Tomo helped launch the Python script FrontrunDarkForest.py that pumped the ETH gas fee to the victim wallet, then the scrolling terminal showing our Kamehameha is standing by.
Catherine clicked “withdraw” , with gas fee 200 gwei, confirmed on Metamask, the transaction broadcasted to the Ethereum network.
The air in the room froze, and we can hear our heartbeat. Confidence has a nasty way of going AWOL at times like this, when all the chips are on the table.
Thousands of CPU miners all over the world across SparkPool, Nanopool, F2Pool mining pools are desperately fighting for this juicy transaction.
30 seconds later, the smart contract withdrew() transaction confirmed, thanks to 200 Gwei premium gas.
Immediately, FrontrunDarkForest.py kicked in, printed out the message:
[INFO] Frontrun success. USDC transferred to SAFE.
In 3 seconds, as confirmed on etherscan, the USDC has arrived in the designated SAFE wallet.
Technically, by 2:15 PM, the IR mission was accomplished. It literally took 33 seconds, to rescue the $1.2 Million Schrodinger’s cat from the dark forest. The world is still peaceful, apparently the demon in Asia is still in sweet dreams.
We waved goodbye to each other, and walked out of my office building. It’s another beautiful sunny day in San Francisco, the osmanthus flowers have that soothing fragrance calming down my nerves, and only as the early Autumn breeze washes over me do I realize my shirt is damp with sweat.
The Tesla tranquilly merged onto highway 101, and started playing my favorite song, David Bowie’s Starman, as in movie Martian. The exotic Dorian key change propels that cosmic vibe, then goes:
There’s a starman waiting in the sky
He’s told us not to blow it
’Cause he knows it’s all worthwhile.
I recalled Catherine’s beaming smile when seeing her $1.2 million cryptos sitting in her hardware wallet address, and she asked: “What would the hacker feel when seeing this?”
The hacker soon will see this unusual Ethereum transaction that happened at lightning speed. He might feel the Starman has rescued the Schrodinger’s cat from his deep dark forest, back to the good hands.
6 — Learning the lesson
- “Prepare for the inevitable incident”, Part 1 in IR bible [1], as recommended by FireEye Mandiant, and AnChain.AI. You won’t be as lucky as Catherine, that happened to know our investor. Budget for it, and make sure at least one Incident Response team is on your contact list.
- Beware of social engineering. Catherine got hacked via a phishing website that allured and stole her private key. She’s not alone, as seen in figure 1. Don’t assume you are always smarter than the hackers. The cliche: You should under no circumstances give away your private key or passphrase, those 12 or 24 words, remember?
- My 3 questions for anonymous DeFi teams: when hundred of million dollar assets staked in your liquidity pool, whom do we contact for authentic customer support? Who do we suggest better UX user experience? Who to insure investors’ assets at stake like FDIC? I don’t see how DeFi is the future banking, unless I get compelling answers on these 3. What do you think?
Acknowledgement:
I would like to thank Daniel Robinson at Paradigm, Sue Xu at Amino Capital for great feedback.
Reference:
[1]. Jason T. Luttgens, Matthew Pepe, Kevin Mandia, “Incident Response & Computer Forensics, 3rd Edition”, ISBN:9780071798686, McGraw-Hill Education, 2014
[2] Dan Robinson, https://medium.com/@danrobinson/ethereum-is-a-dark-forest-ecc5f0505dff
[3] Sam Sun, https://samczsun.com/escaping-the-dark-forest/
