Slope Wallet Sentry Vulnerability — Digital Forensics and Incident Response Report

Preamble

The following statement aims to provide a summary overview of the Slope Wallet vulnerability discovered during the initial root cause analysis of the Solana Wallet hack of August 2nd, 2022. Every description in this factsheet has been confirmed by at least one audit company or is verifiable through publicly available information such as on-chain data. Please reach out to exploit@slope.group in case of inquiries or verification on the details laid out below.

Summary

The Slope Wallet application monitoring logging vulnerability cannot conclusively explain the wallet exploit incident of August 2nd, 2022.

The Slope Sentry vulnerability presents a grave security breach by virtue of collecting sensitive information and storing that data not encrypted on a central server. After 12 days of intensive code and server audits including file, system, memory, network and log forensics by two of the leading crypto auditing firms OtterSec and SlowMist, there is no conclusive evidence of the data on the server ever having been compromised or the data being intercepted in the client-server pipeline.

Slope continues to lead the efforts in helping the affected users recover stolen assets.

1. Outline August 2nd Solana Hack Incident Response:

In the late evening of Tuesday 2nd of August 2022, a total of 9229 wallets The attack lasted a total of 7 hours (22:37 to 05:50 UTC) and drained user assets valued at a total of around 4 million USDC at the time of the hack. The hackers had wallet owner authority when transferring assets to 4 wallets under control of the hackers, implying some sort of private key compromise.

Three hours after the start of the hack, on August 3rd at 01:20 UTC, a Wallet Drain Incident Response Team was assembled by the Solana Foundation. Over 100 individuals from a variety of projects and leading security firms, worked together to analyze possible attack vectors. This extensive investigation lasted for more than 12 hours, and unfortunately a number of sensitive data from the two wallet providers were leaked online and a number of premature and/or misleading public statements were made on the progress of the investigation.

Since a majority of users' reports were reporting hacks using Phantom and Slope mobile wallets and suspecting a supply-chain-attack as the most likely attack vector, a cross-dependency check between the Phantom and Slope mobile applications was initiated. Both applications used application monitoring services provided by Sentry.

At 11:30 UTC it was identified that the Slope specific implementation of Sentry included in Slope Wallet mobile applications (both on Android and iOS) was attempting to send sensitive wallet data to a central server. This data was encrypted via HTTPS TLS and additionally secured via TLS certificate pinning and access to the server hosted by Slope on AliCloud was access controlled. At 11:45 UTC, Slope Finance shut down the application monitoring server, thus closing the vulnerability 15 minutes after it was first reported.

Due to the potentially grave impact of the discovered vulnerability, before the potential size of vulnerability and the vulnerability having been confirmed as the root cause of the hack, at 18:45 UTC, Slope’s official Twitter account officially recommended that ALL Slope Wallet users abandon their wallet addresses and move all assets to newly created wallets as soon as possible.

Having identified a first credible vulnerability, the Solana Incident Response Team abandoned the pursuit of any additional hypotheses for possible attack vectors at 21:28 UTC, without conclusive proof that the uncovered vulnerability would explain the full extent of the hack.

On August 4th at 03:15 UTC, the audit firm OtterSec (www.osec.io) was employed for an extensive server and code audit of Slope. Additional support from the cybersecurity and server audit firm SlowMist (www.slowmist.com) was added at 06:45 UTC. At 16:45 UTC, the digital asset compliance and risk management firm TRM (www.trmlabs.com) was hired to support the asset recovery efforts.

On August 5th at 07:20 UTC, the Slope Wallet mobile applications were pulled from the Android and iOS stores.

1. The Slope Wallet Mobile App Sentry Vulnerability:

1.1. Summary in Simple English

Wallet applications require the use of wallets’ private keys to unlock the wallets and sign transactions. In order to be able to do that, private keys are temporarily loaded into memory and then actively removed from memory.

Sentry Application Monitoring (www.sentry.io) is a leading 3rd party plugin used to debug applications. It helps developers to fix errors in applications by automatically collecting a wide array of data saved in memory and sending this data to a central server where it can be evaluated by the development team. Such services are widely employed in the industry, including at Phantom Wallet and Trust Wallet. Slope employed an on-premise instance of the application monitoring service to debug its mobile applications. Application monitoring was never deployed on the Slope Chrome browser extension.

Sentry provides a number of ways to keep (critical) sensitive data such as the private key from being logged without the knowledge of the developer. It provides automatic keyword filters on the basis of a blacklist in order to avoid inadvertent collection of sensitive data. Additionally, data logged is encrypted during transmission to the server, stored on an encrypted and access-controlled, secure database to avoid unwanted access. These represent four layers of security (collection, transmission, encryption, and access).

Extensive code and server audits by leading audit firms OtterSec (www.osec.io) and SlowMist (www.slowmist.com) took place between August 3rd and August 11th. Ottersec focused on server-side data forensics while SlowMist focused on the full client to server pipeline and the tracking of the stolen funds on-chain. Links to their final findings can be found here: https://osec.io/reports/slope-investigation-report.pdf and here respectively: https://slowmist.medium.com/analysis-of-a-large-scale-attack-on-solana-part-2-ee105d907c12 .

Both audit firms found that, due to two unrelated mistakes in routine feature development, two of the four security layers (collection and encryption) were compromised in Slope mobile wallets for a total of 41 days on client side, and 7 days on server side. In this timeframe, 5,367 wallet private keys were stored on an access-secured database. Out of these, only 1,444 were wallets drained by the hacker. There is no evidence that the remaining 7,785 wallets affected by the hack were ever stored on a Slope server.

Both audit firms conclude that there is no evidence of unwanted access to the server in the timeframe of the vulnerability and that even if access to the server were compromised, it would not be possible for an attacker with direct access to the log server to obtain the full scope of drained wallets. SlowMist additionally concluded that there is no evidence that the data transfer leading to the database (transmission) was compromised.

The audit results therefore remain inconclusive; finding no conclusive evidence linking the application monitoring vulnerability to the August 2nd 2022 exploit.

1.2.Timeline

The following timeline describes the series of events pertaining to Slope’s use of Sentry application monitoring:

10/11/21: Client side: The Sentry SDK was published in Slope Wallet version 1.3. It sent data logs to a Sentry hosted Server. These exports did not include (i.e. filtered out client side) wallet private keys.

12/03/22: Server side: The Sentry server was moved to a solution hosted by Slope in Slope Wallet version 1.4.2 due to increased performance requirements.

01/19/22: Server side: Sentry server migration completed, activation of Sentry data logging.

05/06/22: Server side: Sentry database logging stopped due to errors in the data pipeline (errors in Kafka). Logging was not restarted right away due to upcoming planned maintenance and upgrades in the data analytics pipeline.

06/24/22: Client side: Addition of a toString() method in Slope Wallet version 2.2 that inadvertently bypassed filtering of sensitive parameters by the Sentry blacklist. All log data was encrypted in ciphertext.

07/04/22: Server side: Load balancer update for improved TLS decoding performance.

07/18/22: Client side: Removal of ciphertext Sentry logging in Slope Wallet version 2.2.1.

07/28/22: Server side: Restart of Kafka and thus the entire Sentry data logging pipeline (i.e. start of vulnerability).

08/03/22: Server side: Shutdown of Sentry server (i.e. end of vulnerability).

2. The Slope security audit findings:

2.1.General Scope

• Slope holds an incomplete log of over 1 million (1,214,191) distinct wallet addresses that were imported into a Slope Wallet mobile application since October of 2021. Only 5,033 out of the 9,229 drained wallets (54% of the affected wallets) can be traced to have ever interacted with Slope.
• The Sentry vulnerability can be restricted in scope to only mobile application (APP) users and restricted in time to a window of 7 days (28th of July to 2nd of August 2022).
• The following diagram describes the container setup for the Slope Wallet application:

Fig. 1: Slope Wallet Kubernetes Overview

• The following diagram illustrate the components of the sentry data pipeline:

Fig. 2: Slope Mobile Sentry Data Pipeline

2.2. Database (Postgres & Sentry-Service) Forensics

• Both audit firms completed a review of all database logs and the sentry service configuration.
• The private keys of a total of 6,811 unique wallets (0.4% of the Slope user base on mobile) were found on the Sentry database. Out of these, only 1,444 belonged to the 9,229 affected Solana wallets (15%).
• Data forensics have uncovered no evidence that the remaining 7,785 wallets drained during the hack were ever stored on the Sentry database.
• There is no indication of any log deletion or manipulation taking place. All the Redis pulse messages show that the service was up throughout the entire period while Postgres and Clickhouse logs were paused due to a planned renewal of the analytics pipeline.

Fig. 3: Redis and Postgres Logs for the Slope Sentry Server

Fig. 3: Redis and Postgres Logs for the Slope Sentry Server

Fig. 4: Distribution of Compromised Wallets based by Sentry (event) Date

• The assets of 5,367 wallets (79%) potentially exposed through the Sentry server logging were not drained during the attack. These unaffected wallets held assets valued at over 50,000 USDC at the time of the attack. The hacker ignored these assets while at the same time draining over 300 wallets containing assets valued at less than 20 USDC at the moment of the attack.

2.3. K8s Container (Nginx, Sentry-Relay, Kafka) Forensics

• SlowMist completed a review of all K8s container logs.
• No additional logs were retrieved.
• No evidence of malicious activity.

2.4. Alicloud (Aliyun) Network Forensics

• All the security systems of the Alicloud Container Service for Kubernetes (ACK) were nominal, no abnormalities were detected.

♢ Only HTTPs requests allowed

♢ Istio ingress controller and load balancer nominal

♢ Kubernetes virtual private network nominal

♢ PodSecurityPolicy nominal

♢ Identity management nominal

♢ Runtime monitoring and alerting nominal, no alerts regarding

✦ Malicious mirror

✦ Viruses and malicious programs

✦ Internal intrusion

✦ Container escape

• Server access is protected with 3 factor authentication, all historical login records are recorded, no abnormalities were detected.
• All SSH remote login procedures are analyzed, no abnormalities in the timeframe of the vulnerability.
• No sign of man-in-the-middle attacks:

♢ TLS certificate binding was in place.

♢ DNS hijacking (and other forms of DNS based attacks) have been ruled out by parsing logs queried from AliCloud at htttps://dns.console.aliyun.com/

♢ X.509 certificates are in order.

• No sign of malware identified.

2.5. Source Code Audit

• Sentry was set up to only ingest error level events. A custom parameter filter was employed. This filter was set up to filter out all sensitive data. Due to a parameter concatenation command introduced in Slope Wallet version 2.2.0, the filter was not able to catch all sensitive data.
• A Slope internal code audit was completed. No sign of any additional vulnerabilities was detected.

2.6. Internal Investigation

• The hardware devices of all Slope employees with access to the application monitoring server have been forensically analyzed, no sign of suspicious activity could be detected.

2.7. Ongoing Investigations

• Additional code auditing firms are being engaged for a code audit of the Slope Wallet source code.
• Investigation of any abnormal SSH requests before June 24th, 2022.
• User interviews of potentially affected users that have never used Slope Wallet on a mobile device.
• User interview of users drained by the attack, whose wallet pubkey does not appear in Slope Sentry entries.

Addendum A: Q&A:

1. What was the purpose of switching Sentry to a self-hosted server?

Answer: There are limitations for hosted SaaS Sentry.

At the beginning, we acquired Sentry’s SaaS offering. There is a limit to the number of logs that can be reported each month. With the business scaling, that limit was reached quite quickly, making it impossible to continue to collect errors. The reported logs include two parts: Logs from the productive environment and logs from the development environment. Considering a large amount of reported log data and the high service cost of Sentry’s SaaS offering, we arranged the operation, hosting and maintenance to deploy the service on our own on AliCloud.

2. What is the purpose of adding toString()

Answer: To make debugging more convenient.

The toString() command was used to export all local variables in order to make debugging in the wallet selection more convenient. Code reviews did not catch the fact that this command would export a wallet’s private key and bypass existing security filters.

3. What is the purpose of converting ciphertext into plaintext?

Answer: To make decryption of end-to-end encrypted messaging more performant.

With the introduction of wallet-to-wallet encrypted messaging, two layers of decryption were introduced within the app. In order to facilitate and prevent abnormal bugs, developers decided to uniformly process all in-memory parameters in plaintext.

4. Why were Private Keys logged for a time?

Answer: The data pipeline around the mobile terminals transmit cached data.

For every day following 7/28/22, there are a small number of old logs. Kafka itself will not cache data for a long time (default 7 days). In this case, mobile terminals are turned on for several days after the service is restored, and then the error is sent to the server.

Fig. 5: Distribution of Sentry Timestamps by Nodestore Timestamps