wyc's domain


IBM Data Science Experience: Whole-Cluster Privilege Escalation Disclosure

Posted on February 21, 2017
Tags: security

For the impatient, click here for the original report.

Media coverage can be found at The Register.

This is a disclosure of a privilege escalation vulnerability I found in the IBM Data Science Experience product, which was patched on Feb 15th, 2017. It was a misconfiguration vulnerability with very severe consequences. In short, they left all the Docker TLS keys in the container, which is the same as leaving a jail cell’s keys inside the jail cell. I worked with IBM PSIRT staff to address this. Thank you to Marcia at Zeitgeist Law for helping me think through disclosure issues.

IBM Data Science Experience is an all-in-one data analytics product built on top of IBM’s $300MM investment into Apache Spark. It touts a seamlessly integrated package of big data analytics tools including Spark, Jupyter notebooks, RStudio, Shiny, and more.

What was at stake:

Conditions required:

Timeline

Synopsis

I think that container security will be a big issue for the next 0-3 years. Can containers be used securely and in HIPAA/PCI/HITRUST/etc.-compliant environments? Sure! Just as much as a lot of other software can. But there’s a lot to do to make that happen, and every security control that you need to apply has a compounding p > 0 of being applied incorrectly. Secure defaults can help a lot here, and I think we’re getting there.

As per this vulnerability, it’s simple enough that reading the Docker tutorials would have been sufficient to find/exploit it. Kudos to the IBM security team for taking this seriously and remediating in two weeks instead of the industry norm of 6 months. Here are just some of the (illegal) things that would have been possible through this exploit:

This kind of oversight simply isn’t acceptable for such a high-profile and enterprise product; it tells me that their security review process should be more rigorous. There are people far scarier and more capable than me, and if I could find this in half an hour of educated stumbling, what can those people find if they’re truly determined? I would like to see some real improvements made by this company moving forward, including:

Shameless plug

I think breaking things is fun, but I usually prefer building things. Feel free to reach out if you want to work together.

My original report

(some personal information redacted)

##############################################################################
Vulnerability Report

    Title: Privilege Escalation due to Misconfigured Docker Swarm Services
    Product: IBM Data Science Experience (http://datascience.ibm.com/)

##############################################################################

Summary
-------
The container's access to the Docker Swarm host API was secured via TLS,
however, the keys to access said API were readily available *in the
container*. As is commonly known, access to the Docker host API may be
trivially leveraged into root access on the host machine.


Affected Products
-----------------
- IBM Data Science Experience (http://datascience.ibm.com/)
- Any other accessible services from its machines


Privileges Required
-------------------
- Public Internet access
- Web browser


Concerns
--------
- Root access to container's host machine and (likely) root access to all
  machines on the entire Spark compute cluster network
- R/W access to any connected stores of customer data including network
  drives, LAN-only services, firewall-protected endpoints, etc.
- Installation of rootkits and/or malicious code to steal sensitive business
  data over time


Reproduction
------------
(1) Enter RStudio Web Environment, and focus the cursor on the command line.
(2) Download and Extract Docker:

    > system("wget https://test.docker.com/builds/Linux/x86_64/docker-1.13.1-rc1.tgz")
    > system("tar -xvzf docker*.tgz")

(3) Use docker binary with existing certificates to obtain root-like access to
    the host using volume mounts. You may need to check/correct the naming of the
    pem files for this to work--I'm not sure that I remembered them correctly:

    > system("DOCKER_API_VERSION=1.22 ./docker/docker -H 172.17.0.1 \
              --tlscacert /certs/ca.pem --tlscert /certs/cert.pem \
              --tlskey /certs/key.pem \
              run -v /:/host debian cat /host/etc/shadow")

    This is as good as root access.


Further Untested Implications
-----------------------------
- If all the other Docker Swarm sockets share the same cryptographic handshake
  and are accessible from the internal network, then root access to this one
  box could be leveraged to root the whole network.
- Users could keep on registering new accounts until they are distributed to
  all or nearly all Spark machines on the network, performing the same root
  privilege escalation.
- If customer data is accessible by all nodes, then through exploitation of
  this vulnerability on a single box, it may be feasible to siphon all
  customer data.
- Any other privilege escalations available after obtaining root on a machine


Recommended Remediations
------------------------
These steps may help, but as always, please ensure that they are filtered
through your knowledge of the system archicture to determine if they would be
truly effective:

Fixes and Improvements:

- Remove security certificates found in `/certs/` on Docker containers and
  refactor application architecture as necessary
- More comprehensive sandboxing for environments where remote code execution
  is expected. While containers have some secure defaults, they are not meant
  for security-centric applications, and container security is something that
  must be aggressive pursued. KVM has a much better security record.
- Ensure separation of customer data within data stores
- Consider solutions like AppArmor and SELinux which offer finer-grained
  controls of access
- Lock down `docker pull` on host
- Use a password-protected and TLS-enabled Docker registry if not already

Clean-up:

- Re-image ALL affected machines. Someone may have found this exploit before I
  did and there may already be installed rootkits or malicious software
  services by attackers
- Assessment of all affected customer data and responsible reporting to the
  affected parties


Email with the Context of this Discovery
----------------------------------------

From: Wayne Chang <[email protected]>
To: <REDACTED>@us.ibm.com
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: multipart/alternative; boundary="_----------=_14859852333597655"
Date: Wed, 01 Feb 2017 16:40:33 -0500
Subject: Security Vulnerability

This is a multi-part message in MIME format.

--_----------=_14859852333597655
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="utf-8"

Hi <REDACTED>,



It was a pleasure speaking with you today. Thank you for living up to
IBM's steadfast dedication to security.


I think I may have found a security exploit in an IBM product. I was
at an event called "IBM Open Source Analytics & Data Science
Experience" last night in Atlanta, GA, yesterday on 1/31. We were told
to sign up with free accounts and assured that the technology is
secure and reliable.


I conduct routine security and reliability assessments of potential new
software on behalf of my clients, who have very sensitive data security
concerns. My assessments consist of quick checks for common security
mistakes such as incorrectly configured programs and poor separation of
customer data. To my dismay, one of the checks I performed resulted in
the revelation of a privilege escalation[1] vulnerability having serious
potential for an attacker to impact the stability of the product and
safety of customer data. After this discovery, I immediately disengaged
from the affected product and did not pursue the matter any further, and
will not resume access until this issue is addressed.


I believe in secure software, and I absolutely make every effort to work
with vendors such as yourself to address these issues quietly, safely,
and professionally. With respect to responsible disclosure[2], I'm
looking to you for the best way to proceed next. One of the IBM
representatives mentioned that there may be something like a bug
bounty[3]; if such a thing exists, then I'd be interested.


The website suggests that I should use the following form:
https://www-03.ibm.com/security/secure-engineering/report.html
The same IBM representative from the event recently said over email to
do the same.


Is this the best way, or are you able to get me in touch with someone to
more expediently resolve this potential security issue?


Best,

Wayne Chang

Independent Technology Consultant

WYC Technology, LLC

https://wycd.net


Links:

  1. https://en.wikipedia.org/wiki/Privilege_escalation
  2. https://en.wikipedia.org/wiki/Responsible_disclosure
  3. https://en.wikipedia.org/wiki/Bug_bounty_program