POSTS
Real-life OIDC Security (VII): Responsible Disclosure
This is the final post of a series on Single Sign-On and OpenID Connect 1.0 security. In this post, Responsible Disclosure processes with five vendors and maintainers of popular OpenID Connect implementations are outlined. We reported vulnerabilities and security issues in Amazon Cognito, Bitbucket Server, GitLab, Keycloak, and Salesforce.
The series consists of following posts:
- [Overview] Common Issue Patterns and Derived Security Considerations
- [Implementation] Login Confusion
- [Implementation] Injection of CRLF sequences
- [Implementation] SSRF issues in real-life OIDC implementations
- [Specification] Redirect URI Schemes
- [Specification] Reusable State parameter
- [Responsible Disclosure] Lessons learned during Responsible Disclosure of OIDC/OAuth related issues
As you can see, we structure the series in an [Overview], attacks with concrete examples categorized in [Implementation] and [Specification], and finally lessons learned during the [Responsible Disclosure]. Note that this post gives an overview of the Responsible Disclosure processes that were performed during this research.
Acknowledgment
We made the observations within this post during the execution of my master’s thesis on “Single Sign-On Security: Security Analysis of real-life OpenID Connect Implementations”. The thesis was written at the chair for network and data security of Ruhr University Bochum.
The advisors of my thesis are Dr.-Ing. Christian Mainka (@CheariX), Dr.-Ing. Vladislav Mladenov (@v_mladenov) and Prof. Dr. Jörg Schwenk (@JoergSchwenk). Huge “Thank you” for your continuous support! 🙂
Introduction
Within the last weeks several posts and advisories were published on this blog. The contents of these posts originate from research during my master’s thesis that took place roughly from April 2020 to October 2020.
All vendors were informed about the observations being presented within my thesis and the posts on this blog. In the following, a short overview of the communication and disclosure processes with vendors and maintainers is given. During the research, Proof-of-Concepts with varying details were attached with the initial report. We iterated different levels of details. Two examples for these iterations are likewise presented in the following.
Proof-of-Concepts
It is crucial for an advisory that is sent to a Vulnerability Disclosure Program or Bug Bounty Program to contain a reproducible Proof-of-Concept that clearly highlights what the actual security issue is and how it could be exploited in real-world scenarios.
Even though this might sound trivial in the first place, depending on the test environment that is required to reproduce an issue, there are quite some pitfalls. If there is an external triage team in place (e.g., in case of a “managed” program on Bugcrowd or Hackerone) you literally need to start with the very basic: How should the triage person setup a test environment that runs the vulnerable software?
In case of a self-hosted software, luckily Docker containers were available in our cases that allowed a rather simple setup of this component of the setup.
Beside these general considerations for qualified bug and security issue reports, OpenID Connect scenarios need additional effort. In a regular Single Sign-On scenario, three parties are normally involved: End-User, Service Provider and Identity Provider.
Therefore, if we encounter a vulnerability in an Identity Provider implementation, we need to specify how the triage person may set-up the corresponding Service Provider and vice versa.
Finally, a vulnerability in a rather complex protocol might be harder to spot than a Cross-Site Scripting vulnerability that simply pops an alert(1)
. Therefore, complex vulnerabilities require sufficiently detailed steps-to-reproduce. During our disclosure processes, external references to specifications and security considerations appeared to be useful as attachments, even though the rather complex descriptions needed to be set into context sometimes.
Keycloak-based Proof-of-Concepts
In the beginning, we attached a tutorial on how a developer could set-up a Keycloak instance as Identity Provider with proxies and TLS so that an issue regarding a Service Provider implementation could be validated. This process turned out to be time-consuming and error-prone, as the tutorial needed to be adjusted for each finding.
An exemplary setup section for a finding within Atlassian Bitbucket Server’s OpenID Connect implementation is given in the following:
- Modify your
/etc/hosts
file and add the following lines:
127.0.0.1 keycloak.local
127.0.0.1 bitbucket.local
- Start a reverse proxy (modified from https://github.com/flaccid/docker-tls-proxy) to enable TLS for Bitbucket, as this is required to configure OIDC:
docker run \
-it \
-e UPSTREAM_HOST=host.docker.internal \
-e UPSTREAM_PORT=7990 \
-e FORCE_HTTPS=true \
-e ENABLE_HTTP2=true \
-e ENABLE_WEBSOCKET=true \
-e SELF_SIGNED=true \
-p 7780:80 \
-p 7443:443 \
flaccid/tls-proxy
- Create a x509 certificate to enable using TLS in the following:
openssl req -x509 -sha256 -nodes -newkey rsa:4096 -keyout key.pem -days 730 -out certificate.pem
- Launch a new Bitbucket container using:
docker run --name="bitbucket" -v "$(pwd)"/certificate.pem:/tmp/tls.crt --restart always -e SERVER_PROXY_NAME=bitbucket.local -e SERVER_PROXY_PORT=7443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -p 7990:7990 -p 7999:7999 atlassian/bitbucket-server
Note that we mounted the tls.pem you created earlier, so please make sure to run the above command in the right working directory
- Import the certificate using keytool, the default password is “changeit”:
user@laptop:~$ docker exec -it bitbucket bash
root@1344451d39cf:/var/atlassian/application-data/bitbucket#
root@1344451d39cf:/var/atlassian/application-data/bitbucket# keytool -import -file /tmp/tls.crt -keystore $JAVA_HOME/jre/lib/security/cacerts
Enter keystore password:
Owner: CN=keycloak.local, O=Ruhr-Universität, L=Bochum, ST=NRW, C=DE
Issuer: CN=keycloak.local, O=Ruhr-Universität, L=Bochum, ST=NRW, C=DE
Serial number: a3137647c2161a20
Valid from: Fri Apr 03 09:49:40 GMT 2020 until: Wed Dec 28 09:49:40 GMT 2022
Certificate fingerprints:
MD5: CF:E7:62:11:E3:B6:87:DA:5B:74:47:89:09:88:CB:75
SHA1: A8:50:1A:BC:C2:DF:13:00:CD:8F:76:F6:85:5E:4E:B6:84:4D:1E:04
SHA256: 78:70:0A:CD:1F:BA:91:73:D0:10:16:6E:81:0A:2C:F0:9F:40:F6:BF:B6:E6:16:E2:C0:BE:21:4A:4D:BB:C4:10
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 1
Trust this certificate? [no]: yes
Certificate was added to keystore
root@1344451d39cf:/var/atlassian/application-data/bitbucket#
- Copy the
cacerts
file to your host:
docker cp bitbucket:/opt/java/openjdk/jre/lib/security/cacerts bitbucket-bugcrowd-cacerts
- Mount the obtained file when launching a new instance of Bitbucket. Additionally we need to specify the IP address the hostname
keycloak.local
maps to within the container:
docker run --name="bitbucket" --add-host=keycloak.local:172.17.0.1 -v "$(pwd)"/bitbucket-bugcrowd-cacerts:/opt/java/openjdk/jre/lib/security/cacerts --restart always -e SERVER_PROXY_NAME=bitbucket.local -e SERVER_PROXY_PORT=7443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -p 7990:7990 -p 7999:7999 atlassian/bitbucket-server
- Start keycloak docker as follows:
docker run --restart always -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin -e PROXY_ADDRESS_FORWARDING=true -v "$(pwd)"/certificate.pem:/etc/x509/https/tls.crt -v "$(pwd)"/key.pem:/etc/x509/https/tls.key -p 8080:8080 -p 9990:9990 -p 8443:8443 --name "keycloak_local" jboss/keycloak
As you can see, this setup is quite complex. Therefore, we tried to come up with an easier setup with less unneeded overhead to reproduce security issues.
Custom IdP and SP implementations
To minimize the complexity of the setup, custom NodeJS implementations were sent alongside the advisories for later Responsible Disclosure processes. We created dedicated implementations for the Service Provider and the Identity Provider part of the OIDC setup. Both implementations are publicly available on Github:
- Custom OpenID Connect Service Provider: https://github.com/lauritzh/oidc-custom-sp
- Custom OpenID Connect Identity Provider: https://github.com/lauritzh/oidc-custom-idp
These simple implementations can be easily preconfigured, so that the developer ideally only needs to fire-up one command to have an up-and-running web server that showcases the vulnerability if configured against a vulnerable implementation:
$ node sp.js
Or for the custom Identity Provider:
$ node idp.js
Providing preconfigured VMs
In some rare cases even the setup with preconfigured IdP and SP implementations did not work out. In these cases, we installed all required tools (mainly Docker and Node.js) in a fresh VM, set-up the PoC environment and sent an export of this VM to the triage team as last resort.
As you might anticipate: Even a setup that uses the custom implementations and therefore reduces the overhead is quite time-consuming. Nevertheless, it enabled us to reduce the needed turn-arounds before the triage team was able to reproduce reported issues significantly.
Examples
In the following, exemplary experiences made during our responsible disclosure processes are outlined in short.
Keycloak
Keycloak is an open-source software that is maintained by Red Hat. There is a Jira issue tracker for Keycloak issues that allows to flag reported issues as “Security Sensitive Issue”. Using this channel, 13 tickets for individual findings were opened and discussed with the maintainers and developers of Keycloak. The issues were triaged and categorized by Red Hat employees either as security hardening (violation of specification or general best practices without direct exploitability) or security vulnerability that received a CVE.
All reported and fixed issues that are public can be observed within the public Jira instance.
Bitbucket
Atlassian as vendor of Bitbucket offers multiple communication channels. We decided to use the Bugcrowd program for severe security-relevant findings. After the triage that was performed by Bugcrowd, direct contact was held with Atlassian’s security staff. We filed six reports, from which four reports were accepted and two reports were marked as “not applicable” with regard to their policy. In addition, two non-security-relevant bugs were reported via the Service Desk.
Gitlab
GitLab provides multiple channels to communicate bugs and security issues. For one less relevant finding, a GitLab issue was created on gitlab.com. For the more severe issues, Hackerone was used as a direct communication channel with GitLab’s Security Team. After submission of four issues found within the Service Provider implementation, the security team notified us that they do not consider rogue Identity Providers as part of their threat model and closed all issues accordingly. Highlighting the impact and hints for escalation to Remote Code Execution did not lead to reconsideration. One additional report on an Identity Provider flaw was immediately closed with an unclear explanation. After weeks of discussion, it was reopened and triaged as duplicate, because parts of the issue were covered in a two-year-old unfixed report. As we elaborated further on the impact, there was immediate progress in this case. As a result the issue has been fixed within weeks in v13.2.3 and was acknowledged as CVE-2020-13294. There is also a detailed blog post for this issue.
Salesforce
Even though Salesforce has a Responsible Disclosure Policy, communication with the vendor turned out to be quite challenging. We reported our observations at the End of July. As we did not receive a reply, additional contact attempts were made using different mail addresses, previously known contact persons and the German sales support. The only working communication channel that could be established was through the German sales staff. After repeatedly asking for status updates and offering assistance during triage (e.g., answering further questions or providing test environments for Proof-of-Concepts), the very first direct mail from Salesforce’s security team was received 36 days after our initial mail. The security team claimed that our reports were received and were considered as advisories. Additionally, the Salesforce PSIRT informed us that they consider the cases to be closed and will not perform further investigations so that we likewise did not perform further actions.
Amazon Cognito
As Amazon Cognito is part of AWS, we addressed our reports to aws-security@amazon.com. The Security Staff replied and asked for a call, so that we were able to present an overview of our findings to the AWS Security Staff and Amazon Cognito Developers on August 19th, 2020. We presented a demo on some observations and gave further hints regarding the impact and mitigation of the reported issues.
A second call was held on October 9th, 2020. During this call, the Amazon Cognito Developers focussed on one of the reported issues and asked for further explanation of the impact and steps-to-reproduce.
References
- Custom OpenID Connect Service Provider: https://github.com/lauritzh/oidc-custom-sp
- Custom OpenID Connect Identity Provider: https://github.com/lauritzh/oidc-custom-idp
Thank you for reading this post! If you have any feedback, feel free to reach out via Mastodon, Twitter or LinkedIn. 👨💻
You can directly tweet about this post using this link. 🤓
Special thanks to Louis Jannett (@iphoneintosh) for your feedback on this and all previous posts prior to publication! 🙂