CVE-2016-9015: Python urllib3 1.17 and 1.18 certificate verification failure

[Cory Benfield <cory () lukasa co uk>]

Versions 1.17 and 1.18 of the Python urllib3 library suffer from a vulnerability that can cause them, in certain 
configurations, to not correctly validate TLS certificates. This places users of the library with those configurations 
at risk of man-in-the-middle and information leakage attacks. This vulnerability affects users using versions 1.17 and 
1.18 of the urllib3 library, who are using the optional PyOpenSSL support for TLS instead of the regular standard 
library TLS backend, and who are using OpenSSL 1.1.0 via PyOpenSSL. This is an extremely uncommon configuration, so the 
security impact of this vulnerability is low.

Affected users should upgrade to urllib3 1.18.1, which has been published today and contains only the mitigation for 
this vulnerability on top of the changes in 1.18. If unable to upgrade, users should downgrade their OpenSSL version or 
temporarily stop injecting PyOpenSSL into urllib3 until they are able to upgrade. A more lengthy description of the 
vulnerability follows.

—

This vulnerability was introduced in a substantial refactor of the PyOpenSSL contrib module. During this refactor, a 
branch of code that mapped the Python standard library certificate verification constants (ssl.CERT_NONE, 
ssl.CERT_OPTIONAL, ssl.CERT_REQUIRED) to OpenSSL verification mode flags (SSL_VERIFY_NONE, SSL_VERIFY_PEER, etc.) was 
accidentally lost. This meant that Python standard library constants would be passed directly to OpenSSL via the 
SSL_CTX_set_verify function’s mode argument.

Unfortunately, these Python standard library constants are Python wrappers around the values of a C enumerated type, 
declared like this:

    enum py_ssl_cert_requirements {
        PY_SSL_CERT_NONE,
        PY_SSL_CERT_OPTIONAL,
        PY_SSL_CERT_REQUIRED
    };

Per the standard C enumerated type rules, these constants have the values 0, 1, and 2 respectively. These integers do 
not all map to their OpenSSL verification mode flag equivalents. While PY_SSL_CERT_NONE and SSL_VERIFY_NONE have the 
same value (0) and PY_SSL_CERT_OPTIONAL and SSL_VERIFY_PEER have the same value (1), PY_SSL_CERT_REQUIRED has the value 
2, which maps to SSL_VERIFY_FAIL_IF_NO_PEER_CERT. This flag is defined by the OpenSSL manual page as being meaningless 
on its own, requiring SSL_VERIFY_PEER to also be set in order to have any effect. Additionally, the manual page 
declares that SSL_VERIFY_FAIL_IF_NO_PEER_CERT has no effect in client mode.

In OpenSSL versions prior to 1.1.0, an implementation detail in the OpenSSL codebase would mean that if any nonzero 
value was passed to the mode argument of SSL_CTX_set_verify this would implicitly have the same effect as setting 
SSL_VERIFY_PEER. Essentially, the only value of mode in OpenSSL versions prior to 1.1.0 that would cause certificate 
validation to be disabled was 0 (SSL_VERIFY_NONE).

In the work done for OpenSSL 1.1.0, this implementation detail was changed to check for the SSL_VERIFY_PEER bit 
directly. As the OpenSSL flags are a bit mask, passing PY_SSL_CERT_REQUIRED (2) would *not* have the SSL_VERIFY_PEER 
bit (1) set, which means that OpenSSL would act act as though SSL_VERIFY_PEER was not set. Thus, if 
PY_SSL_CERT_REQUIRED is passed to OpenSSL directly in client mode, OpenSSL 1.0.2 and earlier treat it as equivalent to 
SSL_VERIFY_PEER, whereas OpenSSL 1.1.0 and later treat it as equivalent to SSL_VERIFY_NONE.

This is unquestionably an application error. The OpenSSL documentation is clear that the 
SSL_VERIFY_FAIL_IF_NO_PEER_CERT flag is both meaningless by itself and in client mode. However, OpenSSL’s unexpected 
change in behaviour, combined with the fact that it has no way to report an invalid flag combination to 
SSL_CTX_set_verify, means that this application error leads to a catastrophic silent security failure when used with 
OpenSSL 1.1.0.

The fix for urllib3 is to reintroduce a mapping between the Python standard library enumerated type and the OpenSSL 
flags. Other applications should audit and confirm that they always successfully pass SSL_VERIFY_PEER to 
SSL_CTX_set_verify. The OpenSSL team have been notified of this behaviour and have concluded that it is not a security 
vulnerability in OpenSSL. However, they are considering reverting this change regardless, on the principle that it is 
better to fail closed than to fail open: https://github.com/openssl/openssl/pull/1793

Fortunately, urllib3’s test suite caught this failure, which led to this investigation. Unfortunately, due to the 
relative scarcity of OpenSSL 1.1.0, two released versions of urllib3 had passed before anyone attempted to run urllib3 
with OpenSSL 1.1.0.

The urllib3 team have contacted downstream redistributors for Red Hat and Debian: both distributions are not using 
versions of urllib3 later than 1.16, and so are unaffected. Additionally, the Python Requests library is using urllib3 
version 1.16 and is also not affected.