How we checked 4 servers for the eMudhra CA key leak, and why Node.js still trusted the bad certificates

By OpenKrill. Published 2026-10-21. All checks ran on 2026-10-21, from just before the 01:47 UTC package upgrade until 04:45 UTC, against the 12 test hosts of the public advisory at emudhra.we-have-a-problem.com.

The short answer

Upgrade ca-certificates, then test each TLS client on the host, not only curl. A client is safe when it refuses all 12 advisory test hosts and still connects to the control host. Three kinds of client kept their own root list and stayed open after the system upgrade: Python virtual environments with an old certifi, Node.js builds that bundle their roots, and a Mac security agent that copies the old roots into one bundle for every tool.

What happened

Anyone with a leaked intermediate key can issue a certificate for any domain that chains to a trusted root. The advisory lists the 8 intermediates and publishes 12 test hosts, named {dv,ov,ev}-{g3b,g3c}[-aaa]-emudhra.we-have-a-problem.com, whose chains run through them. If a client connects to one of them without a certificate error, an attacker can impersonate any site to that client.

Half of the 12 hosts chain to the emSign ECC Root CA - G3 root. The -aaa half chain through a cross-sign to the older Comodo AAA root. Current root stores have already dropped the AAA root, so most clients block the -aaa hosts. The non-AAA half is the real test, because it stays open until the client removes emSign ECC Root CA - G3 or blocks the intermediates by key.

So far, Ubuntu ca-certificates 20260816 removes the G3 root, Go 1.27.3 blocks the 8 intermediates inside crypto/x509, and certifi 2026.10.18 drops the root from the Python bundle. Node.js merged its fix in PR 66871 on 2026-10-20, but no release carries it yet.

How we checked 4 hosts

We run an OVH VPS, an Oracle Cloud ARM node, a Linode we share with another project, and a Mac. On each one we tested curl, every Python interpreter a service uses (with and without certifi), a small Go probe built with Go 1.27.1 and with Go 1.27.3, and Node.js.

Node.js was the surprise. The official and NodeSource builds compile in their own copy of the Mozilla roots, and that copy still holds emSign ECC Root CA - G3. The flags --use-system-ca and NODE_USE_SYSTEM_CA=1 look like the fix but aren't. They add the system store to the bundled roots, so the bad root stays and Node still trusted 6 of 12. Only --use-openssl-ca replaces the bundled roots with the OpenSSL store. We added that flag to our Node services with a systemd drop-in, and a daily timer tells us when a Node release carries PR 66871.

The Mac was the hardest host. An endpoint protection agent writes a combined CA bundle and points curl, Node.js and pip at it with CURL_CA_BUNDLE, NODE_EXTRA_CA_CERTS and ~/.curlrc. That bundle copies /etc/ssl/cert.pem, which still holds the G3 root and the old AAA root, so curl trusted the test hosts even though macOS itself blocks them. The fix needs an administrator: remove both roots from /etc/ssl/cert.pem and let the agent rebuild its bundle. That fixes curl and pip. Node.js on the Mac still trusts the G3 root through its bundled roots until a fixed release ships.

Containers were the second surprise. A container reads the root store inside its image, not the host's, so the host upgrade doesn't reach it. On our VPS, 6 containers that make outbound TLS calls still trusted the test hosts after the upgrade. A Ruby app and a curl client trusted all 12, and 3 Node.js apps trusted 6. We mounted the host's patched /etc/ssl/certs and /usr/share/ca-certificates read-only into each container and added --use-openssl-ca for the Node.js apps. After a restart, each one blocked 12 of 12. One more container, our Git server, still holds the root: its setup lives outside this repository, so it stays an open item for its owner.

Check your own host

Run this in a shell. Exit code 60 means curl refused the certificate, which is what you want. Exit code 0 means curl trusted a bad chain.

for k in dv ov ev; do for g in g3b g3c g3b-aaa g3c-aaa; do
  h="$k-$g-emudhra.we-have-a-problem.com"
  curl -s -o /dev/null -m 15 "https://$h/"; echo "$? $h"
done; done
curl -s -o /dev/null -m 15 https://emudhra.we-have-a-problem.com/; echo "$? control (want 0)"

Then test Node.js and Python the same way, because they may not use the store curl uses:

node -e 'require("https").get("https://dv-g3c-emudhra.we-have-a-problem.com/",r=>{r.resume();console.log("TRUSTED: not safe")}).on("error",e=>console.log(/^(UNABLE_TO_GET_ISSUER_CERT(_LOCALLY)?|SELF_SIGNED_CERT_IN_CHAIN|CERT_REVOKED|CERT_UNTRUSTED)$/.test(e.code)?"blocked: "+e.code:"no answer, try again: "+e.code))'
python3 -c 'import certifi,ssl,urllib.request as u; u.urlopen("https://dv-g3c-emudhra.we-have-a-problem.com/", context=ssl.create_default_context(cafile=certifi.where()))'

Run the Python line with each virtual environment's own interpreter. A traceback that ends in CERTIFICATE_VERIFY_FAILED means that environment is safe. If Node.js prints TRUSTED, start it with NODE_OPTIONS=--use-openssl-ca on Linux until your Node line ships the fix. To scan a whole CVE list after a patch day, the CVE Risk Check tool ranks the IDs by CISA KEV and EPSS.

Frequently asked questions

Does upgrading ca-certificates fix every program on the server?

No. It fixes programs that read the system store, such as curl, Ubuntu's own Python and Node.js packages, and Go on Linux. Python virtual environments that use certifi and Node.js builds from nodejs.org or NodeSource keep their own roots and need their own fix.

Why does --use-system-ca not fix Node.js?

It adds the operating system's roots to the roots Node.js bundles. The bundled copy still holds emSign ECC Root CA - G3, so the bad chain still verifies. --use-openssl-ca replaces the bundled roots with the OpenSSL store, which the ca-certificates upgrade has already cleaned.

Which certifi version drops the eMudhra root?

certifi 2026.10.18 on PyPI no longer holds emSign ECC Root CA - G3, and in our test it blocked all 12 advisory test hosts. The project's GitHub tag is named 2026.10.19, but PyPI has no 2026.10.19 package.

Does the host fix reach my Docker or Podman containers?

No. A container uses the CA bundle in its own image. Rebuild the image on a patched base, or mount the host's /etc/ssl/certs and /usr/share/ca-certificates read-only, then test from inside the container.

Do I need to rebuild Go programs?

On Linux a Go program reads the system store, so the ca-certificates upgrade already blocks the bad chains, and our Go 1.27.1 probe passed after it. Rebuild with Go 1.27.3 anyway. It blocks the 8 intermediates by key, which still works if a program loads its own roots.

Sources

• The advisory and its 12 test hosts: emudhra.we-have-a-problem.com, read on 2026-10-21.
• The Go fix: Go 1.27.3 and 1.26.10, listed as fixed releases on the advisory page.
• The Node.js fix: nodejs/node PR 66871, merged on 2026-10-20.
• The Python bundle: certifi on PyPI, version 2026.10.18.
• Our own checks: 4 hosts, 2026-10-21 from just before 01:47 to 04:45 UTC, with the script that ships in our repository as ops/ca-trust-check/check.sh.