Fooling Go's X.509 Certificate Verification

# Fooling Go's X.509 Certificate Verification Source: [https://danielmangum.com/posts/fooling-go-x509-certificate-verification/](https://danielmangum.com/posts/fooling-go-x509-certificate-verification/) Below are two[X\.509](https://en.wikipedia.org/wiki/X.509)certificates\. The first is the Certificate Authority \(CA\) root certificate, and the second is a leaf certifcate signed by the private key of the CA\. `ca\.crt\.pem` ``` -----BEGIN CERTIFICATE----- MIIBejCCASGgAwIBAgIUda4UvlFzwQEO/fD0f4hAnj+ydPYwCgYIKoZIzj0EAwIw EjEQMA4GA1UEAxMHUm9vdCBDQTAgFw0yNjAyMjcxOTQ3NDZaGA8yMTI2MDIwMzE5 NDc0NlowEjEQMA4GA1UEAxMHUm9vdCBDQTBZMBMGByqGSM49AgEGCCqGSM49AwEH A0IABKL5BB9aaQ2TtNgUymEsa/+s2ZlTXVll0N22KKWxh0N/JdgHcjrKfzqRlVrt UN2GXdvsdLOq15TxBq97WvE07lKjUzBRMB0GA1UdDgQWBBTAVEw9doSzY1DuPVxP EnwEp/+VJDAfBgNVHSMEGDAWgBTAVEw9doSzY1DuPVxPEnwEp/+VJDAPBgNVHRMB Af8EBTADAQH/MAoGCCqGSM49BAMCA0cAMEQCIHrSTk/KJHAjn3MC/egvfxMM1NpG GEzMB7EH+VXWz7RfAiAyhwy4E9hc8/qsTI+4iKf2o/zMRu5H2GNJOLqOngglbQ== -----END CERTIFICATE----- ``` `leaf\.crt\.pem` ``` -----BEGIN CERTIFICATE----- MIIBHjCBxAIULE3hvnYxU91g9c9H3+uGCSqXi4MwCgYIKoZIzj0EAwIwEjEQMA4G A1UEAwwHUm9vdCBDQTAgFw0yNjAyMjcxOTQ3NDZaGA8yMTI2MDIwMzE5NDc0Nlow DzENMAsGA1UEAwwEbGVhZjBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKDZ21Yh +1AQp1TrxrS8FquIVEHrFRSXncX9xl5vVhZFqvblzTp2Tg7TER5x7rHG1TIqQL1z xDX4TB+nZOWkyAcwCgYIKoZIzj0EAwIDSQAwRgIhAMeo5t2d1RWL/SB0E+mvvIZP jFT0wDWX1Bm26MtxRcf9AiEApG96fs70WF1JliFgzkTiNvbG7Gj4SvErZ9nNX/Lr PnA= -----END CERTIFICATE----- ``` If you downloaded these certificates, you could visually see that the latter references the former as its Issuer\. If you were to use a tool like`openssl`to verify that the leaf is signed by the private key of root, you would see that it is\. > Unless of course you are reading this blog post from the year 2126 or you have changed the system time on your machine\. If the former, I am exceedingly dissapointed that humanity is still using`openssl`\. ``` openssl verify -CAfile ca.crt.pem leaf.crt.pem ``` Now, if you wanted to write a Go program that verified this chain of trust, it might look something like the following\. `main\.go` ``` package main import ( "encoding/pem" "crypto/x509" "fmt" "os" "time" ) func main() { b, err := os.ReadFile("ca.crt.pem") if err != nil { panic(err) } block, _ := pem.Decode(b) ca, err := x509.ParseCertificate(block.Bytes) if err != nil { panic(err) } b, err = os.ReadFile("leaf.crt.pem") if err != nil { panic(err) } block, _ = pem.Decode(b) lc, err := x509.ParseCertificate(block.Bytes) if err != nil { panic(err) } roots := x509.NewCertPool() roots.AddCert(ca) opts := x509.VerifyOptions{ Roots: roots, CurrentTime: time.Now(), } if _, err := lc.Verify(opts); err != nil { panic(err) } fmt.Println("Certificate verification successful.") } ``` But if you ran that program, you might be surprised to see the following\. ``` panic: x509: certificate signed by unknown authority ``` If you used this CA certificate instead, you would see the expected output\. `ca\.verifies\.crt\.pem` ``` -----BEGIN CERTIFICATE----- MIIBejCCASGgAwIBAgIUda4UvlFzwQEO/fD0f4hAnj+ydPYwCgYIKoZIzj0EAwIw EjEQMA4GA1UEAwwHUm9vdCBDQTAgFw0yNjAyMjcxOTQ3NDZaGA8yMTI2MDIwMzE5 NDc0NlowEjEQMA4GA1UEAwwHUm9vdCBDQTBZMBMGByqGSM49AgEGCCqGSM49AwEH A0IABKL5BB9aaQ2TtNgUymEsa/+s2ZlTXVll0N22KKWxh0N/JdgHcjrKfzqRlVrt UN2GXdvsdLOq15TxBq97WvE07lKjUzBRMB0GA1UdDgQWBBTAVEw9doSzY1DuPVxP EnwEp/+VJDAfBgNVHSMEGDAWgBTAVEw9doSzY1DuPVxPEnwEp/+VJDAPBgNVHRMB Af8EBTADAQH/MAoGCCqGSM49BAMCA0cAMEQCIHrSTk/KJHAjn3MC/egvfxMM1NpG GEzMB7EH+VXWz7RfAiAyhwy4E9hc8/qsTI+4iKf2o/zMRu5H2GNJOLqOngglbQ== -----END CERTIFICATE----- ``` ``` Certificate verification successful. ``` At first glance these certificates appear to be identical\. You could use`openssl`to view the contents of both certificates, and you would get identical output\. ``` openssl x509 -in ca.crt.pem -noout -text ``` ``` Certificate: Data: Version: 3 (0x2) Serial Number: 75:ae:14:be:51:73:c1:01:0e:fd:f0:f4:7f:88:40:9e:3f:b2:74:f6 Signature Algorithm: ecdsa-with-SHA256 Issuer: CN = Root CA Validity Not Before: Feb 27 19:47:46 2026 GMT Not After : Feb 3 19:47:46 2126 GMT Subject: CN = Root CA Subject Public Key Info: Public Key Algorithm: id-ecPublicKey Public-Key: (256 bit) pub: 04:a2:f9:04:1f:5a:69:0d:93:b4:d8:14:ca:61:2c: 6b:ff:ac:d9:99:53:5d:59:65:d0:dd:b6:28:a5:b1: 87:43:7f:25:d8:07:72:3a:ca:7f:3a:91:95:5a:ed: 50:dd:86:5d:db:ec:74:b3:aa:d7:94:f1:06:af:7b: 5a:f1:34:ee:52 ASN1 OID: prime256v1 NIST CURVE: P-256 X509v3 extensions: X509v3 Subject Key Identifier: C0:54:4C:3D:76:84:B3:63:50:EE:3D:5C:4F:12:7C:04:A7:FF:95:24 X509v3 Authority Key Identifier: C0:54:4C:3D:76:84:B3:63:50:EE:3D:5C:4F:12:7C:04:A7:FF:95:24 X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: ecdsa-with-SHA256 Signature Value: 30:44:02:20:7a:d2:4e:4f:ca:24:70:23:9f:73:02:fd:e8:2f: 7f:13:0c:d4:da:46:18:4c:cc:07:b1:07:f9:55:d6:cf:b4:5f: 02:20:32:87:0c:b8:13:d8:5c:f3:fa:ac:4c:8f:b8:88:a7:f6: a3:fc:cc:46:ee:47:d8:63:49:38:ba:8e:9e:08:25:6d ``` However, if you were to compare the bytes of the certificates, you would see that there is a very slight difference; two bytes to be exact\. ``` diff <(openssl x509 -in ca.crt.pem -outform der | xxd) <(openssl x509 -in ca.verifies.crt.pem -outform der | xxd) ``` ``` 4c4 < 00000030: 1231 1030 0e06 0355 0403 1307 526f 6f74 .1.0...U....Root --- > 00000030: 1231 1030 0e06 0355 0403 0c07 526f 6f74 .1.0...U....Root 8c8 < 00000070: 1307 526f 6f74 2043 4130 5930 1306 072a ..Root CA0Y0...* --- > 00000070: 0c07 526f 6f74 2043 4130 5930 1306 072a ..Root CA0Y0...* ``` In both locations, there is a`0x13`byte in the certificate which failed verification with the Go program \(`ca\.crt\.pem`\), and a`0x0c`byte in the certificate that passed \(`ca\.verifies\.crt\.pem`\)\. If you are familiar with X\.509 certificates, you’ll know that they are defined using[Abstract Syntax Notation One \(ASN\.1\)](https://en.wikipedia.org/wiki/ASN.1)and are binary encoded using[Distinguished Encoding Rules \(DER\)](https://en.wikipedia.org/wiki/X.690#DER_encoding)\. They are frequently[Base64 encoded](https://en.wikipedia.org/wiki/Base64)and stored and transmitted as[Privacy\-enhanced Mail \(PEM\)](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail)text files \(which you have already seen in this post\)\. The ASN\.1 specification defines a set of data types, each with an associated tag \(non\-negative integer identifier\), which precedes the length and the value when using DER encoding \(see[this post](https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/)from[Let’s Encrypt](https://letsencrypt.org/)for more information\)\.`openssl`can once again be used to see the data types of different fields in the certificate that is successfully verified\. ``` openssl asn1parse -in ca.verifies.crt.pem ``` ``` 0:d=0 hl=4 l= 378 cons: SEQUENCE 4:d=1 hl=4 l= 289 cons: SEQUENCE 8:d=2 hl=2 l= 3 cons: cont [ 0 ] 10:d=3 hl=2 l= 1 prim: INTEGER :02 13:d=2 hl=2 l= 20 prim: INTEGER :75AE14BE5173C1010EFDF0F47F88409E3FB274F6 35:d=2 hl=2 l= 10 cons: SEQUENCE 37:d=3 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 47:d=2 hl=2 l= 18 cons: SEQUENCE 49:d=3 hl=2 l= 16 cons: SET 51:d=4 hl=2 l= 14 cons: SEQUENCE 53:d=5 hl=2 l= 3 prim: OBJECT :commonName 58:d=5 hl=2 l= 7 prim: UTF8STRING :Root CA 67:d=2 hl=2 l= 32 cons: SEQUENCE 69:d=3 hl=2 l= 13 prim: UTCTIME :260227194746Z 84:d=3 hl=2 l= 15 prim: GENERALIZEDTIME :21260203194746Z 101:d=2 hl=2 l= 18 cons: SEQUENCE 103:d=3 hl=2 l= 16 cons: SET 105:d=4 hl=2 l= 14 cons: SEQUENCE 107:d=5 hl=2 l= 3 prim: OBJECT :commonName 112:d=5 hl=2 l= 7 prim: UTF8STRING :Root CA 121:d=2 hl=2 l= 89 cons: SEQUENCE 123:d=3 hl=2 l= 19 cons: SEQUENCE 125:d=4 hl=2 l= 7 prim: OBJECT :id-ecPublicKey 134:d=4 hl=2 l= 8 prim: OBJECT :prime256v1 144:d=3 hl=2 l= 66 prim: BIT STRING 212:d=2 hl=2 l= 83 cons: cont [ 3 ] 214:d=3 hl=2 l= 81 cons: SEQUENCE 216:d=4 hl=2 l= 29 cons: SEQUENCE 218:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Subject Key Identifier 223:d=5 hl=2 l= 22 prim: OCTET STRING [HEX DUMP]:0414C0544C3D7684B36350EE3D5C4F127C04A7FF9524 247:d=4 hl=2 l= 31 cons: SEQUENCE 249:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Authority Key Identifier 254:d=5 hl=2 l= 24 prim: OCTET STRING [HEX DUMP]:30168014C0544C3D7684B36350EE3D5C4F127C04A7FF9524 280:d=4 hl=2 l= 15 cons: SEQUENCE 282:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints 287:d=5 hl=2 l= 1 prim: BOOLEAN :255 290:d=5 hl=2 l= 5 prim: OCTET STRING [HEX DUMP]:30030101FF 297:d=1 hl=2 l= 10 cons: SEQUENCE 299:d=2 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 309:d=1 hl=2 l= 71 prim: BIT STRING ``` In the diff view of the two CA certificates, the bytes that differed preceded the`Root CA`string in two different places: the Subject and the Issuer, which are the same since this is a self\-signed certificate\. They were also followed by a`0x07`byte, which aligns with the number of characters in`Root CA`\(i\.e\. the length of the value\)\. The differing leading byte suggests differing ASN\.1 data types for these fields\. The CA certificate for which validation is successful uses`UTF8String`\(`0x0c`\), and you can use`openssl`with the failing CA certificate to see that it uses`PrintableString`instead \(`0x13`\)\. ``` openssl asn1parse -in ca.crt.pem ``` ``` 0:d=0 hl=4 l= 378 cons: SEQUENCE 4:d=1 hl=4 l= 289 cons: SEQUENCE 8:d=2 hl=2 l= 3 cons: cont [ 0 ] 10:d=3 hl=2 l= 1 prim: INTEGER :02 13:d=2 hl=2 l= 20 prim: INTEGER :75AE14BE5173C1010EFDF0F47F88409E3FB274F6 35:d=2 hl=2 l= 10 cons: SEQUENCE 37:d=3 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 47:d=2 hl=2 l= 18 cons: SEQUENCE 49:d=3 hl=2 l= 16 cons: SET 51:d=4 hl=2 l= 14 cons: SEQUENCE 53:d=5 hl=2 l= 3 prim: OBJECT :commonName 58:d=5 hl=2 l= 7 prim: PRINTABLESTRING :Root CA 67:d=2 hl=2 l= 32 cons: SEQUENCE 69:d=3 hl=2 l= 13 prim: UTCTIME :260227194746Z 84:d=3 hl=2 l= 15 prim: GENERALIZEDTIME :21260203194746Z 101:d=2 hl=2 l= 18 cons: SEQUENCE 103:d=3 hl=2 l= 16 cons: SET 105:d=4 hl=2 l= 14 cons: SEQUENCE 107:d=5 hl=2 l= 3 prim: OBJECT :commonName 112:d=5 hl=2 l= 7 prim: PRINTABLESTRING :Root CA 121:d=2 hl=2 l= 89 cons: SEQUENCE 123:d=3 hl=2 l= 19 cons: SEQUENCE 125:d=4 hl=2 l= 7 prim: OBJECT :id-ecPublicKey 134:d=4 hl=2 l= 8 prim: OBJECT :prime256v1 144:d=3 hl=2 l= 66 prim: BIT STRING 212:d=2 hl=2 l= 83 cons: cont [ 3 ] 214:d=3 hl=2 l= 81 cons: SEQUENCE 216:d=4 hl=2 l= 29 cons: SEQUENCE 218:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Subject Key Identifier 223:d=5 hl=2 l= 22 prim: OCTET STRING [HEX DUMP]:0414C0544C3D7684B36350EE3D5C4F127C04A7FF9524 247:d=4 hl=2 l= 31 cons: SEQUENCE 249:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Authority Key Identifier 254:d=5 hl=2 l= 24 prim: OCTET STRING [HEX DUMP]:30168014C0544C3D7684B36350EE3D5C4F127C04A7FF9524 280:d=4 hl=2 l= 15 cons: SEQUENCE 282:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints 287:d=5 hl=2 l= 1 prim: BOOLEAN :255 290:d=5 hl=2 l= 5 prim: OCTET STRING [HEX DUMP]:30030101FF 297:d=1 hl=2 l= 10 cons: SEQUENCE 299:d=2 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 309:d=1 hl=2 l= 71 prim: BIT STRING ``` This still doesn’t explain why`openssl`verifies successfully with either CA certificate, while the Go program does not\. To dig further, you can compile and step through the program with`gdb`, starting with a breakpoint on`Verify\(\)`\. ``` gdb main -ex 'b crypto/x509.(*Certificate).Verify' -ex 'run' ``` Stepping through[the function](https://github.com/golang/go/blob/de0d77c6f8c3977d90b5541a34bca9da32494e38/src/crypto/x509/verify.go#L538), you eventually arrive at the point where you are building the candidate certificate chains\. > Add a breakpoint for this function using`b crypto/x509\.\(\*Certificate\)\.buildChains`\. ``` var candidateChains [][]*Certificate if opts.Roots.contains(c) { candidateChains = [][]*Certificate{{c}} } else { candidateChains, err = c.buildChains([]*Certificate{c}, nil, &opts) if err != nil { return nil, err } } ``` As part of evaluating whether the certificate pool provided has a candidate chain,[`findPotentialParents\(\)`is called](https://github.com/golang/go/blob/de0d77c6f8c3977d90b5541a34bca9da32494e38/src/crypto/x509/cert_pool.go#L132)on the`Roots`\(it is also called on the`Intermediates`, but there are no intermediate certificates provided in this example\)\. ``` for _, root := range opts.Roots.findPotentialParents(c) { considerCandidate(rootCertificate, root) } ``` Finally, you arrive at the[source of the failure](https://github.com/golang/go/blob/de0d77c6f8c3977d90b5541a34bca9da32494e38/src/crypto/x509/cert_pool.go#L144)\. A potential parent for the leaf certificate should have a Subject that matches the Issuer of the leaf \(i\.e\. the leaf should refer to it as the certificate that was used for signing\)\. ``` for _, c := range s.byName[string(cert.RawIssuer)] { candidate, constraint, err := s.cert(c) if err != nil { continue } kidMatch := bytes.Equal(candidate.SubjectKeyId, cert.AuthorityKeyId) switch { case kidMatch: matchingKeyID = append(matchingKeyID, potentialParent{candidate, constraint}) case (len(candidate.SubjectKeyId) == 0 && len(cert.AuthorityKeyId) > 0) || (len(candidate.SubjectKeyId) > 0 && len(cert.AuthorityKeyId) == 0): oneKeyID = append(oneKeyID, potentialParent{candidate, constraint}) default: mismatchKeyID = append(mismatchKeyID, potentialParent{candidate, constraint}) } } ``` The keys in the[`byName`map of a`CertPool`](https://github.com/golang/go/blob/de0d77c6f8c3977d90b5541a34bca9da32494e38/src/crypto/x509/cert_pool.go#L18)contain the Subject of the CA certificates\. When using the CA certificate that caused verification failure, stepping through the loop above you can see that there are zero iterations, or, that there are no CA certificates with a Subject that matches the leaf Issuer\. How could that be? The key observation is that the raw Subject and Issuer, the literal bytes, are being used for comparison\. ``` // CertPool is a set of certificates. type CertPool struct { byName map[string][]int // cert.RawSubject => index into lazyCerts // lazyCerts contains funcs that return a certificate, // lazily parsing/decompressing it as needed. lazyCerts []lazyCert // haveSum maps from sum224(cert.Raw) to true. It's used only // for AddCert duplicate detection, to avoid CertPool.contains // calls in the AddCert path (because the contains method can // call getCert and otherwise negate savings from lazy getCert // funcs). haveSum map[sum224]bool // systemPool indicates whether this is a special pool derived from the // system roots. If it includes additional roots, it requires doing two // verifications, one using the roots provided by the caller, and one using // the system platform verifier. systemPool bool } ``` We saw earlier that the two CA certificates differed in the ASN\.1 data types used for their Subject\. Expectedly, if you check the data type of the Issuer in the leaf certificate, you’ll see that it is a`UTF8String`, matching the CA certificate that was verified successfully\. ``` openssl asn1parse -in leaf.crt.pem ``` ``` 0:d=0 hl=4 l= 286 cons: SEQUENCE 4:d=1 hl=3 l= 196 cons: SEQUENCE 7:d=2 hl=2 l= 20 prim: INTEGER :2C4DE1BE763153DD60F5CF47DFEB86092A978B83 29:d=2 hl=2 l= 10 cons: SEQUENCE 31:d=3 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 41:d=2 hl=2 l= 18 cons: SEQUENCE 43:d=3 hl=2 l= 16 cons: SET 45:d=4 hl=2 l= 14 cons: SEQUENCE 47:d=5 hl=2 l= 3 prim: OBJECT :commonName 52:d=5 hl=2 l= 7 prim: UTF8STRING :Root CA 61:d=2 hl=2 l= 32 cons: SEQUENCE 63:d=3 hl=2 l= 13 prim: UTCTIME :260227194746Z 78:d=3 hl=2 l= 15 prim: GENERALIZEDTIME :21260203194746Z 95:d=2 hl=2 l= 15 cons: SEQUENCE 97:d=3 hl=2 l= 13 cons: SET 99:d=4 hl=2 l= 11 cons: SEQUENCE 101:d=5 hl=2 l= 3 prim: OBJECT :commonName 106:d=5 hl=2 l= 4 prim: UTF8STRING :leaf 112:d=2 hl=2 l= 89 cons: SEQUENCE 114:d=3 hl=2 l= 19 cons: SEQUENCE 116:d=4 hl=2 l= 7 prim: OBJECT :id-ecPublicKey 125:d=4 hl=2 l= 8 prim: OBJECT :prime256v1 135:d=3 hl=2 l= 66 prim: BIT STRING 203:d=1 hl=2 l= 10 cons: SEQUENCE 205:d=2 hl=2 l= 8 prim: OBJECT :ecdsa-with-SHA256 215:d=1 hl=2 l= 73 prim: BIT STRING ``` Whether this is the correct behavior has been an[ongoing debate](https://github.com/golang/go/issues/31440)in the Go project, and the matter is complicated by some tools, such as`openssl`as seen in this post, treating different ASN\.1 data types for strings as equivalent when verifying certificates\. Typically you’ll be using the same tooling or services for generating CA certificates and the leaf certificates they sign, so the encoding will likely be consistent\. However, given that leaf certificates are typically much shorter lived than CA certificates, your tooling may evolve over time, potentially causing discrepancies in newly generated leaves\. Though Go’s handling of this scenario results in[fail\-closed](https://authzed.com/blog/fail-open)behavior, it can still cause outages and downtime, making it important to be aware of how you are generating certificates and how they are expected to be verified\.