Basic DNSSEC troubleshooting¶
Some of the troubleshooting tips that will be shown here are focused on the BIND9 DNS server and its tools, but the general principle applies to DNSSEC in all implementations.
Handy “bad” and “good” DNSSEC domains¶
It helps to have some good known domains with broken and working DNSSEC available for testing, so we can be sure our tooling is catching those, and not just failing everywhere. There is no guarantee that these domains will be up forever, and certainly there are more out there, but this list is a good first choice:
These should fail DNSSEC validation:
sigfail.ippacket.stream
These should pass DNSSEC validation:
sigok.ippacket.stream
Logs¶
By default, the BIND9 server will log certain DNSSEC failures, and the journal log should be the first place to check.
For example, if we ask a BIND9 Validating Resolver for the IP address of the www.dnssec-failed.org name, we get a failure:
$ dig @127.0.0.1 -t A www.dnssec-failed.org
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 -t A www.dnssec-failed.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 26260
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 6339d7228b8587f401000000671bc2eb2fe25bdf099ef1af (good)
;; QUESTION SECTION:
;www.dnssec-failed.org. IN A
;; Query time: 460 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Oct 25 16:10:19 UTC 2024
;; MSG SIZE rcvd: 78
That’s a very generic failure: it just says SERVFAIL, and gives us no IP: IN A is empty. The BIND9 logs, however, tell a more detailed story:
$ journalctl -u named.service -f
(...)
named[286]: validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
named[286]: no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 68.87.85.132#53
named[286]: validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
named[286]: no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 68.87.68.244#53
named[286]: validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
named[286]: no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 68.87.76.228#53
named[286]: validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
named[286]: no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 68.87.72.244#53
named[286]: validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
named[286]: no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 69.252.250.103#53
named[286]: broken trust chain resolving 'www.dnssec-failed.org/A/IN': 68.87.72.244#53
Client-side tooling: dig¶
One of the more versatile DNS troubleshooting tools is dig, generally used for interrogating DNS name servers to lookup and display domain information, but its broad functionality makes it a flexible aid for DNS troubleshooting. It provides direct control over setting most of the DNS flags in queries, and displays detailed responses for inspection.
For DNSSEC troubleshooting purposes, we are interested in the following features:
+dnssec: Set the “DNSSEC OK” bit in the queries, which tells the resolver to include in its responses the DNSSEC RRSIG records. This is also shown as adoflag in queries.+cd: This means check disabled and tells the resolver we can accept unauthenticated data in the DNS responses.ad: When included in a response, this flag means authenticated data, and tells us that the resolver who provided this answer has performed DNSSEC validation.@<IP>: This parameter lets us direct the query to a specific DNS server running at the provided IP address.
For example, let’s query a local DNS server for the isc.org type A record, and request DNSSEC data:
$ dig @127.0.0.1 -t A +dnssec +multiline isc.org
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 -t A +dnssec +multiline isc.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25738
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; COOKIE: 8c4c8391524d28af01000000671bd625936966d67a5f7061 (good)
;; QUESTION SECTION:
;isc.org. IN A
;; ANSWER SECTION:
isc.org. 207 IN A 151.101.2.217
isc.org. 207 IN A 151.101.66.217
isc.org. 207 IN A 151.101.130.217
isc.org. 207 IN A 151.101.194.217
isc.org. 207 IN RRSIG A 13 2 300 (
20241107074911 20241024070338 27566 isc.org.
BIl7hov5X11CITexzV9w7wbCOpKZrup3FopjgF+RIgOI
5A8p8l2dJCLp/KBn/G6INj7TOHTtrGs1StTSJVNksw== )
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Oct 25 17:32:21 UTC 2024
;; MSG SIZE rcvd: 231
Let’s unpack this answer for the important troubleshooting parts:
The answer has the
adflag set, meaning this data was authenticated. In other words, DNSSEC validation was successful.The status of the query is
NOERROR, and we have 5 records in the answer section.An RRSIG record for the “A” Resource Record was returned as requested by the
+dnsseccommand-line parameter. This is also confirmed by the presence of the “do” flag in the “OPT PSEUDOSECTION”.
If we repeat this query with a domain that we know fails DNSSEC validation, we get the following reply:
$ dig @127.0.0.1 -t A +dnssec +multiline dnssec-failed.org
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 -t A +dnssec +multiline dnssec-failed.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41300
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; COOKIE: b895f4fe3f3d605401000000671bd719636ef1cfc4e615f3 (good)
;; QUESTION SECTION:
;dnssec-failed.org. IN A
;; Query time: 1355 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Oct 25 17:36:25 UTC 2024
;; MSG SIZE rcvd: 74
This time:
There is no
adflag set in the answer.The status of the query is a generic
SERVFAIL, and zero answers were provided.
We can tell the Validating Resolver (the service running on the @127.0.0.1 address) that we don’t want it to perform DNSSEC validation. We do that by setting the +cd (check disabled) flag. Then things change in our answer:
$ dig @127.0.0.1 -t A +dnssec +cd +multiline dnssec-failed.org
; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> @127.0.0.1 -t A +dnssec +cd +multiline dnssec-failed.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7269
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
; COOKIE: dd66930044348f2501000000671bd808f6852a18a0089b3f (good)
;; QUESTION SECTION:
;dnssec-failed.org. IN A
;; ANSWER SECTION:
dnssec-failed.org. 297 IN A 96.99.227.255
dnssec-failed.org. 297 IN RRSIG A 5 2 300 (
20241111145122 20241025144622 44973 dnssec-failed.org.
fa53BQ7HPpKFIPKyn3Md4bVLawQLeatny47hTq1QouG8
DwyVqmsfs3d5kUTFO5FHdCy4U7o97ODYXiVuilEZS/aZ
n6odin2SCm0so4TnIuKBgZFW41zpI6oIRmIVPv6HLerI
uUxovyMEtaGyd5maNgxGldqLzgWkl18TWALYlrk= )
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Fri Oct 25 17:40:24 UTC 2024
;; MSG SIZE rcvd: 267
Looks like we have some sort of answer, but:
There is no
adflag in the answer, so this data was not authenticated.The status is
NOERROR, and we got two answers.Note there is a
cdflag in the answer, meaning “check disabled”. No attempt at validating the answer was done by the resolver, as requested.
In none of these cases, though, did dig perform DNSSEC validation: it just presented the results provided by the Validating Resolver, which in some cases was no validation at all (via the +cd flag). To perform the validation ourselves, we have to use a different tool.
Digging a bit deeper: delv¶
The delv tool is very similar to dig, and can perform the same DNS queries, but with a crucial difference: it can also validate DNSSEC. This is very useful in troubleshooting, because rather than returning a generic SERVFAIL error when something goes wrong with DNSSEC validation, it can tell us what was wrong in more detail.
But to bring the responsibility of doing DNSSEC validation to the tool itself, we use the +cd flag in our queries, to tell the resolver to not attempt that validation. Otherwise we will just get back a generic SERVFAIL error:
$ delv @127.0.0.1 -t A +dnssec +multiline dnssec-failed.org
;; resolution failed: SERVFAIL
With the +cd flag present, however, delv itself will do the validation. It will fail again, but now with a DNSSEC-specific error:
$ delv @127.0.0.1 -t A +dnssec +cd +multiline dnssec-failed.org
;; validating dnssec-failed.org/DNSKEY: no valid signature found (DS)
;; no valid RRSIG resolving 'dnssec-failed.org/DNSKEY/IN': 127.0.0.1#53
;; broken trust chain resolving 'dnssec-failed.org/A/IN': 127.0.0.1#53
;; resolution failed: broken trust chain
If needed, delv can be told to not perform DNSSEC validation at all, by passing the -i flag. Together with the +cd flag, which instructs the Validating Resolver to not perform validation either, we get this result:
$ delv @127.0.0.1 -i -t A +dnssec +cd +multiline dnssec-failed.org
; answer not validated
dnssec-failed.org. 100 IN A 96.99.227.255
For a good DNSSEC domain, delv will return a validated answer:
$ delv @127.0.0.1 -t A +multiline +cd isc.org
; fully validated
isc.org. 300 IN A 151.101.2.217
isc.org. 300 IN A 151.101.66.217
isc.org. 300 IN A 151.101.130.217
isc.org. 300 IN A 151.101.194.217
isc.org. 300 IN RRSIG A 13 2 300 (
20241107074911 20241024070338 27566 isc.org.
BIl7hov5X11CITexzV9w7wbCOpKZrup3FopjgF+RIgOI
5A8p8l2dJCLp/KBn/G6INj7TOHTtrGs1StTSJVNksw== )
Given that above we used the +cd flag, this means that the validation was done by delv itself. We will get the same result without that flag if the Validating Resolver also succeeds in the DNSSEC validation, and provides an answer.
Incorrect time¶
As with everything related to cryptography, having an accurate measurement of time is of crucial importance. In a nutshell, digital signatures and keys have expiration dates.
An RRSIG record (a digital signature of a Resource Record) has a validity. For example, this record:
noble.example.internal. 86400 IN RRSIG A 13 3 86400 (
20241106131533 20241023195023 48112 example.internal.
5fL4apIwCD9kt4XbzzlLxMXY3mj8Li1WZu3qzlcBpERp
lXPgLODbRrWyp7L81xEFnfhecKtEYv+6Y0Xa5iVRug== )
Has this validity range:
valid until: 20241106131533 (2024-11-06 13:15:33 UTC)
valid since: 20241023195023 (2024-10-23 19:50:23 UTC)
If the DNSSEC validator has an incorrect clock, outside of the validity range, the DNSSEC validation will fail. For example, with the clock incorrectly set to before the beginning of the validity period, delv will complain like this:
$ date
Tue Oct 10 10:10:19 UTC 2000
$ delv @10.10.17.229 -a example.internal.key +root=example.internal +multiline noble.example.internal
;; validating example.internal/DNSKEY: verify failed due to bad signature (keyid=48112): RRSIG validity period has not begun
;; validating example.internal/DNSKEY: no valid signature found (DS)
;; no valid RRSIG resolving 'example.internal/DNSKEY/IN': 10.10.17.229#53
;; broken trust chain resolving 'noble.example.internal/A/IN': 10.10.17.229#53
;; resolution failed: broken trust chain
Any other Validating Resolver will fail in a similar way, and should indicate this error in its logs.
BIND9 itself will complain loudly if it’s running on a system with an incorrect clock, as the root zones will fail validation:
named[3593]: managed-keys-zone: DNSKEY set for zone '.' could not be verified with current keys
named[3593]: validating ./DNSKEY: verify failed due to bad signature (keyid=20326): RRSIG validity period has not begun
named[3593]: validating ./DNSKEY: no valid signature found (DS)
named[3593]: broken trust chain resolving './NS/IN': 199.7.83.42#53
named[3593]: resolver priming query complete: broken trust chain
Third-party Web-based diagnostics¶
There are some public third-party web-based tools that will check the status of DNSSEC of a public domain. Here are some:
https://dnsviz.net/: Returns a graphical diagram showing the chain of trust and where it breaks down, if that’s the case.
https://dnssec-debugger.verisignlabs.com/: A DNSSEC debugger which also shows the chain of trust and where it breaks down, in a table format.