DNSSEC and NSEC

If you just want to set up DNSSEC with SERVFAIL, you can jump to that section

What is DNSSEC and NSEC?

DNSSEC

DNSSEC is a technology that allows the signing of DNS records to verify that the anwser hasn't been tampered with. For that, two new DNS records have been introduced[^1]:

DNSKEY: contains the DNSSEC public key
RRSIG: contains the signature of the of the record

And one new record served at the delegation:

DS (Delegation Signer): contains the hash of the DNSKEY

NSEC

With the DNSSEC records it is possible to sign record like A, AAAA, etc. but not negative responses (e.g. NXDOMAIN).

NSEC (Next Secure) and NSEC3 solve this problem through creating a range of pre-signed subdomains which don't exist. NSEC3 uses subdomain hashes to solve the issue of NSEC domain walking. Note that this still does not prevent zone enumeration.

See this blog post by Bastian Kanbach from the APNIC for more information on NSEC, NSEC3 and subdomain enumeration. Note that SERVFAIL does not use NSEC3 "while lies" due to PowerDNS zone replication not working with it enabled.

How does DNSSEC work?

As stated previously, a zone is secured by signing responses using a key and returning the signature alongside the requested record. The public key is stored in the DNSKEY record. To maintain root of trust, (make sure that the DNSKEY has not been tampered with), a hash of it is stored at the delegation (in the DS record). Then, the delegation (usually) signs the DS record as well with their own key. Walking up the chain, the DS record of the delegation's DNSKEY is stored (and signed) by their delegation, up to the root servers. Since the root servers are at the top of the DNS hierachy, their public keys must be blindly trusted.

example.org. AAAA, RRSIG  <--- requested record
example.org. DNSKEY       <--- public key
             ^
             | authenticates
             |
org.         DS, RRSIG    <--- delegation stores a hash of our DNSKEY
org.         DNSKEY       <--- ... and has their own public key, too
             ^
             | authenticates
             |
.            DS, RRSIG    <--- root zone stores a hash of delegation's DNSKEY
.            DNSKEY       <--- if the private key for this one leaks, buy popcorn
        (root of trust)

Overview of the signing hierachy

The DNSKEY of the root servers is rotated quarterly in the IANAs key signing ceremonies.

DNSSEC with SERVFAIL

In the edit page of a zone, click on the DNSSEC menu at the bottom. There, generate a keypair. We reccomended using either ed25519 or ed448 for the key, but some registrars (and even some delegations) don't allow using those algorithms. In that case, generate an ecdsa384 or ecdsa256 key instead. Only use RSA keys if it's abosloutely nessesary (you can also contact the registrar support about this issue - make a difference!). RSA is a legacy cipher, plus it produces longer keys which will generate bigger overheads.
After generating the keys, go to your registrar's management panel and enter the DS record(s). They should handle forwarding the records to your delegation.
Wait, and check whether your registrar sent you any failure notifications. Some registrar will initially accept anything you paste into the DS field and only fail afterwards - check your e-mail. Some delegations (i.e. .eu) will process your request within a few minutes; For others (i.e. .pl, .de) it may take anywhere between a couple hours to a couple days.

To check if you setup DNSSEC correctly, the Verisign labs DNSSEC debugger is a great tool: https://dnssec-debugger.verisignlabs.com. If you encounter deployment problems on the registrar side, our known quirks page may help.

[^1] - originally introduced in RFC2535, current active version in RFC4034, also contains the definitons for the DS and NSEC records