Update/errata 9/7/2021: Though there are indeed many domains with an SPF record in the CZ ccTLD, the numbers mentioned bellow turned out to be incorrect, due to a calculation error on the part of my source, which only came to light late last night. It turns out that at the time of the scan, there were approximately 1.1 million domains without an SPF record, and only about 300k had the record set (i.e. the ratio was reversed). These numbers are still interesting, though much less optimistic than the originally reported ones...

The Sender Policy Framework (SPF[1]) is a simple but fairly powerful mechanism that may be used (ideally in connection with DKIM[2] and DMARC[3]) to combat phishing to some degree. Basically, it allows a domain name owner to publish a special DNS TXT record containing a list of servers that are authorized to send e-mails for that domain. Existence of this record and its contents is automatically checked by modern e-mail servers whenever they receive an e-mail that appears to come from a certain domain and if the sending server is not on the “approved list” for that domain, the message may be dealt with in a special way – for example marked as potentially fraudulent, moved to a quarantine folder, or even dropped entirely.

The structure of an SPF record is quite simple – after an SPF version specification (v=spf1) one may list “directives”. These are specifications of IP addresses, domain names or other DNS records, that identify valid sources of e-mail for the domain (they may be preceded by a + symbol or – in some instances – an “include” keyword). After the list of “approved senders”, the record should end with a directive that specifies that no other servers may send e-mail for that domain. This may be done by ~all (so called “softfail”) or -all directives, which indicates that no other servers are approved for sending e-mail for that domain.

One might end an SPF record using a “neutral” ?all directive, which would basically mean “an e-mail coming from all other sources, than the ones explicitly listed, should be treated as if this SPF record didn’t exist” (which can be useful for initial deployment or troubleshooting, but defeats the purpose of SPF in any other case, at least when it’s used on its own without DMARC). An SPF record might even end using a +all directive, which would specify that all sources that were not mentioned are explicitly permitted to send messages for the domain. Setting such an SPF record would however be completely nonsensical, since it would basically allow all servers everywhere to legitimately send e-mail on behalf of the domain in question.

A typical SPF record might therefore look like this one, which is currently set for wikimedia.org:

v=spf1 ip4:91.198.174.0/24 ip4:208.80.152.0/22 ip6:2620:0:860::/46 include:_spf.google.com ip4:74.121.51.111 ~all

Some SPF records may include additional mechanisms and be slightly more complex, but for our purposes, the form mentioned above (a list of “valid” senders followed by -all or ~all directive) is quite sufficient.

What’s important to note is that if a record doesn’t specify how e-mail from “all” other senders should be treated (i.e. it doesn’t end in ~all or -all directive), then a ?all directive is implicitly added to its end. This means that a record that is not properly terminated makes very little impact, since it only specifies “allowed” senders, but does not specify that any other sender is not allowed to send e-mails on behalf of the domain.

Although it may seem strange, given what we just mentioned, it is widely accepted that a not insignificant number of SPF records do explicitly end with the ?all directive, which is something that phishing authors sometimes use to their advantage[4,5].

At this point you might reasonably ask what is meant by “a not insignificant number of records”.

Although I can’t offer you hard data for the entire internet, I am in the position to do so for a small part of it – specifically for the .CZ TLD.

The Czech national domain name registry, CZ.NIC (which also operates the Czech National CSIRT) recently created a tool that allows them to scan (literally) all of the domains in the CZ ccTLD space for arbitrary “properties”, such as supported TLS ciphersuits or specific DNS records. And since “misconfigured” SPF records are a pet peeve of mine, ever since I learned about the existence of this tool, I have been bothering a colleague from the National CSIRT, @e2rd_rejthar, with requests for data for SPF-related DNS records. This week, he was kind enough to send them to me, which means I can now share with you fairly exact numbers related to SPF use (though – admittedly – only in a very small part of the internet).

As you may see from the chart, there were many more SPF records than one might have expected (or, at least, that I would have). There were 1,401,785 .CZ domains registered on the day of the scan (August 19th) and these domains used 1,101,489 SPF records in total. Since one may “chain” multiple SPF records for one domain, this does not necessarily mean that over 78.5 % of all CZ domains had SPF record set, but even so, the number is still very high.

Correction - after checking with my colleague, the 1.1M really refers to the number of individual domains with SPFv1 records set, which means that 78.5 % of .cz domains really do have an SPF record.

What’s also surprising is the comparatively small number of “less than optimally set” records. The ?all directive was present in only 35,270 records (~3.2 %) and the “whoever included this should have known better” +all directive was present in only 558 records.

Although the numbers for domains in the .cz TLD are most likely not representative of the wider internet, who knows – maybe there are significantly more SPF records overall than we might expect and perhaps the issue with ?all directives isn’t as commonplace as it used to be… We can only hope.

[1] https://datatracker.ietf.org/doc/html/rfc7208
[2] https://datatracker.ietf.org/doc/html/rfc6376
[3] https://datatracker.ietf.org/doc/html/rfc7489
[4] https://isc.sans.edu/forums/diary/Phishing+email+spoofing+SPFenabled+domain/25426/
[5] https://isc.sans.edu/forums/diary/Agent+Tesla+delivered+by+the+same+phishing+campaign+for+over+a+year/26062/

-----------
Jan Kopriva
@jk0pr
Alef Nula