From 473ee5c2285434473ca21e5b177c2d897924bcd8 Mon Sep 17 00:00:00 2001 From: Warren Kumari Date: Tue, 27 Aug 2024 13:57:10 -0400 Subject: [PATCH 1/2] Add txt file to .gitignore -- this should be generated from XML. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 783cd5b..fb6e585 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ *.swp +i-d/pdns-qof.txt From b7fdcf41875e9489f012437afc752e9bda18aea2 Mon Sep 17 00:00:00 2001 From: Warren Kumari Date: Tue, 27 Aug 2024 13:57:50 -0400 Subject: [PATCH 2/2] Address Eliot's initial ISE comments, move to RFCv3 XML format. --- i-d/pdns-qof.xml | 314 +++++++++++++++++++++++++++-------------------- 1 file changed, 180 insertions(+), 134 deletions(-) diff --git a/i-d/pdns-qof.xml b/i-d/pdns-qof.xml index fca77de..28f7eec 100644 --- a/i-d/pdns-qof.xml +++ b/i-d/pdns-qof.xml @@ -1,30 +1,38 @@ - + + + - - - - + + + + - - - + + + + + ]> - - - - - - - - - + + + + Passive DNS - Common Output Format @@ -97,12 +105,12 @@ - + General Domain Name System Operations dns - This document describes a common output format of Passive DNS Servers that clients can + This document describes a common output format of Passive DNS servers that clients can query. The output format description also includes a common semantic for each Passive DNS system. By having multiple Passive DNS Systems adhere to the same output format for queries, users of multiple Passive DNS servers will be able to combine result sets easily. @@ -111,29 +119,63 @@
Passive DNS is a technique described by Florian Weimer in 2005 in Passive - DNS replication, F Weimer - 17th Annual FIRST Conference on Computer Security. Since - then, multiple Passive DNS implementations were created and have evolved over time. Users of - these Passive DNS servers may query a server (often via WHOIS - or HTTP REST), parse the results, and process them in other - applications. + DNS replication, F Weimer - 17th Annual FIRST Conference on Computer Security. + It is a mechanism for + logging DNS answers in a manner intended to minimize the privacy + implications to users, and is widely by security researchers to investigate + malware (for example to discover command and control servers), and other + security threats. By capturing only the "cache fill" DNS responses + (responses from authoritative servers in response to queries performed by a + recursive resolver when iteratively resolving a name), Passive DNS does + not have access to the client (users) source IP, source port, destination + IP, or destination port. - There are multiple implementations of Passive DNS software. Users of Passive DNS query - each implementation and aggregate the results for their search. This document describes the - output format of four Passive DNS Systems (, - , , and ) - that are in use today and that already share a nearly identical output format. As the format - and the meaning of output fields from each Passive DNS need to be consistent, this document - proposes a solution to commonly name each field along with its corresponding interpretation. - The format follows a simple key-value structure in JSON - format. The benefit of having a consistent Passive DNS output format is that multiple client - implementations can query different servers without having to have a separate parser for - each individual server. passivedns-client currently - implements multiple parsers due to a lack of standardization. The document does not describe - the protocol (e.g. WHOIS, HTTP REST) - nor the query format used to query the Passive DNS. Neither does this document describe - "pre-recursor" Passive DNS Systems. Each of these are separate topics and deserve their own - RFC documents. This document describes the current best practices implemented in various - Passive DNS server implementations. + As these answers are served in response to queries originally + initiated by user devices, the Passive DNS data can be used to detect if + devices using the resolver are connecting to known malicious domains, + without identifying the individual users / devices. In addition, as + answers are responses to queries made by the recursive server itself, + Passive DNS records the answers which are ultimately served to users. + This is important as authoritative servers may serve different answers to + different query addresses, for example to increase performance (e.g Client Subnet in DNS Queries) or to hide + malicious behavior when queried from addresses known to be associated + with security researchers. + + Passive DNS is usually implemented either by capturing DNS response + packets themselves (i.e packets with a destination address of the + recursive resolver, a source port of 53, and the QR bit set to 1) or + by having the DNS software itself log these responses. The latter method + is likely to become more common as recursive to authoritative DNS + communication becomes encrypted. + + + Multiple Passive DNS implementations and services exist. Users of + these Passive DNS services may query a server (often via WHOIS + or HTTP REST), parse the results, and process + them in other applications. Users of Passive DNS query each + implementation and aggregate the results for their search. This document + describes the output format of four Passive DNS Systems (, , , and ) that are + in use today and that already share a nearly identical output format. As + the format and the meaning of output fields from each Passive DNS need to + be consistent, this document proposes a solution to commonly name each + field along with its corresponding interpretation. The format follows a + simple key-value structure in JSON + format. The benefit of having a consistent Passive DNS output format is + that multiple client implementations can query different servers + without having to have a separate parser for each individual server. + passivedns-client currently implements + multiple parsers due to a lack of standardization. The document does + not describe the protocol (e.g. WHOIS, + HTTP REST) nor the query format used to + query the Passive DNS. Neither does this document describe + "pre-recursor" Passive DNS Systems. Each of these are separate topics + and deserve their own RFC documents. This document describes the + current best practices implemented in various Passive DNS server + implementations.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD @@ -142,15 +184,15 @@
-
+
As Passive DNS servers can include protection mechanisms for their operation, results might be different due to those protection measures. These mechanisms filter out DNS answers if they fail some criteria. The bailiwick algorithm protects - the Passive DNS Database from cache poisoning attacks. + the Passive DNS Database from cache poisoning attacks. Another limitation that clients querying the database need to be aware of is that each query simply gets a snapshot-in-time answer at the time of querying. Clients MUST NOT rely on existing answers from different Passive DNS database. Nor should they assume that answers - will be identical across multiple Passive DNS Servers. + will be identical across multiple Passive DNS servers.
@@ -168,8 +210,9 @@
+ + Formal grammar as defined in ABNF
- Formal grammar as defined in ABNF Implementations SHOULD support one or more fields.
- Specifies how many authoritative DNS answers were received at the Passive DNS Server's + Specifies how many authoritative DNS answers were received at the Passive DNS server's collectors with exactly the given set of values as answers (i.e. same data in the answer set - compare with the uniqueness property in "Mandatory Fields"). The number of requests is expressed as a decimal value. This field is represented as a
- An implementer of a passive DNS Server MAY chose to either return time_first and + An implementer of a passive DNS server MAY chose to either return time_first and time_last OR return zone_time_first and zone_time_last. In pseudocode: (time_first AND time_last) OR (zone_time_first AND zone_time_last). In this case, zone_time_{first,last} replace the time_{first,last} fields. However, this is not encouraged since it might be @@ -337,13 +380,12 @@ ws = *(
- An implementer of a passive DNS Server SHOULD serve a document in this Common Output + An implementer of a passive DNS server SHOULD serve a document in this Common Output Format with a MIME header of "application/x-ndjson".
-
@@ -355,39 +397,53 @@ ws = *(
- Passive DNS Servers capture DNS answers from multiple collection points ("sensors") which + Passive DNS servers capture DNS answers from multiple collection points ("sensors") which are located on the Internet-facing side of DNS recursors ("post-recursor passive DNS"). In this process, they intentionally omit the source IP, source port, destination IP and destination port from the captured packets. Since the data is captured "post-recursor", the timing information (who queries what) is lost, since the recursor will cache the results. - Furthermore, since multiple sensors feed into a passive DNS server, the resulting data gets - mixed together, reducing the likelihood that Passive DNS Servers are able to find out much - about the actual person querying the DNS records. In this sense, passive DNS Servers are + Furthermore, since multiple sensors feed into a passive DNS system, the resulting data gets + mixed together, reducing the likelihood that Passive DNS systems are able to find out much + about the actual person querying the DNS records. In this sense, passive DNS systems are similar to keeping an archive of all previous phone books - if public DNS records can be compared to phone numbers - as they often are. Nevertheless, the authors strongly encourage - Passive DNS implementors to take special care of privacy issues. - bortzmeyer-dnsop-dns-privacy is an excellent starting point for this. Finally, the overall + Passive DNS implementors to take special care of privacy issues. Finally, the overall recommendations in RFC6973 should be taken into consideration when designing any application which uses Passive DNS data. + + Passive DNS attempts to collect information necessary for security (such as malware protection) + in as privacy protecting a manner as possible, and is intended to be + used instead of more invasive methods. It does this by only collecting + DNS cache-fill answers, and not any information associated with who caused the + name to be resolved, nor why the name was resolved. Nevertheless, it is possible that + this may still lead to privacy concerns - for example, if Passive DNS records show that + a recursive resolver resolved the name the-mary-and-john-smith-family.example.com, it may be + possible to infer that the Smith family is using that resolver. Operators of Passive DNS + servers should be aware of this and take appropriate steps to limit access to the data. + + Passive DNS operators are encouraged to read and understand + RFC7258 + + In the scope of the General Data Protection Regulation (GDPR - Directive 95/46/EC), - operators of Passive DNS Server needs to ensure the legal ground and lawfulness of its + operators of Passive DNS server needs to ensure the legal ground and lawfulness of its operation.
In some cases, Passive DNS output might contain confidential information and its access - might be restricted. When a user is querying multiple Passive DNS and aggregating the data, + should be restricted. When a user is querying multiple Passive DNS and aggregating the data, the sensitivity of the data must be considered.
+ + Normative References + &RFC2119; &RFC1035; &RFC1034; &RFC3912; &RFC4627; + &RFC3597; &RFC6648; &RFC2234; &RFC6973; &RFC3986; + &RFC7258; - &RFC2119; &RFC1035; &RFC1034; &RFC3912; &RFC4627; - &RFC3597; &RFC6648; &RFC2234; &RFC6973; &RFC3986; - - @@ -397,65 +453,6 @@ ws = *( - - - Black ops 2008: It's the end of the cache as we know it. - - - - - - - - Passive DNS Hardening - - - - - - - - Queries 5 major Passive DNS databases: BFK, CERTEE, DNSParse, ISC, and VirusTotal. - - - - - - - - Representational State Transfer (REST) - - - - - - - - DNSDB API - - - - - - - - pDNS presentation at 4th Centr R&D workshop Frankfurt Jun 5th 2012 - - - - - - - - CIRCL Passive DNS - - - - - Passive DNS server interface using the common output format @@ -464,14 +461,6 @@ ws = *( - - - DNSDB API Client, C Version - - - - - Discussion on the existing implementations of returning either @@ -482,11 +471,68 @@ ws = *( </reference> </references> + <references> + <name>Informative References</name> + &RFC7871; - <references title="Informative References"> - <!-- Here we use entities that we defined at the beginning. --> - <!-- &I-D.narten-iana-considerations-rfc2434bis; --> - <!-- &I-D.draft-bortzmeyer-dnsop-dns-privacy; --> + <reference anchor="BAILIWICK" + target="https://archive.farsightsecurity.com/Passive_DNS/passive_dns_hardening_handout.pdf"> + <front> + <title>Passive DNS Hardening + + + + + + + + Queries 5 major Passive DNS databases: BFK, CERTEE, DNSParse, ISC, and VirusTotal. + + + + + + + + Representational State Transfer (REST) + + + + + + + + DNSDB API + + + + + + + + pDNS presentation at 4th Centr R&D workshop Frankfurt Jun 5th 2012 + + + + + + + + CIRCL Passive DNS + + + + + + + + DNSDB API Client, C Version + + + +