adulau/pdns-qof
1
0
Fork 0
mirror of https://github.com/adulau/pdns-qof.git synced 2024-11-21 17:47:10 +00:00
Code Issues Projects Releases Packages Wiki Activity

Compare commits

base: adulau:a3875a15b4539ee474145b7b3b5d6e4d91d7cae0
Branches Tags
adulau:master
adulau:AddressEliotsInitialComments
adulau:Minor_Grammar
adulau:XML_Formatting_fixes
..
compare: adulau:3c659e961633d157bc626803179c8d92eca127b8
Branches Tags
adulau:master
adulau:AddressEliotsInitialComments
adulau:Minor_Grammar
adulau:XML_Formatting_fixes

5 commits
a3875a15b4 ... 3c659e9616

Author SHA1 Message Date
Alexandre Dulaunoy
3c659e9616
chg: [I-D] regenerated 2024-08-29 11:31:04 +02:00
Alexandre Dulaunoy
a96d6e9f87
chg: [gitignore] the text file generated is used for reader on GitHub 2024-08-29 11:29:03 +02:00
Alexandre Dulaunoy
de636a7937
Merge pull request #33 from adulau/AddressEliotsInitialComments 
Address Eliots initial comments
 2024-08-29 11:28:36 +02:00
Warren Kumari
b7fdcf4187 Address Eliot's initial ISE comments, move to RFCv3 XML format. 2024-08-27 13:57:50 -04:00
Warren Kumari
473ee5c228 Add txt file to .gitignore -- this should be generated from XML. 2024-08-27 13:57:10 -04:00
4 changed files with 464 additions and 363 deletions
Show stats Expand all files Collapse all files

1
.gitignore vendored
View file

@ -1,2 +1 @@
*.swp *.swp

BIN
i-d/pdns-qof.pdf
View file

Binary file not shown.

512
i-d/pdns-qof.txt
View file

@ -5,21 +5,21 @@
Domain Name System Operations A. Dulaunoy Domain Name System Operations A. Dulaunoy
Internet-Draft CIRCL Internet-Draft CIRCL
Intended status: Informational A. Kaplan Intended status: Informational A. Kaplan
Expires: 29 October 2024 Expires: 28 February 2025
P. Vixie P. Vixie
H. Stern H. Stern
Farsight Security, Inc. Farsight Security, Inc.
W. Kumari W. Kumari
Google Google
27 April 2024 27 August 2024
Passive DNS - Common Output Format Passive DNS - Common Output Format
draft-dulaunoy-dnsop-passive-dns-cof-11 draft-dulaunoy-dnsop-passive-dns-cof-12
Abstract Abstract
This document describes a common output format of Passive DNS Servers This document describes a common output format of Passive DNS servers
that clients can query. The output format description also includes that clients can query. The output format description also includes
a common semantic for each Passive DNS system. By having multiple a common semantic for each Passive DNS system. By having multiple
Passive DNS Systems adhere to the same output format for queries, Passive DNS Systems adhere to the same output format for queries,
@ -41,7 +41,7 @@ Status of This Memo
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on 29 October 2024. This Internet-Draft will expire on 28 February 2025.
Copyright Notice Copyright Notice
@ -53,9 +53,9 @@ Copyright Notice
Dulaunoy, et al. Expires 29 October 2024 [Page 1] Dulaunoy, et al. Expires 28 February 2025 [Page 1]
Internet-Draft Passive DNS - Common Output Format April 2024 Internet-Draft Passive DNS - Common Output Format August 2024
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
@ -70,80 +70,114 @@ Internet-Draft Passive DNS - Common Output Format April 2024
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
2. Limitation . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Common Output Format . . . . . . . . . . . . . . . . . . . . 4 3. Common Output Format . . . . . . . . . . . . . . . . . . . . 4
3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 4 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2. ABNF grammar . . . . . . . . . . . . . . . . . . . . . . 4 3.2. ABNF grammar . . . . . . . . . . . . . . . . . . . . . . 4
3.3. Mandatory Fields . . . . . . . . . . . . . . . . . . . . 5 3.3. Mandatory Fields . . . . . . . . . . . . . . . . . . . . 5
3.3.1. rrname . . . . . . . . . . . . . . . . . . . . . . . 5 3.3.1. rrname . . . . . . . . . . . . . . . . . . . . . . . 6
3.3.2. rrtype . . . . . . . . . . . . . . . . . . . . . . . 5 3.3.2. rrtype . . . . . . . . . . . . . . . . . . . . . . . 6
3.3.3. rdata . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3.3. rdata . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3.4. time_first . . . . . . . . . . . . . . . . . . . . . 6 3.3.4. time_first . . . . . . . . . . . . . . . . . . . . . 6
3.3.5. time_last . . . . . . . . . . . . . . . . . . . . . . 6 3.3.5. time_last . . . . . . . . . . . . . . . . . . . . . . 7
3.4. Optional Fields . . . . . . . . . . . . . . . . . . . . . 6 3.4. Optional Fields . . . . . . . . . . . . . . . . . . . . . 7
3.4.1. count . . . . . . . . . . . . . . . . . . . . . . . . 6 3.4.1. count . . . . . . . . . . . . . . . . . . . . . . . . 7
3.4.2. bailiwick . . . . . . . . . . . . . . . . . . . . . . 7 3.4.2. bailiwick . . . . . . . . . . . . . . . . . . . . . . 7
3.5. Additional Fields . . . . . . . . . . . . . . . . . . . . 7 3.5. Additional Fields . . . . . . . . . . . . . . . . . . . . 7
3.5.1. sensor_id . . . . . . . . . . . . . . . . . . . . . . 7 3.5.1. sensor_id . . . . . . . . . . . . . . . . . . . . . . 7
3.5.2. zone_time_first . . . . . . . . . . . . . . . . . . . 7 3.5.2. zone_time_first . . . . . . . . . . . . . . . . . . . 7
3.5.3. zone_time_last . . . . . . . . . . . . . . . . . . . 7 3.5.3. zone_time_last . . . . . . . . . . . . . . . . . . . 8
3.5.4. origin . . . . . . . . . . . . . . . . . . . . . . . 7 3.5.4. origin . . . . . . . . . . . . . . . . . . . . . . . 8
3.5.5. time_first_ms . . . . . . . . . . . . . . . . . . . . 7 3.5.5. time_first_ms . . . . . . . . . . . . . . . . . . . . 8
3.5.6. time_last_ms . . . . . . . . . . . . . . . . . . . . 8 3.5.6. time_last_ms . . . . . . . . . . . . . . . . . . . . 8
3.6. Additional Fields Registry . . . . . . . . . . . . . . . 8 3.6. Additional Fields Registry . . . . . . . . . . . . . . . 8
3.7. Additional notes . . . . . . . . . . . . . . . . . . . . 8 3.7. Additional notes . . . . . . . . . . . . . . . . . . . . 8
3.8. Suggested MIME Types . . . . . . . . . . . . . . . . . . 8 3.8. Suggested MIME Types . . . . . . . . . . . . . . . . . . 8
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 8 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9
6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 8 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 9
7. Security Considerations . . . . . . . . . . . . . . . . . . . 9 7. Security Considerations . . . . . . . . . . . . . . . . . . . 10
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 8. Normative References . . . . . . . . . . . . . . . . . . . . 10
8.1. Normative References . . . . . . . . . . . . . . . . . . 9 9. Informative References . . . . . . . . . . . . . . . . . . . 11
8.2. References . . . . . . . . . . . . . . . . . . . . . . . 10 Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 12
8.3. Informative References . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13
Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12
Dulaunoy, et al. Expires 29 October 2024 [Page 2]
Dulaunoy, et al. Expires 28 February 2025 [Page 2]
Internet-Draft Passive DNS - Common Output Format April 2024 Internet-Draft Passive DNS - Common Output Format August 2024
1. Introduction 1. Introduction
Passive DNS is a technique described by Florian Weimer in 2005 in Passive DNS is a technique described by Florian Weimer in 2005 in
Passive DNS replication, F Weimer - 17th Annual FIRST Conference on Passive DNS replication, F Weimer - 17th Annual FIRST Conference on
Computer Security [WEIMERPDNS]. Since then, multiple Passive DNS Computer Security [WEIMERPDNS]. It is a mechanism for logging DNS
implementations were created and have evolved over time. Users of answers in a manner intended to minimize the privacy implications to
these Passive DNS servers may query a server (often via WHOIS users, and is widely by security researchers to investigate malware
[RFC3912] or HTTP REST [REST]), parse the results, and process them (for example to discover command and control servers), and other
in other applications. security threats. By capturing only the "cache fill" DNS responses
(responses from authoritative servers in response to queries
performed by a recursive resolver when iteratively resolving a name),
Passive DNS does not have access to the client (users) source IP,
source port, destination IP, or destination port.
There are multiple implementations of Passive DNS software. Users of As these answers are served in response to queries originally
Passive DNS query each implementation and aggregate the results for initiated by user devices, the Passive DNS data can be used to detect
their search. This document describes the output format of four if devices using the resolver are connecting to known malicious
Passive DNS Systems ([DNSDB], [DNSDBQ] , [PDNSCERTAT], [PDNSCIRCL] domains, without identifying the individual users / devices. In
and [PDNSCOF]) that are in use today and that already share a nearly addition, as answers are responses to queries made by the recursive
identical output format. As the format and the meaning of output server itself, Passive DNS records the answers which are ultimately
fields from each Passive DNS need to be consistent, this document served to users. This is important as authoritative servers may
proposes a solution to commonly name each field along with its serve different answers to different query addresses, for example to
corresponding interpretation. The format follows a simple key-value increase performance (e.g Client Subnet in DNS Queries [RFC7871]) or
structure in JSON [RFC4627] format. The benefit of having a to hide malicious behavior when queried from addresses known to be
consistent Passive DNS output format is that multiple client associated with security researchers.
implementations can query different servers without having to have a
separate parser for each individual server. passivedns-client Passive DNS is usually implemented either by capturing DNS response
[PDNSCLIENT] currently implements multiple parsers due to a lack of packets themselves (i.e packets with a destination address of the
standardization. The document does not describe the protocol (e.g. recursive resolver, a source port of 53, and the QR bit set to 1) or
WHOIS [RFC3912], HTTP REST [REST]) nor the query format used to query by having the DNS software itself log these responses. The latter
the Passive DNS. Neither does this document describe "pre-recursor" method is likely to become more common as recursive to authoritative
Passive DNS Systems. Each of these are separate topics and deserve DNS communication becomes encrypted.
their own RFC documents. This document describes the current best
practices implemented in various Passive DNS server implementations. Multiple Passive DNS implementations and services exist. Users of
these Passive DNS services may query a server (often via WHOIS
[RFC3912] or HTTP REST [REST]), parse the results, and process them
in other applications. Users of Passive DNS query each
implementation and aggregate the results for their search. This
document describes the output format of four Passive DNS Systems
([DNSDB], [DNSDBQ] , [PDNSCERTAT], [PDNSCIRCL] and [PDNSCOF]) that
are in use today and that already share a nearly identical output
format. As the format and the meaning of output fields from each
Passive DNS need to be consistent, this document proposes a solution
to commonly name each field along with its corresponding
interpretation. The format follows a simple key-value structure in
JSON [RFC4627] format. The benefit of having a consistent Passive
DNS output format is that multiple client implementations can query
different servers without having to have a separate parser for each
Dulaunoy, et al. Expires 28 February 2025 [Page 3]
Internet-Draft Passive DNS - Common Output Format August 2024
individual server. passivedns-client [PDNSCLIENT] currently
implements multiple parsers due to a lack of standardization. The
document does not describe the protocol (e.g. WHOIS [RFC3912], HTTP
REST [REST]) nor the query format used to query the Passive DNS.
Neither does this document describe "pre-recursor" Passive DNS
Systems. Each of these are separate topics and deserve their own RFC
documents. This document describes the current best practices
implemented in various Passive DNS server implementations.
1.1. Requirements Language 1.1. Requirements Language
@ -151,26 +185,18 @@ Internet-Draft Passive DNS - Common Output Format April 2024
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
2. Limitation 2. Limitations
As Passive DNS servers can include protection mechanisms for their As Passive DNS servers can include protection mechanisms for their
operation, results might be different due to those protection operation, results might be different due to those protection
measures. These mechanisms filter out DNS answers if they fail some measures. These mechanisms filter out DNS answers if they fail some
criteria. The bailiwick algorithm [BAILIWICK] protects the Passive criteria. The bailiwick algorithm [BAILIWICK] protects the Passive
DNS Database from cache poisoning attacks [CACHEPOISONING]. Another DNS Database from cache poisoning attacks. Another limitation that
limitation that clients querying the database need to be aware of is clients querying the database need to be aware of is that each query
that each query simply gets a snapshot-in-time answer at the time of simply gets a snapshot-in-time answer at the time of querying.
querying. Clients MUST NOT rely on existing answers from different Clients MUST NOT rely on existing answers from different Passive DNS
Passive DNS database. Nor should they assume that answers will be database. Nor should they assume that answers will be identical
across multiple Passive DNS servers.
Dulaunoy, et al. Expires 29 October 2024 [Page 3]
Internet-Draft Passive DNS - Common Output Format April 2024
identical across multiple Passive DNS Servers.
3. Common Output Format 3. Common Output Format
@ -192,6 +218,14 @@ Internet-Draft Passive DNS - Common Output Format April 2024
Formal grammar as defined in ABNF [RFC2234] Formal grammar as defined in ABNF [RFC2234]
Dulaunoy, et al. Expires 28 February 2025 [Page 4]
Internet-Draft Passive DNS - Common Output Format August 2024
answer = entries answer = entries
entries = * ( entry newline ) entries = * ( entry newline )
entry = ws "{" ws keyvallist ws "}" ws entry = ws "{" ws keyvallist ws "}" ws
@ -215,16 +249,7 @@ Internet-Draft Passive DNS - Common Output Format April 2024
) )
Figure 1
Dulaunoy, et al. Expires 29 October 2024 [Page 4]
Internet-Draft Passive DNS - Common Output Format April 2024
Note that value is defined in JSON [RFC4627] and has the same Note that value is defined in JSON [RFC4627] and has the same
specification as there. The same goes for the definition of string. specification as there. The same goes for the definition of string.
@ -246,6 +271,17 @@ Internet-Draft Passive DNS - Common Output Format April 2024
Senders SHOULD send an array for rdata, but receivers MUST be able to Senders SHOULD send an array for rdata, but receivers MUST be able to
accept a single-string result for rdata. accept a single-string result for rdata.
Dulaunoy, et al. Expires 28 February 2025 [Page 5]
Internet-Draft Passive DNS - Common Output Format August 2024
3.3.1. rrname 3.3.1. rrname
This field returns the name of the queried resource. Represented as This field returns the name of the queried resource. Represented as
@ -268,20 +304,6 @@ Internet-Draft Passive DNS - Common Output Format April 2024
a decimal value (as mentioned above) answer represented as a JSON a decimal value (as mentioned above) answer represented as a JSON
[RFC4627] number. [RFC4627] number.
Dulaunoy, et al. Expires 29 October 2024 [Page 5]
Internet-Draft Passive DNS - Common Output Format April 2024
3.3.3. rdata 3.3.3. rdata
This field returns the resource records of the queried resource. This field returns the resource records of the queried resource.
@ -306,6 +328,16 @@ Internet-Draft Passive DNS - Common Output Format April 2024
timestamp). The time zone MUST be UTC. This field is represented as timestamp). The time zone MUST be UTC. This field is represented as
a JSON [RFC4627] number. a JSON [RFC4627] number.
Dulaunoy, et al. Expires 28 February 2025 [Page 6]
Internet-Draft Passive DNS - Common Output Format August 2024
3.3.5. time_last 3.3.5. time_last
This field returns the last time that the unique tuple (rrname, This field returns the last time that the unique tuple (rrname,
@ -321,23 +353,12 @@ Internet-Draft Passive DNS - Common Output Format April 2024
3.4.1. count 3.4.1. count
Specifies how many authoritative DNS answers were received at the Specifies how many authoritative DNS answers were received at the
Passive DNS Server's collectors with exactly the given set of values Passive DNS server's collectors with exactly the given set of values
as answers (i.e. same data in the answer set - compare with the as answers (i.e. same data in the answer set - compare with the
uniqueness property in "Mandatory Fields"). The number of requests uniqueness property in "Mandatory Fields"). The number of requests
is expressed as a decimal value. This field is represented as a JSON is expressed as a decimal value. This field is represented as a JSON
[RFC4627] number. [RFC4627] number.
Dulaunoy, et al. Expires 29 October 2024 [Page 6]
Internet-Draft Passive DNS - Common Output Format April 2024
3.4.2. bailiwick 3.4.2. bailiwick
The bailiwick is the best estimate of the apex of the zone where this The bailiwick is the best estimate of the apex of the zone where this
@ -365,6 +386,14 @@ Internet-Draft Passive DNS - Common Output Format April 2024
timestamp). The time zone MUST be UTC. This field is represented as timestamp). The time zone MUST be UTC. This field is represented as
a JSON [RFC4627] number. a JSON [RFC4627] number.
Dulaunoy, et al. Expires 28 February 2025 [Page 7]
Internet-Draft Passive DNS - Common Output Format August 2024
3.5.3. zone_time_last 3.5.3. zone_time_last
This field returns the last time that the unique tuple (rrname, This field returns the last time that the unique tuple (rrname,
@ -385,15 +414,6 @@ Internet-Draft Passive DNS - Common Output Format April 2024
that the resolution is in milliseconds since 1st of January 1970 that the resolution is in milliseconds since 1st of January 1970
(UTC). (UTC).
Dulaunoy, et al. Expires 29 October 2024 [Page 7]
Internet-Draft Passive DNS - Common Output Format April 2024
3.5.6. time_last_ms 3.5.6. time_last_ms
Same meaning as the field "time_last", with the only difference, that Same meaning as the field "time_last", with the only difference, that
@ -408,7 +428,7 @@ Internet-Draft Passive DNS - Common Output Format April 2024
3.7. Additional notes 3.7. Additional notes
An implementer of a passive DNS Server MAY chose to either return An implementer of a passive DNS server MAY chose to either return
time_first and time_last OR return zone_time_first and time_first and time_last OR return zone_time_first and
zone_time_last. In pseudocode: (time_first AND time_last) OR zone_time_last. In pseudocode: (time_first AND time_last) OR
(zone_time_first AND zone_time_last). In this case, (zone_time_first AND zone_time_last). In this case,
@ -419,10 +439,17 @@ Internet-Draft Passive DNS - Common Output Format April 2024
3.8. Suggested MIME Types 3.8. Suggested MIME Types
An implementer of a passive DNS Server SHOULD serve a document in An implementer of a passive DNS server SHOULD serve a document in
this Common Output Format with a MIME header of "application/ this Common Output Format with a MIME header of "application/
x-ndjson". x-ndjson".
Dulaunoy, et al. Expires 28 February 2025 [Page 8]
Internet-Draft Passive DNS - Common Output Format August 2024
4. Acknowledgements 4. Acknowledgements
Thanks to the Passive DNS developers who contributed to the document. Thanks to the Passive DNS developers who contributed to the document.
@ -433,134 +460,131 @@ Internet-Draft Passive DNS - Common Output Format April 2024
6. Privacy Considerations 6. Privacy Considerations
Passive DNS Servers capture DNS answers from multiple collection Passive DNS servers capture DNS answers from multiple collection
points ("sensors") which are located on the Internet-facing side of points ("sensors") which are located on the Internet-facing side of
DNS recursors ("post-recursor passive DNS"). In this process, they DNS recursors ("post-recursor passive DNS"). In this process, they
intentionally omit the source IP, source port, destination IP and intentionally omit the source IP, source port, destination IP and
destination port from the captured packets. Since the data is destination port from the captured packets. Since the data is
captured "post-recursor", the timing information (who queries what) captured "post-recursor", the timing information (who queries what)
is lost, since the recursor will cache the results. Furthermore, is lost, since the recursor will cache the results. Furthermore,
since multiple sensors feed into a passive DNS server, the resulting since multiple sensors feed into a passive DNS system, the resulting
data gets mixed together, reducing the likelihood that Passive DNS data gets mixed together, reducing the likelihood that Passive DNS
systems are able to find out much about the actual person querying
the DNS records. In this sense, passive DNS systems are similar to
Dulaunoy, et al. Expires 29 October 2024 [Page 8]
Internet-Draft Passive DNS - Common Output Format April 2024
Servers are able to find out much about the actual person querying
the DNS records. In this sense, passive DNS Servers are similar to
keeping an archive of all previous phone books - if public DNS keeping an archive of all previous phone books - if public DNS
records can be compared to phone numbers - as they often are. records can be compared to phone numbers - as they often are.
Nevertheless, the authors strongly encourage Passive DNS implementors Nevertheless, the authors strongly encourage Passive DNS implementors
to take special care of privacy issues. bortzmeyer-dnsop-dns-privacy to take special care of privacy issues. Finally, the overall
is an excellent starting point for this. Finally, the overall
recommendations in RFC6973 [RFC6973] should be taken into recommendations in RFC6973 [RFC6973] should be taken into
consideration when designing any application which uses Passive DNS consideration when designing any application which uses Passive DNS
data. data.
Passive DNS attempts to collect information necessary for security
(such as malware protection) in as privacy protecting a manner as
possible, and is intended to be used instead of more invasive
methods. It does this by only collecting DNS cache-fill answers, and
not any information associated with who caused the name to be
resolved, nor why the name was resolved. Nevertheless, it is
possible that this may still lead to privacy concerns - for example,
if Passive DNS records show that a recursive resolver resolved the
name the-mary-and-john-smith-family.example.com, it may be possible
to infer that the Smith family is using that resolver. Operators of
Passive DNS servers should be aware of this and take appropriate
steps to limit access to the data.
Passive DNS operators are encouraged to read and understand RFC7258
[RFC7258]
In the scope of the General Data Protection Regulation (GDPR - In the scope of the General Data Protection Regulation (GDPR -
Directive 95/46/EC), operators of Passive DNS Server needs to ensure Directive 95/46/EC), operators of Passive DNS server needs to ensure
the legal ground and lawfulness of its operation. the legal ground and lawfulness of its operation.
Dulaunoy, et al. Expires 28 February 2025 [Page 9]
Internet-Draft Passive DNS - Common Output Format August 2024
7. Security Considerations 7. Security Considerations
In some cases, Passive DNS output might contain confidential In some cases, Passive DNS output might contain confidential
information and its access might be restricted. When a user is information and its access should be restricted. When a user is
querying multiple Passive DNS and aggregating the data, the querying multiple Passive DNS and aggregating the data, the
sensitivity of the data must be considered. sensitivity of the data must be considered.
8. References 8. Normative References
8.1. Normative References
[RFC1034] Mockapetris, P., "Domain names - concepts and facilities",
STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987,
<https://www.rfc-editor.org/info/rfc1034>.
[RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997, DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>. <https://www.rfc-editor.org/info/rfc2119>.
[RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax [RFC1035] Mockapetris, P., "Domain names - implementation and
Specifications: ABNF", RFC 2234, DOI 10.17487/RFC2234, specification", STD 13, RFC 1035, DOI 10.17487/RFC1035,
November 1997, <https://www.rfc-editor.org/info/rfc2234>. November 1987, <https://www.rfc-editor.org/info/rfc1035>.
[RFC3597] Gustafsson, A., "Handling of Unknown DNS Resource Record
(RR) Types", RFC 3597, DOI 10.17487/RFC3597, September
2003, <https://www.rfc-editor.org/info/rfc3597>.
Dulaunoy, et al. Expires 29 October 2024 [Page 9]
Internet-Draft Passive DNS - Common Output Format April 2024
[RFC1034] Mockapetris, P., "Domain names - concepts and facilities",
STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987,
<https://www.rfc-editor.org/info/rfc1034>.
[RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912, [RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912,
DOI 10.17487/RFC3912, September 2004, DOI 10.17487/RFC3912, September 2004,
<https://www.rfc-editor.org/info/rfc3912>. <https://www.rfc-editor.org/info/rfc3912>.
[RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifier (URI): Generic Syntax", STD 66,
RFC 3986, DOI 10.17487/RFC3986, January 2005,
<https://www.rfc-editor.org/info/rfc3986>.
[RFC4627] Crockford, D., "The application/json Media Type for [RFC4627] Crockford, D., "The application/json Media Type for
JavaScript Object Notation (JSON)", RFC 4627, JavaScript Object Notation (JSON)", RFC 4627,
DOI 10.17487/RFC4627, July 2006, DOI 10.17487/RFC4627, July 2006,
<https://www.rfc-editor.org/info/rfc4627>. <https://www.rfc-editor.org/info/rfc4627>.
[RFC3597] Gustafsson, A., "Handling of Unknown DNS Resource Record
(RR) Types", RFC 3597, DOI 10.17487/RFC3597, September
2003, <https://www.rfc-editor.org/info/rfc3597>.
[RFC6648] Saint-Andre, P., Crocker, D., and M. Nottingham, [RFC6648] Saint-Andre, P., Crocker, D., and M. Nottingham,
"Deprecating the "X-" Prefix and Similar Constructs in "Deprecating the "X-" Prefix and Similar Constructs in
Application Protocols", BCP 178, RFC 6648, Application Protocols", BCP 178, RFC 6648,
DOI 10.17487/RFC6648, June 2012, DOI 10.17487/RFC6648, June 2012,
<https://www.rfc-editor.org/info/rfc6648>. <https://www.rfc-editor.org/info/rfc6648>.
[RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 2234, DOI 10.17487/RFC2234,
November 1997, <https://www.rfc-editor.org/info/rfc2234>.
Dulaunoy, et al. Expires 28 February 2025 [Page 10]
Internet-Draft Passive DNS - Common Output Format August 2024
[RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J., [RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J.,
Morris, J., Hansen, M., and R. Smith, "Privacy Morris, J., Hansen, M., and R. Smith, "Privacy
Considerations for Internet Protocols", RFC 6973, Considerations for Internet Protocols", RFC 6973,
DOI 10.17487/RFC6973, July 2013, DOI 10.17487/RFC6973, July 2013,
<https://www.rfc-editor.org/info/rfc6973>. <https://www.rfc-editor.org/info/rfc6973>.
8.2. References [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform
Resource Identifier (URI): Generic Syntax", STD 66,
RFC 3986, DOI 10.17487/RFC3986, January 2005,
<https://www.rfc-editor.org/info/rfc3986>.
[BAILIWICK] [RFC7258] Farrell, S. and H. Tschofenig, "Pervasive Monitoring Is an
Edmonds, R., "Passive DNS Hardening", 2010, Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May
<https://archive.farsightsecurity.com/Passive_DNS/ 2014, <https://www.rfc-editor.org/info/rfc7258>.
passive_dns_hardening_handout.pdf>.
[CACHEPOISONING] [WEIMERPDNS]
Kaminsky, D., "Black ops 2008: It's the end of the cache Weimer, F., "Passive DNS Replication", 2005,
as we know it.", 2008, <http://www.enyo.de/fw/software/dnslogger/
<http://kurser.lobner.dk/dDist/DMK_BO2K8.pdf>. first2005-paper.pdf>.
[DNSDB] Security, F., "DNSDB API", 2013,
<https://api.dnsdb.info/>.
[DNSDBQ] Vixie, P., "DNSDB API Client, C Version", 2018,
<https://github.com/dnsdb/dnsdbq>.
Dulaunoy, et al. Expires 29 October 2024 [Page 10]
Internet-Draft Passive DNS - Common Output Format April 2024
[PDNSCOF] Dulaunoy, D. P. A., "Passive DNS server interface using
the common output format", 2019,
<https://github.com/D4-project/analyzer-d4-passivedns/>.
[github_issue_17] [github_issue_17]
et.al, P. V. W. A. K., "Discussion on the existing et.al, P. V. W. A. K., "Discussion on the existing
@ -568,6 +592,39 @@ Internet-Draft Passive DNS - Common Output Format April 2024
OR time_{first,last}", 2020, OR time_{first,last}", 2020,
<https://github.com/adulau/pdns-qof/issues/17>. <https://github.com/adulau/pdns-qof/issues/17>.
9. Informative References
[RFC7871] Contavalli, C., van der Gaast, W., Lawrence, D., and W.
Kumari, "Client Subnet in DNS Queries", RFC 7871,
DOI 10.17487/RFC7871, May 2016,
<https://www.rfc-editor.org/info/rfc7871>.
[BAILIWICK]
Edmonds, R., "Passive DNS Hardening", 2010,
<https://archive.farsightsecurity.com/Passive_DNS/
passive_dns_hardening_handout.pdf>.
[PDNSCLIENT]
Lee, C., "Queries 5 major Passive DNS databases: BFK,
CERTEE, DNSParse, ISC, and VirusTotal.", 2013,
<https://github.com/chrislee35/passivedns-client>.
Dulaunoy, et al. Expires 28 February 2025 [Page 11]
Internet-Draft Passive DNS - Common Output Format August 2024
[REST] Fielding, R. T., "Representational State Transfer (REST)",
2000, <http://www.ics.uci.edu/~fielding/pubs/dissertation/
rest_arch_style.htm>.
[DNSDB] Security, F., "DNSDB API", 2013,
<https://api.dnsdb.info/>.
[PDNSCERTAT] [PDNSCERTAT]
CERT.at, "pDNS presentation at 4th Centr R&D workshop CERT.at, "pDNS presentation at 4th Centr R&D workshop
Frankfurt Jun 5th 2012", 2012, Frankfurt Jun 5th 2012", 2012,
@ -578,25 +635,8 @@ Internet-Draft Passive DNS - Common Output Format April 2024
Luxembourg, C. -. I. R. C., "CIRCL Passive DNS", 2012, Luxembourg, C. -. I. R. C., "CIRCL Passive DNS", 2012,
<https://www.circl.lu/services/passive-dns/>. <https://www.circl.lu/services/passive-dns/>.
[PDNSCLIENT] [DNSDBQ] Vixie, P., "DNSDB API Client, C Version", 2018,
Lee, C., "Queries 5 major Passive DNS databases: BFK, <https://github.com/dnsdb/dnsdbq>.
CERTEE, DNSParse, ISC, and VirusTotal.", 2013,
<https://github.com/chrislee35/passivedns-client>.
[PDNSCOF] Dulaunoy, D. P. A., "Passive DNS server interface using
the common output format", 2019,
<https://github.com/D4-project/analyzer-d4-passivedns/>.
[REST] Fielding, R. T., "Representational State Transfer (REST)",
2000, <http://www.ics.uci.edu/~fielding/pubs/dissertation/
rest_arch_style.htm>.
[WEIMERPDNS]
Weimer, F., "Passive DNS Replication", 2005,
<http://www.enyo.de/fw/software/dnslogger/
first2005-paper.pdf>.
8.3. Informative References
Appendix A. Examples Appendix A. Examples
@ -606,18 +646,6 @@ Appendix A. Examples
If you query a passive DNS for the rrname www.ietf.org, the passive If you query a passive DNS for the rrname www.ietf.org, the passive
dns common output format can be: dns common output format can be:
Dulaunoy, et al. Expires 29 October 2024 [Page 11]
Internet-Draft Passive DNS - Common Output Format April 2024
{"count": 102, "time_first": 1298412391, "rrtype": "AAAA", {"count": 102, "time_first": 1298412391, "rrtype": "AAAA",
"rrname": "www.ietf.org", "rdata": "2001:1890:1112:1::20", "rrname": "www.ietf.org", "rdata": "2001:1890:1112:1::20",
"time_last": 1302506851} "time_last": 1302506851}
@ -625,9 +653,27 @@ Internet-Draft Passive DNS - Common Output Format April 2024
"rrname": "www.ietf.org", "rdata": "4.31.198.44", "rrname": "www.ietf.org", "rdata": "4.31.198.44",
"time_last": 1389022219} "time_last": 1389022219}
Figure 2
If you query a passive DNS for the rrname ietf.org, the passive dns If you query a passive DNS for the rrname ietf.org, the passive dns
common output format can be: common output format can be:
Dulaunoy, et al. Expires 28 February 2025 [Page 12]
Internet-Draft Passive DNS - Common Output Format August 2024
{"count": 109877, "time_first": 1298398002, "rrtype": "NS", {"count": 109877, "time_first": 1298398002, "rrtype": "NS",
"rrname": "ietf.org", "rdata": "ns1.yyz1.afilias-nst.info", "rrname": "ietf.org", "rdata": "ns1.yyz1.afilias-nst.info",
"time_last": 1389095375} "time_last": 1389095375}
@ -638,6 +684,8 @@ Internet-Draft Passive DNS - Common Output Format April 2024
"rrname": "ietf.org", "rdata": "2001:1890:123a::1:1e", "rrname": "ietf.org", "rdata": "2001:1890:123a::1:1e",
"time_last": 1330209752} "time_last": 1330209752}
Figure 3
Please note that the examples imply that a single query returns a Please note that the examples imply that a single query returns a
single set of JSON objects. For example, two queries were made; one single set of JSON objects. For example, two queries were made; one
query returned a set of two JSON objects and the other query returned query returned a set of two JSON objects and the other query returned
@ -655,7 +703,7 @@ Authors' Addresses
Alexandre Dulaunoy Alexandre Dulaunoy
CIRCL CIRCL
122, rue Adolphe Fischer 122, rue Adolphe Fischer
L-1521 Luxembourg L-L-1521 Luxembourg
Luxembourg Luxembourg
Phone: (+352) 247 88444 Phone: (+352) 247 88444
Email: alexandre.dulaunoy@circl.lu Email: alexandre.dulaunoy@circl.lu
@ -668,18 +716,20 @@ Authors' Addresses
Email: aaron@lo-res.org Email: aaron@lo-res.org
Dulaunoy, et al. Expires 29 October 2024 [Page 12]
Internet-Draft Passive DNS - Common Output Format April 2024
Paul Vixie Paul Vixie
Farsight Security, Inc. Farsight Security, Inc.
11400 La Honda Road 11400 La Honda Road
Woodside, California 94062 Woodside, California 94062
United States of America United States of America
Email: paul@redbarn.org Email: paul@redbarn.org
Dulaunoy, et al. Expires 28 February 2025 [Page 13]
Internet-Draft Passive DNS - Common Output Format August 2024
URI: https://www.farsightsecurity.com/ URI: https://www.farsightsecurity.com/
@ -725,4 +775,10 @@ Internet-Draft Passive DNS - Common Output Format April 2024
Dulaunoy, et al. Expires 29 October 2024 [Page 13]
Dulaunoy, et al. Expires 28 February 2025 [Page 14]

314
i-d/pdns-qof.xml
View file

@ -1,30 +1,38 @@
<?xml version="1.0" encoding="US-ASCII"?> <?xml version="1.0" encoding="utf-8"?>
<?xml-model href="rfc7991bis.rnc"?> <!-- Required for schema validation and schema-aware editing -->
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [ <!DOCTYPE rfc [
<!ENTITY RFC2119 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml">
<!ENTITY RFC2629 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2629.xml">
<!ENTITY RFC1035 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.1035.xml">
<!ENTITY RFC1034 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.1034.xml"> <!ENTITY RFC1034 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.1034.xml">
<!ENTITY RFC4627 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.4627.xml"> <!ENTITY RFC1035 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.1035.xml">
<!ENTITY RFC2119 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml">
<!ENTITY RFC2234 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2234.xml">
<!ENTITY RFC2629 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2629.xml">
<!ENTITY RFC3597 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3597.xml"> <!ENTITY RFC3597 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3597.xml">
<!ENTITY RFC3912 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3912.xml"> <!ENTITY RFC3912 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3912.xml">
<!ENTITY RFC6648 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6648.xml">
<!ENTITY RFC2234 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.2234.xml">
<!ENTITY RFC6973 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6973.xml">
<!ENTITY RFC3986 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3986.xml"> <!ENTITY RFC3986 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3986.xml">
<!ENTITY RFC4627 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.4627.xml">
<!ENTITY RFC6648 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6648.xml">
<!ENTITY RFC6973 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.6973.xml">
<!ENTITY RFC7258 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.7258.xml">
<!ENTITY RFC7871 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.7871.xml">
<!ENTITY I-D.narten-iana-considerations-rfc2434bis SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.narten-iana-considerations-rfc2434bis.xml"> <!ENTITY I-D.narten-iana-considerations-rfc2434bis SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.narten-iana-considerations-rfc2434bis.xml">
<!ENTITY I-D.draft-bortzmeyer-dnsop-dns-privacy SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.draft-bortzmeyer-dnsop-dns-privacy"> <!ENTITY I-D.draft-bortzmeyer-dnsop-dns-privacy SYSTEM "http://xml.resource.org/public/rfc/bibxml3/reference.I-D.draft-bortzmeyer-dnsop-dns-privacy">
]> ]>
<?xml-stylesheet type='text/xsl' href='rfc2629.xslt'?>
<?rfc strict="yes"?>
<?rfc toc="yes"?> <rfc
<?rfc tocdepth="4"?> xmlns:xi="http://www.w3.org/2001/XInclude"
<?rfc symrefs="yes"?> category="info"
<?rfc sortrefs="yes"?> docName="draft-dulaunoy-dnsop-passive-dns-cof-12"
<?rfc compact="yes"?> ipr="trust200902"
<?rfc subcompact="no"?> obsoletes=""
<rfc category="info" docName="draft-dulaunoy-dnsop-passive-dns-cof-12" ipr="trust200902"> updates=""
submissionType="IETF"
xml:lang="en"
version="3">
<!-- ***** FRONT MATTER ***** --> <!-- ***** FRONT MATTER ***** -->
<front> <front>
<title abbrev="Passive DNS - Common Output Format">Passive DNS - Common Output Format</title> <title abbrev="Passive DNS - Common Output Format">Passive DNS - Common Output Format</title>
@ -97,12 +105,12 @@
</address> </address>
</author> </author>
<date day="5" month="June" year="2024" /> <date day="27" month="August" year="2024" />
<area>General</area> <area>General</area>
<workgroup>Domain Name System Operations</workgroup> <workgroup>Domain Name System Operations</workgroup>
<keyword>dns</keyword> <keyword>dns</keyword>
<abstract> <abstract>
<t>This document describes a common output format of Passive DNS Servers that clients can <t>This document describes a common output format of Passive DNS servers that clients can
query. The output format description also includes a common semantic for each Passive DNS query. The output format description also includes a common semantic for each Passive DNS
system. By having multiple Passive DNS Systems adhere to the same output format for queries, system. By having multiple Passive DNS Systems adhere to the same output format for queries,
users of multiple Passive DNS servers will be able to combine result sets easily.</t> users of multiple Passive DNS servers will be able to combine result sets easily.</t>
@ -111,29 +119,63 @@
<middle> <middle>
<section title="Introduction"> <section title="Introduction">
<t>Passive DNS is a technique described by Florian Weimer in 2005 in <xref target="WEIMERPDNS">Passive <t>Passive DNS is a technique described by Florian Weimer in 2005 in <xref target="WEIMERPDNS">Passive
DNS replication, F Weimer - 17th Annual FIRST Conference on Computer Security</xref>. Since DNS replication, F Weimer - 17th Annual FIRST Conference on Computer Security</xref>.
then, multiple Passive DNS implementations were created and have evolved over time. Users of It is a mechanism for
these Passive DNS servers may query a server (often via <xref target="RFC3912">WHOIS</xref> logging DNS answers in a manner intended to minimize the privacy
or HTTP <xref target="REST">REST</xref>), parse the results, and process them in other implications to users, and is widely by security researchers to investigate
applications.</t> malware (for example to discover command and control servers), and other
security threats. By capturing only the "cache fill" DNS responses
(responses from authoritative servers in response to queries performed by a
recursive resolver when iteratively resolving a name), Passive DNS does
not have access to the client (users) source IP, source port, destination
IP, or destination port.</t>
<t> There are multiple implementations of Passive DNS software. Users of Passive DNS query <t>As these answers are served in response to queries originally
each implementation and aggregate the results for their search. This document describes the initiated by user devices, the Passive DNS data can be used to detect if
output format of four Passive DNS Systems (<xref target="DNSDB" />, <xref target="DNSDBQ" /> devices using the resolver are connecting to known malicious domains,
, <xref target="PDNSCERTAT" />, <xref target="PDNSCIRCL" /> and <xref target="PDNSCOF" />) without identifying the individual users / devices. In addition, as
that are in use today and that already share a nearly identical output format. As the format answers are responses to queries made by the recursive server itself,
and the meaning of output fields from each Passive DNS need to be consistent, this document Passive DNS records the answers which are ultimately served to users.
proposes a solution to commonly name each field along with its corresponding interpretation. This is important as authoritative servers may serve different answers to
The format follows a simple key-value structure in <xref target="RFC4627">JSON</xref> different query addresses, for example to increase performance (e.g <xref
format. The benefit of having a consistent Passive DNS output format is that multiple client target="RFC7871">Client Subnet in DNS Queries</xref>) or to hide
implementations can query different servers without having to have a separate parser for malicious behavior when queried from addresses known to be associated
each individual server. <xref target="PDNSCLIENT">passivedns-client</xref> currently with security researchers.</t>
implements multiple parsers due to a lack of standardization. The document does not describe
the protocol (e.g. <xref target="RFC3912">WHOIS</xref>, HTTP <xref target="REST">REST</xref>) <t>Passive DNS is usually implemented either by capturing DNS response
nor the query format used to query the Passive DNS. Neither does this document describe packets themselves (i.e packets with a destination address of the
"pre-recursor" Passive DNS Systems. Each of these are separate topics and deserve their own recursive resolver, a source port of 53, and the QR bit set to 1) or
RFC documents. This document describes the current best practices implemented in various by having the DNS software itself log these responses. The latter method
Passive DNS server implementations. </t> is likely to become more common as recursive to authoritative DNS
communication becomes encrypted.
</t>
<t>Multiple Passive DNS implementations and services exist. Users of
these Passive DNS services may query a server (often via <xref
target="RFC3912">WHOIS</xref>
or HTTP <xref target="REST">REST</xref>), parse the results, and process
them in other applications. Users of Passive DNS query each
implementation and aggregate the results for their search. This document
describes the output format of four Passive DNS Systems (<xref
target="DNSDB" />, <xref target="DNSDBQ" /> , <xref target="PDNSCERTAT"
/>, <xref target="PDNSCIRCL" /> and <xref target="PDNSCOF" />) that are
in use today and that already share a nearly identical output format. As
the format and the meaning of output fields from each Passive DNS need to
be consistent, this document proposes a solution to commonly name each
field along with its corresponding interpretation. The format follows a
simple key-value structure in <xref target="RFC4627">JSON</xref>
format. The benefit of having a consistent Passive DNS output format is
that multiple client implementations can query different servers
without having to have a separate parser for each individual server.
<xref target="PDNSCLIENT">passivedns-client</xref> currently implements
multiple parsers due to a lack of standardization. The document does
not describe the protocol (e.g. <xref target="RFC3912">WHOIS</xref>,
HTTP <xref target="REST">REST</xref>) nor the query format used to
query the Passive DNS. Neither does this document describe
"pre-recursor" Passive DNS Systems. Each of these are separate topics
and deserve their own RFC documents. This document describes the
current best practices implemented in various Passive DNS server
implementations. </t>
<section title="Requirements Language"> <section title="Requirements Language">
<t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD
@ -142,15 +184,15 @@
</section> </section>
</section> </section>
<section title="Limitation"> <section title="Limitations">
<t> As Passive DNS servers can include protection mechanisms for their operation, results <t> As Passive DNS servers can include protection mechanisms for their operation, results
might be different due to those protection measures. These mechanisms filter out DNS answers might be different due to those protection measures. These mechanisms filter out DNS answers
if they fail some criteria. The <xref target="BAILIWICK">bailiwick algorithm</xref> protects if they fail some criteria. The <xref target="BAILIWICK">bailiwick algorithm</xref> protects
the Passive DNS Database from <xref target="CACHEPOISONING">cache poisoning attacks</xref>. the Passive DNS Database from cache poisoning attacks.
Another limitation that clients querying the database need to be aware of is that each query Another limitation that clients querying the database need to be aware of is that each query
simply gets a snapshot-in-time answer at the time of querying. Clients MUST NOT rely on simply gets a snapshot-in-time answer at the time of querying. Clients MUST NOT rely on
existing answers from different Passive DNS database. Nor should they assume that answers existing answers from different Passive DNS database. Nor should they assume that answers
will be identical across multiple Passive DNS Servers. </t> will be identical across multiple Passive DNS servers. </t>
</section> </section>
<section title="Common Output Format"> <section title="Common Output Format">
@ -168,8 +210,9 @@
</section> </section>
<section title="ABNF grammar"> <section title="ABNF grammar">
<!-- "preamble" is deprecated in V3 -->
<t>Formal grammar as defined in <xref target="RFC2234">ABNF</xref></t>
<figure> <figure>
<preamble>Formal grammar as defined in <xref target="RFC2234">ABNF</xref></preamble>
<artwork><![CDATA[ <artwork><![CDATA[
answer = entries answer = entries
entries = * ( entry newline ) entries = * ( entry newline )
@ -266,7 +309,7 @@ ws = *(
<section title="Optional Fields"> <section title="Optional Fields">
<t>Implementations SHOULD support one or more fields.</t> <t>Implementations SHOULD support one or more fields.</t>
<section title="count"> <section title="count">
<t>Specifies how many authoritative DNS answers were received at the Passive DNS Server's <t>Specifies how many authoritative DNS answers were received at the Passive DNS server's
collectors with exactly the given set of values as answers (i.e. same data in the answer collectors with exactly the given set of values as answers (i.e. same data in the answer
set - compare with the uniqueness property in "Mandatory Fields"). The number of set - compare with the uniqueness property in "Mandatory Fields"). The number of
requests is expressed as a decimal value. This field is represented as a <xref requests is expressed as a decimal value. This field is represented as a <xref
@ -328,7 +371,7 @@ ws = *(
</section> </section>
<section title="Additional notes"> <section title="Additional notes">
<t>An implementer of a passive DNS Server MAY chose to either return time_first and <t>An implementer of a passive DNS server MAY chose to either return time_first and
time_last OR return zone_time_first and zone_time_last. In pseudocode: (time_first AND time_last OR return zone_time_first and zone_time_last. In pseudocode: (time_first AND
time_last) OR (zone_time_first AND zone_time_last). In this case, zone_time_{first,last} time_last) OR (zone_time_first AND zone_time_last). In this case, zone_time_{first,last}
replace the time_{first,last} fields. However, this is not encouraged since it might be replace the time_{first,last} fields. However, this is not encouraged since it might be
@ -337,13 +380,12 @@ ws = *(
</section> </section>
<section title="Suggested MIME Types"> <section title="Suggested MIME Types">
<t>An implementer of a passive DNS Server SHOULD serve a document in this Common Output <t>An implementer of a passive DNS server SHOULD serve a document in this Common Output
Format with a MIME header of "application/x-ndjson".</t> Format with a MIME header of "application/x-ndjson".</t>
</section> </section>
</section> </section>
<!-- This PI places the pagebreak correctly (before the section title) in the text output. --> <!-- This PI places the pagebreak correctly (before the section title) in the text output. -->
<?rfc needLines="8"?> <?rfc needLines="8"?>
<section anchor="Acknowledgements" title="Acknowledgements"> <section anchor="Acknowledgements" title="Acknowledgements">
@ -355,39 +397,53 @@ ws = *(
</section> </section>
<section anchor="Privacy" title="Privacy Considerations"> <section anchor="Privacy" title="Privacy Considerations">
<t>Passive DNS Servers capture DNS answers from multiple collection points ("sensors") which <t>Passive DNS servers capture DNS answers from multiple collection points ("sensors") which
are located on the Internet-facing side of DNS recursors ("post-recursor passive DNS"). In are located on the Internet-facing side of DNS recursors ("post-recursor passive DNS"). In
this process, they intentionally omit the source IP, source port, destination IP and this process, they intentionally omit the source IP, source port, destination IP and
destination port from the captured packets. Since the data is captured "post-recursor", the destination port from the captured packets. Since the data is captured "post-recursor", the
timing information (who queries what) is lost, since the recursor will cache the results. timing information (who queries what) is lost, since the recursor will cache the results.
Furthermore, since multiple sensors feed into a passive DNS server, the resulting data gets Furthermore, since multiple sensors feed into a passive DNS system, the resulting data gets
mixed together, reducing the likelihood that Passive DNS Servers are able to find out much mixed together, reducing the likelihood that Passive DNS systems are able to find out much
about the actual person querying the DNS records. In this sense, passive DNS Servers are about the actual person querying the DNS records. In this sense, passive DNS systems are
similar to keeping an archive of all previous phone books - if public DNS records can be similar to keeping an archive of all previous phone books - if public DNS records can be
compared to phone numbers - as they often are. Nevertheless, the authors strongly encourage compared to phone numbers - as they often are. Nevertheless, the authors strongly encourage
Passive DNS implementors to take special care of privacy issues. Passive DNS implementors to take special care of privacy issues. Finally, the overall
bortzmeyer-dnsop-dns-privacy is an excellent starting point for this. Finally, the overall
recommendations in <xref target="RFC6973">RFC6973</xref> should be taken into consideration recommendations in <xref target="RFC6973">RFC6973</xref> should be taken into consideration
when designing any application which uses Passive DNS data.</t> when designing any application which uses Passive DNS data.</t>
<t>Passive DNS attempts to collect information necessary for security (such as malware protection)
in as privacy protecting a manner as possible, and is intended to be
used instead of more invasive methods. It does this by only collecting
DNS cache-fill answers, and not any information associated with who caused the
name to be resolved, nor why the name was resolved. Nevertheless, it is possible that
this may still lead to privacy concerns - for example, if Passive DNS records show that
a recursive resolver resolved the name the-mary-and-john-smith-family.example.com, it may be
possible to infer that the Smith family is using that resolver. Operators of Passive DNS
servers should be aware of this and take appropriate steps to limit access to the data.</t>
<t>Passive DNS operators are encouraged to read and understand
<xref target="RFC7258">RFC7258</xref> </t>
<t>In the scope of the General Data Protection Regulation (GDPR - Directive 95/46/EC), <t>In the scope of the General Data Protection Regulation (GDPR - Directive 95/46/EC),
operators of Passive DNS Server needs to ensure the legal ground and lawfulness of its operators of Passive DNS server needs to ensure the legal ground and lawfulness of its
operation.</t> operation.</t>
</section> </section>
<section anchor="Security" title="Security Considerations"> <section anchor="Security" title="Security Considerations">
<t>In some cases, Passive DNS output might contain confidential information and its access <t>In some cases, Passive DNS output might contain confidential information and its access
might be restricted. When a user is querying multiple Passive DNS and aggregating the data, should be restricted. When a user is querying multiple Passive DNS and aggregating the data,
the sensitivity of the data must be considered.</t> the sensitivity of the data must be considered.</t>
</section> </section>
</middle> </middle>
<!-- *****BACK MATTER ***** --> <!-- *****BACK MATTER ***** -->
<back> <back>
<references>
<name>Normative References</name>
&RFC2119; &RFC1035; &RFC1034; &RFC3912; &RFC4627;
&RFC3597; &RFC6648; &RFC2234; &RFC6973; &RFC3986;
&RFC7258;
<references title="Normative References"><!--?rfc
include="http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml"?--> &RFC2119; &RFC1035; &RFC1034; &RFC3912; &RFC4627;
&RFC3597; &RFC6648; &RFC2234; &RFC6973; &RFC3986; </references>
<references>
<reference anchor="WEIMERPDNS" <reference anchor="WEIMERPDNS"
target="http://www.enyo.de/fw/software/dnslogger/first2005-paper.pdf"> target="http://www.enyo.de/fw/software/dnslogger/first2005-paper.pdf">
<front> <front>
@ -397,65 +453,6 @@ ws = *(
</front> </front>
</reference> </reference>
<reference anchor="CACHEPOISONING" target="http://kurser.lobner.dk/dDist/DMK_BO2K8.pdf">
<front>
<title>Black ops 2008: It's the end of the cache as we know it.</title>
<author fullname="Dan Kaminsky" />
<date year="2008" />
</front>
</reference>
<reference anchor="BAILIWICK"
target="https://archive.farsightsecurity.com/Passive_DNS/passive_dns_hardening_handout.pdf">
<front>
<title>Passive DNS Hardening</title>
<author fullname="Robert Edmonds" />
<date year="2010" />
</front>
</reference>
<reference anchor="PDNSCLIENT" target="https://github.com/chrislee35/passivedns-client">
<front>
<title>Queries 5 major Passive DNS databases: BFK, CERTEE, DNSParse, ISC, and VirusTotal.</title>
<author fullname="Chris Lee" />
<date year="2013" />
</front>
</reference>
<reference anchor="REST"
target="http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm">
<front>
<title>Representational State Transfer (REST)</title>
<author fullname="Roy Thomas Fielding" />
<date year="2000" />
</front>
</reference>
<reference anchor="DNSDB" target="https://api.dnsdb.info/">
<front>
<title>DNSDB API</title>
<author fullname="Farsight Security" />
<date year="2013" />
</front>
</reference>
<reference anchor="PDNSCERTAT"
target="http://www.centr.org/system/files/agenda/attachment/d4-papst-passive_dns.pdf">
<front>
<title>pDNS presentation at 4th Centr R&amp;D workshop Frankfurt Jun 5th 2012</title>
<author fullname="CERT.at" />
<date year="2012" />
</front>
</reference>
<reference anchor="PDNSCIRCL" target="https://www.circl.lu/services/passive-dns/">
<front>
<title>CIRCL Passive DNS</title>
<author fullname="CIRCL -Computer Incident Response Center Luxembourg" />
<date year="2012" />
</front>
</reference>
<reference anchor="PDNSCOF" target="https://github.com/D4-project/analyzer-d4-passivedns/"> <reference anchor="PDNSCOF" target="https://github.com/D4-project/analyzer-d4-passivedns/">
<front> <front>
<title>Passive DNS server interface using the common output format</title> <title>Passive DNS server interface using the common output format</title>
@ -464,14 +461,6 @@ ws = *(
</front> </front>
</reference> </reference>
<reference anchor="DNSDBQ" target="https://github.com/dnsdb/dnsdbq">
<front>
<title>DNSDB API Client, C Version</title>
<author fullname="Paul Vixie" />
<date year="2018" />
</front>
</reference>
<reference anchor="github_issue_17" target="https://github.com/adulau/pdns-qof/issues/17"> <reference anchor="github_issue_17" target="https://github.com/adulau/pdns-qof/issues/17">
<front> <front>
<title>Discussion on the existing implementations of returning either <title>Discussion on the existing implementations of returning either
@ -482,11 +471,68 @@ ws = *(
</reference> </reference>
</references> </references>
<references>
<name>Informative References</name>
&RFC7871;
<references title="Informative References"> <reference anchor="BAILIWICK"
<!-- Here we use entities that we defined at the beginning. --> target="https://archive.farsightsecurity.com/Passive_DNS/passive_dns_hardening_handout.pdf">
<!-- &I-D.narten-iana-considerations-rfc2434bis; --> <front>
<!-- &I-D.draft-bortzmeyer-dnsop-dns-privacy; --> <title>Passive DNS Hardening</title>
<author fullname="Robert Edmonds" />
<date year="2010" />
</front>
</reference>
<reference anchor="PDNSCLIENT" target="https://github.com/chrislee35/passivedns-client">
<front>
<title>Queries 5 major Passive DNS databases: BFK, CERTEE, DNSParse, ISC, and VirusTotal.</title>
<author fullname="Chris Lee" />
<date year="2013" />
</front>
</reference>
<reference anchor="REST"
target="http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm">
<front>
<title>Representational State Transfer (REST)</title>
<author fullname="Roy Thomas Fielding" />
<date year="2000" />
</front>
</reference>
<reference anchor="DNSDB" target="https://api.dnsdb.info/">
<front>
<title>DNSDB API</title>
<author fullname="Farsight Security" />
<date year="2013" />
</front>
</reference>
<reference anchor="PDNSCERTAT"
target="http://www.centr.org/system/files/agenda/attachment/d4-papst-passive_dns.pdf">
<front>
<title>pDNS presentation at 4th Centr R&amp;D workshop Frankfurt Jun 5th 2012</title>
<author fullname="CERT.at" />
<date year="2012" />
</front>
</reference>
<reference anchor="PDNSCIRCL" target="https://www.circl.lu/services/passive-dns/">
<front>
<title>CIRCL Passive DNS</title>
<author fullname="CIRCL -Computer Incident Response Center Luxembourg" />
<date year="2012" />
</front>
</reference>
<reference anchor="DNSDBQ" target="https://github.com/dnsdb/dnsdbq">
<front>
<title>DNSDB API Client, C Version</title>
<author fullname="Paul Vixie" />
<date year="2018" />
</front>
</reference>
</references> </references>
<section anchor="app-additional" title="Examples"> <section anchor="app-additional" title="Examples">