adulau/hashlookup-server
1
0
Fork 0
mirror of https://github.com/adulau/hashlookup-server.git synced 2024-11-21 17:47:06 +00:00
Code Issues Projects Releases Packages Wiki Activity
64 commits 1 branch 2 tags 124 KiB
Commit graph

37 commits

Author SHA1 Message Date
Alexandre Dulaunoy
baf2b82f7b
new: [server] /children and /parents end-points added 
The two new endpoints `children` and `parents` allow to paginate over the large-set of parents or
children.

- The first value is the SHA1 value having children or parents.
- The second value is the number of elements to get (by default is 100 if
the value is set to 0).
- The third value is the cursor to paginate over the element (for
starting the cursor must be set to 0).

A sample usage:

~~~~
adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/0  | jq .
{
  "children": [
    "003587440172055C75130EF1A063C3BB050C3251",
    "007C1E16B3F0F2E48C114E458308397953C7D224",
    "014D1060C674FBBCEAFFD94B85D60AD00618B56B",
    "01A2FACD61D157FC80DD0C5F6B525CC9EDE4B6DE",
    "01D1A98F559966A05923A74EE239C6BBEEB0FDAC",
    "01D381F2FCDD1BDF642AF83C9E96083F2C8D1C03",
    "02B37BA21D1831C120C1C9C1D41893B4DB424EE7",
    "02DED521ADCF17AA8818EA1142F63E05F558E668",
    "0364E0EFE65D9B6502084813189B4D888C117859",
    "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E"
  ],
  "cursor": "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E",
  "total": 774
}
adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E  | jq .
{
  "children": [
    "063EC5526DA21372D77AFC3C40F694478521829B",
    "0647EA948ED37383F74CC68A94E2DC3CBC2A9E4E",
    "0648AAAC06A76A58CB1E999882447BBDEEA42C57",
    "06A62F10F269824FFD75A917A35ACD3F2461981C",
    "0727FE9E2437B15B3F879C7617973AE11E55BA13",
    "074A0CA7131AE8FD9665CFE68A0C124EB6AD0170",
    "075B11AE383071BDA9BE66E336C916F6E6E1F49C",
    "081A336DE7D636F95F0150B7708C614592CBBDAE",
    "08DF546EE44D4B7546FCE5A7B7E284CA35F1B059",
    "0947CE713B69C2318CA684BBB63912621CC17A6A"
  ],
  "cursor": "0947CE713B69C2318CA684BBB63912621CC17A6A",
  "total": 774
}
~~~~
 2022-05-21 17:43:24 +02:00
Alexandre Dulaunoy
74efc57b3d
chg: [kvrocks] update the kvrocks install script 2022-05-21 16:57:06 +02:00
Alexandre Dulaunoy
be67766d01
chg: [server] black -S all the code 2022-01-15 14:56:47 +01:00
Alexandre Dulaunoy
b0d11047bf
chg: [server] update server description 2022-01-15 14:55:09 +01:00
Alexandre Dulaunoy
add29e6ca3
new: [server/api] improved children handling - if too many children are 
returned it's stripped - and a sample is extracted
 2021-12-04 13:59:19 +01:00
Alexandre Dulaunoy
0357542bbd
chg: [import-poc] never flush 2021-12-03 23:40:25 +01:00
Alexandre Dulaunoy
a928c4e51e
new: [api] hashlookup:trust added in the output 
The trust level is calculated based on the number of parent to the file.
If the file has been seen on many sources, the trust level increase.

The scale of the trust level is between 0 and 100. By default, the trust
level is 50 meaning we don't know the trust. Below 50, the file is
suspicious. Above 50, we have evidences that the file is more
legitimate. The calculation is based on the number of parents seen per
file. If a file is seen more often in various sources, it increases the
trust level to reach a maximum of 100.
 2021-12-02 07:33:20 +01:00
Alexandre Dulaunoy
d8ac479d72
chg: [api] expose the total keys of a hashlookup server 2021-11-19 10:26:00 +01:00
Alexandre Dulaunoy
af5552ef75
new: [api] /lookup/sha256 api endpoint added 2021-11-19 07:26:50 +01:00
Alexandre Dulaunoy
a78a99ec44
new: [api] add a hashlookup:parent-total which indicates the cardinality of the parents 
This can be used for the new API endpoint to paginate over large set of
parents.
 2021-10-31 09:04:25 +01:00
Alexandre Dulaunoy
9e79d8ecc7
chg: [server] add auth header in pub-sub 2021-10-25 21:39:38 +02:00
Alexandre Dulaunoy
15e6f461d4
fix: [api/stats] existing hash value from nx removed 2021-09-11 00:02:34 +02:00
Alexandre Dulaunoy
d6fe136421
chg: [api/info] now return the default stats from the new importer 
TODO: add the diff with additional sources and key numbers
 2021-09-09 07:34:37 +02:00
Alexandre Dulaunoy
ac7ec00c97
chg: [config] don't flush db by default 2021-09-06 16:21:34 +02:00
Alexandre Dulaunoy
aa58ad8152
chg: [stats/top] remove recently added hashes from previously nx hash 2021-09-05 21:59:20 +02:00
Alexandre Dulaunoy
6461b91d55
fix: [api/lookup/sha1] missing parents bug 2021-09-05 07:36:44 +02:00
Alexandre Dulaunoy
c470201fd8
fix: [api:md5/sha1] large set of parents are now limited and give a random selection 
TODO: mainly empty files and similar - warning-lists should be added
 2021-09-01 21:36:39 +02:00
Alexandre Dulaunoy
74c0e8c8c2
fix: [api/md5] if there is already more data in default SHA1 we use that one and not the fall-back MD5 lookup 2021-09-01 19:56:50 +02:00
Alexandre Dulaunoy
fe04f1e7dd
chg: [api/md5] data sources with MD5 only hashes 2021-09-01 19:37:21 +02:00
Alexandre Dulaunoy
bb29fcb25c
chg: [api] md5 lookup updated to allow MD5 only records 2021-09-01 19:26:58 +02:00
Alexandre Dulaunoy
c17dbff6a6
new: [api:stats/top] Add a new optional entry to point to get the top 100 of most queried hashes (existing and non-existing) 2021-08-29 14:06:35 +02:00
Alexandre Dulaunoy
4419052c4f
fix: [api:bulk] add proper check of MD5 and SHA1 value before further processing 2021-08-29 12:25:43 +02:00
Alexandre Dulaunoy
a14e5aedf1
new: [api:bulk] add support for pub-sub channel of existing and 
non-existing hashes
 2021-08-29 11:52:07 +02:00
Alexandre Dulaunoy
a8b951cf83
chg: [server] add children hashes if these exist 2021-08-24 14:53:43 +02:00
Alexandre Dulaunoy
02a675d9ea
fix: [api] typo fixed 2021-08-22 23:37:05 +02:00
Alexandre Dulaunoy
a17cd38595
chg: [api] lookup add parent details 2021-08-22 23:23:52 +02:00
Alexandre Dulaunoy
27aa7c4034
fix: [api] fix ttl missing bug 2021-08-22 16:48:06 +02:00
Alexandre Dulaunoy
284e4719c7
new: [feature] session handling added 
A user can now create a session, assign lookup results to a session
and retrieve the lookup session results in one shot.

This partially implement feature requested in issue #2 to support
DFIR sessions.

Thanks to Koen Van Impe for the idea.
 2021-08-14 10:32:00 +02:00
Alexandre Dulaunoy
31ac93e764
new: [pub-sub] add a pub-sub functionality for searched hashes in two 
different channels:
- nx: non-existing hash value
- exist: existing hash value
 2021-08-13 22:42:41 +02:00
Alexandre Dulaunoy
72b462b5ea
new: [statistics] add an optional statistic option in the server to have 
a sorted set of hashes matching and non-matching.
 2021-08-13 22:13:25 +02:00
Alexandre Dulaunoy
fc11f3dd88
chg: [poc-import] JSON updated to add legacy and JSON file jqified 2021-07-29 11:58:25 +02:00
Alexandre Dulaunoy
ef60dd73a9
chg: [requirements] remove standard lib python modules 2021-07-29 11:55:26 +02:00
Koen Van Impe
ca910ff22f Support for import of NSRL datasets in ISO and ZIP format 2021-07-29 10:46:59 +02:00
Koen Van Impe
d280216361 Merge branch 'main' of https://github.com/cudeso/hashlookup-server into main 2021-07-27 18:37:16 +02:00
Koen Van Impe
0edb33ed82 PoC for streamlining import 
PoC to better streamline the import of NSRL data.
Still requires some work but basic concept works.
Currently only tested with Android
 2021-07-27 18:37:12 +02:00
Alexandre Dulaunoy
5932226ece
chg: [install] kvrocks default repo and latest release 2021-07-17 07:59:05 +02:00
Alexandre Dulaunoy
a4b3b7ba60
new: [hashlookup-server] initial import of the code 
- This includes a simple HTTP server for doing bulk and lookup of hashes.
- A simple DNS server to do lookup via DNS
- Various import script for NSRL

This works on a test instance.

TODO:

- Automatic script for NSRL download and import
- Bloomfilter export
- Improved documentation
 2021-07-15 17:49:52 +02:00