adulau/hashlookup-server
1
0
Fork 0
mirror of https://github.com/adulau/hashlookup-server.git synced 2024-11-21 17:47:06 +00:00
Code Issues Projects Releases Packages Wiki Activity
64 commits 1 branch 2 tags 124 KiB
Commit graph

64 commits

This branch
This branch
All branches
Author SHA1 Message Date
Alexandre Dulaunoy
2d168822c9
Set theme jekyll-theme-cayman 2022-05-21 18:07:56 +02:00
Alexandre Dulaunoy
38f8f39ac2
chg: [doc] updated 2022-05-21 18:06:43 +02:00
Alexandre Dulaunoy
fba0d795d0
chg: [doc] API - add /children and /parents example 2022-05-21 17:59:23 +02:00
Alexandre Dulaunoy
baf2b82f7b
new: [server] /children and /parents end-points added 
The two new endpoints `children` and `parents` allow to paginate over the large-set of parents or
children.

- The first value is the SHA1 value having children or parents.
- The second value is the number of elements to get (by default is 100 if
the value is set to 0).
- The third value is the cursor to paginate over the element (for
starting the cursor must be set to 0).

A sample usage:

~~~~
adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/0  | jq .
{
  "children": [
    "003587440172055C75130EF1A063C3BB050C3251",
    "007C1E16B3F0F2E48C114E458308397953C7D224",
    "014D1060C674FBBCEAFFD94B85D60AD00618B56B",
    "01A2FACD61D157FC80DD0C5F6B525CC9EDE4B6DE",
    "01D1A98F559966A05923A74EE239C6BBEEB0FDAC",
    "01D381F2FCDD1BDF642AF83C9E96083F2C8D1C03",
    "02B37BA21D1831C120C1C9C1D41893B4DB424EE7",
    "02DED521ADCF17AA8818EA1142F63E05F558E668",
    "0364E0EFE65D9B6502084813189B4D888C117859",
    "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E"
  ],
  "cursor": "05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E",
  "total": 774
}
adulau@kolmogorov ~ $ curl -s http://127.0.0.1:5000/children/31C43D24d696BC5F5309CCBFA5BDEF65A7170439/10/05C9A276A0E03F7A5F99DE5CC8911583FD8FD60E  | jq .
{
  "children": [
    "063EC5526DA21372D77AFC3C40F694478521829B",
    "0647EA948ED37383F74CC68A94E2DC3CBC2A9E4E",
    "0648AAAC06A76A58CB1E999882447BBDEEA42C57",
    "06A62F10F269824FFD75A917A35ACD3F2461981C",
    "0727FE9E2437B15B3F879C7617973AE11E55BA13",
    "074A0CA7131AE8FD9665CFE68A0C124EB6AD0170",
    "075B11AE383071BDA9BE66E336C916F6E6E1F49C",
    "081A336DE7D636F95F0150B7708C614592CBBDAE",
    "08DF546EE44D4B7546FCE5A7B7E284CA35F1B059",
    "0947CE713B69C2318CA684BBB63912621CC17A6A"
  ],
  "cursor": "0947CE713B69C2318CA684BBB63912621CC17A6A",
  "total": 774
}
~~~~
 2022-05-21 17:43:24 +02:00
Alexandre Dulaunoy
74efc57b3d
chg: [kvrocks] update the kvrocks install script 2022-05-21 16:57:06 +02:00
Alexandre Dulaunoy
439dd83fbf
chg: [requirements] to the hashlookup version of redis-py 
as hashlookup server relies on kvrocks. The `scan` command works
a bit different which causes some issues with the original scan.
Now we use the fork which supports properly the `scan` and the
specific cursor name given by kvrocks.

ref: https://github.com/apache/incubator-kvrocks/discussions/417
 2022-05-21 10:17:58 +02:00
Alexandre Dulaunoy
7beed8849c
chg: [dns] update stats 2022-04-11 17:25:57 +02:00
Alexandre Dulaunoy
dbb5fc918e Set theme jekyll-theme-minimal 2022-01-15 15:12:12 +01:00
Alexandre Dulaunoy
be67766d01
chg: [server] black -S all the code 2022-01-15 14:56:47 +01:00
Alexandre Dulaunoy
b0d11047bf
chg: [server] update server description 2022-01-15 14:55:09 +01:00
Alexandre Dulaunoy
add29e6ca3
new: [server/api] improved children handling - if too many children are 
returned it's stripped - and a sample is extracted
 2021-12-04 13:59:19 +01:00
Alexandre Dulaunoy
0357542bbd
chg: [import-poc] never flush 2021-12-03 23:40:25 +01:00
Alexandre Dulaunoy
a928c4e51e
new: [api] hashlookup:trust added in the output 
The trust level is calculated based on the number of parent to the file.
If the file has been seen on many sources, the trust level increase.

The scale of the trust level is between 0 and 100. By default, the trust
level is 50 meaning we don't know the trust. Below 50, the file is
suspicious. Above 50, we have evidences that the file is more
legitimate. The calculation is based on the number of parents seen per
file. If a file is seen more often in various sources, it increases the
trust level to reach a maximum of 100.
 2021-12-02 07:33:20 +01:00
Alexandre Dulaunoy
d8ac479d72
chg: [api] expose the total keys of a hashlookup server 2021-11-19 10:26:00 +01:00
Alexandre Dulaunoy
af5552ef75
new: [api] /lookup/sha256 api endpoint added 2021-11-19 07:26:50 +01:00
Alexandre Dulaunoy
a78a99ec44
new: [api] add a hashlookup:parent-total which indicates the cardinality of the parents 
This can be used for the new API endpoint to paginate over large set of
parents.
 2021-10-31 09:04:25 +01:00
Alexandre Dulaunoy
9e79d8ecc7
chg: [server] add auth header in pub-sub 2021-10-25 21:39:38 +02:00
Alexandre Dulaunoy
b4c539114f
Merge branch 'main' of github.com:adulau/hashlookup-server into main 2021-09-24 08:17:09 +02:00
Alexandre Dulaunoy
7c174204f7
fix: [dns] records are now too large for single hashlookup record 
reduce to some fields and the HTTP interface the fall-back to get more
info
 2021-09-24 08:16:08 +02:00
Alexandre Dulaunoy
15e6f461d4
fix: [api/stats] existing hash value from nx removed 2021-09-11 00:02:34 +02:00
Alexandre Dulaunoy
d6fe136421
chg: [api/info] now return the default stats from the new importer 
TODO: add the diff with additional sources and key numbers
 2021-09-09 07:34:37 +02:00
Alexandre Dulaunoy
ac7ec00c97
chg: [config] don't flush db by default 2021-09-06 16:21:34 +02:00
Alexandre Dulaunoy
aa58ad8152
chg: [stats/top] remove recently added hashes from previously nx hash 2021-09-05 21:59:20 +02:00
Alexandre Dulaunoy
6461b91d55
fix: [api/lookup/sha1] missing parents bug 2021-09-05 07:36:44 +02:00
Alexandre Dulaunoy
c470201fd8
fix: [api:md5/sha1] large set of parents are now limited and give a random selection 
TODO: mainly empty files and similar - warning-lists should be added
 2021-09-01 21:36:39 +02:00
Alexandre Dulaunoy
74c0e8c8c2
fix: [api/md5] if there is already more data in default SHA1 we use that one and not the fall-back MD5 lookup 2021-09-01 19:56:50 +02:00
Alexandre Dulaunoy
fe04f1e7dd
chg: [api/md5] data sources with MD5 only hashes 2021-09-01 19:37:21 +02:00
Alexandre Dulaunoy
5f93d3e7ab
Merge branch 'main' of github.com:adulau/hashlookup-server into main 2021-09-01 19:27:27 +02:00
Alexandre Dulaunoy
bb29fcb25c
chg: [api] md5 lookup updated to allow MD5 only records 2021-09-01 19:26:58 +02:00
Alexandre Dulaunoy
12b9642610
Update DATABASE.md 2021-08-31 22:47:44 +02:00
Alexandre Dulaunoy
413191c712
Merge pull request #8 from wllm-rbnt/fix 
Fix jq parse error on special chars
 2021-08-31 21:43:45 +02:00
William Robinet
09b56135d1 Fix jq parse error on special chars 2021-08-31 12:11:50 +02:00
Alexandre Dulaunoy
c17dbff6a6
new: [api:stats/top] Add a new optional entry to point to get the top 100 of most queried hashes (existing and non-existing) 2021-08-29 14:06:35 +02:00
Alexandre Dulaunoy
4419052c4f
fix: [api:bulk] add proper check of MD5 and SHA1 value before further processing 2021-08-29 12:25:43 +02:00
Alexandre Dulaunoy
a14e5aedf1
new: [api:bulk] add support for pub-sub channel of existing and 
non-existing hashes
 2021-08-29 11:52:07 +02:00
Alexandre Dulaunoy
ecc2baf2f9
chg: [doc] added keys related to packages 2021-08-24 22:53:53 +02:00
Alexandre Dulaunoy
cea5524490
chg: [req] added 2021-08-24 14:57:15 +02:00
Alexandre Dulaunoy
1a94f6c246
fix: [req] moved 2021-08-24 14:56:56 +02:00
Alexandre Dulaunoy
a8b951cf83
chg: [server] add children hashes if these exist 2021-08-24 14:53:43 +02:00
Alexandre Dulaunoy
e9fb182c82
Update DATABASE.md 2021-08-23 09:17:20 +02:00
Alexandre Dulaunoy
02a675d9ea
fix: [api] typo fixed 2021-08-22 23:37:05 +02:00
Alexandre Dulaunoy
a17cd38595
chg: [api] lookup add parent details 2021-08-22 23:23:52 +02:00
Alexandre Dulaunoy
8504b492f7
chg: [doc] children added 2021-08-22 19:20:41 +02:00
Alexandre Dulaunoy
670e3dd52c
chg: [doc] parent added 2021-08-22 19:18:21 +02:00
Alexandre Dulaunoy
27aa7c4034
fix: [api] fix ttl missing bug 2021-08-22 16:48:06 +02:00
Alexandre Dulaunoy
ab1b74e109
chg: [doc] add the default keys used 2021-08-22 16:20:55 +02:00
Alexandre Dulaunoy
3a8185dd33
Merge branch 'main' of github.com:adulau/hashlookup-server into main 2021-08-22 16:14:02 +02:00
Alexandre Dulaunoy
99609502cb
chg: [doc] document basic storage format 2021-08-22 16:13:41 +02:00
Alexandre Dulaunoy
a0c858884a
Update README.md 2021-08-18 10:21:07 +02:00
Alexandre Dulaunoy
d43a693a7d
Typo fixed 2021-08-18 10:17:22 +02:00