15 commits

Author	SHA1	Message	Date
Alexandre Dulaunoy	793e7ae9c5	chg: [langdetect] detection of language before further processing ... Before processing the text, we use cld3 to detect the language and compare if the foreseen spacy model to be used.	2020-10-09 20:47:43 +02:00
Alexandre Dulaunoy	98a8d8275e	chg: [output] make readable table-like with bold headers ... Official request from @C00kie-	2020-10-09 18:36:33 +02:00
Alexandre Dulaunoy	2c295e79cf	chg: [export] fix CSV export ... TODO: Review escaping in CSV	2020-10-09 17:19:11 +02:00
Alexandre Dulaunoy	49be2bf809	new: [output] readable output to help analyst reading the output ... First version based on @C00kie- feedback. Potential improvement could be a more tabular representation.	2020-10-09 07:48:06 +02:00
Alexandre Dulaunoy	9364c75477	chg: [score] scores are now integer	2020-10-09 07:27:08 +02:00
Alexandre Dulaunoy	193ad08144	fix: [bug] punctuation was not part of OOV and were not accounted	2020-10-09 07:25:26 +02:00
Alexandre Dulaunoy	ef5011a64f	chg: [cleanup] key names used in redis has been simplified	2020-10-09 07:18:16 +02:00
Alexandre Dulaunoy	10049a69b6	new: [option] --binary to dump in binary format instead of UTF-8	2020-10-08 23:30:57 +02:00
Alexandre Dulaunoy	26244739dd	new: [option] Don't flush the redisdb, useful when you want to process multiple files and aggregate the results.	2020-10-08 23:22:00 +02:00
Alexandre Dulaunoy	949e41d19f	new: [lemmatized/verbatim] displaying verbatim or lemmatized version is now an option	2020-10-08 23:13:51 +02:00
Alexandre Dulaunoy	3d71d9288e	chg: [args] add an option to force the language	2020-10-01 23:06:39 +02:00
Alexandre Dulaunoy	3a09abc80c	new: [output] JSON export added	2020-09-21 07:50:57 +02:00
Alexandre Dulaunoy	3c3760019e	chg: [feature] add punct statistics for the oov (but the punct in ... spacy.io seems super buggy or incorrect)	2020-08-20 14:33:15 +02:00
Alexandre Dulaunoy	526f88071c	new: [feature] -s option to display the overall statistics of different tokens seen	2020-08-20 13:28:49 +02:00
Alexandre Dulaunoy	dd7c796460	new: [napkin] first release ... Napkin is a Python tool to produce statistical analysis of a text. Analysis features are : - Verbs frequency - Nouns frequency - Digit frequency - Labels frequency such as (Person, organisation, product, location) as defined in spacy.io [named entities](https://spacy.io/api/annotation#named-entities) - URL frequency - Email frequency - Mention frequency (everything prefixed with an @ symbol) - Out-Of-Vocabulary (OOV) word frequency meaning any words outside English dictionary	2020-08-19 17:33:04 +02:00