2020-08-18 14:49:24 +00:00
# napkin-text-analysis
2020-08-19 15:33:04 +00:00
2020-10-09 19:52:00 +00:00
2020-08-19 15:33:04 +00:00
Napkin is a Python tool to produce statistical analysis of a text.
Analysis features are :
- Verbs frequency
- Nouns frequency
- Digit frequency
- Labels frequency such as (Person, organisation, product, location) as defined in spacy.io [named entities ](https://spacy.io/api/annotation#named-entities )
- URL frequency
- Email frequency
- Mention frequency (everything prefixed with an @ symbol)
- Out-Of-Vocabulary (OOV) word frequency meaning any words outside English dictionary
2020-10-08 21:13:51 +00:00
Verbs and nouns are in their lemmatized form by default but the option `--verbatim` allows to keep the original inflection.
2020-10-08 21:22:00 +00:00
Intermediate results are stored in a Redis database to allow the analysis of multiple text files.
2020-08-19 15:33:04 +00:00
# requirements
- Python >= 3.6
- spacy.io
2020-10-09 19:52:00 +00:00
- redis (a redis server running on port 6380 is required)
- pycld3
- tabulate
2020-08-19 15:33:04 +00:00
# how to use napkin
~~~~
2020-11-22 13:22:18 +00:00
usage: napkin.py [-h] [-v V] [-f F] [-t T] [-s] [-o O] [-l L] [-i]
[--verbatim] [--no-flushdb] [--binary] [--analysis ANALYSIS]
2020-10-11 09:04:30 +00:00
[--disable-parser] [--disable-tagger]
2020-10-13 05:41:01 +00:00
[--token-span TOKEN_SPAN] [--table-format TABLE_FORMAT]
2020-11-22 13:22:18 +00:00
[--full-labels]
2020-08-19 15:33:04 +00:00
Extract statistical analysis of text
optional arguments:
2020-10-11 09:24:17 +00:00
-h, --help show this help message and exit
-v V verbose output
-f F file to analyse
-t T maximum value for the top list (default is 100) -1 is
no limit
-s display the overall statistics (default is False)
-o O output format (default is csv), json, readable
-l L language used for the analysis (default is en)
2020-11-22 13:22:18 +00:00
-i Use stdin instead of a filename
2020-10-11 09:24:17 +00:00
--verbatim Don't use the lemmatized form, use verbatim. (default
is the lematized form)
--no-flushdb Don't flush the redisdb, useful when you want to
process multiple files and aggregate the results. (by
default the redis database is flushed at each run)
--binary set output in binary instead of UTF-8 (default)
--analysis ANALYSIS Limit output to a specific analysis (verb, noun,
hashtag, mention, digit, url, oov, labels, punct).
(Default is all analysis are displayed)
--disable-parser disable parser component in Spacy
--disable-tagger disable tagger component in Spacy
--token-span TOKEN_SPAN
Find the sentences where a specific token is located
2020-10-13 05:41:01 +00:00
--table-format TABLE_FORMAT
set tabulate format (default is fancy_grid)
2020-11-22 13:22:18 +00:00
--full-labels store each label value in a ranked set (default is
False)
2020-08-19 15:33:04 +00:00
~~~~
# example usage of napkin
2020-10-11 09:24:17 +00:00
## Generate all analysis for a given text
2020-08-19 15:33:04 +00:00
A sample file "The Prince, by Nicoló Machiavelli" is included to test napkin.
2020-10-09 19:52:00 +00:00
`python3 ./bin/napkin.py -o readable -f samples/the-prince.txt -t 4`
2020-08-19 15:33:04 +00:00
Example output:
~~~~
2020-10-09 19:52:00 +00:00
╒══════════════════╕
2020-10-13 05:41:01 +00:00
│ Top 4 of verb │
╞══════════════════╡
│ 207 occurences │
├──────────────────┤
│ will │
├──────────────────┤
│ 137 occurences │
├──────────────────┤
│ can │
├──────────────────┤
│ 116 occurences │
├──────────────────┤
│ make │
├──────────────────┤
│ 106 occurences │
├──────────────────┤
│ may │
├──────────────────┤
│ 102 occurences │
├──────────────────┤
│ would │
╘══════════════════╛
╒══════════════════╕
│ Top 4 of noun │
2020-10-09 19:52:00 +00:00
╞══════════════════╡
2020-10-13 05:41:01 +00:00
│ 206 occurences │
├──────────────────┤
│ prince │
├──────────────────┤
│ 120 occurences │
├──────────────────┤
│ man │
2020-10-09 19:52:00 +00:00
├──────────────────┤
2020-10-13 05:41:01 +00:00
│ 108 occurences │
2020-10-09 19:52:00 +00:00
├──────────────────┤
2020-10-13 05:41:01 +00:00
│ state │
├──────────────────┤
│ 90 occurences │
├──────────────────┤
│ people │
├──────────────────┤
│ one │
2020-10-09 19:52:00 +00:00
╘══════════════════╛
2020-10-13 05:41:01 +00:00
╒═════════════════════╕
│ Top 4 of hashtag │
╞═════════════════════╡
╘═════════════════════╛
╒═════════════════════╕
│ Top 4 of mention │
╞═════════════════════╡
╘═════════════════════╛
╒═══════════════════╕
│ Top 4 of digit │
╞═══════════════════╡
│ 1 occurences │
├───────────────────┤
│ 99775 │
├───────────────────┤
│ 84116 │
├───────────────────┤
│ 750175 │
├───────────────────┤
│ 6221541 │
├───────────────────┤
│ 57037 │
╘═══════════════════╛
2020-10-09 19:52:00 +00:00
╒═════════════════════════════════════════╕
│ Top 4 of url │
╞═════════════════════════════════════════╡
2020-10-13 05:41:01 +00:00
│ 5 occurences │
├─────────────────────────────────────────┤
│ www.gutenberg.org │
├─────────────────────────────────────────┤
│ 2 occurences │
├─────────────────────────────────────────┤
│ www.gutenberg.org/donate │
├─────────────────────────────────────────┤
2020-10-09 19:52:00 +00:00
│ 1 occurences │
├─────────────────────────────────────────┤
│ www.gutenberg.org/license │
├─────────────────────────────────────────┤
│ www.gutenberg.org/contact │
├─────────────────────────────────────────┤
│ http://www.gutenberg.org/5/7/0/3/57037/ │
╘═════════════════════════════════════════╛
2020-10-13 05:41:01 +00:00
╒═════════════════╕
│ Top 4 of oov │
╞═════════════════╡
│ 9 occurences │
├─────────────────┤
│ Sforza │
├─────────────────┤
│ 7 occurences │
├─────────────────┤
│ Fermo │
├─────────────────┤
│ 6 occurences │
├─────────────────┤
│ Vitelli │
├─────────────────┤
│ Pertinax │
├─────────────────┤
│ Orsinis │
╘═════════════════╛
╒════════════════════╕
│ Top 4 of labels │
╞════════════════════╡
│ 339 occurences │
├────────────────────┤
│ PERSON │
├────────────────────┤
│ 305 occurences │
├────────────────────┤
│ GPE │
├────────────────────┤
│ 197 occurences │
├────────────────────┤
│ CARDINAL │
├────────────────────┤
│ 189 occurences │
├────────────────────┤
│ ORG │
├────────────────────┤
│ 131 occurences │
├────────────────────┤
│ NORP │
╘════════════════════╛
2020-10-09 19:52:00 +00:00
╒═══════════════════╕
2020-10-13 05:41:01 +00:00
│ Top 4 of punct │
2020-10-09 19:52:00 +00:00
╞═══════════════════╡
2020-10-13 05:41:01 +00:00
│ 3440 occurences │
├───────────────────┤
├───────────────────┤
│ 144 occurences │
├───────────────────┤
├───────────────────┤
│ 32 occurences │
2020-10-09 19:52:00 +00:00
├───────────────────┤
├───────────────────┤
2020-10-13 05:41:01 +00:00
│ 26 occurences │
2020-10-09 19:52:00 +00:00
├───────────────────┤
├───────────────────┤
2020-10-13 05:41:01 +00:00
│ 11 occurences │
├───────────────────┤
│ 1.F.3 │
╘═══════════════════╛
╒═══════════════════╕
│ Top 4 of email │
╞═══════════════════╡
│ 1 occurences │
2020-10-09 19:52:00 +00:00
├───────────────────┤
2020-10-13 05:41:01 +00:00
│ gbnewby@pglaf.org │
2020-10-09 19:52:00 +00:00
╘═══════════════════╛
2020-08-19 15:33:04 +00:00
~~~~
2020-10-11 09:24:17 +00:00
## Extract the sentences associated to a specific token
`python3 ./bin/napkin.py -o readable -f samples/the-prince.txt -t 4 --token-span "Vitelli"`
~~~~
2020-10-13 05:41:01 +00:00
╒═════════════════════════════════════════════════════════════════════════╕
│ Top 4 of span for Vitelli │
╞═════════════════════════════════════════════════════════════════════════╡
│ 1 occurences │
├─────────────────────────────────────────────────────────────────────────┤
│ This duke entered │
│ Romagna with auxiliary troops, leading forces composed entirely of │
│ French soldiers, and with these he took Imola and Forli; but as they │
│ seemed unsafe, he had recourse to mercenaries, and hired the Orsini and │
│ Vitelli; afterwards finding these uncertain to handle, unfaithful and │
│ dangerous, he suppressed them, and relied upon his own men. │
├─────────────────────────────────────────────────────────────────────────┤
│ The Florentines appointed Paolo Vitelli their captain, │
│ a man of great prudence, who had risen from a private station to the │
│ highest reputation. │
├─────────────────────────────────────────────────────────────────────────┤
│ Nevertheless, Messer Niccolo Vitelli has been seen in │
│ our own time to destroy two fortresses in Città di Castello in order │
│ to keep that state. │
├─────────────────────────────────────────────────────────────────────────┤
│ And the │
│ difference between these forces can be easily seen if one considers │
│ the difference between the reputation of the duke when he had only the │
│ French, when he had the Orsini and Vitelli, and when he had to rely │
│ on himself and his own soldiers. │
├─────────────────────────────────────────────────────────────────────────┤
│ And that his foundations were │
│ good is seen from the fact that the Romagna waited for him more than a │
│ month; in Rome, although half dead, he remained secure, and although │
│ the Baglioni, Vitelli, and Orsini entered Rome they found no followers │
│ against him. │
╘═════════════════════════════════════════════════════════════════════════╛
2020-10-11 09:24:17 +00:00
~~~~
2020-10-13 05:46:21 +00:00
## Specify table format
In `readable` output, the format can be set to any of the tabulate format supported. If you want the top 10 of out-of-vocabulary
words from the text in GitHub markdown format.
`python3 ./bin/napkin.py -o readable -f samples/the-prince.txt -t 10 --analysis oov --table-format github`
| Top 10 of oov |
|------------------|
| 9 occurences |
| Sforza |
| 7 occurences |
| Fermo |
| 6 occurences |
| Vitelli |
| Pertinax |
| Orsinis |
| Colonnas |
| Bentivogli |
| Agathocles |
| 5 occurences |
| Oliverotto |
| Cæsar |
| Commodus |
2020-10-11 12:32:00 +00:00
# overview of processing in napkin
2020-10-11 12:32:45 +00:00
2020-10-11 12:32:00 +00:00
2020-10-09 19:52:00 +00:00
# what about the name?
The name 'napkin' came after a first sketch of the idea on a napkin. The goal was also to provide a simple text analysis tool which can be run on the corner of table in a kitchen.
2020-08-19 15:33:04 +00:00
# LICENSE
napkin is free software under the AGPLv3 license.
~~~~
Copyright (C) 2020 Alexandre Dulaunoy
Copyright (C) 2020 Pauline Bourmeau
~~~~
