Adrien Barbaresi

trafilatura. Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

German-NLP. Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

simplemma. Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

courlan. Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

htmldate. Fast and robust date extraction from web pages, with Python or on the command-line

py3langid. Faster, modernized fork of the language identification tool langid.py

geokelone. integrates spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization

german-reddit. Extraction of a German Reddit Corpus

awesome-crawler. A collection of awesome web crawler,spider in different languages

rust-primes. Rust code and command-line utility to compute and visualize prime sequences

tweets-tools. Diverse tools used with Twitter data

flux-toolchain. Filtering and Language-identification for URL Crawling Seeds (FLUCS) a.k.a. FLUX-Toolchain

awesome-web-scraping. List of libraries, tools and APIs for web scraping and data processing.

vardial-experiments. Experiments conducted on the occasion of the VarDial shared tasks

jparser. A readability parser which can extract title, content, images from html pages

jlcl-style. Experiments to modernize the LaTeX class of the JLCL

nltk_data. NLTK Data

coronakorpus. Material zum Aufbau eines deutschsprachigen COVID-19-Webkorpus / Building a corpus in German dedicated to coronavirus

awesome-digital-humanities. Software for humanities scholars using quantitative or computational methods.

toponyms. Old prototype for toponym extraction in historical texts written in German

zeitcrawler. Automatically exported from code.google.com/p/zeitcrawler

microblog-explorer. Perform crawls of social networks (identi.ca, reddit, friendfeed) to gather internal and external links and identify their language

url-compressor. A fast pattern-based URL compression for lists of links