Helge Holzmann

ArchiveSpark. An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

Web2Warc. An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

HadoopConcatGz. A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

internetarchive-transfer-scripts. Scripts to transfer archive.org collections, using https://github.com/jjjake/internetarchive

HadoopWebGraph. A Hadoop input format to use gaphs in WebGraph's BV format with Hadoop and Spark.

IABooksOnArchiveSpark. Analyze digitized books from the Internet Archive remotely with ArchiveSpark

Exspec. Don't write specs anymore, just save 'em while testing your code interactively. Specs will become a byproduct.

Micrawler. Create and cite micro Web archives with semantics as temporal representations of objects / entities / concepts on the Web

ArchiveSpark-server. A server application that provides a Web service API for ArchiveSpark to be used by third-party applications to integrate temporal Web archive data with a flexible, easy-to-use interface.

FEL4ArchiveSpark. Yahoo's Fast Entity Linker for ArchiveSpark

MHLonArchiveSpark. Work with Medical Heritage Library collections using ArchiveSpark

ArchiveSpark-AUT-bridge. The compatibility layer between ArchiveSpark and The Archives Unleashed Toolkit (AUT)

ArchiveSpark-Zeppelin-Docker. ArchiveSpark with Zeppelin as ready-to-use Docker image

ArchiveSpark2Triples. Convert web archives to RDF triples with ArchiveSpark

WarcPartitioner. Partition (W)ARC Files by MIME Type and Year

ArchiveSpark-docker. ArchiveSpark on Docker

ArchivePig. An Apache Pig framework that facilitates access to Web Archives, enables easy data extraction as well as derivation.

MapReduceLecture. A lecture on MapReduce with example code