Josh Wills

buenavista. A Postgres Proxy Server in Python

exhibit. A prototype of Hive UDFs/UDTFs that execute nested SQL queries within rows.

duckdbt. The Modern Data Stack in a Python package

de4ml. Supporting materials/code examples for my course in data engineering for machine learning.

avro-json. Utilities for converting to and from JSON from Avro records via Hadoop streaming or Hive.

geojson. Scala library for working with GeoJSON records using Esri's Geometry API for Java

shift_yourself_left. Python

target-duckdb. A Singer.io target for DuckDB

driskill. Either[Hotel in Austin, Prototype of a Scala Distributed Collections API]

nba_monte_carlo. The Modern Data Stack in a (Smaller) Box

supernova. A starter kit for working with supernova schemas.

lineage. An R package for tracking the transformations applied to the vectors in a data frame.

mz-fastapi. A FastAPI utility for building HTTP endpoints powered by Materialize TAIL queries

jaffle_shop_duckdb. Get started with dbt in less than 1 minute from `git clone` to `dbt docs serve` for free!

hive-scd. A new kind of slowly changing dimension pattern for Apache Hive.

dbt-buenavista. The dbt adapter for a Buena Vista database proxy server

dplyrimpaladb. R dplyr connector for ImpalaDB

crunch-demo. A demo application for getting started with Apache Crunch.

s3-demo. Demo dbt-duckdb against localstack w/the new fsspec config options in version 1.4.1

attribution. MapReduce job for creating multitouch attribution models.

datafu. Hadoop library for large-scale data processing

avroplay. Me messing around with some Avro stuff

saferdd. Tools for working with dirty data in Apache Spark.

hanukkahofdata. My solutions to the 2023 Hanukkah of Data

kitten. The fast and fun way to write YARN applications.

dbt-mysql. MySQL plugin for dbt

cdh-mapreduce-ext. Classes in the new mapreduce.* API that are not part of CDH3 yet.

dbt-external-tables. dbt macros to stage external sources

spatial-framework-for-hadoop. The Spatial Framework for Hadoop allows developers and data scientists to use the Hadoop data processing system for spatial data analysis.

harlequin. The DuckDB IDE for Your Terminal.

dbt-fusion. The next-generation engine for dbt

COVIDScenarioPipeline. Public shared code for doing scenario forecasting and creating reports for various governmental entities.

armeria. Asynchronous RPC/REST library built on top of Java 8, Netty, HTTP/2, Thrift and gRPC

litData. Transform datasets at scale. Optimize datasets for fast AI model training.

data_pipeline. Code for the data processing pipeline

autolabel. Label, clean and enrich text datasets with LLMs.

hosprunner. R

incubator-airflow. Apache Airflow (Incubating)

pants. The Pants Build System

deepdive. DeepDive

mz-hack-day-2022. Official repo for the Materialize + Redpanda + dbt Hack Day 2022, including a sample project to get everyone started!

spark-dataflow. Provides a Spark backend for executing Dataflow pipelines.

avro-json-serde. A wrapper that uses the Hive AvroSerDe to deserialize data as JSON for use with Hive Streaming

RecordBreaker. Record Breaker

brickhouse. Hive UDF's for the data warehouse

scoobi. A Scala productivity framework for Hadoop.