Less Wright

Ranger-Deep-Learning-Optimizer. Ranger - a synergistic optimizer using RAdam (Rectified Adam), Gradient Centralization and LookAhead in one codebase

Ranger21. Ranger deep learning optimizer rewrite to use newest components

Best-Deep-Learning-Optimizers. Collection of the latest, greatest, deep learning optimizers (for Pytorch) - CNN, NLP suitable

mish. Mish Deep Learning Activation Function for PyTorch / FastAI

res2net-plus. Res2Net architecture with improved stem and Mish activation function

Ranger-Mish-ImageWoof-5. Repo to build on / reproduce the record breaking Ranger-Mish-SelfAttention setup on FastAI ImageWoof dataset 5 epochs

FAdam_PyTorch. an implementation of FAdam (Fisher Adam) in PyTorch

training-detr. Unofficial Colab on how to train DETR, the intelligent object detector, with your own dataset. DETR = Detection Transformer

transformer_central. Various transformers for FSDP research

triton_kernels_for_fun_and_profit. Custom kernels in Triton language for accelerating LLMs

Ranger22. Testing various improvements to Ranger21 for 2022

mrnet-fastai. Deep Learning CNN using FastAI for the Stanford MRNet Knee MRI diagnosis challenge

Thunder-Detr. (unofficial) - customized fork of DETR, optimized for intelligent obj detection on 'real world' custom datasets

AdamW-Triton-PyTorch. Can AdamW written in Triton be as performant as fused CUDA impl?

MARS-AdamW-PyTorch. unofficial implementation of MARS-AdamW in PyTorch

t5_11. housing our model example of fine tuning an 11B t5 with FSDP

fsdp_llm. FSDP optimizations for LLM training

hyper_efficient_optimizers. Development of hyper efficient optimizers that can match/exceed AdamW, while using reduced memory

transformer_framework. framework for plug and play of various transformers (vision and nlp) with FSDP

LightRelu. Customized PyTorch implementation of LiSHT (linear scaled hyperbolic tangent) activation function for deep learning

FTSwishPlus. FTSwish with mean shifting added to increase performance

auto-adaptive-ai. auto adaptive framework for intrinsic hyperparameter selection, adaptive padding, normalized weights

TRelu. An improved activation function for deep learning - Threshold Relu, or TRelu

fsdp_review. Some eval and profile routines for fsdp

torchchat. Run PyTorch LLMs locally on servers, desktop and mobile

muon_optimizer. Muon optimizer with changes proposed by Moonlight

PolarBearLLM. testing new TransFormer, MoE, and TransNormer features

expert-token-resonance-pytorch. Unofficial implementation of Expert Token Resonance in PyTorch (and later Triton)

EfficientNet-PyTorch. Unofficial port of Google's new EfficientNet to Pytorch and FastAI

RangerQH-Testing. Repo for running RangerQH + Res2NetPLus with LIP Pooling

AutoOpt-for-FastAI. Integrate Ebay's AutoOpt Deep Learning Optimizer into the FastAI framework

4bit_optimizers. 4bit optimizers for PyTorch

sigma_reparam. Sigma Reparam for Transformers (based on Apple's paper)

bfloat_optimizer. Pure bfloat AdamW+ tweaks

facial-keypoint-detection. Facial keypoint detection CNN - custom architecture using partial convolution padding

QuantFour_AdamW_Cuda. Fused 4bit AdamW in Cuda

perception_tools. additional utils for working with Unity perception package

adopt. Official Implementation of "ADOPT: Modified Adam Can Converge with Any β2 with the Optimal Rate"

skycraft2. Minecraft in the sky, written in Python

unet-seg. Jupyter Notebook

AutoGPTQ_Triton. An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

FTSwish. Flattened Threshold Swish Activation function - PyTorch implementation

vision_mamba. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

style-transfer-vgg. Artistic Style transfer using VGG19

coordinate_clipped_Optimizers. coordinate wise clipped Optimizers in PyTorch

cuSZp2. AD-AE version of SC24 cuSZp2 paper

T-MAC. Low-bit LLM inference on CPU/NPU with lookup table

triton_flashv2_alibi. working repo for Triton based Flash2 supporting alibi pos embeddings

dion. Dion optimizer algorithm

Adam-mini. Code for Adam-mini: Use Fewer Learning Rates To Gain More

Atom. Codebase for Atom

rmsnorm_explore. Triton RMSNorm dev

cuda_float_compress. Python package for compressing floating-point PyTorch tensors

nano-vllm-rust. port of nano-vllm to rust language