A curated field guide to running, adapting, measuring, and learning about AI on hardware you control. Every entry links to its canonical home, code repository, and primary sources.
Installable applications for downloading models, chatting locally, and exposing local endpoints.
Desktop application and local server for discovering, downloading, and running local language models on macOS, Windows, and Linux.
Local-first desktop AI application that runs open models on-device and can connect to remote model providers.
Desktop chat application and SDK ecosystem for running language models privately on consumer hardware.
Inference engines, model servers, and portable runtimes for CPUs, GPUs, browsers, and edge devices.
C and C++ runtime for local and cloud LLM and VLM inference across CPU, Metal, CUDA, HIP, Vulkan, SYCL, and other backends.
Local model runner and server for building with open models on macOS, Windows, Linux, and Docker.
Composable local AI engine exposing OpenAI-, Anthropic-, and ElevenLabs-compatible APIs for language, vision, voice, image, and video models.
High-throughput library and server for LLM inference with continuous batching, PagedAttention, quantization, distributed parallelism, and OpenAI-compatible APIs.
Serving framework for language and multimodal models with a high-performance runtime and frontend language for model programs.
Python package for generating text, quantizing, and fine-tuning large language models on Apple silicon with MLX.
GPU-focused inference library and quantization format for running transformer language models efficiently on NVIDIA hardware.
NVIDIA toolkit for optimizing and serving large language model inference with TensorRT on NVIDIA GPUs.
Machine-learning compilation runtime for deploying language models across GPUs, CPUs, browsers, and native applications.
ONNX Runtime library providing generative-model loops, tokenization, sampling, KV-cache management, and model execution.
Single-binary local inference server and interface built on llama.cpp with text-generation and multimodal features.
OpenAI-compatible web server for ExLlamaV2 and ExLlamaV3 model inference.
Array framework for machine learning research on Apple silicon with unified memory and composable transformations.
High-performance C and C++ implementation of OpenAI Whisper speech recognition for local inference.
Packages model weights and a llama.cpp-based runtime into portable executables for local model distribution and execution.
Rust and HIP inference engine for running language models on AMD RDNA and CDNA GPUs with an OpenAI-compatible server.
Speculative-inference server for heterogeneous machines and consumer NVIDIA and AMD GPUs, with custom kernels and OpenAI-compatible serving.
Toolkit for optimizing and running AI inference across Intel CPUs, GPUs, and NPUs, including generative-model workloads.
On-device AI runtime and deployment stack for executing PyTorch models across mobile, embedded, and edge hardware.
Self-hosted workspaces and web interfaces for using local models, documents, agents, and tools.
Self-hosted web interface for local Ollama models and OpenAI-compatible APIs, with model, agent, tool, RAG, and multi-user features.
All-in-one desktop and self-hosted application for chatting with documents, using agents, and connecting local or remote models.
Self-hosted multi-provider AI chat platform supporting agents, tools, model endpoints, and local inference servers.
Gradio web interface for running and experimenting with local text-generation models through multiple loaders.
Control planes, proxies, and distributed-serving systems for operating one or many model servers.
Lightweight proxy that starts and stops local model servers on demand and exposes an OpenAI-compatible endpoint.
Kubernetes-native distributed inference serving stack for large language models, built around vLLM and intelligent scheduling.
Open-source platform for managing GPU clusters and deploying model inference services across heterogeneous hardware.
Libraries for adapting models, reducing memory use, and preparing efficient local artifacts.
Toolkit for memory-efficient fine-tuning and reinforcement learning of open language models.
Weight-only quantization toolkit for compressing language and multimodal models with low tuning cost.
Library providing k-bit optimizers, matrix multiplication, and quantization primitives for PyTorch models.
Model quantization and inference library implementing GPTQ-family post-training quantization across multiple backends.
Unified framework for fine-tuning language and multimodal models with scalable training methods and a browser-based interface.
Reproducible harnesses and public leaderboards for measuring inference performance and model or agent quality.
Public leaderboard comparing real LLM inference benchmarks, runtimes, recipes, and configurations on NVIDIA DGX Spark hardware.
Community platform for local-LLM speed tests and quality benchmarks across models, hardware, engines, and quantizations.
Benchmark executable included with llama.cpp for measuring model load, prompt processing, and token-generation performance.
Command-line benchmarking application for measuring throughput and latency of generative-AI inference services.
Framework for running and evaluating agents in containerized task environments, including Terminal-Bench.
Benchmark dataset and harness for evaluating AI agents on real terminal tasks in isolated environments.
Open evaluation platform and reproducible harness for comparing language models on standardized benchmarks.
Unified framework for evaluating local and hosted language models across a large collection of reproducible academic benchmark tasks.
Dataset and reproducible Docker evaluation harness for measuring whether language-model agents can resolve real-world software issues.
Contamination-resistant benchmark and evaluation code for measuring language-model code-generation ability with continuously collected programming problems.
Primary collections and platforms for finding, versioning, and distributing models and datasets.
Open initiative producing small, task-specific machine-learning models distributed in ONNX format for local use on devices, servers, and browsers.
Git-based platform for discovering, versioning, sharing, and collaborating on models, datasets, and ML applications.
Model-as-a-Service ecosystem and hubs for discovering, versioning, downloading, running, fine-tuning, and evaluating models and datasets.
Open learning material for understanding, adapting, and deploying language models.
Free course covering transformer models, Hugging Face libraries, fine-tuning, datasets, tokenizers, demos, and advanced LLM topics.
Community-maintained maps of the local-AI ecosystem and places to discover related projects.
Curated directory of open-source and local AI inference engines, interfaces, platforms, developer tools, agents, training projects, leaderboards, and communities.