Llama Cpp Portable Llm Inference

---
name: "llama.cpp Portable LLM Inference Engine in C/C++"
slug: "llama-cpp-portable-llm-inference"
description: "llama.cpp is a high-performance C/C++ implementation for running LLM inference across diverse hardware. It supports GGUF model quantization, GPU acceleration on NVIDIA/AMD/Apple Silicon, and provides both a CLI and an OpenAI-compatible HTTP server for local model serving."
github_stars: 100939
verification: "security_reviewed"
source: "https://github.com/ggml-org/llama.cpp"
category: "Developer Tools"
framework: "Multi-Framework"
tool_ecosystem:
  github_repo: "ggml-org/llama.cpp"
  github_stars: 100939
---

# llama.cpp Portable LLM Inference Engine in C/C++

llama.cpp is a high-performance C/C++ implementation for running LLM inference across diverse hardware. It supports GGUF model quantization, GPU acceleration on NVIDIA/AMD/Apple Silicon, and provides both a CLI and an OpenAI-compatible HTTP server for local model serving.

## Installation

Use the upstream install or setup path that matches your environment:
- Run with Docker - see our [Docker documentation](docs/docker.md)

Requirements and caveats from upstream:
- Python: [ddh0/easy-llama](https://github.com/ddh0/easy-llama)
- Python: [abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
- Node.js: [withcatai/node-llama-cpp](https://github.com/withcatai/node-llama-cpp)

Basic usage or getting-started notes:
- Install llama.cpp using [brew, nix or winget](docs/install.md)
- Download pre-built binaries from the [releases page](https://github.com/ggml-org/llama.cpp/releases)
- Build from source by cloning this repository - check out [our build guide](docs/build.md)

- Source: https://github.com/ggml-org/llama.cpp
- Extracted from upstream docs: https://raw.githubusercontent.com/ggml-org/llama.cpp/HEAD/README.md

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/llama-cpp-portable-llm-inference/)