Skip to content

Docs: consolidate build-from-source and GPU backend guide #2314

@cschanhniem

Description

@cschanhniem

Problem

The build-from-source documentation is spread across README, CONTRIBUTING.md, and many GitHub issues (e.g. #2013, #695, #721, #756). Users with newer GPUs (RTX 50-series, sm_120) or uncommon CUDA versions repeatedly hit the same build failures because the docs lack:

  1. A CMAKE_ARGS mapping table showing which flag to set for each GPU backend
  2. CUDA version compatibility notes (e.g., CUDA 12.9+ requires specific cmake workarounds)
  3. The minimum/maximum supported CUDA toolkit versions
  4. How to verify the build actually used the GPU (simple Python snippet to check backend)

Suggested improvements

In docs/build.md (or a new docs/gpu.md):

| Backend | CMAKE_ARGS                                    | GPU Requirements              |
|---------|-----------------------------------------------|-------------------------------|
| CUDA    | -DGGML_CUDA=ON                                | NVIDIA GPU, CUDA >= 11.3      |
| Metal   | -DGGML_METAL=ON (default on macOS)            | Apple Silicon or AMD GPU      |
| Vulkan  | -DGGML_VULKAN=ON                              | Vulkan 1.2+ capable GPU       |
| SYCL    | -DGGML_SYCL=ON -DGGML_SYCL_TARGET=INTEL       | Intel Arc / Iris Xe           |
| HIP     | -DGGML_HIPBLAS=ON                             | AMD ROCm-compatible GPU       |

And a troubleshooting section for the most common build errors (CMake not finding CUDA, nvcc version mismatch, sm_120 not recognized).

Value

Consolidating this reduces the "can't install with GPU" issues that currently fill the tracker.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions