|
GPULlama3.java is a JVM-native LLM inference and serving engine. You write and ship plain Java; TornadoVM JIT-compiles the hot transformer kernels to CUDA, PTX, OpenCL, or Apple Metal at runtime — no JNI glue, no second toolchain, no native rebuild per GPU. One Serve it behind an OpenAI-compatible API, embed it in LangChain4j or Quarkus, or run it from the CLI in one line. |
Builds on Llama3.java by Alfonso² Peterssen. Earlier Llama2 work: llama2.tornadovm.
- 🟦 Pure Java, all the way down. Transformer kernels are written in Java and accelerated by TornadoVM — no CUDA C, no hand-written JNI. Debug and build with the toolchain you already have.
- 🌍 Write once, run on any GPU. NVIDIA (CUDA / PTX), Intel & AMD (OpenCL), Apple Silicon (Metal). Backend is auto-detected from your TornadoVM SDK — switch with a flag, not a rebuild.
- 🔌 Drop-in for the Java AI stack. Official LangChain4j provider (since v1.7.1) and Quarkus inference engine.
- ⚡ Built to serve. OpenAI-compatible HTTP server, llama-bench-style benchmarking, and tensor-core (MMA) batch prefill (see Serving).
- 📦 Many models, one runtime. Llama 3, Mistral, Qwen 2.5 / Qwen 3, Phi-3, IBM Granite 3.3 / 4.0, DeepSeek-R1-Distill — all in GGUF.
# 1. Install a TornadoVM SDK (bundles the GPU runtime)
curl -s "https://get.sdkman.io" | bash && source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install tornadovm
tornado --devices # confirm your GPU is listed
# 2. Run a model on the GPU — no build required, via JBang
jbang gpullama3@beehive-lab -m beehive-llama-3.2-1b-instruct-fp16.gguf -p "Explain GPU acceleration in one sentence."Grab a ready-to-run model from the Hugging Face collections below.
GPULlama3.java is growing into a serving engine — the vLLM-style path for the JVM:
- 🌐 OpenAI-compatible server —
llama-tornado --serverexposes/v1/chat/completionsand/v1/completionswith streaming and zero external dependencies. Point any OpenAI client atlocalhost. See OpenAI-compatible server below. - 🎯 Tensor-core (MMA) batch prefill on the CUDA backend, FP16 & Q8_0 —
--with-prefill-decode --batch-prefill-size N. - 📈 llama-bench-style benchmarking —
llama-tornado --benchreports a pp/tg matrix with avg±stddev in md/csv/json/jsonl/sql. See Benchmarking below. - 🧮 On-device greedy sampling (landing next) — argmax on the GPU keeps logits device-side, cutting device→host traffic by ~500× per token. (PR #134)
- 📚 Static batched decode (landing next) — B independent sequences per step for up to 41× aggregate throughput (Llama & Qwen3). (PR #129)
Since LangChain4j v1.7.1, GPULlama3.java is an officially supported model provider — no glue code, GPU-accelerated out of the box.
GPULlama3ChatModel model = GPULlama3ChatModel.builder()
.modelPath(modelPath)
.temperature(0.9) // more creative
.topP(0.9) // more variety
.maxTokens(2048)
.onGPU(Boolean.TRUE) // false → lightweight CPU llama3.java
.build();📖 LangChain4j docs · 🚀 Agentic workflow demo
<!-- JDK 21 -->
<dependency>
<groupId>io.github.beehive-lab</groupId>
<artifactId>gpu-llama3</artifactId>
<version>0.5.0</version>
</dependency>
<!-- JDK 25 → use version 0.5.0-jdk25 -->
- Java 21 — required for the Vector API & TornadoVM (Java 25 supported via the
-jdk25artifact /llamaTornadoscript). - TornadoVM with an OpenCL, PTX, CUDA, or Metal backend.
llama-tornado/llamaTornadoauto-detect whichever backend your installed SDK was built with. - GCC/G++ 13+ — to build TornadoVM's native components.
TornadoVM is distributed via the official website and SDKMAN!. Pick a package matching your OS, architecture, and backend (opencl, ptx).
sdk install tornadovm
tornado --devices # verifygit clone https://github.com/beehive-lab/GPULlama3.java.gitUse the llama-tornado script with --gpu. The backend (OpenCL, PTX, CUDA, or Metal) is auto-detected from
your installed TornadoVM SDK (TORNADOVM_HOME/etc/tornado.backend) — no need to select it manually. If your
SDK was built with more than one backend, force one with --opencl, --ptx, --cuda (NVIDIA), or --metal
(Apple Silicon); forcing a backend that isn't part of the installed SDK errors out.
# Basic GPU inference — backend auto-detected
./llama-tornado --gpu --verbose-init \
--model beehive-llama-3.2-1b-instruct-fp16.gguf \
--prompt "Explain the benefits of GPU acceleration."
# Force a specific backend (only needed for multi-backend SDKs)
./llama-tornado --gpu --cuda \
--model beehive-llama-3.2-1b-instruct-fp16.gguf \
--prompt "Explain the benefits of GPU acceleration."Swap in any tested model — e.g. beehive-llama-3.2-3b-instruct-fp16.gguf or ...-8b-....
Same backend auto-detection as llama-tornado. A single-file Java 25 launcher that replaces the Python
script (needs java 25+ on your PATH):
./llamaTornado --gpu --verbose-init --metal \
--model Mistral-7B-Instruct-v0.3.Q8_0.gguf --prompt "what is java"Script-like startup à la Jlama, powered by JBang:
curl -Ls https://sh.jbang.dev | bash -s - app setup
# From the catalog
jbang gpullama3@beehive-lab -m model.gguf -p "Tell me a joke"
jbang app install gpullama3@beehive-lab && gpullama3 -m model.gguf -p "Hello!"
# Or the local script, interactive
jbang LlamaTornadoCli.java -m beehive-llama-3.2-1b-instruct-fp16.gguf --interactiveFully containerized GPU inference via pre-built images (docker-gpullama3.java):
| Backend | Image |
|---|---|
| OpenCL | beehivelab/gpullama3.java-nvidia-openjdk-opencl |
| PTX (CUDA) | beehivelab/gpullama3.java-nvidia-openjdk-ptx |
docker run --rm -it --gpus all -v "$PWD":/data \
beehivelab/gpullama3.java-nvidia-openjdk-opencl \
/gpullama3/GPULlama3.java/llama-tornado \
--gpu --verbose-init \
--model /data/Llama-3.2-1B-Instruct.FP16.gguf --prompt "Tell me a joke"GGUF models, ready to download:
| Family | Collection |
|---|---|
| Llama 3.2 | llama3-gpullama3java |
| IBM Granite 4.0 | granite-40-language-models |
| IBM Granite 3.3 | granite-33-language-models |
| Qwen 2.5 | qwen-25-gpullama3java |
| Qwen 3 | qwen-3-gpullama3java |
| Phi-3 | phi-3-gpullama3java |
| Mistral | mistral-gpullama3java |
| DeepSeek-R1-Distill-Qwen | deepseek-r1-distill-qwen |
Formats: GGUF · FP16 (full), Q8_0 & Q4_0 (partial).
Default device allocation is 14GB. Larger models need more — raise it with --gpu-memory:
| Model size | Recommended | Flag |
|---|---|---|
| 1B | 14GB (default) | — |
| 3–7B | 15GB+ | --gpu-memory 15GB |
| 8B+ | 20GB+ | --gpu-memory 20GB |
./llama-tornado --gpu --model beehive-llama-3.2-3b-instruct-fp16.gguf \
--prompt "Tell me a joke" --gpu-memory 15GBStill out of memory? Use Q4_0 instead of Q8_0, or close other GPU apps. The error to look for:
TornadoOutOfMemoryException: Unable to allocate ... bytes of memory.
To increase the maximum device memory, use -Dtornado.device.memory=<X>GB
--show-command prints the exact Java + JVM invocation used under the hood, so you can replicate it in IntelliJ, Maven, Gradle, or any launcher:
llama-tornado --gpu --model beehive-llama-3.2-1b-instruct-fp16.gguf \
--prompt "tell me a joke" --show-command📋 Full command-line options (llama-tornado --help)
usage: llama-tornado [-h] --model MODEL_PATH [--prompt PROMPT] [-sp SYSTEM_PROMPT]
[--temperature TEMPERATURE] [--top-p TOP_P] [--seed SEED] [-n MAX_TOKENS]
[--stream STREAM] [--echo ECHO] [-i] [--instruct]
[--server] [--port PORT] [--bench] [--bench-args BENCH_ARGS]
[--gpu] [--opencl] [--ptx] [--cuda] [--metal]
[--gpu-memory GPU_MEMORY] [--heap-min HEAP_MIN] [--heap-max HEAP_MAX]
[--debug] [--profiler] [--profiler-dump-dir DIR]
[--print-bytecodes] [--print-threads] [--print-kernel] [--full-dump]
[--show-command] [--execute-after-show]
[--with-prefill-decode] [--batch-prefill-size N] [--cuda-graphs]
[--opencl-flags FLAGS] [--max-wait-events N] [--verbose]
LLaMA Configuration: --prompt, -sp/--system-prompt, --temperature (0.0–2.0, default 0.1),
--top-p (default 0.95), --seed, -n/--max-tokens (default 512),
--stream (default True), --echo (default False), --suffix (FIM/Codestral)
Mode Selection: -i/--interactive, --instruct (default)
OpenAI server: --server (run the HTTP server instead of inference), --port (default 8080)
Benchmark: --bench (llama-bench-style matrix), --bench-args="..." (see Benchmarking below)
Hardware: --gpu, --opencl/--ptx/--cuda/--metal (auto-detected; force one of the installed
backends), --gpu-memory (default 14GB), --heap-min/--heap-max (default 20g)
Debug & Profiling: --debug, --profiler, --profiler-dump-dir,
--print-bytecodes, --print-threads, --print-kernel, --full-dump, --verbose-init
Command Display: --show-command, --execute-after-show
Prefill-Decode: --with-prefill-decode, --batch-prefill-size N (tensor-core MMA batch prefill,
FP16 & Q8_0)
Advanced: --cuda-graphs (PTX backend only), --opencl-flags (default: -cl-denorms-are-zero
-cl-no-signed-zeros -cl-finite-math-only), --max-wait-events (default 32000), --verbose/-v
# Peek at what TornadoVM is doing
./llama-tornado --gpu --model model.gguf --prompt "..." --print-kernel # generated GPU kernel
./llama-tornado --gpu --model model.gguf --prompt "..." --print-bytecodes # TornadoVM bytecodes
./llama-tornado --gpu --model model.gguf --prompt "..." --debug --full-dump # everything- ✅ GGUF models — full FP16, partial Q8_0 / Q4_0.
- ✅ Chat, instruction, and interactive modes (
--interactive,--instruct). - ✅ Automatic backend detection —
llama-tornado/llamaTornadodetect and use whichever backend (OpenCL, PTX, CUDA, or Metal) your installed TornadoVM SDK was built with; override with--opencl/--ptx/--cuda/--metal. - ✅ Cross-platform: NVIDIA (OpenCL · PTX · CUDA), Intel (OpenCL), Apple (OpenCL · Metal).
- ✅ Serving — OpenAI-compatible API, llama-bench-style benchmarking, tensor-core (MMA) batch prefill.
- 🧩 Coming next — static batched decode, on-device sampling (preview; see Serving).
📄 Transformer optimizations in TornadoVM · 🧭 Project roadmap
Partially funded by EU Horizon Europe & UKRI grants (most recent first): AERO 101092850 · P2CODE 101093069 · ENCRYPT 101070670 · TANGO 101070052.