Llama3.java

Practical Llama 3, 3.1 and 3.2 inference implemented in a single Java file.

This project from Alfonso² Peterssen is on GitHub.

Installation and test

Download Llama3.java:

git clone https://github.com/mukel/llama3.java.git

Download Llama:

cd llama3.java
# Llama 3.2 (3B)
curl -L -O https://huggingface.co/mukel/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_0.gguf

Execute:

java --enable-preview --source 21 --add-modules jdk.incubator.vector Llama3.java -i --model Llama-3.2-3B-Instruct-Q4_0.gguf

For more options see documentation on GitHub. For more modells see Alfonso² Peterssen page on HuggingFace.

Qwen 2.5-Coder

Download Qwen:

cd llama3.java
# Llama 3.2 (3B)
curl -L -O https://huggingface.co/mukel/Qwen2.5-Coder-3B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-3B-Instruct-Q4_0.gguf

Execute:

java --enable-preview --source 21 --add-modules jdk.incubator.vector Llama3.java -i --model Qwen2.5-Coder-3B-Instruct-Q4_0.gguf

Unfortunatly it failed:

java --enable-preview --source 21 --add-modules jdk.incubator.vector Llama3.java -i --model Qwen2.5-Coder-3B-Instruct-Q4_0.gguf
WARNING: Using incubator modules: jdk.incubator.vector
Note: Llama3.java uses preview features of Java SE 21.
Note: Recompile with -Xlint:preview for details.
Parse Qwen2.5-Coder-3B-Instruct-Q4_0.gguf: 318 millis
Load LlaMa model: 258 millis
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.lang.Integer.intValue()" because the return value of "java.util.Map.get(Object)" is null
        at com.llama4j.ModelLoader.loadModel(Llama3.java:696)
        at com.llama4j.ModelLoader.loadModel(Llama3.java:686)
        at com.llama4j.Llama3.main(Llama3.java:274)