Garlic LLM Runs Qwen3.5-9B Offline on a Jailbroken PS4 at About 35 Tokens per Second

garlic llm

Developer Leonardo Alegría has released Garlic LLM, an unofficial open-source homebrew app that runs Alibaba’s Qwen3.5-9B large language model directly on the GPU of a jailbroken PlayStation 4. It works entirely on the console, with no PC and no internet connection. Beyond text chat, the app includes offline speech recognition and voice synthesis, and the console can run the Python code the model writes at the press of a single button.

In text chat the model generates 31 to 39 tokens per second, and on code it reaches 43 tokens per second. For comparison, the same Qwen3.5-9B ran at 60 tokens per second on a laptop GeForce RTX 3070 Ti with 8 GB of memory in a test by the Architecture & Performance blog, while on the neural processing unit (NPU) of a Rockchip RK3588 board it managed 3.2 tokens per second, according to Qengineering.

The code is available on GitHub, and the all-in-one package with the models (about 5.7 GB) can be downloaded from Hugging Face.

How to run a neural network without standard ML libraries

The PS4 gives homebrew software no compute API for using the GPU outside of games and no machine learning libraries. To get around this, the developer had to build the whole inference stack at the lowest level:

  • Direct GPU commands. All of the model’s math (matrix multiplication, attention, normalization) was written from scratch as OpenCL C kernels. The LLVM compiler toolchain then turns them into machine code for the AMD GCN graphics architecture used by the console’s chip.
  • Bypassing the graphics driver. The app writes low-level command packets (PM4) itself and sends them straight to the GPU, skipping the PS4 operating system’s graphics driver.
  • Mixed quantization. To fit the model into the console’s memory, which is modest by today’s standards, the developer compressed the weights unevenly:
    • Most parameters are stored in a 4-bit format (Q4).
    • The two heaviest matrices in each layer (the gate and up projections) are squeezed further, down to 3 bits (Q3). According to the author, keeping the whole model in 4 bits would take about 384 MiB more memory than the console has free.
  • Compressed conversation memory (KV cache). The attention cache, where the model keeps the context of the conversation, is quantized to 4 bits as well. That is what makes a 16K-token context window fit, while the full Qwen3.5-9B supports up to 262,144 tokens natively.

The project is named after Garlic, the PS4’s name for the region of memory the GPU reads directly. According to the GitHub description, the model weights take up 4.32 GiB, with another 0.20 GiB for the multi-token prediction (MTP) head. It drafts up to two tokens ahead, and the full model verifies them in a single pass, which noticeably speeds up generation.

Performance: PS4 vs. a laptop GPU and embedded hardware

Device and setup Runtime and quantization Generation speed
PS4 (Garlic LLM) – text chat Custom GCN kernels, Q4/Q3 weights, MTP speculative decoding 35 tokens/s (range 31–39)
PS4 (Garlic LLM) – voice mode Same, but the MTP memory goes to the voice models 22 tokens/s
Laptop with GeForce RTX 3070 Ti (8 GB) llama.cpp, Unsloth UD-Q4_K_XL (5.32 bits per weight) 60 tokens/s
Rockchip RK3588 board (NPU) RKLLM, 8-bit weights and activations (W8A8) 3.2 tokens/s

Data: the Garlic LLM repository on GitHub, Architecture & Performance and Qengineering. For PS4 text chat, 35 tokens/s is the midpoint of the 31–39 tokens/s range given by the author; the measurements were taken by different people under different conditions.

Key takeaways

  • Hardware efficiency. By the editors’ estimate (35 / 3.2 ≈ 10.9), the PS4 runs the model about 11 times faster than the dedicated NPU board based on the RK3588. In the editors’ assessment, part of the gap comes from the 8-bit W8A8 format, which means processing twice as much weight data as the console’s 4-bit weights. The console trails the RTX 3070 Ti laptop GPU by only about 1.7x (60 vs. 35 tokens/s, editorial estimate).
  • What multi-token prediction (MTP) adds. In voice mode, the memory reserved for MTP is taken by the voice models, and speed drops from 31–39 to about 22 tokens/s. If the whole difference is due to MTP, speculative decoding speeds up generation by roughly 1.4–1.8x (editorial estimate: 31 / 22 ≈ 1.41; 39 / 22 ≈ 1.77).

Voice, requirements and the model itself

  • Voice mode. Speech is recognized by Parakeet TDT 0.6B v3, and answers are spoken by the Chatterbox engine. Spoken replies are available in seven languages: Spanish, English, Portuguese, Russian, Italian, French and German.
  • Requirements. The console needs custom firmware with the GoldHEN exploit enabled. The developer tested the app on a base PS4 and a PS4 Slim with firmware 11.52; users also report it working on a PS4 Pro (25–45 tokens/s), but the author has not tested it on a Pro.
  • What the model can do. In Alibaba’s official benchmarks, the original Qwen3.5-9B scores 82.5 on the MMLU-Pro academic benchmark, ahead of GPT-OSS-120B (80.8) and GPT-OSS-20B (74.8). On code generation (LiveCodeBench v6) it scores 65.6, versus 74.6 for GPT-OSS-20B and 82.7 for GPT-OSS-120B. These numbers apply to the full-size model, not the 3–4-bit version running on the console.

What’s next: an API for your home network

The developer plans to ship an OpenAI-compatible API in the next release. That will turn a jailbroken PS4 into a local model server that other devices on the home network can query. The limits: requests are handled one at a time, and tool calling is not supported.


bnr2mob
Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted