Home
Services
Local AI Deployment Test Results Image & Video NeoCloud Results Offline Private Security System Digital Transformation Professional Photography
Team
Bruce Experience Education Contact

Local AI Test Results

Real-world speed measurements from our two test workstations: a dual Radeon RX 7900 XTX build and an RTX 3080 Ti + RTX 3080 build. Every model runs on our own hardware, and the launchers, settings and raw results are published on GitHub.Real-world speeds from our AMD and NVIDIA test workstations.

Real-World Test Results: AMD Radeon

Results from the current build of our AMD workstation, from its latest runs: short tool-call checks on November 9, 2026, in the final card layout, with prefill calculated from time to first token. The launchers and configuration are published on GitHub.A real-world example of our testing results.

The Machine

Two Radeon GPUs, 48 GB of VRAM

An Intel Core i5-13600KF on an ASUS B760M board with 64 GB of DDR5 and two AMD Radeon RX 7900 XTX cards, on Ubuntu. Both cards sit behind a Broadcom PCIe switch at x8 each, and direct GPU-to-GPU transfers through the switch are turned on (about 13.8 GB/s each way when tested). Every model is split across both cards.Core i5 with two Radeon RX 7900 XTX cards behind a PCIe switch on Ubuntu, each model split across both.

Ubuntu2× RX 7900 XTXPCIe switchROCm 7.2
Runtimes

Five Engines, One API

SGLang and vLLM on ROCm with tensor parallelism across both cards, llama.cpp on HIP for mixture-of-experts models, and Strata and Maya on HIP for models streamed from the SSD, each pinned to a tested version. One engine runs at a time behind an authenticated OpenAI-compatible gateway that starts or switches to whichever model a request names, so the same chat and agent tools reach every model.SGLang, vLLM, llama.cpp, Strata and Maya across both cards, one at a time behind one authenticated API that loads the requested model.

SGLangvLLMllama.cppStrata
Speed & Checks

Empero at 173 Tokens/s, 256K Context

Empero, a mixture-of-experts model kept entirely in GPU memory, runs at 173 tokens per second on the current build with a 256K context. Nine model routes are qualified on this build: Qwen3.8-27B on SGLang and vLLM, Nemotron Bolt and Ripple-65B in GPU memory, and Qwen3.8-Flash-Next, DeepSeek V4.1 Flash and GLM-5.3-Flash streamed from the SSD by Strata and Maya.Empero at 173 tokens per second with a 256K context on the current build, and 9 model routes qualified.

VRAM-only MoE256K contextSSD streaming
Our AMD test workstation in its current layout: an open white case with a Corsair CPU cooler, and in front of it a Sapphire Radeon RX 7900 XTX and a second RX 7900 XTX on an external PCIe switch board.
The current build: both RX 7900 XTX cards on the PCIe switch board in front of the case.
Grouped by where each model’s weights live while running, then by model where a model has several quantizations or engines, and otherwise by engine, fastest decode first. One row per model and engine: the best measured result for that engine on the current build. Measured November 9 and 18, 2026; see How we test below.
Model / quantEngineContextPrefill t/sTime to first tokenDecode t/s
VRAM-only models
Qwen3.8-27B
INT421.3 GB · SATA SSD · short tool-call replyTool callsSGLang TP2 + DFlash2160K1,736input ÷ time to first token3.86 s211.2
INT420.1 GB · SATA SSD · short tool-call replyTool callsvLLM TP2160K1,766input ÷ time to first token3.64 s155.1
llama.cpp
Empero Qwen3.8-35B-A3B IQ4_XS19.6 GB · SATA SSD · short tool-call replyTool callsllama.cpp HIP + MTP3256K2,893input ÷ time to first token2.22 s173.3
vLLM
Nemotron 3.5 Bolt 74B AutoRound W4A1643.7 GB · SATA SSD · short tool-call replyTool callsvLLM TP2 + Triton int4 MoE256K1,963input ÷ time to first token3.55 s40.1
Ripple-65B Preview GPTQ W4A1635.2 GB · SATA SSD · short tool-call replyTool callsvLLM TP264K764input ÷ time to first token8.18 s28.0
MoE models with SSD streaming
Strata
Qwen3.8-Flash-Next IQ3_XXS79.1 GB · NVMe SSD · short tool-call replyTool callsStrata HIP + MTP256K740input ÷ time to first token8.38 s86.3
DeepSeek V4.1 Flash 1017
UD-IQ1_S233.0 GB · NVMe SSD · short tool-call replyTool callsMaya HIP32K35input ÷ time to first token182.22 s6.8
UD-IQ1_S233.0 GB · NVMe SSD, shared with Maya · short tool-call replyTool callsStrata HIP32K111input ÷ time to first token57.21 s5.6
Maya
GLM-5.3-Flash IQ2_XXS97.4 GB · NVMe SSD · short tool-call replyTool callsMaya HIP32K57input ÷ time to first token105.75 s5.2

Quality ranking

Ranked by correct answers out of 216 items sampled from public benchmarks (MMLU-Pro, MATH-500, AIME, BigCodeBench, IFEval, BFCL and MuSR), one attempt each at temperature 0 with thinking off, through the normal API on November 11 to 17, 2026; category columns count correct answers. Each model is listed once, with its best-scoring engine (DeepSeek also runs on Strata and Qwen3.8-27B on vLLM, each scoring within the margin). Ranks 1 to 3 are not statistically separated, nor are ranks 4 to 7. Ripple’s score is a lower bound: 65 of its answers hit the length limit while still reasoning. The AMD and NVIDIA tests use different items, so their scores are not comparable.
Model / quantRankScore (95% interval)Code (20)Instructions (32)Knowledge (60)Math (40)Reasoning (32)Tools (32)
Qwen3.8-Flash-Next IQ3_XXSStrata (RDNA3 fork)178.2%72.3–83.2%123048332323
DeepSeek V4.1 Flash 1017 UD-IQ1_SMaya277.8%71.8–82.8%122851292226
Qwen3.8-27B INT4SGLang375.5%69.3–80.7%83147321827
Empero Qwen3.8-35B-A3B IQ4_XSllama.cpp469.9%63.5–75.6%112547241628
Nemotron 3.5 Bolt 74B AutoRound W4A16vLLM566.2%59.7–72.2%72346301522
GLM-5.3-Flash IQ2_XXSMaya663.0%56.3–69.1%102543112126
Ripple-65B Preview GPTQ W4A16vLLM762.0%55.4–68.2%51742321424
Test configuration
ProcessorIntel Core i5-13600KF
MotherboardASUS TX GAMING B760M WIFI
Memory64 GB DDR5 (61.6 GiB usable)
Graphics2× AMD Radeon RX 7900 XTX, 24 GB each (48 GB in total)
GPU linksBoth cards behind a Broadcom PEX88096 PCIe switch at x8 each (Gen4 x16 uplink); direct GPU-to-GPU transfers are on, about 13.8 GB/s each way when tested
PowerTwo power supplies: one for the PCIe switch board and both cards, one for the rest of the system
Model storageNVMe and SATA SSDs
Operating systemUbuntu, Linux kernel 7.0
GPU softwareROCm 7.2.3 (HIP)
Results measured onQwen3.8-Flash-Next: November 18, 2026, on the Strata RDNA3 fork with no fixed expert RAM budget. All other rows: the current build in its November 7 card layout, November 9, 2026, one short tool-call request each, with direct GPU-to-GPU transfers on (GLM-5.3-Flash with a longer watchdog interval while it loads)
How we test
  • One model is loaded at a time, split across both cards. Model loading and warm-up runs are excluded from every figure.
  • Every row uses the latest run on the current build. Where the latest run did not measure a value, the cell shows the most recent earlier current-build result and is labelled with its date.
  • Prefill and time to first token (November 9): one real tool-call request of about 5,900 to 7,000 input tokens, with 25 tool definitions, through the normal API. The client records only time to first token (from sending the request to the first content, including tool-call parsing) and the total request time, so prefill is calculated as input ÷ time to first token.
  • Decode (November 9): the tool-call reply’s output tokens divided by the time from the first token to the end of the request. The reply is a tool call of about 54 to 171 tokens, so it is a short burst rather than a sustained rate.
  • Context is the window the installed profile is configured for.
  • Checks passed: Tool calls means the model completed a tool round trip and a follow-up answer through the normal API.
  • Since November 9, 2026, engine updates are checked every day and installed by hand instead of by a nightly updater: each new version is taken unless it really breaks something (a failed tool-call check, a crash, under half the current speed, or more than three times the time to first token).

Setup, launchers and full results on GitHub →

NVIDIA Hardware & Results

Current hardware · Ubuntu: RTX 3080 Ti + RTX 3080 · 22 GB VRAM

Our NVIDIA test workstation. All results below were measured on November 5 and 6, 2026, on its current build with 96 GB of RAM.Our NVIDIA test workstation.

The Machine

Two GeForce Cards, 22 GB of VRAM

An Intel Core i9-13900KS on an MSI Z690 board with 96 GB of DDR5, on Ubuntu 26.04 with NVIDIA driver 615. The RTX 3080 Ti 12 GB and RTX 3080 10 GB each have their own PCIe 4.0 x8 link to the CPU, with no direct GPU-to-GPU path, and the processor’s integrated graphics drives the display. Most profiles split the model across both cards.Core i9, 96 GB of RAM, and an RTX 3080 Ti and RTX 3080 on Ubuntu.

Ubuntu 26.04RTX 3080 Ti + 3080Core i9-13900KS96 GB RAM
Runtimes

Five Engines, One API

HyperQwen on vLLM and SGLang for dense Qwen3.8-27B, kept entirely in GPU memory; and llama.cpp, Strata and Maya for mixture-of-experts models that stream their experts from system memory and the SSD. A model router behind an authenticated OpenAI-compatible API loads one model at a time.HyperQwen, SGLang, llama.cpp, Strata and Maya, one model at a time behind one authenticated API.

HyperQwenSGLangllama.cppStrataMaya
Speed & Checks

Up to 84 Tokens/s on 22 GB

Strata streams the IQ3_XXS build of Qwen3.8-Flash-Next from the SSD at 84 tokens per second with a 128K context. Nemotron Bolt, in NVIDIA’s native 2-bit build with part of it in system memory, runs at 68 tokens per second with a 160K context, and HyperQwen runs the 4-bit Qwen3.8-27B entirely in VRAM at 63 tokens per second with an 84K context.Up to 84 tokens per second streaming from the SSD, 68 for Bolt and 63 entirely in VRAM.

128K ContextAll in VRAMSSD Streaming
Our NVIDIA test workstation in its current build: an open white case with a Thermalright CPU cooler and two lit memory sticks above two GeForce RTX graphics cards installed directly in the motherboard.
The current build: the RTX 3080 Ti and RTX 3080 in the motherboard’s own slots, under a Thermalright CPU cooler.
Grouped by where each model’s weights live while running, then by model where a model has several quantizations or engines, and otherwise by engine, fastest decode first. One row per model and engine, with every value in a row from the same run. Measured November 5 and 6, 2026; see How we test below.
Model / quantEngineContextPrefill t/sTime to first tokenDecode t/s
VRAM-only models
Qwen3.8-27B
AutoRound INT415.8 GB · SATA SSD · short tool-call replyTool callsHyperQwen vLLM TP284K2,094.62.80 s62.7
AutoRound INT415.8 GB · SATA SSD, shared with HyperQwen · short tool-call replyTool callsSGLang TP256K2,130.12.90 s60.12
Models partly in system memory
llama.cpp
Nemotron 3.5 Bolt 74B native IQ222.0 GB · SATA SSD · short tool-call replyTool callsllama.cpp CUDA160K673.89.60 s67.8
MoE models with SSD streaming
Qwen3.8-Flash-Next
IQ3_XXS75.8 GB · NVMe SSD · short tool-call replyTool callsStrata CUDA128K1,058.05.40 s83.7
UD-IQ4_XS-v3193.7 GB · NVMe SSD · short tool-call replyTool callsStrata CUDA128K759.97.63 s43.7
Maya
GLM-5.3-Flash IQ2_XXS94.7 GB · NVMe SSD · short tool-call replyTool callsMaya CUDA64K550.910.06 s14.2
DeepSeek V4.1 Flash 1017
UD-IQ1_S233.0 GB · NVMe + SATA SSDs · short tool-call replyTool callsStrata CUDA32K87.165.59 s12.00
UD-IQ1_S233.0 GB · NVMe + SATA SSDs, shared with Strata · short tool-call replyTool callsMaya CUDA32K209.428.84 s8.3

Quality ranking

Ranked by the average of five public-benchmark accuracies on 390 items (MMLU-Pro 210, AIME 2025 30, LiveCodeBench 60, BFCL single-turn 60 and multi-turn 30), one attempt each with thinking off, through the normal API from November 10 to 12, 2026. No model is statistically separated from the one next to it, where the same weights run on two engines, only the higher score is listed (Qwen3.8-27B also runs on HyperQwen and DeepSeek on Maya, each scoring the same); DeepSeek is ahead of both Flash-Next quantizations, GLM and Bolt, and Bolt is behind every model except GLM. Flash-Next UD-IQ4_XS-v31 was added on November 18 from its trial run outside the router (same 390 items and launch settings, November 17 and 18); it is not separated from IQ3_XXS or from Qwen3.8-27B. The AMD and NVIDIA tests use different items, so their scores are not comparable.
Model / quantRankScore (95% interval)MMLU-ProAIME 2025LiveCodeBenchBFCL single-turnBFCL multi-turn
DeepSeek V4.1 Flash 1017 UD-IQ1_SStrata173.0%67.3–78.4%86.7%76.7%55.0%90.0%56.7%
Qwen3.8-27B AutoRound INT4SGLang268.1%62.3–73.9%85.7%66.7%50.0%91.7%46.7%
Qwen3.8-Flash-Next UD-IQ4_XS-v31Strata365.5%60.2–70.8%87.6%80.0%58.3%78.3%23.3%
Qwen3.8-Flash-Next IQ3_XXSStrata462.0%56.5–67.5%85.2%73.3%58.3%73.3%20.0%
GLM-5.3-Flash IQ2_XXSMaya557.2%52.1–62.3%86.2%6.7%55.0%91.7%46.7%
Nemotron 3.5 Bolt 74B native IQ2llama.cpp653.9%48.7–59.4%76.2%50.0%33.3%90.0%20.0%
Test configuration
ProcessorIntel Core i9-13900KS, with integrated graphics for the display
MotherboardMSI MPG Z690 Carbon WiFi
Memory96 GB DDR5 (about 91.8 GiB usable)
GraphicsNVIDIA GeForce RTX 3080 Ti 12 GB + RTX 3080 10 GB (22 GB)
GPU linksEach card on its own PCIe 4.0 x8 link to the CPU; no direct GPU-to-GPU path, so the cards exchange data through system memory
Model storageNVMe system drive for the streamed models; Samsung 870 QVO SATA SSD for Qwen3.8-27B, Nemotron Bolt and part of DeepSeek
Operating systemUbuntu 26.04, Linux kernel 7.0
GPU softwareNVIDIA driver 615.71.09, at stock power limits
Results measured onHyperQwen: November 9, 2026, after updating to vLLM 0.33.0. Qwen3.8-Flash-Next: November 16, after updating to Strata 0.2.15. Qwen3.8-Flash-Next UD-IQ4_XS-v31: November 18, first request after a fresh load, same Strata engine. GLM and DeepSeek on Maya: November 18, after updating to Maya 1.1.67. Nemotron Bolt: November 13, after updating to llama.cpp 1.0.0, with its prompt cache in RAM. SGLang and DeepSeek on Strata: November 5, in a fresh rerun of every model. All on the current build with 96 GB of RAM
How we test
  • One model is loaded at a time, split across both cards. Model loading is excluded, and the operating system’s file cache is not flushed.
  • Each row is one real tool-call request of about 5,400 to 6,400 input tokens and a few dozen output tokens through the normal API, followed by an exact follow-up answer; we keep the fastest successful run.
  • Prefill and decode are the engine’s own counters; time to first token runs from sending the request to the first content, including tool-call parsing.
  • Context is the window the installed profile runs with.
  • Checks passed: Tool calls means the model completed a tool round trip and an exact follow-up answer through the normal API.
  • Since November 9, 2026, engine updates are checked every day and installed by hand instead of by a nightly updater: each new build is staged separately, checked with the same short tool-call test before and after the switch, and kept unless it really breaks (a failed check, under half the current speed, or more than three times the time to first token).

Setup and full results on GitHub →

Image and Video Generation

Image and video diffusion models measured in ComfyUI on our AMD and NVIDIA workstations: Qwen-Image, FLUX 3 Image Klein, MiniMax H3, Wan 2.2 and LTX 2.3.Image and video models measured in ComfyUI.

See the image and video results →

Ready to get started?

Tell us about your environment, we’ll design a deployment that keeps your data yours.

Get in Touch