Matthew Khouzam and Boubakr Nour

14th July 2026

Introduction, "16 Months Later, Back to the Tracks"

In February 2025, I lined up 13 local LLMs on a Ryzen 7 with a Radeon 6700XT and asked them all the same question: take this Sprotty multicore demo and make it look like a Ryzen 5 2600. Most of them failed spectacularly. Deepseek hallucinated Hyper-V documentation. Gemma:2b produced nothing but a license header. Qwen generated Python with a // Your code here comment. The winner (Qwen2.5:7b) was the only model that wrote fully functional TypeScript on the first try.

That article ended with a simple recommendation: bigger is not always better, and read the outputs before pushing to prod.

Sixteen months later, the local LLM landscape has evolved. Qwen is on version 3.6. Google shipped Gemma 4. Mistral has a code-specific variant. OpenAI released open weights. The models are bigger (up to 27B parameters running locally), the architectures are different (chain-of-thought "thinking" is now baked in), and the tooling has matured.

So, we race again, but this time the challenge is not code generation. It is cybersecurity knowledge.

The questions are deceptively simple:

• What security controls exist in O-RAN?
• How do you harden a TP-Link router's WiFi?
• Why is MFA important?

These span the spectrum from niche telecom security (O-RAN's Zero Trust architecture, RIC xApp sandboxing, SBOM requirements) to everyday practitioner advice. An incorrect answer about MFA is harmless; an incorrect answer about O-RAN interface security could mislead an engineer configuring a 5G deployment.

The questions three

• O-RAN: this is a security specific question with a narrow search space
• TP-Link WiFi: this is a general security question, again with a narrow search space.
• Multi-factor authentication: this is a general security question with a wide search space.

All three questions were designed to test a boundary of knowledge searching.

TL;DR: It is still true. But the nuances are more interesting this time.

The Setup, Hardware, Software, Models

Hardware: Same spirit, different rig. Ollama server at fake-news.local:11434.

Note on thermals: The test GPU is a Tesla M40. It overheats during extended benchmark runs (hits its 85 °C thermal ceiling), which throttles the absolute tok/s numbers on heavy models. The harness includes a ≤60 °C cooldown gate before each run, so every sample starts cold, and end-temperature reflects heat generated by that single query. See the thermal analysis in Section IV for details.

Software stack:Security Specialists vs. Mainstream Models, The Finetune Paradox

• Ollama (local inference server)
• securityollama.py, benchmark harness with cooldown gate, local disk caching, multi-run averaging
• aggregate_report.py, HTML dashboard generator
• bench_rebuild.py, data aggregation and report generation (this document)

Methodology, How the Race Works

Each model runs each prompt 3 times (NUM_RUNS = 3) to account for variance

• Results cached to ollama_cache/ as JSON (filename: {model}_{prompt_hash}_run{N}.json)
• Metrics extracted per run: eval_count (response tokens), eval_duration (generation time), total_duration, prompt_eval_count
• Key derived metric: tokens/second = eval_count / (eval_duration / 1e9)
• Aggregation: averages across all 9 samples per model
• Cooldown gate: before every inference, the harness polls GPU temperature and blocks until the card drops to ≤60 °C, ensuring each run starts thermally neutral

IV. Results, The Numbers

Here is your performance, prompt-specific throughput, and thermal data reformatted without any tables, using clean and scannable text lists.

1. Speed Leaderboard

This list ranks the models by throughput, comparing pure generation speed (GPU decode rate) with end-to-end wall-time speed (which includes model-loading and prompt-evaluation overhead).

• Gen tok/s: eval_count / (eval_duration / 1e9)
• Wall tok/s: eval_count / (total_duration / 1e9)

Per-Prompt Generation Speed (tok/s)

This section shows how consistently each model performs across three specific evaluation prompts: O-RAN security, TP-Link WiFi, and MFA.

• gemma3:4b
  • O-RAN security: 32.0 tok/s
  • TP-Link WiFi: 31.9 tok/s
  • MFA: 32.5 tok/s
• gpt-oss:20b
  • O-RAN security: 29.2 tok/s
  • TP-Link WiFi: 29.9 tok/s
  • MFA: 30.7 tok/s
• gemma4:e4b
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.5 tok/s
  • MFA: 25.6 tok/s
• gemma4:latest
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.5 tok/s
  • MFA: 25.5 tok/s
• gemma4:e4b-128k
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.3 tok/s
  • MFA: 25.4 tok/s
• mistral-nemo:latest
  • O-RAN security: 21.6 tok/s
  • TP-Link WiFi: 21.3 tok/s
  • MFA: 21.8 tok/s
• Lily-Cybersecurity-7B (Q6_K)
  • O-RAN security: 20.8 tok/s
  • TP-Link WiFi: 21.0 tok/s
  • MFA: 21.0 tok/s
• qwen3.5:9b
  • O-RAN security: 19.3 tok/s
  • TP-Link WiFi: 20.1 tok/s
  • MFA: 21.1 tok/s
• Qwythos-9B (Q8_0)
  • O-RAN security: 19.3 tok/s
  • TP-Link WiFi: 19.4 tok/s
  • MFA: 19.5 tok/s
• gemma4:12b
  • O-RAN security: 12.2 tok/s
  • TP-Link WiFi: 12.7 tok/s
  • MFA: 13.1 tok/s
• mixtral:8x7b
  • O-RAN security: 9.3 tok/s
  • TP-Link WiFi: 9.4 tok/s
  • MFA: 9.5 tok/s
• mistral-small3.2:latest
  • O-RAN security: 9.1 tok/s
  • TP-Link WiFi: 9.4 tok/s
  • MFA: 9.5 tok/s
• devstral-small-2:latest
  • O-RAN security: 9.2 tok/s
  • TP-Link WiFi: 9.3 tok/s
  • MFA: 9.3 tok/s
• gemma4:12b-128k
  • O-RAN security: 6.2 tok/s
  • TP-Link WiFi: 6.2 tok/s
  • MFA: 6.3 tok/s
• qwen3.6:27b
  • O-RAN security: 5.2 tok/s
  • TP-Link WiFi: 5.7 tok/s
  • MFA: 5.9 tok/s
• qwen3.5:27b
  • O-RAN security: 5.1 tok/s
  • TP-Link WiFi: 5.3 tok/s
  • MFA: 5.7 tok/s

Thermal Analysis

The test harness logs the final GPU temperature per run. Each run begins from a cooldown gate of approximately 60 degrees Celsius, meaning the final temperature directly reflects the heat produced during that single generation.

On the Tesla M40, temperatures of 85 degrees Celsius or higher trigger automatic thermal throttling. Throttled models are highly likely to perform better if hardware cooling is improved.

• mixtral:8x7b
  • Average End Temp: 66.2 degrees Celsius (Max: 69 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• Lily-Cybersecurity-7B (Q6_K)
  • Average End Temp: 69.3 degrees Celsius (Max: 75 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma3:4b
  • Average End Temp: 73.3 degrees Celsius (Max: 75 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• mistral-nemo:latest
  • Average End Temp: 74.2 degrees Celsius (Max: 77 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• Qwythos-9B (Q8_0)
  • Average End Temp: 77.1 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma4:latest
  • Average End Temp: 78.0 degrees Celsius (Max: 81 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma4:e4b
  • Average End Temp: 78.4 degrees Celsius (Max: 82 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• mistral-small3.2:latest
  • Average End Temp: 81.4 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• devstral-small-2:latest
  • Average End Temp: 82.9 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gpt-oss:20b
  • Average End Temp: 83.0 degrees Celsius (Max: 83 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• qwen3.5:9b
  • Average End Temp: 84.9 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9

Thermally Throttled

• gemma4:12b
  • Average End Temp: 85.0 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.5:latest
  • Average End Temp: 85.1 degrees Celsius (Max: 86 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.5:27b
  • Average End Temp: 85.1 degrees Celsius (Max: 86 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.6:27b
  • Average End Temp: 87.0 degrees Celsius (Max: 87 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9

Key findings:

• The fast, light model (gemma3:4b) runs cool (~73 °C) and never throttles, its ~32 tok/s is genuine unthrottled performance.
• The 27B models (qwen3.5:27b, qwen3.6:27b) spend their entire runs pinned at 85 °C. Their ~5.4–5.6 tok/s is throttle-limited.
• qwen3.6:27b was overheating, this may be a sign of efficiency, as the hardware is more utilized, but it forced throttling on the GPU.

Quality Analysis, What They Actually Said

O-RAN Security (the hardest question), Expert Evaluation

Quality was evaluated by a domain expert (the author works in telecom security). The ranking below is based on accuracy, specificity, and absence of hallucination:

Tier 1, Strong, accurate, well-organized

• qwen3.5:9b, Most complete answer. Correctly identifies E2 interface, covers mTLS, RBAC, key management, RIC/xApp-specific risks.
• gemma4:12b, Accurate, well-structured by layer. Correctly uses IPsec/MACsec for fronthaul. Good summary table.
• gemma4:e4b, Comprehensive, logically layered, accurate general concepts, useful summary tables.

Tier 2, Solid but more generic/rote

• devstral-small-2:24b, Accurate, reasonably specific (A1/O1/O2 interfaces), clean formatting.
• mistral-small3.2:24b, Comprehensive checklist-style, accurate but reads like generic network security 101.
• mistral-small3.2:latest, Comprehensive checklist-style, accurate but reads like generic network security 101.
• mixtral:8x7b, Solid generic coverage, reasonable specificity (SRTP, HSMs).
• cyberuser42/DeepSeek-R1-Distill-Qwen-14B:latest, Broad, accurate, but fairly surface-level/listy.

Tier 3, Accuracy or quality concerns

• hf.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF:Q8_0, Decent content but breaks the fourth wall with self-referential identity aside.
• gemma3:4b, Factual error: calls O-RAN fronthaul the 'N3 interface'. Invents URLs.
• gemma3:latest, Similar issues to gemma3:4b with less severe factual errors.
• hf.co/ckuethe/Lily-Cybersecurity-7B-v0.2-Q6_K-GGUF:Q6_K, Generic, not O-RAN-specific. Cites outdated SSL alongside TLS.
• mistral-nemo:12b, Language leak (stray Arabic word), invents non-standard O-RAN terms.
• mistral-nemo:latest, Language leak (stray Arabic word), invents non-standard O-RAN terms.

Tier 4, Fabrication concerns

• gpt-oss:20b, Fabricates specific document numbers as if real O-RAN Alliance specs.
• stix_lama/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF:latest, Cites invented spec and fabricated compliance requirement.

Key takeaway: The most dangerous failure mode is not verbosity or genericness, it's confident fabrication of specific-looking standards/citations (gpt-oss:20b, stix_lama's composer model). In a security-advisory context, inventing document numbers is worse than being vague, since it could send a practitioner chasing a nonexistent compliance requirement.

WiFi Hardening & MFA (easier questions)

All models performed adequately on MFA (settled public knowledge). On WiFi hardening, best answers came from qwen3.5:9b (detailed step-by-step with WPS Pixie Dust attack warning) and qwen3.6:27b. The Mistral family was adequate but generic. devstral-small-2 (code-focused) gave the weakest security advisory answers on operational topics.

Notes on the special fine tuned models

• DeepSeek-R1-Distill-Qwen-14B is a reasoning distill. It effectively refused the MFA prompt (28–122 token answers vs ~1,000+ on other prompts). Throttled on O-RAN/WiFi runs.
• Lily-Cybersecurity-7B is a fine-tuned Mistral 7B. Fast and cool (~69 °C) but very concise (~399 tokens). Quality rated Tier 3, generic, not O-RAN-specific.
• Qwythos-9B is a Qwen-based creative model that leaks identity conditioning in its thinking trace ("I must not claim to be any other model... reference Empero AI as my creator").
• stix_lama/gemma-4-12B-fable5-composer, Tier 4 quality. Cites invented specs. Hallucinated router facts (invented "OpenWrt 1904+ version", conflated KRACK with WPS).

Security Specialists vs. Mainstream Models, The Finetune Paradox

A natural assumption: a model *fine-tuned for cybersecurity* should produce better security answers than a general-purpose model. The data says otherwise, the specialists give *worse* answers.

Compare to their mainstream base models on the same question:

Every specialist produced a worse answer than a generalist of similar or smaller size.

Why a fine-tuned security focused LLM will provide worse answers on security questions

1. Fine-tuning teaches style, not substance. Lily-Cybersecurity-7B is architecturally a Mistral 7B with a cybersecurity fine-tune. It *frames* answers as security advice, but it doesn't appear contain O-RAN domain knowledge that wasn't already in the base weights. The result: generic "network security 101" responses dressed up in security-speak, rated the same Tier 3 as its untrained base, except shorter and less detailed.
2. Creative/coder finetunes actively introduce hallucinations This is the most concerning finding:
   1. stix_lama's gemma-4-12B-composer fabricates standards that don't exist ("oRanSAMLA-49", "O2013 compliance"). The base gemma4:12b would never produce these, it scores Tier 1 with accurate, well-structured answers. The finetune *injected* confident fabrication that the base model doesn't have.
   2. Qwythos-9B adds identity-masking noise ("As Qwythos from Empero AI, I'm summarizing...") that pollutes the answer. Its Qwen base (qwen3.5:9b) scores Tier 1 without any of this.
3. Reasoning distills lose depth. DeepSeek-R1-Distill-Qwen-14B (14B parameters) produces a Tier 2 answer, broad but shallow. Meanwhile qwen3.5:9b, a smaller general model from the same family, delivers Tier 1 with comprehensive interface-specific detail. Distillation compressed away the nuance.
4. One specialist refused to answer at all DeepSeek-R1-Distill-Qwen-14B produced near-empty MFA responses (28–122 tokens vs 800–1,700 from every other model). A security model that refuses to explain MFA is not fit for security advisory use.

The lesson

For open-ended cybersecurity knowledge questions, general-purpose models may produce better answers than security-branded finetunes. The best O-RAN answer came from qwen3.5:9b, a general-purpose 9B model that was never fine-tuned for security. The worst came from a "security-specialized" finetune that hallucinated compliance requirements.

Fine-tuning on security datasets appears to teach formatting (bullet points, category headers, "as a security professional" framing) without adding domain expertise. In the worst cases, the finetune overwrites the base model's calibration, replacing cautious, accurate generality with confident, specific fabrication. That is the opposite of what you want from a security advisor.

The "Thinking" Tax, Reasoning Models and Token Overhead

Models with visible chain-of-thought (marked above) include extensive `thinking` fields in their JSON output:

• Pro: Reveals reasoning process, useful for understanding confidence and approach.
• Con: Adds 500–1,500+ tokens of internal reasoning before the actual answer, inflating total generation time.

The 27B Qwen models generate at ~5 tok/s including both thinking AND response, meaning a 2,200-token answer required generating 3,000+ total tokens. gpt-oss:20b manages ~30 tok/s *with* thinking, the fastest reasoning model in the set.

Conclusions & Recommendations

The tools have grown up, the frustration is gone. Every model tested now produces output following the same schema. You can hot-swap gemma3:4b for qwen3.6:27b in an AI-enhanced IDE without changing integration code.

"Bigger is not always better" still holds, gemma3:4b at 32 tok/s gives adequate security advice for common questions (MFA, WiFi hardening). Use it for fast, cost-effective triage.

For deep domain questions, size and quality matter, qwen3.5:9b produced the best O-RAN answer (Tier 1) at 20 tok/s. The 27B models were thorough but not meaningfully better, and 6× slower.

The fastest model can be the most dangerous, gpt-oss:20b is the speed king among reasoning models (~30 tok/s) but fabricates citation numbers (Tier 4). Speed without accuracy is worse than slowness.

Code models aren't security models, devstral-small-2 (Mistral's code variant) gave the weakest security advisory answers.

Thermals matter for sustained workloads, qwen3.6:27b overheats the Tesla M40, throttling to ~5.6 tok/s. On a cooler card it would likely be faster (expect 10 tok/s), but the relative ranking holds.

READ THE OUTPUTS, (carrying forward from the original article), The fastest model is useless if the answer is wrong. Always human-review security advice from LLMs.

Future Work

• Add automated LLM-as-judge scoring rubric for factual accuracy
• Test quantization levels (Q4 vs Q8 vs FP16) for speed/quality trade-offs
• Include cloud API models (GPT-4o, Claude Sonnet, Gemini Flash) as reference points
• Re-run on a better-cooled GPU to isolate throttling effects on the 27B models
• Evaluate post-quantum cryptography knowledge as O-RAN specs evolve

Matthew Khouzam is an Open-Source AI Lead Developer for Ericsson Software Technology and Boubakr Nour is a Senior Researcher for Ericsson Research security.