Matthew Khouzam and Boubakr Nour

14th July 2026

I. Introduction, "16 Months Later, Back to the Tracks"

In February 2025, I lined up 13 local LLMs on a Ryzen 7 with a Radeon 6700XT and asked them all the same question: take this Sprotty multicore demo and make it look like a Ryzen 5 2600. Most of them failed spectacularly. Deepseek hallucinated Hyper-V documentation. Gemma:2b produced nothing but a license header. Qwen generated Python with a // Your code here comment. The winner (Qwen2.5:7b) was the only model that wrote fully functional TypeScript on the first try.

That article ended with a simple recommendation: bigger is not always better, and read the outputs before pushing to prod.

Sixteen months later, the local LLM landscape has evolved. Qwen is on version 3.6. Google shipped Gemma 4. Mistral has a code-specific variant. OpenAI released open weights. The models are bigger (up to 27B parameters running locally), the architectures are different (chain-of-thought "thinking" is now baked in), and the tooling has matured.

So, we race again, but this time the challenge is not code generation. It is cybersecurity knowledge.

The questions are deceptively simple:

• What security controls exist in O-RAN?
• How do you harden a TP-Link router's WiFi?
• Why is MFA important?

These span the spectrum from niche telecom security (O-RAN's Zero Trust architecture, RIC xApp sandboxing, SBOM requirements) to everyday practitioner advice. An incorrect answer about MFA is harmless; an incorrect answer about O-RAN interface security could mislead an engineer configuring a 5G deployment.

The questions three

• O-RAN: this is a security specific question with a narrow search space
• TP-Link WiFi: this is a general security question, again with a narrow search space.
• Multi-factor authentication: this is a general security question with a wide search space.

All three questions were designed to test a boundary of knowledge searching.

What's different this time:

Feb 2025 RaceJuly 2026 RaceDomainTypeScript code generationSecurity advisory knowledgeModels13 models, 1.5B–14B params18 models, 4B–27B paramsPrompts1 (inferred, hard)3 (explicit, varying difficulty)Runs1 per model3 per prompt × 3 prompts = 9 per modelEvaluation"Did it compile?" (subjective)tok/s + response quality (quantitative + qualitative)ToolingManual (Eclipse Theia chat)Automated harness with caching & reporting (still verified with Eclipse Theia chat)

TL;DR: It is still true. But the nuances are more interesting this time.

II. The Setup, Hardware, Software, Models

Hardware: Same spirit, different rig. Ollama server at fake-news.local:11434.

Note on thermals: The test GPU is a Tesla M40. It overheats during extended benchmark runs (hits its 85 °C thermal ceiling), which throttles the absolute tok/s numbers on heavy models. The harness includes a ≤60 °C cooldown gate before each run, so every sample starts cold, and end-temperature reflects heat generated by that single query. See the thermal analysis in Section IV for details.

Software stack:

• Ollama (local inference server)
• securityollama.py, benchmark harness with cooldown gate, local disk caching, multi-run averaging
• aggregate_report.py, HTML dashboard generator
• bench_rebuild.py, data aggregation and report generation (this document)

Models under test (18 models, multiple families):

Google Gemma Family

Thinking: No

• gemma3:4b (~4B) | Gemma 3

Thinking: Yes

• gemma4:e4b (~12B) | Gemma 4
• gemma4:latest (~27B) | Gemma 4
• gemma4:e4b-128k (~12B) | Gemma 4, 128k context
• gemma4:12b (12B) | Gemma 4
• gemma4:12b-128k (12B) | Gemma 4, 128k context

Mistral Family

Thinking: No (Standard & Code)

• mistral-nemo:latest (12B) | Standard Mistral
• mixtral:8x7b (8x7B) | Mistral Mixture of Experts (MoE)
• mistral-small3.2:latest (24B) | Standard Mistral
• devstral-small-2:latest (~24B) | Mistral Code-focused

Thinking: No (Specialized Finetune)

• hf.co/ckuethe/Lily-Cybersecurity-7B-v0.2-Q6_K-GGUF:Q6_K (7B) | Cybersecurity finetune

Alibaba Qwen Family

Thinking: Yes

• qwen3.5:9b (9B) | Qwen 3.5
• qwen3.5:27b (27B) | Qwen 3.5
• qwen3.5:latest (~27B) | Qwen 3.5
• qwen3.6:27b (27B) | Qwen 3.6

OpenAI Open Weights

Thinking: Yes

• gpt-oss:20b (20B)

III. Methodology, How the Race Works

Reference: securityollama.py

• Each model runs each prompt 3 times (NUM_RUNS = 3) to account for variance
• Results cached to ollama_cache/ as JSON (filename: {model}_{prompt_hash}_run{N}.json)
• Metrics extracted per run: eval_count (response tokens), eval_duration (generation time), total_duration, prompt_eval_count
• Key derived metric: tokens/second = eval_count / (eval_duration / 1e9)
• Aggregation: averages across all 9 samples per model
• Cooldown gate: before every inference, the harness polls GPU temperature and blocks until the card drops to ≤60 °C, ensuring each run starts thermally neutral

IV. Results, The Numbers

Here is your performance, prompt-specific throughput, and thermal data reformatted without any tables, using clean and scannable text lists.

1. Speed Leaderboard

This list ranks the models by throughput, comparing pure generation speed (GPU decode rate) with end-to-end wall-time speed (which includes model-loading and prompt-evaluation overhead).

• Gen tok/s: eval_count / (eval_duration / 1e9)
• Wall tok/s: eval_count / (total_duration / 1e9)

Rank 1: gemma3:4b

• Throughput: 32.1 Gen tok/s | 30.3 Wall tok/s
• Response Stats: 1,110 average tokens in 36.7s
• Features: Thinking: No | Throttled: No

Rank 2: gpt-oss:20b

• Throughput: 29.9 Gen tok/s | 28.0 Wall tok/s
• Response Stats: 1,812 average tokens in 66.4s
• Features: Thinking: Yes | Throttled: No

Rank 3: gemma4:e4b

• Throughput: 25.4 Gen tok/s | 23.6 Wall tok/s
• Response Stats: 1,709 average tokens in 72.9s
• Features: Thinking: Yes | Throttled: No

Rank 4: gemma4:e4b-128k

• Throughput: 25.3 Gen tok/s | 24.0 Wall tok/s
• Response Stats: 1,697 average tokens in 71.3s
• Features: Thinking: Yes | Throttled: No

Rank 5: mistral-nemo:latest

• Throughput: 21.6 Gen tok/s | 19.4 Wall tok/s
• Response Stats: 500 average tokens in 26.8s
• Features: Thinking: No | Throttled: No

Rank 6: Lily-Cybersecurity-7B (Q6_K)

• Throughput: 21.0 Gen tok/s | 20.3 Wall tok/s
• Response Stats: 399 average tokens in 19.7s
• Features: Thinking: No | Throttled: No

Rank 7: qwen3.5:latest

• Throughput: 20.5 Gen tok/s | 20.2 Wall tok/s
• Response Stats: 2,173 average tokens in 108.6s
• Features: Thinking: Yes | Throttled: YES (Thermal ceiling reached)

Rank 8: qwen3.5:9b

• Throughput: 20.2 Gen tok/s | 19.6 Wall tok/s
• Response Stats: 2,359 average tokens in 121.7s
• Features: Thinking: Yes | Throttled: No

Rank 9: Qwythos-9B (Q8_0)

• Throughput: 19.4 Gen tok/s | 18.7 Wall tok/s
• Response Stats: 920 average tokens in 49.1s
• Features: Thinking: Yes | Throttled: No

Rank 10: gemma4:12b

• Throughput: 12.7 Gen tok/s | 12.0 Wall tok/s
• Response Stats: 1,511 average tokens in 126.5s
• Features: Thinking: Yes | Throttled: YES (Thermal ceiling reached)

Rank 11: mixtral:8x7b

• Throughput: 9.4 Gen tok/s | 8.4 Wall tok/s
• Response Stats: 377 average tokens in 57.0s
• Features: Thinking: No | Throttled: No

Rank 12: mistral-small3.2:latest

• Throughput: 9.3 Gen tok/s | 8.6 Wall tok/s
• Response Stats: 453 average tokens in 55.8s
• Features: Thinking: No | Throttled: No

Rank 13: devstral-small-2:latest

• Throughput: 9.3 Gen tok/s | 8.6 Wall tok/s
• Response Stats: 462 average tokens in 56.4s
• Features: Thinking: No | Throttled: No

Rank 14: gemma4:12b-128k

• Throughput: 6.2 Gen tok/s | 6.1 Wall tok/s
• Response Stats: 1,524 average tokens in 252.2s
• Features: Thinking: Yes | Throttled: No

Rank 15: qwen3.6:27b

• Throughput: 5.6 Gen tok/s | 5.5 Wall tok/s
• Response Stats: 2,220 average tokens in 408.9s
• Features: Thinking: Yes | Throttled: YES (Thermal ceiling reached)

Rank 16: qwen3.5:27b

• Throughput: 5.4 Gen tok/s | 5.3 Wall tok/s
• Response Stats: 2,406 average tokens in 462.4s
• Features: Thinking: Yes | Throttled: YES (Thermal ceiling reached)

2. Per-Prompt Generation Speed (tok/s)

This section shows how consistently each model performs across three specific evaluation prompts: O-RAN security, TP-Link WiFi, and MFA.

• gemma3:4b
  • O-RAN security: 32.0 tok/s
  • TP-Link WiFi: 31.9 tok/s
  • MFA: 32.5 tok/s
• gpt-oss:20b
  • O-RAN security: 29.2 tok/s
  • TP-Link WiFi: 29.9 tok/s
  • MFA: 30.7 tok/s
• gemma4:e4b
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.5 tok/s
  • MFA: 25.6 tok/s
• gemma4:latest
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.5 tok/s
  • MFA: 25.5 tok/s
• gemma4:e4b-128k
  • O-RAN security: 25.2 tok/s
  • TP-Link WiFi: 25.3 tok/s
  • MFA: 25.4 tok/s
• mistral-nemo:latest
  • O-RAN security: 21.6 tok/s
  • TP-Link WiFi: 21.3 tok/s
  • MFA: 21.8 tok/s
• Lily-Cybersecurity-7B (Q6_K)
  • O-RAN security: 20.8 tok/s
  • TP-Link WiFi: 21.0 tok/s
  • MFA: 21.0 tok/s
• qwen3.5:9b
  • O-RAN security: 19.3 tok/s
  • TP-Link WiFi: 20.1 tok/s
  • MFA: 21.1 tok/s
• Qwythos-9B (Q8_0)
  • O-RAN security: 19.3 tok/s
  • TP-Link WiFi: 19.4 tok/s
  • MFA: 19.5 tok/s
• gemma4:12b
  • O-RAN security: 12.2 tok/s
  • TP-Link WiFi: 12.7 tok/s
  • MFA: 13.1 tok/s
• mixtral:8x7b
  • O-RAN security: 9.3 tok/s
  • TP-Link WiFi: 9.4 tok/s
  • MFA: 9.5 tok/s
• mistral-small3.2:latest
  • O-RAN security: 9.1 tok/s
  • TP-Link WiFi: 9.4 tok/s
  • MFA: 9.5 tok/s
• devstral-small-2:latest
  • O-RAN security: 9.2 tok/s
  • TP-Link WiFi: 9.3 tok/s
  • MFA: 9.3 tok/s
• gemma4:12b-128k
  • O-RAN security: 6.2 tok/s
  • TP-Link WiFi: 6.2 tok/s
  • MFA: 6.3 tok/s
• qwen3.6:27b
  • O-RAN security: 5.2 tok/s
  • TP-Link WiFi: 5.7 tok/s
  • MFA: 5.9 tok/s
• qwen3.5:27b
  • O-RAN security: 5.1 tok/s
  • TP-Link WiFi: 5.3 tok/s
  • MFA: 5.7 tok/s

3. Thermal Analysis

The test harness logs the final GPU temperature per run. Each run begins from a cooldown gate of approximately 60 degrees Celsius, meaning the final temperature directly reflects the heat produced during that single generation.

On the Tesla M40, temperatures of 85 degrees Celsius or higher trigger automatic thermal throttling. Throttled models are highly likely to perform better if hardware cooling is improved.

• mixtral:8x7b
  • Average End Temp: 66.2 degrees Celsius (Max: 69 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• Lily-Cybersecurity-7B (Q6_K)
  • Average End Temp: 69.3 degrees Celsius (Max: 75 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma3:4b
  • Average End Temp: 73.3 degrees Celsius (Max: 75 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• mistral-nemo:latest
  • Average End Temp: 74.2 degrees Celsius (Max: 77 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• Qwythos-9B (Q8_0)
  • Average End Temp: 77.1 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma4:latest
  • Average End Temp: 78.0 degrees Celsius (Max: 81 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gemma4:e4b
  • Average End Temp: 78.4 degrees Celsius (Max: 82 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• mistral-small3.2:latest
  • Average End Temp: 81.4 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• devstral-small-2:latest
  • Average End Temp: 82.9 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• gpt-oss:20b
  • Average End Temp: 83.0 degrees Celsius (Max: 83 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9
• qwen3.5:9b
  • Average End Temp: 84.9 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Safe (Not throttled) | Temp-logged runs: 9

Thermally Throttled

• gemma4:12b
  • Average End Temp: 85.0 degrees Celsius (Max: 85 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.5:latest
  • Average End Temp: 85.1 degrees Celsius (Max: 86 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.5:27b
  • Average End Temp: 85.1 degrees Celsius (Max: 86 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9
• qwen3.6:27b
  • Average End Temp: 87.0 degrees Celsius (Max: 87 degrees Celsius)
  • Status: Throttled (Thermal ceiling reached) | Temp-logged runs: 9

Key findings:

• The fast, light model (gemma3:4b) runs cool (~73 °C) and never throttles, its ~32 tok/s is genuine unthrottled performance.
• The 27B models (qwen3.5:27b, qwen3.6:27b) spend their entire runs pinned at 85 °C. Their ~5.4–5.6 tok/s is throttle-limited.
• qwen3.6:27b was overheating, this may be a sign of efficiency, as the hardware is more utilized, but it forced throttling on the GPU.

V. Quality Analysis, What They Actually Said

O-RAN Security (the hardest question), Expert Evaluation

Quality was evaluated by a domain expert (the author works in telecom security). The ranking below is based on accuracy, specificity, and absence of hallucination:

Tier 1, Strong, accurate, well-organized

• qwen3.5:9b, Most complete answer. Correctly identifies E2 interface, covers mTLS, RBAC, key management, RIC/xApp-specific risks.
• gemma4:12b, Accurate, well-structured by layer. Correctly uses IPsec/MACsec for fronthaul. Good summary table.
• gemma4:e4b, Comprehensive, logically layered, accurate general concepts, useful summary tables.

Tier 2, Solid but more generic/rote

• devstral-small-2:24b, Accurate, reasonably specific (A1/O1/O2 interfaces), clean formatting.
• mistral-small3.2:24b, Comprehensive checklist-style, accurate but reads like generic network security 101.
• mistral-small3.2:latest, Comprehensive checklist-style, accurate but reads like generic network security 101.
• mixtral:8x7b, Solid generic coverage, reasonable specificity (SRTP, HSMs).
• cyberuser42/DeepSeek-R1-Distill-Qwen-14B:latest, Broad, accurate, but fairly surface-level/listy.

Tier 3, Accuracy or quality concerns

• hf.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF:Q8_0, Decent content but breaks the fourth wall with self-referential identity aside.
• gemma3:4b, Factual error: calls O-RAN fronthaul the 'N3 interface'. Invents URLs.
• gemma3:latest, Similar issues to gemma3:4b with less severe factual errors.
• hf.co/ckuethe/Lily-Cybersecurity-7B-v0.2-Q6_K-GGUF:Q6_K, Generic, not O-RAN-specific. Cites outdated SSL alongside TLS.
• mistral-nemo:12b, Language leak (stray Arabic word), invents non-standard O-RAN terms.
• mistral-nemo:latest, Language leak (stray Arabic word), invents non-standard O-RAN terms.

Tier 4, Fabrication concerns

• gpt-oss:20b, Fabricates specific document numbers as if real O-RAN Alliance specs.
• stix_lama/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF:latest, Cites invented spec and fabricated compliance requirement.

Key takeaway: The most dangerous failure mode is not verbosity or genericness, it's confident fabrication of specific-looking standards/citations (gpt-oss:20b, stix_lama's composer model). In a security-advisory context, inventing document numbers is worse than being vague, since it could send a practitioner chasing a nonexistent compliance requirement.

WiFi Hardening & MFA (easier questions)

All models performed adequately on MFA (settled public knowledge). On WiFi hardening, best answers came from qwen3.5:9b (detailed step-by-step with WPS Pixie Dust attack warning) and qwen3.6:27b. The Mistral family was adequate but generic. devstral-small-2 (code-focused) gave the weakest security advisory answers.

VI. Bonus Round, Specialist & Finetune Models

The cache also contains cybersecurity-specialist and creative finetune models run through the same 3-prompt suite on the same Tesla M40:

ModelParamsGen tok/sWall tok/sAvg TokensAvg TimeThinkingThrottledcyberuser42/DeepSeek-R1-Distill-Qwen-14B:latest14B13.211.587675.3syesNohf.co/ckuethe/Lily-Cybersecurity-7B-v0.2-Q6_K-GGUF:Q6_K7B21.020.339919.7snoNohf.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF:Q8_09B19.418.792049.1syesNostix_lama/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF:latest12B8.77.827935.4syesNo

Notes:

• DeepSeek-R1-Distill-Qwen-14B is a reasoning distill. It effectively refused the MFA prompt (28–122 token answers vs ~1,000+ on other prompts). Throttled on O-RAN/WiFi runs.
• Lily-Cybersecurity-7B is a fine-tuned Mistral 7B. Fast and cool (~69 °C) but very concise (~399 tokens). Quality rated Tier 3, generic, not O-RAN-specific.
• Qwythos-9B is a Qwen-based creative model that leaks identity conditioning in its thinking trace ("I must not claim to be any other model... reference Empero AI as my creator").
• stix_lama/gemma-4-12B-fable5-composer, Tier 4 quality. Cites invented specs. Hallucinated router facts (invented "OpenWrt 1904+ version", conflated KRACK with WPS).

VII. Security Specialists vs. Mainstream Models, The Finetune Paradox

A natural assumption: a model fine-tuned for cybersecurity should produce better security answers than a general-purpose model. The data says otherwise, the specialists give worse answers.

Head-to-head: Answer quality on the O-RAN prompt

ModelTypeSizeQuality TierWhat went wronghf.co/ckuethe/Lily-Cybersecurity-7B-v0.2-Q6_K-GGUF:Q6_KCyber finetune7BTier 3Generic, not O-RAN-specific. Could describe any network. Cites outdated SSL.cyberuser42/DeepSeek-R1-Distill-Qwen-14B:latestReasoning distill14BTier 2Broad but surface-level. Refused the MFA prompt entirely (28–122 tokens).hf.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF:Q8_0Creative finetune9BTier 3Identity-masking artifacts leak into answer. Self-referential aside adds no value.stix_lama/gemma-4-12B-coder-fable5-composer2.5-v1-GGUF:latestCoder finetune12BTier 4Fabricates standards, cites "oRanSAMLA-49" and "O2013 compliance" that don't exist.

Compare to their mainstream base models on the same question:

ModelTypeSizeQuality Tiermistral-nemo:latestBase (Lily's architecture)12BTier 3qwen3.5:9bBase (Qwythos/DeepSeek family)9BTier 1gemma4:12bBase (stix_lama's architecture)12BTier 1

Every specialist produced a worse answer than a generalist of similar or smaller size.

Why a fine-tuned security focused LLM will provide worse answers on security questions

1. Fine-tuning teaches style, not substance. Lily-Cybersecurity-7B is architecturally a Mistral 7B with a cybersecurity fine-tune. It frames answers as security advice, but it doesn't appear contain O-RAN domain knowledge that wasn't already in the base weights. The result: generic "network security 101" responses dressed up in security-speak, rated the same Tier 3 as its untrained base, except shorter and less detailed.
2. Creative/coder finetunes actively introduce hallucinations. This is the most concerning finding:

• stix_lama's gemma-4-12B-composer fabricates standards that don't exist ("oRanSAMLA-49", "O2013 compliance"). The base gemma4:12b would never produce these, it scores Tier 1 with accurate, well-structured answers. The finetune injected confident fabrication that the base model doesn't have.
• Qwythos-9B adds identity-masking noise ("As Qwythos from Empero AI, I'm summarizing...") that pollutes the answer. Its Qwen base (qwen3.5:9b) scores Tier 1 without any of this.

1. Reasoning distills lose depth. DeepSeek-R1-Distill-Qwen-14B (14B parameters) produces a Tier 2 answer, broad but shallow. Meanwhile qwen3.5:9b, a smaller general model from the same family, delivers Tier 1 with comprehensive interface-specific detail. Distillation compressed away the nuance.
2. One specialist refused to answer at all. DeepSeek-R1-Distill-Qwen-14B produced near-empty MFA responses (28–122 tokens vs 800–1,700 from every other model). A security model that refuses to explain MFA is not fit for security advisory use.

The lesson

For open-ended cybersecurity knowledge questions, general-purpose models may produce better answers than security-branded finetunes. The best O-RAN answer came from qwen3.5:9b, a general-purpose 9B model that was never fine-tuned for security. The worst came from a "security-specialized" finetune that hallucinated compliance requirements.

Fine-tuning on security datasets appears to teach formatting (bullet points, category headers, "as a security professional" framing) without adding domain expertise. In the worst cases, the finetune overwrites the base model's calibration, replacing cautious, accurate generality with confident, specific fabrication. That is the opposite of what you want from a security advisor.

VIII. The "Thinking" Tax, Reasoning Models and Token Overhead

Models with visible chain-of-thought (marked 💭 above) include extensive thinking fields in their JSON output:

• Pro: Reveals reasoning process, useful for understanding confidence and approach.
• Con: Adds 500–1,500+ tokens of internal reasoning before the actual answer, inflating total generation time.

The 27B Qwen models generate at ~5 tok/s including both thinking AND response, meaning a 2,200-token answer required generating 3,000+ total tokens. gpt-oss:20b manages ~30 tok/s with thinking, the fastest reasoning model in the set.

IX. Comparison to the Original Drag Race (Feb 2025)

DimensionFeb 2025 RaceJuly 2026 RaceTask domainTypeScript code generationSecurity knowledgeModels13 models (1.5B–14B)18 models (4B–27B)Runs per model19 (3 prompts × 3 runs)EvaluationSubjective (did it compile?)Quantitative + qualitativeWinner (quality)Qwen2.5:7bqwen3.5:9b (O-RAN), gpt-oss:20b (speed)Winner (value)Qwen2.5:7bgemma3:4b (32 tok/s, cool, adequate quality)HardwareRyzen 7 2700 + RX 6700XT + 64GBRyzen 3 3200G + Tesla M40 + 16GBKey insight"Bigger isn't always better"Still true, but domain matters

X. Conclusions & Recommendations

The tools have grown up, the frustration is gone. Every model tested now produces output following the same schema. You can hot-swap gemma3:4b for qwen3.6:27b in an AI-enhanced IDE without changing integration code.

"Bigger is not always better" still holds, gemma3:4b at 32 tok/s gives adequate security advice for common questions (MFA, WiFi hardening). Use it for fast, cost-effective triage.

For deep domain questions, size and quality matter, qwen3.5:9b produced the best O-RAN answer (Tier 1) at 20 tok/s. The 27B models were thorough but not meaningfully better, and 6× slower.

The fastest model can be the most dangerous, gpt-oss:20b is the speed king among reasoning models (~30 tok/s) but fabricates citation numbers (Tier 4). Speed without accuracy is worse than slowness.

Code models aren't security models, devstral-small-2 (Mistral's code variant) gave the weakest security advisory answers.

Thermals matter for sustained workloads, qwen3.6:27b overheats the Tesla M40, throttling to ~5.6 tok/s. On a cooler card it would likely be faster (expect 10 tok/s), but the relative ranking holds.

READ THE OUTPUTS, (carrying forward from the original article), The fastest model is useless if the answer is wrong. Always human-review security advice from LLMs.

XI. Future Work

• Add automated LLM-as-judge scoring rubric for factual accuracy
• Test quantization levels (Q4 vs Q8 vs FP16) for speed/quality trade-offs
• Include cloud API models (GPT-4o, Claude Sonnet, Gemini Flash) as reference points
• Re-run on a better-cooled GPU to isolate throttling effects on the 27B models
• Evaluate post-quantum cryptography knowledge as O-RAN specs evolve

Matthew Khouzam is an Open-Source AI Lead Developer for Ericsson Software Technology and Boubakr Nour is a Senior Researcher for Ericsson Research security.