CyberSecQwen-4B: A Specialized Cyber Security Model Achieved with 4B Parameters

A new cyber security specialized small language model “CyberSecQwen-4B” has been released from Hugging Face. This model is specialized in defensive cyber security tasks and achieves performance that surpasses an 8B general-purpose model with only 4B parameters.

Conventional frontier models can handle a wide range of tasks, but have high API costs and all prompts are sent to external data centers. Furthermore, they are trained to reject complex edge cases that are actually handled in defensive tasks, such as incident reports, attacker-level payloads, and vulnerability disclosure drafts.

Performance Advantage of Specialized Models

CyberSecQwen-4B has shown excellent results in evaluation on CTI-Bench, compared to Cisco’s Foundation-Sec-Instruct-8B model. In CTI-MCQ (2,500 items), it recorded a score of 0.5868 ± 0.0029, exceeding the 8B model’s 0.4996 by +8.7 points.

In CTI-RCM (1,000 CVE→CWE items), it recorded 0.6664 ± 0.0023, maintaining 97.3% accuracy compared to the 8B model’s 0.6850. Despite having half the number of parameters, the performance difference is only 1.9 points.

This result demonstrates that a 4B model specialized in a narrow domain can achieve performance comparable to a general-purpose 8B model. (Source: CyberSecQwen-4B: Why Defensive Cyber Needs Small, Specialized, Locally-Runnable Models)

Single-GPU Training on AMD MI300X

The entire training pipeline of CyberSecQwen-4B is completed on a single AMD Instinct MI300X 192GB instance. The combination of 192GB HBM3 memory and ROCm 7’s vLLM stack enables training without using complex optimization techniques such as quantization tricks, gradient checkpoints, and model splitting.

In this environment provided through the AMD Developer Cloud, all processes from training to adapter integration and evaluation can be performed on a single GPU instance. This allows for efficient development while avoiding the complexity of managing multiple GPU clusters and distributed training.

Practical Introduction Requirements

CyberSecQwen-4B is designed to run on 12GB consumer-grade GPU cards. Running a 70B general-purpose model on 4 GPUs locally is difficult in practice, even if it is “local”. On the other hand, running a 4B general-purpose model on a single consumer-grade GPU is easy to introduce, but it cannot achieve the performance required for actual tasks, inferior to an 8B specialized model.

CyberSecQwen-4B has proven that a carefully adjusted 4B specialized model can achieve performance comparable to or surpassing an 8B specialized model in a narrow range of threat intelligence tasks, such as CWE classification, CVE-to-CWE mapping, and structured CTI Q&A. (Source: CyberSecQwen-4B: Why Defensive Cyber Needs Small, Specialized, Locally-Runnable Models)

Summary

  • By downloading CyberSecQwen-4B from Hugging Face and running it locally on a 12GB GPU environment, CTI analysis tasks can be automated without external API costs.
  • By training a specialized 4B model using the AMD MI300X environment, it is possible to achieve performance that surpasses large general-purpose models in specific domains.
  • By adopting the CTI-Bench evaluation protocol, it is possible to objectively measure the performance of in-house cyber security models and compare them with existing solutions.
  • By utilizing ROCm 7’s vLLM stack, efficient model development can be achieved on a single GPU without complex distributed training settings.