Qwen3.8-2.4T-A95B Architecture Details

Qwen3.8-2.4T-A95B is a 2.4 trillion parameter open-weight model that adopts a hybrid architecture where 95 billion parameters are activated per token. This model employs a fine-grained Mixture of Experts (MoE) architecture, which improves routing efficiency and expertise by distributing capacity across 512 small experts.

The architecture features the following:

  • Gated DeltaNet Layer (69/92 layers): uses linear attention and replaces KV cache with a fixed-size memory
  • Gated Attention Layer (23/92 layers): achieves high-precision token interactions with full quadratic attention
  • Context Window: natively supports 262,144 tokens, expandable to 1,010,000 tokens

This design limits computation and memory for processing long texts of nearly one million tokens, optimizing it for agent workloads. (Source: Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM)

Model Deployment Flow

The deployment of Qwen3.8-2.4T-A95B is carried out using Amazon SageMaker HyperPod and vLLM through the following steps:

  1. Cluster Provisioning: prepare ml.p6-b300 instances (8×NVIDIA B300 Blackwell Ultra GPU)
  2. vLLM Setup: enable NVFP4 quantization, embedded reading, tool calls, and native Multi-Token Prediction (MTP) speculative decoding
  3. OpenAI Compatible Endpoint Construction: automate model loading and endpoint setup

This process optimizes the operational cost of large models while achieving high-performance inference. (Source: Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM)

Reading Control and Agent Workload Optimization

Qwen3.8 is designed for agent execution, allowing reading control using the reasoning_effort parameter (low/medium/high). This setting enables flexible resource adjustment and optimization for the following use cases:

  • Multi-Step Coding: code generation involving multiple steps
  • Autonomous Tool Usage: task automation through external API integration
  • Long-Term Horizon Planning: strategic decision-making across multiple interactions

This provides flexibility in trading off computational resources and reading depth. (Source: Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM)

Summary

  • Deploy Qwen3.8-2.4T-A95B on SageMaker HyperPod: optimize large model operational costs with vLLM using NVFP4 quantization settings
  • Utilize Reading Control Parameters: adjust reasoning_effort to optimize computational resources for multi-step tasks
  • Optimize for Agent Workloads: achieve long-term horizon processing with a hybrid architecture of Gated DeltaNet and Gated Attention
  • Construct OpenAI Compatible Endpoints: automate endpoint setup using SageMaker’s managed services
  • Expandable Context Window: support native 262K tokens and expandable 1M tokens, enhancing long text processing