Full Specifications:

Dell PowerEdge XE9680 Build and Design

The PowerEdge XE9680 is an imposing piece of hardware, measuring 10.36 inches (263.20mm) in height, 18.97 inches (482.00mm) in width, and 39.71 inches (1008.77mm) in depth with its bezel attached. When fully loaded, it weighs 251.44 lbs (114.05 kg). The GPU selection will have the final say on weight, with the NVIDIA H100/H200 model coming in at 238 lbs, while the AMD MI300X unit tips the scales at 251 lbs.

This was the first server that required careful thought to load properly into our testing environment. When you consider server weight and the number of people needed to rack hardware, there is some wiggle room to go outside the boundaries, but at a certain point, one or two people aren’t lifting it alone. Dell is kind enough to give you a “lift table” to help you understand how this platform fits in. For all those wondering, Kevin loaded the XE9680 into the rack himself.

Despite its complexity and Dell’s recommendation for specialized service technicians, the XE9680 features remarkably user-friendly service elements. The server’s panels include detailed service instructions and clear graphics, making maintenance procedures surprisingly approachable for experienced IT staff. These visual guides proved invaluable during our hands-on time with the system, allowing us to service various components confidently.

After opening the cover on the PowerEdge XE9680, once you get past the numerous power cables from the small power substation on top, it looks a lot like a PowerEdge R760. Our unit was powered by two Intel Xeon Platinum 8468 processors, each with 48 cores at 2.1Ghz. Each processor offers 80 PCIe lanes, which flow through quite a few PCIe switches in this unit to support the GPUs, NICs, and other hardware loaded into the XE9680.

One of the most impressive engineering features is the PCIe Switch Board (PSB) design. These boards provide connectivity for up to 10 additional full-height, half-length PCIe cards (two of which can exceed 75W power draw) and integrate directly with the GPU baseboard. This direct integration enables GPU-direct technology, allowing SSDs and network cards to communicate directly with the GPUs, bypassing the CPU and reducing latency for I/O-intensive AI workloads.

Each expansion slot supports a full PCIe Gen5 x16 interface, including the two lower slots on the far left and right of the layout. While the upper eight slots are connected through their own PSB, the two lower slots connect directly to the PCIe Base Board (PBB). These two slots also support high-power draw cards. Additionally, it should be noted that the PCIe layout varies slightly depending on the GPU type chosen for the PowerEdge XE9680. The AMD-equipped models don’t support SmartNIC/DPUs, and the Intel Gaudi3 models have two slots blocked off due to airflow issues.

Cooling is another area where Dell’s engineering expertise shines. The system employs up to 16 high-performance gold-grade fans—six in the mid-tray and ten at the rear. The PowerEdge XE9680 supports a wide range of installation scenarios, with ambient temperatures ranging from 10 to 35C (30C with the Intel Gaudi3 GPUs). At full tilt, the server moves an impressive 1,200CFM into the hot aisle.

This robust cooling solution handles even the most demanding thermal loads, including the AMD MI300X, Intel Gaudi3, or NVIDIA H100 GPUs, while maintaining optimal operating temperatures. The PowerEdge XE9680 sings quite a bit under load in terms of noise output. Dell offers a full acoustical specification sheet for the XE9680 under different situations, but it’s pretty easy to say it will be a loud platform under load.

Management

The XE9680’s management capabilities are built around Dell’s enterprise-proven iDRAC9, which provides comprehensive server lifecycle management and monitoring. This iteration of iDRAC brings several AI-optimized features, including detailed GPU telemetry, power consumption analytics, and extensive thermal monitoring designed for high-density AI workloads.

The platform’s management stack is particularly noteworthy for AI infrastructure deployments. Through iDRAC9’s RESTful API with Redfish support, organizations can programmatically monitor and manage GPU utilization, memory bandwidth, and thermal conditions – critical metrics for maintaining optimal AI training and inference performance. The system’s integration with OpenManage Enterprise enables fleet-wide management of multiple XE9680s through a unified console, which is essential for large-scale AI clusters.

Security and compliance are foundational elements of the management architecture. The platform implements Silicon Root of Trust and Secure Component Verification, ensuring hardware integrity from boot-up through operation. These features are especially valuable when running sensitive AI workloads or handling proprietary model weights.

The predictive failure analysis capability, powered by CloudIQ integration, uses machine learning to forecast potential hardware issues before they impact workloads. This proactive approach is especially crucial for long-running AI training jobs, where unexpected downtime can result in days of lost computation. When combined with Dell’s ProSupport Plus service, this predictive capability triggers automatic case creation and parts dispatch, often resulting in preventive maintenance before system degradation occurs.

For organizations requiring integration with existing management tools, the XE9680 supports various management frameworks through OpenManage integrations, including ServiceNow and BMC TrueSight, allowing seamless incorporation into established IT service management workflows.

The iDRAC9 interface provides detailed real-time monitoring of critical components through an intuitive dashboard. GPU monitoring displays comprehensive metrics, including temperature, power consumption, and utilization rates across all eight accelerators, essential for optimizing AI workload distribution.

The storage monitoring interface offers instant visibility into drive health, temperature, and performance metrics across the NVMe array, which is particularly valuable when managing high-throughput inference caches and training datasets.

Memory, Storage, and Scale

The eight AMD MI300X GPUs inside the Dell PowerEdge XE9680 represent a significant leap in GPU memory capacity, offering 192GB of HBM3 memory per card compared to NVIDIA H200’s 141GB. This 36% increase in memory capacity isn’t just a number on a spec sheet – it’s critical for large language model deployment.

This massive pool of memory, coupled with the MI300X’s 5.3 TB/s memory bandwidth, enables organizations to run multiple instances of smaller models or partition larger models across GPUs while maintaining high throughput and low latency.

To put this in perspective, Meta’s Llama 3.1 405B model, which requires north of 1TB of VRAM in BF16, can be comfortably distributed across a single XE9680 with MI300X GPUs without quantization and full 128k context length. This eliminates potential quality loss associated with quantization techniques and allows for more Tokens/Second compared to having the model distributed over two servers.

To maximize our storage footprint, we used the Solidigm 61.44TB drives to serve as a sophisticated extension to the memory, bridging the gap between high-speed GPU memory and traditional storage. The SSDs excel at storing key-value cache pairs during inference, effectively extending the GPU’s memory capacity for long-context generations. Their massive capacity and NVMe performance make them ideal for quick model weight access, enabling efficient model switching and warm starts.

In applications like the Metrum AI deployment we detail below, the SSDs pull double duty as the storage backend for vector databases, delivering the performance needed for real-time similarity searches while maintaining the capacity for extensive embedding storage.

The value of these high-capacity drives extends beyond inference to training workflows. They provide ideal local storage for queuing training batches, reducing network overhead by keeping data closer to the compute resources. During training, these drives excel at storing model checkpoints locally, which is critical for maintaining training progress and enabling quick recovery. This local storage strategy also helps to optimize network utilization by reducing the immediate network traffic after each processed layer and batch.

While the 61.44TB capacity across eight bays in the XE9680 sounds promising, there’s much more capacity coming. With Solidigm’s newly announced 122.88TB drive, the storage density in the XE can be doubled to nearly a petabyte for further training optimizations and longer-lived inference caches.

Metrum AI Healthcare Assistant – Revolutionizing Patient Care

The healthcare sector consistently faces a challenge in managing time-consuming patient documentation and record management, which often detracts from direct patient care. Metrum AI’s Healthcare Assistant, deployed on Dell PowerEdge XE9680 servers with AMD accelerators, exemplifies how advanced AI infrastructure can transform healthcare workflows, enhancing efficiency and improving patient outcomes.

The system utilizes Llama 3.1 70B Instruct as its primary language model, renowned for its comprehension of medical contexts. This allows it to process complex patient data easily. This language model is paired with the gte-v1.5 embedding model and Milvus Vector DB, providing a robust foundation for natural language processing and contextual understanding essential for handling medical data.

Metrum AI’s Healthcare Assistant also includes a multimodal approach incorporating HistoGPT for histopathology image analysis and OpenAI’s Whisper for real-time transcription of physician notes. Together, these models streamline clinical workflows, allowing doctors to speak naturally while the system transcribes, categorizes, and integrates information into patient records in real-time.

Metrum AI recognizes that even though individual patient data may be relatively small, the combined storage demands of high-traffic hospitals can escalate to hundreds of terabytes. The Dell PowerEdge XE9680 can address this with its local onboard NVMe storage. Our configuration offers eight 2.5″ U.2 NVMe storage bays operating at PCIe Gen4 speeds. While we tested the XE9680 with 61.44TB Soldigim D5-P5336 QLC SSDs, this capacity can scale even further. Soldigim recently launched their new D5-P5336 122.88TB QLC models, which doubles the capacity of their already massive SSDs while maintaining the same performance.

Metrum provided estimates of how patient data translates over time over different scenarios. When you work that out into total storage capacity, you can see how many additional patients a unit could support using the highest-capacity SSDs. Taking the estimated data footprint per patient and going against the usable capacity for each SSD (57TB for the 61TB SSD and 114TB for the 122TB SSD), we can see that having dense SSDs greatly increases what you can store on the server in a meaningful way per year.

While the initial 1-year estimates seem pretty high, it’s important to note that patient data isn’t static. You will have new data captured and new visits scheduled, growing the demand for storage. This is where storage plays a significant role in the medical imaging space. Additional storage capacity directly affects how many patients a solution can effectively support.

The Dell PowerEdge XE9680, fitted with AMD MI300X accelerators and integrated with Metrum AI’s Healthcare Assistant, provides a scalable and efficient solution for healthcare providers. By automating time-consuming tasks and enabling rapid access to critical insights, this setup allows clinicians to focus more on patient care while managing growing demands. Through seamless integration of AI components across language, image, and voice modalities, the Healthcare Assistant represents a significant advancement in AI-driven healthcare solutions, reducing administrative burdens and improving overall patient outcomes.

Conclusion