“Custom” Dell PowerEdge R760 Server

As the saying goes, the third time’s a charm. This isn’t our first rodeo with smashing records with Pi; we took lessons from our first two iterations to build the best Pi platform. Our first build leveraged a 2U server with 16 NVMe bays and three internal SSD sleds. With 30.72TB Solidigm P5316 SSDs, we contained the swap storage for y-cruncher, but we had to leverage an HDD-based storage server for the output file. It was less than optimal, especially during the end of the write-out phase. Our 2nd platform used the same server, with an external NVMe JBOF attached, which gave us additional NVMe bay—but at the cost of sensitive cabling and unbalanced performance. The downside to both platforms was needing to rely on external hardware throughout the entire y-cruncher run at the cost of added power and additional points of failure.

For this run, we wanted to leverage one all-direct-NVMe single server and have enough space for our y-cruncher swap storage and output storage under one sheet-metal roof. Enter the Dell PowerEdge R760 with the 24-bay NVMe Direct Drives backplane. This platform leverages an internal PCIe switch to get all NVMe drives talking to the server simultaneously, bypassing any need for additional hardware or RAID devices. We then pieced together a PCIe riser configuration from multiple R760s in our lab environment, giving us four PCIe slots in the rear for additional U.2 mounted NVMe SSDs. A bonus was taking larger heatsinks off another R760, giving us as much turbo-boost headroom as possible. Direct Liquid Cooling came into our lab a month too late to be implemented in this run.

“The StorageReview Lab Team’s calculation of pi to over 202 trillion digits, achieved using 5^th Gen Intel Xeon processor, underscores the power and efficiency of these CPUs. Leveraging the increased core count and advanced performance features of the 5^th Gen Xeon processor, this milestone sets a new benchmark in computational mathematics and continues to pave the way for innovations across various scientific and engineering workloads,” said Suzi Jewett, Intel general manager for 5^th Gen Intel Xeon Processor Products

While you technically could order a Dell configuration precisely like the one used in this run, it was not something they had lying around and needed to be pieced together. (Maybe Michael will run a limited edition “Pi” batch of R760s with this exact config, custom paint, and the SR logo.)

The power supply size was also critical to this run. While most would immediately think the CPUs draw most of the power, having 28 NVMe SSDs under one roof is a considerable power impact. Our build leveraged the 2400W PSUs, which, as it turned out, only barely worked. We had a few near-critical level power draw moments where we would have been underpowered if the system had dropped one power supply connection. This hit early on; power consumption skyrocketed while the CPU loads peaked, and the system increased I/O activity to all SSDs. If we had to do this again, the 2800W models would have been preferred.

Performance Specs

Technical Highlights

• Total Digits Calculated: 202,112,290,000,000
• Hardware Used: Dell PowerEdge R760 with 2x Intel Xeon 8592+ CPUs, 1TB DDR5 DRAM, 28x Solidigm 61.44TB P5336
• Software and Algorithms: y-cruncher v0.8.3.9532-d2, Chudnovsky
• Data Storage: 3.76PB written per Drive, 82.7PB across the 22 disks for swap array
• Calculation Duration: 100.673 Days

y-cruncher Telemetry

• Logical Largest Checkpoint: 305,175,690,291,376 ( 278 TiB)
• Logical Peak Disk Usage: 1,053,227,481,637,440 ( 958 TiB)
• Logical Disk Bytes Read: 102,614,191,450,271,272 (91.1 PiB)
• Logical Disk Bytes Written: 88,784,496,475,376,328 (78.9 PiB)
• Start Date: Tue Feb 6 16:09:07 2024
• End Date: Mon May 20 05:43:16 2024
• Pi: 7,272,017.696 seconds, 84.167 Days
• Total Computation Time: 8,698,188.428 seconds, 100.673 Days
• Start-to-End Wall Time: 8,944,449.095 seconds, 103.524 Days

The largest known digit of Pi is 2, at position 202,112,290,000,000 (two hundred two trillion, one hundred twelve billion, two hundred ninety million).

Broader Implications

While calculating pi to such a vast number of digits might appear to be an abstract challenge, the practical applications and techniques developed during this project have far-reaching implications. These advancements can enhance various computational tasks, from cryptography to complex simulations in physics and engineering.

The recent 202 trillion digit pi calculation highlights significant advancements in storage density and total cost of ownership (TCO). Our setup achieved an astonishing 1.720 petabytes of NVMe SSD storage within a single 2U chassis. This density represents a leap forward in data storage capabilities, especially considering the total power consumption peaked at only 2.4kW under full CPU and drive load.

This energy efficiency contrasts traditional HPC record runs that consume significantly more power and generate excessive heat. Power consumption increases exponentially when you factor in additional nodes for scale-out storage systems if you need to expand low-capacity shared storage compared to high-density local storage. Heat management is critical, especially for smaller data centers and server closets. Cooling traditional HPC record systems is no small feat, requiring data center chillers that can draw more power than the equipment running alone. By minimizing power consumption and heat output, our setup offers a more sustainable and manageable solution for small businesses. As a bonus, most of our run was performed with fresh air cooling.

To put this into perspective, imagine the challenges those running with networked shared storage and un-optimized platforms face. Those setups would require one or more data center chillers to keep temps in check. In these environments, every watt saved translates to less cooling required and lower operational costs, making our high-density, low-power approach an ideal choice. Another critical benefit to running a lean and efficient platform for a record run is protecting the entire setup with battery backup hardware. As mentioned earlier, you would need battery backups for compute servers, switching, storage servers, chillers, and water pumps to keep it alive for a good chunk of the year.

Overall, this record-breaking achievement showcases the potential of current HPC technologies and underscores the importance of energy efficiency and thermal management in modern computing environments.

Ensuring Accuracy: The Bailey–Borwein–Plouffe Formula

Calculating pi to 202 trillion digits is a monumental task, but ensuring the accuracy of those digits is equally crucial. This is where the Bailey–Borwein–Plouffe (BBP) formula comes into play.

The BBP formula allows us to verify pi’s binary digits in hexadecimal (base 16) format without needing to compute all preceding digits. This is particularly useful for cross-checking sections of our massive calculation.

 

Two of the verification computations.

Here’s a simplified explanation:

1. Hexadecimal Output: We first generate pi’s digits in hexadecimal during the main calculation. The BBP formula can compute any arbitrary individual digit of pi in base 16 directly. You can do this with other programs like GPUPI, but y-cruncher also has a built-in function. If you prefer an open-source approach, the formulas are well-known.
2. Cross-Verification: We can compare these results with our main calculation by calculating specific positions of pi’s hexadecimal digits independently with the BBP formula. If they match, it strongly indicates that our entire sequence is correct. We did this cross-check over six times; here are two of them.

For instance, if our primary calculation produces the same hexadecimal digits as those obtained from the BBP formula at various points, we can confidently assert the accuracy of our digits. This method is not just theoretical; it’s been practically applied in all significant pi calculations, ensuring robustness and reliability in results.

R= Official Run Result, V= Verification Result

• R: f3f7e2296 822ac6a8c9 7843dacfbc 1eeb4a5893 37088*
• V: *3f7e2296 822ac6a8c9 7843dacfbc 1eeb4a5893 370888

Astute readers will note that the verifications from the screenshots and the above comparison are a bit shifted(*). While not necessary, since the hex would be affected at the end, we also spot-checked a few other locations (like 100 Trillion and 105 Trillion digits) to ensure the run matched. While it is theoretically possible to calculate any decimal digit of pi using a similar method, it is unclear if that would have precision past a mere 100 Million digits or be even computationally efficient to do so, rather than do the Chudnovsky math and get all of them. (If Eric Weisstein sees this, reach out; I’d like to take a stab at it.)

By integrating this mathematical cross-checking process, we can assure the integrity of our record-breaking 202 trillion digit pi computation, demonstrating our computational precision and commitment to scientific accuracy.

The Road Ahead