The Cambridge-Cranfield HPCF currently has two main computers: Franklin a 900 processor Sun and Maxwell, a cluster with 128 Opteron processors.

Franklin (Sunfire 15K)

The Cambridge-Cranfield HPCF Sun Galaxy-class configuration supercomputer is based on the SunFire F15K node, nine of which will be installed at Cambridge with 100 CPUs and one at Cranfield with 72 CPUs. Each node has 288 GB of main memory, of which at least 256 Gbytes should be available for applications. The CPUs are 1.2 GHz UltraSPARC III Cu each with a peak of 2.4 Gflops; 1 Gflops should be attainable on suitable codes. Each node is a shared memory system (SMP) allowing each CPU access to all the available memory on the node using fast memory buses.

The nodes are connected by the new SunFire Link interconnect which enables to system to scale exceptionally well up to eight nodes. A quote from the Sun press release on 18th November 2002 describes the main properties of the interconnect:

The high-bandwidth, low-latency interconnect enables customers to build superclusters that can function as a collective -- virtualized -- system capable of linear scaling as nodes are added. The new technology sets industry high marks for cluster performance with sustained 2.8 gigabytes per second bandwidth and MPI latency rates under four microseconds. With Sun Fire Link interconnect, Sun has increased speeds at which data is shared, while delivering bandwidth an order of magnitude greater than standard, commodity interconnects. Additionally, Sun has cut hardware latency to one microsecond, a speed competitors have yet to offer.

The data transfer rate between CPUs within a node is of the same order as the Fire Link interconnect rate: about 3 GB/s. A severe and realistic test of the system is the bandwidth for the all-to-all MPI command. Within one node this bandwidth is 7 GB/s and for three nodes connected by the Fire Link 7 GB/s was also achieved showing the scalability of this configuration for a realistic test. In many cases, we should expect even better performance for the interconnect.

The very low latencies quoted above, both within a node and over the Fire Link between nodes, is also an extremely important feature for maintaining data transfer efficiency. In comparison, the Hitachi SR2201 had a crossbar network with a latency of 14 μs.

The Cambridge systems will initially be linked in 3 groups of three with the new Fire Link technology. This delivers about 3 GB/s per node, confirmed by our benchmarks, which should give good scalability. So the maximum job size supported initially is 24 hours at a realistic performance of 300 Gflops with 150 GB/s intranode memory bandwidth communicating with 800 GB of memory. The available disk space is about 6 TB.

Sun Microsystems are committed to a partnership with the CCHPCF and the Universities which goes beyond system support. They are keen to interact with users on all levels to mutual benefit and to support the wider use of high performance computing in research. In the press release they confirm this position:

Galaxy-class configuration customers in the education market will also join an elite community of Sun customers worldwide as a Center of Excellence (COE), gaining expanded opportunities for partnership with Sun and other institutions participating in the Center of Excellence program.

The full press release can be seen at www.sun.com/smi/Press/sunflash/2002-11/sunflash.20021118.3.html There will be a number of Sun tuning courses for users which will be announced at the appropriate times.

                                  
CPU:          9 x 100 x UltraSPARC III Cu, 1.2 GHz, 2 floating point pipelines
GFLOPS:       900 x 2.4 = 2160
Caches:       32 KB 4-way associative instruction cache
              64 KB 4-way associative data cache
              8 MB secondary cache
Memory:       288 GB per node, 2592 GB total
Disk:         4 TB local RAID disk

Franklin's queues currently allow jobs of up to 24 hours and 96 processors (240 Gflops and 288 GB).

Biography of Rosalind Franklin

Maxwell (Sun Opteron cluster)

Maxwell, the new HPCF cluster, is based on 64 Sun V20z dual processor nodes. Each node has two 248 Opteron processors running at 2.2 GHz, each with a peak of 4.4 Gflops, so 2 Gflops per processor should be attainable on suitable codes. Each CPU has its own memory controller so the memory architecture is non-uniform. The kernel does a reasonable job of keeping processes on the same CPU as the memory. Measurements show each CPU can get 3.6 GB/s scaling fairly smoothly to 7 GB/s per node.

Of the 64 nodes, one is used as a log-in node. The remaining 63 nodes are used for jobs. Of these nodes, 47 have 4 Gb while the remaining 16 have 8 GB. The nodes are connected by a dedicated gigabit ethernet network for running MPI. This has a reasonable bandwidth of about 80 MB/s but a latency of the order of 17 μs. The 16 large memory nodes are also connected by a high performance Infiniband interconnect which provides a bandwidth of 800 MB/s with 5 μs latency.

CPU:       64 x 2 x AMD Opteron 248, 2.2 GHz, 2 floating point pipelines
GFLOPS:    128 x 4.4 = 563
Caches:    64 KB 2-way associative instruction cache
           64 KB 2-way associative data cache
           1 MB 16-way associative secondary cache
Memory:    4 GB minimum per node, 320 GB total
Disk:      Access to 4 TB RAID disk on Franklin

Maxwell's queues currently allow jobs of up to 24 hours and 64 processors (280 Gflops and 128 GB).

For more information about clusters in general, see the report of a Meeting on Clusters hosted by the CCHPCF on 6 December 2004.

Biography of James Clerk Maxwell