Operating Environment
From UF HPC Wiki
The University of Florida Phase I and II cluster has an up-to-date operating environment to offer our users the flexibility needed for operation of their projects.
Contents |
Operating System
- CentOS 4.5 (Kernel - 2.6.18-8.1.14.el5.L-1642)
- Lustre Filesystem
- How to create a System mirror
- Torque queueing system
- Cisco IOS
Monitoring Software
System Software
- GNU Compiler Suite (3.4.6-8)
- gcc
- g++
- g77
- GNU Compiler Suite (4.1.1-53)
- gcc
- g++
- gfortran
- Intel Compiler Suite (9.1.044)
- icc
- ifort
- Intel Compiler Suite (10.1.015)
- icc
- ifort
- OFED 1.2 IB Stack
- Kernel IB
- OpenMPI (1.2.2)
- MVAPICH (0.9.9)
- MVAPICH2 (0.9.8)
Hardware
Compute Nodes
- 202 Compute Nodes (3 Racks)
- Rackable Systems C1000 1U Chassis
- 4 AMD Opteron Cores (2 x 275, 2.2GHz)
- 4 GB DDR400 RAM
- 300 GB SATA Disk (local storage)
- 1 Cisco 4x Infiniband HCA (x8 PCI-E)
- DC Power
- 120 Compute Nodes (2 Racks)
- Rackable Systems C1000 1U Chassis
- 4 AMD Opteron Cores (2 x 275, 2.2GHz)
- 8GB DDR400 RAM
- 80GB SATA Disk (local storage)
- 1 Mellanox Technologies MT25208 4X Infinband HCA (8X PCI-E)
- AC Power
- 80 Compute Nodes (1 Rack)
- Rackable Systems C1000 1U Chassis
- 4 AMD Opteron Cores (2 x 275, 2.2GHz)
- 8GB DDR400 RAM
- 80GB SATA Disk (local storage)
- AC Power
- 2 "Fat" Nodes
- Rackable Systems C4002 4U Chassis
- 8 AMD Opteron Cores (4 x 275, 2.2GHz)
- 32 GB DDR400 RAM
- 300 GB Ultra-320 SCSI Disk (Local Storage)
- AC Power
- 112 Compute Nodes (4 Racks)
- Penguin Relion 1670 Servers (Operating Manual)
- 8 Intel E5462 Cores (2 x E5462, 2.80GHz)
- 1600 MHz FSB
- 800MHz DDR2 RAM (PC2 6400)
- 64 GB (21 Nodes)
- 32 GB (91 Nodes)
- 250GB SATA Disk (Local Storage)
- AC Power
- 4 nVidia Tesla GPU Host Nodes
- SuperMicro 6025 SuperServer
- 4 Intel 5462 Cores (1 x E5462, 2.80GHz)
- 16 GB RAM (PC2 6400)
- 250 GB SATA Disk (Local Storage)
- 2x Tesla C1060 GPUs
- AC Power
I/O Nodes and Storage (UF HPC Scratch)
- 1 Metadata Server (Lustre MDS)
- Penguin Altus 1650
- 4 AMD Opteron 2212 HE (2.0 GHz)
- 8 GB RAM
- 4 750 GB SATA II Disks (dual mirrors)
- 1 Mellanox IB HCA (4x SDR)
- 8 I/O Nodes (Lustre OSSs)
- Rackable Systems C3002 3U Chassis
- 4 AMD Opteron Cores (2 x 275, 2.2GHz)
- 4 GB DDR400 RAM
- 300 GB SATA Disk (System Disk)
- 3 QLogic 2340 FC HCAs (2 Gb/s)
- 1 Cisco 4x Infiniband HCA (x8 PCI-E)
- Storage Subsystem: 43 TB (Raw)
- 12 OmniStore 4932F RAID Controllers (1 GB Cache)
- 144 300 GB dual-ported FC Disks (10,000 RPM)
- 2 Cisco MDS 9216i FC Switches
- 1 Cisco 3012 FC to IB Gateway Switch
- 1.5+ GB/sec Sustainable Random Access I/O
I/O Nodes and Storage (CRN Scratch)
- 1 Metadata Server (Lustre MDS)
- Penguin Altus 1650
- 4 AMD Opteron 2212 HE (2.0 GHz)
- 8 GB RAM
- 4 750 GB SATA II Disks (dual mirrors)
- 1 Mellanox IB HCA (4x SDR)
- 2 I/O Nodes (Lustre OSSs)
- Penguin Altus 2650SA
- 4 AMD Opteron Cores (2 x 2350, 2.0 GHz)
- 16 GB DDR400 RAM
- 2 160 GB SATA II Disk (mirrored)
- 3 QLogic 2462 FC HCAs (4 Gb/s)
- 1 Mellanox InfiniHost III Ex Infiniband HCA (x8 PCI-E)
- 1 Chelsio S310E 10GbE NIC
- Storage Subsystem: 104 TB (Raw)
- 6 RAID Inc. Falcon III 24-Bay RAID Chassis
- 6 Falcon III RAID Controllers (2 GB Cache)
- 144 750 GB SATA II Disks
- 1 Cisco 3012 FC to IB Gateway Switch
- 1.8+ GB/sec Sustainable Random Access I/O
Networking
New Physics Building
- InfiniBand Interconnect
- 50% Blocking Fat Tree Topology
- 2 Cisco 7008 Core IB Switches (96 Ports 4x IB)
- 14 Cisco 7000 Leaf IB Switches (24 x 4x SDR)
- 8 Silverstorm 9024S IB Leaf Switches (24 x 4X SDR/DDR)
- 1 Cisco 3012 EtherNet/Fibre Channel Gateway
- 4 FC Gateway Modules
- 6 EtherNet Gateway Modules
- Gigabit Ethernet Interconnect (Management)
- Cisco 6506 Gigabit Ethernet Switch
- Cisco 4948 Gigabit Ethernet Switch
- 8 3Com 2848 48-Port Gigabit Ethernet Switches
- 4 Foundry FastIron Edge X448 Ethernet Switches
Larsen Hall
- InfiniBand Interconnect
- 50% Blocking Fat Tree Topology
- Voltaire 2012
- 2 Fabric Controllers (Spines)
- 3 24 Port (4x DDR IB) Modules
- 2 20210G IB/Ethernet Gateway Modules
- 2 10 Gb Ethernet (XFP Slots)
- 20 Ports (4x DDR IB)
- Force-10 C150 Ethernet Switch
- 1 RP Module
- 1 8 Port 10 Gb Ethernet (XFP Slots) Module
Node Performance
Penguin Relion 1670
- 64 GB RAM (See configuration details above)
- OpenMPI 1.2.2 (--btl sm,self)
- Goto BLAS (1.26, -lgoto_penrynp-r1.26)
mpirun --mca btl sm,self -hostfile mpihosts -np 8 ./xhpl
============================================================================
HPLinpack 1.0a -- High-Performance Linpack benchmark -- January 20, 2004
Written by A. Petitet and R. Clint Whaley, Innovative Computing Labs., UTK
============================================================================
An explanation of the input/output parameters follows:
T/V : Wall time / encoded variant.
N : The order of the coefficient matrix A.
NB : The partitioning blocking factor.
P : The number of process rows.
Q : The number of process columns.
Time : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.
The following parameter values will be used:
N : 90800
NB : 184
PMAP : Row-major process mapping
P : 2
Q : 4
PFACT : Right
NBMIN : 4
NDIV : 2
RFACT : Right
BCAST : 2ringM
DEPTH : 0
SWAP : Spread-roll (long)
L1 : transposed form
U : transposed form
EQUIL : yes
ALIGN : 8 double precision words
----------------------------------------------------------------------------
- The matrix A is randomly generated for each test.
- The following scaled residual checks will be computed:
1) ||Ax-b||_oo / ( eps * ||A||_1 * N )
2) ||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 )
3) ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo )
- The relative machine precision (eps) is taken to be 1.110223e-16
- Computational tests pass if scaled residuals are less than 16.0
============================================================================
T/V N NB P Q Time Gflops
----------------------------------------------------------------------------
WR03R2R4 90800 184 2 4 6185.92 8.068e+01
----------------------------------------------------------------------------
||Ax-b||_oo / ( eps * ||A||_1 * N ) = 0.0030523 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_1 * ||x||_1 ) = 0.0029648 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) = 0.0004746 ...... PASSED
============================================================================
Finished 1 tests with the following results:
1 tests completed and passed residual checks,
0 tests completed and failed residual checks,
0 tests skipped because of illegal input values.
----------------------------------------------------------------------------
End of Tests.
============================================================================
