Operating Environment

From UF HPC Wiki

Revision as of 19:53, 28 October 2009 by Hpc (Talk | contribs)
Jump to: navigation, search

The University of Florida Phase I and II cluster has an up-to-date operating environment to offer our users the flexibility needed for operation of their projects.

Contents

Operating System

Monitoring Software

System Software

  • GNU Compiler Suite (3.4.6-8)
    • gcc
    • g++
    • g77
  • GNU Compiler Suite (4.1.1-53)
    • gcc
    • g++
    • gfortran
  • Intel Compiler Suite (9.1.044)
    • icc
    • ifort
  • Intel Compiler Suite (10.1.015)
    • icc
    • ifort
  • OFED 1.2 IB Stack
    • Kernel IB
    • OpenMPI (1.2.2)
    • MVAPICH (0.9.9)
    • MVAPICH2 (0.9.8)

Hardware

Compute Nodes

  • 202 Compute Nodes (3 Racks)
    • Rackable Systems C1000 1U Chassis
    • 4 AMD Opteron Cores (2 x 275, 2.2GHz)
    • 4 GB DDR400 RAM
    • 300 GB SATA Disk (local storage)
    • 1 Cisco 4x Infiniband HCA (x8 PCI-E)
    • DC Power
  • 120 Compute Nodes (2 Racks)
    • Rackable Systems C1000 1U Chassis
    • 4 AMD Opteron Cores (2 x 275, 2.2GHz)
    • 8GB DDR400 RAM
    • 80GB SATA Disk (local storage)
    • 1 Mellanox Technologies MT25208 4X Infinband HCA (8X PCI-E)
    • AC Power
  • 80 Compute Nodes (1 Rack)
    • Rackable Systems C1000 1U Chassis
    • 4 AMD Opteron Cores (2 x 275, 2.2GHz)
    • 8GB DDR400 RAM
    • 80GB SATA Disk (local storage)
    • AC Power
  • 2 "Fat" Nodes
    • Rackable Systems C4002 4U Chassis
    • 8 AMD Opteron Cores (4 x 275, 2.2GHz)
    • 32 GB DDR400 RAM
    • 300 GB Ultra-320 SCSI Disk (Local Storage)
    • AC Power
  • 112 Compute Nodes (4 Racks)
    • Penguin Relion 1670 Servers (Operating Manual)
    • 8 Intel E5462 Cores (2 x E5462, 2.80GHz)
    • 1600 MHz FSB
    • 800MHz DDR2 RAM (PC2 6400)
      • 64 GB (21 Nodes)
      • 32 GB (91 Nodes)
    • 250GB SATA Disk (Local Storage)
    • AC Power
  • 4 nVidia Tesla GPU Host Nodes
    • SuperMicro 6025 SuperServer
    • 4 Intel 5462 Cores (1 x E5462, 2.80GHz)
    • 16 GB RAM (PC2 6400)
    • 250 GB SATA Disk (Local Storage)
    • 2x Tesla C1060 GPUs
    • AC Power

I/O Nodes and Storage (UF HPC Scratch)

  • 1 Metadata Server (Lustre MDS)
    • Penguin Altus 1650
    • 4 AMD Opteron 2212 HE (2.0 GHz)
    • 8 GB RAM
    • 4 750 GB SATA II Disks (dual mirrors)
    • 1 Mellanox IB HCA (4x SDR)
  • 8 I/O Nodes (Lustre OSSs)
    • Rackable Systems C3002 3U Chassis
    • 4 AMD Opteron Cores (2 x 275, 2.2GHz)
    • 4 GB DDR400 RAM
    • 300 GB SATA Disk (System Disk)
    • 3 QLogic 2340 FC HCAs (2 Gb/s)
    • 1 Cisco 4x Infiniband HCA (x8 PCI-E)
  • Storage Subsystem: 43 TB (Raw)
    • 12 OmniStore 4932F RAID Controllers (1 GB Cache)
    • 144 300 GB dual-ported FC Disks (10,000 RPM)
    • 2 Cisco MDS 9216i FC Switches
    • 1 Cisco 3012 FC to IB Gateway Switch
    • 1.5+ GB/sec Sustainable Random Access I/O

I/O Nodes and Storage (CRN Scratch)

  • 1 Metadata Server (Lustre MDS)
    • Penguin Altus 1650
    • 4 AMD Opteron 2212 HE (2.0 GHz)
    • 8 GB RAM
    • 4 750 GB SATA II Disks (dual mirrors)
    • 1 Mellanox IB HCA (4x SDR)
  • 2 I/O Nodes (Lustre OSSs)
    • Penguin Altus 2650SA
    • 4 AMD Opteron Cores (2 x 2350, 2.0 GHz)
    • 16 GB DDR400 RAM
    • 2 160 GB SATA II Disk (mirrored)
    • 3 QLogic 2462 FC HCAs (4 Gb/s)
    • 1 Mellanox InfiniHost III Ex Infiniband HCA (x8 PCI-E)
    • 1 Chelsio S310E 10GbE NIC
  • Storage Subsystem: 104 TB (Raw)
    • 6 RAID Inc. Falcon III 24-Bay RAID Chassis
    • 6 Falcon III RAID Controllers (2 GB Cache)
    • 144 750 GB SATA II Disks
    • 1 Cisco 3012 FC to IB Gateway Switch
    • 1.8+ GB/sec Sustainable Random Access I/O

Networking

New Physics Building

  • InfiniBand Interconnect
    • 50% Blocking Fat Tree Topology
    • 2 Cisco 7008 Core IB Switches (96 Ports 4x IB)
    • 14 Cisco 7000 Leaf IB Switches (24 x 4x SDR)
    • 8 Silverstorm 9024S IB Leaf Switches (24 x 4X SDR/DDR)
    • 1 Cisco 3012 EtherNet/Fibre Channel Gateway
      • 4 FC Gateway Modules
      • 6 EtherNet Gateway Modules
  • Gigabit Ethernet Interconnect (Management)
    • Cisco 6506 Gigabit Ethernet Switch
    • Cisco 4948 Gigabit Ethernet Switch
    • 8 3Com 2848 48-Port Gigabit Ethernet Switches
    • 4 Foundry FastIron Edge X448 Ethernet Switches

Larsen Hall

  • InfiniBand Interconnect
  • 50% Blocking Fat Tree Topology
  • Voltaire 2012
    • 2 Fabric Controllers (Spines)
    • 3 24 Port (4x DDR IB) Modules
    • 2 20210G IB/Ethernet Gateway Modules
      • 2 10 Gb Ethernet (XFP Slots)
      • 20 Ports (4x DDR IB)
  • Force-10 C150 Ethernet Switch
    • 1 RP Module
    • 1 8 Port 10 Gb Ethernet (XFP Slots) Module

Node Performance

Penguin Relion 1670

  • 64 GB RAM (See configuration details above)
  • OpenMPI 1.2.2 (--btl sm,self)
  • Goto BLAS (1.26, -lgoto_penrynp-r1.26) 
mpirun --mca btl sm,self -hostfile mpihosts -np 8 ./xhpl
============================================================================
HPLinpack 1.0a  --  High-Performance Linpack benchmark  --   January 20, 2004
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Labs.,  UTK
============================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :   90800 
NB     :     184 
PMAP   : Row-major process mapping
P      :       2 
Q      :       4 
PFACT  :   Right 
NBMIN  :       4 
NDIV   :       2 
RFACT  :   Right 
BCAST  :  2ringM 
DEPTH  :       0 
SWAP   : Spread-roll (long)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

----------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual checks will be computed:
   1) ||Ax-b||_oo / ( eps * ||A||_1  * N        )
   2) ||Ax-b||_oo / ( eps * ||A||_1  * ||x||_1  )
   3) ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo )
- The relative machine precision (eps) is taken to be          1.110223e-16
- Computational tests pass if scaled residuals are less than           16.0

============================================================================
T/V                N    NB     P     Q               Time             Gflops
----------------------------------------------------------------------------
WR03R2R4       90800   184     2     4            6185.92          8.068e+01
----------------------------------------------------------------------------
||Ax-b||_oo / ( eps * ||A||_1  * N        ) =        0.0030523 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_1  * ||x||_1  ) =        0.0029648 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) =        0.0004746 ...... PASSED
============================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
----------------------------------------------------------------------------

End of Tests.
============================================================================
Retrieved from "http://wiki.hpc.ufl.edu/index.php/Operating_Environment"