Operating Environment

From UF HPC Wiki

The University of Florida Phase I and II cluster has an up-to-date operating environment to offer our users the flexibility needed for operation of their projects.

Contents 1 Operating System 2 Monitoring Software 3 System Software 4 Hardware 4.1 Compute Nodes 4.2 Networking 4.2.1 New Physics Building 4.2.2 Larsen Hall 5 Node Performance 5.1 Penguin Relion 1670

Operating System

• CentOS 4.5 (Kernel - 2.6.18-8.1.14.el5.L-1642)
• Lustre Filesystem
  • How to create a System mirror
• Torque queueing system
• Cisco IOS

Monitoring Software

• Nagios

System Software

• GNU Compiler Suite (3.4.6-8)
  • gcc
  • g++
  • g77

• GNU Compiler Suite (4.1.1-53)
  • gcc
  • g++
  • gfortran

• Intel Compiler Suite (9.1.044)
  • icc
  • ifort

• Intel Compiler Suite (10.1.015)
  • icc
  • ifort

• Math Libraries

• OFED 1.2 IB Stack
  • Kernel IB
  • OpenMPI (1.2.2)
  • MVAPICH (0.9.9)
  • MVAPICH2 (0.9.8)

• Lustre 1.6.4.2 (Parallel/Cluster File System)
• MySQL
• Pico or Nano

Hardware

Compute Nodes

Type	#	Processor	Cores	Memory	Infiniband	Total Cores	Total Memory
Compute	202	Two dual core 2.2GHz AMD Opteron 275	4	4 GB	YES	202 x 4 = 808	202 x 4 = 808
Compute	120	Two dual core 2.2GHz AMD Opteron 275	4	8 GB	YES	120 x 4 = 480	120 x 8 = 960
Compute	80	Two dual core 2.2GHz AMD Opteron 275	4	8 GB	NO	80 x 4 = 320	80 x 8 = 640
Compute	2	Four dual core 2.2GHz AMD Opteron 875	8	16 GB	YES	2 x 8 = 16	2 x 16 = 32
Compute	91	Two quad core 2.8GHz Intel E5462	8	32 GB	YES	91 x 8 = 728	91 x 32 = 2912
Compute	21	Two quad core 2.8GHz Intel E5462	8	64 GB	YES	21 x 8 = 168	21 x 64 = 1344
Compute	4	One quad core 2.8GHz Intel E5462	4	16 GB	YES	4 x 4 = 16	4 x 16 = 64
MDS	2	Two dual core 2.0GHz AMD Opteron 2212HE	4	8 GB	YES	2 x 4 = 8	2 x 8 = 16
OSS	8	Two dual core 2.2GHz AMD Opteron 275	4	4 GB	YES	8 x 4 = 32	8 x 4 = 32
OSS	2	Two quad core 2.0GHz AMD Opteron 2350	4	16 GB	YES	2 x 4 = 8	2 x 16 = 32

Networking

New Physics Building

• InfiniBand Interconnect
  • 50% Blocking Fat Tree Topology
  • 2 Cisco 7008 Core IB Switches (96 Ports 4x IB)
  • 14 Cisco 7000 Leaf IB Switches (24 x 4x SDR)
  • 8 Silverstorm 9024S IB Leaf Switches (24 x 4X SDR/DDR)
  • 1 Cisco 3012 EtherNet/Fibre Channel Gateway
    • 4 FC Gateway Modules
    • 6 EtherNet Gateway Modules

• Gigabit Ethernet Interconnect (Management)
  • Cisco 6506 Gigabit Ethernet Switch
  • Cisco 4948 Gigabit Ethernet Switch
  • 8 3Com 2848 48-Port Gigabit Ethernet Switches
  • 4 Foundry FastIron Edge X448 Ethernet Switches

Larsen Hall

• InfiniBand Interconnect
• 50% Blocking Fat Tree Topology
• Voltaire 2012
  • 2 Fabric Controllers (Spines)
  • 3 24 Port (4x DDR IB) Modules
  • 2 20210G IB/Ethernet Gateway Modules
    • 2 10 Gb Ethernet (XFP Slots)
    • 20 Ports (4x DDR IB)

• Force-10 C150 Ethernet Switch
  • 1 RP Module
  • 1 8 Port 10 Gb Ethernet (XFP Slots) Module

Node Performance

Penguin Relion 1670

• 64 GB RAM (See configuration details above)
• OpenMPI 1.2.2 (--btl sm,self)
• Goto BLAS (1.26, -lgoto_penrynp-r1.26)

mpirun --mca btl sm,self -hostfile mpihosts -np 8 ./xhpl
============================================================================
HPLinpack 1.0a  --  High-Performance Linpack benchmark  --   January 20, 2004
Written by A. Petitet and R. Clint Whaley,  Innovative Computing Labs.,  UTK
============================================================================

An explanation of the input/output parameters follows:
T/V    : Wall time / encoded variant.
N      : The order of the coefficient matrix A.
NB     : The partitioning blocking factor.
P      : The number of process rows.
Q      : The number of process columns.
Time   : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

The following parameter values will be used:

N      :   90800 
NB     :     184 
PMAP   : Row-major process mapping
P      :       2 
Q      :       4 
PFACT  :   Right 
NBMIN  :       4 
NDIV   :       2 
RFACT  :   Right 
BCAST  :  2ringM 
DEPTH  :       0 
SWAP   : Spread-roll (long)
L1     : transposed form
U      : transposed form
EQUIL  : yes
ALIGN  : 8 double precision words

----------------------------------------------------------------------------

- The matrix A is randomly generated for each test.
- The following scaled residual checks will be computed:
   1) ||Ax-b||_oo / ( eps * ||A||_1  * N        )
   2) ||Ax-b||_oo / ( eps * ||A||_1  * ||x||_1  )
   3) ||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo )
- The relative machine precision (eps) is taken to be          1.110223e-16
- Computational tests pass if scaled residuals are less than           16.0

============================================================================
T/V                N    NB     P     Q               Time             Gflops
----------------------------------------------------------------------------
WR03R2R4       90800   184     2     4            6185.92          8.068e+01
----------------------------------------------------------------------------
||Ax-b||_oo / ( eps * ||A||_1  * N        ) =        0.0030523 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_1  * ||x||_1  ) =        0.0029648 ...... PASSED
||Ax-b||_oo / ( eps * ||A||_oo * ||x||_oo ) =        0.0004746 ...... PASSED
============================================================================

Finished      1 tests with the following results:
              1 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
----------------------------------------------------------------------------

End of Tests.
============================================================================