Operating Environment

From UFRC
Jump to: navigation, search

The University of Florida Phase I and II cluster has an up-to-date operating environment to offer our users the flexibility needed for operation of their projects.

System Software

Operating System

  • RHEL6 (Kernel - 2.6.32-220.17.1.el6.x86_64 )

File System

  • Lustre 1.8.6
    • /scratch/ufhpc (230 TB)
    • /cms (1,100 TB)

Compilers

Math Libraries

  • MKL
    • Composer XE 2011 SP1

GPU Development

Batch System

Monitoring

InfiniBand Stack

  • Native RHEL6

MPI Implementations

Editors

Hardware

Clusters and Hosts

Type # Processor Cores Memory GPUs IB Total Cores Total Memory Notes
Phase II
Compute 202 2 AMD Opteron 275 (2.2 GHz) 4 4 GB 0 SDR 202x4=808 202x4=808 GB
Compute 2 4 AMD Opteron 875 (2.2 GHz) 8 16 GB 0 SDR 2x8=16 2x16=32 GB:
Compute 120 2 AMD Opteron 275 (2.2 GHz) 4 8 GB 0 SDR 120x4=480 120x8=960 GB
Phase III
Compute 91 2 Intel E5462 (2.8 GHz) 8 32 GB 0 DDR 91x8=728 91x32=2912 GB
Compute 21 2 Intel E5462 (2.8 GHz) 8 64 GB 0 DDR 21x8=168 21x64=1344 GB
GPU 4 1 Intel E5462 (2.8 GHz) 4 16 GB 2 DDR 4x4=16 4x16=64 GB 1
Phase IV
Compute 128 2 AMD Opteron 4184 (2.8GHz) 12 32 GB 0 QDR 128x12=1536 128x32=4096 GB
Rack Six
Compute 68 2 AMD Opteron 6220 (3.0 GHz) 16 64 GB 0 - 68x16=1088 68x64=4,352 GB
Rack 23
Compute 16 2 Intel X5675 (3.0 GHz) 12 96 GB 0 QDR 16x12=192 16x96=1,536 GB
GPU 10 2 Intel E5-2643 (3.3 GHz) 8 64 GB 4 FDR 10x8=80 10x64=640 GB 2
Rack 24
Compute 8 2 Intel X5675 (3.0 GHz) 12 96 GB 0 QDR 8x12=96 8x96=768 GB
GPU 16 2 AMD Opteron 6220 (3.0 GHz) 16 32 GB 2 QDR 16x8=128 16x64=1024 GB 2
Cobra
GPU 4 1 Intel E5675 (3.0GHz) 6 24 GB 2 QDR 4x6=24 4x24=96 GB 3
Vette
GPU 1 2 Intel E5620 (2.4GHz) 8 24 GB 8 - 1x8=8 1x24=24 GB 3
Big Mem
Compute 1 2 Intel X7560 (2.3GHz) 16 128 GB 0 - 1x16=16 1x128=128 GB
Compute 1 4 AMD Opteron 6174 (2.2GHz) 48 512 GB 0 - 1x48=48 1x512=512 GB
Compute 1 8 Intel E7-8850 (2.0GHz) 80 1024 GB 0 - 1x80 1x1024=1024 GB
Wukong
Compute 16 2 Intel E5420 (2.5GHz) 8 16 GB 0 - 16x8=128 16x16=1024 GB
Kong Ming
Compute 32 2 AMD Opteron 2427 (2.2GHz) 12 32 GB DDR 32x12=384 32x32=1024 GB
  1. S1070 GPUs, 4GB Device RAM
  2. M2090 GPUs, 6GB Device RAM
  3. M2070 GPUs, 6GB Device RAM

Networks and Interconnects

New Physics Building (Phase II)

  • InfiniBand Interconnect
    • 50% Blocking Fat Tree Topology
    • 2 Cisco 7008 Core IB Switches (96 Ports 4x IB)
    • 14 Cisco 7000 Leaf IB Switches (24 x 4x SDR)
    • 8 Silverstorm 9024S IB Leaf Switches (24 x 4X SDR/DDR)
    • 1 Cisco 3012 EtherNet/Fibre Channel Gateway
      • 4 FC Gateway Modules
      • 6 EtherNet Gateway Modules
  • Gigabit Ethernet Interconnect (Management)
    • Cisco 6506 Gigabit Ethernet Switch
    • 8 3Com 2848 48-Port Gigabit Ethernet Switches
    • 4 Foundry FastIron Edge X448 Ethernet Switches

Larsen Hall (Phase III)

  • InfiniBand Interconnect
  • 50% Blocking Fat Tree Topology
  • Voltaire 2012
    • 2 Fabric Controllers (Spines)
    • 3 24 Port (4x DDR IB) Modules
    • 2 20210G IB/Ethernet Gateway Modules
      • 2 10 Gb Ethernet (XFP Slots)
      • 20 Ports (4x DDR IB)
  • Force-10 C150 Ethernet Switch
    • 1 RP Module
    • 1 8 Port 10 Gb Ethernet (XFP Slots) Module

Storage

HPC (Shared)

  • File System: /scratch/hpc
  • Lustre 1.8
  • 32 Lustre OSTs (RAID LUNs)
  • 230 TB Usable Storage

CMS (Dedicated)

  • File System: /cms
  • Lustre 1.8
  • 12 Lustre OSSs (Servers)
  • 101 Lustre OSTs (RAID LUNs)
  • 1,100 TB Usable Storage