Operating Environment

From UFRC
Jump to: navigation, search

The University of Florida Phase I and II cluster has an up-to-date operating environment to offer our users the flexibility needed for operation of their projects.

System Software

Operating System

  • RHEL6 (Kernel - 2.6.32-220.17.1.el6.x86_64 )

File System

  • Lustre 1.8.6
    • /scratch/ufhpc (230 TB)
    • /cms (1,100 TB)

Compilers

  • Intel Compiler Suite
    • Composer XE 2011 SP1
    • Composer XE 2013 (sp1.0.080)
    • Composer XE 2013 (sp1.3.174)

Math Libraries

  • MKL
    • Composer XE 2011 SP1
    • Composer XE 2013 (sp1.0.080)
    • Composer XE 2013 (sp1.3.174)

GPU Development

Batch System

Monitoring

InfiniBand Stack

  • Native RHEL6

MPI Implementations

Editors

Hardware

Clusters and Hosts (Current)

Type # Processor Cores Memory GPUs IB Total Cores Total Memory Notes
HiPerGator 2.0
Compute 916 2 Intel E5-2698v3 (2.3GHz) 32 128 GB 0 FDR 32x916=29312 128x916=117248GB
GPU 8 2 Intel E5-2683v3 (2.0GHz) 28 128 GB 2 FDR 28x8=224 128x8=1024GB 3
GPU 8 2 Intel E5-2683v3 (2.0GHz) 28 128 GB 2 FDR 28x8=224 128x8=1024GB 5
GPU 8 2 Intel E5-2683v3 (2.0GHz) 28 128 GB 2 FDR 28x8=224 128x8=1024GB 3,4
GPU 8 2 Intel E5-2683v3 (2.0GHz) 28 128 GB 2 FDR 28x8=224 128x8=1024GB 3,5
HiPerGator
Compute 256 4 AMD Opteron 6378 (2.4GHz) 64 256 GB 0 FDR 64x256=16384 256x256=65536GB
Rack c10
Compute 68 2 AMD Opteron 6220 (3.0 GHz) 16 64 GB 0 - 68x16=1088 68x64=4,352 GB
Rack c11
Compute 16 2 Intel X5675 (3.0 GHz) 12 96 GB 0 QDR 16x12=192 16x96=1,536 GB
GPU 10 2 Intel E5-2643 (3.3 GHz) 8 64 GB 4 FDR 10x8=80 10x64=640 GB 1
Rack c12
Compute 8 2 Intel X5675 (3.0 GHz) 12 96 GB 0 QDR 8x12=96 8x96=768 GB
GPU 16 2 AMD Opteron 6220 (3.0 GHz) 16 32 GB 2 QDR 16x8=128 16x64=1024 GB 1
Cobra
GPU 4 1 Intel E5675 (3.0GHz) 6 24 GB 2 QDR 4x6=24 4x24=96 GB 2
Vette
GPU 1 2 Intel E5620 (2.4GHz) 8 24 GB 8 - 1x8=8 1x24=24 GB 2
Big Mem
Compute 1 2 Intel X7560 (2.3GHz) 16 128 GB 0 - 1x16=16 1x128=128 GB
Compute 1 4 AMD Opteron 6174 (2.2GHz) 48 512 GB 0 - 1x48=48 1x512=512 GB
Compute 1 8 Intel E7-8850 (2.0GHz) 80 1024 GB 0 - 1x80 1x1024=1024 GB
Phase IV
Compute 128 2 AMD Opteron 4184 (2.8GHz) 12 32 GB 0 QDR 128x12=1536 128x32=4096 GB
  1. nVidia Tesla M2090 GPUs, 6GB Device RAM
  2. nVidia Tesla M2070 GPUs, 6GB Device RAM
  3. nVidia Tesla K80 GPU, 12GB Device RAM
  4. nVidia Grid K2 GPU
  5. Intel Xeon Phi 5100 Co-processor

Clusters and Hosts (Retired)

Type # Processor Cores Memory GPUs IB Total Cores Total Memory Notes
Phase II
Compute 202 2 AMD Opteron 275 (2.2 GHz) 4 4 GB 0 SDR 202x4=808 202x4=808 GB
Compute 2 4 AMD Opteron 875 (2.2 GHz) 8 16 GB 0 SDR 2x8=16 2x16=32 GB
Compute 120 2 AMD Opteron 275 (2.2 GHz) 4 8 GB 0 SDR 120x4=480 120x8=960 GB
Phase III
Compute 91 2 Intel E5462 (2.8 GHz) 8 32 GB 0 DDR 91x8=728 91x32=2912 GB
Compute 21 2 Intel E5462 (2.8 GHz) 8 64 GB 0 DDR 21x8=168 21x64=1344 GB
GPU 4 1 Intel E5462 (2.8 GHz) 4 16 GB 2 DDR 4x4=16 4x16=64 GB 1
Wukong
Compute 16 2 Intel E5420 (2.5GHz) 8 16 GB 0 - 16x8=128 16x16=1024 GB
Kong Ming
Compute 32 2 AMD Opteron 2427 (2.2GHz) 12 32 GB 0 DDR 32x12=384 32x32=1024 GB
  1. S1070 GPUs, 4GB Device RAM

Networks and Interconnects

New Physics Building (Phase II)

  • InfiniBand Interconnect
    • 50% Blocking Fat Tree Topology
    • 2 Cisco 7008 Core IB Switches (96 Ports 4x IB)
    • 14 Cisco 7000 Leaf IB Switches (24 x 4x SDR)
    • 8 Silverstorm 9024S IB Leaf Switches (24 x 4X SDR/DDR)
    • 1 Cisco 3012 EtherNet/Fibre Channel Gateway
      • 4 FC Gateway Modules
      • 6 EtherNet Gateway Modules
  • Gigabit Ethernet Interconnect (Management)
    • Cisco 6506 Gigabit Ethernet Switch
    • 8 3Com 2848 48-Port Gigabit Ethernet Switches
    • 4 Foundry FastIron Edge X448 Ethernet Switches

Larsen Hall (Phase III)

  • InfiniBand Interconnect
  • 50% Blocking Fat Tree Topology
  • Voltaire 2012
    • 2 Fabric Controllers (Spines)
    • 3 24 Port (4x DDR IB) Modules
    • 2 20210G IB/Ethernet Gateway Modules
      • 2 10 Gb Ethernet (XFP Slots)
      • 20 Ports (4x DDR IB)
  • Force-10 C150 Ethernet Switch
    • 1 RP Module
    • 1 8 Port 10 Gb Ethernet (XFP Slots) Module

Storage

HPC (Shared)

  • File System: /scratch/hpc
  • Lustre 1.8
  • 32 Lustre OSTs (RAID LUNs)
  • 230 TB Usable Storage

CMS (Dedicated)

  • File System: /cms
  • Lustre 1.8
  • 12 Lustre OSSs (Servers)
  • 101 Lustre OSTs (RAID LUNs)
  • 1,100 TB Usable Storage