NCHMF Update

May, 2014
Des Albert
Pre-sales Engineer
APMEA

Agenda
Solutions for Earth System Modelling
Experience with UM
Cray XC 30 Overview
Aries Interconnect
Network Topology
Performance & Benchmarks
Questions / Discussion

Cray Solutions for Earth System Modeling

High-resolution numerical weather prediction and earth
system climate modeling are grand challenge problems.
The ability to perform weather and climate simulations at a
range of spatial and temporal scales is critical:
Results in exponential increases in computational and storage and

data management needs.

Cray systems are used in a wide range of areas and

configurations:

5/7/2014

Operational numerical weather prediction

Numerical weather prediction research
Climate change research
As model development platforms for extreme scale architectures
Compute and storage sizes range from Terascale to Petascale
Cray Confidential

Crays Presence in the Weather and Climate

Community
Earth System Modeling is a key area on Cray Systems
worldwide:
Dedicated operational NWP and research centers
Multi-disciplinary research centers
From Teraflops to Petaflops
Recent Cray XC30 wins at DWD, ECMWF, FMI and NEA

NAVO

MeteoSwiss

NRL

5/7/2014

TerraMe
Laboratory

Cray Confidential

FMI

NAMEM
Mongolia

DWD - Deutscher Wetterdienst

Germany's National Meteorological Service, the
Deutscher Wetterdienst (DWD), is responsible for
meeting meteorological requirements arising from all
areas of economy and society in Germany.

Cray XC30 XCW and XCE

Two Cray XC30 systems
1 production, 1 research/backup
2 Sonexion storage systems, 3 PB
Applications:
Numerical weather prediction
Focus on higher resolution and
accurate global/regional forecasts
Climate research

Testimonial

System Overview

XCW and XCE

It is imperative that we equip our
researchers and scientists with
scalable, productive, and above all,
highly reliable supercomputing
systems. The Cray XC30
supercomputers will be valuable
resources for us, and we are pleased
to be working with Cray."
Prof. Dr. Gerhard Adrian, President of DWD

5/7/2014

Cray Confidential

ECMWF UK
European Centre for Medium-Range Weather Forecasting

System Overview

The new Cray facility will help improve ECMWFs forecasting capabilities by providing high
performance computing to support:
Increased resolution and model enhancements.
Development of ensemble-based data assimilation methods in conjunction with the
Ensemble Prediction System.
Better representation of physical processes and further increased use of satellite
observation data.
Key acquisition criteria included:
Sustained performance
System functionality and reliability
Total cost of ownership
Ventus & Anemos

5/7/2014

Cray XC30 Ventus & Anemos

2 XC30 Systems (liquid cooled)
1 XC30-AC (air cooled)

Integrated Sonexion storage

Testimonial

Applications include weather

forecasting and climate research.

"I am looking forward to working with

Cray to enable us to continue
improving the skill of our weather
predictions. The Cray system will be a
strong foundation to meet our
strategic, operational and research
objectives."
Alan Thorpe, ECMWF Director-General

Cray Confidential

Singapore National Environment Agency (NEA)

NEA recently selected Cray for the
provision of their next
supercomputing facility.
NEA is the leading public
organization responsible for
improving and sustaining a clean
and green environment in
Singapore.
Provides timely weather
information to support public
safety and socio-economic
activities, including haze alerts.
55+ Tflop/s Cray XC30.

5/7/2014

Cray Confidential

NCSA National Center for Supercomputing Applications

Blue Waters, located at the University of Illinois at Urbana-Champaign, is one of the most
powerful supercomputers in the world, and is the fastest supercomputer on a university campus
anywhere in the world. Scientists and engineers across the country use the power of Blue Waters
to tackle a wide range of challenging problems, from predicting the behavior of complex biological
systems to simulating the evolution of the cosmos.

5/7/2014

Cray XK7 Blue Waters

Peak performance estimated > 13 PF
300+ compute/storage cabinets
25,000+ total compute nodes including
both CPUs and accelerators
Largest integrated high-performance
computing & storage system
More than 1.5 PB of memory
More than 25 PB of disk storage
Up to 500 PB of tape storage

Testimonial

System Overview

Blue Waters

Cray Confidential

Blue Waters is a national resource

that will allow researchers access to
the most powerful computational
resources available today, furthering
research across all scientific
disciplines and enabling the
investigation of problems not possible
before.
Farnam Jahanian, Head of NSFs Directorate for
Computer and Information Science and Engineering

Some Additional Recent Installations and Wins

Leadership facilities worldwide that support significant
earth system modeling communities, including

5/7/2014

NCSA Blue Waters Cray XK7: >13 PF

ORNL Titan Cray XK7: >27 PF
NERSC Edison Cray XC30: >2 PF
EPSRC/EPCC Archer Cray XC30

Cray Confidential

Selected Cray Customers with UM

Development platform for UM
Strong application support s not only for Operational
Center but for Research Center

5/7/2014

KMA CoE-ESM, Cray XT5/XE6

NEA Cray XC30
EPSRC/EPCC Archer Cray XC30
Recent Win at SAWS Cray XC30

Cray Confidential

10

Project UPSCALE: UK Climate Research Team

Wins Major Computing Award on the HLRS XE6
UPSCALE project:
UK on PRACE: weather-resolving Simulations of Climate for globAL

Environmental risk

The team comprises climate scientists at the UK Met

Office and National Centre for Atmospheric Science
(NCAS)
Led by Pier Luigi Vidale, Willis Professor of Climate System Science

and Climate Hazards at the University of Reading's Meteorology

department and Director of the Weather and Climate Hazards
Laboratory.

Granted substantial computing resources on the HLRS

Hermit XE6 by the Partnership for Advanced Computing in
Europe (PRACE) Access Committee.

5/7/2014

Cray Confidential

11

5/7/2014

12

 Todays science is:

Data-intensive
Data-driven
Compute-intensive
Multi-scale, multi-physics
Resource requirements are
driven by multiple dimensions.
Data Tsunami is defying
standard approaches to
interpretation
Volume and complexity of
data are too much for either
humans or current
technologies for effective
analysis

Fidelity / Resolution

Data and Computational Drivers

Data and Compute

Resources

Figure adopted from

Computing Issues for
WCRP Weather and
Climate Modeling,
James Kinter and Michael
Wehner

Data driven discovery and advanced analytics are rapidly

becoming a competitive differentiators providing insight and
predictive capabilities.
5/7/2014

Cray Confidential

13

Petascale towards Pre-Exascale

Petascale systems have been key:
In enabling transformational capabilities across
a broad range of science and engineering,
including industry
Key to provide capabilities of immense socio-

economic benefit to a broad range

consequence questions.

Fully coupled biogeochemistryphysical climate simulation

As development platforms for preparing earth

system models for extreme scale capabilities.

NCSA Blue Waters and ORNL Titan will further

push the boundary of Petascale computing.
Exascale applications oriented efforts will play a
key role to prepare for the eventual availability of
systems.
For modelers either you have run at 100,000+
cores, or you have not.

HIV immature virion simulations

Kinetic simulations of the magnetosphere

5/7/2014

Cray Confidential

Slide 14
14
Climate Specialists Workshop

State-of-the-Art Earth System Modeling

Several efforts around the world that are pushing the
boundaries of science and computational science.
Addressing a number of challenges:
System scale
Disruptive technologies such as accelerators
Changes in system characteristics including greater parallelism,

component performance balance, memory size, I/O,

Power consumption
Programmability, maintainability and portability.

Roughly two broad categories of effort:

Maximizing use of Petascale Facilities science and development

uses
Pre-exascale Development Algorithmic design for massive for
massive parallelism, resource constraints and robustness.
Today much of the effort is focused on the effective use of accelerators

Cray experts and developers are involved in a wide range

of efforts around the world.
5/7/2014

Cray Confidential

15

Selected NCSA Blue Waters Earth Sciences Petascale

Computing Resource Allocations (PRACs)

Supercomputing at the Sustained Petascale Performance

Understanding
tornadoes and their
parent supercells
through ultra-high
resolution
simulation/analysis
Principal Investigator:
Robert Wilhelmson,
University of Illinois at
Urbana-Champaign.

Using petascale
computing capabilities
to address climate
change uncertainties
Principal Investigators:
Donald Wuebbles and
Xin-Zhong Liang,
University of Illinois at
Urbana-Champaign

Testing hypotheses
about climate prediction
at unprecedented
resolutions on the NSF
Blue Waters system
Principal Investigators:
Benjamin Kirtman,
University of Miami;
William Large, University
Corporation For
Atmospheric Research;
David Randall, Colorado
State University; Cristiana
Stan, Institute of Global
Environment and Society

Petascale Design and

Management of Satellite
Assets to Advance
Space Based Earth
Science
Principal Investigator :
Patrick Reed

Transformational Science Across a Range of Areas

5/7/2014

Cray Confidential

16

Petascale WRF Hurricane Sandy Simulation on

Blue Waters
Largest ever storm prediction model using real data of
over 4 billion points used to simulate the landfall of
Hurricane Sandy with WRF application.
Grid size of 9120x9216x48 @ 500m resolution

NCSA Blue Waters Cray XK7 sustained 285 Tflops using

437,760 cores simulating an 18-hr forecast.
Performance will enable research into potential for greater
accuracy when predicting exact landfall time and place,
and wind and water damage.

5/7/2014

Cray Confidential

17

WRF Hurricane Sandy Simulation on Blue

Waters

5/7/2014

Cray Confidential

18

Earth
Sciences

Auto/
Aero

Energy

Life
Sciences

Financial
Services

Storage/Data

Analysis

Computation

Cray Industry Solutions

Industry-specific solutions that leverage Cray

expertise across all technologies and products
19

Computation

Analytics

Storage & Data Management

Big Data

Supercomputing

Cray Supercomputing

20

Extended Portfolio of Cray Computing Solutions

Cray CS300 Series

Cluster Supercomputers

Cray XC30 Series

Supercomputers

Capacity Computing Focus

Capability Computing Focus

Price/Performance/Watt
Flexible system configurations
Industry Standards Technologies
Manageability and Reliability
Modular Scalability

Application Scalability
HPC Optimized HW, SW & IP
Price/Performance
Roadmap Upgradability
Reliability/Availability/Serviceability

Scaling Across the Performance Spectrum

21

Cray CS300-AC Cluster Supercomputer

Overview
Flexible and Scalable
Up to 80 nodes/rack configured with 2
CPUs/node
Interconnect: 3D Torus/Fat Tree, single/dual
rail, QDR/FDR, IB/GigE
Supports a wide range of parallel Storage
File Systems

Manageable
Integrated with Crays HPC Cluster Software
Stack
Includes Crays Advanced Cluster Engine
Management software
Includes network, server, cluster and storage
management
Performance and Flexibility

End-To-End Supercomputer Solutions

Air-Cooled Solution

From design, integration, delivery to on-site

installation and customized services

22

Cray CS300-LC Cluster Supercomputer

Overview
Energy-Efficient Architecture

Liquid-Cooled architecture
Uses warm water instead of chilled water
Directly cools the processor and memory
Uses low-pressure isolated secondary loop
Leak detection and prevention features
integrated with remote monitoring and
reporting

Flexible and Scalable

Up to 60 nodes/rack configured with 2
CPUs/node
Interconnect: 3D Torus/Fat Tree, single/dual
rail, QDR/FDR, IB/GigE
Supports a wide range of parallel Storage File
Systems
Less Energy Lower TCO Faster ROI

End-To-End Supercomputer Solutions

Liquid-Cooled Solution

From design, integration, delivery to on-site

installation and customized services

23

Introducing the Cray XC30TM Supercomputer Series

Adaptive
Supercomputing

Scalable
Performance

Comprehensive
HPC Integration

Flexible Processor
Options & Upgrades

Enhanced Aries
Interconnect

HW & Networking

Hybrid Systems

Global Network
Bandwidth

Adaptive Network
Routing
Advanced Adaptive
Programming Tools

HPC Development
Tools

SW Environment &
Partner Ecosystem
Storage
Reliability &
Resiliency

Cray Linux
25

Cray Aries Custom Interconnect

26

Aries
Aries is the Cray custom

interconnect ASIC used in

the Cray XC product family
40nm process
Die size: 16.6 x 18.9mm
Gate count: 217M
184 lanes of high speed

SerDes
30 optical network lanes
90 electrical network lanes
64 PCI Express lanes

27

Cray XC30 System Building Blocks

Compute
Blade
4 Compute
Nodes

Chassis
Rank 1
Network
16 Compute
Blades
No Cables
64 Compute
Nodes

Group
Rank 2
Network
Passive
Electrical
Network
2 Cabinets
6 Chassis
384 Compute
Nodes

System
Rank 3
Network
Active Optical
Network
Hundreds of
Cabinets
Up to 10s of
thousands of
nodes

28

Cray XC30 Modular Blades

29

Cascade Compute Blade

30

Cray XC30 Fully Populated

Compute Blade
SPECIFICATIONS
Module power:

2014 Watts

PDC max. power:

900 Watt

Air flow req.:

275 cfm

Size:

2.125 in x 12.95 in x 33.5 in

Weight:

<40 lbm

31

PDCs are Upgradeable to New Technology

32

Cray XC30 Quad Processor Daughter Card

Intel Processor (4)

Voltage Reg (2)

Southbridge (2)

DDR Memory (16)

33

Cray XC I/O Module

SPECIFICATIONS
Module power:
PDC max. power:

PCIe
Card
Slots

Riser
Assembly

225 Watt

Air flow req.:

275 cfm

Size:

2.125 in x 12.95 in x 33.5 in

Weight:

Intel 2600 Series

Processor

1650 Watts

35 lbs

Aries

34

Cray XC30 Dragonfly

Topology

35

Dragonfly Network Concept

Goals:
Provide scalable global bandwidth
Exploit low cost of short electrical links
Reduce the required number of global optical hops
Avoid the need for external router cabinets
Dragonfly concept
Construct groups of locally-connected nodes
Treat the group as single super node with very high
radix
Pool all the optical links coming out of the group into a
single dimension
Create a single all-to-all optical stage among the groups

Cray Inc. Proprietary Not For Public Disclosure

Cray XC30 Rank1 Network

o Chassis with 16 compute blades

o 128 Sockets
o Inter-Aries communication over
backplane
o Per-Packet adaptive Routing

37

Cray XC30 Architecture

& Packaging

38

Cray XC30 Rank-2 Copper Network

2 Cabinet
Group
768 Sockets

6 backplanes
connected with
copper cables in a 2cabinet group:
Black Network

16 Aries connected
by backplane
Green Network

4 nodes
connect to a
single Aries

Active optical
cables interconnect
groups
Blue Network
39

Cray XC30 Network

The Cray XC30 system is built around the idea of optimizing
interconnect bandwidth and associated cost at every level

Rank-1
PC Board

Rank-2
Passive CU

Rank-3
Active Optics
40

Copper & Optical Cabling

Optical
Connections

Copper
Connections

41

Cray XC30 Routing

Minimal routes
between any two
nodes in a group
are just two hops
Non-minimal route
requires up to four
hops.
With adaptive routing
we select between
minimal and nonminimal paths based
on load

D
R

The Cray XC30 Class2 Group has sufficient

bandwidth to support
full injection rate for all
384 nodes with nonminimal routing

42

Cray XC30 Network Overview Rank-3 Network

An all-to-all pattern is wired between the

groups using optical cables (blue

network)
Up to 240 ports are available per 2cabinet group
The global bandwidth can be tuned by
varying the number of optical cables in
the group-to-group connections

Group 0

Group 1

Group 2

Group 3

Example: An 4-group system is interconnected with 6

optical bundles. The bundles can be configured between
20 and 80 cables wide

43

Aries Adaptive Routing Algorithm

44

Why is the Dragonfly topology a good idea?

Scalability
Topology scales to very large systems

Performance
More than just a case of clever wiring, this

topology leverages state-of-the-art adaptive

routing that Cray developed with Stanford
University
Smoothly mixes small and large messages
eliminating need for a 2nd network for I/O
traffic

Simplicity
Implemented without external switches
No HBAs or separate NICs and Routers

Price/Performance
Dragonfly maximizes the use of backplanes

and passive copper components

Dragonfly minimizes the use of active optical
components
45

MPI Bandwidth
OSU MPI Multiple Bandwidth / Message Rate Test
12

10

Bandwidth (GB/s)

Aries
FDR

16

64

256

1K
4K
Message size (bytes)

16K

64K

256K

1M

4M

46

MPI Latency
OSU MPI Multi Latency Test
2.50

2.00

Latency (us)

Aries
FDR

1.50

1.00

0.50

0.00

16
64
Message size (bytes)

256

1K

47

What makes a Cray Aries network better?

Scalability to > 500,000 X86 Cores

Cray users run large jobs 20-50% of system size is common

Many examples of 50K-250K MPI tasks per job
Optimized collectives MPI_Allreduce in particular

Optimized short transfer mechanism (FMA)

Provides global access to memory, used by MPI and PGAS

High issue rate for small transfers: 8-64 byte put/get and amo in particular

HPC optimized network

Small packet size 64-bytes

Router bandwidth >> injection bandwidth
Adaptive Routing & Dragonfly topology

Connectionless design

Doesnt depend on a connection cache for performance

Limits the memory required per node

Fault tolerant design

Link level retry on error

Adaptive routing around failed links
Network reconfigures automatically (and quickly) if a component fails
End to end CRC check with automatic software retry in MPI
48

Benchmarks

49

MPI Bandwidths
18

Bandwidth (GBytes/sec)

16
pingpong

14

unidirectional

12

bidirectional

10
8
6
4
2
0

64

128

256

512

1K

2K

4K

8K

16K

32K

64K 128K 256K 512K

1M

Message Size (bytes)

50

MPI_Allreduce Latency
35
30

Time (microseconds)

25
20

Software
Aries CE

15
10
5
0

2000

4000

6000
8000
Number of Processes

10000

12000
51

WRF NewConus 2.5KM Benchmark

18000

16000

14000

12000
XE6 MC12 2.1GHz Gemini 1.1

GFLOP/s

XT6 MC12 2.1GHz SeaStar

IBM Power6

10000

Intel Nehalem 6-core 2.93GHz

Intel Nehalem QC 2.8GHz
IBM BG/P

8000

IBM BG/L
Cascade SB8 2.6GHz

6000

4000
- Computation and halo exchange costs only
- http://www.mmm.ucar.edu/wrf/WG2/benchv3/

2000

0
0
5/7/2014

2000

4000

6000

8000

10000
Cores

12000

14000

16000

18000

20000

Pete Johnsen, Cray, Inc.

52

Build & Go Example ROMS Ocean Model

Build & Go

18.00

Simulated Years / Day

16.00

FDR IB Cluster
Cray XC30

14.00

FDR IB Cluster

12.00
10.00

Item

FDR IB Cluster
FDR IB Cluster

Cray XC30

Processor

Intel E5-2670
Intel E5-2670

Same

6.00

Compiler

Intel 13.1
Intel 13.1

Same

4.00

Compiler
Options

-heap-arrays -ip - Same

-heap-arrays -ip
-O3 -xW free
O3

2.00

Topology

Full Fat Tree

Full Fat Tree

Dragonfly

0.00

MPI
Libraries

Intel 4.0
Intel 4.0

Cray MPICH2
6.0

Math
Libraries

Intel MKL 10.3

Intel MKL 10.3

Cray Libsci
12.1

OS

RHEL 6.2
RHEL 6.2

CLE 5.1

8.00

512

1024

1536

Cores

SC 2013

2048

Copyright 2013 Cray Inc. - Confidential and Proprietary

53

One Benefit of Dragonfly:

Performance independent of placement

Sandia miniApp, models performance of CTH

Running on 2256 node CSCS system ( global bandwidth)
Four concurrent 512 node parallel jobs, different layouts
Runtime in seconds for 100 cycles

1
1 1 1 1 1
1 1 1 1 1
1
1 1 1 1 1
1 1 1 1 1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1
1
1
1
1

1
1 1 1 1 2
2 2 2 2 3
3
4 4 4 4 4
1 1 1 1 1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

2
1 1 1 1 1
2 2 2 2 2
3
4 4 4 4 4
2 2 2 2 2

2
1
2
3
4
2

2
1
2
3
4
2

2
1
2
3
4
2

2
1
2
3
4
2

2
1
3
3
4
2

2
1
3
3
4
2

2
2
3
4
4
2

2 2
2
3 3
4 4
4 4
2 2

2
2
3
4
4
2

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

2
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

3
2
3
4
4
4

1
2
3
4
4
4

1
2
3
4
4
4

1
2
3
4
4
4

1
2
3
4
4
4

1
2
3
4
4
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

1
2
3
4
2
4

3
1 4 3 3 3
3 1 2 1 3
3
4 3 2 4 3
1 1 2 4 1

1
3
2
3
4
3

3
4
3
4
3
4

2
2
4
2
1
4

1
3
4
4
2
4

3
3
4
4
3
2

2
4
2
4
2
2

3
4
1
3
4
1

3
3
3
3
1
1

1
4
4
4
2
1

1
4
1
3
4
1

4
2
1
4
2
4

4
3
1
4
2
4

1
2
4
4
3
4

3
2
1
3
3
1

3
3
2
1
2
4

4
2
1
1
4
2

2
3
4
2
1
2

1
1
2
3
1
3

2
1
1
1
3
3

3
1
3
4
4
1

2
1
2
2
4
1

2
1
4
1
2
4

2
2
1
2
1
4

3
2
2
2
1
2

2
2
1
1
2
2

3
2
2
2
3
1

1
2
4
4
1
1

4
1
4
2
2
4

2
4
3
1
1
1

4
1
2
4
4
1

3
4
3
4
4
1

4
2
1
2
2
2

1
4
2
1
1
2

3
3
1
2
2
2

1
1
1
1
3
3

4
2
2
4
4
1

2
2
1
1
3
2

1
3
1
4
1
4

1
2
1
3
2
1

4
4
1
4
1
1

1
2
4
1
4
1

3
3
1
2
2
3

2
4
4
4
2
4

3
4
3
2
4
2

3
3
1
2
4
2

2
3
3
4
1
4

3
1
2
4
2
1

4
4
1
4
4
3

4
2
3
1
4
4

3
2
2
4
4
3

2
3
2
2
2
4

1
3
4
3
1
4

3
3
1
1
3
4

3
1
4
4
3
4

3
2
1
2
3
1

1
4
4
3
1
1

1
3
1
1
4
4

2
2
3
2
3
4

4
1
4
2
3
3

1
1 1 1 1 1
1 1 1 1 1
1
2 2 2 2 2
2 2 2 2 2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1 1
1
1 1
2 2
2 2
2 2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

2
3 3 3 3 3
4 4 4 4 4
1
2 2 2 2 2
3 3 3 3 3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

3
4
4
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3 3
4
1 1
2 2
3 3
4 4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

4
2 2 2 2 2
3 3 3 3 3
3
3 3 3 3 3
4 4 4 4 4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

4
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
4

1
2
3
3
3
3

1
2
3
3
3
3

1
2
3
3
3
3

1
2
3
3
3
3

1
2
3
3
3
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1 1
2
3 3
3 3
4 4
3 3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
2
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
3

1
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

2
3
3
3
4
4

3
4 3 1 2 1
1 2 3 2 1
2
1 1 1 1 1
3 3 4 2 4

3
1
4
2
1
3

4
4
3
2
3
2

2
2
1
2
1
1

1
4
2
2
3
2

4
3
2
3
4
4

2
3
4
2
4
3

3
4
4
4
4
2

3
3
4
3
2
1

1
4
3
2
1
3

3
3
3
3
1
4

3
3
3
4
4
3

4
2
3
4
2
4

3
3
2
4
1
3

1
4
2
2
1
1

1
1
4
2
2
3

3
2
4
1
4
4

4
4
2
1
3
3

1
2
4
2
4
1

3
4
4
3
1
4

2
4
1
2
3
4

2
3
2
3
4
2

4
1
3
1
2
2

3
2
2
1
2
2

2
4
1
4
4
1

3
1
2
2
1
2

4
1
4
1
1
3

1
3
1
2
2
1

2
4
2
2
2
2

3
3
1
1
1
1

2
3
1
2
4
3

1
4
3
1
4
4

2
4
2
2
3
1

1 4
2
4 2
1 4
3 2
4 2

4
4
2
3
2
1

3
4
4
2
3
2

2
4
3
3
4
1

1
1
1
3
3
4

3
4
2
2
3
2

2
2
4
2
1
2

1
4
3
1
1
4

4
4
3
3
4
2

2
2
2
4
2
3

3
4
4
1
1
4

4
4
4
2
2
2

3
1
1
3
2
2

4
2
1
3
2
2

2
3
2
4
2
4

4
1
2
2
3
3

4
3
1
2
1
3

4
2
1
4
3
1

1
1
4
4
1
1

3
1
4
3
4
3

2
3
2
4
2
1

2
1
3
2
1
1

3
2
2
3
3
1

2
3
4
1
1
3

3
4
3
3
2
3

4
3
1
1
3
4

2
2 2 2 2 2
2 2 2 2 2
2
2 2 2 2 2
2 2 2 2 2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

2
2
2
2
2
2

4
1 1 1 1 1
2 2 2 2 2
3
4 4 4 4 4
1 1 1 1 1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

4
1 1 1 1 1
3 3 3 3 3
1
3 3 3 3 3
2 2 2 2 2

4
1
3
1
3
2

4
1
3
1
3
2

4
1
3
1
3
2

4
1
3
1
3
2

4
1
3
3
3
2

4
1
3
3
3
2

4
1
3
3
3
4

4
1
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
3
4

4
4
1
3
1
4

4
4
1
3
1
4

4
4
1
3
1
4

4
4
1
3
1
4

4
4
1
3
1
4

4
2
1
3
1
4

4
2
1
3
1
4

4
2
1
3
1
2

4
2
1
3
1
2

4
2
1
3
1
2

4
2
1
3
1
2

4
2
1
3
1
2

4
2
1
3
1
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
2

4
3
1
3
2
4

4
3
1
3
2
4

4
3
1
3
2
4

4
3
1
3
2
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
4
4

4
3
1
3
2
4

1
3
1
3
2
4

1
3
1
3
2
4

1
3
1
3
2
4

1
3
1
3
2
4

1
3
1
3
2
4

4
3 1 3 1 4
3 1 4 2 3
4
4 2 2 3 2
3 3 1 4 1

2
1
1
3
4
4

2
1
4
3
1
1

1
1
3
2
2
1

1
3
2
2
4
4

3
2
3
1
2
4

3
3
4
3
2
1

4
2
1
3
4
3

1
3
4
3
4
3

2
4
2
3
2
3

2
1
4
4
1
1

2
4
3
2
4
1

2
4
3
4
3
2

3
2
4
2
2
2

2
2
4
3
3
2

2
1
4
3
2
1

1
1
4
3
4
3

2
2
2
4
3
4

1
4
4
4
2
2

2
3
2
2
3
1

1
4
3
3
4
1

2
3
4
2
3
1

4
1
3
4
4
1

1
3
4
2
3
3

3
2
4
4
1
4

1
4
4
2
4
4

3
4
1
1
4
1

3
2
2
1
1
1

3
2
2
1
1
4

2
4
1
4
1
3

3
1
2
3
4
2

4
3
1
4
4
1

1
2
3
4
2
2

2
3
2
1
4
1

2
4
3
1
1
4

1
4
1
3
1
2

1
1
3
3
3
1

3
1
2
4
4
4

1
1
4
2
1
1

4
3
1
4
4
2

4
2
2
2
2
3

3
1
4
2
3
4

1
2
3
4
3
4

3
1
1
2
3
4

3
1
1
2
1
3

1
2
4
4
4
1

3
2
3
1
2
4

4
1
1
2
1
2

3
4
1
2
4
3

3
1
3
2
2
1

1
1
1
2
3
3

4
4
1
1
1
4

4
1
3
1
4
2

1
2
1
2
2
2

3
1
1
1
2
2

2
4
1
4
4
2

4
2
3
2
2
3

1
1
1
3
4
3

4
3
3
3
3
2

3
3
4
1
1
1

2
3 3 3 3 3
3 3 3 3 3
3
3 3 3 3 3
3 3 3 3 3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

3
3
3
3
3
3

2
3 3 3 3 3
4 4 4 4 4
1
2 2 2 2 2
3 3 3 3 3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

4
4 4 4 4 4
3 3 3 3 3
3
1 1 2 4 4
4 4 4 2 2

4
4
3
3
4
2

4
4
3
1
4
2

4
4
3
1
4
2

4
4
3
1
4
2

4
4
3
1
1
2

4
4
3
1
1
2

4
4
3
1
1
2

4
4
3
1
1
2

4
4
3
1
1
2

4
4
3
1
1
2

4
4
3
1
4
2

4
4
3
1
4
2

4
4
3
1
4
2

4
4
3
1
4
2

3
4
3
1
4
2

3
4
3
1
4
2

3
4
3
1
4
2

3
4
3
1
4
2

3
4
3
1
4
2

3
4
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
2

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

3
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

4
3
3
1
4
1

2
2 1 2 4 4
1 1 1 4 1
2
2 1 4 3 4
3 4 1 2 3

1
4
3
1
2
3

1
1
4
2
4
1

2
1
3
2
1
2

1
3
1
1
4
3

2
3
3
3
3
1

3
3
1
4
1
1

4
3
1
2
2
4

4
3
1
1
1
3

2
1
1
1
2
1

4
4
4
1
4
4

4
2
2
3
4
1

2
3
1
3
3
2

3
1
3
1
2
3

3
3
1
3
3
3

2
2
2
3
2
3

3
4
4
4
4
1

3
1
4
2
1
2

2
2
2
2
4
1

3
2
3
2
1
4

2
3
1
2
4
2

4
3
4
3
2
2

1
1
4
2
2
2

2
2
2
1
1
3

4
3
1
4
4
1

4
3
3
4
2
3

1
2
2
1
4
1

4
2
2
2
1
1

4
1
3
2
2
1

2
3
3
3
4
2

1
2
2
3
4
1

1
4
3
4
4
1

2
4
4
2
3
4

1
4
4
2
3
2

3
1
3
4
4
2

4
1
3
1
4
1

3
1
4
4
3
4

4
3
4
4
1
2

1
4
1
1
3
1

3
3
3
3
4
3

2
3
3
4
1
4

4
2
3
3
1
2

4
4
3
1
2
2

1
2
1
3
2
1

1
4
1
1
3
4

3
1
2
2
1
1

3
4
3
3
3
3

1
1
1
1
3
2

4
4
3
2
1
1

2
4
3
1
3
3

1
3
1
4
4
2

4
4
2
4
3
2

3
4
2
2
3
4

2
2
4
2
2
3

4
4
2
3
1
4

2
1
3
1
2
4

3
3
3
2
2
4

3
4
2
2
1
4

2
3
4
2
4
4

2
4
3
4
2
2

3
3 3 3 3 3
3 3 3 3 3
3
4 4 4 4 4
4 4 4 4 4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

3
3
3
4
4
4

4
1 1 1 1 1
2 2 2 2 2
3
4 4 4 4 4
1 1 1 1 1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

4
1
2
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
3
4
1

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
2
3
4
1
2

1
4 4 4 4 4
2 2 2 2 2
1
2 2 2 2 2
1 1 1 1 1

1
4
2
1
2
1

1
4
2
1
2
1

1
4
2
1
2
1

1
4
2
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
1
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
1

1
4
3
3
2
4

1
4
3
3
2
4

1
4
3
3
2
4

1
4
3
3
2
4

1
4
3
3
2
4

1
2
3
3
2
4

1
2
3
3
1
4

4
2
3
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
4
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
2
1
2
1
1

4
3 1 2 3 4
3 1 2 3 3
4
2 3 2 2 2
2 3 1 1 1

4
1
2
3
4
4

2
4
1
1
1
4

1
3
2
3
2
1

3
4
3
1
3
3

2
3
4
4
2
1

1
2
1
2
4
2

4
1
2
3
1
1

4
1
4
2
3
4

2
1
4
3
4
4

1
2
2
3
2
4

4
2
1
2
3
2

2
2
4
3
1
2

3
4
4
3
4
4

3
4
3
3
4
1

2
3
3
2
4
4

1
3
4
2
1
2

2
4
4
1
2
1

3
3
4
4
3
4

1
1
3
2
1
4

4
3
4
4
3
1

2
4
1
3
4
1

3
4
3
4
1
2

2
4
2
2
3
1

2
1
4
1
4
4

3
1
3
4
3
3

2
3
3
4
3
4

2
3
1
3
4
1

3
3
3
1
2
2

3
1
3
2
1
1

4
1
1
4
1
2

3
1
3
2
3
3

4
1
2
1
3
3

4
3
2
2
4
1

2
3
1
2
2
4

4
4
2
1
2
3

3
3
4
2
1
3

1
2
1
2
3
4

2
4
1
2
4
2

3
4
4
1
3
1

2
2
1
2
1
1

4
3
2
4
4
4

4
2
1
3
4
3

4
2
2
1
4
4

3
4
2
3
1
2

1
4
4
2
3
4

4
3
1
4
3
2

4
2
4
4
3
4

1
4
2
1
1
2

1
2
4
3
1
2

3
4
4
1
4
4

3
2
2
1
2
3

2
4
3
4
4
4

2
4
4
2
1
1

1
3
3
1
1
2

4
2
4
4
4
1

4
3
1
3
1
3

4
2
4
3
4
4

2
1
3
1
2
4

4
2
1
1
2
2

4
4 4 4 4 4
4 4 4 4 4
4
4 4 4 4 4
4 4 4 4 4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

4
4
4
4
4
4

2
3 3 3 3 3
4 4 4 4 4
1
2 2 2 2 2
3 3 3 3 3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

2
3
4
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
1
2
3

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

3
4
1
2
3
4

1
1 1 1 1 1
2 2 2 2 2
2
2 2 2 2 2
2 2 2 2 2

1
1
2
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
4
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
1
2

1
1
1
2
1
2

1
1
1
2
1
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
1
2
2
2

1
1
2
2
2
2

1
1
2
2
2
2

1
1
2
2
2
2

1
1
2
2
2
2

2
3 3 4 2 4
4 4 3 2 2
2
3 1 2 2 1
1 1 3 2 2

4
4
2
4
3
3

4
3
3
4
3
3

1
3
2
1
4
3

2
3
2
3
3
1

2
3
2
4
3
2

1
4
1
2
2
1

4
2
4
3
2
2

3
1
3
2
3
2

1
1
2
4
1
1

3
1
4
4
2
2

3
2
3
1
2
3

3
3
4
4
4
1

3
4
4
3
2
3

4
2
2
2
4
3

1
4
3
2
2
3

3
2
4
2
2
1

2
4
3
3
4
2

2
3
4
1
1
3

2
1
2
1
3
3

4
3
3
2
3
1

3
4
3
2
2
1

3
1
1
2
3
3

3
1
3
4
4
3

2
3
2
4
2
3

3
2
1
4
1
1

2
3
1
2
2
3

1
3
2
4
1
1

1
4
3
1
3
3

3
1
2
4
1
3

4
1
1
2
3
3

1
1
4
3
1
1

1
3
4
4
1
3

2
3
1
1
3
3

3
4
2
3
3
1

4
2
3
3
2
3

4
2
2
1
4
3

1
3
4
4
3
1

4
2
2
3
1
3

3
3
3
4
4
1

4
4
3
2
4
1

2
1
4
3
1
1

3
1
4
4
1
3

2
4
4
2
1
3

3
1
1
2
4
1

2
3
2
4
4
1

4
3
4
2
3
3

3
2
4
1
1
1

1
4
2
1
1
1

2
4
3
4
3
3

1
4
4
3
3
1

2
2
4
2
2
1

4
3
4
4
1
1

1
3
3
3
2
1

2
3
2
3
1
1

3
3
3
3
2
1

3
1
2
2
2
1

1
2
3
2
3
1

3
4
1
3
1
1

4
1
3
2
2
1

Blocks of 512 nodes

69.0

68.8

68.9

68.9

Blocks of 64 nodes
69.4

69.4

69.4

69.5

Average of 32 nodes
69.7

69.7

69.4

69.7

Random nodes
70.9

71.0

70.6

70.5

Application Explorations for Future Interconnects, R.F. Barrett, C.T. Vaughan, S.D. Hammond, and
D.Roweth, Workshop on Large-Scale Parallel Processing, at the International Parallel and Distributed
Processing Symposium: IPDPS 2013, Boston, MA, 2013.

SC 2013

Copyright 2013 Cray Inc. - Confidential and Proprietary

54

Performance independent of placement

New results following upgrade of the CSCS system
28 cabinets, ~5000 nodes, ~40,000 cores
Four concurrent jobs ranging in size from 128 to 8192 ranks
System in production use at the time (80-95% load)
Runtime for 4 simultaneous miniGhost jobs
70
68

Run time (seconds)

66
64
62
60
58

Block layout

56

Random(64) layout

54
52
50

128

256

512

1024

2048

4096

8192

Job size (ranks) with 8 processes per node

5/7/2014

55

Mix of application and streaming I/O traffic

Analysis of the impact of checkpoint I/O traffic on performance
Four jobs spread out across the whole machine with block(64) layout, three

miniGhost plus one performing many-to-few I/O

This is work in progress
Runtime for 4 simultaneous jobs, 3 miniGhost + checkpoint I/O
450

66

400

64

350

62

300

60

250

58

200

I/O bandwidth
miniGhost(4)
miniGhost(3) + I/O

56

150

54

100

52

I/O bandwidth (GB/s)

500

68

Run time (seconds)

70

50

50

256

512

1024

2048

4096

8192

Job size (ranks) with 8 processes per node

5/7/2014

56

Cray Software Stack

57

Cray Linux Environment Focus

Performance
Maximize compute cycles delivered to applications while also
providing necessary services
Lightweight operating system on compute node
Standard Linux environment on service nodes
Optimize network performance through close interaction with
hardware

Stability and Resiliency

Correct defects which impact stability
Implement features to increase system and application robustness

Scalability
Scale to large system sizes without sacrificing stability
Provide system management tools to manage complicated
systems
58

CLE - Operating Systems and I/O

Compute
node

Compute Node Linux

Service
node OS

Standard Linux distro

IO Device drivers

File
systems

Networking

Third-party
extensions

Lustre
NFS

Operating
system
services

Core specialization
Dynamic Shared
Library (DSL) support
Cluster Compatibility
Mode (CCM)
Data Virtualization
Service (DVS)

System
management

Application-Level
Placement Scheduler
(ALPS)
Node Health Checker
(NHC)
Cray Management
Services (CMS)
Command interface

HSN: Gemini and Aries

(GNI, DMAPP)
IBGNI (IB verbs ->
Gemini/Aries), used by CCM
TCP/IP

GPFS
Work load managers

Hardware
Supervisory
System

Handling errors,
resiliency
Event routing
Booting the system

59

XE6 and XC30 Software Components

Generational Commonality
XE6 Software Stack

MPI

UPC & CAF PGAS

Languages

SHMEM

User-level

Distributed Memory API (DMAPP)

Kernel-level Generic Network

Interface (kGNI)

OS Bypass

CLE Linux Core

OS Bypass

Kernel

ioctl or
syscall

Generic Network Interface (uGNI)

Programming
Models

Generic
APIs

PGAS/SHMEM
Applications

Aries-aware optimized PE components

MPI

User-level

Distributed Memory API (DMAPP)

Generic Network Interface (uGNI)

Kernel

Cray Linux
& Network
Driver

Gemini - Hardware Abstraction Layer (HAL)

UPC & CAF PGAS

Languages

SHMEM

CLE Linux Core

Kernel-level Generic Network

Interface (kGNI)

OS Bypass

Gemini-aware optimized PE components

MPI
Applications

OS Bypass

User
Applications

PGAS/SHMEM
Applications

ioctl or
syscall

MPI
Applications

XC30

Aries - Hardware Abstraction Layer (HAL)

Network
Gemini Network

Aries Network

Common Base
60

The Cray System

User Applications, Programs, & Scripts

Programmers
Cray Programming
Environment

Administrators

Programming Models
Library Based

Compilers

CUDA

MPI

Directive Based
SHMEM

OpenACC

Cray
System
Mgmt

OpenMP

UPC

CAF

esFS
esLogin
esMS

Linux

Chapel

External
Server
Mgmt

System
Config,
Boot
Manager,
&
Health
Services

Language Based

Linux

Debugging Tools
Performance Tools
Modules

CUDA

Development tools

Scientific

MPI

Scientific

MPI

Other

X86-64 Enhanced Libraries

Enhanced Accelerator Libraries

Workload Manager
uGNI

Linux Environment

Cray APIs

DMAPP

ALPS Infrastructure

ALPS Launch

MOM

Linux

CLE

CLE

CLE

CLE

CLE

CLE

CLE

CLE

CLE

CLE

gNI

gNI

gNI

gNI

gNI

gNI

gNI

gNI

gNI

gNI

Cray High Speed Network

External
Login
Servers

Cray
X86-64
I/O
Nodes

Cray
GPU
Compute
Nodes

Cray
GPU
Compute
Nodes

Cray
GPU
Compute
Nodes

Cray
GPU
Compute
Nodes

Cray
x86-64
Compute
Nodes

Cray
x86-64
Compute
Nodes

Cray
x86-64
Compute
Nodes

Cray
x86-64
Compute
Nodes

Cray
X86-64
I/O
Nodes

System
Management
Workstation

External
Cluster
Manager

SMW

BMC

Globally Addressable Memory

Service

Accelerator

x86

x86

Boot RAID

Compute
x86

MDS

OST

Lustre
esFS

Legend

Site Network
Cray Content

GPU

x86

Memory

Memory

NFS

GPFS

OST

PCIe

(DVS)

Panasas

Lustre
Appliance

Customer
Filesystem

3rd Party Content

Community Content

61

Cray Software Ecosystem

GNU

Cray Linux
Environment
CrayPAT
Cray Apprentice2
Reveal

Cray Iterative
Refinement Toolkit
Cray PETSc, CASK

DVS

62

Cray XC30 User Environment

GNU

Cray Linux
Environment
CrayPAT
Cray Apprentice2
Reveal

Load & Go

Build & Go

Cray Iterative
Refinement Toolkit
Cray PETSc, CASK

DVS

Tune & Go

Code & Go

An Adaptive Linux OS optimized specifically

for HPC

ESM Extreme Scalability

CCM Cluster Compatibility

Mode

Mode

No compromise scalability
Low-Noise Kernel for scalability
Native Comm. & Optimized MPI
Application-specific performance
tuning and scaling

No compromise compatibility
Fully standard x86/Linux
Standardized Communication Layer
Out-of-the-box ISV Installation
ISV applications simply install and
run

CLE run mode is set by the user on a job-by-job basis to provide full flexibility
64

Cray Integrated Programming Environment

3. Static Analysis

Performance Analysis Overview

Executing Application

Runtime Information

High Level Profile/ Tracing

Performance Problem Analyzer

2. Compiler
Export/Import
Program Analyses

Performance
Feedback
Application
Source code

1. Applications

Database
containing
application
information

Performance Analysis
Results

Queries for
Application
Optimization

5. Program Analysis Tools

Source-to-Source optimizations

65

Cray Programming Environment Distribution

Focus on Performance and Productivity
Programming
Languages

Fortran

Programming
models
Distributed
Memory
(Cray MPT)
MPI
SHMEM

Cray Compiling
Environment
(CCE)

Tools

Optimized Scientific
Libraries

Environment setup

Compilers

LAPACK

I/O Libraries

NetCDF

Modules
ScaLAPACK
Debuggers

HDF5
BLAS (libgoto)

GNU

Modules

Shared Memory
OpenMP 3.0
OpenACC
C++
PGAS & Global
View
Python

UPC (CCE)
CAF (CCE)
Chapel

DDT
3rd Party
Compilers

lgdb
Debugging Support
Tools

Abnormal
Termination
Processing

Iterative
Refinement
Toolkit
Cray Adaptive
FFTs (CRAFFT)
FFTW

STAT

Cray PETSc
(with CASK)

Performance Analysis

Cray Trilinos
(with CASK)

CrayPat

Cray developed
Licensed ISV SW
3rd party packaging
Cray added value to 3rd party

Cray
Apprentice2
Scoping Analysis

Reveal
66

Tune & Go Example ROMS Ocean Model

Build & Go
Tune & Go

18.00
18.00

Cray XC30

Simulated Years / Day

16.00
16.00
14.00
14.00

FDR IB Cluster
FDR IB Cluster
Cray XC30

12.00
12.00

Cray XC30 + Rank

FDR IB Cluster
Optimization

10.00
10.00
8.00
8.00
6.00
6.00
4.00
4.00
2.00
2.00
0.00
0.00

00

512
512

1024
1024

1536
1536

Cores

SC 2013

2048
2048

Item
Item
Item
Processor
Processor
Processor
Compiler
Compiler
Compiler
Compiler
Compiler
Compiler
Options
Options
Options
Topology
Topology
Topology
MPI
MPI
MPI
Libraries
Libraries
Libraries
Math
Math
Math
Libraries
Libraries
Libraries
OS
OS
OS
Tuning

Copyright 2013 Cray Inc. - Confidential and Proprietary

FDR IB Cluster
FDR IB Cluster
FDR IB Cluster
Intel E5-2670
Intel E5-2670
Intel E5-2670
Intel 13.1
Intel 13.1
Intel 13.1
-heap-arrays -ip -heap-arrays -ip --heap-arrays
O3 -xW free -ip
O3 -xW free
O3 -xW free
Full Fat Tree
Full Fat Tree
Full Fat Tree
Intel 4.0
Intel 4.0
Intel 4.0
Intel MKL 10.3
Intel MKL 10.3
Intel MKL 10.3
RHEL 6.2
RHEL 6.2
RHEL 6.2

Cray XC30
Cray XC30
Same
Same
CCE 8.1
Same
Default flags
Same
Dragonfly
Dragonfly
Cray MPICH2
Cray MPICH2
6.0
6.0
Cray Libsci
Cray Libsci 12.1
12.1
CLE 5.1
CLE 5.1
MPI Rank
Reordering

67

Storage

68

Cray Storage Solution Depth

Cray

or

HPC
Cluster

IBM, HP, Dell,

many others

Cray Data Movers (esDM)

QDR or FDR InfiniBand

SAN

Cray Sonexion

Cray Storage Management

External OSS, MDS Servers
(esFS, esMS)

HSM

Tape

Cray Development
Environment
(esLogin)

SFA12K

E-Series

69

Sonexion System Components

Metadata
Management Unit

Scalable Storage
Units

April 2013

Cray Inc.

70

Sonexion: Only Three Components

MMU: Metadata Management Unit

Fully integrated metadata module

Lustre Metadata software

Metadata disk storage
Dual redundant management servers
Metadata storage target RAID

SSU: Scalable Storage Unit

Fully integrated storage module

Storage controller, Lustre server

Disk controller, RAID engine
High speed storage
Provides both capacity and performance

Fully prepared rack

Prewired for InfiniBand, Ethernet and power

Ready for instant expansion

71

The Cray Sonexion 1600

Enhanced Metadata Controller
FDR IB, GigE switches
Exclusively
designed for
Lustre
Fully modular
and scalable
Performance
scales with
capacity

Integrated controller & storage

Linear speed & capacity scaling

Up to 6 storage modules per rack

Each SSU delivers >5GB/sec

72

Updated Dashboard Overview

Node Status

FS Throughput

Inventory

Statistics

Cray Inc. Confidential: Storage and DM

73

BoM Plans

Discussion

5/7/2014

Cray Inc. Property

74

Questions?