High Performance Computing: Planing, usage, and Application

Yudi Rosandi
Workshop on High Performance Computing: from design to application

Outline
Planing
Hardware components Infrastructures Operating system Basic knowledge

Usage

Cluster Application
Simulation, research and development Data processing, forecasting

Meet some Beowulf clusters....

Meet some Beowulf clusters....

Meet some Beowulf clusters....

Meet some Beowulf clusters....

Planing

Cluster
collection of computers interconected one and another via network working together on a single program or interdependent different programs

Planing of a cluster
What is the purpose? How much budget do we have? What is the expected outcome?

High availibility cluster

Load Balancing
multi-user environtment

Compute cluster (HPC)

single parallel program run in many procs

Budget...LOW
build from scratch
chalenging precious experience troublesome but interresting freedom most problems comes from networking cable mesh high maintenance cost high power consumtion (depends on hardware selection)

Pitfalls

Budget...HIGH
high-end cluster
blade server ready built racks production ready integrated high-speed lan (infiny-band) good cooling system low power consumption

What we get

HPC Outcome
Research
General sciences
fast publication many ready to use programs ability to solve complex problems insight to unknown aspects in science parallel programming computer administration networking user management benchmarking and optimization

Computer science

Low research cost!

HPC Outcome
Production
fast computation fast decission making increasing services high availability

Hardware components

What kind of hardware do we need?

Anything...

Hardware requirements
Computing nodes
fast computer fast network interface sufficient memory local scratch storage dedicated, reliable large storage sufficient memory (+8Gb) high-speed network connection (channel bonding, etc)

Storage server

Hardware requirements
Head server(s): reliable, not necessarily fast
dhcp nis login queuing note:
these servers may be separated individual servers, combined servers in one machine, or virtualized (e.g. Xen) Login server may need special consideration!

Common cluster for high performance computing

Needed infrastructure
Room temperature control Stabil electricity Power consumption networking backbone

Operating system
Things to consider
stability reliability managability accessibility security

Operating system: LINUX

Why????

Operating system: LINUX

stability (OK) reliability (OK) managability (OK) accessibility (OK) security (OK)

Cluster Usage

Basic knowledge to use a cluster computing facility

know how to:
make remote connection use a shell (csh,sh,bash,etc) file-system: locate directories, file permission submit/manage programs write a parallel program hacking running malicious program robbing memory spamming file-system

don't do:

Know your limits! Respect other users!

Remote login
Accesible from internet
static address dedicated

Secured internet connection

key authentication firewall

Remote login

Remote login
Files for ssh connection (under ~/.ssh/)
id_rsa (@client)
identity file, must be kept 'only' in the connecting computer SECRET! don't transfer via email, etc!

id_rsa.pub
public key, to be distributed to all servers

known_hosts (@client)
list of host keys of known servers

authorized_keys, authorized_keys2 (@server)

list of public keys of clients/user those allowed to connect.

config (@client)

local user specific configuration

command to create id keys: ssh-keygen

Shell
elements
variables program controls redirection pipes

customizing shell
make your self convenient personal commands, path, etc creating job-script building embarrassingly parallel programs automating processes

basic scripting knowledge

Shell: bash
Standard in Linux Complete Flexible Advanced Demo/example....
declaring variable simple program controls redirection and pipes

Application

Animation movies! (@ pixar)

Animation movies! (@ pixar)

Material science (@ TU-KL)

Material science (@ TU-KL)

a glance at computing center

Application
incomplete examples... Research:
modelling nano-sciences complex-system image rendering

Industry

Many others...

data processing

Thank you