INTRODUCTION TO PARALLEL

COMPUTING
B S RAMANJANEYULU

System Software Development Group,

CDAC, Bangalore.

1
 Presentation Outline

• Need for Parallel Computing

• Requirements of Parallel Computing
• Parallel Computing Terminology
• Parallel computer architectures
• Designing parallel algorithms
• Architectural taxonomy (SISD, SIMD, MISD and
MIMD)
• Symmetric multiprocessing (SMP)
• Clusters
• Parallel programming models
2
 How to Run Applications Faster?

 There are 3 ways to improve performance:

• Work Harder
• Work Smarter
• Get Help (multiple workers)
 Computer Analogy
• Use faster hardware: e.g. reduce the time per instruction
• Optimized algorithms and techniques.
• Multiple computers to solve problem.

3
 Sequential vs. Parallel

Sequential

Parallel

4
 Sequential vs. Parallel (Contd…)

 Traditional sequential programs execute one instruction at a time

using one processor
 Parallelism implies executing tasks simultaneously (on multiple
processors) to complete the job faster
 Parallelism can be done by:
− Breaking up the task into smaller tasks
− Assigning the smaller tasks to multiple workers (processors) to
work on simultaneously
− Coordinating the workers (processors)
− Parallel problem solving is natural. Examples: Building
construction; Automobile manufacturing

5
 The Need For Faster Machines

 Grand Challenge Problems:

 Climate Modeling
 Computational Fluid dynamics
 Combustion Systems
 Human Genome
 Structural Mechanics
 Molecular Modeling
 Astrophysical Calculations
 Seismic Data Processing

6
 Data Parallelism
Example:

if CPU=“1" then
start=1
end=50
else if CPU=“2" then
start=51
end=100
end if

do
i = start , end
Task on d(i)
end do
7
 Task Parallelism
Task parallelism

 Multiple tasks executing concurrently is called task parallelism.

 All the CPUs execute separate code blocks simultaneously.

Example:

if CPU=“1" then
do “Task 1”
else if CPU=“2" then
do “Task 2”
end if

8
 Definition

Definition :

 In computer architecture point of view, a parallel

computer is a “Collection of processing elements that
communicate and co-operate to solve large problems
fast”.
 When this architecture is combined with a parallel
algorithm, we get the ‘parallel computing system’.

9
 Sequential vs. Parallel Computing

SEQUENTIAL COMPUTING

 Fetch/Store

 Compute

PARALLEL COMPUTING

 Fetch/Store

 Compute
 communicate

10
 Execution Time

• Sequential system
– Execution time as a function of size of input
• Parallel system
– Execution time as a function of input size,
and number of processors used

11
 Terminology of Parallel Computing

Speedup : Speedup ‘Tp’ is defined as the ratio of the serial

runtime of the best sequential algorithm for solving a problem to
the time taken by the parallel algorithm to solve the same
problem on ‘p’ processors.
Tp=T(seq) / T(parallel)
The ‘p’ processors used by the parallel algorithm are assumed to
be identical to the one used by the sequential algorithm

Efficiency: Ratio of speedup to the number of processors.

Efficiency = Tp / P

12
 Terminology of Parallel Computing
Throughput (in FLOPS): (Contd…)

It is obtained by taking the clock rate of the given system and

dividing it by the number of clock cycles a floating point
instruction requires.

Cost : Cost of solving a problem on a parallel system is the

product of parallel runtime and the number of processors used ,
i.e., E = p.Sp

13
 Requirements for Parallel Computing

 Multiple processors
(The workers)

 Network
(Link between workers)

 OS support

14
 Requirements for Parallel Computing (Contd…)

Parallel Programming Paradigms

 Message Passing (MPI , PVM )
 Data Parallel (Fortran 90/High Performance Fortran )
 Multi-Threading
 Hybrid
 Others (OpenMP, shmem)
 Decomposition of the problem into pieces that multiple
workers can perform.

15
 Issues in Parallel Computing

• Parallel computer architectures

• Efficient parallel algorithms
• Parallel programming models
• Parallel computer languages
• Methods for evaluating parallel algorithms
• Parallel programming tools

16
 Designing Parallel Algorithms

 Detect and exploit any inherent parallelism in an existing

sequential Algorithm

 Invent a new parallel algorithm

 Adopt from another parallel algorithm that solves a similar

problem

17
 Decomposition Techniques

Decomposition Techniques

The process of splitting the computations in a problem into a set of

concurrent tasks is referred to as decomposition.

 Decomposing a problem effectively is of paramount importance

in parallel computing.

 Without a good decomposition, we may not be able to achieve a

high degree of concurrency.

 Decomposing a problem must ensure good load balance.

18
 Decomposition Techniques (Contd…)

What is meant by good decomposition?

 It should lead to high degree of concurrency (fine-granularity).

 The interaction among tasks should be as little as possible

(coarse-granularity).

•The ratio between computation and communication is known as granularity.

19
 Success depends on the combination of

 Architecture, Compiler, Choice of Right Algorithm

 Portability, Maintainability, and Efficient implementation

20
 Architectural Taxonomy

Flynn's taxonomy uses the relationship of program instructions

to program data. The four categories are:
 SISD – Single Instruction, Single Data Stream
 SIMD – Single Instruction, Multiple Data Stream
 MISD - Multiple Instruction, Single Data Stream
(no practical examples)
 MIMD - Multiple Instruction, Multiple Data Stream

21
 SISD Model features
 Not a parallel computer
 Conventional serial, scalar von Neumann computer
 A single instruction is issued in each clock cycle
 Each instruction operates on a single (scalar) data element
 Performance measured in MIPS
 Examples: most PCs and single CPU workstations

22
 SIMD Model features

Also von Neumann architectures but

more powerful instructions
Each instruction may operate on more
than one data element
Usually intermediate host executes
program logic and broadcasts
instructions to other processors
Examples: Array Processors and Vector
Processors (used in the supercomputers of
1970’s and 80’s

23
 MIMD Model features

 Parallelism achieved by connecting multiple processors together

 Each processor executes its own instruction stream independent of other
processors on unique data stream
 Advantages
 Processors can execute multiple job streams simultaneously
 Each processor can perform any operation regardless of what
other processors are doing
Disadvantages
Load balancing overhead - synchronization needed to coordinate
processors at end of parallel structure in a single application
Can be difficult to program 24
MIMD Block Diagram

25
MIMD Classification

26
Parallel Computer Architecture Memory Models

Shared Memory Distributed Memory

27

Hybrid Memory
Symmetric Multiprocessors (SMP)

28
 Symmetric Multiprocessors (SMP)
(Contd…)

•Uses commodity microprocessors with on-chip and off-chip

cache.
•Processors are connected to a shared memory through a high-
speed bus
•Single address space.
•Easy application development.
•Difficult to scale.
•Difficult to repair/ replace the faulty node (when compared to
clusters)

29
SMP, MPP and clusters

30
 Competing Architectures

• Massively Parallel Processors (MPP)-proprietary systems built for

specific purposes
– high cost and a low performance/price ratio.
• Symmetric Multiprocessors (SMP)
– suffers from scalability
• Distributed Systems
– difficult to extract high performance.
• Clusters
– High Performance Computing--- With Commodity Processors
– High Availability Computing --- for Critical Applications

31
 What is a Cluster?

 A cluster is a type of parallel or distributed processing system,

which consists of a collection of interconnected stand-
alone/complete computers cooperatively working together as a
single, integrated computing resource.

 A typical cluster consists of:

• Faster, closer connection Network than a typical LAN
• Low latency communication protocols
• Looser connection than SMP

32
 Motivation for using Clusters

 The communications bandwidth between

workstations is increasing as new networking
technologies and protocols are implemented in
LANs and WANs.

 Workstation clusters are easier to integrate into

existing networks than special parallel computers.

33
Cluster Computer Architecture

34
 Components of Cluster Computers
• Multiple High Performance Computers
– PCs
– Workstations
– SMPs
• State-of-the-art Operating Systems
– Layered
– Micro-kernel based
• High Performance Networks/Switches
– Gigabit Ethernet
– PARAMNet
– Myrinet
• Network Interface Cards (NICs)
• Fast Communication Protocols and Services
– Active Messages (AM)
– Virtual Interface Architecture (VIA)
35
 Components of Cluster Computers (Contd…)

• Parallel Programming Environments and Tools

– Compilers
– PVM [Parallel Virtual Machine]
– MPI [Message Passing Interface]
• Applications

– Sequential
– Parallel or Distributed

36
 Parallel programming models -- MPI, PVM and OpenMP

•MPI – Messaging Passing Interface

•PVM – Parallel Virtual Machine
•Both MPI and PVM are based on message passing mechanism.
•Both MPI and PVM can be used with shared-memory and
distributed memory architectures.
•MPI
- MPI is mainly for data-parallel problems.
- Collective and asynchronous operations are more powerful
in MPI.
•OpenMP – Open Multiprocessing
- OpenMP is thread-based multiprocessing.
37

- OpenMP – more suitable to SMP systems.

 Features of CDAC’s PARAM Supercomputers

 Distributed memory at system level and Shared memory at Node level.

 Nodes connected by low latency high throughput System Area

Networks PARAMNet and Fast/Gigabit Ethernet.

 Standard Message Passing interface (MPI) i.e. SUN MPI, IBM MPI,
Public Domain MPI and C-DAC’s own MPI (CMPI).

 C-DAC’s High Performance Computing and Communication Software

(HPCC) for Parallel Program Development and run time support.

38
 References

• http://www.llnl.gov/computing/tutorials/parallel_comp/
• Tutorials located in the Maui High Performance
Computing Center's "SP Parallel Programming
Workshop".
• Linux Parallel procesing HOW TO from
http://www.tldp.org/HOWTO/Parallel-Processing-
HOWTO.html

39
Thank you.

40