GCC Lab Manual

DR.
SIVANTHI ADITANAR COLLEGE OF ENGINEERING
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING
CS6712 GRID AND CLOUD COMPUTING LABORATORY
LAB MANUAL
CLASS : IV CSE A & B
FACULTY IN CHARGE
Mr.D.KESAVARAJA
Ms.A.ANNIE JESUS SUGANTHI RANI
HOD/CSE

LTPC
0 03 2
OBJECTIVES:
The student should be made to:
Be exposed to tool kits for grid and cloud environment.
Be familiar with developing web services/Applications in grid framework
Learn to run virtual machines of different configuration.
Learn to use Hadoop
LIST OF EXPERIMENTS:
GRID COMPUTING LAB
Use Globus Toolkit or equivalent and do the following:
1. Develop a new Web Service for Calculator.
2. Develop new OGSA-compliant Web Service.
3. Using Apache Axis develop a Grid Service.
4. Develop applications using Java or C/C++ Grid APIs
5. Develop secured applications using basic security mechanisms available in Globus
Toolkit.
6. Develop a Grid portal, where user can submit a job and get the result. Implement it with
and without GRAM concept.
CLOUD COMPUTING LAB

Use Eucalyptus or Open Nebula or equivalent to set up the cloud and demonstrate.
1. Find procedure to run the virtual machine of different configuration. Check how many
virtual machines can be utilized at particular time.
2. Find procedure to attach virtual block to the virtual machine and check whether it holds
the data even after the release of the virtual machine.
3. Install a C compiler in the virtual machine and execute a sample program.
4. Show the virtual machine migration based on the certain condition from one node to the
other.
5. Find procedure to install storage controller and interact with it.
6. Find procedure to set up the one node Hadoop cluster.
7. Mount the one node Hadoop cluster using FUSE.
8. Write a program to use the API's of Hadoop to interact with it.
9. Write a wordcount program to demonstrate the use of Map and Reduce tasks
TOTAL: 45 PERIODS
OUTCOMES:
At the end of the course, the student should be able to
Use the grid and cloud tool kits.
Design and implement applications on the Grid.
Design and Implement applications on the Cloud.
LIST OF EQUIPMENT FOR A BATCH OF 30 STUDENTS:
SOFTWARE:
Globus Toolkit or equivalent
Eucalyptus or Open Nebula or equivalent
HARDWARE
Standalone desktops 30 Nos
Dept of CSE, DR.SACOE Page 1

TABLE OF CONTENTS
LIST OF EXPERIMENTS
Ex. No EXPERIMENT PAGE No.

CLOUD COMPUTING LAB
3
1 FIND PROCEDURE TO SET UP THE ONE NODE HADOOP CLUSTER
9
2 TO MOUNT THE ONE NODE HADOOP CLUSTER USING FUSE
TO WRITE A WORD COUNT PROGRAM TO DEMONSTRATE THE 12

3
USE OF MAP AND REDUCE TASKS
WRITE A PROGRAM TO USE THE API'S OF HADOOP TO INTERACT 16

4
WITH IT
19
5 INSTALLATION OF VIRTUAL MACHINE
22
6 VIRTUAL BLOCK ATTACHMENT
24
7 INSTALLATION OF C COMPILER
26
8 VIRTUAL MACHINE MIGRATION
PROCEDURE TO INSTALL STORAGE CONTROLLER AND 30

9
INTERACT WITH IT
GRID COMPUTING LAB 32
34
1 DEVELOP NEW WEB SERVICE FOR CALCULATOR
2 DEVELOP NEW OGSA-COMPLIANT WEB SERVICE 38
42
3 DEVELOPING NEW GRID SERVICE
44
4 DEVELOP APPLICATIONS USING JAVA - GRID APIS
DEVELOP SECURED APPLICATIONS USING BASIC SECURITY IN 45

5
GLOBUS
49
6 DEVELOP A GRID PORTAL
FACULTY IN CHARGE HOD/CSE

CLOUD COMPUTING LAB
Ex.No:1 FIND PROCEDURE TO SET UP THE ONE NODE HADOOP CLUSTER
AIM:
To find procedure to set up the one node Hadoop cluster
INTRODUCTION:
Apache Hadoop is an open-source software framework for distributed storage and distributed
processing of very large data sets on computer clusters built from commodity hardware.
All the modules in Hadoop are designed with a fundamental assumption that hardware failures
are common and should be automatically handled by the framework.
The core of Apache Hadoop consists of a storage part, known as Hadoop Distributed File
System (HDFS), and a processing part called MapReduce.
Hadoop splits files into large blocks and distributes them across nodes in a cluster.
The base Apache Hadoop framework is composed of the following modules:

Hadoop Common it contains the Java libraries and utilities needed by other Hadoop modules.
HDFS Hadoop Distributed File System A Java based scalable file system distributed across
multiple nodes.
MapReduce YARN framework for parallel big data processing.
Hadoop YARN: A framework for cluster resource management.
STEPS:
Step 1: Install Java on CentOS 7
Step 2: Install Hadoop Framework in CentOS 7
Step 3: Configure Hadoop in CentOS 7
Step 4: Format Hadoop Namenode
Step 5: Start and Test Hadoop Cluster
Step 6: Browse Hadoop Services
PROCEDURE:
1. Login into CentOS in student login. (Password:student)

2. Right click on the desktop and open Konsole
3. To login into root user, Type
$ su
Give the password as root
4. Open File Manager root Desktop (Check for jdk and hadoop files)
Step 1: Install Java on CentOS 7
5. To install the java package give the command:

# rpm -Uvh jdk-8u91-linux-x64.rpm
Step 2: Install Hadoop Framework in CentOS 7
6. Create a new user account on your system without root powers (well use it for Hadoop
installation path and working environment)
The new account home directory will reside in /opt/hadoop063 directory.
# useradd -d /opt/hadoop063 hadoop063
# passwd hadoop063
7. To extract the hadoop archive file:

# tar xfz hadoop-2.7.2.tar.gz
Then copy the directory content to hadoop account home path.

# cp -rf hadoop-2.7.2/* /opt/hadoop063/
To change the permission for the copied files

# chown -R hadoop063:hadoop063 /opt/hadoop063/
(chown -R username:password /opt/foldername/)
8. Login with hadoop user

# su hadoop063
Configure Hadoop and Java Environment Variables on your system by editing

the .bash_profile file.
$ vi .bash_profile
Append the following lines at the end of the .bash_profile file:
## JAVA env variables

export JAVA_HOME=/usr/java/default
export PATH=$PATH:$JAVA_HOME/bin
export
CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
## HADOOP env variables

export HADOOP_HOME=/opt/hadoop063
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
9. To initialize the environment variables and to check their status, issue the below commands:
$ source .bash_profile
$ echo $HADOOP_HOME
$ echo $JAVA_HOME
10. Finally, configure ssh key based authentication for hadoop account by running the below
commands
Also, leave the passphrase filed blank in order to automatically login via ssh.
$ ssh-keygen -t rsa
(Just give enter)
Step 3: Configure Hadoop in CentOS 7
11. To setup Hadoop cluster on a single node in a pseudo distributed mode edit its configuration
files.

(The location of hadoop configuration files is $HADOOP_HOME/etc/hadoop/, which is

represented as hadoop account home directory (/opt/hadoop063/) path.
Logged in with user hadoop063 and start editing the following configuration file)
Edit core-site.xml file. This file contains information about the port number used by Hadoop
instance, file system allocated memory, data store memory limit and the size of Read/Write
buffers.
$ vi etc/hadoop/core-site.xml
Add the following properties between <configuration> ... </configuration> tags.
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000/</value>
</property>
</configuration>
12. Edit hdfs-site.xml file. This file contains information about the value of replication
data, namenode path and datanode path for local file systems.
$ vi etc/hadoop/hdfs-site.xml
Here add the following properties between <configuration> ... </configuration> tags.
/opt/volume/ directory is used to store our hadoop file system.
Replace the dfs.data.dir and dfs.name.dir values accordingly.
<configuration>
<property>
<name>dfs.data.dir</name>
<value>file:///opt/volume/datanode</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///opt/volume/namenode</value>
</property>
</configuration>
13. Weve specified /opt/volume/ as our hadoop file system storage, therefore we need to create
those two directories (datanode and namenode) from root account and grant all permissions to
hadoop account by executing the below commands.
$ su
Password: root
# mkdir -p /opt/volume/namenode
# mkdir -p /opt/volume/datanode
# chown -R hadoop063:hadoop063 /opt/volume/
# ls -al /opt/ //Verify permissions
# exit //To exit from root account

14. Create mapred-site.xml file to specify that we are using yarn MapReduce framework.
$ vi etc/hadoop/mapred-site.xml
Add the following lines to mapred-site.xml file:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
15. Edit yarn-site.xml file
$ vi etc/hadoop/yarn-site.xml
Add the following lines to yarn-site.xml file between <configuration> ... </configuration>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
16. Set Java home variable for Hadoop environment by editing the below line from hadoop-env.sh
file.
$ vi etc/hadoop/hadoop-env.sh
Change the export JAVA_HOME line as below (You are pointing to your Java system path).
export JAVA_HOME=/usr/java/default/
17. Open slaves file
$ vi etc/hadoop/slaves
Check for the following line

localhost
Step 4: Format Hadoop Namenode
18. Once hadoop single node cluster has been setup its time to initialize HDFS file system by
formatting the /opt/volume/namenode storage directory with the following command:
$ hdfs namenode -format

Step 5: Start and Test Hadoop Cluster
19. The Hadoop commands are located in $HADOOP_HOME/sbin directory. In order to start
Hadoop services run the below commands on your konsole:
$ start-dfs.sh
$ start-yarn.sh
Check the services status with the following command.
$ jps
Alternatively, you can view a list of all open sockets for Apache Hadoop on your system using
the ss command.
$ ss -tul
$ ss -tuln // Numerical output
20. To test hadoop file system cluster create a random directory in the HDFS file system and copy a
file from local file system to HDFS storage (insert data to HDFS).
$ hdfs dfs -mkdir /my_storage

$ hdfs dfs -put LICENSE.txt /my_storage
To view file content inside HDFS file system:
$ hdfs dfs -cat /my_storage/LICENSE.txt
To list a directory inside HDFS files system:
$ hdfs dfs -ls /my_storage/
To retrieve data from HDFS to our local file system:
$ hdfs dfs -get /my_storage/ ./
//Get the full list of HDFS command options by issuing:
$ hdfs dfs -help
Step 6: Browse Hadoop Services

21. In order to access Hadoop services, open Firefox and type the following
http://127.0.0.1:50070
In Hadoop page Select Utilities Browse your files

Click my_storage, LICENSE.txt will be shown.

OUTPUT:
RESULT:
Thus the Apache Hadoop Cluster was configured and results were checked and verified.

Ex: No: 2 TO MOUNT THE ONE NODE HADOOP CLUSTER USING FUSE
AIM:
To mount the one node Hadoop cluster using FUSE on CentOS
FUSE:
FUSE permits you to write down a traditional user land application as a bridge for a conventional
file system interface.
The hadoop-hdfs-fuse package permits you to use your HDFS cluster as if it were a conventional
file system on Linux.
Its assumed that you simply have a operating HDFS cluster and grasp the hostname and port that
your NameNode exposes.
The Hadoop fuse installation and configuration with Mounting HDFS, HDFS mount using fuse is
done by following the below steps.
Step 1. Required Dependencies
Step 2. Download and Install FUSE
Step 3. Install RPM Packages
Step 4. Modify HDFS FUSE
Step 5. Check HADOOP Services
Step 6. Create a Directory to Mount HADOOP
Step 7. Modify HDFS-MOUNT Script
Step 8. Create softlinks of LIBHDFS.SO
Step 9. Check Memory Details
PROCEDURE:
Step 1: Required Dependencies

Hadoop single / multinode Cluster (started mode)
jdk (preinstalled)
Step 2: Download and install fuse

Login as hadoop user to a node in hadoop cluster (master / datanode)
Download hdfs-fuse from following location
#wget https://hdfs-fuse.googlecode.com/files/hdfs-fuse-0.2.linux2.6-gcc4.1-x86.tar.gz
Check whether the downloaded file is available in Home

Extract hdfs-fuse-0.2.linux2.6-gcc4.1-x86.tar.gz
#tar -zxvf hdfs-fuse-0.2.linux2.6-gcc4.1-x86.tar.gz
Step 3: Install rpm packages

Switch to root user to install following rpm packages
fuse-libs-2.8.3-4.el6.i686
fuse-2.8.3-4.el6.i686
fuse-devel-2.8.3-4.el6.i686
#su root
#yum install fuse*
#chmod +x /usr/bin/fusermount
Step 4: Modify hdfs fuse

After installation of rpm packages, switch back to hadoop user
# su hadoop063

Modify hdfs fuse configuration / environmental variables

$cd hdfs-fuse/conf/
Add following lines in hdfs-fuse.conf

$vi hdfs-fuse.conf
export JAVA_HOME=/usr/java/default # JAVA HOME path

export HADOOP_HOME=/opt/hadoop063 # hadoop installation home path
export FUSE_HOME=/opt/hadoop063 #fuse installation path
export HDFS_FUSE_HOME=/opt/hadoop063/hdfs-fuse # fuse home path
export HDFS_FUSE_CONF=/opt/hadoop063/hdfs-fuse/conf # fuse configuration path
LogDir /tmp LogLevel LOG_DEBUG Hostname 127.0.0.1 # hadoop master node
IP Port 9099 # hadoop port number
Step 5: Check hadoop services

Verify hadoop instance is running
$cd ..
$ jps
2643 TaskTracker
4704 Jps
2206 NameNode
2516 JobTracker
2432 SecondaryNameNode
2316 DataNode
Step 6: Create a directory to mount hadoop

Create a folder to mount hadoop file system to it
#mkdir /opt/hadoop063/hdfsmount
# cd
#pwd
Step 7: Modify hdfs-mount script

Switch to hdfc fuse binary folder in order to run mount script.
#cd hdfs-fuse/bin/
Modify hdfs-mount script to show jvm path location and other environmental settings
$ vi hdfs-mount
JAVA_JVM_DIR=/usr/java/default/jre/lib/amd64/server
export JAVA_HOME=/usr/java/default
export HADOOP_HOME=/opt/hadoop063
export FUSE_HOME=/opt/hadoop063
export HDFS_FUSE_HOME=/opt/hadoop063/hdfs-fuse
export HDFS_FUSE_CONF=/opt/hadoop063/hdfs-fuse/conf
Step 8: Create softlinks of libhdfs.so

Create softlinks of libhdfs.so
# cd /opt/hadoop063/hdfs-fuse/lib/
# ln -s /opt/hadoop063/hadoop-1.2.1/c++/Linux-i386-32/lib/libhdfs
Mount HDFS file system to /home/hadoop/hdfsmount

#./hdfs-mount /opt/hadoop063/hdfsmount
or

$./hdfs-mount -d /opt/hadoop063/hdfsmount (-d option to enable debug)
Step 9: Check memory details
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_hadoop-lv_root 50G 1.4G 46G 3% /
tmpfs 504M 0 504M 0% /dev/shm
/dev/sda1 485M 30M 430M 7% /boot
/dev/mapper/vg_hadoop-lv_home 29G 1.2G 27G 5% /home
hdfs-fuse 768M 64M 704M 9% /home/hadoop/hdfsmount
$ ls /opt/hadoop063/hdfsmount/
tmp user
Use below fusermount command to unmount hadoop file system
$fusermount -u /opt/hadoop063/hdfsmount
RESULT:
Thus, the fuse mount is ready to use as local file system.

Ex: No: 03 TO WRITE A WORD COUNT PROGRAM TO DEMONSTRATE

THE USE OF MAP AND REDUCE TASKS
AIM:
To write a word count program to demonstrate the use of Map and Reduce tasks.
WORD COUNT:
The WordCount program reads text files and counts how often words occur.
The input is text files and the output is text files, each line of which contains a word and the count
of how often it occurred, separated by a tab.
Each mapper takes a line as input and breaks it into words. It then emits a key/value pair of the
word and 1. Each reducer sums the counts for each word and emits a single key/value with the
word and sum.
As an optimization, the reducer is also used as a combiner on the map outputs. This reduces the
amount of data sent across the network by combining each word into a single record.
STEPS:
Make a directory for mapreduce in your hadoop user and enter into mapreduce directory.
Open an editor (vi or nano), and write the code for WordCountMapper.java,
WordCountReducer.java, WordCount.java
WordCountMapper.java Contains the map function implementation.
WordCountReducer.java Contains the reduce function implementation.
WordCount.java Contains the code coordinating the execution of the map
and reduce functions.
To compile the WordCount program, execute the following commands in the mapreduce
folder
$ javac -cp hadoop-core-1.0.4.jar *.java
$ jar cvf WordCount.jar *.class
The first command compiles the program using the classes developed by Hadoop
(i.e., hadoop-core-1.0.4.jar)
The second command creates a jar file called WordCount.jar
To run the WordCount program in Hadoop
$ start-all.sh
The command starts the Hadoop services.
Then prepare the input for the WordCount program:

$ hdfs dfs -mkdir /input
$ vi MyFile.txt
Sometimes you win
Sometimes you learn
$ hdfs dfs -put MyFile.txt /input
The former command creates a directory called input in the Hadoop Distributed File System
The second command will copy MyFile.txt into the input folder in HDFS.
Finally execute the following commands:
$ bin/hadoop jar WordCount.jar WordCount /input /output
$ hdfs dfs -get /output/*
The first command run the WordCount program in Hadoop.
Note that the command specifies the names of:
the class where the main method resides (cf. the WordCount.java file).

the HDFS folder where the input files resides.

the HDFS folder that will contain the output files.
The second command display the output folder from HDFS to your machine.
The result of the WordCount program will be found in a file (probably) called part-00000 also.
PROGRAM:
Mapper Class - WordCountMapper.java
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Mapper;
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>

{
//hadoop supported data types
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException
{
//taking one line at a time and tokenizing the same
String line = value.toString();
StringTokenizer tokenizer = newStringTokenizer(line);
//iterating through all the words available in that line and forming the key value pair
while (tokenizer.hasMoreTokens())
{
word.set(tokenizer.nextToken());
//sending to output collector which inturn passes the same to reducer
context.write(word, one);
}
}
}
Reducer Class - WordCountReducer.java
import java.util.Iterator;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Reducer;
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable>

{
//Reduce method for just outputting the key from mapper as the value from mapper is just an empty
string
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
InterruptedException
{
int sum = 0;
/*iterates through all the values available with a key and add them together and give the

final result as the key and sum of its values*/

for (IntWritable value : values)
{
sum += value.get();
}
context.write(key, new IntWritable(sum));
}
}
Driver Class - WordCount.java
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class WordCount extends Configured implements Tool

{
public int run(String[] args) throws Exception
{
//getting configuration object and setting job name
Configuration conf = getConf();
Job job = new Job(conf, "Word Count hadoop-0.20");
//setting the class names

job.setJarByClass(WordCount.class);
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
//setting the output data type classes

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
//to accept the hdfs input and output dir at run time
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(String[] args) throws Exception {

int res = ToolRunner.run(new Configuration(), newWordCount(), args);
System.exit(res);
}
}

WORKING LOGIC:
OUTPUT:
$ hdfs dfs -get /output/*
Sometimes 2
you 2
win 1
learn 1
RESULT:
Thus, a word count program to demonstrate the use of Map and Reduce tasks was successfully
executed and verified.

Ex: No: 4 WRITE A PROGRAM TO USE THE API'S OF HADOOP

TO INTERACT WITH IT
AIM:
To write a program to use the API's of Hadoop to interact with it.
DESCRIPTION:
Hadoops org.apache.hadoop.fs.FileSystem is generic class to access and manage HDFS
files/directories located in distributed environment.
Files content are stored inside datanode with multiple equal large sizes of blocks.
Namenode keep the information of those blocks and Meta information.
FileSystem read and stream by accessing blocks in sequence order.
FileSystem first get blocks information from NameNode then open, read and close one by one.
It opens first block once it is completed, it closes the block and opens the next block.
HDFS replicate the block to give higher reliability and scalability.
If one of the datanode is temporarily down (fails) then it moves to other cluster datanode.
FileSystem uses FSDataOutputStream and FSDataInputStream to write and read the contents in
stream.
Configuration class passes the Hadoop configuration information to FileSystem.
It loads the core-site and core-default.xml through class loader and keeps Hadoop configuration
information such as fs.defaultFS, fs.default.name etc.
FileSystem uses Java IO FileSystem interface mainly DataInputStream and DataOutputStream for
IO operation.
Delete method on FileSystem remove the file/directory permanently
STEPS:
Open a Konsole, Login as 'root' user
Make a directory named 'HDFSJava' (Any name)
# mkdir -p /opt/hadoop063/HDFSJava
Copy the hadoop-0.20.2-core.jar and commons-logging-1.1.1.jar and put it in to the created
directory
# cp /home/student/Desktop/hadoop-0.20.2-core.jar/* /opt/hadoop063/HDFSJava
# cp /home/student/Desktop/commons-logging-1.1.1.jar/* /opt/hadoop063/HDFSJava
Change to HDFSJava directory
# cd /opt/hadoop063/HDFSJava
Extract the two jar files
# jar -xvf hadoop-0.20.2-core.jar
# jar -xvf commons-logging-1.1.1.jar
Open an editor (vi or nano), type the java coding.
# vi HDFSWordCounter.java
Type the following commands
# source .bash_profile
# start-all.sh
# jps
Check whether datanode and namenode are running
Compile the java program
# javac -cp hadoop-0.20.2-core.jar HDFSWordCounter.java
Run the java program
# java HDFSWordCounter

PROGRAM:
//hadoop imports
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
public class HDFSWordCounter

{
//change this to string arg in main
public static final String inputfile = "hdfsinput.txt";
public static final String inputmsg = "Count the amount of words in this sentence!\n";
public static void main(String [] args) throws IOException

{
// Create a default hadoop configuration
Configuration config = new Configuration();
config.addResource(new Path(etc/hadoop/core-site.xml)); // Parse created config to the HDFS
FileSystem fs = FileSystem.get(config);
// Specifies a new file in HDFS.

Path filenamePath = new Path(inputfile);
System.out.println(filenamePath.toString());
try
{
// if the file already exists delete it.
if (fs.exists(filenamePath))
{
//remove the file
fs.delete(filenamePath, true);
}
//FSOutputStream to write the inputmsg into the HDFS file
FSDataOutputStream fin = fs.create(filenamePath);
fin.writeUTF(inputmsg);
fin.close();
//FSInputStream to read out of the filenamePath file

FSDataInputStream fout = fs.open(filenamePath);
String msgIn = fout.readUTF();
//Print to screen
System.out.println(msgIn);
fout.close();
}
catch (IOException ioe)
{
System.err.println("IOException during operation " + ioe.toString());
System.exit(1);
}
}
}

OUTPUT:
# java HDFSWordCounter
hdfsinput.txt
Count the amount of words in this sentence!
RESULT:
Thus The hdfsinput file has been written to the hdfs and output is displayed.

EX. NO. : 5 INSTALLATION OF VIRTUAL MACHINE
AIM:
To find procedure to run the virtual machine of different configuration. Check how many virtual
machines can be utilized at particular time.
KVM:
In computing, virtualization refers to the act of creating a virtual (rather than actual) version of
something, including virtual computer hardware platforms, operating systems, storage devices, and
computer network resources. Kernel-based Virtual Machine (KVM) is a virtualization infrastructure for
the Linux kernel that turns it into a hypervisor.
STEPS FOR KVM INSTALLATION:
1. To run KVM, you need a processor that supports hardware virtualization. So check that your
CPU supports hardware virtualization
egrep -c '(vmx|svm)' /proc/cpuinfo
If 0 - CPU doesn't support hardware virtualization 1 -

CPU support hardware virtualization
2. To see if your processor is 64-bit, egrep -

c ' lm ' /proc/cpuinfo
If 0 - printed, it means that your CPU is not 64-bit. 1- is

64-bit.
3. $ ls /lib/modules/3.16.0-3 generic/kernel/arch/x86/kvm/kvm kvm-
amd.ko kvm-intel.ko kvm.ko
4. $ ls /dev/kvm
/dev/kvm
5. Install Necessary Packages using the following commands,
qemu-kvm
libvirt-bin
bridge-utils
virt-manager
qemu-system

6. Creating VMs
virt-install --connect
connect qemu:///system -n hardy -r 512 -f hardy1.qcow2 -s 12 -cc ubuntu-
ubuntu
14.04.2-server-amd64.iso --vnc
-- --noautoconsole --os-type linux --os-variant
variant ubuntuHardy
OUTPUT:
1. New virtual machine is created using KVM:

RESULT:
Thus the virtual machine of different configuration is created successfully.

EX. NO.:6 VIRTUAL BLOCK ATTACHMENT
AIM :
To find procedure to attach virtual block to the virtual machine and check whether it holds the
data even after the release of the virtual machine.
PROCEDURE:
This experiment is to be performed through portal. Login into Openstack portal, in instances,
create virtual machines.
In Volumes, create storage block of available capacity. Attach / Mount the storage block volumes
to virtual machines, unmount the volume and reattach it.
Volumes are block storage devices that you attach to instances to enable persistent storage. You
can attach a volume to a running instance or detach a volume and attach it to another instance at
any time. You can also create a snapshot from or delete a volume. Only administrative users can
create volume types.
Create a volume
1. Log in to the dashboard.
2. Select the appropriate project from the drop down menu at the top left.
3. On the Project tab, open the Compute tab and click Volumes category.
4. Click Create Volume.
In the dialog box that opens, enter or select the following values.
Volume Name: Specify a name for the volume.
Description: Optionally, provide a brief description for the volume.
Volume Source: Select one of the following options:
o No source, empty volume: Creates an empty volume. An empty volume does not
contain a file system or a partition table.
o Image: If you choose this option, a new field for Use image as a source displays.
You can select the image from the list.
o Volume: If you choose this option, a new field for Use volume as a
source displays. You can select the volume from the list. Options to use a
snapshot or a volume as the source for a volume are displayed only if there are
existing snapshots or volumes.
Type: Leave this field blank.
Size (GB): The size of the volume in gibibytes (GiB).
Availability Zone: Select the Availability Zone from the list. By default, this value is set
to the availability zone given by the cloud provider (for example, us-west or apac-south).
For some cases, it could be nova.
5. Click Create Volume.
The dashboard shows the volume on the Volumes tab.
Attach a volume to an instance

After you create one or more volumes, you can attach them to instances. You can attach a volume to
one instance at a time.
4. Select the volume to add to an instance and click Manage Attachments.
5. In the Manage Volume Attachments dialog box, select an instance.
6. Enter the name of the device from which the volume is accessible by the instance.
7. Click Attach Volume.
The dashboard shows the instance to which the volume is now attached and the device name.

You can view the status of a volume in the Volumes tab of the dashboard. The volume is either
Available or In-Use.
Now you can log in to the instance and mount, format, and use the disk.
Detach a volume from an instance

3. On the Project tab, open the Compute tab and click the Volumes category.
4. Select the volume and click Manage Attachments.
5. Click Detach Volume and confirm your changes.
A message indicates whether the action was successful.
Create a snapshot from a volume

4. Select a volume from which to create a snapshot.
5. In the Actions column, click Create Snapshot.
6. In the dialog box that opens, enter a snapshot name and a brief description.
7. Confirm your changes.
The dashboard shows the new volume snapshot in Volume Snapshots tab.
Edit a volume
4. Select the volume that you want to edit.
5. In the Actions column, click Edit Volume.
6. In the Edit Volume dialog box, update the name and description of the volume.
7. Click Edit Volume.
Delete a volume
When you delete an instance, the data in its attached volumes is not deleted.
4. Select the check boxes for the volumes that you want to delete.
5. Click Delete Volumes and confirm your choice.
A message indicates whether the action was successful.
RESULT:
Thus the new virtual block is successfully added to existing virtual machine.

EX. NO. : 7 INSTALLATION OF C COMPILER
AIM:
To find procedure to attach virtual block to the virtual machine and check whether it holds the data
even after the release of the virtual machine.
PROCEDURE:
1. Install a C compiler in the virtual machine and execute a sample program.

Through Openstack portal create virtual machine. Through the portal connect to virtual
machines. Login to VMs and install c compiler using commands.
Eg : apt-get install gcc
2. Show the virtual machine migration based on the certain condition from one node to the other.
To demonstrate virtual machine migration, two machines must be configured in one cloud. Take
snapshot of running virtual machine and copy the snapshot file to the other destination machine and
restore the snapshot. On restoring the snapshot, VM running in source will be migrated to destination
machine.
1. List the VMs you want to migrate, run:
$ nova list
2. After selecting a VM from the list, run this command where VM_ID is set to the ID in the list
returned in the previous step:
$ nova show VM_ID
3. Use the nova migrate command.
$ nova migrate VM_ID
4. To migrate an instance and watch the status, use this example script:
#!/bin/bash
# Provide usage
usage() {
echo "Usage: $0 VM_ID"
exit 1
}
[[ $# -eq 0 ]] && usage
# Migrate the VM to an alternate hypervisor
echo -n "Migrating instance to alternate host"
VM_ID=$1
nova migrate $VM_ID
VM_OUTPUT=`nova show $VM_ID`
VM_STATUS=`echo "$VM_OUTPUT" | grep status | awk '{print $4}'`
while [[ "$VM_STATUS" != "VERIFY_RESIZE" ]]; do
echo -n "."

sleep 2
VM_OUTPUT=`nova show $VM_ID`
VM_STATUS=`echo "$VM_OUTPUT" | grep status | awk '{print $4}'`
done
nova resize-confirm $VM_ID
echo " instance migrated and resized."
echo;
# Show the details for the VM
echo "Updated instance details:"
nova show $VM_ID
# Pause to allow users to examine VM details
read -p "Pausing, press <enter> to exit."
RESULT:
Thus the C Compiler is installed successfully and executed a sample C program.

EX. NO. : 8 VIRTUAL MACHINE MIGRATION
AIM:
To show the virtual machine migration based on the certain condition from one node to the
other.
STEPS TO MIGRATE THE VIRTUAL MACHINE:
1. Open virt-manager
2. Connect to the target host physical machine
Connect to the target host physical machine by clicking on the File menu, then click Add
Connection.

3. Add connection
The Add Connection window appears.
Enter the following details:
Hypervisor: Select QEMU/KVM.
Method: Select the connection method.
Username: Enter the username for the remote host physical machine. Hostname: Enter
the hostname/IP address for the remote host physical machine.
Click the Connect button. An SSH connection is used in this example, so the specified user's password
must be entered in the next step.

4. Migrate guest virtual machines
Open the list of guests inside the source host physical machine (click the small triangle on the left of the
host name) and right click on the guest that is to be migrated (guest1-rhel6-64
( in this example) and click
Migrate.
In the New Host field, use the drop-down

drop down list to select the host physical machine you wish to migrate the
guest virtual machine to and click Migrate.

A progress window will appear.
virt-manager now displays the newly migrated guest virtual machine running in the destination host. The
guest virtual machine that was running in the source host physical machine is now listed in the Shutoff
state.
RESULT:
Thus the virtual machine is migrated from one node to another node successfully.

EX. No: 9 PROCEDURE TO INSTALL STORAGE CONTROLLER

AND INTERACT WITH IT
AIM :
To find procedure to install storage controller and interact with it.
PROCEDURE:
Storage controller will be installed as Swift and Cinder components when installing Openstack.
The ways to interact with the storage will be done through portal.
OpenStack Object Storage (swift) is used for redundant, scalable data storage using clusters of
standardized servers to store petabytes of accessible data. It is a long-term storage system for
large amounts of static data which can be retrieved and updated.
OpenStack Object Storage provides a distributed, API-accessible storage platform that can be
integrated directly into an application or used to store any type of file, including VM images,
backups, archives, or media files. In the OpenStack dashboard, you can only manage containers
and objects.
In OpenStack Object Storage, containers provide storage for objects in a manner similar to a
Windows folder or Linux file directory, though they cannot be nested. An object in OpenStack
consists of the file to be stored in the container and any accompanying metadata.
Create a container
Log in to the dashboard
2. On the Project tab, open the Object Store tab and click Containers category.
3. Click Create Container.
4. In the Create Container dialog box, enter a name for the container, and then click Create
Container.
You have successfully created a container.
Upload an object
4. Select the container in which you want to store your object.
5. Click Upload Object.
The Upload Object To Container: <name> dialog box appears. <name> is the name of
the container to which you are uploading the object.
6. Enter a name for the object.
7. Browse to and select the file that you want to upload.
8. Click Upload Object.
You have successfully uploaded an object to the container.

Manage an object
To edit an object
4. Select the container in which you want to store your object.
5. Click the menu button and choose Edit from the dropdown list.
The Edit Object dialog box is displayed.
6. Browse to and select the file that you want to upload.
7. Click Update Object.
RESULT:
Thus the procedure to install storage controller and interact with it was done successfully.

GRID COMPUTING LAB
GLOBUS TOOLKIT
INTRODUCTION
The open source Globus Toolkit is a fundamental enabling technology for the "Grid," letting
people share computing power, databases, and other tools securely online across corporate,
institutional, and geographic boundaries without sacrificing local autonomy.
The toolkit includes software services and libraries for resource monitoring, discovery, and
management, plus security and file management.
In addition to being a central part of science and engineering projects that total nearly a half-
billion dollars internationally, the Globus Toolkit is a substrate on which leading IT companies
are building significant commercial Grid products.
INTALLATION PROCEDURE FOR GLOBUS TOOLKIT
MANDATORY PREREQUISITE:
Linux 64 bit Operating System
Download all softwares

1. Apache-tomcat-7.0.67-tar.gz
2. Apache-ant-1.9.6.bin.tar.gz
3. Junit 3.8.1.zp
4. Jdk-8u60-linux-x64.gz
Copy downloads to /usr/local and type the following commands
1. cp /home/stack/downloads/* /usr/local
2. pwd
3. tar zxvf jdk-8u60-linux-x64.gz

cd jdk1.8.0_60/
pwd
export JAVA_HOME=/usr/local/grid/SOFTWARE/jdk1.8.0_60/bin/
cd ..
4. tar zxvf apache-ant-1.9.6-bin.tar.gz

pwd
export ANT_HOME=/usr/local/grid/SOFTWARE/apache-ant-1.9.6
cd ..
5. tar zxvf apache-tomcat-7.0.67.tar.gz

cd apache-tomcat-7.0.67/
pwd
export CATALINA_HOME=/usr/local/grid/SOFTWARE/apache-tomcat-7.0.67
cd ..

6. unzip junit3.8.1.zip
cd junit3.8.1
pwd
export JUNIT_HOME=/usr/local/grid/SOFTWARE/junit3.8.1
cd ..
pwd
7. dpkg -i globus-toolkit-repo_latest_all.deb
8. apt-get update
9. apt get install globus-data-management-client

apt get install globus-gridftp
apt get install globus-gram5
apt get install globus-gsi
apt get install myproxy
apt get install myproxy-server
apt get install myproxy-admin
10. grid-cert-info -subject
11. grid-mapfile-add-entry {-------output of grid-cert-info -subject-----} gtuser

grid-proxy-init -verify -debug
12. service globus-gridftp-server start

service globus-gridftp-server status
13. myproxy-logon -s {name}
14. service globus-gatekeeper start
service globus-gatekeeper statusmyproxy-logon -s
15. globus-job-run name /bin/hostname

EX: NO: 01 DEVELOP NEW WEB SERVICE FOR CALCULATOR
AIM:
To Develop new web service for calculator using Globus toolkit
PROCEDURE :
When you start Globus toolkit container, there will be number of services starts up. The service for this
task will be a simple Math service that can perform basic arithmetic for a client.
The Math service will access a resource with two properties:
1. An integer value that can be operated upon by the service

2. A string values that holds string describing the last operation
The service itself will have three remotely accessible operations that operate upon value:
(a) add, that adds a to the resource property value.
(b) subtract that subtracts a from the resource property value.
(c) getValueRP that returns the current value of value.
Usually, the best way for any programming task is to begin with an overall description of what you
want the code to do, which in this case is the service interface. The service interface describes how what
the service provides in terms of names of operations, their arguments and return values. A Java
interface for our service is:
public interface
Math
{ public void add(int a);
public void subtract(int a);
public int getValueRP();
}
It is possible to start with this interface and create the necessary WSDL file using the standard Web
service tool called Java2WSDL. However, the WSDL file for GT 4 has to include details of resource
properties that are not given explicitly in the interface above.
Hence, we will provide the WSDL file. Step 1 Getting the Files All the required files are provided and
comes directly from [1]. The MathService source code files can be found from
http://www.gt4book.com (http://www.gt4book.com/downloads/gt4book-examples.tar.gz)
A Windows zip compressed version can be found at

http://www.cs.uncc.edu/~abw/ITCS4146S07/gt4book-examples.zip. Download and uncompress the file
into a directory called GT4services. Everything is included (the java source WSDL and deployment
files, etc.):
WSDL service interface description file -- The WSDL service interface description file is provided
within the GT4services folder at: GT4Services\schema\examples\MathService_instance\Math.wsdl
This file, and discussion of its contents, can be found in Appendix A. Later on we will need to modify
this file, but first we will use the existing contents that describe the Math service above.
Service code in Java -- For this assignment, both the code for service operations and for the resource
properties are put in the same class for convenience. More complex services and resources would be
defined in separate classes.
The Java code for the service and its resource properties is located within the GT4services folder at:

GT4services\org\globus\examples\services\core\first\impl\MathService.java.
Deployment Descriptor -- The deployment descriptor gives several different important sets of
information about the service once it is deployed. It is located within the GT4services folder at:
GT4services\org\globus\examples\services\core\first\deploy-server.wsdd. Step 2 Building the Math
Service It is now necessary to package all the required files into a GAR (Grid Archive) file.
The build tool ant from the Apache Software Foundation is used to achieve this as shown overleaf:
Generating a GAR file with Ant (from http://gdp.globus.org/gt4- tutorial/multiplehtml/ch03s04.html)

Ant is similar in concept to the Unix make tool but a java tool and XML based. Build scripts are
provided by Globus 4 to use the ant build file. The windows version of the build script for MathService
is the Python file called globus-build-service.py, which held in the GT4services directory. The build
script takes one argument, the name of your service that you want to deploy. To keep with the naming
convention in [1], this service will be called first.
In the Client Window, run the build script from the GT4services directory with: globus-build-service.py
first The output should look similar to the following: Buildfile: build.xml
BUILD SUCCESSFUL
Total time: 8 seconds
During the build process, a new directory is created in your GT4Services directory that is named build.
All of your stubs and class files that were generated will be in that directory and its subdirectories.
More importantly, there is a GAR (Grid Archive) file called
org_globus_examples_services_core_first.gar. The GAR file is the package that contains every file that
is needed to successfully deploy your Math Service into the Globus container.
The files contained in the GAR file are the Java class files, WSDL, compiled stubs, and the deployment
descriptor. Step 3 Deploying the Math Service If the container is still running in the Container
Window, then stop it using Control-C.

To deploy the Math Service, you will use a tool provided by the Globus Toolkit called globus-deploy-
gar. In the Container Window, issue the command: globus-deploy-gar
org_globus_examples_services_core_first.gar Successful output of the command is :
The service has now been deployed.
Check service is deployed by starting container from the Container Window:
You should see the service called MathService.

Step 4 Compiling the Client A client has already been provided to test the Math Service and is located
in the GT4Services directory.
Step 5 Start the Container for your Service Restart the Globus container from the Container Window
with: globus-start-container -nosec if the container is not running.
Step 6 Run the Client To start the client from your GT4Services directory, do the following in the
Client Window, which passes the GSH of the service as an argument: java -classpath
build\classes\org\globus\examples\services\core\first\impl\:%CLASSPATH%
org.globus.examples.clients.MathService_instance.Client
http://localhost:8080/wsrf/services/examples/core/first/MathService which should give the output:
Current value: 15 Current value: 10

Step 7 Undeploy the Math Service and Kill a Container Before we can add functionality to the Math
Service (Section 5), we must undeploy the service.
In the Container Window, kill the container with a Control-C.
Then to undeploy the service, type in the following command: globus-undeploy-gar
org_globus_examples_services_core_first which should result with the following output: Undeploying
gar... Deleting /.
Undeploy successful
RESULT:
Thus to develop new web service for calculator using Globus toolkit was done successfully.

EX.NO. 02: DEVELOP NEW OGSA-COMPLIANT WEB SERVICE
AIM
To develop a new OGSA-Compliant Web service in Grid Service using .NET language.
PROCEDURE
Developed by the Global Grid Forum

Aims to define a common, standard, open architecture for Grid Applications.
Defines a set of rules that make up a grid service.
Sharing and Coordinated use of diverse resources in Dynamic VO's
Two Technologies to build OGSA:
Globus Tool Kit

Community based Open Architecture, open set of services and software libraries that support Grids
and Grid applications.
Web Services
Popular Standards based framework for accessing network applications.
Technique for describing software components to be accessed.
Define methods for accessing and discovering the components.
Its another distributed computing technology which allows us to create client/server apps.
Grid Services and OGSA
OSGA
OGSA defines the semantics of a Grid Service instance.
Defines the basic behavior and does not specify what a service does and how it does it.
Doesn't address issues of implementation programming model, language, tools and execution
environment.
By Defining a set of service semantics OGSA specifies interactions between services in a manner
independent of any hosting environment.
To implement an OGSA-compliant GIS by exploiting .NET technologies. The implementation of Grid

Services with .NET technologies, management of the dynamic nature of the information describing the re-
sources (e.g. CPU usage, Memory usage, etc.), extension of the MS directory service functionalities (e.g.
Active Directory) in order to implement the OGSA Index Service functionalities.
The first issue is related to the implementation of Grid Service Specification prescribed in a MS .NET
language . In the framework of the GRASP project, we have selected the implementation of Grid Service
Specification provided by the Grid Computing Group of the Virginia University, named OGSI.NET.
To manage the dynamic nature of information describing the resources, GT3 leverages on Service
Data Providers. In the MS environment, we rely on Performance Counters and Windows Management
Instrumentation (WMI) architecture to implement the Service Data Providers. For each component of a
MS system we have a performance object (e.g. Processor Object) gathering all the performance data of

the related entity. Each performance object provides a set of Performance Counters that retrieves specific
performance data regarding the resource associated to the performance object. For example, the
%ProcessorTime is a Performance Counter of the Processor Object representing the percentage of time
during which the processor is executing a thread. The performance counters are based on services at the
operating system level, and they are integrated in the .NET platform. In fact, .NET Framework provides a
set of APIs that allows the management of the performance counters.
To perform the collection and provisioning of the performance data to an index service, we
leverage on Windows Management Instrumentation (WMI) architecture. WMI is a unifying architecture
that allows the access to data from a variety of underlying technologies. WMI is based on the Common
Information Model (CIM) schema, which is an industry standard specification.
Figure 1: WMI Architecture
Distributed Management Task Force (DMTF).WMI provides a three-tiered approach for collecting and
providing management data. This approach consists of a standard mechanism for storing data, a standard
protocol for obtaining and distributing management data, and a WMI provider. A WMI provider is a
Win32 Dynamic-Link
Library (DLL) that supplies instrumentation data for parts of the CIM schema. Figure 3 shows the
architecture of WMI.
When a request for management information comes from a consumer (see Figure 3) to the CIM Object
Manager (CIMON), the latter evaluates the re-quest, identifies which provider has the information, and
returns the data to the consumer. The consumer only requests the desired information, and never knows
the information source or any details about the way the information data are extracted from the underlying
API. The CIMOM and the CIM repository are implemented as a system service, called WinMgmt, and
are accessed through a set of Component Object Model (COM) interfaces.
To implement an OGSA-compliant Index Service, we exploit some Active Directory (AD)

features . AD is a directory service designed for distributed networking environments providing secure,

structured, hierarchical storage of information about interesting objects, such as users, computers,
services, inside an enterprise network. AD provides a rich support for locating and working with these
objects, allowing the organizations to e ciently share and manage information about network resources
and users. It acts as the central authority for network security, letting the operating system to readily
verify a user identity and to control his/her access to network resources.
Our goal is to implement a Grid Service that, taking the role of a consumer (see Figure 1), queries at
regular intervals the Service Data Providers of a VO (see Figure 2) to obtain resources information,
collect and aggregate these information, and allows to perform searches, among the resources of an
organization, matching a specified criteria (e.g. to search for a machine with a specified number of
CPUs). In our environment this Grid Service is called Global Information Grid Service (GIGS) (see
Figure 2).
Figure 2: Example of GIS in MS Environment

The hosts that run the GIGS have to be Domain Controllers (DC). A DC is a server computer,
running on Microsoft WindowsNT, Windows2000, or Windows Server2003 family operating systems,
that manages security for a domain. The use of a DC permits us to create a global catalog of all the
objects that reside in an organization, that is the primary goal of AD services. This scenario is depicted in
Figure 3 (a) where black-coloured machines run GIGS and the other ones are Service Data Providers.
Obviously, black-coloured hosts could also be Service Data Providers.
Figure 3: Service Provider GIGS strucure

In order to avoid that the catalog grows too big and becomes slow and clumsy, AD is partitioned
into units, the triangles in Figure 3 (a). For each unit there is at least a domain controller. The AD
partitioning scheme emulates the Windows 2000 domain hierarchy (see Figure 3 (b)). Consequently, the
unit of partition for AD services is the domain. GIGS has to implement an interface in order to obtain,
using a publish/subscribe method, a set of data from Service Data Providers describing an active directory
object. Such data are then recorded in the AD by using Active Directory Service Interface (ADSI), a
COM based interface to perform common tasks, such as adding new objects.
After having stored those data in AD, the GIGS should be able to query AD for retrieving such data. This
is obtained exploiting the Directory Services.
RESULT
Thus the program for developing OGSA- Complaint web service was successfully executed.

EX. NO:3 DEVELOPING NEW GRID SERVICE
AIM:
To develop new Grid Service
PROCEDURE :
1. Setting up Eclipse, GT4, Tomcat, and the other necessary plug-ins and tools
2. Creating and configuring the Eclipse project in preparation for the source files
3. Adding the source files (and reviewing their major features)
4. Creating the build/deploy Launch Configuration that orchestrates the automatic generation of the
remaining artifacts, assembling the GAR, and deploying the grid service into the Web services
container
5. Using the Launch Configuration to generate and deploy the grid service
6. Running and debugging the grid service in the Tomcat container
7. Executing the test client
8. To test the client, simply right-click the Client.java file and select Run > Run... from the pop-up
menu (See Figure 27).
9. In the Run dialog that is displayed, select the Arguments tab and enter
http://127.0.0.1:8080/wsrf/services/examples/ProvisionDirService in the Program Arguments:
textbox.
10. Run dialog
11. Run the client application by simply right-clicking the Client.java file and selecting Run > Java
Application

OUTPUT
Run Java Application
RESULT:
Thus to develop new Grid Service was done successfully.

Ex. No:4 DEVELOP APPLICATIONS USING JAVA - GRID APIS
AIM :
To develop Applications using Java Grid APIs
PROCEDURE:
1. Build a server-side SOAP service using Tomcat and Axis

2. Create connection stubs to support client-side use of the SOAP service
3. Build a custom client-side ClassLoader
4. Build the main client application
5. Build a trivial compute task designed to exercise the client ClassLoader
6. Test the grid computing framework
OUTPUT
RESULT:
Thus to develop Applications using Java Grid APIs was done successfully.

Ex. No:5 DEVELOP SECURED APPLICATIONS USING BASIC SECURITY IN GLOBUS
AIM :
To Develop secured applications using basic security in Globus
PROCEDURE:
Mandatory prerequisite:
Tomcat v4.0.3
Axis beta 1
Commons Logging v1.0
Java CoG Kit v0.9.12
Xerces v2.0.1
1.Installing the software Tomcat and deploy Axis on Tomcat
2. Install libraries to provide GSI support for Tomcat
Copy cog.jar, cryptix.jar, iaik_ssl.jar, iaik_jce_full.jar, iaik_javax_crypto.jar to Tomcat's

common/lib directory.
Check that log4j-core.jar and xerces.jar (or other XML parser) are in Tomcat's common/lib
directory.
Copy gsicatalina.jar to Tomcat's server/lib directory.
3. Deploy GSI support in Tomcat
Edit Tomcat's conf/server.xml

Add GSI Connector in <service> section:

<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="8443" minProcessors="5" maxProcessors="75"
enableLookups="true" authenticate="true"
acceptCount="10" debug="1" scheme="httpg" secure="true">
<Factory className="org.globus.tomcat.catalina.net.GSIServerSocketFactory"
cert="/etc/grid-security/hostcert.pem"
key="/etc/grid-security/hostkey.pem"
cacertdir="/etc/grid-security/certificates"
gridmap="/etc/grid-security/gridmap-file"
debug="1"/>
</Connector>

If you are testing under a user account, make sure that the proxy or certificates and keys
are readable by Tomcat. For testing purposes you can use user proxies or certificates
instead of host certificates e.g.:
<Connector className="org.apache.catalina.connector.http.HttpConnector"
port="8443" minProcessors="5" maxProcessors="75"
enableLookups="true" authenticate="true"
acceptCount="10" debug="1" scheme="httpg" secure="true">
<Factory className="org.globus.tomcat.catalina.net.GSIServerSocketFactory"
proxy="/tmp/x509u_up_neilc"
debug="1"/>
</Connector>
If you do test using user proxies, make sure the proxy has not expired!
Add a GSI Valve in the <engine> section:
<Valve className="org.globus.tomcat.catalina.valves.CertificatesValve"
debug="1" />
4. Install libraries to provide GSI support for Axis
Copy gsiaxis.jar to the WEB-INF/lib directory of your Axis installation under Tomcat.
5. Set your CLASSPATH correctly
You should ensure that the following jars from the axis/lib directory are in your classpath:
o axis.jar
o clutil.jar
o commons-logging.jar
o jaxrpc.jar
o log4j-core.jar
o tt-bytecode.jar
o wsdl4j.jar
You should also have these jars in your classpath:
o gsiaxis.jar
o cog.jar
o xerces.jar (or other XML parser)
6. Start the GSI enabled Tomcat/Axis server
Start up Tomcat as normal

Check the logs in Tomcat's logs/ directory to ensure the server started correctly. In particular check that:
apache_log.YYYY-MM-DD.txt does not contain any GSI related error messages
catalina.out contains messages saying "Welcome to the IAIK ... Library"
catalina_log.YYYY-MM-DD.txt contains messages saying "HttpConnector[8443] Starting
background thread" and "HttpProcessor[8443][N] Starting background thread"
localhost_log.YYYY-MM-DD.txt contains a message saying "WebappLoader[/axis]: Deploy
JAR /WEB-INF/lib/gsiaxis.jar"
7. Writing a GSI enabled Web Service
7.1. Implementing the service

The extensions made to Tomcat allow us to receive credentials through a transport-level security
mechanism. Tomcat exposes these credentials, and Axis makes them available as part of the
MessageContext.
Alpha 3 version
Let's assume we already have a web service called MyService with a single method, myMethod. When a
SOAP message request comes in over the GSI httpg transport, the Axis RPC despatcher will look for the
same method, but with an additional parameter: the MessageContext. So we can write a new myMethod
which takes an additional argument, the MessageContext.
This can be illustrated in the following example:
package org.globus.example;
import org.apache.axis.MessageContext;
import org.globus.axis.util.Util;
public class MyService {
// The "normal" method

public String myMethod(String arg) {
System.out.println("MyService: http request\n");
System.out.println("MyService: you sent " + arg);
return "Hello Web Services World!";
}
// Add a MessageContext argument to the normal method

public String myMethod(MessageContext ctx, String arg) {
System.out.println("MyService: httpg request\n");
System.out.println("GOT PROXY: " + Util.getCredentials(ctx));
}
Beta 1 version
In the Beta 1 version, you don't even need to write a different method. Instead the Message Context is put
on thread local store. This can be retrieved by calling MessageCOntext.getCurrentContext():
package org.globus.example;
import org.apache.axis.MessageContext;
import org.globus.axis.util.Util;
public class MyService {
// Beta 1 version
public String myMethod(String arg) {
System.out.println("MyService: httpg request\n");
// Retrieve the context from thread local

MessageContext ctx = MessageContext.getCurrentContext();
System.out.println("GOT PROXY: " + Util.getCredentials(ctx));
}

}
Part of the code provided by ANL in gsiaxis.jar is a utility package which includes the getCredentials()
method. This allows the service to extract the proxy credentials from the MessageContext.
7.2. Deploying the service
Before the service can be used it must be made available. This is done by deploying the service. This can
be done in a number of ways:
1. Use the Axis AdminClient to deploy the MyService classes.
2. Add the following entry to the server-config.wsdd file in WEB-INF directory of axis on Tomcat:
3. <service name="MyService" provider="java:RPC">
4. <parameter name="methodName" value="*"/>
5. <parameter name="className" value="org.globus.example.MyService"/>
6. </service>
8. Writing a GSI enabled Web Service client

As in the previous example, this is very similar to writing a normal web services client. There are some
additions required to use the new GSI over SSL transport:
Deploy a httpg transport chain
Use the Java CoG kit to load a Globus proxy
Use setProperty() to set GSI specifics in the Axis "Property Bag":
o globus credentials (the proxy certificate)
o authorisation type
o GSI mode (SSL, no delegation, full delegation, limited delegation)
Continue with the normal Axis SOAP service invokation:
o Set the target address for the service
o Provide the name of the method to be invoked
o Pass on any parameters required
o Set the type of the returned value
o Invoke the service
You can invoke this client by running:
java org.globus.example.Client -l httpg://127.0.0.1:8443/axis/servlet/AxisServlet "Hello!"
RESULT:
Thus to develop secured applications using basic security in Globus was done successfully.

Ex. No:6 DEVELOP A GRID PORTAL
AIM :
To Develop a Grid portal, where user can submit a job and get the result. Implement it with
and without GRAM concept.
PROCEDURE:
Step 1. Building the GridSphere distribution requires 1.5+. You will also need Ant 1.6+ available
at http://jakarta.apache.org/ant.
Step 2. You will also need a Tomcat 5.5.x servlet container available at
http://jakarta.apache.org/tomcat. In addition to providing a hosting environment for GridSphere,
Tomcat provides some of the required XML (JAR) libraries that are needed for compilation.
Step 3. Compiling and Deploying
Step 4. The Ant build script, build.xml, uses the build.properties file to specify any compilation
options. Edit build.properties appropriately for your needs.
Step 5. At this point, simply invoking "ant install" will deploy the GridSphere portlet container to
Tomcat using the default database. Please see the User Guide for more details on configuring the
database.
Step 6. The build.xml supports the following basic tasks:
install -- builds and deploys GridSphere, makes the documentation and installs the database
clean -- removes the build and dist directories including all the compiled classes
update -- updates the existing source code from CVS
compile -- compiles the GridSphere source code
deploy -- deploys the GridSphere framework and all portlets to a Tomcat servlet container located
at $CATALINA_HOME
create-database - creates a new, fresh database with original GridSphere settings, this wipes out
your current database
docs -- builds the Javadoc documentation from the source code
To see all the targets invoke "ant --projecthelp".
Step 7. Startup Tomcat and then go to http://127.0.0.1:8080/gridsphere/gridsphere to see the
portal.
RESULT:
Thus to develop a Grid portal, where user can submit a job and get the result using GRAM
concept was done successfully.

GCC Lab Manual

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

GCC Lab Manual

Uploaded by

Copyright:

Available Formats

DR.

SIVANTHI ADITANAR COLLEGE OF ENGINEERING

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

CS6712 GRID AND CLOUD COMPUTING LABORATORY

CLASS : IV CSE A & B

CS6712 GRID AND CLOUD COMPUTING LABORATORY

CLOUD COMPUTING LAB

LIST OF EQUIPMENT FOR A BATCH OF 30 STUDENTS:

Dept of CSE, DR.SACOE Page 1

Ex. No EXPERIMENT PAGE No.

TO WRITE A WORD COUNT PROGRAM TO DEMONSTRATE THE 12

WRITE A PROGRAM TO USE THE API'S OF HADOOP TO INTERACT 16

PROCEDURE TO INSTALL STORAGE CONTROLLER AND 30

GRID COMPUTING LAB 32

2 DEVELOP NEW OGSA-COMPLIANT WEB SERVICE 38

DEVELOP SECURED APPLICATIONS USING BASIC SECURITY IN 45

FACULTY IN CHARGE HOD/CSE

Dept of CSE, DR.SACOE Page 2

CLOUD COMPUTING LAB

Ex.No:1 FIND PROCEDURE TO SET UP THE ONE NODE HADOOP CLUSTER

The base Apache Hadoop framework is composed of the following modules:

1. Login into CentOS in student login. (Password:student)

Step 1: Install Java on CentOS 7

5. To install the java package give the command:

Step 2: Install Hadoop Framework in CentOS 7

Dept of CSE, DR.SACOE Page 3

# tar xfz hadoop-2.7.2.tar.gz

Then copy the directory content to hadoop account home path.

To change the permission for the copied files

8. Login with hadoop user

Configure Hadoop and Java Environment Variables on your system by editing

Append the following lines at the end of the .bash_profile file:

## JAVA env variables

## HADOOP env variables

Step 3: Configure Hadoop in CentOS 7

Dept of CSE, DR.SACOE Page 4

(The location of hadoop configuration files is $HADOOP_HOME/etc/hadoop/, which is

Add the following properties between <configuration> ... </configuration> tags.

Replace the dfs.data.dir and dfs.name.dir values accordingly.

Dept of CSE, DR.SACOE Page 5

Add the following lines to mapred-site.xml file:

15. Edit yarn-site.xml file

17. Open slaves file

Check for the following line

Step 4: Format Hadoop Namenode

$ hdfs namenode -format

Dept of CSE, DR.SACOE Page 6

Step 5: Start and Test Hadoop Cluster

Check the services status with the following command.

$ hdfs dfs -mkdir /my_storage

To view file content inside HDFS file system:

$ hdfs dfs -cat /my_storage/LICENSE.txt

To list a directory inside HDFS files system:

$ hdfs dfs -ls /my_storage/

To retrieve data from HDFS to our local file system:

$ hdfs dfs -get /my_storage/ ./

//Get the full list of HDFS command options by issuing:

$ hdfs dfs -help

Step 6: Browse Hadoop Services

In Hadoop page Select Utilities Browse your files

Dept of CSE, DR.SACOE Page 7

Dept of CSE, DR.SACOE Page 8

Step 1: Required Dependencies

Step 2: Download and install fuse