PROJECT REPORT ON

FINAL REPORT ON MINI-PROJECT

BHARATI VIDYAPEETHS COLLEGE OF ENGINEERING

A-4,Paschim Vihar, Rohtak Road, New Delhi

AFFILIATED TO
GURU GOBIND SINGH INDRAPRASTHA UNIVERSITY (G.G.S.I.P.U.)

Submitted to:

Submitted by:

IT DEPARTMENT

Bhavna Nakra (25211503110)

Diksha Mahajan (25011503110)
Surbhi Gupta (10511503110)

Acknowledgement

We are overwhelmed in all humbleness and gratefulness to acknowledge our depth to all
those who have helped us to put these ideas, well above the level of simplicity and into
something concrete.
We are very thankful to our guide Ms. Parul Yadav for approving of our project with
encouragement and her valuable help. She was always there to show us the right track when
we needed help. With the help of her valuable suggestions and guidance, we were able to
perform this project work.
We would also like to thank our teachers and friends, who often helped and gave us support
at critical junctures during the making of this project.

Bhavna Nakra
Diksha Mahajan
Surbhi Gupta

Table of Contents

Chapter-1: Introduction

Chapter-2: Features and Controls

Chapter-3: Concept Overview

Chapter-4: Source Code

Chapter-5: Output Screens

Chapter-6: Future Scope

Chapter-7: References

Page | 1

Chapter-1
Introduction
In the second module of our minor project we have moved onto Image drawing and Gesture
recognition. This has been implemented using OpenCV Version 2.2.0 and CodeBlocks 10.05.

OpenCV 2.2.0
The openCV version 2.2.0 was developed as a development in the 2.x series. The key new
features include the new modular structure of the library, the new feature detector/descriptor
framework, Latent SVM (Support Vector Machine) detector and Gradient Boosting Trees,
opencv-gpu - CUDA module created with support by NVidia, Android port created with
support by Google, new Qt backend in highgui with lots of nice features, improved
documentation with links to Wiki and hundreds of bug fixes.

Code::Blocks
It is a free and open source, cross-platform IDE which supports
multiple compilers including GCC and Visual C++. It is developed in C+
+ using wxWidgets as the GUI toolkit. Using a plugin architecture, its capabilities and
features are defined by the provided plug-ins. Currently, Code::Blocks is oriented
towards C and C++. In some cases installing third-party SDKs or frameworks is
necessary.Code::Blocks is being developed for Windows, Linux, and Mac OS X and has been
ported to FreeBSD
The latest stable version as of December 2012 is Code::Blocks 12.11, was released on
December 6, 2012.
Using the above library and platform, we have implemented a gesture drawing mechanism
that applies the colour detection and object tracking that we have studied in the first report.
This remote sketching takes yellow colour as a pen to draw lines of desired colour (Red,
Green or Blue) and thickness. The program also allows one to erase the drawing, clear the
screen or exit the program without touching.

Page | 2

Chapter-2
Features and Controls
The Remote Sketching program will offer the following:

Noise Handing:
- using area to ignore the noise
- using magnitude limit to skip long distance points
- using 5x5 median filter to decrease the background noise

Control features:
- choosing line colours
- changing line thickness based on Y coordinates
- clearing the screen + saving the image
- exiting the program
- saving the image (new)
The above features and controls are incorporated in the code using the concepts and logic
described next.

Page |3

Chapter- 3
Concept Overview
The following functions and concepts are used in the code:

1. cvCreateImage: IplImage* cvCreateImage(CvSize size, int depth, int channels)

This function creates an image header and allocates the image data.
size Image width and height
depth Bit depth of image elements.
channels Number of channels per pixel. This function only creates images with interleaved
channels.

2. cvCvtColor: void cvCvtColor(const CvArr* src, CvArr* dst, int code)

Converts an image from one color space to another.
src The source 8-bit (8u), 16-bit (16u) or single-precision floating-point (32f) image
dst The destination image of the same data type as the source. The number of channels may
be different
code Color conversion operation that can be specified.

3. cvInRangeS:void
er, CvArr* dst)

cvInRangeS(const CvArr* src, CvScalar lower, CvScalar upp

Checks that array elements lie between two scalars.

src The first source array

lower The inclusive lower boundary
upper The exclusive upper boundary
dst The destination array, must have 8u or 8s type
The function does the range check for every element of the input array

4. cvQueryframe:IplImage* cvQueryFrame(CvCapture* capture)

Grabs and returns a frame from a camera or file.
Capture- Video Capturing structure. The function cvQueryFrame grabs a frame from a
camera or video file, decompresses it and returns it. This function is just a combination of
GrabFrame and RetrieveFrame , but in one call. The returned image should not be released or
modified by the user. In the event of an error, the return value may be NULL.

5. cvFlip: cvFlip(IntPtr src,IntPtr dst,int flipMode) Causes flipping to avoid the

mirroring problem.
src (IntPtr)-Source array.
dst (IntPtr)-Destination array.
flipMode(Int32)-Specifies how to flip the array. flip_mode = 0 means flipping around x-axis,
flip_mode > 0 (e.g. 1) means flipping around y-axis and flip_mode < 0 (e.g. -1) means
flipping around

both axes.

Page |4

6. cvSmooth: cvSmooth Method (src, dst, type, param1, param2, param3, param4)

It will smoothen the image using one of several methods. Every of the methods has some
features and restrictions listed below:

Blur with no scaling works with single-channel images only and supports accumulation of 8bit to 16-bit format (similar to cvSobel and cvLaplace) and 32-bit floating point to 32-bit
floating-point format.
Simple blur and Gaussian blur support 1- or 3-channel, 8-bit and 32-bit floating point
images. These two methods can process images in-place.
Median and bilateral filters work with 1- or 3-channel 8-bit images and cannot process
images in-place.
Here we have used the median filter, which is a nonlinear digital filtering technique, often
used to remove noise. Such noise reduction is a typical pre-processing step to improve the
results of later processing (for example,edge detection on an image). Median filtering is very
widely used in digital image processing because, under certain conditions, it preserves edges
while removing noise. The main idea of the median filter is to run through the signal entry by
entry, replacing each entry with the median of neighbouring entries. The pattern of
neighbours is called the "window", which slides, entry by entry, over the entire signal.

7. Moments: Calculates all moments up to third order of a polygon or

rasterized shape
void cvMoments( const CvArr* arr, CvMoments* moments, int isBinary=0 );
arr-Image (1-channel or 3-channel with COI set) or polygon (CvSeq of points of a vector of
points).
Moments- Pointer to returned moment state structure
isBinary- (For images only) If the flag is non-zero, all the zero pixel values are treated as
zeroes, all the others are treated as ones.

The function cvMoments calculates spatial and central moments up to the third order and
writes them to moments. The moments may be used then to calculate gravity center of the
shape, its area, main axises and various shape characeteristics including 7 Hu invariants.
GetSpatialMoment: Retrieves spatial moment from moment state
structure

double cvGetSpatialMoment( CvMoments* moments, int j, int i );

moments- The moment state, calculated by cvMoments.
j- x-order of the retrieved moment, j >= 0.
i-

y-order of the retrieved moment, i >= 0 and i + j <= 3.

The function cvGetSpatialMoment retrieves the spatial moment, which in case of image
moments is defined as:
Mji=sumx,y(I(x,y)xjyi)
where I(x,y) is the intensity of the pixel (x, y).
GetCentralMoment: Retrieves central moment from moment state
structure
double cvGetCentralMoment( CvMoments* moments, int j, int i );
moments- Pointer to the moment state structure.
j- x-order of the retrieved moment, j >= 0.
i-y-order of the retrieved moment, i >= 0 and i + j <= 3.
The functioncvGetCentralMoment retrieves the central moment, which in case of image
moments is defined as:
ij=sumx,y(I(x,y)(x-xc)j(y-yc)i),

Page |5
where xc=M10/M00, yc=M01/M00 - coordinates of the gravity center

In simple words, the moment of 0th degree for a black/white image with black=0 and
white=1: this is simply the sum of the pixels, i.e. the number of white pixels. Moment of 1st
degree for x-axis and some particular point X on the x-axis: this is the sum of the white pixel
distances from X. I.e. it is the sum of their positions w.r.t. X. If we divide this by the number

of white pixels (0th moment) you get the average white pixel position wrt. X. Similarly for yaxis, the moments can be found.

8. cvPutText: void cvPutText(CvArr* img, const char* text, CvPoint org, const
CvFont* font, CvScalar color)
Draws a text string.
img Input image
text String to print
org Coordinates of the bottom-left corner of the first letter
font Pointer to the font structure
color Text color
The function renders the text in the image with the specified font and color. The printed text
is clipped by the ROI rectangle. Symbols that do not belong to the specified font are replaced
with the symbol for a rectangle.

9. cvAdd:cvAdd(src1,src2,dst)
Adds one array to another one: dst(I)=src1(I)+src2(I) if mask(I)!=0All the arrays must have
the same type, except the mask, and the same size (or ROI size)
src1 (IntPtr)-The first source array.
src2 (IntPtr)-The second source array.
dst (IntPtr)-The destination array.
mask (IntPtr)-Operation mask, 8-bit single channel array; specifies elements of destination
array to be changed.

10. CvAnd:void cvAnd(const CvArr* src1, const CvArr* src2, CvArr* dst, const
CvArr* mask=NULL)
Calculates per-element bit-wise conjunction of two arrays.
src1-First source array
src2-Second source array
dst-Destination source array

mask-Operation mask, 8-bit single channel array; specifies elements of the destination array
to be changed.
The function calculates per-element bit-wise logical conjunction of two arrays:
dst(I)=src1(I)&src2(I) if mask(I)!=0
In the case of floating-point arrays their bit representations are used for the operation. All the
arrays must have the same type, except the mask, and the same size.
11. cvOverlayImage: cvOverlayImage(IplImage* src, IplImage* overlay, CvPoint location,
CvScalar S, CvScalar D)
This is used to overlay an image onto another at a specified point.
12. cvSaveImage(const char* filename, const CvArr* image) : Saves an image to a specified
file. filename Name of the file. image Image to be saved.

Page |6
13.
Parameters:
14.
The function cvSaveImage saves the image to the specified file. The image format is chosen
based on the filename extension. Only 8-bit single-channel or 3-channel (with BGR
channel order) images can be saved using this function.
15. cvGet2D/ cvSet2D: cvGet2D( IntPtr arr, int idx0, int idx1 ) : Returns/ Sets a
particular array element. cvSet2D( IntPtr arr, int idx0, int idx1, MCvScalar value )
Parameters
arr (IntPtr)
Input array. Must have a single channel
idx0 (Int32)
The first zero-based component of the element index
idx1 (Int32)
The second zero-based component of the element index

Page |7

Chapter-4
Source Code
#include <opencv/cv.h>
#include <opencv/highgui.h>
#include <iostream>
#include <cmath>

using namespace std;

#define blue CV_RGB(0,0,255)

#define green CV_RGB(0,255,0)
#define red CV_RGB(255,0,0)
#define white CV_RGB(255,255,255)

#define black CV_RGB(0,0,0)

void cvOverlayImage(IplImage* src, IplImage* overlay, CvPoint location, CvScalar S,

CvScalar D)
{
int x,y,i;

for(x=0;x < overlay->width - 10 ;x++)

{
if(x+location.x>=src->width) continue;
for(y=0;y < overlay->height - 10 ;y++)
{
Page |8

if(y+location.y>=src->height) continue;
CvScalar source = cvGet2D(src, y+location.y, x+location.x);
CvScalar over = cvGet2D(overlay, y, x);
CvScalar merged;
for(i=0;i<4;i++)
merged.val[i] = (S.val[i]*source.val[i]+D.val[i]*over.val[i]);
cvSet2D(src, y+location.y, x+location.x, merged);
}
}
}

void ClearScreen(IplImage* imgScribble, IplImage* imgDrawing)

{
cvSet(imgScribble, black);
cvSet(imgDrawing, white);
}

IplImage* GetThresholdedImage(IplImage* img, CvScalar& lowerBound, CvScalar&

upperBound)
{
// Convert the image into an HSV image
IplImage* imgHSV = cvCreateImage(cvGetSize(img), 8, 3);
cvCvtColor(img, imgHSV, CV_BGR2HSV);

Page |9

IplImage* imgThreshed = cvCreateImage(cvGetSize(img), 8, 1);

cvInRangeS(imgHSV, lowerBound, upperBound, imgThreshed);

cvReleaseImage(&imgHSV);
return imgThreshed;
}

int main()
{
// controls
double area_limit = 700;

CvScalar lowerBound = cvScalar(20, 100, 100); // yellow

CvScalar upperBound = cvScalar(30, 255, 255);

// defaults
int lineThickness = 2;
CvScalar lineColor = blue;

CvCapture* capture = 0;
capture = cvCaptureFromCAM(0);
if(!capture)
{
P a g e | 10

cout << "Could not initialize capturing...\n";

return -1;
}

// This image holds the "scribble" data...

// the tracked positions of the pointer object
IplImage* imgScribble = NULL;

IplImage* imgColorPanel = 0;
imgColorPanel = cvLoadImage( "pp.png",CV_LOAD_IMAGE_UNCHANGED); // load
the panel image. (This is a png image, not designed/included in the source code!)

if(!imgColorPanel)
{
cout << "panel is not found !!! \n";
return -1;
}

IplImage* imgDrawing = 0;
imgDrawing = cvCreateImage( cvSize(cvQueryFrame(capture)>width,cvQueryFrame(capture)->height),
cvQueryFrame(capture)->depth,

//Bit depth per channel

3 //number of channels
);
cvSet(imgDrawing, white);
P a g e | 11

CvFont font, fontbig;

cvInitFont( &font, CV_FONT_HERSHEY_COMPLEX, 1, .6, 0, 2, CV_AA);
cvInitFont( &fontbig, CV_FONT_HERSHEY_COMPLEX, 3, .6, 0, 3, CV_AA);

int confirm_close = 10, confirm_clear = 20; // counters for clear and exit confirmation
char buffer [50]; // buffer for cvPutText
int image_num = 0; // to keep track of image numbers for saving
int posX = 0;
int posY = 0;

while(true)

{
IplImage* frame = 0;
frame = cvQueryFrame(capture);
if(!frame)
break;
cvFlip(frame,NULL,1); // flip the frame to overcome mirroring problem

// If this is the first frame, we need to initialize it

if(imgScribble == NULL)
imgScribble = cvCreateImage(cvGetSize(frame), 8, 3);
// cvSet( imgScribble, cvScalar(0,0,0));
P a g e | 12

// Median filter to decrease the background noise

cvSmooth( frame, frame,
CV_MEDIAN,
5, 5 //parameters for filter, in this case it is filter size
);

// Holds the thresholded image (tracked color -> white, the rest -> black)
IplImage* imgThresh = GetThresholdedImage(frame,lowerBound,upperBound);

// Calculate the moments to estimate the position of the object

CvMoments *moments = (CvMoments*)malloc(sizeof(CvMoments));
cvMoments(imgThresh, moments, 1);

// The actual moment values

double moment10 = cvGetSpatialMoment(moments, 1, 0);
double moment01 = cvGetSpatialMoment(moments, 0, 1);
double area = cvGetCentralMoment(moments, 0, 0);

// Holding the last and current positions

int lastX = posX;
int lastY = posY;
P a g e | 13

posX = 0;
posY = 0;

if(moment10/area>=0 && moment10/area < 1280 && moment01/area >=0 &&

moment01/area < 1280
&& area>area_limit /* to control the limit */ )
{
posX = moment10/area;
posY = moment01/area;
}

CvPoint cvpoint = cvPoint(150,30); // location of the text

if(posX < 90 && posY > 400) // clear
{
lineColor = white; // white color works as eraser
cvPutText( frame, "Eraser selected.", cvpoint, &font, white );
sprintf (buffer, "Clearing the screen in %d",confirm_clear); // count-down for clearing
the screen
cvPutText( frame, buffer, cvPoint(150,70), &font, red );
confirm_clear--;
if(confirm_clear < 0) // confirm in 10 frames before clearing
{
P a g e | 14

confirm_clear = 20;
sprintf (buffer, "d0%d.jpg",image_num++);
cvSaveImage(buffer ,imgDrawing); // save the frame into an image
ClearScreen(imgScribble,imgDrawing);
cvPutText( frame, "Cleared the screen.", cvPoint(150,110), &font, white );
}
}
else if(posX > 540 && posY > 360) // blue
{
lineColor = blue;

cvPutText( frame, "Blue color selected.", cvpoint, &font, blue );

}

else if(posX > 540 && posY > 200 && posY < 280) // green
{
lineColor = green;
cvPutText( frame, "Green color selected.", cvpoint, &font, green );
}

else if(posX > 540 && posY < 120) // red

{
lineColor = red;
cvPutText( frame, "Red color selected.", cvpoint, &font, red );
}
P a g e | 15

else if(posX > 0 && posX < 90 && posY > 0 && posY < 120) // exit
{
sprintf (buffer, "EXITING in %d",confirm_close);
cvPutText( frame, buffer, cvpoint, &font, red );
confirm_close--;
if(confirm_close < 0) // confirm in 10 frames before exit
break;
}
else if(posX < 90 && posY > 130 && posY < 390) // line thickness
{

lineThickness = 6 - ( posY/60-1 ); // change the thickness of line from 1 - 5 based on

posY
}

sprintf (buffer, "%d",lineThickness);

cvPutText( frame, buffer, cvPoint(40,255), &fontbig, lineColor );

double magnitude = sqrt( pow(lastX-posX,2) + pow(lastY-posY,2) );

// We want to draw a line only if its a valid position
//if(lastX>0 && lastY>0 && posX>0 && posY>0)
if(magnitude > 0 && magnitude < 100 && posX > 120 && posX<530)
{
P a g e | 16

// Draw a line from the previous point to the current point

cvLine(imgDrawing, cvPoint(posX, posY), cvPoint(lastX, lastY), lineColor,
lineThickness,CV_AA);
}

// Add the scribbling image and the frame...

cvAdd(imgDrawing, imgScribble, imgDrawing);

// Combine everything in frame

cvAnd(frame, imgDrawing, frame);

cvOverlayImage(frame, imgColorPanel, cvPoint(20, 20), cvScalar(1.5,1.5,1.5,1.5),

cvScalar(0.75,0.75,0.75,0.75));

cvShowImage("Threshold", imgThresh);
cvShowImage("Drawing", imgDrawing);

cvShowImage("Video", frame);
//cvShowImage("Video", imgColorPanel);
cvOverlayImage(frame, imgColorPanel, cvPoint(20, 20), cvScalar(2.0,2.0,2.0,2.0),
cvScalar(0.1,0.1,0.1,0.1));
//cvOr(frame, imgColorPanel, frame);
int c = cvWaitKey(10);
if(c==27) //ESC key
break;
P a g e | 17

//else if(c==49) // 1 key

cvReleaseImage(&imgThresh);
delete moments;
}

cvReleaseCapture(&capture);
//

cvReleaseImage(&imgColorPanel);
cvReleaseImage(&imgScribble);

return 0;
}

Page | 18

Chapter- 5
Output Screens
The Panel

Gesture drawing

P a g e | 19

Threshold Image

Erasing operation

P a g e | 20
Cleared window

Exiting the program

P a g e | 21

Chapter-6
Future Scope
The Gesture Drawing program can be equipped with the following add-ons to help develop
interactive game software to offer a different and a better experience.
1.
2.
3.
4.
5.

Mouse clicks using gestures and color detection

Keyboard operation using gestures, to operate games etc
Flick and pinch to zoom-in or zoom-out images
Face detection
Simple game implementation

Chapter-7
References
1.
2.
3.
4.

http://8a52labs.wordpress.com

E-book Learning openCV by O Reilly

http://docs.opencv.org/doc/tutorials/tutorials.html

http://www.youtube.com/user/18F4550videos
5. http://stackoverflow.com/questions