Level 3b - Live Multiple Face Detection

Posted December 30, 2011

PROBLEM STATEMENT:
Make a Windows Form Application in C# using EmguCV library, such that 1- it gets an input image from a live web camera OR from some folder by loading/reading it. 2- detects faces (one or many) in the input image. 3- and shows these detected faces in the image by marked rectangle/ box .

LEARNING OBJECTIVE:

SNAPSHOT OF FINAL OUTPUT:

PRELIMINARIES TO FACE DETECTION: =============================================================== ==========

In my earlier post Getting Ready For Face Detection , i had collected and shared most of the basic concepts behind face detection that you need to know before implementing face detection. If you have not read that post then I strongly suggest you read it before diving into the implementation details, to avoid any sort of confusions as you proceed with this tutorial. Because that post provides you with brief 1. 2. 3. 4. 5. 6. materials to read, about: Machine learning and its relation to face detection. What is face detection?What are the different approaches to face detection? Which approach is best of all? Our Selected approach: Viola Jones Algorithm. Reasons for selecting Viola-Jones method Concept of how Viola Jones algorithm works The Haar Cascade

=============================================================== ========== IMPLEMENTED FACE DETECTION -- IN A NUTSHELL: =============================================================== ==========

Let me summarize for you:

In a nutshell, we will implement Viola-Jones algorithm because its the easiest ready to use face detection method which is supported by OpenCV/EmguCV and has great results.

In this method, OpenCV (and therefore EmguCV) uses a type of face detector called a Haar Cascade classifier the one introduced by Viola and Jones. The Haar Cascade is a classifier (detector) trained on thousands of human faces. This training data is stored in an XML file, and is later used by the classifier during detection. NOTE: 'Classifier' means the same thing as 'detector'! The image, on which detection must happen, could come from a local hard drive or directly from a camera. When the Viola-Jones object(face) classifier/detector runs over the image, it uses the trained data stored in an XML file's sometime earlier. With help of this data, the classifier decides if the identified object is a the object in question or not. in case of face detection, it decides if its Face or Not Face. Viola Jones Classification assumes a fixed scale for the object(face in our case), say

50x50 pixels. Since the object(face) in an image might be smaller or larger than this, the classifier runs over the image several times, to search for objects(faces) across a range of scales. This may seem an enormous amount of processing, but thanks to algorithmic tricks, classification is very fast, even when it's applied at several scales. The parameters to the classifier in the code control and "tune" the classifier's properties of detection. -----------------------------------------------------------------------------------------------------------------------------------------

Question: "Does Viola-Jones Classification/Detection work in the same way for any object? be it a face, eye, car, ball etc etc?! Answer: Indeed! Viola-Jones' Haar Classifier is a generic detector, whose classification depends on the trained data stored in the XML it uses. so whatever object was used to train this XML, the classifier will detect that object in the image.e.g, if XML holds the trained data for human faces, then Viola-Jones Haar Classifier becomes "face detector". but if XML had data stored about a car, then the classifier works as "Car detector". ----------------------------------------------------------------------------------------------------------------------------------------Now, lets begin with implementing Viola-Jones method of face detection, and as we do, well gain a better understanding of how it all fits together: the Haar classifier + the trained XML file + the parameters that tune the classifier.

============================================================== ===========

STEP BY STEP TUTORIAL:

=============================================================== ==========
I'll be taking the Camera Capture application, that we had last modified in Level 2 tutorial, to a more advanced level in this tutorial. For any details on Camera Capture application, or to start a project from scratch, please see Level 1 and then Level 2 to avoid any confusion or lack of information on the application during this tutorial. NOTE FOR THOSE WHO ARE implementing face detection on an image stored in some folder: you DON'T need Level 1 or 2's Camera Capture application; Feel free to use this ready-to-use EmguCV project in VS2010 and add the face detection code to it as mentioned in this tutorial. I suggest you to code behind the start button.

STEP 1: DECLARING THE CLASSIFIER - Declare an object of class HaarCascade ---------------------------------------------------------------------------------------------------------------------------------The Viola-Jones classifier/detector used in EmguCV for detecting faces in an image is called HaarCascade. Lets declare a global object of class HaarCascade and call it haar. Were making it global to allow any and all functions in our code to use it:
private HaarCascade haar;

(Please stream all the following videos at 720p for best visibility of code.)

STEP 2: LOAD THE HaarCascade XML file ---------------------------------------------------------------------------------------------------------------------------------A classifier uses data stored in an XML file to decide how to classify each image location. So naturally, haar will need some XML file to load trained data from. You'll need to tell the classifier (haar object in this case) where to find this data file you want it to use. It's better to locate the XML file we want to use and make sure our path to it is correct, before we code the rest of our face-detection program. -----------------------------------------------------------------------------------------------------------------------------------------

Question: Do I have to first train an XML file for face or object detection myself?! Answer: For face detection, NO! Not at ALL! Please do NOT be mistaken. You do NOT have to train an XML file for face detection yourself!! Thanks to Viola & Jones and other people who had taken on this strenuous job, they have already trained XMLs with thousands of face samples. So those XML files are bound to give good results, up to 100% as Ive seen. All you need is to use these available trained XML files. However if its an object other than a face that youre trying to detect, say, a car, THEN yes! You may need to train an XML file ONLY IF you do not find any such XML over the internet. Download these XMLs & see if there's the one you were looking for. ----------------------------------------------------------------------------------------------------------------------------------------I'll be using the XML file called haarcascade_frontalface_alt_tree.xml It had the best results, as far as I have tried, for frontal face detection. Your experience might differ depending on your requirements, so you may freely try other xml files available for face detection. You canDownload Here the various XML files I had found for frontal face detection in OpenCV setup and EmguCV setup folders. The haar object we declared earlier will hold the data loaded from this XML file. I prefer to load the XML file once and for all, when the windows form loads, so lets initialize our haar object in the Form_Load function (CameraCapture_Load() in our 'Live face detection' case) as follows:
// adjust path to find your XML file haar = new HaarCascade("haarcascade_frontalface_alt_tree.xml");

NOTE: If you're using ReadyEmguCVProject.sln as i provided above, please double click the 'ReadyEmguCVProject windows form' so that it creates and takes you to the load function. -----------------------------------------------------------------------------------------------------------------------------------------

Question: Why is there no path specified for XML file in your code? Where have you put this file? Answer: Thats because I had copied the XML file to the Debug folder of this project and am using it from there in my code. This way you need not specify any path to a file thats in your debug folder, you only have to specify the full file name with its extension. This also removes file location dependency for your XML; now that its with your other debug files, it makes no difference where you move the original XML file or your Debug folder to. Your code will execute without trouble! -----------------------------------------------------------------------------------------------------------------------------------------

STEP 3: SET THE IMAGE SOURCE FOR FACE DETECTION --Selecting 'where' face detection must happen in the code. ---------------------------------------------------------------------------------------------------------------------------------Face detection will naturally need an image to process. Here comes the question that where and when does the face detection take place? Should it be an image you saved earlier from somewhere, or should it be a live image from a web camera connected to your application? That all depends on you! And the choice you make is what decides where your face detection lines of code fit into the rest of the code. Since weve decided here to detect faces from a live image stream, well add the code inside our ProcessFrame() function, right after the command that fetches the frames from our web camera.

ALTERNATIVE: You can instead read an image from some folder and apply face detection on it. Youll have to load your image into ImageFrame instead of capturing an image from a web cam. For that, instead of code that fetches captured image, use the following lines of code :
//Read an image from hard disk at location e.g E:\IMAGES\ Image InputImg = Image.FromFile(@"E:\IMAGES\MyPic.jpg"); Image<Bgr,byte> ImageFrame = new Image<Bgr,byte>(new Bitmap(InputImg));

And then insert the face detection code after you have set this image source. NOTE: the @ sign in the image path above code makes path valid without having to write double slashes in path. Remove @ sign and see the error. -----------------------------------------------------------------------------------------------------------------------------------------

STEP 4: INSERT THE FACE DETECTION CODE: ---------------------------------------------------------------------------------------------------------------------------------After setting the source of image (where ours is live image captured by web camera), insert the code that performs face detection functions:

COPY-PASTE THIS CODE(as given in video):

-------------------------------------------------------------------------------------------------------------------------FOR LIVE FACE DETECTION: Copy-Paste This code into the ProcessFrame() function of Camera Capture Application.OR in the corresponding place in your own application. FOR FACE DETECTION ON LOADED IMAGE: Copy-Paste this code behind the 'Start' button of ReadyEmguCVProject Application. OR in the corresponding place in your own application. --------------------------------------------------------------------------------------------------------------------------

UNDERSTAND THE CODE:

-------------------------------------------------------------------------------------------------------------------------Let us understand step-wise and in detail what this code is doing:

4.1. Convert frame (Image) captured by Web camera to Grey-Scale: After loading/reading the input image, it must be converted to gray-scale so the detector can work on it. the following command converts a color EmguCV Image (ImageFrame) to a gray-scale EmguCV Image (grayFrame):
//convert the image to gray scale Image<Gray, byte> grayframe = ImageFrame.Convert<Gray, byte>();

4.2. Call the Detector -Detect faces from the Grey-Scale image After converting the image to gray-scale, the following command is the call to run the face detector on this gray-scale image (called grayframe here).
//detect faces from the gray-scale image and store into an array of type 'var',i.e 'MCvAvgComp[] var faces = grayframe.DetectHaarCascade(haar, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(25, 25))[0];

This function, DetectHaarCascade(), takes up to five parameters, the first being the classifier youre using (haar for HaarCascade in our case), rest of the four parameters are used for Tuning" the detector. And by tuning I mean, to detect a large face? small face? Set is not face criterion to a high threshold? or Lower threshold? etc etc. the parameters in our code is sort of default values. -----------------------------------------------------------------------------------------------------------------------------------------

AFTER, completely implementing the face detection in this tutorial, I highly recommend you to check out "tuning" the face detector by simply changing the parameter values in the call to DetectHaarCascade(). and see the change in detection result yourself! So that you have a better understanding of what they do to the detector. To see how to change the values and what are the valid values? please read Tuning the Detector by Setting its Parameters which explains what each of these 4 parameters does and what valid values you can assign to them. -----------------------------------------------------------------------------------------------------------------------------------------

To count the total number of faces detected in one call is simple, after detecting faces in the image, just add this:
int TotalFaces = faces.Length; only display it: // you can use TotalFaces from here on, i'll

MessageBox.Show("Total faces detected: " + TotalFaces.ToString());

faces in our code IS the array after all which has all the detected faces, yes? all you need is its length. Thanks to Wendy for asking this very useful question :) ----------------------------------------------------------------------------------------------------------------------------------------4.3. Mark detected faces within colored rectangle drawing a rectangle on the detected face is NOT the job of the classifier/detector! Youre going to have to do it yourself explicitly. In the above code, the following lines do this job:
//draw a green rectangle on each detected face in image foreach (var face in faces) { ImageFrame.Draw(face.rect, new Bgr(Color.Green), 3); }

4.4. Display the image with marked Detect faces Now that our faces have been detected and marked with rectangles, time to display the image with detections!
//show the image in the EmguCV ImageBox CamImageBox.Image = ImageFrame;

-----------------------------------------------------------------------------------------------------------------------------------------

ALTERNATIVE: To Display an EmguCV image in Windows Form Picture Box, use this code instead of the one in 4.4 above:
//Show the image in Windows Form PictureBox called "pictureBox1"

pictureBox1.Image = ImageFrame.ToBitmap();

----------------------------------------------------------------------------------------------------------------------------------------STEP 5: Add Missing 'Emgu.Cv.CvEnum' directive before debugging.

----------------------------------------------------------------------------------------------------------------------------------

MISSING DIRECTIVE: Do NOT forget to add the 'Emgu.Cv.CvEnum' directive

STEP 6 : Debug the program!

---------------------------------------------------------------------------------------------------------------------------------STEP 7: Handle Error 2: missing .dlls

----------------------------------------------------------------------------------------------------------------------------------

HANDLE ERROR: .DLL files are missing?! - ADD the new/more .dll files Required for face detection functioning!

STEP 8: Successful Debugging! ---------------------------------------------------------------------------------------------------------------------------------STEP 9: FINAL - "TUNE" YOUR DETECTION PARAMETERS!

A COMMON PROBLEM is: "What are the BEST parameter values for my detection?" OR "Face detection result is not up to my expectations/needs! How do I improve the detection results?!" Please Read the solution here. I HIGHLY RECOMMENDED THAT YOU READ THE NEXT POST, ITS AN INTEGRAL PART OF THIS TUTORIAL! OTHERWISE YOU WILL MISS OUT THOSE GREAT TIPS THAT CAN SAVE YOU FROM "OH NO, NOW HOW TO DO THIS" WORRIES!

=============================================================== ==========

SOURCE CODE - Live face detection

(VS 2010)

NOTE: if you're a beginner to Face detection, then i HIGHLY stress you read and follow the whole tutorial to understand and learn this project. The source code is here only to HELP you! -----------------------------------------------------------------------------------------------------------------------------------------------------