Image Processing :Assignment 4 Error analysis in diameter estimation

Date: 04.04.2011 Name: Laura Stoilescu Address: Calslaan 28-5, Zip 7522MC, Enschede Email: l.i.stoilescu@student.utwente.nl Student Number: S1070053 Educational Program: Electrical Engineering- Telecommunication Networks

The purpose of the assignment is to statistically analyse the results of the diameter estimator developed in exercise 3 for assessing its performance. To achieve that, we realise the performance evaluation for different values of the standard deviation and vessel diameter D. 1. Create a function get_diameter which implements the steps realised in exercise 3

From the previous exercise we know that 1.5 mm corresponds to 37.5 pixels therefore, for 1mm we will have 25 pixels. The 250x200 image corresponds to 10x8 mm2. I have implemented the diameter estimation algorithm for different images (various values for the nominal diameter):
clc, close all, clear all Dmin=1; Dmax=8; step=0.5; [IM IMsize] = create_pic(Dmin, Dmax, step); diam=zeros(1, IMsize); op= 'c'; sigma=4; for k=1:IMsize diam(k) = get_diameter(IM{k},sigma,op); end

where create_pic is the function used for creating different diameters test images:
function [IM,IMsize] = create_pic (Dmin, Dmax, step) if nargin~=3, error('Wrong number of input arguments') end D = Dmin:step:Dmax; [IMsize ~] = size(D'); IM=cell(1,IMsize); k=1; for i=1: IMsize IM{k}=im_bloodvessel(D(i)); k=k+1; end end

and get_diameter is the required function:

function [diam] = get_diameter(im,sigma,op) % get_diameters % computes the mean diameter of an object

% diam = get_diameters(im,sigma,op) calculates the mean diameter % in milimeters of the blood vessel from a given image % im - the image for which we apply the function; % sigma - represents the dispersion of the Gaussian PSFs used % to calculate the derivatives needed for edge detection. % op - the type of edge detection which is to be used: % 'c'- Canny edge detector using the gradient angle and % magnitude for finding edges. % 'm'- Marr-Hildreth edge detector, using the Laplacian % instead of the gradient map % get_diameters implements hysteresis thresholding with % thresholds properly chosen for the artificially created blood % vessel images. % Compute edge map thresholds =[0.2 0.001]; edge_map=ut_edge(im,op,'s',sigma,'h',thresholds); % Find local diameters local_diam=zeros(1,size(im,2)); [l c]=find(edge_map); c_length=size(c,1)/2 j=1; for i=1:c_length local_diam(i)=l(j+1)-l(j); j=j+2; end % Find mean diameter in mm diam=mean(local_diam)/25; % Display overlay marr=im; canny=im; thresholds=[0.2 0.001]; edge_map1=ut_edge(im,'m','s',2,'h',thresholds); edge_map2=ut_edge(im,'c','s',2,'h',thresholds); marr(edge_map1==1)=256; canny(edge_map2==1)=256; figure() subplot(1,2,1), imshow(marr,[],'Border','tight'),title('Marr - Hildreth'); subplot(1,2,2), imshow(canny,[],'Border','tight'),title('Canny'); end

The function is called for 19 images in which the vessel diameter being successively increased by 0.5 from 1 mm to 10 mm. was chosen 5, according to the value from the last exercise. As shown before, the Canny edge detector proves to be a lot more efficient than the Marr Hildreth one, the differences being noticeable at least for all values of D. The differences between the given values and the estimated ones are much larger for Marr-Hildreth estimators as it can be seen in Figure 3 and Figure 4. Comparing the edge maps obtained for each detector, both of them have pretty low performance when considering small diameters but that can be solved by picking another value.

Figure 1: The computed edge maps for Dmin=1 mm using both algorithms with =2

Figure 2: The computed edge maps for Dmin=1 mm using both algorithms with =5

Sigma = 5; Canny detector 10 9 8 7 6 5 4 3 2 1 0 Estimated diameters Real diameters

10

Figure 3: The estimated diameters (blue circles) versus the real ones (red squares) using Canny algorithm

Figure 4: The estimated diameters (blue circles) versus the real ones (red squares) using Marr- Hildreth algorithm

2. Apply the function get_diameters for =2 and identify the irregularities. Modify the function adding another output parameter which would indicate the success of the estimation Considering a tolerated error of 10% or 0.1 of the real diameter, for a standard deviation of =2, we get substantially worse results in comparison to the previous case =5 where we got even 100% successful estimations with the same tolerance level. For Canny Edge detectors we have figures Figure 6 and

Figure 5: Canny Edge Detector, Tolerance 10%: Success rate 40 %; Errors

Canny Edge Detector, 10%, Sigma = 2 8 7 6 5 4 3 2 1 0

15.50 %;

Figure 6: Canny Edge Detector, Tolerance 40%: Success rate 66.67 %; Errors 17.38 %;

Marr-Hildreth Edge Detector, 10%, Sigma = 2 8 7 6 5 4 3 2 1 0

Figure 7: Marr- Hildreth Edge detector, Tolerance 10 %: Success rate 0%; Error 68.19 %;

Figure 8: Marr- Hildreth Edge detector, Tolerance 40 %: Success rate 13.33%; Error 63.39 %;

As it can be deduced, the performance is much worse for these settings. A smaller standard deviation results in the detection of many fake edges which affects computing the diameter. The distance between the edges will be a lot smaller since there are many intermediary points which are considered to be edge points. Because of the low scale, the PSF covers a small area of the image and lots of small irregularities are found. A low dispersion means high sensitivity to small features. Blurring the image by using a higher scale solves the issue, because at scale t features smaller than are basically blurred away.

These irregularities can be found by counting the number of edge pixels on each column. If there are more than 2 edge pixels on any column, irregularities exist that would give an incorrect estimate for the diameter. Thus, the success variable will be set to 0. To summarize, irregularities exist when a column of the edge map contains more than 2 edge pixels. This condition is equivalent to counting the number of non zero elements from the edge map matrix. Normally this number should equal 400, or, more generally, double the width of the image, in pixels. If the number is lower or higher than that, either there are missing pixels or there are false positives. The Matlab code for this condition is:
% Test for irregularities num_non_zero=size(find(edge_map),1);

If num_non_zero is exactly 400, the function continues, otherwise it exits. The function was tested on various realizations of the blood vessel image and the condition worked as planned.
function [diam success] = get_diameters(im,sigma,op) % Compute edge map thresholds=[0.2 0.001]; edge_map= ut_edge (im,op,'s',sigma,'h',thresholds); % Test for irregularities num_non_zero=size(find(edge_map),1); if num_non_zero==400 % Find local diameters local_diam=zeros(1,size(im,2)); [l c]=find(edge_map); c_length=size(c,1)/2; % always an even number j=1; for i=1:c_length local_diam(i)=l(j+1)-l(j); j=j+2; end % Find mean diameter in mm diam=mean(local_diam)/25; success=true; else diam=0; success=false; return end end

This is another version of the function, initially I used double values for the success variable to compute the success rate and the total errors with respect to the tolerance margin set.
diff=abs(real-diam); if (diff<(0.1*real)) success=1; else % the error

success=0; end

so that:
Error = (sum(diff)/sum(d_t))*100; SuccessRate = (sum(success)/IMsize)*100;

3. Create a structure for counting the number of successes at increasing sigma values. For each sigma values run multiple trials and store the diameter in case of success. At this stage, the array containing the estimated diameters for each sigma value should be emptied at the end of the inner loop.
clc sigma=2:5:25; sigsize=size(sigma,2); ntrials=3; for i=1:sigsize s=zeros(2,ntrials); d=zeros(2,ntrials); for j=1:ntrials % s and d are vectors im=im_bloodvessel(D); % Test for various diameters [d(1,j) s(1,j)]=get_diameters(im,sigma(i),'m'); % Marr - Hildreth [d(2,j) s(2,j)]=get_diameters(im,sigma(i),'c'); % Canny end % Two vectors are needed because their size can differ [~ , ~, d_mh] = find(d(1,:)); % finds the non-zero elements [~ , ~, d_c] = find(d(2,:)); success(1,i)=sum(s(1,:))/ntrials; sdev(1,i)=std(d_mh); bias(1,i)=abs(mean(d_mh-D)); success(2,i)=sum(s(2,:))/ntrials; sdev(2,i)=std(d_c); bias(2,i)=abs(mean(d_c-D)); disp(['Iteration ',num2str(i)]); end

Variable success counts the number of successful calls to the function get_diameter. The vectors d and s store both successful and unsuccessful results because they need to be emptied at the beginning of every iteration of the external loop. Only storing successful results would have meant that s and d may have different sizes at every external iteration, which is very inefficient in Matlab. Running the code gives the following results: Sigma Successful calls 2 0 7 2 12 3 17 3 22 1

With small additions, this basic structure will represent the framework on which a statistical evaluation can be performed. 4. Extend the code such as the success rate is estimated and stored as a function of . Estimate and store the standard deviation and the bias as well. The following three lines estimate and store the success rate, the standard deviation and the bias.
[~ , ~, diam]=find(d); % finds the non-zero elements in d and stores them in diam success(i)=sum(s)/no_trials; std_deviation(i)=std(diam); bias(i)=abs(mean(diam)-1.5));

5. Apply the performance evaluation for both Marr Hildreth and Canny edge detectors and plot their success rate, standard deviation and bias as a function of .

The code was modified a bit in order to handle the evaluation of both algorithms. The final version is given below.
clc close all clear all sigma=2:1:25; D=1.5; no_iterations=size(sigma,2); no_trials=100; success=zeros(2,no_iterations); bias=zeros(2,no_iterations); std_deviation=zeros(2,no_iterations); tic for i=1:no_iterations s=zeros(2,no_trials); d=zeros(2,no_trials); for j=1:no_trials % s and d are vectors im=im_bloodvessel(D); % Test for various diameters [d(1,j) s(1,j)]=get_diameters(im,sigma(i),'m'); % Marr - Hildreth [d(2,j) s(2,j)]=get_diameters(im,sigma(i),'c'); % Canny End % Two vectors are needed because their size can differ [~ , ~, d_mh]=find(d(1,:)); % finds the non-zero elements [~ , ~, d_c]=find(d(2,:)); success(1,i)=sum(s(1,:))/no_trials; std_deviation(1,i)=std(d_mh); bias(1,i)=abs(mean(d_mh-D)); success(2,i)=sum(s(2,:))/no_trials; std_deviation(2,i)=std(d_c);

bias(2,i)=abs(mean(d_c-D)); disp(['Iteration ',num2str(i)]); end disp(['Total time: ',num2str(toc)]);

figure(1),title('Success rate as a function of \sigma') ylabel('Success rate') xlabel('\sigma') hold on plot(sigma,success(1,:),'LineWidth',2) plot(sigma,success(2,:),'--r','LineWidth',3) hold off legend('Marr-Hildreth','Canny'); grid minor figure(2),title('Standard deviation as a function of \sigma') ylabel('Standard deviation') xlabel('\sigma') hold on plot(sigma,std_deviation(1,:),'LineWidth',2) plot(sigma,std_deviation(2,:),'--r','LineWidth',3) hold off legend('Marr-Hildreth','Canny'); grid minor figure(3),title('Bias as a function of \sigma') ylabel('Bias') xlabel('\sigma') hold on plot(sigma,bias(1,:),'LineWidth',2) plot(sigma,bias(2,:),'--r','LineWidth',3) hold off legend('Marr-Hildreth','Canny'); grid minor success, bias and std_deviation are vectors of 2 lines and no_iterations columns. The first line

corresponds to the evaluation of the Marr Hildreth algorithm and the second line corresponds to the evaluation of the Canny algorithm. The bias was calculated as the absolute difference between the real diameter and the estimated diameter. For each value of , 100 trials were run. As can be seen from the graph in Figure 9, the Canny edge detector has a definite lead regarding the success rate until = 11. From that point on, the difference between the two edge detection methods become negligible.

Success rate as a function of 0.9 0.8 0.7 0.6

Success rate

Marr-Hildreth Canny

0.5 0.4 0.3 0.2 0.1 0

10

15

20

25

Figure 9: The success rates of the Marr - Hildreth and Canny edge detectors as functions of , for D = 1.5 mm

Standard deviation as a function of 0.05 Marr-Hildreth Canny 0.045

Standard deviation

0.04

0.035

0.03

0.025

0.02

10

15

20

25

Figure 10: Standard deviations of the Marr - Hildreth and Canny edge detectors as functions of , for D = 1.5 mm

As Figure 10 shows, the standard deviation of Marr Hildreth is not defined until = 6. This is because Marr Hildreth has a non zero success rate only starting with = 6. For = 7, the 100 sample estimates computed with Marr Hildreth have smaller variance than those computed with Canny. This changes however for every other value of . It can be safely concluded that the Canny method has less variance of its estimates, therefore being more consistent.
Bias as a function of 0.8 0.7 0.6 0.5
Bias

Marr-Hildreth Canny

0.4 0.3 0.2 0.1 0

10

15

20

25

Figure 11: Biases (in mm) of the Marr - Hildreth and Canny edge detectors as functions of , for D = 1.5 mm

Bias is a very important measure, as it shows the difference between the actual diameter and the estimated one. As stated above, for each value, bias was calculated as the absolute difference between the mean value of the sample estimates and the actual diameter. It is very interesting to see that the two algorithms have basically the same bias, i.e. the same mean error. Figure 11 might be misleading however, suggesting that the two algorithms yield the same error. This is not entirely true, because the Canny edge detector is more consistent over the number of trials than the Marr Hildreth one. This means that there is a higher probability that a single trial of the Marr Hildreth algorithm to give a larger error (bias) than a single trial of the Canny algorithm. In any case, both algorithms exhibit a minimum bias of 0.025 mm for = 14 and a maximum bias of 0.75 mm for = 25. The results lead to the conclusion that the Canny edge detector is at least as good as the Marr Hildreth one. Even if the success rate is mostly the same for values higher than 11, the Canny edge detector exhibits less variance while having the same bias as Marr Hildreth and, moreover, it works for smaller dispersions of the PSF.

6. Repeat the previous evaluation for D = 3 millimeters.

Success rate as a function of 1 0.9 0.8 0.7
Success rate

Marr-Hildreth Canny

0.6 0.5 0.4 0.3 0.2 0.1 0

10

15

20

25

Figure 12: The success rates of the Marr - Hildreth and Canny edge detectors as functions of , for D = 3 mm

Standard deviation as a function of 0.032 0.03 0.028

Standard deviation

Marr-Hildreth Canny

0.026 0.024 0.022 0.02 0.018 0.016

10

15

20

25

Figure 13: The standard deviations of the Marr - Hildreth and Canny edge detectors as functions of , for D = 3 mm

Bias as a function of 0.5 Marr-Hildreth Canny

0.45

0.4

0.35
Bias

0.3

0.25

0.2

10

15

20

25

Figure 14: Biases (in mm) of the Marr - Hildreth and Canny edge detectors as functions of , for D = 3 mm

The results are basically the same for D = 3 mm. The Canny edge detector has the upper hand in both success rate and variance. Both algorithms have the same bias. This time however, the minimum bias is at = 25 and has a value of 0.19 mm, while maximum bias is of 0.42 mm at = 15. It is clear that the Canny edge detector is better than the Marr Hildreth one. After = 11 however, the differences between the two algorithms become small, the Canny edge detector having a slight advantage because of the lower variance. However, there is another difference which might be relevant in some cases: the processing time needed for each algorithm. The following code was executed for calculating the processing time:

clc close all sigma=2:2:25; D=1.5; no_iterations=size(sigma,2); no_trials=10; tic for i=1:no_iterations for j=1:no_trials % Choose between Marr - Hildreth and Canny % get_diameters(im,sigma(i),'m'); % Marr - Hildreth get_diameters(im,sigma(i),'c'); % Canny end

end disp(['Total time marr m: ',num2str(toc)]);

The results obtained were: 14.2101 seconds for the Marr Hildreth algorithm. 24.7611 seconds for the Canny algorithm.

There were 12 values of used and 10 trials per value were executed. This gives a total of 120 calls to get_diameters. Given that the additional processing time required for checking the conditions of the two loops is the same for both algorithms, we can safely ignore it. This gives a mean of: 0.1184 seconds for a run of the Marr Hildreth algorithm. 0.2063 seconds for a run of the Canny algorithm.

This means that the Marr Hildreth algorithm is 42.61 % faster than the Canny algorithm. This could be significant in some applications and it could be a factor to consider.

7. Discuss the results and explain the behavior of the algorithm.

In my view, the biggest advantage that this diameter estimation method has is that it is bullet proof: it never gives false positives. It detects irregularities in its output and only returns valid estimates. On the other hand, the biggest disadvantage is that it needs a priori information: without some idea about the possible value of the actual diameter, the algorithm might return poor estimates. For example, as Figure 11 shows, the bias hits a maximum of 0.75 mm for = 25. This gives a relative error of 50 %, which is rather high for medicinal purposes. Things get a little better when the blood vessel is thicker, as the maximum bias of 0.42 mm at = 15 is equivalent to a relative error of only 14 %, 3.5 times lower than the maximum relative bias for D = 1.5 mm. The minimum bias for D = 3 mm is of 0.19 mm at = 25 and corresponds to a 4.67 % relative error. The minimum bias for D = 1.5 mm is of 0.025 mm for = 14 and corresponds to a relative error of 1.67 %. These impressive results show that the algorithm has a sweet spot for every diameter, where the bias is minimum for a given value. Given some a priori information, the algorithm performs very well. The fact that the bias depends on the standard deviation of the PSFs used is normal: is a measure of the scale at which the algorithm looks for changes in luminosity. If is very small, the noise will stand out from the background and the algorithm will detect the noise contour in addition to the contour of the blood vessel. This gives the so called irregularities in the returned edge maps and this is why a minimum value for is needed in order to get relevant results. The bias still depends on the chosen even after crossing this minimum value is due to the OTF of the imaging system with which the image was captured. The blood vessel itself is not very sharp and this can easily be concluded from the fact that it is quite difficult for even a human observer to pinpoint the exact contour of the blood vessel. , the scale at which the algorithm looks for changes in gray levels, is therefore a measure for the threshold at which the shape of the blood vessel is assumed to start. This is due to the fact that gives the

dispersion of the PSFs used to compute both the Laplacian (for Marr Hildreth) and the gradient magnitude (for Canny). The higher this value is, the more ground the PSF covers, so the derivative in a given pixel is dependent on a larger surface. Thus, as increases, the first and second order derivatives become more sensitive to changing gray levels. Because of that, the Laplacian and the gradient magnitude become smoother and are more sensitive. This basically lowers the relative gray level at which relevant change in gray levels is assumed to exist, thus increasing the estimated diameter. The quantitative coronary angiography method gives good and useful diameter estimates when provided with some a priori information regarding the actual diameter of the blood vessel. Given this information, a suitable range of dispersions can be chosen for the PSF, minimizing bias and ensuring the validity of the estimate.