Jacob Chandran

English 11 AP
5/27/16
IM 6

Final Product (Experiment code)

Below is the code that I have refined and produced that trains a machine to recognize a
certain database of images, which is like a basic memory. Then, using a neural network,
LIBLINEAR, which is the model used in this experiment, different attributes and motifes of the
known object database, known to humans as adjectives, are assembled and used to classify
unknown objects that are then introduced to the system. In this experiment,the unknown class
was directly contrasted with known classes, and then grafted (this would be temporary in a real
world situation) onto the known class with the most simularities. Then, in order to see if the
unknown class had a certain attribute, one would test to see if the known class onto which the
unknown class was drafted had the attribute.
Next steps: The code below is a surface level method to train and test data using LIBLINEAR
which uses a neural network, however, in order to increase accuracy and refinement, one would
have to code the neural network and its parameters oneself. The subject of neural networks is
very complex and fairly new to the field of artificial intelligence so coding and understanding a
neural network will probably take time. However, if I would be able to code a neural network to
classify objects, The neural network would eventually optimize itself by fitting which mimicks
the way the brain and motor muscles learn. The uses and applications of neural networks are
boundless but the main task is to create a platform that allows for flexability and optimized
results whichever direction is ultimately taken.
%
% Attribute Classification Using the Indirect Method
%

Jacob Chandran
English 11 AP
5/27/16
IM 6
% Read overFeat features from .mat files, extract train, test data
% and train a one vs. all SVM Multiclassifier
%
% Jacob Chandran
% Dec 9, 2015
%
% attr_218
% attributes: {218x1 cell} the questions
% classes: {57x1 cell} the words
% classIDs: {57x1 cell} the wordnet id
% mean: [57x218 double] the scalar value between -1 and +1 for
% whether the class embodies the attribute
% dis06: [57x218 double] thresholded versions of mean (uses 0.6)
% dis08: [57x218 double] thresholded versions of mean (uses 0.8)

clear;
close all;
%
% load('/home/chandje1/Desktop/OVERFEAT/attr_218.mat')
% addpath(overfeatpath);
% addpath(liblinearpath);
%%%%%%%%%%%%%%%%%%%%%%%%% Definitions %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%
overfeatpath = '/home/chandje1/Desktop/OVERFEAT/mat/'
attributepath = '/home/chandje1/Desktop/human218-features/'
liblinearpath = '/home/chandje1/liblinear-2.1/matlab/'
addpath(overfeatpath);
addpath(attributepath);
addpath(liblinearpath);
load('/home/chandje1/Desktop/OVERFEAT/attr_218.mat')
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% End Definitions %%%%%%%%%%%%
%%%%%%%%%%%
% Set parameters
numOverFeat = 4096; % No of OverFeat Features
% attr_218
% attributes: {218x1 cell} the questions
% classes: {57x1 cell} the words
% classIDs: {57x1 cell} the wordnet id
% mean: [57x218 double] the scalar value between -1 and +1 for
% whether the class embodies the attribute

Jacob Chandran
English 11 AP
5/27/16
IM 6
% dis06: [57x218 double] thresholded versions of mean (uses 0.6)
% dis08: [57x218 double] thresholded versions of mean (uses 0.8)
load('Intel218Questions.mat')
%Question
workingQuestions = 0; skippedQuestions = 0;
QuestionsLength = length(Intel218Questions);
cumTestIdx = 1;
for curQuestion = 1: QuestionsLength
Intel218Questions(curQuestion);
% curQuestion
% Specify Train and Test Classes
% training classes
QuestionIndex = curQuestion;
values = attr_218.dis06(:,QuestionIndex);
positive_values = find(values == 1);
negative_values = find(values == -1);
positive = length(positive_values);
negative = length(negative_values);
fprintf('No of positive, negative values %d %d\n',positive,negative);
if (positive >= 10) && (negative >= 10)
thisWorks= curQuestion;
workingQuestions = workingQuestions+1;
fprintf('Current Question, workingQuestions %d %d\n',thisWorks,workingQuestions);
positive_values'
negative_values'
fprintf('Positive Classes \n');
attr_218.classes(positive_values(1:5))
fprintf('Negative Classes \n');
attr_218.classes(negative_values(1:5))
%dbstack, keyboard
% Positive for attribute
trainClassIndex = 1;
for trainclass15 = 1:5
trainingClasses{trainClassIndex,trainclass15} = [curQuestion;
attr_218.classes(positive_values(trainclass15))];
end
% Negative for attribute
for trainclass610 = 1:5
trainingClasses{trainClassIndex,trainclass610 + 5} = [curQuestion;
attr_218.classes(negative_values(trainclass610))];
end
trainingClasses
%dbstack,keyboard

Jacob Chandran
English 11 AP
5/27/16
IM 6
nTrainClasses = size(trainingClasses,2);
for trnIdx = 1:nTrainClasses
qIndex = cell2mat(trainingClasses{trnIdx}(1));
className = char(trainingClasses{trnIdx}(2));
fprintf('Train Index, Question Index, Class Name %d %d %s\n', trnIdx, qIndex, className);
trainingClassIndex = find(strcmpi(attr_218.classes,className));
fileID = attr_218.classIDs{trainingClassIndex};
fileName = sprintf('%s%s.mat',overfeatpath,fileID);
fprintf('Loading Training File No Class Name, Feature File %d %s %s\n',i,className, fileName);
if not(exist(fileName,'file'))
fprintf('ERROR : File, %s, does not exist\n',fileName);
dbstack, keyboard
else
temp = load(fileName);
featureArrayTrain{trnIdx} = temp.X;
labelArrayTrain{trnIdx} = temp.Y;
end
end
numImagesTrain = 0;
for idx = 1:length(featureArrayTrain)
numImagesTrain=numImagesTrain+size(featureArrayTrain{idx},1);
end
fprintf('TRAINING: We have %d total images with overfeat data for %d
classes.\n',numImagesTrain,length(featureArrayTrain))
% ==================================================================
% Create the training image sets
% ==================================================================
% Organize the full X matrix (before sampling train and test sets
X = zeros(nTrainClasses,numOverFeat);
Y = zeros(numImagesTrain,1); % the training class labels (0-indexed)
place = 1;
for trnIdx = 1:nTrainClasses
curdata = featureArrayTrain{trnIdx};
curLabel = labelArrayTrain{trnIdx};
X(place:(place+size(curdata,1)-1),:) = curdata;
% Y(place:(place+size(curdata,1)-1)) = ones(size(curdata,1),1)*(i-1);
Y(place:(place+size(curdata,1)-1)) = curLabel;
place = place+size(curdata,1);
end
% ensuring the rows of X are normalized (they were already practically
% normalized)
normsX = sqrt(sum(X.^2,2));
X = X./repmat(normsX,1,size(X,2));
% ====================================================================
% Create LIBLINEAR SVM Model

Jacob Chandran
English 11 AP
5/27/16
IM 6
% ====================================================================
% trainData = sparse(X);
% model = cell(numClasses,1);
% for k=1:numClasses
% model{k} = train(double(trainLabel==k), trainData);
% end
fprintf('Training LIBLINEAR MULTI-CLASS CLASSIFIER ...... \n');
model = train(Y, sparse(X));
%dbstack,keyboard
% =================================================================
% TESTING CLASSES
% =================================================================
%
test classes
%
positive for attribute
testClassIndex = 1;
for testclass15 =1:5
testClasses{testClassIndex,testclass15} = [curQuestion;attr_218.classes(positive_values(testclass15 +
5))];
end
%
%
negative for attribute
for testclass610 = 6:10
testClasses{testClassIndex,testclass610} =
[Intel218Questions(curQuestion);attr_218.classes(negative_values(testclass610))];
end
testClasses
%dbstack, keyboard
%%%%%%%%%%%%%%%% Test Class
% For each TEST class find the file id of the class, and load the feature
% data for the class
% Do prediction for the class and compute class probabilities
nTestClasses = size(testClasses,2);
for tstIdx = 1:nTestClasses
qTIndex = cell2mat(testClasses{tstIdx}(1));
classNameTest = char(testClasses{tstIdx}(2));
fprintf('Test Index, Question Index, Class Name %d %d %s\n', tstIdx, qTIndex, classNameTest);
testClassIndex = find(strcmpi(attr_218.classes,classNameTest));
fileID = attr_218.classIDs{testClassIndex};
fileName = sprintf('%s%s.mat',overfeatpath,fileID);
fprintf('Loading Testing File No Feature File %d %s\n',tstIdx,fileName);
if not(exist(fileName,'file'))
fprintf('ERROR : File, %s, does not exist\n',fileName);
dbstack, keyboard
else
temp = load(fileName);

Jacob Chandran
English 11 AP
5/27/16
IM 6
X = temp.X;
Y = temp.Y;
end
% ensuring the rows of X are normalized (they were already practically
% normalized)
normsX = sqrt(sum(X.^2,2));
X = X./repmat(normsX,1,size(X,2));
fprintf('Testing LIBLINEAR CLASSIFIER .............................\n');
fprintf('Test Class No, Test Class %g %s\n', tstIdx, (testClasses{tstIdx}{2}));
[predict_label, accuracy, prob_estimates] = predict(Y, sparse(X), model); %prob_estimates

probability = [0,0];
min_list = min(prob_estimates); %prob_estimates
probability(:,cumTestIdx) = probability(:,cumTestIdx)- repmat(min_list(cumTestIdx),
size(probability(:,cumTestIdx)));
max_list = max(prob_estimates);
probability(:,cumTestIdx) = 2*probability(:,cumTestIdx) ./ repmat(max(max_list(cumTestIdx)),
size(probability(:,cumTestIdx)));
while cumTestIdx < probability
cumTestIdx = cumTestIdx+1;
end
% Rescaling
lastshift = probability - 1;
probability = abs(lastshift);
probability = abs(lastshift);
%normalize the feature vectors
list_len = sqrt(sum(probability.^2,2));
probability = probability./repmat(list_len,1,size(probability,2));
end
%dbstack, keyboard

else
skippedQuestions = skippedQuestions +1;
fprintf('Skipped question No, No of Skipped Questions %d %d\n',curQuestion,skippedQuestions);
%dbstack, keyboard
end %else
end % Question
fprintf('No of Questions, workingQuestions, skippedQuestions %d %d
%d\n',QuestionsLength,workingQuestions,skippedQuestions);

% Analyse Experiment Results

%

Jacob Chandran
English 11 AP
5/27/16
IM 6
% Attribute Evaluation
%
% Analyze Cumulative Results from running program attr_experiment1.m
%
% Jacob Chandran
%
% attr_218
% attributes: {218x1 cell} the questions
% classes: {57x1 cell} the words
% classIDs: {57x1 cell} the wordnet id
% mean: [57x218 double] the scalar value between -1 and +1 for
% whether the class embodies the attribute
% dis06: [57x218 double] thresholded versions of mean (uses 0.6)
% dis08: [57x218 double] thresholded versions of mean (uses 0.8)
% Randomly Choose 20 TEST Classes from 57 Classes
% For Each Question, Keep Same Test Class, Find train classes that are not
% in test class, train and predict using test class
clear all; close all;
%%%%%%%%%%%%%%%%%%%%%%%% Definitions %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
overfeatpath
= '\\aplfsfox.jhuapl.edu\redd_projects\Information_Sciences\ZeroShot\DataSets\
ImageNet\ImageNet-data\OVERFEAT\mat\';
attributepath
= '\\aplfsfox.jhuapl.edu\redd_projects\Information_Sciences\ZeroShot\DataSets\
human218-features';
%liblinearpath = addpath('c:\Program Files\liblinear-2.1\matlab\');

Jacob Chandran
English 11 AP
5/27/16
IM 6
addpath(overfeatpath);
addpath(attributepath);
%addpath(liblinearpath);
load('\\aplfsfox.jhuapl.edu\redd_projects\Information_Sciences\ZeroShot\DataSe
ts\ImageNet\ImageNet-data\OVERFEAT\attr_218.mat');
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% End Definitions %%%%%%%%%%%%%%%%%%%%%%%
%
% Set parameters
numOverFeat = 4096; % No of OverFeat Features
load('Intel218Questions.mat');
% Analyze Cumulative results
resultFile = 'AttrExperiment_2016_2_1_1721.mat';
eval(sprintf('load %s ',resultFile));
workingQuestions = length(cumulativeResult);
fprintf('No of working questions %d\n',workingQuestions);
nZeroAttr = 0;
trnClassWts = [1 1 1 1 1 -1 -1 -1 -1 -1];
for cIdx = 1: workingQuestions
questionIdx = cumulativeResult{cIdx}{1};
%fprintf('Current Question: %d
%s\n',cIdx,char(Intel218Questions(questionIdx)));
fprintf('%d %s\n',cIdx,char(Intel218Questions(questionIdx)));
% fprintf('%s\n',char(Intel218Questions(questionIdx)));
attrList{cIdx} = char(Intel218Questions(questionIdx));
trainClassesA = cumulativeResult{cIdx}{2};

Jacob Chandran
English 11 AP
5/27/16
IM 6
testClassesA = cumulativeResult{cIdx}{3};
testClassIdxA = cumulativeResult{cIdx}{4};
% scoreQuestion = cumulativeResult{cIdx}{5};
% scoreQuestionWt = cumulativeResult{cIdx}{6};
% scoreQuestionAvWt = cumulativeResult{cIdx}{7};
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Extracting cumulative scaled prob estimate for each image
% in each class - Output nImages x no of train classes
cumScaledProbEstimates = cumulativeResult{cIdx}{8};
nTestClasses = length(cumScaledProbEstimates);
zeroAttrIdx = 0;
posAttrTh = 0.6; negAttrTh = -0.6; %thresholds
for tstIdx = 1:nTestClasses
classNameTest = char(testClassesA(tstIdx));
% fprintf('Test Index, Question Index, Class Name %d %d %s\n', tstIdx,
questionIdx, classNameTest);
tstClNameIdx = strmatch(classNameTest, attr_218.classes);
tstAttrValue = attr_218.dis06(tstClNameIdx,questionIdx);
scaledProbEst = cumScaledProbEstimates{tstIdx};
spEBuf = repmat(trnClassWts, size(scaledProbEst,1), 1);
scaledProbEstA = scaledProbEst.*spEBuf;
sumScaledProbE = sum(scaledProbEstA,2);
predAttrVal = zeros(size(sumScaledProbE));
posSumProbEIdx = find(sumScaledProbE > posAttrTh);

Jacob Chandran
English 11 AP
5/27/16
IM 6
zeroSumProbEIdx = find(sumScaledProbE < posAttrTh & sumScaledProbE >
negAttrTh);
negSumProbEIdx = find(sumScaledProbE < negAttrTh);
predAttrVal(posSumProbEIdx) = 1;
predAttrVal(negSumProbEIdx) = -1;
predAttrVal(zeroSumProbEIdx) = 0;
correctCount = length(find(predAttrVal==tstAttrValue));
errorCount = length(find(predAttrVal~=tstAttrValue));
percError = errorCount/(errorCount+correctCount)*100;
percErrorBuf(cIdx,tstIdx) = percError;
zeroAttrIdx = zeroAttrIdx+1;
% cumErrorResultAttr{cIdx} = {questionIdx, errorBuff, correctBuff};
end
end
percErrorBuf = percErrorBuf(:,
[10,12,13,15,16,17,19,3,6,18,1,2,4,5,9,11,14,20,7,8]);
imagesc(percErrorBuf);
colorbar
xlabel('Test Classes')
ylabel('Attributes')
title('Attribute Classification Error Plot');
set(gca,'XTick',1:workingQuestions)
% set(gca,'XTickLabel',
{'comb','pot','gravy','bathtub','flag','sugar','daisy','acorn','ball',...
% 'ape','bench','puppy','goat','lamp','zebra','otter',
'rabbit','lemonade','monkey','ski'})

Jacob Chandran
English 11 AP
5/27/16
IM 6
testClassesA = testClassesA(1,
[10,12,13,15,16,17,19,3,6,18,1,2,4,5,9,11,14,20,7,8]);
set(gca,'XTickLabel',testClassesA)
% yTick = [1 workingQuestions];
% yTick = [1 11 21 31 41 51 61 71 81];
%yTick = [1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81];
%yTick = [1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70
73 76 79 81];
yTick = [2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80];
idx = 1;
%attrTicList = attrList(1, [1,4,5,6,7,8,15,17,28,29,30,31,32,33,34,37,39,40,
13,45,2,3,9,10,11,12,14,16,18,19,20,21,22 21 22 23 24 25 26 27 36 38]);
%Unspaced list just to see colors more clearly
attrTicList = attrList(1, [1,4,5,6,7,8,15,17,28,29,30,31,32,33,34,37,39,40,
13,45,2,3,9,10,11,12,14,16,18,19,20,21,22 23 24 25 26 27 36 38 39 41 42 43 44
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81]);
% for ii = 1:length(yTick)
% attrTicList{ii} = attrList{yTick(ii)};
% end
% attrTicList = attrList(yTick);
set(gca,'YTick',1:81) %yTick
set(gca,'YTickLabel',attrTicList)
set(gca,'XTickLabelRotation',90);
for i = 1:workingQuestions
questionMeans(i) = mean(percErrorBuf(i,:));

Jacob Chandran
English 11 AP
5/27/16
IM 6
end
for i = 1:nTestClasses
testClassMeans(i) = mean(percErrorBuf(:,i));
end

The above figure shows the plot of the percentage error of attribute vs. test class. The x
axis corresponds to 20 test classes and the y axis corresponds to the correct attribute for lowest
classifcation error. Of the 81 attributes used in the experiment, 24 attributes correctly the
underlying objects Classification error varied from 17% to 46 %. Based on the results of this
experiment, visual attributes seem to outperform non visual attributes. Attributes like, Is it
brown (43.44%) or Does it cast a shadow (25%) consistantly produced less error than classes
like Can you buy it or is it tasty. One reason that visual attributes may produce less error
than nonvisual attributes is that they are directly verifyable based on an image of the object,
which is how the system is trained. An attribute such as can you buy it is relative and can be
affected by environmental conditions not actively observable in an image. In addition, most

Jacob Chandran
English 11 AP
5/27/16
IM 6

visual attributes contain fewer conditions than nonvisual attributes, which require much more
information than an image can provide in order to verify certain qualities. In conclusion, when
trying to accurately describe an object, the visual and directly observable characteristics are those
that can be most accurately predicted rather than those that have specific cases based on
environmental and outer conditions not observable from a simple image. Also, those attributes
that depend on the least amount of conditions offer less room for error in classification. In order
to reduce error, attribute questions should be as direct as possible, not leaving any room for
possible misinterpretation or variability.

Even though 17% error is not a huge problem, a neural network coded for the zero shot
problem can reach accuracy percentages of 99% which is far more reliable than 83% accuracy.
Also, using a custom neural network offers many more possible customizations which could lead
to less error or the adoption of a smaller data set with just as accurate results. The code I have
used this year has been helpful to understand how exactly a machine learns and used a pretrained
neural network, however, the next step is to create a custom neural network that can be much
more agile and flexable to changes in the data or changes in objective.