End of the Road

Hello all, this is the final blog post for the project.
Thank you for following this blog.

The Best Results first

Here are the extracted control points on top of the manually selected ones.

yellow - my extraction approach

light blue - manually selected points

• You might notice that there is a fundamental difference in the control points.  I am using the mouth corners, and no nose base.

After ignoring the mouth corners and nose base, I am able to achieve the following performance

Subject M1 4th surprise face VS 43 subjects( 13 M, 30 F)

Just to reiterate, this the performance I was achieving using the manually selected control points. 

43 subjects( 13 M, 30 F)

A probem emerges:

• for some images, the 3d camera was able to capture some of the slope inside the nostril.  This causes the steepest edge detected to lie inside the nose as opposed to on the actual control point.

  

Other Trials

Here are some of the not so good results:

 

Here are some better results

This is the accuracy of the system I was trying to achieve.  I hoped that by using 3-dimensional data, I would have higher accuracy simply because I had higher resolution.  Although the paper I followed does claim that the previous statement is true, I was unable to reproduce such stunningly high results using my more artistic approach.  I think if I would have been more sophisticated with the nose width points, the creativity of the eye width control points would have been just as accurate as needed.

Thank you.

Control Point Extraction

The Final Control Points:

Control Points were extracted in this order:

The order is important because the accuracy of points 3-5 is dependent on point 2.  All points are dependent of point 1.

1. Nose Tip
2. Nose Width Points (Left & Right)
3. Inner Eye Corner (Left & Right)
4. Outer Eye Corner (Left & Right)
5. Mouth Corners (Left & Right)

The Control Points were extracted using different methods:

Nose Tip - The greatest z-axis value

Nose Width - A search area is defined where the Control Point is expected to be.  The Control Point is determined by finding the steepest curve.

InnerEye - A search area is defined directly above the Nose Width Point.  The point with the lowest Z-Coordinate is determined to be the Control Point

OuterEye - The distance between the two InnerEye points is added to each control Point to determine the OuterEye point.

MouthCorner - Half the distance between from the InnerEye to the OuterEye is added to the InnerEye.  A straight line is drawn downward in the y-direction to define a new search area.  Within this region, the point with the lowest z-value is estimated to be the final control point.

New Approaches to Normalization

The first thing we did was to change our normalization algorithm.  Last week, we translated, rotated, scaled, and sheared each face so as to align them all on top of another.  This week, we undid the shearing.  Instead, 

• We computed the SVD of the Affine Matrix ( A = U * S * V')
• We found the estimated rigid rotation matrix R = U * V' (excluding S, which is the scaling component of equation). 

A. Subject: Male #1, Expression: Surprised, Sample #1 - BLACK

B. Subject: Male #1, Expression: Surprised, Sample #4 - BLUE

(B is transformed using R and translation to A)

Here are the results after running the Matching against the new normalization methods.  

• With 23 subjects or less, we achieved perfect sensitivity vs specificity. 
•  As we increased the number of subjects into the system, we observed more error until we reached a threshold at 43 subjects where we no longer observed a loss in precision as we added more subjects.  
• There are more False Negatives than False Positives.

Results of matching the new normalization:

The data consists of 13 males, 20 females and 13 males, 30 females respectively

33 Subjects (13 M 20 F)

 43 Subjects (13 M 30 F)

Alignment Issue Update

Last week, we present some statistics showing how well fiducial points were matching between samples of the same subjects for the same facial expressions (4 samples only, using a simplified L.O.O). However, after being informed that our fiducial points may not be aligned we were advised to look into this more.

Update:
It turns out the fiducial points are not aligned. Here is a image showing 4 examples of the same subject, same expression, without alignment and with alignment of 8 fiducial points that correspond to raw point clouds (using 3d affine transforms, mapping FP #1, FP #3, FP #4 -> FP #2):

Black: FP #2
Red: FP #1
Green: FP #3
Blue: FP #4
(8 Fiducial Points, Female subject: #1, Expression (Angry): #1, Binghamton Database)




Below is an image of the 2 sets of 83 control points corresponding a point cloud face that was preprocessed before and after they were affine transformed (but of the same subject, same expression, same number of samples):

Red: FP #2
Blue: FP #1
Purple: FP #3
Teal: FP #4




We also re-computed statistics on all the "aligned" points for a 10 females and 10 males where each male/female has 4 samples of 7 expressions, however, we did not have enough time to conjure up presentable data.

Control Point Matching for a Single Subject, Single Expression

Subject 43

Happiness:
4th image of happiness compared against:

1. happy images 1-3 of same subject
2. happy images 1-3 of all other subjects

k=1:

subject 43 score 2

subject 10 score 3

subject 11 score 1

subject 12 score 2

subject 13 score 0

subject 14 score 10

subject 15 score 3

subject 16 score 8

subject 17 score 0

subject 18 score 0

subject 19 score 6

subject 20 score 1

subject 21 score 0

subject 22 score 2

subject 23 score 9

subject 24 score 4

subject 25 score 1

subject 26 score 2

subject 27 score 1

subject 28 score 1

subject 29 score 3

subject 30 score 0

subject 31 score 3

subject 02 score 4

subject 03 score 5

subject 34 score 1

subject 01 score 4

subject 06 score 9

subject 07 score 3

subject 04 score 1

subject 05 score 2

subject 40 score 0

subject 41 score 2

subject 08 score 6

subject 09 score 1

subject 44 score 9

subject 45 score 0

subject 46 score 3

subject 47 score 5

subject 48 score 0

subject 49 score 3

subject 50 score 4

subject 51 score 4

subject 52 score 1

subject 53 score 6

subject 54 score 7

subject 55 score 5

subject 56 score 1

subject 32 score 3

subject 33 score 2

subject 35 score 4

subject 36 score 6

subject 37 score 4

subject 38 score 2

subject 39 score 1

subject 42 score 3

k = 2:

subject 43 score 0
subject 10 score 1
subject 11 score 0
subject 12 score 0
subject 13 score 0
subject 14 score 8
subject 15 score 3
subject 16 score 3
subject 17 score 0
subject 18 score 0
subject 19 score 5
subject 20 score 0
subject 21 score 0
subject 22 score 1
subject 23 score 5
subject 24 score 1
subject 25 score 1
subject 26 score 2
subject 27 score 0
subject 28 score 1
subject 29 score 1
subject 30 score 0
subject 31 score 0
subject 02 score 2
subject 03 score 2
subject 34 score 0
subject 01 score 1
subject 06 score 4
subject 07 score 1
subject 04 score 1
subject 05 score 1
subject 40 score 0
subject 41 score 0
subject 08 score 3
subject 09 score 0
subject 44 score 7
subject 45 score 0
subject 46 score 0
subject 47 score 2
subject 48 score 0
subject 49 score 0
subject 50 score 2
subject 51 score 2
subject 52 score 0
subject 53 score 4
subject 54 score 5
subject 55 score 5
subject 56 score 0
subject 32 score 0
subject 33 score 2
subject 35 score 2
subject 36 score 5
subject 37 score 1
subject 38 score 1
subject 39 score 1
subject 42 score 1

k = 3:

subject 43 score 0
subject 10 score 0
subject 11 score 0
subject 12 score 0
subject 13 score 0
subject 14 score 6
subject 15 score 2
subject 16 score 2
subject 17 score 0
subject 18 score 0
subject 19 score 5
subject 20 score 0
subject 21 score 0
subject 22 score 0
subject 23 score 3
subject 24 score 0
subject 25 score 1
subject 26 score 2
subject 27 score 0
subject 28 score 1
subject 29 score 0
subject 30 score 0
subject 31 score 0
subject 02 score 2
subject 03 score 1
subject 34 score 0
subject 01 score 0
subject 06 score 1
subject 07 score 0
subject 04 score 0
subject 05 score 1
subject 40 score 0
subject 41 score 0
subject 08 score 2
subject 09 score 0
subject 44 score 4
subject 45 score 0
subject 46 score 0
subject 47 score 0
subject 48 score 0
subject 49 score 0
subject 50 score 2
subject 51 score 1
subject 52 score 0
subject 53 score 3
subject 54 score 3
subject 55 score 2
subject 56 score 0
subject 32 score 0
subject 33 score 2
subject 35 score 1
subject 36 score 2
subject 37 score 1
subject 38 score 1
subject 39 score 0
subject 42 score 1
-----------------------------------------------------

Sadness:
4th image of sadness compared against:

1. sad images 1-3 of same subject
2. sad images 1-3 of all other subjects

k = 1:

subject 43 score 3
subject 10 score 2
subject 11 score 2
subject 12 score 7
subject 13 score 1
subject 14 score 9
subject 15 score 2
subject 16 score 1
subject 17 score 0
subject 18 score 0
subject 19 score 0
subject 20 score 2
subject 21 score 1
subject 22 score 2
subject 23 score 11
subject 24 score 5
subject 25 score 2
subject 26 score 1
subject 27 score 3
subject 28 score 2
subject 29 score 1
subject 30 score 1
subject 31 score 5
subject 02 score 2
subject 03 score 5
subject 34 score 0
subject 01 score 0
subject 06 score 6
subject 07 score 3
subject 04 score 3
subject 05 score 1
subject 40 score 1
subject 41 score 5
subject 08 score 3
subject 09 score 4
subject 44 score 6
subject 45 score 6
subject 46 score 3
subject 47 score 5
subject 48 score 1
subject 49 score 3
subject 50 score 4
subject 51 score 3
subject 52 score 0
subject 53 score 12
subject 54 score 7
subject 55 score 0
subject 56 score 2
subject 32 score 2
subject 33 score 6
subject 35 score 2
subject 36 score 2
subject 37 score 1
subject 38 score 0
subject 39 score 2
subject 42 score 1

k = 2:


subject 43 score 2
subject 10 score 9
subject 11 score 7
subject 12 score 1
subject 13 score 5
subject 14 score 7
subject 15 score 1
subject 16 score 7
subject 17 score 1
subject 18 score 7
subject 19 score 6
subject 20 score 3
subject 21 score 5
subject 22 score 2
subject 23 score 7
subject 24 score 1
subject 25 score 2
subject 26 score 7
subject 27 score 5
subject 28 score 1
subject 29 score 16
subject 30 score 5
subject 31 score 1
subject 02 score 10
subject 03 score 7
subject 34 score 4
subject 01 score 3
subject 06 score 9
subject 07 score 4
subject 04 score 3
subject 05 score 4
subject 40 score 1
subject 41 score 1
subject 08 score 8
subject 09 score 5
subject 44 score 1
subject 45 score 5
subject 46 score 6
subject 47 score 12
subject 48 score 5
subject 49 score 4
subject 50 score 7
subject 51 score 7
subject 52 score 6
subject 53 score 4
subject 54 score 5
subject 55 score 9
subject 56 score 8
subject 32 score 11
subject 33 score 11
subject 35 score 8
subject 36 score 5
subject 37 score 5
subject 38 score 11
subject 39 score 2
subject 42 score 0



-----------------------------------------------------

All Emotions:
4th image of Avg Face compared against:

1. avg images 1-3 of same subject
2. avg images 1-3 of all other subjects

k = 20

subject 43 score 2
subject 10 score 9
subject 11 score 7
subject 12 score 1
subject 13 score 5
subject 14 score 7
subject 15 score 1
subject 16 score 6
subject 17 score 1
subject 18 score 7
subject 19 score 5
subject 20 score 4
subject 21 score 5
subject 22 score 0
subject 23 score 4
subject 24 score 4
subject 25 score 1
subject 26 score 3
subject 27 score 1
subject 28 score 0
subject 29 score 11
subject 30 score 5
subject 31 score 2
subject 02 score 6
subject 03 score 9
subject 34 score 5
subject 01 score 3
subject 06 score 9
subject 07 score 4
subject 04 score 4
subject 05 score 4
subject 40 score 0
subject 41 score 1
subject 08 score 7
subject 09 score 4
subject 44 score 0
subject 45 score 2
subject 46 score 5
subject 47 score 6
subject 48 score 4
subject 49 score 4
subject 50 score 6
subject 51 score 5
subject 52 score 6
subject 53 score 4
subject 54 score 7
subject 55 score 6
subject 56 score 6
subject 32 score 4
subject 33 score 6
subject 35 score 8
subject 36 score 4
subject 37 score 4
subject 38 score 9
subject 39 score 0
subject 42 score 0

Control Point Matching: Different Approaches & Results

Proposed Algorithm1
Euclidean Distance:

1. Each Emotion has an average and Standard Deviation

We decided to take the set of feature points for a person's average face.  A person's average face is the pointwise average of all their facial expressions (neutral, angry, disgust, fear, happiness, sadness, and surprise).  Next we calculated the distance between a single subject's neutral expression, and their average expression.  The average distance is 22,955.  Then we calculated the distance between a single subject's neutral expression, and a random subject's neutral expression.  The average distance is 23,426.

To narrow in on accuracy, we next tried Neutral vs Neutral.
Obviously for the same subject the distance was 0.
For one subject vs. a random subject, the distance was 5,553.

It's starting to look like facial expression plays a larger role in face detection than we thought.

Happy VS Happy:

• sameSubject VS sameSubject: 0
• subject1 VS randomSubject: 17,680.857
• mean across 7 trials was 17,680.857
• std deviation across 7 trials was 787.105

From here on, Euclidean Distance & Standard Deviation are about the same for any facial expression except Neutral.

Angry VS Angry:

• subject1 VS randomSubject: 17,993

Disgust VS Disgust:

• subject1 VS randomSubject: 18,418

Fear VS Fear:

• subject1 VS randomSubject: 18,673

Sadness VS Sadness:

• subject1 VS randomSubject: 18,816

Surprise VS Surprise:

• subject1 VS randomSubject: 18,096

Neutral VS Neutral (only one image per subject):

• subject1 VS randomSubject: 5,979
• Here, the mean is: 5,655.85 and the StdD:179.556

(Maybe also compare some results comparing DIFFERENT emotions for SAME subjects)


(Maybe also compare some results comparing DIFFERENT emotions for DIFFERENT subjects)


Proposed Algorithm2
Standard Deviation:

1. Each Control Point has an average and standard deviation

We also looked at combining the average of each control point with the standard deviation, and creating a lower and upper bound based on [mean-(k)stdD, mean+(k)stdD] where k is a variable for us to adjust in order to make a more or less forgiving system.

For subject 1:
Control Point 1:

• mean is [-1.629351072, -43.134416, 49.60222000000001]
• stdD is [ 0.4779777856, 4.3027568, 2.2643159999999978]

Control Point 2:

• mean is [-30.102224, -39.13870000000001, 42.339132]
• stdD is [0.5331751999999994, 4.513219999999998, 2.0564536]

Control Point 3:

• mean is [33.45903199999999, -42.588708, 49.45645599999999]
• stdD is [0.7318063999999985, 4.601218400000002, 2.6821088000000017]

Control Point 4:

• mean is [61.85497199999999, -36.155116, 42.42556799999999]
• stdD is [0.48691439999999775, 4.5829768, 2.818026400000002]

Control Point 5:

• mean is [5.943795720000001, -77.130488, 31.952824000000007]
• stdD is [1.002333744, 4.269662400000001, 2.6875951999999983]

Control Point 6:

• mean is [30.396192000000006, -76.80582400000002, 31.9487]
• stdD is [0.24202159999999892, 3.7177551999999965, 3.0918799999999997]

Control Point 7:

• mean is [18.4926776, -85.00685599999998, 55.480132]
• stdD is [0.7604155200000001, 4.755828800000003, 2.8384736000000004]

Control Point 8:

• mean is [18.6188916, -85.170976, 37.27458399999998]
• stdD is [0.45469831999999993, 4.861604800000001, 2.827723200000004]

The idea now is to iterate over the control points of an incoming face, and check if it falls in the range of [mean +- (k)stdD] for each control point of each subject.  In order to separate our training data from our testing data, we are thinking of using half of a subjects emotions to test, and the other half of the emotions to train.  Hopefully this will avoid the problem of splitting our data into two similar halves.

In order to test, maybe we can test all males VS all females expecting to see a lot of failures


Originally we planned our Training Data VS Testing Data to be half the females & half the males VS the other two halves.  As of right now, our Training Data VS Testing Data is Female VS Male.  Maybe different advantages of both.  Could we use 13 images to train and 13 to test?

Week 4

Normalization of training data complete.  Ready to move onto control point extraction.  Using homogeneous coordinates and rotation matrices to manipulate our data.

Some difficulties included finding the nose tip.  This was not as easy as our paper suggested it would be.  Solved this problem by rotating by a hard-coded amount right now in order to avoid locating the nose tip.  Scaling doesn't seem to be an issue at the moment.

Looking forward, we will need to implement orthogonal distance fitting (plane fitting) and min max filtering for region selection.  We will need to divide the face into rectangular grids in order to perform plane fitting.  Some methods we are looking at include Gauss-Newton algorithm, Steepest Gradient Descent, Levenberg-Marquardt procedure, and simplex method.