Control Point Extraction

The Final Control Points:

Control Points were extracted in this order:

The order is important because the accuracy of points 3-5 is dependent on point 2.  All points are dependent of point 1.

1. Nose Tip
2. Nose Width Points (Left & Right)
3. Inner Eye Corner (Left & Right)
4. Outer Eye Corner (Left & Right)
5. Mouth Corners (Left & Right)

The Control Points were extracted using different methods:

Nose Tip - The greatest z-axis value

Nose Width - A search area is defined where the Control Point is expected to be.  The Control Point is determined by finding the steepest curve.

InnerEye - A search area is defined directly above the Nose Width Point.  The point with the lowest Z-Coordinate is determined to be the Control Point

OuterEye - The distance between the two InnerEye points is added to each control Point to determine the OuterEye point.

MouthCorner - Half the distance between from the InnerEye to the OuterEye is added to the InnerEye.  A straight line is drawn downward in the y-direction to define a new search area.  Within this region, the point with the lowest z-value is estimated to be the final control point.

New Approaches to Normalization

The first thing we did was to change our normalization algorithm.  Last week, we translated, rotated, scaled, and sheared each face so as to align them all on top of another.  This week, we undid the shearing.  Instead, 

• We computed the SVD of the Affine Matrix ( A = U * S * V')
• We found the estimated rigid rotation matrix R = U * V' (excluding S, which is the scaling component of equation). 

A. Subject: Male #1, Expression: Surprised, Sample #1 - BLACK

B. Subject: Male #1, Expression: Surprised, Sample #4 - BLUE

(B is transformed using R and translation to A)

Here are the results after running the Matching against the new normalization methods.  

• With 23 subjects or less, we achieved perfect sensitivity vs specificity. 
•  As we increased the number of subjects into the system, we observed more error until we reached a threshold at 43 subjects where we no longer observed a loss in precision as we added more subjects.  
• There are more False Negatives than False Positives.

Results of matching the new normalization:

The data consists of 13 males, 20 females and 13 males, 30 females respectively

33 Subjects (13 M 20 F)

 43 Subjects (13 M 30 F)

Alignment Issue Update

Last week, we present some statistics showing how well fiducial points were matching between samples of the same subjects for the same facial expressions (4 samples only, using a simplified L.O.O). However, after being informed that our fiducial points may not be aligned we were advised to look into this more.

Update:
It turns out the fiducial points are not aligned. Here is a image showing 4 examples of the same subject, same expression, without alignment and with alignment of 8 fiducial points that correspond to raw point clouds (using 3d affine transforms, mapping FP #1, FP #3, FP #4 -> FP #2):

Black: FP #2
Red: FP #1
Green: FP #3
Blue: FP #4
(8 Fiducial Points, Female subject: #1, Expression (Angry): #1, Binghamton Database)




Below is an image of the 2 sets of 83 control points corresponding a point cloud face that was preprocessed before and after they were affine transformed (but of the same subject, same expression, same number of samples):

Red: FP #2
Blue: FP #1
Purple: FP #3
Teal: FP #4




We also re-computed statistics on all the "aligned" points for a 10 females and 10 males where each male/female has 4 samples of 7 expressions, however, we did not have enough time to conjure up presentable data.

Control Point Matching for a Single Subject, Single Expression

Subject 43

Happiness:
4th image of happiness compared against:

1. happy images 1-3 of same subject
2. happy images 1-3 of all other subjects

k=1:

subject 43 score 2

subject 10 score 3

subject 11 score 1

subject 12 score 2

subject 13 score 0

subject 14 score 10

subject 15 score 3

subject 16 score 8

subject 17 score 0

subject 18 score 0

subject 19 score 6

subject 20 score 1

subject 21 score 0

subject 22 score 2

subject 23 score 9

subject 24 score 4

subject 25 score 1

subject 26 score 2

subject 27 score 1

subject 28 score 1

subject 29 score 3

subject 30 score 0

subject 31 score 3

subject 02 score 4

subject 03 score 5

subject 34 score 1

subject 01 score 4

subject 06 score 9

subject 07 score 3

subject 04 score 1

subject 05 score 2

subject 40 score 0

subject 41 score 2

subject 08 score 6

subject 09 score 1

subject 44 score 9

subject 45 score 0

subject 46 score 3

subject 47 score 5

subject 48 score 0

subject 49 score 3

subject 50 score 4

subject 51 score 4

subject 52 score 1

subject 53 score 6

subject 54 score 7

subject 55 score 5

subject 56 score 1

subject 32 score 3

subject 33 score 2

subject 35 score 4

subject 36 score 6

subject 37 score 4

subject 38 score 2

subject 39 score 1

subject 42 score 3

k = 2:

subject 43 score 0
subject 10 score 1
subject 11 score 0
subject 12 score 0
subject 13 score 0
subject 14 score 8
subject 15 score 3
subject 16 score 3
subject 17 score 0
subject 18 score 0
subject 19 score 5
subject 20 score 0
subject 21 score 0
subject 22 score 1
subject 23 score 5
subject 24 score 1
subject 25 score 1
subject 26 score 2
subject 27 score 0
subject 28 score 1
subject 29 score 1
subject 30 score 0
subject 31 score 0
subject 02 score 2
subject 03 score 2
subject 34 score 0
subject 01 score 1
subject 06 score 4
subject 07 score 1
subject 04 score 1
subject 05 score 1
subject 40 score 0
subject 41 score 0
subject 08 score 3
subject 09 score 0
subject 44 score 7
subject 45 score 0
subject 46 score 0
subject 47 score 2
subject 48 score 0
subject 49 score 0
subject 50 score 2
subject 51 score 2
subject 52 score 0
subject 53 score 4
subject 54 score 5
subject 55 score 5
subject 56 score 0
subject 32 score 0
subject 33 score 2
subject 35 score 2
subject 36 score 5
subject 37 score 1
subject 38 score 1
subject 39 score 1
subject 42 score 1

k = 3:

subject 43 score 0
subject 10 score 0
subject 11 score 0
subject 12 score 0
subject 13 score 0
subject 14 score 6
subject 15 score 2
subject 16 score 2
subject 17 score 0
subject 18 score 0
subject 19 score 5
subject 20 score 0
subject 21 score 0
subject 22 score 0
subject 23 score 3
subject 24 score 0
subject 25 score 1
subject 26 score 2
subject 27 score 0
subject 28 score 1
subject 29 score 0
subject 30 score 0
subject 31 score 0
subject 02 score 2
subject 03 score 1
subject 34 score 0
subject 01 score 0
subject 06 score 1
subject 07 score 0
subject 04 score 0
subject 05 score 1
subject 40 score 0
subject 41 score 0
subject 08 score 2
subject 09 score 0
subject 44 score 4
subject 45 score 0
subject 46 score 0
subject 47 score 0
subject 48 score 0
subject 49 score 0
subject 50 score 2
subject 51 score 1
subject 52 score 0
subject 53 score 3
subject 54 score 3
subject 55 score 2
subject 56 score 0
subject 32 score 0
subject 33 score 2
subject 35 score 1
subject 36 score 2
subject 37 score 1
subject 38 score 1
subject 39 score 0
subject 42 score 1
-----------------------------------------------------

Sadness:
4th image of sadness compared against:

1. sad images 1-3 of same subject
2. sad images 1-3 of all other subjects

k = 1:

subject 43 score 3
subject 10 score 2
subject 11 score 2
subject 12 score 7
subject 13 score 1
subject 14 score 9
subject 15 score 2
subject 16 score 1
subject 17 score 0
subject 18 score 0
subject 19 score 0
subject 20 score 2
subject 21 score 1
subject 22 score 2
subject 23 score 11
subject 24 score 5
subject 25 score 2
subject 26 score 1
subject 27 score 3
subject 28 score 2
subject 29 score 1
subject 30 score 1
subject 31 score 5
subject 02 score 2
subject 03 score 5
subject 34 score 0
subject 01 score 0
subject 06 score 6
subject 07 score 3
subject 04 score 3
subject 05 score 1
subject 40 score 1
subject 41 score 5
subject 08 score 3
subject 09 score 4
subject 44 score 6
subject 45 score 6
subject 46 score 3
subject 47 score 5
subject 48 score 1
subject 49 score 3
subject 50 score 4
subject 51 score 3
subject 52 score 0
subject 53 score 12
subject 54 score 7
subject 55 score 0
subject 56 score 2
subject 32 score 2
subject 33 score 6
subject 35 score 2
subject 36 score 2
subject 37 score 1
subject 38 score 0
subject 39 score 2
subject 42 score 1

k = 2:


subject 43 score 2
subject 10 score 9
subject 11 score 7
subject 12 score 1
subject 13 score 5
subject 14 score 7
subject 15 score 1
subject 16 score 7
subject 17 score 1
subject 18 score 7
subject 19 score 6
subject 20 score 3
subject 21 score 5
subject 22 score 2
subject 23 score 7
subject 24 score 1
subject 25 score 2
subject 26 score 7
subject 27 score 5
subject 28 score 1
subject 29 score 16
subject 30 score 5
subject 31 score 1
subject 02 score 10
subject 03 score 7
subject 34 score 4
subject 01 score 3
subject 06 score 9
subject 07 score 4
subject 04 score 3
subject 05 score 4
subject 40 score 1
subject 41 score 1
subject 08 score 8
subject 09 score 5
subject 44 score 1
subject 45 score 5
subject 46 score 6
subject 47 score 12
subject 48 score 5
subject 49 score 4
subject 50 score 7
subject 51 score 7
subject 52 score 6
subject 53 score 4
subject 54 score 5
subject 55 score 9
subject 56 score 8
subject 32 score 11
subject 33 score 11
subject 35 score 8
subject 36 score 5
subject 37 score 5
subject 38 score 11
subject 39 score 2
subject 42 score 0



-----------------------------------------------------

All Emotions:
4th image of Avg Face compared against:

1. avg images 1-3 of same subject
2. avg images 1-3 of all other subjects

k = 20

subject 43 score 2
subject 10 score 9
subject 11 score 7
subject 12 score 1
subject 13 score 5
subject 14 score 7
subject 15 score 1
subject 16 score 6
subject 17 score 1
subject 18 score 7
subject 19 score 5
subject 20 score 4
subject 21 score 5
subject 22 score 0
subject 23 score 4
subject 24 score 4
subject 25 score 1
subject 26 score 3
subject 27 score 1
subject 28 score 0
subject 29 score 11
subject 30 score 5
subject 31 score 2
subject 02 score 6
subject 03 score 9
subject 34 score 5
subject 01 score 3
subject 06 score 9
subject 07 score 4
subject 04 score 4
subject 05 score 4
subject 40 score 0
subject 41 score 1
subject 08 score 7
subject 09 score 4
subject 44 score 0
subject 45 score 2
subject 46 score 5
subject 47 score 6
subject 48 score 4
subject 49 score 4
subject 50 score 6
subject 51 score 5
subject 52 score 6
subject 53 score 4
subject 54 score 7
subject 55 score 6
subject 56 score 6
subject 32 score 4
subject 33 score 6
subject 35 score 8
subject 36 score 4
subject 37 score 4
subject 38 score 9
subject 39 score 0
subject 42 score 0