Section A

# Three-dimensional Face Recognition based on Feature Points Compression and Expansion

Andy Kyung-yong Yoon1,*,2, Ki-cheul Park2, Sang-min Park2, Duck-kyo Oh2, Hye-young Cho2, Jung-hyuk Jang2, Byounghee Son3
3Dongguk University, Rep. of Korea, diana@dongguk.edu
*Corresponding Author : Andy Kyung-yong Yoon, Professional School of San Martin University, Peru, xperado@usmp.pe

© Copyright 2019 Korea Multimedia Society. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Jun 19, 2019; Accepted: May 24, 2019

Published Online: Jun 30, 2019

## Abstract

Many researchers have attempted to recognize three-dimensional faces using feature points extracted from two-dimensional facial photographs. However, due to the limit of flat photographs, it is very difficult to recognize faces rotated more than 15 degrees from original feature points extracted from the photographs. As such, it is difficult to create an algorithm to recognize faces in multiple angles. In this paper, it is proposed a new algorithm to recognize three-dimensional face recognition based on feature points extracted from a flat photograph. This method divides into six feature point vector zones on the face. Then, the vector value is compressed and expanded according to the rotation angle of the face to recognize the feature points of the face in a three-dimensional form. For this purpose, the average of the compressibility and the expansion rate of the face data of 100 persons by angle and face zone were obtained, and the face angle was estimated by calculating the distance between the middle of the forehead and the tail of the eye. As a result, very improved recognition performance was obtained at 30 degrees of rotated face angle.

Keywords: face recognition; face detection; feature vector; vector compression and expansion; 3D face generation; 3D face recognition

## I. INTRODUCTION

The first step in recognizing a face is to determine if a face exists in the image captured by the camera. If there is a face in the image, the position and size of the face are searched to determine whether or not the face is recognizable. If it is recognizable, it searches for 'eye' as the first step of recognition. Common face detection methods can be classified into a knowledge-based method, a feature-based method, a template-matching method, and an appearance-based method [1].

The knowledge-based method assumes that a person's face consists of two eyes, one nose and mouth, and each face element has a certain distance and position. And it is a method to detect faces considering the relation between these elements. Feature-based methods infer face size and position using face-specific features such as facial features, color, shape, and size [2], [3].

It detects faces through inferred data, and also detects faces through distance between facial elements, position on face, and so on. The template matching method is to create a basic template for the face, and then analyze the inputted face image to create a standard template for the face. Then, the standard template and the input face are compared and detected. The appearance-based method is to detect a face using a model learned by a set of learning images. This method uses statistical numerical values to detect face parts in complex images.

The final goal of this study is to improve the recognition rate of faces at various angles. That is, the face which is even in various angle changing environments can be recognized.

As a method of realizing this technology, it is to implement an algorithm that inputs two-dimensional flat photograph, extracting a feature point, and then recognizes faces rotated in three-dimension using extracted feature point. In order to realize this technology, vector compression and expansion methods are used. That is, it recognizes the face rotated in the up, down, left, and right direction only by the feature points of the flat photograph. That is, the feature point vector of the face is classified into six vector zones, and the rotation angle is obtained by increasing or decreasing the vector zones according to the angle of rotation of the face. In this way, the recognition can be attempted by coinciding with the feature points of the face rotating in three dimensions. By doing this, it recognizes faces rotated in top, bottom, left, and right with only the feature points of a flat photograph.

## II. THE SHAPE OF THE HEAD ANALYSIS

The short-head is usually eyes are higher than the ears, and the distance between the eyes and ears is so close that glasses or sunglasses are not suitable. This is the reason why glasses flow down well. The long head type has lower ears than ears, and the distance between the eyes and ears is long, so the sunglasses are well suited. If an Oriental has a nose plastic surgery, it can be recognized immediately, it because of the difference in the shape of head. In this way, when the nose is raised from the face with short forehead and back, the face harmony is broken [5][6].

A face which is narrow width and long between the forehead and the back is expressed three-dimensional at any angle. Statistically, the face shape that is classified as long -head in Korea is less than 5%, and the long-head is three-dimensional face that everyone feels difference. However, long-head shape is not always nice and it is not always bad for short-head shape [7].

In the face recognition system, the feature points are extracted slightly differently at the long-head and short-head shape. Therefore, there is a significant difference in the variable values of the face zones even in the multi-angle rotated faces of the upper, lower, left, and right sides.

The Westerner’s face has a big three-dimensional than Oriental. Thus, the nose is relatively high, and the eyes are relatively deep. As a result, eyes are dented and bigger than Oriental.

This shape of face is very suitable for extracting feature points, but the useless ratio of feature points is relatively high when the generating a side face’s feature point. The reason is that distinct feature points disappear from the side face. Therefore, it has been proven by data that it is very difficult to generate feature point from the side faces of Westerners.

2.2 Face recognition model

As can be seen in Figure 2, the side faces of Western and Oriental people are distinctly different. The shape of the Westerner’s mouth is a clockwise depression from the nose to the mouth and then coming out of the jaw again. However, the shape of the Oriental’s mouth appears to protrude from the nose to the jaw in the counterclockwise direction [8].

Fig. 2. Side face of Westerner and Oriental.

It is not a problem to recognize these faces with mouth shapes by feature points extracted from front face only. However, it can be seen that an algorithm must be differently applied to Orientals and Westerners in order to recognize side faces or slope faces. In other words, it can be seen that the feature point compression and expansion algorithm is more suitable than the feature point generation algorithm for Oriental's faces, and the feature point generation algorithm has a higher recognition rate on the faces of the Westerners.

## III. FEATURE POINTS COMPRESSION & EXPANSION

3.1 Algorithm Overview

The feature point compression and expansion algorithm is inventive and new algorithm developed while carrying out this study. This algorithm can provide the fastest solution for rotated face recognition. The basic feature of this algorithm is to divide the feature points extracted from the face into 6 zones as shown in Fig. 3.

Fig. 3. Six zones division on the face.

In left-right rotation, compression or expansion is divided into left and right. Thus, six zones are again grouped into two zones, one is grouped into three zones on the left, and the other zone is grouped into three zones on the right.

Therefore, in the left and right rotation, the zone number 1, 3, 5 are bound one group, and the zone number 2, 4, 6 become another group and are compressed or expanded. When rotating up and down, it should be grouped into three zones. At this time, the zone number 1, 2 are bound one group, the zone number 3, 4 are bound second group, and the zone number 5,6 are bound another group.

Each zone is interlocked or separately compressed and expanded. In each zone, extracted feature points are located, and the points are stored as vector values. Compression and expansion are not manipulations on simple position values. This is done by calculating a vector value.

Fig. 4 is an Illustration for the left-right rotation. As shown in the figure, when rotating left, the left zone is compressed and the right zone is expanded. Conversely, when rotated to the right, the right zone is compressed and the left zone is expanded.

Fig. 4. Illustration for left-right rotation.

Fig. 5 is an Illustration for the case of up and down rotation. As shown in the figure, when the head is tilted down, the lower zone is compressed and the upper zone is expanded. The intermediate zone is compressed and expanded selectively.

Fig. 5. Illustration for up-down rotation.

Conversely, when the head is lifted up, the upper zone is compressed and the lower zone is expanded. The intermediate zone is compressed and extended selectively.

3.2 Geometric Properties of Vectors

Since the compression and expansion of vectors are the basic framework of the algorithm, the basic algorithm, the linear operator, has been developed as follows.

3.2.1. Contraction and Dilation on R2

When k is a non-negative scalar,

The liner operator

$T\left(x,y\right)=\left(kx,ky\right)$

is said to be a scaling operator with factor k.

The reason, T (x, y) = (kx, ky) is a linear operator because it is a linear equation with kx, ky.

Here, if 0≤k<1,

the operator T is called contraction.

If k>1,

the operator T is called dilation.

A common geometric feature of contraction and expansion is preserving the direction of the vector. The difference is to reduce the size of the vector in the case of contraction, but to increase the size of the vector in the case of dilation.

3.2.2. Compression and Expansion on R2

For a linear operator T (x, y) = (kx, y),

when k is scalar, not a negative,

If 0≤k <1,

the geometry on R2 is compressed in the x-axis direction.

If k> 1,

the geometry on R2 extends in the x-axis direction.

Therefore, for the linear operator

$T\left(x,y\right)=\left(kx,y\right)$

If 0≤k<1, the operator T is referred to as compression in the x-direction by k,

If k> 1,

the operator T is expanded by k in the x direction (expansion in the x-direction with factor k).

Similarly, when k is scalar, not a negative,

For the linear operator

$T\left(x,y\right)=\left(x,ky\right)$

If 0≤k<1,

the geometry on R2 has the geometric characteristic of being compressed in the y-axis direction,

If k> 1,

the geometry of the figure on R2 extends in the y-axis direction.

Therefore, for the linear operator

$T\left(x,y\right)=\left(x,ky\right)$

If 0≤k<1,

the operator T is referred to as compression in the y direction by k,

If k> 1,

the operator T is said to expand in the y direction by k.

## IV. EXPERIMENT

4.1 Representative Model

One hundred Korean faces were photographed for the experiment of vector compression and expansion algorithm, which is for multi-angle face recognition.

The feature points were extracted based on the captured front face in the photograph, and based on them, the compression and expansion rates, number of useless feature points, and useless feature points rate for the up & down rotation angles of 15° and the left & right rotation angles of 15° and 30° respectively were obtained.

As a result, representative face models were selected, which was the closest to the average of the ratio of each feature point and the useless feature point.

Therefore, the feature points of the representative face model are extracted, and the compression and expansion rates of each angle of up, down, left, and right, the number of feature points that are useless, and the useless ratio are analyzed. The reason for analyzing useless feature points is that they are closely related to the threshold of recognition rate.

Fig. 6 shows the representative face model and extracted feature points. Although the face selected as the representative model is slightly different from the traditional Korean face type, and according to the distribution of the feature points, it shows the future Korean female characteristics face type.

Fig. 6. Representative Face Model.

In addition, the analysis of facial recognition feature point data has distinctive features from Western female and can be positioned as a unique Korean beauty figure.

The total number of feature points of the representative model face is 128, and the number of feature points for each zone and the size of each zone by feature points are shown in Table 1.

Table 1. Feature point information of Face Model.
Zone Horizontal width Vertical width Feature points
1 4.6 3.48 40
2 4.19 3.48 39
3 6.26 2.60 12
4 5.09 2.41 12
5 5.26 4.68 14
6 4.26 4.89 11
Total 128
4.2 Feature Points Compression & Expansion

A total of 128 feature points are somewhat larger than the average feature points of Koreans. The reason seems to be that the outline of the face is futuristic, and it possesses a clear aspect ratio. The reason why the number of feature points is evenly distributed in the right and left sections is the data proving that the face is symmetrical. The reason why the feature points are distributed widely in the zone 1, 2 is analyzed because it is a distinctive shape.

Fig. 7 shows the representative model’s multi-angle. Fig. 8 shows extracted feature point that is replaced with the face angle of the representative model in Fig. 7. Therefore, in Fig. 8, the two left feature points show rotation angles of 30 ° and 15 °, the right feature points show the rotation angles of 15 ° and 30 °, and the up & down rotation angles are 15 °. Fig. 8 shows the state of compression and expansion in the direction of rotation from the original extracted feature point which located in the center.

Fig. 7. Representative Model’s Multi-angle.
Fig. 8. Representative Model’s Multi-angle Feature point.

Yellow is the feature point within the threshold, and red is a useless feature point. If the total number of feature point does not exceed 35% of useless feature points, there is no problem recognizing the face.

Table 2 and Fig. 9 shows the number of useless feature points for each zone. Table 3 and Fig. 10 shows the distribution of compression and expansion rates by zone.

Table 2. Useless ratio of Feature points.
Zone L-15 L-30 R-15 R-30 D U
1 22.5 27.5 22.5 27.5 37.5 30
2 23.1 28.2 25.6 28.2 35.9 33.3
3 25 50 33.3 33.3 16.7 25
4 16.7 50 50 33.3 8.3 33.3
5 28.6 28.6 21.4 35.7 35.7 28.6
6 27.3 27.3 27.3 36.4 36.4 36.4
Fig. 9. Useless Feature Point rate.
Table 3. Compression & Expansion rate distribution
Zone L-15 L-30 R-15 R-30 D U
1 (23.5) (28.5) 16.5 17.4 8.6 11.2
2 10.0 16.5 (19.1) (22.0) 6.6 9.8
3 (35.9) (43.1) 8.6 14.7 11.2 21.5
4 17.1 26.5 (12.0) (22.0) 4.1 19.5
5 (38.4) (41.4) 20.9 26.6 (9.2) 12.0
6 22.5 33.8 (27.7) (38.5) (7.4) 16.4
Fig. 10. Compression & Expansion rate distribution.

In this study, it was tried to find the compression and expansion mean values according to the face rotation angle. Therefore, the average value of 100 persons face was investigated in order to obtain the average compression and expansion rate in up, down, left and right rotation.

It was found that the compression and expansion rates vary widely depending on the width and length of the face. The average value showed a large deviation according to the size of the face, and it was not a meaningful value.

However, the result obtained incidentally was that the expansion rate on the opposite side of the zone to be compressed was 10 to 15% smaller on average than the compression rate.

That is, if the compression rate of the zone 1 is 30%, the expansion rate of the zone 2 is 25 ~ 27% on average. That is, the compression ratio and the expansion rate are not proportional.

Based on these results, further study is needed to recognize the multi-angle rotation faces by this vector compression and expansion algorithms. It is needed to find the average parameter for the compression and expansion rate according to face width and length.

## V. CONCLUSION

Recognizing faces rotated by multiple angles using only feature points extracted from a flat photo is one of the most difficult assignments. Various algorithms have been devised and attempted to solve this assignment.

Especially, in this study, it was tried to estimate and convert the three-dimensional shape by adjusting the interval of the feature points by the rotation angle of the face.

Since the feature points can’t be adjusted one by one, they are grouped and the feature point intervals are adjusted via group adjustment.

The compression and expansion rates determined by the face rotation angle. However, this value was found to vary widely depending on the width and length of the face. Therefore, it is considered that the average value obtained from 100 face data is not enough to be used as the average parameter of this algorithm.

In order to realize effective three-dimensional face recognition using this algorithm, more face feature point data should be analyzed and established.

Through this study, it was obtained a basic algorithm that can recognize three-dimensional faces through vector compression and expansion. Further studies should continue to attempt to obtain an average parameter. If the average parameter is obtained, the face is considered to be recognized at a rotation angle larger than the current maximum recognizable face rotation angle.

## REFERENCES

[1].

M.Akhil jabbar, B.L Deekshatulua, Priti Chandra, “Classification of Heart Disease Using K- Nearest Neighbor and Genetic Algorithm” Procedia Technology 10 ( 2013 ) 85 – 94.

[2].

Mohammed Hazim Alkawaz, Dzulkifli Mohamad, Ahmad Hoirul Basori, Tanzila Saba, "Blend Shape Interpolation and FACS for Realistic Avatar" Springer 3D Res (2015) 6:6.

[3].

Henry A. Rowley, Shumeet Baluja, Takeo Kanade, “Neural Network Based Face Detection” Computer Vision and Pattern Recognition, 1996, Carnegie Mellon University.

[4].

Cheol-woong Lee, Ilmin Kim, Sea-Hong Cho, “Designing and Implementing 3D virtual Face Aesthetic Surgery System”, Journal of Digital Contents Society Vol. 9 No. 4 Dec. 2008(pp. 751-758).

[5].

Yi Jung-Hoon, Lee Chan, Rhee Phill-Kyu, “A Mapping Algorithm for Real Time Animation Based in Facial Features”, The Journal of the Korea information Processing Society Vol.7 No.2,pp. 919-922, 2000.

[6].

Dong-Hoon Kim, Dae-Kyu Shin, Min-Young Eum, Hyun-Sool Kim, Sang-Hui Park, "Face Recognition using Face Region Information in the Dynamic Link Architecture”, The KIEE Summer Conference 2002, pp 2583-2585. Jul 2002.

[7].

Gi-Young Go, Doo-Young Kim, "Rotation and Scale Invariant Face Detection Using Log-polar Mapping and Face Features”," The Journal of The Korea Institute of Convergence Signal Processing, Vol 6, No. 1, pp-15-22, Jan. 2005.

[8].

Kyunghee Lee, Hyeran Gyun, Chansup Chung, “Facial Feature Extraction for Face and Facial Expression Recognition”, The Journal of Korean Society for Emotion and Sensibility, pp. 25-29, 1998.

## Authors

Andy Kyung-yong Yoon

Andy Kyung-yong Yoon completed his MS and Ph.D. degree from the Yonsei University, Korea. He is currently a Professor in the Professional school of Electronic Engineering at San Martin University, Peru, and also a CTO of NEOSECU, YAP Company, Korea where his main activities include research, undergraduate and postgraduate training. His research interests include Mobile Agent Systems and AI related O2O application, and Indoor Positioning System.

Ki-cheul Park

Ki-cheul Park graduated from Kwangwoon University and a bachelor of electronics engineering. He participated in the TV development project of Samsung Electronics and is currently CEO of Neosecu Co., Ltd. in Korea.

Duck-kyo Oh

Duck-kyo Oh graduated from Kwangwoon University and a bachelor of computer science and engineering. He served as General Manager of Beijing Inca Co., Ltd., and is currently the director of development at Neosecu Co., Ltd. in Korea.

Sang-min Park

Sang-min Park graduated from Kwangwoon University in Korea and studied Electronics at University of Nevada, Las Vegas and a bachelor of electronic & communication engineering. He is currently an intern at the Neosecu Co., Ltd. In Korea.

Hye-young Cho

Hye-young Cho graduated from Dongmyung University and a bachelor of computer science and engineering. She is currently head of development at Neosecu Co., Ltd in Korea.

Jung-hyuk Jang

Jung-hyuk Jang graduated from Semyung University and a bachelor of electronics engineering. He is currently a researcher at the Neosecu Co., Ltd. in Korea.

Byounghee Son

Byounghee Son completed her MS and Ph.D. degree from the Yonsei University, Korea. She is currently a Professor in the Division of Computer, Information Communication Engineering at Dongguk University, Korea. She is interested in AI and IoT solution.