YAN Limin，DU Bin，GUO Qiang，PAN Hao

?

Recognize the Tilt Fingertips by Partial Scan Algorithm

YAN Limina,b，DU Bina,b，GUO Qianga,b，PAN Haoa,b

( a. Key Laboratory of Advanced Displays and System Application, Ministry of Education,b. Microelectronics Research and Development Center, Shanghai University, Shanghai 200072, China )

To meet the needs of finger interaction system and the recognition of tilt fingertips, a fast and accurate method is described to detect the position of fingertips and real-time recognize tilt fingertip gesture. This method uses YCbCr color-space-segmentation-algorithm to devise complexion roughly. After that, preprocessing regions are segmented carefully by the concept of "proportion of perimeter and acreage" to eliminate interference region other than the hand skin color. In addition, obtain the finger profile by binomial method of least squares fitting algorithm. Besides, the improved convex algorithms are employed to complete the fingertips detection and correct tilt angle. Finally, fingertips are recognized by means of partial scan method. Experimental results demonstrate that the proposed method can realize the recognition refers to easy tilt fingertips of zero to night, and reach the high recognition rate of 95.7%. Apart from this, stable performance are achieved．

fingertips recognition; complexion; position the fingertips; convex; partial scan

0 Introduction

Hands are the most flexible muscles of the human body, and the fingers are the most important part of it. They can help us to complete daily work. In addition, rich information can be achieved. Finger is a humanized and free media which can communicate directly with computer. Human-computer interaction of fingers based on computer vision technology is one of indispensable methods to realize emerging human-computer interaction technology[1].

Currently, most researchers put the focus away from identification of body, arms and other large-scale, and concentrate on a more detailed finger gesture recognition. Yang Bo[2]and etc. obtain the relative spacing and spatial distribution of the relative density about the knuckles to identify the finger. They improve the recognition rate about only a small numbers of finger gestures; Lin Qiangshui[3]and etc., who collected the information of geometrical feature analysis, use a random decision tree classification, but the real time performance is not good enough.

Foreign researchers research on the field for further, and a large number of classic algorithms are proposed. The Oka K[4]designed the interactive desktop system with the infrared thermal imager, that senses the temperature of manpower in order to carry out segmentation and tracking with the limits of the uncontrolled temperature sensitivity. Ren[5]and others analyzed the fingers[6]through the establishment of three-dimensional hand image. Nevertheless, the large amount of calculation couldn’t meet the requirement of real-time.

In addition, researchers at home and abroad conduct meticulous researches under the condition of facing towards the camera. However, in daily life, a certain angle the fingertips leaned is closer to the actual situation. The existing algorithm of tilt fingers’ recognition rate is extremely low. To solve this problem, this paper repairs the tilt fingers after identifying the fingers. That can greatly improve the recognition rate, and are moresuitable with the habits of everyday life.

This paper includes the following four parts: coarse segmentation with skin color, region segmentation, finger detection and tilt correction, and finger gesture analysis. In this paper, the algorithm greatly reduces the computation complexity and also enhances the efficiency of the algorithm for real-time processing.

1 Skin Color Coarse Segmentation

The coarse segmentation is the process of extracting the background and the skin color region of the RGB image obtained from the camera, and extracts the parts of the hand according to the morphology. The collected video frames match with YCbCr model, a model of skin segmentation.

YCbCr model can effectively solve the problem of separation of skin color region, which greatly overcome the problems of separated skin points and separating the color of skin incompletely. In addition, the value of Y, Cb, Cr can be respectively composed of R, G, B value through linear transformation with high computing efficiency.

Usually, the region can be considered a normal skin color area when 77≤b≤127, 133≤r≤173[7]. In this paper, to capture the skin color information more accurately, the threshold region is 85≤b≤118, 145≤r≤168, on the basis of the adaptive threshold adjustment.

2 Detailed Region Segmentation

Detailed region segmentation aims to completely separate fingers and hands away from other skin regions, and obtain accurate information of fingers and hands eventually. Calculate area ratio of each connected area for the proportion of the total skin. If the percentage is less than 10%, the connected area is judged to be interference. Skin color region will change in a certain degree. In this paper, remove noise interference through corrosion, expansion, median and Gaussian filter, and conduct smoothing preprocessing to get effective skin region.

According to “Proportion of perimeter and acreage” concept in article [8], that means the ratio of the circumference and the area about connected region. Among them, the bigger ratio shows that the larger the ductility. In the actual test, the appropriate proportion can exactly identify the hand area in skin region as well as the face and etc. In order to avoid floating deviation by the distance with camera, they use the C2/S value as a screening threshold. The appropriate threshold of [24.6, 33.3] can separate of the hand effectively by a large number of experiments, shown in figure 1(a). Then, obtain the binary figure by the hand, shown in Figure 1(b).

The Binomial method of least squares fitting algorithm[9]eliminates the interference and influence of arbitrary angle between the palm and fingers, which can get a more stable profile and better location about fingertips. The process results are shown in figure 1(c).

Fig.1 Hand region division

3 Fingertip Positioning and Tilt Correction

Fingertip position and image tilt correction are the important parts. The accuracy of the fingertip is related to the image's tilt correction, which ultimately affects the results of the analysis.

The separation of the fingers, fingertips and other areas is the basis for their accurate identification and tracking. According to different test conditions, algorithms can be roughly divided into three main algorithms[10]: convexity-defects-based(CDB) algorithm, center-of-mass-based(CMB) algorithm and geometry-based(GB) algorithm.

In the calculation of the corresponding hand convex envelope, process the points on the convex envelope, and further filtrate and exclude the interference based on the CDB algorithm. The processing time of the algorithm is too short to be ignored, which can also be more accurate to identify the fingertip. The improved CDB algorithm has good performance and good stability, and can greatly improve the accuracy of the identification of potential.

3.1Convex Envelope

Analysis found that the convex envelope algorithm can effectively solve the closed problem of curve’s distribution. As a subset of the plane convex (), If and only if for any two points,∈convex (), the line segmentare fully contained in the convex ()[11], as shown in figure 2.

Fig.2 Schematic of the convex envelope

For a finite number of points in a given plane, then they can find out the set of minimum point connected into a convex polygon. After that, the several points are within the polygon or on the line segment of polygon. Accordingly, this convex polygon is convex envelope of the given points.

3.2Improved Convex Envelope Algorithm

As shown in figure 3, for a group of points set, calculate the only convex (). All vertexes of the minimum convex polygon stem from set. Generally, fingers are several endpoints farthermost from the palm. So, come up with convex envelope endpoint or kick point, which includes the candidate points of the fingertips. Specific process is as follows:

Step1: All points in setare ordered from small to large based on the value of-coordinate. Select the smallest point fromcoordinates as the reference point0. If it appears several points of samevalue, then select the point with the minimum value as corresponding reference point0;

Step2: Number the angles from small to large between the horizontalaxis andunder the conditions of counterclockwise traversal scan. Shown in figure 3,D

Step3:0and1are the initial points, therefore themust be on the line segment of convex envelope. For given counterclockwise scanning, if the cross-product of theandis negative, in other words, the direction of the rotation is consistent with the scan direction,Pwill be ruled out. If the correspondingPandP-1has been excluded, substitute them withP-2,P-3and so on in turn postpone. Finally, determine whether point is on the convex envelope one by one;

Step4: The11must be on the convex envelope, too. Then connect these points shown in figure 3.

In figure 3, when considering2, the cross-product ofandis positive.is clockwise rotation relative to, contrary to given counterclockwise scanning direction. So1is retained, which means1is an endpoint. Then, what about3, the cross-product ofandis negative, therefore2is ruled out. About4, by the judge before, the2has been removed, so the judgment of cross-product ofandinstead ofand. Obviously, thus3is reserved for endpoint. Accordingly, obtain all the endpoints on the convex hull, polygon {0,1,3,6,9,11}, as shown in figure 3.

Fig.3 Convex envelope algorithm

This algorithm processes a frame image focusing on area of hands. The results are shown in figure 4 (a) and (b). Figure 4 (a) shows, simple convex envelope algorithm contains fingertips within the endpoint, but still can’t rule out the other non-fingertips endpoint. Figure 4 (b) shows, all the vertices of convex are open hands’ finger points. This paper improves the convex envelope algorithm using “curvature threshold value method” for the secondary screening. Specific steps are as follows:

Fig.4 Convex algorithm results and improved convex algorithm results

Step1: Define that the points of set, which are both on the palm outline and on the convex envelope. About all thePpoints on set, attain theandby respectively connecting pointandof its left and right sides near the hand on the contour;

Step2: Set 90° as the preset thresholdfor more accuracy results and remarkable effects. Calculate the anglebetween vectorand, if＜, the points are fingertips. Otherwise, these points are rejected out of set. Eventually, fingertip points and the other two wrist endpoints are achieved.

Figure 4(c), (d) are the results by improved convex algorithm of fingertips’ convex on (a) and (b). In the figure,,andare the fingertips, and pointsand￠are not. After calculating, all points on convex envelope are meet the set threshold to get fingertips on the convex envelope.

3.3Detection and Rotation of Angle of Inclination

Measure the center of the handin the image center of gravity to establish-axis and-axis. Convex envelope algorithm can be relatively easy to obtain each digital endpoint in the frame image, namely, the fingertip. Connect theand fingers and get polar angle1,2,3, …,ζ, where=1, 2, 3, 4, 5. Based on the average valueof the polar angle,rotate the image by formula (2):

In figure 5(a), clockwise direction is positive and they process the image rotation for (-90°). Then rotate the image that faces the camera. To facilitate the next scan, the results shows in figure 5(b).

Fig.5 Schematic of tilt angle detecting

4 Recognition Algorithm of Fingers’ Scan Method

Palm rectangular area can be obtained by the regional fine splitting and recorded its four endpoint pixel coordinates(1,1),(2,2),(3,3) and(4,4), showed in figure 6(a). Therefore, the hand’s width is equal to=(2-1)and high is equal to= (1-3). At the moment, record the pixels(x,y) on the left line. By this way, it can greatly reduce the scanning area, the corresponding amount of computation and improve the recognition rate on a rectangular frame.

Fig.6 Schematic of partial scanning method.

Algorithm provides that each scan begin from line, and divides the rectangular frame of hands into 20 equal portions. So the algorithm scans from (1,1-×/20) to (2,1-×/20) line by line. The variableis set to 0 at the beginning. And then, the variableis incremented by 1 after each scan. According to morphology, golden Ratio of finger length accounted for the entire palm is 1:2.44. Scanning range is defined as 70%×for reducing the relative scanning, namely,≤14. Minimum ordinate of the scanning lines is1-14×/20, as shown as figure 6(a). This threshold can reduce the computation time with more accurate recognition. Procedure is as follows:

Step1: Initialize FingerNum to 0 and scan the Rectangle progressively. When the scan line is tangent to the contour or both of them are intersected and it appears intersect pixels in pairs, the FingerNum+1. What’s more, record the width of intersections in pares and store in a two-dimensional array of Count[][], Whereis theth finger scanned from top to bottom;

Step2: Progressive line scan down to judge the size of the finger width. If Count[][]>2.5×Count[][1], stop scan and record thethfingers’ width. The width is too large to exceed the base of the finger and enter into the palm. Else, saved the data Count[][];

Step3: Calculate new average finger width Count[] by each finger recorded Count[][]. Calculation the value of Count[]/Count[1]. In case of Count[]/Count[1]≥1.5，that means two fingers are closed to each other. Then, FingerNum+1, as figure 6(b) shows;

Step4: If FingerNum = 1, there may occur two values, that are 1 and 7. If Count[]>/3, FingerNum=7, otherwise FingerNum=1;

Step5: If FingerNum=2, 2, 6, 8 and 9 may be the values. And ify>1-0.5×in Left_M，this implies that thumbs have appeared. Assumed that fingertip distance is (2-1), it affirms that FingerNum=6, otherwise, FingerNum=8. When the thumb is not extended, if theof Count[1][] and Count[2][] are equal with each other, FingerNum=2, otherwise FingerNum=9.

5 Results and Analysis

Experiments of this article is configured as desktop with Intel (R) Core (TM) i3-4160 processor, the 8G Kingston DDR3 memory and Windows 7 64-bit operating system. The article creates different finger image library, which can recognize simple finger number actions 0~9.

Figure 7 shows legend of digital libraries 0~9, and experimental results. Final recognition effect will be shown through the static recognition rate and run-time properties.

Fig.7 Self dataset

Table 1 displays results of 0~9 static finger gestures recognition by the use of this algorithm. The experiments sample the data from ten different people by ten any direction of gestures. The average recognition rate is capable of coming to 95.7% of which the recognition rate of 1 is 100%.

Table 1 Gesture recognition by article’s algorithm

Table 2 shows that performance comparison between the algorithms. We can see,recognition rate and the running time has a more significant advantage with great improvement using the proposed algorithm. At the same time, the running time only adds an additional 0.1 ms within float range of ±10%. Consequently,optimization algorithm of this article has considerable advantages in terms of recognition rate and run time, and can accurately identify any directions for the camera.

Table 2 Performance comparison with traditional algorithms of [10]

6 Conclusion

In this paper, based on the four parts: coarse segmentation with skin color, region segmentation, finger detection and tilt correction, and finger gesture analysis, paper achieves good results upon independent data. Moreover, tilt finger is close to the daily life and the actual situation that is practical significance. Compared with the traditional algorithm, algorithm in this article has more obvious advantages, for instance, improving the recognition rate, speeding up the whole processing, enabling more accurate static gesture recognition and further promoting the exploration and research trends referring to gesture recognition with good portability.

[1] LIU Lin，GENG Junmei，GU Guohua，. Human Detection Based on Contour Features and Neural Networks [J]. Opto-Electronic Engineering(S1003-501X)，2014，41(7)：50-56.

[2] YANG Bo，SONG Xiaona，F(xiàn)ENG Zhiwide，.Gesture Recognition in Complex Background Base on Distribution Features of Hand [J]. Journal of Computer Aided Design and Computer Graphics(S1003-9775)，2010，22(10)：1841-1848.

[3] LIN Shuiqiang，WU Yadong，CHEN Yonghui. Gesture Recognition Method Based on geometric Characteristics [J]. Computer Engineering and Design(S1000-7024)，2014，35(2)：636-640.

[4] Oka K，Sato Y，Koike H. Real-time Fingertip Tracking and Gesture Recognition [J]. IEEE Computer Graphics and Applications(S0272-1716)，2002，22(6)：64-71.

[5] REN Zhou，YUAN Junsong，ZHANG Zhengyou. Robust Hand Gesture Recognition Based on Finger-earth Mover's Distance with a Commodity Depth Camera [C]// Proceedings of the 19th ACM international conference on Multimedia，ACM，2011：1093-1096.

[6] YU Bo，CHEN Yongqiang，WANG Shuangyi，. Static Gesture Recognition Algorithm Based on Characteristics of Finger Angle [J]. Journal of Xihua University: Natural Science(S1673-159X)，2014，33(1)：69-71.

[7] Rahman M A，Purnama E，Purnomo M H.Simple method of human skin detection using HSV and YCbCr color spaces [C]// IEEE International Conference on Intelligent Autonomous Agents, Networks and Systems，2014：58-61.

[8] XU Xuyan，ZHANG Qi，WU Xia. Reserarrch of Algorithm For Real-time Gesture Recognition Based on OpenCV [J]. Information Technology(S1009-2552)，2013，4(4)：99-102.

[9] LIU Fu，LIU Huiying，GAO Lei，. Hand Shape Recognition Based on Fusion Features of Fingers and Particle Swarm Optimization [J]. Optics and Precision Engineering(S1004-924X)，2015，23(6)：1774-1782.

[10] Maisto M，Panella M，Liparulo L，. An accurate algorithm for the identification of fingertips using an RGB-D camera [J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems(S2156-3357)，2013，3(2)：272-283.

[11] NI Jinsong，WANG Yihuai. Algorithm Study for Closed Principle Curves Learning Which Starts with Convex Envelope Hull [J]. Computer Science(S1002-137X)，2007，34(4)：166-170.

基于局部掃描法對(duì)傾斜指勢(shì)的識(shí)別

嚴(yán)利民a，b，杜 斌a，b，郭 強(qiáng)a，b，潘 浩a，b

( 上海大學(xué) a. 新型顯示技術(shù)及應(yīng)用集成教育部重點(diǎn)實(shí)驗(yàn)室；b. 微電子研究與開(kāi)發(fā)中心，上海 200072 )

為滿足手指交互系統(tǒng)的需要，并能夠達(dá)到對(duì)傾斜指勢(shì)進(jìn)行準(zhǔn)確識(shí)別的要求，本文介紹了一種快速、準(zhǔn)確對(duì)指尖檢測(cè)定位并實(shí)時(shí)識(shí)別傾斜指勢(shì)的方法。該方法利用YCbCr顏色空間分割算法對(duì)膚色聚類進(jìn)行粗分割，然后運(yùn)用“周積比”概念對(duì)預(yù)處理區(qū)域進(jìn)行細(xì)化分割，剔除除手部以外的膚色干擾區(qū)域并利用最小二乘法二項(xiàng)式擬合算法獲取手指輪廓。采用改進(jìn)的凸包絡(luò)優(yōu)化算法完成指尖的檢測(cè)及傾斜修正。最后，進(jìn)行局部掃描獲得最終的指勢(shì)識(shí)別。實(shí)驗(yàn)表明本文介紹的方法能實(shí)現(xiàn)簡(jiǎn)單傾斜指勢(shì)0~9的識(shí)別且識(shí)別率高達(dá)95.7%，穩(wěn)定性較好。

指勢(shì)識(shí)別；膚色；指尖定位；凸包絡(luò)；局部掃描

TP391.4

A

1003-501X(2016)12-0147-07

10.3969/j.issn.1003-501X.2016.12.023

2016-01-05；

2016-06-14

國(guó)家自然科學(xué)基金(61376028)資助項(xiàng)目

嚴(yán)利民(1971-)，男(漢族)，江蘇吳縣人。副教授，主要研究工作是從事集成電路應(yīng)用、智能檢測(cè)和控制方面的研究。

杜斌(1992-)，男(漢族)，江蘇徐州人。碩士，主要研究工作是研究方向?yàn)閯?dòng)作識(shí)別與智能控制。E-mail：18717865628@163.com。