The hearing-impaired community uses gestures to communicate. Gestures can also be used in interactions between man and computer. However, gestures become increasingly complicated in a comparatively complex environment. A recognition algorithm with a choice of function based on the improved genetic algorithm is proposed to improve the ability to identify gestures. The recognition process includes retailing, extraction, and feeding functions before classifying the neural network. After learning gestures, the proposed method is compared with traditional methods that use the classic genetic algorithm. The proposed method demonstrates the effect of optimization and sensitivity of the function.