Skin detection is a useful and popular method to identify and recognize human body parts, faces, naked people, and skin diseases and retrieve people in multimedia databases. Therefore, finding a suitable method to divide the pixels of an image into different skin groups can be very important. One of the most common methods is based on convolutional neural networks (CNN). However, the process of training a CNN is a challenging issue. Various optimization strategies have recently been used to optimize CNN biases and weights, such as the firefly algorithm (FA) and ant colony optimization (ACO). In this study, we use a well-known nature-inspired technique called Chimp optimization algorithm (ChOA) to train a classical LeNet-5 CNN structure for skin detection. The proposed skin classification algorithms operate directly on the RGB and HVS color space. The results clearly show that the proposed algorithm significantly improves the performance of a convolutional neural network.