Hand gestures are a well-known and intuitive method of human-computer interaction. The majority of the research has concentrated on hand gesture recognition from the RGB images, however, little work has been done on recognition from videos. In addition, RGB cameras are not robust in varying lighting conditions. Motivated by this, we present the video based hand gestures recognition using the depth camera and a light weight convolutional neural network (CNN) model. We constructed a dataset and then used a light weight CNN model to detect and classify hand movements efficiently. We also examined the classification accuracy with a limited number of frames in a video gesture. We compare the depth camera’s video gesture recognition performance to that of the RGB camera. We evaluate the proposed model’s performance on edge computing devices and compare to benchmark models in terms of accuracy and inference time. The proposed model results in an accuracy of 99.48% on the RGB version of test dataset and 99.18% on the depth version of test dataset. Finally, we compare the accuracy of the proposed light weight CNN model with the state-of-the hand gesture classification models.