Hand Action Recognition

Abstract

This paper presents a new dataset for hand action detection for manipulating (assembling and dismantling) mechanical devices and an action detection model based on Transformers. An entry in this dataset is a first-person-view video segment that shows hands performing an action. These hands may utilize a tool and act on an object of the device. These actions were categorized into 12 classes for simple representation. The deep learning model extracts features from each frame in a video, adds position embedding, and feeds the obtained feature vectors to a Transformer Encoder. The output vector goes through a fully connected network to obtain the final class. We have implemented our model and trained it using the presented dataset. We experimentally evaluate the learning and obtain encouraging results.