Deep Learning-Enabled Multitask System for Exercise Recognition and Counting

Abstract

Exercise is a prevailing topic in modern society as more people are pursuing a healthy lifestyle. Physical activities provide unimaginable benefits to human well-being from the inside out. 2D human pose estimation, action recognition and repetitive counting fields developed rapidly in the past several years. However, few works combined them together as a whole system to assist people in evaluating body poses, recognizing exercises and counting repetitive actions. The existing methods estimate pose positions first, and utilize human joints locations in the other two tasks. In this thesis, we propose a multitask system covering the three domains. Different from the methodology used in the literature, heatmaps which are the byproducts of 2D human pose estimation models are adopted for exercise recognition and counting. Recent heatmap processing methods are proven effective in extracting dynamic body pose information. Inspired by this, we propose a new deep-learning multitask model of exercise recognition & repetition counting, and apply these approaches to the multitask for the first time. To meet the needs of the multitask model, we create a new dataset Rep-Penn with action, counting and speed labels. A two-stage training strategy is applied in the training process. Our multitask system can estimate human pose, identify physical activities and count repeated motions. We achieved 95.69% accuracy in exercise recognition on Rep-Penn dataset. The multitask model also performed well in repetitive counting with 0.004 Mean Average Error (MAE) and 0.997 Off-By-One (OBO) accuracy on Rep-Penn dataset. Compared with existing frameworks, our method obtained state-of-the-art results.