A Framework for Realistic Clothed Avatar Reconstruction and Animation

Abstract

The rise of metaverse technologies has sparked a growing need for reconstructing 3D realistic avatars, which could give users a real experience in various fields, such as fashion, movies, games, and so on. An avatar is essentially a digital replication of a physical human and plays a crucial role in this evolving field. Traditional approaches rely on expensive equipment and advanced algorithms for 3D avatar reconstruction, which is impractical for users. Subsequently, methods are based on images or videos reconstructing avatars without texture or avatars with texture but not animation. Therefore, this paper proposes a comprehensive framework for realistic clothed avatar reconstruction and animation utilizing a single video as input. The framework comprises four modules: video acquisition, video preprocessing, avatar reconstruction, and avatar animation. Videos can be captured in any wild environment with different devices, such as cameras or phones. 2D keypoints and masks are extracted through deep learning models and input into the reconstruction module to generate a precise 3D avatar mesh model and a full texture image. Avatar animation is achieved through an auto-rigger system and a bake action tool. Meanwhile, extensive experiments are conducted to evaluate the impact of four factors on avatar reconstruction performance: mask generation method, background, speed, and rotation. The results indicate that rotation has the most significant effect on avatar reconstruction, followed by the mask generation method, background, and speed. Finally, a four-layer metaverse is proposed and developed using blockchain and artificial intelligence (AI) technologies to verify the availability and generality of our avatar framework. Through this exploration, we not only showcase the potential of our approach but also underscore the convergence of digital realms and emerging technologies that are reshaping the future of human interaction and expression.