Commonsense Knowledge for 3D Modeling: A Machine Learning Approach

Abstract

Common‒sense knowledge is a collection of non‒expert and agreed‒upon facts and information about the world shared among most people past early childhood based on their experiences. It includes uses of objects, their properties, parts and materials, their locations, spatial arrangements among them; location and duration of events; arguments, preconditions and effects of actions; urges and emotions of people, etc. In creating 3D worlds and especially text‒ to‒scene and text‒to‒animation systems, this knowledge is essential to eliminate the tedious and low‒level tasks, thus allowing users to focus on their creativity and imagination. We address tasks related to five categories of common‒sense knowledge that is required by such systems including: (1) spatial role labeling to automatically identify and annotate a set of spatial signals within a scene description in natural language; (2) grounding spatial relations to automatically position an object in 3D world; (3) inferring spatial relations to extract symbolic spatial relation between objects to answer questions regarding a 3D world; (4) recommending objects and their relative spatial relations given a recent manipulated object to auto‒complete a scene design; (5) learning physical attributes (e.g., size, weight, and speed) of objects and their corresponding distribution. We approach these tasks by using deep learning and probabilistic graphical models and exploit existing datasets and web content to learn the corresponding common‒sense knowledge.