Towards Subjective Multimedia Summarization Framework for Sporting Event in the Context of Digital Twins

Abstract

Real-world events generate a massive amount of traffic on social media with live, moment-to-moment accounts as any given situation unfolds. The data generated on social media is loaded with the sentiment, opinions, and reactions of the public towards the events. Browsing the event-related data in its raw form would be an overwhelming task due to the extensive amount of data, making the search for any specific updates or useful information an especially daunting and time-consuming endeavor. Event-related data, if effectively summarized, could generate a comprehensive overview of the event in terms of what happened and how people reacted at the time. Unfortunately, most of the event-based summarization systems concentrate their effort on simply describing what happened during the event; ignoring a valuable resource of emotional reactions that portrays the event from varying perspectives. Accordingly, in this thesis, we introduce an event-based summarization approach that incorporates multimedia data, sentiment, and human reactions with respect to their point of view in order to generate a concise subjective multimedia summary of the event. In order to achieve our goal, we introduce popularity prediction and sentiment analysis models, both of which are essential in our summarization approach. As the event unfolds over time, we utilize the popularity prediction model to extract a representative set of images for the event to be included in the summary based on the predicted popularity scores. Multi-modality features are exploited to develop our model, including various levels of visual features, textual features, and contextual features. In order to track users' opinions and changes in their sentiment in correlation with the occurrence of sub-events, we develop a sentiment classification model that effectively recognizes the sentiment conveyed in sport conversations. Due to the lack of a manually annotated sentiment dataset, we propose a new, manually labeled football-based sentiment dataset. We also create an automatically generated sentiment-lexicon specifically for the football domain. We assess the performance of our developed models and generated summary through extensive experimental evaluation. We explore the impact of different features on the performance of the popularity prediction and sentiment models. We also leverage the knowledge of sport fans to evaluate the generated subjective summarization through an online user-based survey. The experiment results confirm the effectiveness of our proposed solution for event-based summarization by considering multimedia data, sentiment, and people's subjective views of events.