Multi-Task Deep Learning for Affective Content Detection from Text

Abstract

Deep learning (DL) is a subset of machine learning and artificial intelligence. It has broad adaptability to most types of tasks, including but not limited to text classification and identification of objects in images and videos. It can generate more powerful models compared with legacy machine learning methods. Multi-task Learning (MTL) is an approach that improves generalization by using the domain information which is contained in the training signals of related tasks as an inductive bias. When we apply Multi-task Learning to Deep Learning, the method is called Multi-task Deep Learning.

We focus on deep learning for natural language processing, in particular on how multi-task learning can be used to improve the performance on several tasks at the same time. We present two experiments that deploy Multi-task Deep Learning for detecting affect information from texts.

In experiment 1, we propose a hard parameter sharing multi-task deep learning model for the task of detecting happiness ingredients. For training Deep Learning classifiers, the two primary classes, "agency" and "social", meaning whether the author is in control or the moment involves other people, are treated as two separate tasks while "concept", meaning the categories of the moment, is serverd as an auxiliary task. Then, we train a multi-task deep learning classifier to see if the shared knowledge among the three tasks can be used to improve the overall results. In addition, we compare several models that use different kinds of word embeddings: different dimensions of the vectors, fixed versus trainable embeddings, initialized randomly or with pre-trained embeddings.

In experiment 2, we compare several different multi-task deep learning models on the task of six labels: Information_disclosure, Emotional_disclosure, Support, General_support, Info_support, and \textitEmo_support. The labels mean that the texts contain informational or emotional disclosure of a person, or express informational or emotional supprtiveness, which can also be catchphrases. We propose a novel way to employ the multi-task deep learning model for the task of detecting disclosure and support, called Venn-diagram-based fragment MTL model. We calculate all possible logical relations between the six labels, represented in a Venn diagram. Based on it, the six labels are distributed to multiple fragment layers. Then, a multi-task deep neural network is built on these layers.

We showed that our multi-task learning model has a stronger ability to represent multi-label tasks over multiple single-task learning models, and using pre-trained trainable embeddings with auxiliary tasks can get the best results. Furthermore, among different multi-task deep learning structures, our model based on Venn diagrams achieved better performance than regular multi-task deep learning and obtained the best results in the CL-Aff Shared Task 2020 for the disclosure labels.