Multimodal Machine Learning:A Survey and Taxonomy

Abstract aim: build models that can process and relate information from multiple modalities new taxonomy:representation, translation, alignment, fusion, and co-learning. INTRODUCTION                                                                                                                                  three modalities:                                                                                                                                   (nlp)natural language which can be both written or spoken; (cv) visual signals which are often represented with images or videos; (sr) vocal signals which encode sounds and para-verbal information such as prosody and vocal expressions challenges: representation: the complementarity and redundancy translation:translate one modality to another( open-ended or subjective) alignment: measure similarity between different modalities and deal with possible long range dependencies and ambiguities fusion: join information from two or more modalities to perform a prediction co-learning: co -training, conceptual grounding, and zero shot learning Table 1: A summary of applications enabled by multimodal machine learning. For each application area we identify the core technical challenges that need to be addressed in order to tackle it.

2 A PPLICATIONS : A HISTORICAL PERSPECTIVE 略 3 M ULTIMODAL R EPRESENTATIONS Representing multiple modalities poses many diffificulties: how to combine the data from heterogeneous sources; how to deal with different levels of noise; and how to deal with missing data. a number of properties for good representations: smoothness(类比nlp的平滑), temporal and spatial coherence, sparsity, and natural clustering amongst others additional desirable properties for multimodal representations: similarity in the representation space should reflect the similarity of the corresponding concepts, the representation should be easy to obtain even in the absence of some modalities it should be possible  to fifill-in missing modalities given the observed ones. 3.1 Joint Representations（ project unimodal representations together into a multimodal
space ） Mathematically, the joint representation is expressed as: xm = f(x1, . . . , xn)        (1) Neural networks To construct a multimodal representation using neural networks each modality starts with several individual neural layers fol lowed by a hidden layer that projects the modalities into a joint space.The joint multimodal representation is then be passed through multiple hidden layers itself or used directly for prediction.（usually be trained end to end） advantages: neural network based joint rep resentations comes from their often superior performance and the ability to pre-train the representations in an unsu pervised manner. The performance gain is, however, depen dent on the amount of data available for training. disadvantages: model not being able to handle missing data                                                                              naturally deep networks are often diffificult to train Probabilistic graphical models advantages: generative nature, which allows for an easy way to deal with missing data ( even if a whole modality is missing, the model has a natural way to cope) generate samples of one modality in the presence of the other one, or both modalities from the representation s the representation can be trained in an unsupervised manner enabling the use of unlabeled data disadvantages: the diffificulty of training them( high computational cost, and the need to use approximate variational training methods ) Sequential Representation(varying length sequences) 略 3.2 Coordinated Representations ( learn separate representations for each modality but coordinate them through a constraint ) While coordinated representation is as follows: f ( x 1 ) ∼ g ( x 2 )        (2) Similarity models（ minimize the distance between modal ities in the coordinated space ） structured coordinated space（ Structured coordinated spaces are commonly used in cross-modal hashing ） Hashing enforces certain constraints on the resulting multimodal space: 1) it has to be an N -dimensionalHamming space — a binary representation with controllable number of bits; 2) the same object from different modalities has to have a similar hash code; 3) the space has to be similarity-preserving. 4 T RANSLATION Given an entity in one modality the task is to generate the same entity in a different modality. 4.1 Example-based Retrieval-based models ( directly use the retrieved translation without modifying it ) Combination-based models ( rely on more complex rules to create translations based on a number of retrieved instances ) 4.2 Generative approaches Grammar-based models Encoder-decoder models Continuous generation models 5 A LIGNMENT 5.1 Explicit alignment Unsupervised Supervised 5.2 Implicit alignment Graphical models Neural networks         

 

6 F USION 6.1 Model-agnostic approaches 6.2 Model-based approaches Multiple kernel learning (MKL) Graphical models Neural Networks 7 C O - LEARNING 7.1 Parallel data         Co-training Transfer learning 7.2 Non-parallel data Transfer learning Conceptual grounding Zero shot learning 7.3 Hybrid data