Relaxed Invariant Representation for Unsupervised Domain Adaptation

Abstract

The success of supervised machine learning relies on the availability of a large amount of annotated training data from different domains, which is often cost-ineffective to collect, and unrealistic in many scenarios. Unsupervised domain adaptation (UDA) aims to overcome this problem by transferring predictive models trained on a labelled source domain to an unlabelled target domain, with the difficulty of resolving distributional shift between domains. To bridge this distribution gap, recent advances in deep learning focus on learning representations that are invariant across domains. However, such an approach may fail to generalize well to target domains and may even considerably deteriorate adaptability, due to the existence of an inherent trade-off between adaptability and invariance. Building on advances in deep generative models, this thesis aims to relax the learning of invariant representations, and to develop efficient algorithms for UDA. This thesis comprises two parts. The first part introduces the problem of learning invariant representations. In particular, we mathematically derive a lower bound on the joint probability distribution of the source and target domains as a framework for UDA and theoretically discuss how this bound can be used to relax the invariance in representation learning. Following this motivation, in the second part, we design a simple, yet efficient algorithm to address the challenges of forcing too much invariance in domain distributional matching. We empirically show how the trade-off between adaptability and invariant representation can be mitigated with an invertible architecture between the representation and predictor models while learning the invariant representation. The experiments are run on public benchmark problems and the results show that the proposed method relaxes the excessive invariance effectively and outperforms the existing domain adaptation approaches.