Improving Non-Transferable Representation Learning by Harnessing Content and Style

Non-transferable learning (NTL) aims to restrict the generalization of models toward the target domain(s). To this end, existing works learn non-transferable representations by reducing statistical dependence between the source and target domain. However, such statistical methods essentially neglect to distinguish betweenstylesandcontents, leading them to inadvertently fit (i) spurious correlation betweenstylesandlabels, and (ii) fake independence betweencontentsandlabels. Consequently, their performance will be limited when natural distribution shifts occur or malicious intervention is imposed. In this paper, we propose a novel method (dubbed as H-NTL) to understand and advance the NTL problem by introducing a causal model to separately modelcontentandstyleas two latent factors, based on which we disentangle and harness them as guidances for learning non-transferable representations with intrinsically causal relationships. Specifically, to avoid fitting spurious correlation and fake independence, we propose a variational inference framework to disentangle the naturally mixedcontent factorsandstyle factorsunder our causal model. Subsequently, based on dual-path knowledge distillation, we harness the disentangled twofactorsas guidances for non-transferable representation learning: (i) we constraint the source domain representations to fitcontent factors(which are the intrinsic cause oflabels), and (ii) we enforce that the target domain representations fitstyle factorswhich barely can predict labels. As a result, the learned feature representations follow optimal untransferability toward the target domain and minimal negative influence on the source domain, thus enabling better NTL performance. Empirically, the proposed H-NTL significantly outperforms competing methods by a large margin.