Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

Locatello et al., 2019


  • Raises concerns about the authenticity of recent progress in the unsupervised learning of disentangled representations
  • Show theoretically that unsupervised learning is impossible without inductive biases
  • Empirical results show that increased disentanglement does not reduce sample complexity of downstream learning
  • Disentanglement learning should be explicit about inductive biases, supervision, and concrete benefits of the learned representation
  • Links: [ website ] [ pdf ]


  • Core assumption in representation learning: high-dimensional real-world observations are generated from much lower dimensional, semantically meaninful latent variable
    • Disentangled representations should therefore separate out these distinct factors of variation in the data
    • Additional assumption that disentangled representations will be useful for downstream tasks
  • Independent component analysis (ICA) also aims to uncover independent components of the input
    • Limited utility for non-linear cases


  • Considered the following methods, based on VAE loss with regularizer:
    • $\beta$-VAE: constrain capacity of bottleneck with hyperparameter in front of KL regularizer
    • AnnealedVAE: gradually increases bottleneck capacity
    • FactorVAE: penalize total correlation with adversarial training
    • $\beta$-TCVAE: penalize correlation with biased MC estimator
    • DIP-VAE-II: penalize mismatch between posterior and prior
  • Each method uses same architecture, optimizer, and hyperparameters for optimizer and batch size


  • Datasets:
    • Deterministic function of latent variable
      • dSprites
      • Cars3D
      • SmallNORB
      • Shapes3D
    • Stochastic:
      • Color-dSprites: random color
      • Noisy-dSprites: white shapes on noisy background
      • Scream-dSprites: background replaced with random patch with random tint from The Scream painting
  • Metrics of disentanglement:
    • BetaVAE: accuracy of linear classifier on predicting index of fixed factor of variation
    • FactorVAE: majority vote classifier on different feature vector, addresses issues with BetaVAE
    • Mutual Information Gap (MIG): normalized gap in MI between highest and second highest coordinate in representation
    • Modularity: each dimension of representation depends on at most one factor of variation
    • DCI Disentanglement: entropy of distribution from normalizing importance of repsentation dimensions for predicting variation factors
    • SAP score: average difference of prediciton error of two most predictive latent dimensions
  • Proof that for any marginal distribution of input data, there exists generative models with latent variables disentangled from the learned representation, but aso ones that are completely entangled
    • Correct model cannot be determined from just the input distribution
  • Results on Color-dSprites show that, in general, the methods produce an aggregated posterior whose individual dimensions are uncorrelated, but not for dimensions of the mean representation
  • With the exception of Modularity, all metrics seem to be correlated accross multiple datasets
  • Calculate FactorVAE for each method on Cars3D while varying hyperparameters and random seed:
    • Large overlap between models suggest hyperparameters and random seed more importaant than specific objective function
    • There is significant variation from random seed alone
  • Probability of a selected model performing better than a random model on a random dataset and metric is basically at chance
  • Plot of sample efficiency vs FactorVAE score does not show a strong correlation


  • Easy to draw incorrect conclusions from results using only a few methods, metrics, and datasets
  • Unsupervised model selection remains an open problem
  • Poor correlation of sample complexity vs disentanglement might just be due to the tested models’ inability to reliably produce disentangled representations
Elias Z. Wang
Elias Z. Wang
AI Researcher | PhD Candidate