LOTUS Theorem in original GANs formulation.

Computer Vision Machine Learning Generative Adversarial Networks

How do we apply the Law Of The Unconcious Statistician to Ian Goodfellow’s original GANs value function formulation?

Andrea Bonvini https://github.com/andreabonvini
12-11-2020

In this brief we want to prove a passage of the original GANs Paper by Ian Goodfellow. Specifically, we want to prove that the following equation is satisfied:

\[ \int_{\mathbf{z}}p_{z}(\mathbf{z})\log(1-\mathcal{D}(\mathcal{G}(\mathbf{z})))d\mathbf{z} = \int_{\mathbf{x}}p_{g}(\mathbf{x}) \log(1-\mathcal{D}(\mathbf{x})) d\mathbf{x} \] For a continuous random variable \(\mathbf{z}\), let \(\mathbf{x} = \mathcal{G}(\mathbf{z})\) and \(\mathbf{z}=\mathcal{G}^{-1}(\mathbf{x})\) suppose that \(\mathcal{G}\) is differentiable and that its inverse \(\mathcal{G}^{-1}\) is monotonic. By the formula for inverse functions and differentiation we have that \[ \frac{d\mathbf{z}}{d \mathbf{x}}\cdot \frac{d \mathbf{x}}{d \mathbf{z}} = 1 \] \[ \frac{d\mathbf{z}}{d \mathbf{x}} = \frac{1}{\frac{d \mathbf{x}}{d \mathbf{z}}}\\ \] \[ \frac {d\mathbf{z}} {d\mathbf{x}} = \frac {1} {\frac{d\mathcal{G}(\mathcal{G}^{-1}(\mathbf{x}))}{d(\mathcal{G}^{-1}(\mathbf{x}))}} \] \[ \frac {d\mathbf{z}} {d\mathbf{x}} = \frac {1} {\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))} \] \[ d\mathbf{z} = \frac {1} {\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))}d\mathbf{x} \] so that by a change of variables we can rewrite everything in function of \(\mathbf{x}\), \[ \int_{-\infty}^{\infty} p_{z}(\mathbf{z}) \log \left( 1-\mathcal{D}(\mathcal{G}(\mathbf{z})) \right) d\mathbf{z} = \] \[ \int_{-\infty}^{\infty} p_{z}(\mathbf{z}) \log \left( 1-\mathcal{D}(\mathbf{x}) \right) \frac{1}{\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))}d\mathbf{x}= \] \[ \int_{-\infty}^{\infty} \color{blue}{p_{z}(\mathbf{z})} \log \left( 1-\mathcal{D}(\mathbf{x}) \right) \color{blue}{ \frac{1}{\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))}}d\mathbf{x}= \] \[ \int_{-\infty}^{\infty} \color{blue}{p_{z}(\mathbf{z})} \color{blue}{ \frac{1}{\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))}} \log \left( 1-\mathcal{D}(\mathbf{x}) \right) d\mathbf{x}= \] Now, notice that the cumulative distribution function \(P_\mathbf{X}(\mathbf{x}):\mathbb{R}^n\to[0,1]\) is \(P_\mathbf{X}(\mathbf{x})=Pr(\mathbf{X}<\mathbf{x})=Pr(X_1\leq x_1,X_2\leq x_2,\dots,X_n\leq x_n)\), we observe that:

\[ P_\mathbf{X}(\mathbf{x})= \] \[ Pr(\mathbf{X}<\mathbf{x})= \] \[ Pr(\mathcal{G}(\mathbf{Z})\leq\mathbf{x})= \] \[ Pr(\mathbf{Z}\leq\mathcal{G}^{-1}(\mathbf{x}))= \] \[ P_{\mathbf{Z}}(\mathcal{G}^{-1}(\mathbf{x})) \] From here \[ P_\mathbf{X}(\mathbf{x})=P_{\mathbf{Z}}(\mathcal{G}^{-1}(\mathbf{x}))\\ P_\mathbf{X}(\mathbf{x})=P_{\mathbf{Z}}(\mathbf{z}) \] We take the derivative w.r.t \(\mathbf{x}\): \[ \frac{dP_\mathbf{X}(\mathbf{x})}{d\mathbf{x}} =\frac{P_{\mathbf{Z}}(\mathbf{z})}{d\mathbf{x}} \] \[ \frac{dP_\mathbf{X}(\mathbf{x})}{d\mathbf{x}} =\frac{P_{\mathbf{Z}}(\mathbf{z})}{d\mathbf{z}}\frac{d\mathbf{z}}{d\mathbf{x}} \] \[ p_{X}(\mathbf{x}) = p_{\mathbf{z}}(\mathbf{z})\frac{1}{\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))} \] For clarity we rename \(p_{X}(\mathbf{x})\) as \(p_{g}(\mathbf{x})\) since it represents the distribution learned from our generator \(\mathcal{G}\), we have \[ \color{blue}{p_{g}(\mathbf{x}) = p_{\mathbf{z}}(\mathbf{z})\frac{1}{\mathcal{G}^{'}(\mathcal{G}^{-1}(\mathbf{x}))}} \]

Citation

For attribution, please cite this work as

Bonvini (2020, Dec. 11). Last Week's Potatoes: LOTUS Theorem in original GANs formulation.. Retrieved from https://lastweekspotatoes.com/posts/2021-07-22-lotus-theorem-in-original-gans-formulation/

BibTeX citation

@misc{bonvini2020lotus,
  author = {Bonvini, Andrea},
  title = {Last Week's Potatoes: LOTUS Theorem in original GANs formulation.},
  url = {https://lastweekspotatoes.com/posts/2021-07-22-lotus-theorem-in-original-gans-formulation/},
  year = {2020}
}