All about Diffusion models

 

• Classfier gudiance
• What is score function?: https://ericmjl.github.io/score-models/notebooks/01-score-function.html

The gradient of the log probability density with respect to data (https://medium.com/@aminamollaysa/score-based-generative-models-bfe88808dc81)

A Pedagogical Introduction to Score Models - 2  Score Functions

• What is maximum likelihood estimation? (including why likelihood not probability) https://towardsdatascience.com/probability-concepts-explained-maximum-likelihood-estimation-c7b4342fdbb1

Probability concepts explained: Maximum likelihood estimation

• Diffusion guidance (+ why add additional noise, "Temperature" in deep learning)? https://sander.ai/2022/05/26/guidance.html

Guidance: a cheat code for diffusion models

To sample from a diffusion model, an input is initialised to random noise, and is then iteratively denoised by taking steps in the direction of the score function (i.e. the direction in which the log-likelihood increases fastest), with some additional noise mixed in to avoid getting stuck in modes of the distribution

 

"Temperature"?: The temperature parameter in deep learning is used to control the randomness or uncertainty of the model's predictions. It is often used in the context of generating text or other types of data. A higher temperature value will result in more diverse and creative outputs, while a lower temperature value will result in more conservative and predictable outputs. The temperature parameter is typically used in conjunction with a softmax function to adjust the probabilities of each possible output.

즉, temperature가 높으면 uncertainty도 높다. Guidance: a cheat code for diffusion models에서 쓰임새를 보면 감마가 inverse temperature paramter라고 나온다. 감마가 1 초과인 경우 distribution 을 mode의 중심이 더 뾰족하고 (다만 이 지점의 probability가 더 높아지는건 아니다. probability <= 1 이니까. 더 작은 probability 들이 더 많이 작아지는것) probability 가 상대적으로 더 작은곳들의 값들이 더 낮아진다. 즉 uncertainty가 줄어든다. 고로 감마값이 커질수록 uncertainty가 낮아지므로 감마는 inverse temperature paramter.

 

• Why intractable?: https://xyang35.github.io/2017/04/14/variational-lower-bound/#ba2014multiple

Understanding the Variational Lower Bound

• Why need ELBO after KL? Why still intractable if only use KL?: https://mpatacchiola.github.io/blog/2021/01/25/intro-variational-inference.html

Evidence, KL-divergence, and ELBO

 

• What is Variational Inference?
  https://stats.stackexchange.com/questions/340955/what-does-variational-mean

What does "variational" mean?

https://mpatacchiola.github.io/blog/2021/01/25/intro-variational-inference.html

Evidence, KL-divergence, and ELBO

• Interpreting conditonal distribution

with code: https://towardsdatascience.com/variational-autoencoder-demystified-with-pytorch-implementation-3a06bee395ed

Variational Autoencoder Demystified With PyTorch Implementation.