Comparing Kullback-Leibler divergence values

In summary, the conversation discusses the evaluation of two survival models in R, using Kullback-Leibler divergence to compare the simulated survival time datasets (`dat.s1` and `dat.s2`) with the observed survival time dataset (`dat.obs`). While the initial results show that `dat.s2` is a better match, a visualization of the density plots reveals that `dat.s1` aligns better with the observation. This discrepancy may be due to the non-commutative nature of KL-divergence and the different objective functions it represents. The discussion also suggests exploring the alternative direction of fitting `dat.obs` to `dat.s1` and `dat.s2` to better align with the research question.
  • #1
nigels
36
0
I’m currently evaluating the "realism" of two survival models in R by comparing the respective Kullback-Leibler divergence between their simulated survival time dataset (`dat.s1` and `dat.s2`) and a “true”, observed survival time dataset (`dat.obs`). Initially, directed KLD functions show that `dat.s2` is a better match to the observation:

> library(LaplacesDemon)
> KLD(dat.s1, dat.obs)$sum.KLD.py.px
[1] 1.17196
> KLD(dat.s2, dat.obs)$sum.KLD.py.px
[1] 0.8827712​

However, when I visualize the densities of all three datasets, it seems quite clear that `dat.s1` (green) better alignes with the observation:

> plot(density(dat.obs), lwd=3, ylim=c(0,0.9))
> lines(density(dat.s1), col='green')
> lines(density(dat.s2), col='purple')​

What is the cause behind this discrepancy? Am I applying KLD incorrectly due to some conceptual misunderstanding?
 

Attachments

  • KL_nonsense_sof.png
    KL_nonsense_sof.png
    14.2 KB · Views: 747
Physics news on Phys.org
  • #2
Keep in mind that the KL-divergence is non-commutative, and different "orders" correspond to different objective functions (and different research questions). The way you're fitting it (that is, KL(Q||P), where Q is being fit to P) is trying to match regions of high density, and it does seem to be the case that the highest probability mass in your "worse fitting" model coincides with the highest probability mass in your target better than does the "better fitting" model. There's a fairly good discussion related to the topic here:

https://stats.stackexchange.com/questions/188903/intuition-on-the-kullback-leibler-kl-divergence
and
http://timvieira.github.io/blog/post/2014/10/06/kl-divergence-as-an-objective-function/

The other direction may actually be closer to what you're interested in:
KLD(dat.obs, dat.s1)$sum.KLD.py.px
KLD(dat.obs, dat.s2)$sum.KLD.py.px
 
Last edited:

Related to Comparing Kullback-Leibler divergence values

1. What is Kullback-Leibler divergence?

Kullback-Leibler divergence, also known as relative entropy, is a measure of how one probability distribution differs from another. It is often used in statistics and information theory to compare the similarity between two sets of data or probability distributions.

2. How is Kullback-Leibler divergence calculated?

Kullback-Leibler divergence is calculated by taking the sum of the probability of each event in one distribution multiplied by the logarithm of the ratio of that probability to the corresponding probability in the other distribution. This calculation can also be interpreted as the amount of information lost when using one distribution to approximate the other.

3. What is the significance of comparing Kullback-Leibler divergence values?

Comparing Kullback-Leibler divergence values allows us to quantitatively measure the similarity or difference between two sets of data or probability distributions. This can be useful in a variety of fields, such as machine learning, data analysis, and information theory.

4. How do you interpret Kullback-Leibler divergence values?

A higher Kullback-Leibler divergence value indicates a greater difference between two distributions, while a lower value indicates a higher similarity. It is important to note that Kullback-Leibler divergence is not a symmetric measure, meaning that the order in which the distributions are compared can affect the resulting value.

5. Can Kullback-Leibler divergence be negative?

Yes, Kullback-Leibler divergence can be negative. This occurs when the probability of an event in one distribution is greater than the probability of the same event in the other distribution. However, a negative value may not always make sense in certain contexts, so it is important to consider the interpretation of the results in relation to the specific problem being studied.

Hot Threads

Recent Insights

Back
Top