Sampling from a population

[hoffmann]

i have a population of items and i calculate how similar each item is to the rest of the items in the population. i store these values in a symmetric nxn similarity matrix. when i look at the distribution of scores, they follow a beta distribution -- most similarity scores are close to 0 (the items are dissimilar) and then they tail off and fewer and fewer are closer to 1 (the items are very similar).

i want to generate samples from this distribution -- i want to group items together whose distribution of scores is also beta, but only with a lot of the mass centered around the mean and tailing off on both ends (like a hump). these groupings of beta distributions should (i'm pretty sure) naturally form in my data, so my question is: is there a method to sample from my population that gives me a smaller population with a given distribution? any starting point would help...thanks!

 

[mmwave]

Thank you for sharing your interesting findings about the distribution of similarity scores in your population. I can offer some insights and suggestions on how you can generate samples from this distribution.

Firstly, I would like to clarify that the beta distribution is a continuous probability distribution that takes values between 0 and 1, which seems to be a suitable choice for your similarity scores. However, in order to generate samples from this distribution, you will need to specify the parameters of the beta distribution. These parameters will determine the shape of the distribution and the location of the "hump" that you mentioned.

One approach could be to fit a beta distribution to your existing similarity scores data and use the estimated parameters to generate new samples. This can be done using statistical software such as R or Python. Alternatively, if you have a large enough sample of similarity scores, you can use the method of moments to estimate the parameters of the beta distribution.

Another approach could be to use a simulation method such as Monte Carlo sampling. In this method, you can specify the desired beta distribution and generate a large number of random samples from it. You can then select a smaller subset of these samples to form your new population.

It is also worth considering if there are any underlying factors or variables that may affect the distribution of similarity scores in your population. For example, if your items can be categorized into different groups, you can generate samples from each group separately to ensure that the distribution of scores is similar within each group.

In summary, there are various methods that you can use to generate samples from your population with a desired distribution. I hope these suggestions will help you in your research. Best of luck!