Best way to quantify how well a model fits data

[omega_minus]

TL;DR Summary

I have recorded time domain data and a computer model (finite difference time domain). I want to know the best method to quantify how well the model fits the recorded data.

Hello,

I have recorded time-domain data from an experiment. I am using a finite difference time domain algorithm to simulate the experiment. The model has been tuned to resemble the data fairly well but I'd like to be able to quantify the fit. Most of what I see online seems to be about random data and quantifying its randomness. My data is not random, it's deterministic. Other than point wise subtraction, is there a standard approach to compare them?
One issue I see is that the small amplitude signals look like they disagree with the model more than large ones (the model gets better looking for big amplitudes). But just subtracting them leads to bigger differences for the larger signals due to their amplitudes, but it seems like they should have smaller error since the model "appears" to fit better.

Any help is appreciated!

 

[Dale]

Are you comparing models or just wanting a number for one model without respect to any other?

 

[omega_minus]

Thanks for the quick reply Dale! The "model" I have is a simulation from a computer. So it is the one that I'm going with since the actual experiment is too complicated for curve fitting from initial principles. That said, I'd like to compare how well the model fits as the conditions of the experiment change. Specifically, it seems to fit well for higher magnetic fields but gets a little worse for weaker ones. But each "model" is a rerun of the computer model under the new magnetic field.
Is that what you're asking?

 

[hutchphd]

How do you "adjust" the model? Are there a fixed number of arbitrary parameters to play with?

 

[omega_minus]

There are lots of parameters unfortunately. Its a model for magnetic nanoparticles. You can put arbitrary numbers of particles in any locations with various anisotropies, radii saturation magnetization, etc. I’ve narrowed it down to only adjusting positions to imitate the inter-particle interactions I am seeing in my data and it’s working out well. But because they are positions defined in Cartesian space, the number of combinations is virtually unlimited.

 

[Dale]

It seems like a simple R^2 should do it.

 

[hutchphd]

So you are adjusting the position of N particles is 3 dimensions ?

 

[FactChecker]

One issue I see is that the small amplitude signals look like they disagree with the model more than large ones (the model gets better looking for big amplitudes). But just subtracting them leads to bigger differences for the larger signals due to their amplitudes, but it seems like they should have smaller error since the model "appears" to fit better.

You should look at the percent errors and see if that fits with your intuition of "better fit". That would compensate for the larger errors from the larger true values.
It sounds like you have two sources of error -- one that is proportional to the time increments, and one that is not. By considering the percentage error, you will compensate for the proportionality and that will leave the non-proportional errors to appear larger when the time data is smaller.

 

[omega_minus]

So you are adjusting the position of N particles is 3 dimensions ?

Yes. The code allows for a random placing of N particles in an L^3 sized box. I just maintain the number density the I need and am able to keep the simulation representative of my sample. The parts I am adjusting are lengths and positions of chains of particles, distances between their constituent chains and interchain distances. For the most part this has been able to simulate the actual data quite well.

 

[omega_minus]

It seems like a simple R^2 should do it.

Great thank you! I'll do that and see what I get.

 

[omega_minus]

You should look at the percent errors and see if that fits with your intuition of "better fit". That would compensate for the larger errors from the larger true values.

Thank you, I'll try this. I initially tried something similar but I was left wondering how exactly to do it. Given that it's a continuous data set should I find the percent error at each point in time and then sum them? I'd also thought about the difference between the two signals as a signal itself and then trying to analyze that too. I'm more accustomed to finding errors between a data set and a linear fit (for example). The whole part about two signals, neither one being mathematically well-defined functions was confusing as to a best-practices approach.

 

[FactChecker]

I would try taking the logarithm of both the true and estimated values and then calculate the sum squared errors. (That is treating the error as an error in a multiplying factor.)

 

[hutchphd]

This is a new technique to me and frankly I don't understand it at all. Is there a handy reference?

 

[omega_minus]

This is a new technique to me and frankly I don't understand it at all. Is there a handy reference?

If you’re interested in seeing the finite difference time domain model that I used, a paper written on it can be found here:
https://arxiv.org/pdf/1408.0908.pdf

 

[hutchphd]

I really meant the logarithmic difference method suggested by @FactChecker. Usually his (her?) suggestions are on the money but this one I don't understand.
I will check the paper. I did a similar kind of calculation to locate N identical light points in 3space from 2 stereo photos of the N points . I just minimized the rms (##R^2##) variation. But I only had N=20.

 

[omega_minus]

I really meant the logarithmic difference method suggested by @FactChecker. Usually his (her?) suggestions are on the money but this one I don't understand.
I will check the paper. I did a similar kind of calculation to locate N identical light points in 3space from 2 stereo photos of the N points . I just minimized the rms (##R^2##) variation. But I only had N=20.

Oh I see, sorry about that. I too was confused about the logarithm thing. I’ll watch for a response from them and thanks to you and everyone else for their input.

 

[FactChecker]

Oh I see, sorry about that. I too was confused about the logarithm thing. I’ll watch for a response from them and thanks to you and everyone else for their input.

I might not have an accurate picture of the situation. I was thinking of the model ##Y=\epsilon b X^a##, where ##\epsilon## is log-normal, which can be converted to a linear regression model ##\ln(Y) = a \ln(X) +ln(b) + \epsilon_{normal}##. The fact that the directly-measured errors seem to be proportional to the ##X## values made me think of that.

 

[omega_minus]

Ok I did the R^2 calculation recommended by @Dale at the results are great. They look like I thought they should. I’d never used R^2 on anything other than linear regression but it was an easy thing to code for two waveforms as an output of my model. Thanks again for everyone’s input!

 

[Stephen Tashi]

I want to know the best method to quantify how well the model fits the recorded data.

What is your definition of "best"? If we have two methods of quantifying how well a model fits data, what criteria do we use to say that one method is the "better" one?

 

[omega_minus]

What is your definition of "best"? If we have two methods of quantifying how well a model fits data, what criteria do we use to say that one method is the "better" one?

Well that’s really my question. It’s easy enough to go online and look at various metrics and their definitions but I haven’t ever seen two time domain signals compared in a paper before. I was really just asking people who write scientific papers what would be a useful comparison if they were reading it. That is, which one would be more helpful in understanding how well the model had worked.

 

[Jarvis323]

I'm not familiar with the physics or the type of model you're using. But wouldn't you need to take into account the bias/variance tradeoff.

https://en.wikipedia.org/wiki/Bias–variance_tradeoff

I guess you are thinking that because your data is deterministic, you don't have to worry about over fitting? But the purpose of the model is to predict outputs you haven't computed right? And those outputs are unknown, so as far as you are concerned they should be treated like they are random (as far as modelling goes) right?

So you should somehow account for that. Otherwise, you can just use a model that will exactly match the data you have with absolutely no error. But that model won't work for the data you don't have.

https://en.wikipedia.org/wiki/Overfitting

Maybe you can reason about it based on assumptions, and by analyzing the residuals, etc. You might have some known or assumed mathematical/statistical relationship between the model and the physics that you can rely on.

https://en.wikipedia.org/wiki/Statistical_model_validation

Or/still you can try tuning the model based on some of your data, and then see what kind of error you get on the data that was not used at all for tuning the model. This would give you an idea how well your model will perform when put into actual use. This might be complicated sometimes, since not all unseen solutions/data points might be equal in terms of how well your model would predict them, so how do you know you're being fair, or what is fair? You have to come up with a good way to argue that the data you tested on is representative, and the performance results you get will be valid only for predicting data similar to that test data.

https://en.wikipedia.org/wiki/Resampling_(statistics)
https://en.wikipedia.org/wiki/Cross-validation_(statistics)

If you keep trying to tune your model to get better and better performance based on the same data, then you might be falling into the circular analysis pitfall.

https://en.wikipedia.org/wiki/Circular_analysis

Ultimately, you usually don't want the model to fit the data you have as best as possible. It's actually usually a red flag if your model fits your in-sample data too well (it means it probably doesn't predict out of sample data well). The error metric you use is just one tool in the validation process.