Statistical ensemble interpretation done right

[lodbrok]

Uncertainty ~ Is a measure of (an obsevers/agents) predictive abiliy, typically measured by some "statistical uncertainty" score, say some confidence interval as per some confidence level, in predicting the future (for example the outcome of an experiment), or the responses from the environment. But without specifying WHY.

Ignorance - supposedly indicates that the explanation of lack of certainty/confidence is because the agent/observer is uninformed but where we at least in principle could have been informed. Ie that the information it needs is in principle information theoretically and computationally accessible in a given time scale.

Try to define it mathematically, and you'll see the problem immediately. But logically, a lack of predictive ability is equivalent to a lack of knowledge. What you call a "statistical uncertainty score" is what I would call a "probability". The WHY is already included. You can't predict it because you don't know how to do it.
Not knowing how to do it is Ignorance. Therefore "uncertainty NOT due to ignorance" is contradictory. Uncertainty means ignorance.

 

[PeterDonis]

Uncertainty means ignorance.

In the sense you have used the term--you don't know how to do it--yes. But the obvious next question is, why don't you know how to do it? There are two possible reasons:

(1) You don't know how to do it because you haven't yet figured out how. But it is possible to do it--you just need to figure out how.

(2) You don't know how to do it because it is impossible to do it. No matter how much effort you expend, you will never be able to figure out how to do it because it is impossible.

Your quoted statement above does not help at all to distinguish between these two possibilities.

 

[lodbrok]

In the sense you have used the term--you don't know how to do it--yes. But the obvious next question is, why don't you know how to do it? There are two possible reasons:

(1) You don't know how to do it because you haven't yet figured out how. But it is possible to do it--you just need to figure out how.

(2) You don't know how to do it because it is impossible to do it. No matter how much effort you expend, you will never be able to figure out how to do it because it is impossible.

Your quoted statement above does not help at all to distinguish between these two possibilities.

You can't prove a negative, so you can't know that it's impossible to know how to do it. Therefore (2) should never come up in reasonable discussions of physics.

 

[PeterDonis]

You can't prove a negative, so you can't know that it's impossible to know how to do it.Therefore (2) should never come up in reasonable discussions of physics.

I disagree. For example, (2) is our current belief regarding traveling faster than light. You could quibble and say that we might discover a method of FTL travel some time in the future, but our best current theories of physics say it's impossible--not just that we haven't figured out how yet, but that there is no way to do it at all. Reasonable discussions of our best current theories make use of statements like this all the time.

 

[Fra]

You can't prove a negative, so you can't know that it's impossible to know how to do it. Therefore (2) should never come up in reasonable discussions of physics.

A common method is proof by contradiction.

Sure bell ansatz does not cover all possibilities, but it is an example of a proof by contradiction, as QM(and more important -experiments) does not obey the implied inequality that follows from explaining it all by an uninformed physicist assuming bell style local realism.

/Fredrik

 

[lodbrok]

I disagree. For example, (2) is our current belief regarding traveling faster than light. You could quibble and say that we might discover a method of FTL travel some time in the future, but our best current theories of physics say it's impossible--not just that we haven't figured out how yet, but that there is no way to do it at all. Reasonable discussions of our best current theories make use of statements like this all the time.

There are a lot of things that we can be certain are impossible. But claiming to know that something cannot be known is quite different than a claim postulating that you can't travel faster than the speed of light. But even if it is granted that the original claim was just a postulate that the uncertainty in QT is not due to ignorance (afaik there's no such postulate in QT), there's still a burden to define those terms in a non-contradictory way and I haven't seen such definitions.

 

[lodbrok]

A common method is proof by contradiction.

Sure bell ansatz does not cover all possibilities, but it is an example of a proof by contradiction, as QM(and more important -experiments) does not obey the implied inequality that follows from explaining it all by an uninformed physicist assuming bell style local realism.

/Fredrik

Yes, it proves that those possibilities it covers are inconsistent (contradictory) with its assumptions. It doesn't prove that it is impossible to find a possibility that is consistent with its assumptions.

 

[vanhees71]

You don't see the contradiction between "complete knowledge" and "uncertainty"? How exactly does uncertainty arise when you have "complete knowledge"?

If I know that a quantum system is prepared in a pure state, then I have "complete knowledge" about the system, but I don't know, which measurement result I get when measuring an observable for whose representing operator this state is not an eigenstate. So in QT, even with complete knowledge, there's uncertainty about the outcome of measurements.

There are also the uncertainty relations. E.g., for momentum and position you get ##\Delta x \Delta p_x \geq \hbar/2##, which implies that neither position nor momentum can ever be determined, although either of them can be as precisely determined as you like, but then the other becomes very uncertain and vice versa.

Mathematically:
Certainty = Probability 1 or Probability 0 = Complete Knowledge
Uncertainty = 0< Probability < 1
"Complete Knowledge" and "Certainty" mean the same thing. If you disagree, provide a consistent mathematical definition for both.

Not in QT. The point is that in a given pure state ("complete knowledge") never all observables take determined values.

The second part of your statement (bold) illustrates the problem. A pure state does not mean "complete knowledge" of the system it means complete information about the preparation procedure. That's why non-compatible observables are uncertain.

Within QT knowing that the system is prepared in a pure state means that you have "complete knowledge". It's simply not possible to prepare the system "more precisely" in any sense than preparing it in a pure state. That's also why the von Neumann entropy of a pure state is 0, i.e., then there's no "missing information".

 

[Fra]

I would say that the overall "uncertainty" in predictions has at least two sources, beyond the simple "ignorance" we discussed....

1) Non-commutativity of what we want to predict, this is new to QM

2) Uncertainty because in a real inference you never attain infinite confidence in finite time, this we have also in classical mechanics

(1) this is not even a postualte, it follows from the NEW definition of observables. Those that keep thinking p and x can be known with arbitrary precision in QM, are simply confusing the classical mechanics _definiton_ of the generalized momenat, with the quantum mechanical defintions with conjugate variables. This is a simple property of the fourier transform defining all conjugate variables.

(2) A "pure state" in QM, means we have infinide confidence in the state preparation. This is of course not possible exactly. It's a fiction, just as is zero entropy. From discussions on here, we clearly disagree wether this is a "practical issue" of no fundamental importance for understanding the nature of interactions or not. I think this is a practical issue in classical mechanics, but at least not in the way I "interpret" QM in the context of trying to acheive unification, including gravity. I find that this "practical issue" in rather a potential natural regulator, that can bring order to things which today is a mess of divergences and fine tuning. At least I think it's an objective possibility, we can't rule it out.

In this thread, my main focus was on (2), which per see, has nothing to do with bells theorem. I think it's helpful to distinguish the cases here.

/Fredrik

 

[Morbert]

A pure state does not mean "complete knowledge" of the system it means complete information about the preparation procedure.

I think this statement needs to be refined, as many different procedures can produce the same state. The state does not tell us the name of the physicist doing the preparing, for example.

An instrumentalist might say something like a pure state is complete insofar as it minimizes von Neumann entropy.

 

[PeterDonis]

There are a lot of things that we can be certain are impossible.

Which contradicts your previous claim.

But claiming to know that something cannot be known

Nobody made such a claim. The claim that was made is that it might be the case that that something cannot be known. Nobody claimed that we know that is the case. But you were claiming that "something cannot be known" was not even a possibility at all. Now you are contradicting yourself.

is quite different than a claim postulating that you can't travel faster than the speed of light.

It isn't a postulate in relativity, it's a consequence of the theory. If the theory eventually turns out to be only an approximation to some other theory, that some other theory might say something different about this.

But even if it is granted that the original claim was just a postulate that the uncertainty in QT is not due to ignorance (afaik there's no such postulate in QT), there's still a burden to define those terms in a non-contradictory way and I haven't seen such definitions.

I don't know what you're talking about. Such definitions have been given in this thread. You might not like them, but that doesn't mean they aren't there.

 

[vanhees71]

Which contradicts your previous claim.Nobody made such a claim. The claim that was made is that it might be the case that that something cannot be known. Nobody claimed that we know that is the case. But you were claiming that "something cannot be known" was not even a possibility at all. Now you are contradicting yourself.

I'd say the "standard claim" of QT is not only that "something cannot be known" (i.e., the values of all observables of any given system at once) but it says that indeed in Nature there's no state that the values of all observable of a system cannot take determined values. It's not about subjective ignorance, for whatever reason, but about objective behavior of Nature.

Of course, this claim can never be proven mathematically. No physical content, in fact, can be proven by pure math/thought, but we can check it by experiments/observations in Nature.

Bell's work is so important in this matter, because it translated a vague philosophical claim by EPR into a clear scientific hypothesis, which can be tested against the predictions of QT, i.e., the assumption of "realism" (i.e., all values of all observables always take determined values) and "locality" (i.e., that properties of far-distant parts of any system are independent, which should rather be called "separability" than "locality", as discussed many times, controversally, in this forum), leading to Bell's inequalities, which are violated by the predictions of QT for entangled quantum systems.

All corresponding tests lead to the conclusion that QT is right (Nobel prize in physics 2022).

It isn't a postulate in relativity, it's a consequence of the theory. If the theory eventually turns out to be only an approximation to some other theory, that some other theory might say something different about this.

I thought we discuss QT here. The quibbles of EPR of course apply in both non-relativistic QM and relativistic QFT.

Of course, as any physical theory also QT is open for revision, as soon as new, reproducible empirical facts become known, which cannot be explained by it. It's pretty clear that QT is not complete, as long as we don't have a consistent quantum theory including the gravitational interaction. Unfortunately there are no empirical hints at, how a more comprehensive theory might look like.

 

[PeterDonis]

I thought we discuss QT here.

QFT, which depends on SR, is part of QT.

 

[A. Neumaier]

You don't know how to do it because it is impossible to do it. No matter how much effort you expend, you will never be able to figure out how to do it because it is impossible.

No, but you can consider that as a possibility if you've been trying and trying and not succeeding.

But considering it as a possibility is different from knowing that it is impossible. Thus if we don't know how we also don't know whether it is impossible to know or whether we just haven't been able to figure it out how.

 

[A. Neumaier]

t says that indeed in Nature there's no state that the values of all observable of a system cannot take determined values. It's not about subjective ignorance, for whatever reason, but about objective behavior of Nature.

This is correct only when you replace Nature by QT, and 'values of observables' by 'eigenvalues of operators corresponding to observables'. Equating these is an interpretation, not as claim of QT !

 

[Lord Jestocost]

QFT, which depends on SR, is part of QT.

The most important "philosophical" or rather "interpretative" questions raised by quantum mechanics also remain with the transition to quantum field theory.

 

[vanhees71]

This is correct only when you replace Nature by QT, and 'values of observables' by 'eigenvalues of operators corresponding to observables'. Equating these is an interpretation, not as claim of QT !

That's part of the minimal statistical interpretation. A mathematical scheme without any relation to observable facts about Nature is not a physical theory. "Values of observables" are readings on a measurment device.

 

[A. Neumaier]

That's part of the minimal statistical interpretation.

But the minimal interpretation is an interpretation. QT has different interpretations. For example, what you claim is wrong in the maximal, thermal interpretation.

A mathematical scheme without any relation to observable facts about Nature is not a physical theory.

But a minimal interpretation is deliberately silent (by minimality) about anything more than guaranteed by the observations. Assertions about irreducible randomness are no longer minimal.

"Values of observables" are readings on a measurment device.

The only thing standard QT says about these is that in experiments that can be described well by idealized von-Neumann-measurements, these values are determined stochastically by Born's rule. The latter contains no statement about the reasons for the observed randomness.

 

[vanhees71]

But the minimal interpretation is an interpretation. QT has different interpretations. For example, what you claim is wrong in the maximal, thermal interpretation.

What is the maximal, thermal interpretation? Is it what you hitherto called "thermal interpretation"? With this I still have my quibbles of understanding. So I can't really comment on it.

But a minimal interpretation is deliberately silent (by minimality) about anything more than guaranteed by the observations. Assertions about irreducible randomness are no longer minimal.

In the minimal statistical interpretation you claim that all there is are the probabilities for the outcomes of measurements, and nothing else, not described by QT, which implies that there is "irreducible randomness". I always thought that's why many (usually philosophically inclined) physicists have their epistemological problems with it. Einstein seems to have been a proponent of this interpretation, but at the same time he was dissatisfied with this conclusion and thus thought QT to be "incomplete", i.e., there should indeed be a deterministic (or "realistic") description, and that's why he looked for his unified classical field theories.

The only thing standard QT says about these is that in experiments that can be described well by idealized von-Neumann-measurements, these values are determined stochastically by Born's rule. The latter contains no statement about the reasons for the observed randomness.

The reasons for physical theories are usually that it works, i.e., it's in accordance with the observations, and that's the case also for QT ;-)).

 

[A. Neumaier]

What is the maximal, thermal interpretation? Is it what you hitherto called "thermal interpretation"?

It is just the thermal interpretation, with 'maximal' added for emphasis and contrast, since it contains the minimal interpretation, but also much more that is not addressed in the latter.

In the minimal statistical interpretation you claim that all there is are the probabilities for the outcomes of measurements,

Yes.

and nothing else,

No. This is not the minimal interpretation, but your addition to it. It makes an absolute, empirically not checkable and hence purely philosophical statement about Nature.

The minimal interpretation drops this, and hence is more minimal than what you like to call minimal. It is the consensus part where all interpretations agree.

not described by QT, which implies that there is "irreducible randomness". I always thought that's why many (usually philosophically inclined) physicists have their epistemological problems with it.

Yes, because it is an extremely strong additional philosophical assumption, not needed to explain the success of QT!

The reasons for physical theories are usually that it works, i.e., it's in accordance with the observations, and that's the case also for QT ;-)).

For this one only needs the minimal interpretation, and not your nonminimal addition 'and nothing else'!

 

[Morbert]

In the minimal statistical interpretation you claim that all there is are the probabilities for the outcomes of measurements, and nothing else, not described by QT, which implies that there is "irreducible randomness". I always thought that's why many (usually philosophically inclined) physicists have their epistemological problems with it. Einstein seems to have been a proponent of this interpretation, but at the same time he was dissatisfied with this conclusion and thus thought QT to be "incomplete", i.e., there should indeed be a deterministic (or "realistic") description, and that's why he looked for his unified classical field theories

One charge against the minimal ensemble interpretation noted by Home and Whitaker is that the interpretation dismisses foundational problems rather than solves them.

Though Squires considers: “This is a perfectly reasonable view, and it may be the correct one to take”, it cannot be said he actually supports it: “We must not, however, go on to claim that we have solved the problems... We have merely ignored them. We do not only have experimental results for ensembles. Individual systems exist and the problems arise when we observe them. It is possible to argue that quantum theory says nothing about such individual systems but, even if this is true, the problems do not go away.”

[edit] - Broken link. Here is the explicit link https://citeseerx.ist.psu.edu/viewd...E21C5C27?doi=10.1.1.675.655&rep=rep1&type=pdf

 

[vanhees71]

Which problems? If you simply accept that QT is a complete description, then you must simply accept the "irreducible randomness", and there are no more problems, which only arise, because you claim that QT were incomplete, because you think that Nature must be deterministic, and thus only a deterministic theory can be complete, i.e., the only "allowed randomness" is due to ignorance, as in classical statistical physics.

Of course, you can never prove that any theory is a complete description, because it may well be that you find some empirical evidence that clearly contradicts the theory, and then you need to find a new theory.

In the case of QT it's also clear that it's intrinsically incomplete, because there's no satisfactory theory of the gravitational interaction compatible with it, but this may or may not be related to the probabilistic Nature of QT, i.e., whether a more comprehensive theory, including gravitation, will be deterministic again or not, one cannot say without having found this theory.

 

[A. Neumaier]

If you simply accept that QT is a complete description, then you must simply accept the "irreducible randomness"

No. The thermal interpretation accepts the first but not the second, because, compared to Born's rule it has a more comprehensive interpretation of the relation between QT and experiment.

Of course, you can never prove that any theory is a complete description,

This is why a minimal interpretation cannot accept this as part of the interpretation. Since you accept it, your interpretation is not minimal.

 

[Morbert]

Which problems? If you simply accept that QT is a complete description, then you must simply accept the "irreducible randomness", and there are no more problems, which only arise, because you claim that QT were incomplete, because you think that Nature must be deterministic, and thus only a deterministic theory can be complete, i.e., the only "allowed randomness" is due to ignorance, as in classical statistical physics.

Of course, you can never prove that any theory is a complete description, because it may well be that you find some empirical evidence that clearly contradicts the theory, and then you need to find a new theory.

In the case of QT it's also clear that it's intrinsically incomplete, because there's no satisfactory theory of the gravitational interaction compatible with it, but this may or may not be related to the probabilistic Nature of QT, i.e., whether a more comprehensive theory, including gravitation, will be deterministic again or not, one cannot say without having found this theory.

I am fairly sympathetic to the position that QT can be considered complete under the appropriate interpretations. More specifically, if we conceptualize an ensemble, the state is a complete description of the ensemble. Note that this is distinct from saying a single system is completely described by the state. This relations diagram conveys Ballentine's position at least.

 

[PeterDonis]

considering it as a possibility is different from knowing that it is impossible

Agreed. But it can still be considered as a possibility in a reasonable discussion of physics. The post by @lodbrok that I responded to was denying that.

 

[PeterDonis]

If you simply accept that QT is a complete description

The minimal interpretation makes no such claim. It does not say that QT is complete or that it is incomplete. It is simply silent on all such matters. So, as @A. Neumaier has already commented, you are clearly not using the minimal interpretation.

 

[vanhees71]

This is again splitting hairs. I only said that it is a logical possibility to assume that indeed not all observables take always predetermined values. Then QT, in the minimal statistical interpretation, can be "considered complete". I did not claim that QT were complete. IMHO none of our current theories is complete (but for different reasons than the "foundational problems" of QT, i.e., the lack of a description of the gravitational interaction that is consistent with Q(F)T).

 

[PeterDonis]

This is again splitting hairs.

No, it isn't. It is, once again, observing that you refuse to be consistent in your use of terms.

I only said that it is a logical possibility to assume that indeed not all observables take always predetermined values. Then QT, in the minimal statistical interpretation, can be "considered complete".

Which, again, is not the minimal interpretation. The minimal interpretation does not say anything about "logical possibilities" or whether QT can be "considered complete". It is silent on all such matters.

It is very confusing to me that you, who have repeatedly claimed that you have no interest in interpretation discussions, continue to post in them, without, apparently, even understanding your own viewpoint.

I did not claim that QT were complete.

Really?

If you simply accept that QT is a complete description

I wonder if you even read what you post.

 

[Lynch101]

Which problems? If you simply accept that QT is a complete description, then you must simply accept the "irreducible randomness", and there are no more problems, which only arise, because you claim that QT were incomplete, because you think that Nature must be deterministic, and thus only a deterministic theory can be complete, i.e., the only "allowed randomness" is due to ignorance, as in classical statistical physics.

Of course, you can never prove that any theory is a complete description, because it may well be that you find some empirical evidence that clearly contradicts the theory, and then you need to find a new theory.

In the case of QT it's also clear that it's intrinsically incomplete, because there's no satisfactory theory of the gravitational interaction compatible with it, but this may or may not be related to the probabilistic Nature of QT, i.e., whether a more comprehensive theory, including gravitation, will be deterministic again or not, one cannot say without having found this theory.

What is it a complete description of though? Is it a complete description of a statistical sample (or "ensemble") or is it a complete description of each individual element of the statistical sample (or "ensemble")?

 

[PeterDonis]

What is it a complete description of though? Is it a complete description of a statistical sample (or "ensemble") or is it a complete description of each individual element of the statistical sample (or "ensemble")?

Since this thread is about the statistical ensemble interpretation, we are discussing here interpretations of the first type (with the caveat that "ensemble" is not the same thing as "statistical sample"--this was discussed earlier in the thread). However, AFAIK not all versions of the statistical ensemble interpretation claim that the description given by QT of the statistical ensemble is complete.

 

[Lynch101]

Since this thread is about the statistical ensemble interpretation, we are discussing here interpretations of the first type (with the caveat that "ensemble" is not the same thing as "statistical sample"--this was discussed earlier in the thread). However, AFAIK not all versions of the statistical ensemble interpretation claim that the description given by QT of the statistical ensemble is complete.

Cheers PD. I read through the thread alright and saw the discussion about the use of the term "ensemble". I put it in brackets and quote marks moreso because Vanhees uses the term in place of statistical sample.

To what extent would you say the SEI (or perhaps some expressions of it) are actually interpretations as distinct from "shut up an calculate"?

 

[PeterDonis]

I put it in brackets and quote marks moreso because Vanhees uses the term in place of statistical sample.

And, as I said, that has been corrected in previous discussion. The quantum state in the statistical ensemble interpretation represents, as the name says, the ensemble (with the definition given in Ballentine that I posted earlier), which, as I said, is not the same thing as the finite statistical sample that we actually get from running experiments.

To what extent would you say the SEI (or perhaps some expressions of it) are actually interpretations as distinct from "shut up an calculate"?

This question is unanswerable. If you have a specific reference from the literature or a specific post earlier in this thread, you can ask about that.

 

[vanhees71]

What is it a complete description of though? Is it a complete description of a statistical sample (or "ensemble") or is it a complete description of each individual element of the statistical sample (or "ensemble")?

Of course, probabilistic notions make only sense for statistical samples (as proxies of an ensemble). That's very accurate lnguage, and we agreed to use it in this thread. In everyday discussions among physicsts (particularly experimentalists) "ensemble" is the usual lingo. Nobody talks about "statistical samples", but that's admittedly imprecise and may lead to confusion when it comes to fine details about interpretation.

 

[Lord Jestocost]

“The standard empirical interpretation of quantum mechanics is already statistical. However, a statistical (ensemble) interpretation can also be treated as a semantic interpretation which provides an understanding of empirical data. In contrast to the standard (Copenhagen) interpretation, the statistical interpretation does not refer to an individual object but it refers to a collective (ensemble) of similarly prepared ones.”

Alexander Pechenkin in “The Statistical (Ensemble) Interpretation of Quantum Mechanics” (Chapter 50 of “The Oxford Handbook of the History of Quantum Interpretations”, Oxford University Press (2022))

 

[Lynch101]

Since this thread is about the statistical ensemble interpretation, we are discussing here interpretations of the first type (with the caveat that "ensemble" is not the same thing as "statistical sample"--this was discussed earlier in the thread). However, AFAIK not all versions of the statistical ensemble interpretation claim that the description given by QT of the statistical ensemble is complete.

In those versions of the SEI which claim that the description of the statistical ensemble (given by QT) is complete, do they say anything about the process by which the statistical ensemble becomes populated, as in, the process whereby the individual elements of the ensemble come to be part of the ensemble?

Am I using the correct terminology when I say that the finite statistical sample used in experiments, acts as a proxy for testing the predictions QT makes with regard to an abstract ensemble?

The thing I'm trying to get at is, I know there is an experimental process which gives rise to the statistical sample. I'm just wondering if the aforementioned (or indeed any) versions of SEI describe both the process and the ensemble (where the statistical sample acts as a proxy for the ensemble), or does it just describe the ensemble?