|
Author
|
Topic: A 4th Law of Thermodynamics
|
Pim van Meurs
Member
Member # 541
|
posted 09. November 2003 01:20
Erik: Pim van Meurs, my confusion comes down to your very imprecise way of expressing yourself and the fact that you do not clarify yourself. Your latest reply quotes some of my comments on Shannon entropy and thermodynamical entropy. Then you proceed to comment on Shannon information and "thermodynamical information" (what is that?) as if that had something to do with the quoted statement.
I thought I had fixed these references, sorry but the response remains basically the same. Let's not get distracted by these issues.
Erik: I question the coherence of your ideas, so any reply that takes such liberties with the terminology is worthless.
I am sorry that you are confused by what I see to be minor distractions. The basic argument remains the same, there is a direct simple mapping between shannon and thermodynamical entropy which shows that entropy in shannon sense can be expressed in thermodynamical terms as well.
I refer you to Schneider's excellent paper on this issue.
quote:
H Uncertainty of the machines microstates
Sb: The Boltzmann-Gibbs entropy of a physical system, such as a molecular machine
Sb=kb ln(2) H
The only difference between uncertainty and entropy for the microstates of a macromolecule is in the units of measure, bits versus joules per respectively
Or the following paper The Boltzmann/Shannon entropy as a measure of correlation
quote:
Jaynes showed how Shannon's theory could be merged with statistical mechanics, leading to the conceptualization of the thermodynamic principle
of maximum entropy as a principle expressing that the distribution of energy among the microstates should be that distribution which is least-biased, given the constraint of a specified temperature.
Or
A short account of a connection of Power Laws to the Information Entropy
quote:
Since the Shannon entropy is equivalent to the Boltzmann's entropy under equilibrium, non interacting conditions, we interpret this result as the complex system making use of its intra-interactions and its non equilibrium in order to keep the equilibrium Boltzmann's entropy constant on the average, thus enabling it an advantage at surviving over less ordered systems, i.e. hinting towards an "Evolution of Structure".
and
quote:
The nomenclature "entropy" was adopted because
the Shannon entropy was shown by Jaynes (1957) to be identical to the classical Boltzmann entropy.
link
I found the following comment intruiging as it pertains to the SLOT
quote:
Maximization of S subject to available information yields the least-biased probability assignment over the quantum states of the system. Since the theoretical function S in the form (7) is invariant under unitary transformation, it is often argued that this expression cannot describe the second law. But Jaynes (Ref.34) later demonstrated that, in fact, it is just this property that allows one to derive the second law, which is a statement about experimental entropy.
Link
[ 09. November 2003, 02:07: Message edited by: Pim van Meurs ]
IP: Logged
|
|
Doron Shadmi
Member
Member # 965
|
posted 09. November 2003 05:59
High Entropy means maximum level of redundancy and uncertainty, which are based on the highest symmetry-degree of some system.
For example let us say that there is a piano with 3 notes and we call it 3-system :
DO=D , RE=R , MI=M
The highest Entropy level of 3-system is the most left information's-tree,
where each key has no unique value of its own, and vice versa.
code:
<-Redundancy->
M M M ^<----Uncertainty
R R R | R R
D D D | D D M D R M
. . . v . . . . . .
| | | | | | | | |
3 = | | | |___|_ | |___| |
| | | | | | |
|___|___|_ |_______| |_______|
| | |
An example of 4-notes piano:
DO=D , RE=R , MI=M , FA=F
code:
------------>>>
F F F F F F F F
M M M M M M M M
R R R R R R R R R R R R R R
D D D D D D D D D R D D D D D D
. . . . . . . . . . . . . . . .
| | | | | | | | | | | | | | | |
| | | | |__|_ | | |__| | | |__|_ |__|_
| | | | | | | | | | | |
| | | | | | | | | | | |
| | | | | | | | | | | |
|__|__|__|_ |_____|__|_ |_____|__|_ |_____|____
| | | |
4 =
M M M
R R R R R R R
D R D D D R D R D D D F D D M F
. . . . . . . . . . . . . . . .
| | | | | | | | | | | | | | | |
|__| |__|_ |__| |__| | | | | |__|_ | |
| | | | | | | | | | |
| | | | |__|__|_ | |_____| |
| | | | | | | |
|_____|____ |_____|____ |________| |________|
| | | |
D R M F
. . . .
| | | |
|__| | |
| | |
|_____| |
| |
|________|
|
These examples are based on the idea of associations between opposite (or complementary) concepts, and in this case the concepts are discreteness {...} and continuum {___}.
No one of these concepts alone can be an environment for complex phenomenon.
[ 09. November 2003, 06:22: Message edited by: Doron Shadmi ]
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 09. November 2003 13:21
Pim van Meurs, am I discussing with someone capable of thinking and expressing own thoughts or am I discussing with a quote-miner incapable of responding in any other way than by listing quotes of peripheral relevance?
Concerning a particular mix-up between "entropy" concepts and "information" concepts, Pim van Meurs wrote: "I thought I had fixed these references, sorry but the response remains basically the same. Let's not get distracted by these issues."
The problem is that these issues are not mere distractions. You can barely write a sentence without getting something wrong. This would not be a problem if it is was obvious what you actually meant and how to rewrite your claims in a correct form. But it appears impossible to find an coherent interpretation of what you really mean. If you tried to get it right, you would discover that your statements are mainly of two types: either exceedingly trivial (and true) or very wrong. However, because of your insistence to avoid any clarifications and instead reply using lists of quotes that you've mined from the net, you will likely not discover this yourself.
quote:
Erik: I question the coherence of your ideas, so any reply that takes such liberties with the terminology is worthless.
Pim van Meurs: I am sorry that you are confused by what I see to be minor distractions. The basic argument remains the same, there is a direct simple mapping between shannon and thermodynamical entropy which shows that entropy in shannon sense can be expressed in thermodynamical terms as well.
OK, Pim van Meurs. I argue that your ideas are incoherent. You reply that this is "minor distraction", refuse to clarify anything, and repeat your old claims. What do you hope to accomplish with that reply?
In response to my challenge for Pim van Meurs to show in detail and his own words how Dembski's LCI can be translated into the second law of thermodynamics, I got five quotes without any significant explanation by Pim as to why he thinks the quotes are relevant and exactly what he think they show. The first quote comes from a web page by Schneider, where he "derives" an exceedingly trivial relation between the thermodynamical entropy and the Shannon entropy of the particular stochastic variable MICROSTATE (defined above).
The second quote is from a paper that shows that if we transform stochastic variables in a way that destroys correlations between them, then the joint (Shannon) entropy of the transformed variables must be larger than the joint entropy of the original variables. This means that we may regard joint entropy as an inverse measure of correlation. Sure we can, but how is that relevant to anything discussed in this thread? The quote from that article is about Ed Jaynes's approach to statistical mechanics. To understand that quote we should remember that Jaynes was a hardcore Bayesian. Before his recent departure, he struggled to finish a book on Bayesian statistics. We may note Chapter 2 of that book, which presents the R.T. Cox's argument that the simplest consistent way of reasoning about propositions, whose truth or falsity are uncertain (in the sense of being unknown), is to use Bayesian inference. That is, we should assign probabilities to statements and consider probability theory to be a kind of logic of statements, whose truth values are more or less unknown. This reduces to ordinary logic if we only care about inferences that hold with probability one. Thus, Jaynes considered probability to be a measure of human certainty, i.e. for him the statement "the probability of the event E is 0.46" is not a statement about the real-world, but rather an autobiographical statement meaning roughly "given my personal state of knowledge, my degree of certainty that E will occur is 0.46". When doing Bayesian inference, we need to figure out likelihoods and a priori probabilities. Statistical hypotheses are normally formulated in such a way that the likelihood is obvious. Thus, the use of likelihoods is uncontroversial. A Bayesian also needs to figure out the a priori probability of the hypothesis of interest. This is a matter of considerable controversy and the use of (apparently arbitrarily/subjectively chosen) a priori probabilities is the target of much criticism of Bayesian inference. Jaynes, being a hardcore Bayesian, felt the need to provide good answers to those who disagreed with his view of the meaning of probability and how inference should be done. It appears that he wanted people to agree that his version of Bayesianism was the way to reason about uncertainty. Jaynes's answer to those who found the use of a priori probabilities arbitrary was that those probabilities can and should be determined from general considerations, such as invariance under transformations and the Principle of Maximum Entropy. The latter states that we should pick the a priori probabilities that maximize the Shannon entropy of the possible outcomes, subject to the constraint that the probabilities must be consistent with our a priori knowledge.
A possible objection to Jaynes's view is this: Surely at least some probabilities are properties of the real world rather than of our state of knowledge? Surely the probabilities of statistical mechanics and quantum mechanics are not merely representations of our ignorance? Interested readers may check Chapter 10 of the book mentioned above for Jaynes's rather lame attempt to deal with QM. He was more successful with statistical mechanics. In the article
Jaynes E. T. (1957) "Information Theory and Statistical Mechanics", Physical Review, 106(4):620-630
he showed that statistical mechanics can be considered as merely a particular application of Bayesian inference using a MaxEnt prior. Jaynes recovered all of equilibrium statistical mechanics as Bayesian inferences resulting from the a priori probability distribution over energy states that has the maximum Shannon entropy consistent with some physical a priori knowledge. It should be noted that Jaynes considered statistical mechanics to be just another application of MaxEnt Bayesianism (hence the title of section 3 of the paper: "Application to statistical mechanics"). He did not, as Pim might think, consider Shannon entropy to be a special case of thermodynamical entropy. Here are some representative quotes to verify my above description of Jaynes:
quote:
"It is now evident how to solve our problem; in making inferences on the basis of partial information we must use that probability distribution which has maximum entropy subject to whatever is known. This is the only unbiased assignment we can make; to use any other would amount to arbitrary assumption of information which by hypothesis we do not have." (p. 623)
"The principle of maximum entropy may be regarded as an extension of the principle of insufficient reason (to which it reduces in case no information is given except enumeration of the possibilities x_i), with the following essential difference. The maximum-entropy distribution may be asserted for the positive reason that it is uniquely determined as the one which is maximally noncommittal with regard to missing information, instead of the negative one that there was no reason to think otherwise. Thus the concept of entropy supplies the missing criterion of choice which Laplace needed to remove the apparent arbitrariness of the principle of insiffucient reason, and in addition shows precisely how this principle is to be modified in case there are reasons for 'thinking otherwise.'" (p. 623)
"It is interesting to note the ease with which these rules of calculation are set up when we make entropy the primitive concept. Conventional arguments, which exploit all that is known about the laws of physics, in particular the constants of motion, lead to exactly the same predictions that one obtains directly from maximizing the entropy." (p. 624)
"Many of the propositions of statistical mechanics are capable of two different interpretations. The Maxwellian distribution of velocities in a gas is, on the one hand, the distribution that can be realized in the greatest number of ways for a given total energy; on the other hand, it is a well-verified experimental fact. Fluctuations in quantities such as the density of a gas or the voltage across a resistor represent on the one hand the uncertainty of our predictions, on the other a measurable physical phenomenon. Entropy as a concept may be regarded as a measure of our degree of ignorance as to the state of a system; on the other hand, for equilibrium conditions it is an experimentally measurable quantity, whose most important properties were first found empirically. It is this last circumstance that is most often advanced as an argument against the subjective interpretations of entropy.
The relation between and maximum-entropy inference and experimental facts may be clarified as follows. We frankly recognize that the probabilities involved in prediction based on partial information can have only a subjective significance, and that the situation cannot be altered by the device of inventing a fictitious ensemble, even though this enables us to give the probabilities a frequency interpretation. One might then ask how such probabilities could be in any way relevant to the behavior of actual physical systems. A good answer to this is Laplace's famous remark that probability theory is nothing but 'common sense reduced to calculation.' If we have little or no information relevant to a certain question, common sense tells us that no strong conclusions either way are justified. The same thing must happen in statistical inference, the appearance of a broad probability distribution signifying the verdict, 'no definite conclusion.'" (p. 626)
"The essential point in the arguments presented above is that we accept the von-Neumann--Shannon expression for entropy, very literally, as a measure of the amount of uncertainty represented by a probability distribution; thus entropy becomes the primitive concept with which we work, more fundamental even than energy. If in addition we reinterpret the prediction problem of statistical mechanics in the subjective sense, we can derive the usual relations in a very elementary way without any consideration of ensembles or appeal to the usual arguments concerning ergodicity or equal a priori probabilities. The principles and mathematical methods of statistical mechanics are seen to be of much more general applicability than conventional arguments would lead one to suppose. In the problem of prediction, the maximization of entropy is not an application of a law of physics, but merely a method of reasoning which ensures that no unconscious arbitrary assumptions have been introduced." (p. 629-630)
The last quote is his conclusion section, and we may take particular notice of the last sentence.
The third quote comes from an article attempting to establish the basics of an as-yet unfinished theory of certain complicated systems. The authors try to derive general predictions from the general principle that the thermodynamical entropy is constant. The quoted statement means that the Shannon entropy of a system's state is equivalent to its thermodynamical entropy. This is clear from their actual equations. If you, Pim, had also provided detailed calculations you could also afford to neglect to write out, in your main text, which stochastic variable a Shannon entropy concerns. But since your posts only contain unclear non-mathematical statements, you should resist that temptation.
The last quote concerns the von Neumann entropy of a physical system. I have to wonder what you are trying to say by including that quote (do you know what a density matrix is?). Although the trace of the operator rho log(rho) can be rewritten in a way that looks like a Shannon entropy*, the von Neumann entropy is introduced in QM to remedy certain flaws of the Shannon entropy (namely that the latter cannot distinguish coherent superpositions from statistical mixtures).
Erik
* Diagonalize the operator rho and remember that the eigenvalues of rho are probabilities defining the statistical mixture.
[ 09. November 2003, 13:39: Message edited by: Erik ]
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 09. November 2003 17:04
Erik: Pim van Meurs, am I discussing with someone capable of thinking and expressing own thoughts or am I discussing with a quote-miner incapable of responding in any other way than by listing quotes of peripheral relevance?
Lets not distract from these issues by using such unwarranted ad hominems Erk. Your posting consists of many such unnecessary snipes.
Examples: "do you know what a density matrix is?). ", "But since your posts only contain unclear non-mathematical statements, you should resist that temptation.", "However, because of your insistence to avoid any clarifications and instead reply using lists of quotes that you've mined from the net, you will likely not discover this yourself"
You complained about me getting things 'wrong' thus I circumvented this spiral of avoidance and went straight to the source.
I provided my references, you still focus on what you consider to be confusing statements on my part such as "He did not, as Pim might think, consider Shannon entropy to be a special case of thermodynamical entropy". Lets forget about what I said and focus on the equivalency of Shannon and Boltzmann entropy lest we run the risk of having this thread spiral out of control in ad hominems.
For this reason I am more than willing to accept your premise that you consider my explanations to be confusing. In fact I am even willing to accept that they are confusing since this is brainstorming.
The issue I am interested in is the (existance of an) equivalence between Shannon and Boltzmann entropy. I assume we share this interest?
So is the following wrong?
quote:
The nomenclature "entropy" was adopted because the Shannon entropy was shown by Jaynes (1957) to be identical to the classical Boltzmann entropy.
The paper on correlation and Shannon/Boltzmann entropy suggests that the SLOT may not limited to just Boltzmann entropy.
[ 09. November 2003, 18:11: Message edited by: Pim van Meurs ]
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 10. November 2003 17:41
quote:
Erik: Pim van Meurs, am I discussing with someone capable of thinking and expressing own thoughts or am I discussing with a quote-miner incapable of responding in any other way than by listing quotes of peripheral relevance?
Pim van Meurs: Lets not distract from these issues by using such unwarranted ad hominems Erk. Your posting consists of many such unnecessary snipes.
It is not an unwarranted question. I want to know if you are ever going to post some of your own reasoning or if you will continue to merely list quotes indefinitely. The internet is full of quotes that seem interesting to you, yet whose precise meaning eludes you, and I cannot address them all. You will never run out of quotes. I can, however, comment on your reasoning (or at least I could, were you determined to avoid any clarifications).
The highly offensive thing about your debating tactic is the systematic asymmetry in workload. You post short unsubstantiated claims. I try to explain in detail what's wrong and what's unclear. You reply with links and quotes, without any non-trivial comments of your own. I read all of your links and quotes and explain to you what they really mean. You completely ignore my corrections (it would be fine if you disputed them using arguments that you understand and can express in your own words, but you ignore them) and repeat your old claims. I find that style very disrespectful, in a much less superficial way than an ad hominem argument--intended or perceived--would have been.
quote:
Pim van Meurs: The issue I am interested in is the (existance of an) equivalence between Shannon and Boltzmann entropy. I assume we share this interest?
Well, I already know exact relation between
(i) the mathematical form of Shannon and Boltzmann entropy,
(ii) the mathematical meaning of Shannon and Boltzmann entropy, and
(iii) which properties of thermodynamical entropy that are general consequences of its mathematical form and which properties that depend on physical results (such as energy conservation, equations of motions, etc.) that have no analogues in general.
I post in this thread to counter-balance some incorrect claims about these relations.
quote:
Pim van Meurs: So is the following wrong?
"The nomenclature 'entropy' was adopted because the Shannon entropy was shown by Jaynes (1957) to be identical to the classical Boltzmann entropy."
It depends on exactly what you think it means. Do you think it is only convention among MaxEnt Bayesians? Do you think it means that the properties of thermodynamical entropy will automatically hold for any Shannon entropy? Do you think the authors take "entropy" to be a property of the real world or a property of human knowledge/ignorance?
quote:
Pim van Meurs: The paper on correlation and Shannon/Boltzmann entropy suggests that the SLOT may not limited to just Boltzmann entropy.
The paper suggests that:
(i) "The Second Law of Thermodynamics may be rephrased to state that correlations are highly unlikely to arise spontaneously, and that the natural course of evolution of a systems is one in which correlations diminish."
(ii) "Thinking about entropy as a measure of correlation leads to a key implicit assumption in both Boltzmann's theory and Shannon's theory: the individual (micro)states are assumed to be uncorrelated. [...] In communication theory and statistical mechanics, this assumption may in certain circumstances be valid, but where this assumption breaks down severely is the case when we attempt to take the limit to a continuous set of states."
I have some reservations about (i) and (ii), but for the purposes of this discussion I'm content to note that neither (i) nor (ii) is in any way a suggestion that we can transfer any property of thermodynamical entropy (e.g. non-decreasing in time for isolated systems) to any Shannon entropy.
------------------------------------
Finally, I observe that there are now two versions of the second law of thermodynamics that figures in this thread. The first version is the ordinary one in thermodynamics/statistical mechanics that takes the thermodynamical entropy of a system to be a property of the system, and it follows from it that the entropy of an isolated system will never decrease as long as it remains isolated (save for small fluctuations). This version is associated with irreversible physical dynamics and an arrow of time.
The second version is Jaynes's version. As already mentioned, he considered thermodynamical entropy to be a measure of human knowledge/ignorance, and using a priori knowledge of physics he recovered equilibrium statistical mechanics. His version of the second of law thermodynamics is a statement, not about physical systems, but about our knowledge/ignorance of the state of a physical system. This version of the second law is about irreversible increase of ignorance about the state of a system. It is not associated with an arrow of time; in fact entropy increases both forward and backward in time!!! That is, if the state of a system is partially or fully specified at a given time, then the uncertainty in our predictions of its state will increase both backwards and forwards in time (because the physical a priori knowledge that Jaynes exploits is time-symmetric).
When you refer to "the second law of thermodynamics", which of these two versions are you referring to? The objective arrow-of-time version or the subjective human ignorance version?
Erik
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 10. November 2003 18:30
Your continued ad hominem attacks make me conclude that further discussion with you does not serve the brainstorming nature of this forum.
If you consider the workload to be highly asymetric then you may want to consider that you any such effort is voluntary. And if this offends you then I point out to you that you have the option not to participate or limit your efforts.
Sorry that you seem to be unfamiliar with the true concepts of brainstorming.
[ 10. November 2003, 18:33: Message edited by: Pim van Meurs ]
IP: Logged
|
|
Rex Kerr
Member
Member # 632
|
posted 10. November 2003 19:02
Brainstorming isn't necessary when the answers are already known.
It's not yet clear to me, however, which answers are already known. For example, Erik posted a list of conditions for specified intelligence to be the same as thermodynamic entropy. But this didn't indicate under which conditions s.i. is nondecreasing as a consequence of the nondecreasing nature of thermodynamic entropy.
For instance, suppose we group microstates in sets of ten, completely at random, and let membership in one of these sets be our random variable. Then the second law of thermodynamics implies that the Shannon entropy is nondecreasing also.
Now, I will grant that specified information is broken because Dembski's presentation of probability is broken (e.g. by treating all outcomes as equiprobable), so that getting specified information to match up with anything is difficult at best. However, simply pointing out this flaw is uninformative; anyone familiar with classical or quantum statistical mechanics or information theory should be able to see how to fix Dembski's treatment to make it look a lot more like Shannon entropy, as it should.
So the question is: what is the consequence for Shannon entropy of the second law of thermodynamics? Specifically, what types of groupings of microstates are needed in order for the law to continue to apply, or something like the law to approximately apply? If Erik already knows the answer, hopefully he can tell us. If not, I think that is the question that is most useful to focus on.
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 10. November 2003 23:11
Rex: Brainstorming isn't necessary when the answers are already known.
Fair enough, perhaps I am just expecting a certain level of civility that I felt lacking in my interactions with 'Erik'.
IP: Logged
|
|
Moderator
Administrator
Member # 1
|
posted 11. November 2003 08:41
I'm going to have to side with Erik here, Pim. He has hit the nail on the head. To have meaningful discussions/brainstorms, you need to properly engage:
quote:
The highly offensive thing about your debating tactic is the systematic asymmetry in workload. You post short unsubstantiated claims. I try to explain in detail what's wrong and what's unclear. You reply with links and quotes, without any non-trivial comments of your own. I read all of your links and quotes and explain to you what they really mean. You completely ignore my corrections (it would be fine if you disputed them using arguments that you understand and can express in your own words, but you ignore them) and repeat your old claims. I find that style very disrespectful, in a much less superficial way than an ad hominem argument--intended or perceived--would have been.
Many people on this forum, not just Erik, have found your discussion style highly disrespectful, not because of viscious personal attacks but because of the way you trivialize the discussion by slapping up links, making assertions, etc.
Brainstorms moderators have been driving at this for some time, though Erik has articulated the problem in a much clearer way. Take note.
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 11. November 2003 17:59
quote:
Rex Kerr: For example, Erik posted a list of conditions for specified intelligence to be the same as thermodynamic entropy. But this didn't indicate under which conditions s.i. is nondecreasing as a consequence of the nondecreasing nature of thermodynamic entropy.
That's because the conditions require that the system is isolated and in its maximum entropy state.
quote:
Rex Kerr: For instance, suppose we group microstates in sets of ten, completely at random, and let membership in one of these sets be our random variable. Then the second law of thermodynamics implies that the Shannon entropy is nondecreasing also.
Since you began that paragraph with a "for instance" (implying that you are exemplifying a case when "specified information" is nondecreasing as a result of the second law of thermodynamics), I feel compelled to point out that "specified information" is not Shannon entropy.
Suppose we have twenty microstates. Initially the probabilities of the microstates are
0.5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.5.
Let's say that I have listed them in an order so that the first ten are for the first partition and the last ten are for the second partition. Let's call the microstate M and the other s.v. X (see below). Then initially we have
H(M) = 0.693 nats,
H(X) = 0.693 nats.
Now we wait for a while and the next time we check the probabilities of the microstates are
0.14, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.02, 0.5
The entropies then become
H(M) = 2.03 nats,
H(X) = 0.627 nats.
We see that H(M) has more than doubled, yet H(X) has decreased. Therefore your statement that the second law implies that both entropies must increase is wrong.
quote:
Rex Kerr: Now, I will grant that specified information is broken because Dembski's presentation of probability is broken (e.g. by treating all outcomes as equiprobable), so that getting specified information to match up with anything is difficult at best. However, simply pointing out this flaw is uninformative; anyone familiar with classical or quantum statistical mechanics or information theory should be able to see how to fix Dembski's treatment to make it look a lot more like Shannon entropy, as it should.
If your criterion for what counts as "fixing" Dembski's "specified information" is how similar the modification is to Shannon entropy, then I suppose I agree. However, Dembski's "specified information" was introduced for rhetorical reasons, not because the logarithm of the probability of a specification means something special.
Although Dembski's examples are cases where a uniform probability distribution is assumed, I don't think the intention is that his methods and conclusions should be limited to only uniform probability distributions (lest it would be weird formulate the design inference method as a method for eliminating an entire set of chance hypotheses). We could subdivide the specification into smaller parts, or even outcomes, and look at the probabilities for the parts. However, in Dembski's method only the probability of the whole specification is part of the decision criterion (this particular aspect is very sensible, I must say). If you modify "specified information" to make it depend on the probabilities of parts of the specification, then you break the (trivial) relation between the LCI and the Explanatory Filter.
quote:
Rex Kerr: So the question is: what is the consequence for Shannon entropy of the second law of thermodynamics? Specifically, what types of groupings of microstates are needed in order for the law to continue to apply, or something like the law to approximately apply? If Erik already knows the answer, hopefully he can tell us. If not, I think that is the question that is most useful to focus on.
Consider the sample space {1,2,...,W}. We define a stochastic variable M that takes on the same value as the outcome, i.e. if the outcome is 17 then M = 17. I encourage the reader to think of M as the microstate of a system. The probability distribution over M will be denoted p = (p(1),...,p(W)), so that
p(m) = Pr(M = m).
We also think of the sample space as divided into V different partitions C(1),...,C(V) (it is in the nature of partitions that they do not overlap and that they together make up the entire sample space). We now define the stochastic variable X, which takes the value k if the outcome (or value of M) is in the k:th partition. Thus, the event {X = k} is exactly the partition C(k). We denote the probability over X by q = (q(1),...,q(V)), so that
q(k) = Pr(X = k) = Pr(M in C(k)).
Having defined this notation, I think it is a good start to ask and answer these questions:
Question 1: What is the largest value H(X) can have when H(M) is maximal? The maximum H(M) = log(W) is attained only when p is a uniform distribution, i.e. p(m) = 1/W for all m. This immediately gives q(k) = |C(k)| / W. The entropy of X is given by
H(X) = log(V) - D(q||u),
where u = (1/V,...,1/V) is the uniform probability distribution over X and D(q||u) is the Kullback-Leibler divergence (or relative entropy). The Kullback-Leibler divergence is never negative and it is only zero when q = u.
Question 2: What is the largest value H(M) can have when H(X) is maximal? The maximum value H(X) = log(V) is attained when q is uniform. This allows for some freedom in the choice of p -- as long as the probabilities in partition C(k) add upp to q(k), for all partitions, the choice is arbitrary. The distribution that maximizes H(M) is the distribution that is uniform within each partition, so now p will be that distribution. If M = m belongs to the partition C(k), then
p(m) = q(k) / |C(k)|.
And the largest possible value of H(M) is
H(M) = log(W) - D(p||u') = log(V) + (1/V) sum log(C(k))
where u' = (1/W,...,1/W) is the uniform distribution over M.
The lesson from these questions is that H(X) and H(M) can be maximal simultaneously if and only if all partitions have the same number of elements.
--------------------------
Because of the way M and X are defined (knowing M is to know everything there is to know), we have that the conditional entropy H(X | M) = 0. The joint entropy can be written in two ways
H(M,X) = H(M) + H(X | M) = H(M) + 0
H(M,X) = H(X) + H(M | X)
it follows from this that
H(M) = H(X) + H(M | X).
The entropy H(M) consists of two components. The components are not completely independent, so some changes in p will result in one increasing and the other decreasing. Some changes will only affect one of them, and still other changes may increase or decrease both of them. The first term, H(X), measures the uniformity of q, i.e. the uniformity between partitions. The second term, H(M | X), measures the uniformity within partitions.
The same insight can be arrived at in a different way. We know that the sum of all probabilities is one. Taking the differential of this sum gives
sum p(m) = 1,
sum dp(m) = 0.
It follows from this that the differential of the entropy of M is
dH(M) = -sum d(p(m) log(p(m))) = -sum log(p(m)) dp(m)
(N.B. The above equation presupposes that we use base e logarithms.) And, analogously,
dH(X) = -sum log(q(k)) dq(k).
Even if dH(M) > 0 it is not necessarily true that dH(X) > 0. A simple example is if there is no "probability flow" between partitions so that all dq(k) = 0. It can also happen that dH(M) > 0 and 0 > dH(X) (see the example above with twenty microstates and two partitions). Think of it as a linear equation system, with dH(M), dH(X), and p fixed. As long as dH(M) and dH(X) are sufficiently small, we will in most cases be able to find a Deltap that results in the entropy changes we wished for.
This is not a complete answer to your question, but I hope it is enough to discourage general claims about the relation between H(M) and H(X).
Erik
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 11. November 2003 22:01
Erik,
What was the "trivial" relation between the LCI, and the explanatory filter?
[ 11. November 2003, 22:01: Message edited by: gedanken ]
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 13. November 2003 11:27
Gedanken, the relation between the LCI and the Explanatory Filter is that the LCI just says that the Explanatory Filter cannot fail. When we use the Explanatory Filter, we assume for the sake of the argument that an observed event was due to a particular non-ID hypothesis. We compute the probability of the event, under the assumption that it was due to our non-ID hypothesis. If this probability is lower than a threshold probability, Dembski's decision rule tells us to reject our particular non-ID hypothesis (and if we've rejected all non-ID hypotheses in this fashion, the decision rule tells us to conclude that the event was due to intelligent design).
The "specified information" of the event (w.r.t. a non-ID hypothesis) is just the (negative) logarithm of the probability of the specification. If the numerical value is larger than the (negative) logarithm of the threshold probability, then we say that the "specified information" is "complex", i.e. the event has CSI. An event has CSI w.r.t. a hypothesis precisely when the Explanatory Filter would reject the hypothesis.
Thus, the LCI, which asserts that natural processes cannot generate CSI, is just the assertion that natural processes cannot generate false positives of the Explanatory Filter.
For a different interpretation of the LCI, see Richard Wein's comments. Wein commented that it can be interpreted in the way I have outlined above, but he apparently considered this to be so trivial that it didn't need any further comments. Instead, he discussed the uniform-probability interpretation of the LCI. Based on Dembski's alleged proof of LCI, I think the only possible interpretation is that the LCI is indeed a disguised version of the older Law of Small Probabilities.
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 13. November 2003 20:55
quote:
[T]he relation between the LCI and the Explanatory Filter is that the LCI just says that the Explanatory Filter cannot fail.
I was wondering if some direct and simple relationship like would not be assumed.
The reason I asked is that I suspected there might be a relation to my study of the reliability of the EF (in termf of "false positives"). That study shows the conditions in which the EF does fail. Not appropriate for here, and understanding how to relate these mathematically may be difficult, but I am thinking that the study of failure rates of the EF can be related mathematically to the "4th law" issue.
IP: Logged
|
|
|
Printer-friendly view of this topic