|
Author
|
Topic: A 4th Law of Thermodynamics
|
Erik
Member
Member # 160
|
posted 23. October 2003 12:41
Pim van Meurs wrote: "My present feelings are that ignoring specification, the link between LCI and SLOT seems trivial. Including specification seems to make LCI a subset of the SLOT."
1. And what is the trivial link between the so-called LCI and 2LOT? Is the "link" that "specified information" and entropy are completely independent? Is it that, in an isolated system,
("specified information") + (conversion constant) * (entropy) = constant?
Does the link hold regardless of which events and chance hypotheses we consider? I become suspicious when you claim a trivial link, yet don't mention what the link is.
2. How can we even talk about the LCI if we ignore "specification"? The LCI is about "specified information", which in turn is about (among other things) "specification". If we ignore "specification", then it seems meaningless to talk about the LCI.
Rex Kerr wrote: "It's easier to use both the second and third laws. Information restricts the number of microstates available. Therefore, the third law implies information entropy. Restricting microstates takes energy; the second law says energy is conserved, so therefore information is conserved or lost."
I'm afraid I don't follow your reasoning. Perhaps you just wrote your post too quickly and made a few mistakes when translating your thoughts into an ISCID Brainstorming post (that happens to me frequently), but here are some problems:
1'. What do you mean by "information" and in what sense does it restrict the number of microstates available? Examples of possible interpretations: "information" is meant in the subjective Bayesian sense of anything that makes us change our subjective probabilities, "information" means a decrease in the Shannon entropy of some stochastic variable, etc.
2'. Implies is a relation between two propositions. It is not a relation between a proposition (e.g. a law of thermodynamics) and a quantity (e.g. information entropy), so what does this it mean to say "the third law implies information entropy"?
The details of this topic are so important that non-detailed posts are very uninformative. It would be very helpful if you gave a specific example of what you mean. It doesn't have to be a sophisticated example; a detailed analysis of the relation between the entropy and "specified information" for an ideal gas, a single quantum mechanical particle-in-a-box, etc. and some comments on how the example generalizes to any system would probably greatly clarify your reasoning.
Erik
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 23. October 2003 14:59
When I stated that the link between CSI and entropy seems trivial I was refering to the link established by Tom Schneider between Shannon entropy and thermodynamical entropy.
Link
S =k_b ln(2) H where S is thermodynamic and H is shannon entropy
Specification, if Sobel is correct that it is trivial to provide for a specification would make the difference between specified and nonspecified shannon entropy irrelevant. Specification would make LCI merely a subset of the SLOT.
Is specification relevant for LCI? Is only specified information subject to this 'law'? I do not believe so.
IP: Logged
|
|
Rex Kerr
Member
Member # 632
|
posted 23. October 2003 20:41
After looking at that link, I don't think there's anything left for me to do. I was about to rederive the same result in a very similar (probably less elegant) way.
To deal with specification, we note that
(1) The existence of specification has no impact on the physical evolution of a system (by definition, since specifications are not that way because of any regularity, and are independent of the event).
(2) The information in a specified event is information.
By (2), specified information is information, and by (1) the analysis is the same for each, so specified information is (on average) nonincreasing for any constant set of specifications.
(For silly, not deep reasons, though. If there is a claim that a specification X for system S can somehow mutate into a different specification Y for system T, and avoid the normal lossy/stochastic mechanisms, then an argument for this is required.)
Note that specified information is not absolutely conserved in the sense that energy is, only on average. (Likewise, entropy is an average phenomenon.)
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 23. October 2003 22:54
I suddenly have a hunch about a statistical relationship to the result I am close to demonstrating about coincidences and the "explanatory filter".
(That result is that the explanatory filter is incredibly sensitive to "missing distributions" not considered. It is in fact as sensitive to those missing distributions as is the factor 'alpha' required to be lower than 0.5 by "ProbRes" factors. In other words if ProbRes factors show that one must control for coincidences by a factor beta, then 'alpha' should be set to 0.5/beta, and this is simply a different way of saying exactly the formula used by Dr. Dembski, not something new except for giving a name for a term. Beta=M*N. So the EF is sensitive to missing distributions, amplifying them by a factor of Beta--or to put it differently in a set of Beta possible alternatives generated by ProbRes cases, that at least one may 'hit' is of greater probability by factor Beta=M*N, based on ProbRes factors M,N.)
The thought is that the distributed randomness will balance with the ProbRes factors which should be considered in specification when applied to an 'information' basis, just as it was needed in the "explanatory filter' orientation.
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 24. October 2003 07:50
Pim van Meurs, Dembski's "specified information" must not be confused with the Shannon information of some stochastic variable. They are not the same. You also asked and answered this rhetorical question: "Is specification relevant for LCI? Is only specified information subject to this 'law'? I do not believe so."
Well, I disagree. Here's a quote from the book "No Free Lunch" (p. 160):
quote:
"The general form of the Law of Conservation of Information may now be stated as follows:
Law of Conservation of Information. Given an item of CSI, call it B = (T2,E2), for which E2 arose by natural causes, any event E1 causally upstream from E2 that under the operation of natural causes is sufficient to produce E2 belongs to an item of CSI, call it A = (T1,E1), such that
I(A&B) = I(A) mod UCB
where by definition the quantity of information in an item of specified information is the quantity of information in the conceptual component (i.e., I(A) =(def) I(T1) and I(A&B) =(def) I(T1&T2))."
Here T1 and T2 are specifications (of E1 and E2, respectively) and "I(A&B) = I(A) mod UCB" is Dembski's unorthodox way of expressing the inequality
I(A) + UCB >= I(A&B).
I happen to doubt that "specified information" is a meaningfully defined concept (and to the extent that it is, I happen to think that the LCI is false), but I note that Dembski only claims that the LCI holds for "specified information".
I do not know exactly what you mean by "it is trivial to provide for a specification", but even if you mean that we can always find a specification T2 such that T2 = E2, it still doesn't follow that the LCI is a special case of the 2LOT. First of all, the events E1 and E2 may have nothing to do with thermodynamics. Second, even if we can choose the specification T2 = E2, we do not have to choose that specification. Third, even if the events E1 and E2 are about thermodynamics and even if E2 is its own specification it is still unclear exactly how the LCI relates to the 2LOT, because I(A&B), unlike entropy, is not defined as a weighted sum of logarithms*.
Erik
* Entropy is defined in terms of the individual probabilities of all outcomes, and it depends on how the sample space is divided into outcomes. Dembski's "quantity of specified information" is defined in terms of the probability of a single event (the specification) and, ignoring the fact that the rejection function may change when we change sample space, it does not depend on how the sample space is divided into outcomes.
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 24. October 2003 07:57
Rex Kerr, imagine my surprise when I saw that your reply contains no clarifications. No one is able to express his/her applications of Dembski's concept in a way that is both clear and correct. For instance, what does the statement
"The information in a specified event is information."
mean? Is it the (incorrect) statement that "specified information" is the same as Shannon information?
You also seem to think that Schneider's relation between thermodynamical entropy and Shannon entropy always holds. It is in fact an exceedingly trivial special case. Shannon entropy, like variance and expectation value, is a quantity that is defined for stochastic variables. Each stochastic variable has its own Shannon entropy. Thermodynamical entropy is defined as the Shannon entropy of a particular stochastic variable, namely a variable that indexes the quantum mechanical energy eigenstate of a system. Only for this particular stochastic variable is the thermodynamical entropy directly related to the Shannon entropy. When we study stochastic variables that have nothing to do with QM energy eigenstates, there is no relation between Shannon entropy and thermodynamical entropy. (In fact, as seen from the receiving end of a communication channel, the Shannon entropy of the message that is sent will on average decrease*.)
Erik
* It is true that H(X) >= H(X|Y).
Edited to correct a mistaken claim.
[ 24. October 2003, 09:09: Message edited by: Erik ]
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 24. October 2003 15:15
Erik: Pim van Meurs, Dembski's "specified information" must not be confused with the Shannon information of some stochastic variable. They are not the same. You also asked and answered this rhetorical question: "Is specification relevant for LCI? Is only specified information subject to this 'law'? I do not believe so."
I would disagree, specified information ala Dembski is nothing more than Shannon information with an independent specification.
There are various issues to consider here: 1) Is Sobel correct that specification is trivial for most cases and thus specification is irrelevant? 2) Does specification make the information somehow non Shannon like?
As to 2), specification does not change the nature of the beast, at most it may restrict what is specified and what is not but the general observation remains that Dembski's information is just Shannon entropy. Note that Dembski's information is NOT Shannon information since Shannon information is expressed as a change in Shannon entropy.
I showed how thermodynamic entropy and information entropy can be related thus the link between Shannon entropy and entropy seems to be trivial.
One of the problems with this whole excercise is the vagueness of the derivation of LCI. But I believe that a convincing argument can and has been made which relates LCI and SLOT.
Erik: In fact, as seen from the receiving end of a communication channel, the Shannon entropy of the message that is sent will on average decrease*
Indeed, which shows the Maxwell Demon like behavior of for instance RMNS which can inject information into an open system at a significant cost however resolving the Maxwell Demon paradox. Since Dembski's LCI applies to closed systems Shannon information can not increase.
[ 24. October 2003, 15:40: Message edited by: Pim van Meurs ]
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 24. October 2003 17:10
Erik wrote quote:
(In fact, as seen from the receiving end of a communication channel, the Shannon entropy of the message that is sent will on average decrease*.)
That is for a communication channel in which the only relevant variables are the original signal at the source and stochastic noise in the channel (and receiver components prior to decoding). In this case, the mutual information of the source and receiver is the correlation of the signal exiting the source with the signal exiting the decoder of the receiver (receiver components prior to the decoding component are part of the channel).
In biological evolution, what is of interest is the information transmitted from (source) parent generation to (receiver) offspring generation, and the communication channel (imperfect reproduction with heritable stochastic variation and selection) is not a passive (linear or nonlinear) device carrying a signal affected only by stochastic noise in the channel. Under those conditions, it is not at all clear that "It is true that H(X) >= H(X|Y)," even on average, unless the source is considered as the parent generation plus non-stochastic processes operating within the channel right up to the decoder output, where the "decoder" is the offspring generation's reproduction in turn (it thereby becoming the source for the next generation). It's a little hard to think about a channel as just passing a signal plus stochastic noise when the channel is itself effectively part of the source.
RBH
[ 24. October 2003, 17:16: Message edited by: RBH ]
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 24. October 2003 17:26
But I think this gets back to Eric's point about clear definitions.
For "information" might be described as "correct data", wherein any deviation from original data was not "information". But it could be described in terms of any constructed code that produces a functioning system. That is different, as the noise actually becomes "information" in the context. It is opposed to the original sender's desire if the sender desires clear communication. But if the orientation is different, the measure of some sort of value (which "information is") might be different from error free reception.
I have produced several useful concepts by misunderstanding someone elses statement. (e.g. mutating that original code.) But then I applied the apparent meaning in a context of my own understanding, producing a new concept that was neither mine nor the original sender's.
[ 24. October 2003, 17:29: Message edited by: gedanken ]
IP: Logged
|
|
Rex Kerr
Member
Member # 632
|
posted 24. October 2003 21:54
Erik says:
quote:
When we study stochastic variables that have nothing to do with QM energy eigenstates, there is no relation between Shannon entropy and thermodynamical entropy.
Quite right (assuming you include the traditional "microstates" definition under the heading "QM energy eigenstates"). So, what is a stochastic variable that has nothing to do with QM energy eigenstates?
Careful now. When you say "nothing", I expect that you mean it.
IP: Logged
|
|
Erik
Member
Member # 160
|
posted 26. October 2003 11:16
quote:
Pim van Meurs: I would disagree, specified information ala Dembski is nothing more than Shannon information with an independent specification.
[...]
As to 2), specification does not change the nature of the beast, at most it may restrict what is specified and what is not but the general observation remains that Dembski's information is just Shannon entropy. Note that Dembski's information is NOT Shannon information since Shannon information is expressed as a change in Shannon entropy.
Is your claim that "specified information" is Shannon information (as stated in the first paragraph)? Or is your claim that "specified information" is Shannon entropy (as stated in the third paragraph)?
Regardless of which, here's a suitable example to analyze: Consider any tombstone with an inscription. For instance, let's suppose that the inscription says "Here rests John Doe, born May 2 1923, died January 4 1978". As the sample space, we choose the set of all sequences of English letters, digits and punctuation characters that are exactly 57 symbols long. As a straw-man hypothesis that we pit our ID hypothesis against, we pick a uniform probability distribution over the sample space. Regardless of which specification we choose (e.g. we might feel tempted to claim that our knowledge of the English language and John Doe's life explicitly and univocally identifies a rejection function on the sample space), the amount of "specified information" can never be larger than the probability of the actual inscription that we observe. For our particular sample space and hypothesis, the amount of "specified information" is never larger than
log2(70^57) ~ 350
regardless of which specification we pick (I assumed 70 different symbols: upper and lower case letters, ten digits, and a few punctuation characters; the exact numbers are not very important...). The number of possible QM energy eigenstates (aka "microstates") that the tombstone can be in given it's energetic constraints, is much higher than 2^350. Furthermore, the thermodynamical entropy of the tombstone is highly dependent on the temperature, whereas the the "specified information" is highly robust against normal variations in temperature. We can imagine the tombstone being slowly cooled to 150 Kelvin and then slowly heated to 600 Kelvin. The difference in entropy at the lower and higher temperature will be considerable, but the difference in "specified information" is zero (given the choice of sample space and hypothesis).
You are of course free to argue that for a particular choice of sample space, hypothesis, and specification the "specificied information" will be proportional to the thermodynamic entropy, but this is certainly not the case for all choices. This means that Dembski's LCI cannot possibly be a special case of the 2LOT. At best, the 2LOT is a special case of the LCI.
quote:
Pim van Meurs: I showed how thermodynamic entropy and information entropy can be related thus the link between Shannon entropy and entropy seems to be trivial.
Shannon entropy is a property of a stochastic variable. Every stochastic variable variable has it's own Shannon entropy and the relation
(thermodynamical entropy) = k ln(2) (Shannon entropy)
certainly cannot hold for all stochastic variable. So for which stochastic variable does the trivial relation hold? Let X denote the number of millions of votes (rounded off to an integer) that president Bush will get in the next American presidential election. What is the relation between H(X) and the thermodynamical entropy of the United States?
quote:
Erik: In fact, as seen from the receiving end of a communication channel, the Shannon entropy of the message that is sent will on average decrease*
Pim van Meurs: Indeed, which shows the Maxwell Demon like behavior of for instance RMNS which can inject information into an open system at a significant cost however resolving the Maxwell Demon paradox. Since Dembski's LCI applies to closed systems Shannon information can not increase.
The fact I mentioned has little to do with Maxwell's demon. The former is about the Shannon entropy of a message, the latter is about the Shannon entropy of the exact energy eigenstate of a system.
quote:
RBH: Under those conditions, it is not at all clear that "It is true that H(X) >= H(X|Y)," even on average, unless the source is considered as the parent generation plus non-stochastic processes operating within the channel right up to the decoder output, where the "decoder" is the offspring generation's reproduction in turn (it thereby becoming the source for the next generation).
1. The inequality H(X) >= H(X|Y) is a mathematical theorem that holds for any discrete variables X, Y and for any probability distribution over them.
2. The inequality is the average. The conditional entropy H(X|Y=y) can be both lower and higher than H(X), but the average
H(X|Y) = H(X|Y=y1) P(Y=y1) + H(X|Y=y2) P(Y=y2) + ...
is always lower.
3. You are probably really thinking about a time series X(1), X(2), X(3), ... of stochastic variables and it is quite true that H(X(t)) need not be either lower or higher than H(X(t+1)) in general.
quote:
Rex Kerr: Quite right (assuming you include the traditional "microstates" definition under the heading "QM energy eigenstates"). So, what is a stochastic variable that has nothing to do with QM energy eigenstates?
Careful now. When you say "nothing", I expect that you mean it.
"QM energy eigenstate" is the modern definition of microstate, at least for the purposes of statistical mechanics at a fundamental level.
I meant what I wrote, but it is possible that I did not mean the same thing you expect. I wrote: "Each stochastic variable has its own Shannon entropy. Thermodynamical entropy is defined as the Shannon entropy of a particular stochastic variable, namely a variable that indexes the quantum mechanical energy eigenstate of a system. Only for this particular stochastic variable is the thermodynamical entropy directly related to the Shannon entropy. When we study stochastic variables that have nothing to do with QM energy eigenstates, there is no relation between Shannon entropy and thermodynamical entropy."
Perhaps you expect, given that last sentence, an example of a stochastic variable that has literally nothing to do with any QM energy eigenstate. However, that last sentence occured in the context of a remark that the Shannon entropies of different stochastic variables need not be directly related. A stochastic variable that has nothing to do with QM energy eigenstates is not necessary for a counter-example to the claimed direct relation, it's just a idealization that we can approximate. We do not need a particularly good approximation for the relation to fail. Here are some specific examples for you:
1. Let X denote a s.v. which can only take on the value 0. Thus, H(X) = 0.
2. Consider a single hydrogen atom in the ordinary approximation (no spin-coupling to the nucleus). That is, the energy eigenstates can be indexed by four quantum numbers (energy level, total angular momentum, angular momentum in the z-direction, spin in the z-direction). Let Sz be a s.v. taking the value 1 if a measurement of the spin in the up/down-direction gives the result "up" and -1 if the result is "down". How is H(Sz) related to the thermodynamical entropy?
3. Same hydrogen atom. Let Sx be a s.v. taking the value -1 if a measurement of the spin in the left/right-direction gives the result "left" and 1 if the result is "right". Is H(Sx) proportional to the thermodynamical entropy?
4. Let P denote the parity of the state of a system, e.g. a hydrogen atom or a particle in a box. Is H(P) proportional to the thermodynamical entropy?
5. Pick any library. Let N denote the number of occurances of the letter 'e' in a randomly chosen book. How is H(N) related to the thermodynamical entropy of a book? How is H(N) related to the thermodynamical entropy of the library? Is the relation a simple proportionality relation, where the proportionality constant is k ln(2)?
6. Let S be the thermodynamical entropy of a system. How is H(S) related to S?
The bottom line is, first, that the relation
S = k ln(2) H(X),
where S is the thermodynamical entropy and X is any stochastic variable is completely indefensible as a general relation. And, second, that if we allow ourselves to write "Shannon entropy" without having a clear idea of which stochastic variable we are referring to, we will not be able to reach justified conclusions.
Erik
IP: Logged
|
|
Pim van Meurs
Member
Member # 541
|
posted 26. October 2003 12:24
Erik: Furthermore, the thermodynamical entropy of the tombstone is highly dependent on the temperature, whereas the the "specified information" is highly robust against normal variations in temperature. We can imagine the tombstone being slowly cooled to 150 Kelvin and then slowly heated to 600 Kelvin. The difference in entropy at the lower and higher temperature will be considerable, but the difference in "specified information" is zero (given the choice of sample space and hypothesis).
You are now confusing to separate concepts, the Shannon entropy of the message versus the termodynamical entropy of the tombstone.
you state that the general relationship between Shannon entropy and thermodynamical entropy is indefensible and yet the derivation of said relationship is straightforward WHEN applied to the same entities namely the entropy of the information/complexity in question.
I believe that Erik's flaw is exemplified by some of the examples he uses such as
quote:
Pick any library. Let N denote the number of occurances of the letter 'e' in a randomly chosen book. How is H(N) related to the thermodynamical entropy of a book?
Erik suggests there is a relationship between H(N), the shannon entropy of the text versus the thermodynamical entropy of the book.
quote:
Since the Shannon entropy is equivalent to the Boltzmann’s entropy under equilibrium,...
Link
quote:
Inspired by the observations of Ochs 2, we would like to show that the Boltzmann/Shannon formula for entropy may be viewed as a measure of correlation, by showing that for a class of transformations which destroys correlations between variables in a probability distribution, the Boltzmann/Shannon formula for the entropy is non-decreasing.
source
The latter one is very interesting since it shows the equivalency of these entropies under transformations which destroy correlation resulting in non-decreasing entropy. They also argue that quote:
The Second Law of Thermodynamics may be rephrased to state that correlations are highly unlikely to arise spontaneously, and that the
natural course of evolution of a system is one in which correlations diminish.
Thus processes which increase correlations such as selection seem to be sufficient to explain the decrease in entropy in a system.
Scientific American link shows basically the same argument I have made about the entropy calculations
quote:
Even when reduced to common units, however, typical values of the two entropies differ vastly in magnitude. A silicon microchip carrying a gigabyte of data, for instance, has a Shannon entropy of about 1010 bits (one byte is eight bits), tremendously smaller than the chip's thermodynamic entropy, which is about 1023 bits at room temperature. This discrepancy occurs because the entropies are computed for different degrees of freedom. A degree of freedom is any quantity that can vary, such as a coordinate specifying a particle's location or one component of its velocity. The Shannon entropy of the chip cares only about the overall state of each tiny transistor etched in the silicon crystal--the transistor is on or off; it is a 0 or a 1--a single binary degree of freedom. Thermodynamic entropy, in contrast, depends on the states of all the billions of atoms (and their roaming electrons) that make up each transistor. As miniaturization brings closer the day when each atom will store one bit of information for us, the useful Shannon entropy of the state-of-the-art microchip will edge closer in magnitude to its material's thermodynamic entropy. When the two entropies are calculated for the same degrees of freedom, they are equal.
[ 26. October 2003, 16:17: Message edited by: Pim van Meurs ]
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 26. October 2003 18:40
Erik wrote quote:
1. The inequality H(X) >= H(X|Y) is a mathematical theorem that holds for any discrete variables X, Y and for any probability distribution over them.
2. The inequality is the average. The conditional entropy H(X|Y=y) can be both lower and higher than H(X), but the average
H(X|Y) = H(X|Y=y1) P(Y=y1) + H(X|Y=y2) P(Y=y2) + ...
is always lower.
3. You are probably really thinking about a time series X(1), X(2), X(3), ... of stochastic variables and it is quite true that H(X(t)) need not be either lower or higher than H(X(t+1)) in general.
I am quite definitely thinking of a times series: I specified information transmission through (and mutual information across) generations as the problem in biology. Evolution is a process that generates a time series, and it is information transmission through that series that is of interest.
RBH
IP: Logged
|
|
Rex Kerr
Member
Member # 632
|
posted 27. October 2003 00:33
For the systems Erik describes:
- Something that cannot vary has no Shannon entropy. So yes, H(X) is and should be zero in this case. Likewise, there is one microstate, and thermodynamic entropy is zero.
- Assuming equal energies for both spin states, S = k_B * sigma = k_B log g(N,U) = k_B log 2 and H(X) = - 0.5 log 0.5 + 0.5 log 0.5 = log 2
- Same thing.
- This is exactly covered by Schneider's derivation that was linked to above, unless I misunderstand what you're asking for.
- Okay, this breaks down. However, H(N) isn't really the information present. In fact, there is much more inherent in the physical system, but we don't care about it.
- I'm not sure what to do with H(S), aside from noting that there is much more information in a system, in general, than a scalar representing its entropy. That's interesting, actually. If I get time, I'll think about it more.
Case (5) is the interesting one. Here we have an encoding that, as thermodynamic entropy increases, could apparently decrease Shannon entropy. For example, as "e" starts to decay, it might start to look like "c" instead, a lower-probability and therefore higher-information character. But all we've really learned is that we're not encoding information efficiently; we use quintillions of ink molecules to represent "e" instead of a few spin states or position of a molecule.
Thermodynamic entropy is the limiting case as information representation becomes maximally efficient.
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 27. October 2003 13:55
Rex wrote quote:
Thermodynamic entropy is the limiting case as information representation becomes maximally efficient.
That is a very interesting remark. I strongly encourage you to work through some of the implications of it.
RBH
IP: Logged
|
|
|
Printer-friendly view of this topic