|
Author
|
Topic: Validation (from Stone Circles thread)
|
GP
Member
Member # 570
|
posted 31. January 2003 11:35
quote:
As I've discussed the proposal with Bill Dembski, Wesley Elsberry, and others, we've talked about a one-year time frame for the bit string experiment.
Thank you Paul. I want to pursue this matter just a bit further, if you will. After the year is over, what are the possible resulting answers that you're expecting? It seems to me that at present you will have to include at least 3 possibilities: design, not-design, and "I don't know." But, what is the nature of the design detector -- is it a machine that implements EF (for example), or is it a machine-man duo, or is it a man (this list is not exhaustive, of course)? I ask because the "I don't know" answer seems to require a subjective evaluation. In other words, it seems to require an ad hoc criteria for terminating the search for natural phenomena.
One more question. How do you know that the encoding processes do not impart CSI, in light of Dembski's displacement thesis?
[ 31. January 2003, 11:38: Message edited by: GP ]
IP: Logged
|
|
Paul A. Nelson
Member
Member # 26
|
posted 31. January 2003 12:14
GP wrote:
quote:
It seems to me that at present you will have to include at least 3 possibilities: design, not-design, and "I don't know."
I think that's exactly right.
GP wrote:
quote:
But, what is the nature of the design detector -- is it a machine that implements EF (for example), or is it a machine-man duo, or is it a man (this list is not exhaustive, of course)? I ask because the "I don't know" answer seems to require a subjective evaluation. In other words, it seems to require an ad hoc criteria for terminating the search for natural phenomena.
No -- a one-year period is just an arbitrary cut-off point. The experiment has to start, and end, within some reasonable time interval. But remember, the point of the experiment is to test design detection methods. Logically speaking, one can never exhaust "the search for natural phenomena" (by which I assume you mean the search for natural causes for any event). But that's not what is at issue in the experiment.
quote:
One more question. How do you know that the encoding processes do not impart CSI, in light of Dembski's displacement thesis?
This is an important question that must be settled prior to the experiment. Obviously, a Geiger counter is an intelligently-caused artifact, as would be the encoding (the ones-and-zeros translation) of the decay pattern of radium or some other radioactive element. But since the datum of interest is the decay pattern itself, a natural event, the experimental design or definition of "naturally caused" would take the necessary human role (of gathering data and encoding it) into account.
If an ID advocate claimed that a false positive for radioactive decay (for instance) wasn't really a false positive, because of the intervention of human beings in encoding the data, I'd consider that an abject failure of his design detection method, coupled with a gross misunderstanding of the experimental set-up and its definitions of terms. If radioactive decay comes out as "designed," that ID detection method would be a loser.
[ 31. January 2003, 12:16: Message edited by: Paul A. Nelson ]
IP: Logged
|
|
GP
Member
Member # 570
|
posted 31. January 2003 12:30
Paul, thank you again for your patience in answering my questions. You wrote: quote:
But that's not what's at issue in the experiment.
in response to my concern about an ad hoc termination to satisfying the first node of the EF detection scheme. I am now a little confused. Is it inevitable that an implementation of design detection will require some "arbitrary"/ad-hoc decision (such as a cut-off point) at certain steps? It sounds to me like this is at the very heart of the matter. One way that I see this manifest is that you acknowledge "[l]ogically speaking, one can never exhaust 'the search for natural phenomena'." But isn't this problem inherent in the way design is formulated -- namely as the rejection of natural causation? Without satisfying at least this constraint, what use would be the design detector?
I am happy to hear that we are in agreement about my CSI displacement concern. I'd like to hear how one addresses the issue, however my time is up for today. I would love to pick up on this discussion point later.
IP: Logged
|
|
charlie d.
Member
Member # 159
|
posted 31. January 2003 12:37
quote:
As for encoding the bacterial flagellum -- I don't know how one would do that. Bacterial motors and flagella are pretty complex systems. But that shouldn't be an obstacle, because we've got an infinitely rich pool of other natural patterns, much easier to encode, that could be used in the experiment.
There you go, Paul, you have identified one clear possible source of bias in the "string" test. As you correctly point out, encoding only works for patterns that are simple enough. However, the use of the EF is really only interesting when applied to things, like the flagellum and other biological systems, that are in fact very complex and not easily encoded.
So, how can one avoid the risk that the EF can correctly evaluate simple bit strings, but not complex systems, or vice-versa? On the one hand, the rate of false positives could perhaps increase at complexity levels higher than a simple encodable pattern (with the filter being "overwhelmed", in a sense). On the other hand, maybe the encoding will make everything, designed or not, look like a relatively simple algorithm, giving rise to high rates of false negatives, making the filter appear useless when in fact, when correctly applied to complex systems, it could still be 100% accurate.
See the point? You are running a control experiment on a sample population that differs significantly (in this case in complexity, which is certainly one of the variables to consider when attempting a design inference) from what will ultimately be your experimental population. Methodologically, it does not make much sense, IMO.
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 31. January 2003 14:04
Paul A Nelson said:
quote:
Gedanken and RBH -- I've read through your posts twice, and I confess that I just don't understand your worries. Put that down to my stupidity if you like. The bit string proposal really isn't that difficult or ambiguous. Here's a parallel situation. Some years ago, a physicist (I think he was a physicist) claimed to be able to identify LP recordings, by composer, opus, etc., simply from examining the vinyl discs (with the labels covered, of course). James Randi was skeptical, so he arranged a test, assembling a collection of LPs with their labels carefully masked. The physicist correctly identified every single recording -- that is, every single recording of classical music. Randi had included some rock albums, as wild cards, and the physicist said, "I don't know what these are -- sorry." Randi wrote up the results for Skeptical Inquirer, acknowledging that the guy could do what he claimed to do.
This is an area close to my own area of expertise, so let me both ask some questions and make comments.
First of all the identification is done by way of the music that is encoded on the vinyl by way of the analog wiggles in the groove.
I don’t know exactly how the identification was done. A simple method would be simply to play the record. But another way would be to optically analyze the grove pattern, first by digitizing an image and using a simple automated algorithm to extract the ‘wiggles’ in the grooves.
The point of this is that only vinyl records are used, so a single simple algorithm can be used to essentially extract the groove patterns into a digitized format.
Now whether an optical technique or a direct playing of the record was used, it does not matter.
(So question one: How was the waveform extracted? I assume that it was extracted. If a digital image was used, this does not deny the waveform was extracted -- it simply places the waveform into a different but essentially equivalent type of bit encoding, and at least segments of the analog waveforms can be extracted from the image format. So how was this done?)
From this point I will assume that the music waveforms were essentially digitized. It does not matter (as I said) that a visual image technique would have been used.
AH I have found a reference:
http://www.skepdic.com/vinylvision.html
The technique was visual, not computer. So I don’t understand how this was to be related to the “bit encoding” into digital format.
Let’s continue as though this was to be done in a digital format, with recognition to be done by computer rather than by human.
The digitizing could be done with exactly the same parameters being extracted. The technique was to identify patterns of low and high activity on the concentric grooves as they spiral in, giving a profile of the sould levels of the music in space as an equivalent to the time function profile.
Thus the technique is a transformation in a very rigid and regular translation from grove spiral information to a profile. In pattern recognition this is known as “feature extraction”. The key is the translation is by a fixed and regular pattern that takes not intelligence and few if any decisions. It can be easily and completely automated.
I have made the point elsewhere that the problem is extracting the probability of occurrence of the original pattern so as to transport it to the bit pattern to be analyzed. Are you simply implying that we transport pictures of the objects? Is that sufficient?
How could you gain (as charlie d points out) the calculations that have been done to find the so-called probability of the flagella, based solely on a picture? The regularized extraction technique of a digital image of a vinyl recording and the pattern recognition that can be easily accomplished because of that regularity is not a good example to suggest that the more complex problem can be achieved in an objective manner.
To put this differently, a strictly regular process could be used to generate bit patterns from objects -- like taking digital pictures? That would be a low information technique (as someone else is fond of saying). But it is low information specifically because it strips off some of the information, such as history.
So would the following be acceptable examples, and are the “correct answers” as follows:
Pictures of washers, encoded as digital pictures -- and expected answer should be “designed”
Pictures of rings of saturn encoded as digital pictures -- and expected answer should be “natural”
Picture of a single “stone circle” -- and expected answer should be “natural”.
Picture of a crop circle, but one that is of washer shape -- and expected answer should be “designed”
Picture of a cross section of the ring portion of the flagellum -- now what?
IP: Logged
|
|
Paul A. Nelson
Member
Member # 26
|
posted 31. January 2003 14:14
GP wrote:
quote:
I am now a little confused. Is it inevitable that an implementation of design detection will require some "arbitrary"/ad-hoc decision (such as a cut-off point) at certain steps? It sounds to me like this is at the very heart of the matter. One way that I see this manifest is that you acknowledge "[l]ogically speaking, one can never exhaust 'the search for natural phenomena'." But isn't this problem inherent in the way design is formulated -- namely as the rejection of natural causation?
Well, logically speaking, nothing compels one to go past the first node of the EF. The universe of possible natural causes for any event cannot be exhaustively searched. Nevertheless, we do in fact pass the first node all the time -- that is, we ordinarily judge natural causes insufficient to explain an event. You've done this yourself hundreds, probably thousands, of times. The inference is no more ad hoc than any other decision procedure.
Let's suppose that one of the bit strings encodes a sonnet by William Shakespeare. You discover this, but refuse to say that the string is "intelligently caused," because there is a probability (greater than zero) that the sonnet might have been caused by a tumbling bottle of ink, somewhere near London in 1596, blown over by the wind. I think most people would find that argument counterintuitive, to say the least, but it's a logically possible move.
For design (as a real cause, and scientific explanation) to have any empirical content, it must have a contrast class -- namely, natural causes. Now, if one requires the logically impossible, i.e., that we exhaust the universe of possible natural causes, one cannot infer design.
But we do infer design. Therefore, one need not exhaust all possible natural causes.
Charlie wrote:
quote:
As you correctly point out, encoding only works for patterns that are simple enough.
That's not what I said. I said I didn't know how to encode a flagellum -- having given the problem all of two minutes of thought. But I'm quite certain that very complex patterns indeed could be encoded, with some hard work.
Charlie wrote:
quote:
However, the use of the EF is really only interesting when applied to things, like the flagellum and other biological systems, that are in fact very complex and not easily encoded.
I don't know about that. The base sequence of Mycoplasma would be (relatively) easy to encode, and it's whomping complex.
quote:
So, how can one avoid the risk that the EF can correctly evaluate simple bit strings, but not complex systems, or vice-versa? On the one hand, the rate of false positives could perhaps increase at complexity levels higher than a simple encodable pattern (with the filter being "overwhelmed", in a sense). On the other hand, maybe the encoding will make everything, designed or not, look like a relatively simple algorithm, giving rise to high rates of false negatives, making the filter appear useless when in fact, when correctly applied to complex systems, it could still be 100% accurate.
Indeed. All possible outcomes. But then, that's exactly why we would do the experiment, right? To see what would happen.
quote:
See the point? You are running a control experiment on a sample population that differs significantly (in this case in complexity, which is certainly one of the variables to consider when attempting a design inference) from what will ultimately be your experimental population. Methodologically, it does not make much sense, IMO.
Then I'd expect you to debunk the experiment after it was run. ![[Big Grin]](biggrin.gif)
Gedanken, please don't take my mention of the LP-identification test as a literal parallel. I was making only a general methodological point about the importance of testing extraordinary claims for one's ability to discriminate patterns.
I've used up my 3-posts-per-day. More tomorrow.
[ 31. January 2003, 14:57: Message edited by: Paul A. Nelson ]
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 31. January 2003 15:53
I'll divide this post into three parts, for (maybe?) some clarity.
Validation and Control Conditions
Paul wrote quote:
Gedanken and RBH -- I've read through your posts twice, and I confess that I just don't understand your worries.
I don't have "worries." I'm concerned with appropriate validation of methodologies. In part my concern stems from an experience several decades ago when I was peripherally involved (please don't ask me how I got into this project - blame a former student) in evaluating claims for remote viewing by psychics. (Before someone blows a gasket, let me say that I am emphatically not arguing that ID is directly analogous to remote viewing by psychics.) The core question was how to score (as "hits" or "misses") the psychics' drawings and descriptions of the remote sites they were allegedly psychically viewing. It turned out that when appropriate control conditions were used in the scoring procedure, their responses did not reliably discriminate between sites they had allegedly viewed and those they had not been instructed to view. Interestingly, one argument that was offered in defense of the psychics was the claim that they were using another parapsychological power, telepathy, to descry the alternative non-viewed control sites in the mind of the researcher who selected them (in a double-blind arrangement) and those telepathic images then contaminated the psychics' 'view' of the target sites, thereby muddying the scoring of their responses.
The "scoring" of hits in remote viewing is always subjective: a human judge looks at the picture the psychic drew, looks at the site allegedly viewed, and decides on some criteria whether the picture matches the site well enough to be scored a "hit." Two obvious things contaminate that procedure: the (frequent) lack of objective criteria for assessing matches, and the lack of an appropriate control condition. That's where my concern lies in design detection with the EF.
I am interested in validating methodologies, not "testing" them. Consider the development of spectroscopy. Before using Fraunhofer lines to infer the composition of stars, and ultimately using their relative shifts of position in spectra to measure the velocity of recession of distant astronomical objects, the techniques were validated by testing them on terrestrial elements and compounds of known composition to learn the 'signatures' of those materials in the patterns of lines in the spectrum of the light they emitted when heated. That's the model I've had in mind as I've harped on validating design detection and designer discrimination techniques on objects and processes of known provenance. That's the general model I intend to follow when I begin validating the designer-discrimination methods I'm looking at.
"Validity" and "reliability" are different concepts. "Validity" refers to the question of whether this measuring device measures what it purports to measure, what I think it measures. Do the Fraunhofer lines actually allow me to discriminate among heated substances? Do their positions in the spectra actually correlate highly with the composition of test materials?
"Reliability" refers to the consistency of measurements across occasions. If I measure a specific substance repeatedly without varying the conditions surrounding the measuring process, do the measured positions of the several Fraunhofer lines cluster around the same locations with some stable (and therefore specifiable) error distribution?
An acceptable observational technique or classifying device has to have data telling us about both its validity and reliability. The present question is whether the EF is both reliable and valid in its classifications.
Paul's anecdote about the physicist whose method could discriminate among classical composers is interesting, but incomplete. I was unable to find the story in the Skeptical Inquirer's back issues at least not under an easily recognizable title. It does nicely illustrate the control condition issue, though. I can think of at least two quite different ways of accomplishing what the physicist is said to have done, but both require that the discrimination method be 'trained' on exemplars of the set of phenomena among which discriminations will be made. I don't mean they must be trained on the specific test items, but on items from the same classes. Given Randi's insertion of rock records in the test set, any algorithm I can think of would have the same outcome: "Unclassified."
Design detection and designer discrimination are classification tasks, and classification tasks require that somewhere along the line the methodology has to 'know' about the relevant classes. The EF has three classes - regularity, chance, and design. To this point it has not been applied on any scale at all to objects of known provenance or objects of unknown provenance, at least not in the way Dembski has told us it should be applied. There are a handful of examples that are repeatedly cited - the Caputo case and the bacterial flagellum primary among them - but none have been added to that list in several years. PCID has published no formal applications of the EF and none appear in informal venues like these web boards.
Validating a design detection method requires that it be applied to an array of phenomena of known provenance, to assess the method's behavior under known circumstances. Otherwise we have no way of evaluating either its reliability or its validity. We have no reason at all to trust its classifications.
Applying the EF to 'natural' phenomena
In the Stone Circles OP Paul wrote quote:
Some have suggested that stone circles and other "false positive" phenomena provide telling counterexamples to Dembski's explanatory filter. Well, Bill has been my design-theoretic colleague for 12 years, and we spent a lot of time early on (notably, at Cambridge UK in the summer of 1992, with Steve Meyer), going over problem cases such as stone circles. Indeed, it was the very existence of such phenomena that led us to the need for a method or inferential procedure like the explanatory filter.
But later in the same thread he wrote quote:
It's hard to use stone circles (i.e., sorted pattern grounds) now as a test of the filter, for obvious reasons.
I don't see such phenomena as "counter-examples." Along with other participants in several of these threads, I see them (along with the Oklo reactors, solar eclipses, and similar phenomena) as appropriate objects (along with other classes of phenomena) for validating design detection methods. What are the "obvious reasons" that prevent the EF from being applied to one of the phenomena that stimulated its invention, but do not also prevent it from being applied to considerably more complicated structures like the bacterial flagellum?
Inferring "regularity."
In response to GP, Paul wrote quote:
Well, logically speaking, nothing compels one to go past the first node in the EF. The universe of possible natural causes for any event cannot be exhaustively searched. Nevertheless, we do in fact pass the first node all the time -- that is, we ordinarily judge natural causes insufficient to explain an event. You've done this yourself hundreds, probably thousands, of times. The inference is no more ad hoc than any other decision procedure.
This is a deceptive analogy. We do not pass the first node for a given human artifact because we make a decision that natural causes are "insufficient" to account for the object; we pass it because we know they are inappropriate to account for this object. We pass the regularity (natural causes) node "all the time" in those cases where we know, based on prior experience and learning, that a given phenomenon is highly likely to be a member of the class of human-designed objects or processes. But the EF is not intended to classify objects of known provenance; it is intended to classify those of unknown provenance where we do not know whether they are due to regularity, chance, or (non-human) design. So the analogy with our everyday judgements of (not inferences to) human design is fatally flawed: We are classifying on the basis of our prior knowledge and experience of human designs. The EF is supposed to allow us to infer non-human design. If I were theologically inclined I would be very interested in (and quite possibly pretty worked up about) a technique that actually detected human-like designs in biology.
As I understand it, the EF is fundamentally a probability discriminator. The folding of "regularity" and "chance" together into a single class in TDI is to enable a comparison of the probability that the occurrence under analysis is an instance of one of the two of those classes with the UPB, and if the UPB is passed, one infers design (the EF never evaluates the probability of the design hypothesis). According to the decision rules of the EF, if just one potential "regularity" hypothesis in the distribution of all possible "regularity or chance" alternatives has a probability greater than 1 in 10^150, one cannot infer design. If there is just one teeny weeny hump in the otherwise uniform (flat) PDF over all alternatives in the "regularity or chance" universe, one can't infer design. That potential hypothesis with the PDF hump doesn't have to be true, it doesn't even have to have been tested, it merely has to have just enough plausibility to bring its probability to a value greater than 1 in 10^150. I genuinely do not understand how one could say with a straight face that there is anything at all that could get past the "regularity or chance" nodes to make it to the promised land of design. That is, as far as I can see, in the absence of real good validation and reliability studies, all EF classification decisions on objects not known on other grounds to be of intelligent design must be to regularity or chance, with no residue left over to be classified design.
Debunking experiments
In response to charlie d's remark about control conditions and methodological problems, Paul wrote quote:
Then I'd expect you to debunk the experiment after it was run.
As I used to tell my students, it's much better to design it right in the first place than have me rip it up afterwards.
RBH
Edited to add a line left out whilst cutting and pasting. ![[Frown]](frown.gif)
[ 31. January 2003, 16:19: Message edited by: RBH ]
IP: Logged
|
|
andyg
Member
Member # 415
|
posted 31. January 2003 16:39
I think Charlie D's point is an excellent one - how does one "reduce" a comlex structure into a bit string in such a way as to avoid bias? How could we reasonably expect the filter to distinguish between:
- a randomly generated series of 0s and 1s and
- a bacterial flagellum reduced to a series of 0s and 1s
When we don't know how to express a "flagellum" in bits?
What aspects of the flagellum would we encode? Its dimensions? Its amino acid composition? The sequences of the genes that encode it? Its X-ray diffraction co-ordinates? The x-ray diffraction coordinates of the DNA that encodes it? An atomic force image of it inserted into the cell membrane? Or what?
AndyG
[ 31. January 2003, 19:10: Message edited by: andyg ]
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 31. January 2003 19:30
Paul,
So would the following be acceptable examples, and are the “correct answers” as follows:
Digital pictures of washers, encoded as digital pictures -- and expected answer should be “designed”
Digital pictures of rings of saturn encoded as digital pictures -- and expected answer should be “natural”
Digital picture of a single “stone circle” -- and expected answer should be “natural”.
Digital picture of a crop circle, but one that is of washer shape -- and expected answer should be “designed”
Digital picture of a cross section of the ring portion of the flagellum -- now what?
(Making too many points allows the respondent to pick and choose which to answer -- must learn)
A main point in asking is to try to determine what constitutes a digital string that is not designed.
IP: Logged
|
|
|
Printer-friendly view of this topic