|
Author
|
Topic: Validating Design Discrimination Methodologies
|
RBH
Member
Member # 380
|
posted 17. February 2003 08:11
Rex wrote quote:
RBH, that's a very interesting report--even if the loss of implementation details frustrates me.
However, ever the contrarian, I have some data sets to throw at you that I believe will confuse your algorithm--or at least will broaden the definition of what it means for something to be "designed".
Is it easy enough for me to simply give you the data sets and have you run them through? Or should I instead suggest a way to generate them? It's probably a better test if you don't know what the sequences are (and I just label them "generation process #1" and so on). But if the system isn't at the stage yet where that kind of testing is appropriate (or if you can't reveal what an input format might look like), just let me know and I'll describe the generation methods instead.
It isn't there yet. Sorry. I'm still sorting out several problems, including what is probably the most difficult, seeing if there is a standardized representation/encoding scheme that can be mechanized over as wide a range of potential phenomena as possible. And validation, not 'test,' is still my sole emphasis. I'm feeling my way through, trying to maintain some focus on the end point(s) so the earlier stages don't take me down a garden path and I end up being able to look only at the data the system was developed on. As, if, and when I'm ready to start some blind testing, I'll invite your data sets with pleasure.
Gedanken: That sounds real interesting from the hints you've given. Please do keep us posted. I wholly understand the constraints you're operating under: I am, too.
RBH
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 18. July 2003 17:36
Others are working along similar lines, it appears. This recent report of a computer algorithm capable of learning to discriminate the gender of authors of texts is another indication that Multiple Designers Theory is a viable research enterprise.
My own work on designer discrimination methodologies has been going slowly, but steadily. It is becoming less likely that I will be able to publicly report it for a while, but as and if I can, I will.
RBH
[ 18. July 2003, 17:39: Message edited by: RBH ]
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 28. August 2003 14:08
RBH, I have a question. Earlier (page 3) you reported:
quote:
I took heat for "designer-centric extremism" on the grounds that to do research on the nature, identity, and properties of the designer(s) requires pre-knowledge of, or presuppositions about, those very properties. As I report above, that criticism is unfounded: It is possible to statistically distinguish between samples of the products of different designers with no assumptions at all about the nature, properties, or identities of the designers, and without any knowledge of the design process itself. All that is required is the samples of products themselves. That is not a trivial result.
Neural network, fuzzy logic, and other discipline writers sometimes speak about “model free estimators” of a transfer or recognition function. (e.g. See Kosko, Neural Networks and Fuzzy Systems, 1992, pp 13, 25-26.) This is in essence what you seem to be claiming for your system.
However I have often questioned the “fuzzy” definition of “model free” that is involved. For example if one has knowledge from experience (e.g. training data) one can give some credence to different design styles recognition actually implying different human designers. (I’m not being in any way specific on how one does this pattern recognition except that it is purely a function of the input once the mapping function or algorithm is specified. No function is “purely” a function of the inputs, since the function itself is a specification and that specification comes from somewhere and thus is not “purely” independent of any other source of information.)
A system, using some measurement technique, could easily do clustering of like signals. But that the clusters represent the output of distinct designers, as opposed to distinct design modes of the same entity, seems to be a question that must be related to some sort of training data in order for the methodology (algorithm) to be validated as distinguishing on some relevant dataset.
But having been trained on such a dataset (in the design of the algorithm itself) it no longer is truly “model free” in the sense of being completely independent of known characteristics of distinctness of designers. Such testing, proving successful in distinguishing human designers, must incorporate in some form statistically collected information (by the “debugging” or research test-modify-feedback-recode process) that notes that humans don’t by the measurement technique employed tend to have such distinct modes that cluster.
So my point would be that by researching on a GA algorithm that distinguishes human designers, the method may appear to be “model free” yet is not in fact independent of characteristics of humans. The subtle background introduction of pattern recognition technique into the method may be shown to be quite opaque in terms of recognizable orientation toward human designers – but are they really free of such bias?
(I think that this same criticism can be made of Dembski’s “specification” aspect of the EF, can it inherently ever really be “independent of the event”? Don’t misunderstand me, I am not arguing that SUDID is not a subset of MDT.)
[PS don’t anybody welcome me back – I’m only back when underemployed in productive activity.]
[ 28. August 2003, 17:47: Message edited by: gedanken ]
IP: Logged
|
|
RBH
Member
Member # 380
|
posted 28. August 2003 23:14
gedanken wrote quote:
A system, using some measurement technique, could easily do clustering of like signals. But that the clusters represent the output of distinct designers, as opposed to distinct design modes of the same entity, seems to be a question that must be related to some sort of training data in order for the methodology (algorithm) to be validated as distinguishing on some relevant dataset.
Not so much trained on an existing known dataset but tested on it. Whether it can make the discrimination between products of multiple designers and products of distinct design themes by a single designer is a question yet to be addressed. I'm still working (sporadically, I admit, in the face of heavy interference from other stuff) on standardizing the representation scheme so as to mechanize it. In the ideal case it would be cleanly 'model free;' whether that's achievable is still up for grabs. Your remarks that quote:
The subtle background introduction of pattern recognition technique into the method may be shown to be quite opaque in terms of recognizable orientation toward human designers - but are they really free of such bias?
may have force; there may well be subtle biases as a consequence of the development-test-revise cycles.
No, I don't think Dembski's "specification" can ever be independent of the event. I think it's considerably less 'model-free' than a mechanized discrimination mechanism that does not (explictly at least) depend on subjective human judgements.
RBH
IP: Logged
|
|
gedanken
Member
Member # 594
|
posted 29. August 2003 11:28
Hi, RBH, the remainder of your post indicates you see exactly what I was getting at.
quote:
Not so much trained on an existing known dataset but tested on it.
As you noted but did not state explicitly, I am looking at a different feedback loop from the “training” feedback process which may be inherent in the system’s normal operation. This feedback loop involves the designer of the system (possibly subconsciously) including information from the environment on “what works”.
This process analysis actually implies that the environment is adding “intelligence” to the system, as opposed to you the “designer” of the program including that intelligence strictly as a replication of your own. The selection (meta-)process is clearly done by your intelligent actions in researching the design of the system. Changes to the design are somewhat equivalent to “mutations”, but are guided by your own thinking processes. But once again not in absence of input from the environment. (In fact I might claim that it is entirely dependent on your experience, physical state, and environmental feedback as inputs to your (meta-)system designing the program. Many of your concepts may have been based on observations on how other systems found in nature operate, added to the analysis of many generations of such observers of nature and their similar extraction of information from nature.)
I am in fact claiming that if humans regularly exhibited a form of “schizophrenic” character in design, that you would have to adjust your detection scheme to distinguish different persons from the different “characters” in the individual. Since this is not a regular characteristic of human design, you don’t have to account for this. But you would not in my opinion be able to make the system recognize distinct designers (as opposed to simply clustering design concepts) unless you take experience with real human design as feedback. It can’t be “subtle”, it is strongly characteristic, or it can’t work on human designers.
[ 29. August 2003, 11:36: Message edited by: gedanken ]
IP: Logged
|
|
Rex Kerr
Member
Member # 632
|
posted 29. August 2003 16:49
You raise an interesting point, gedankin. Perhaps the appropriate approach would be to measure (or at least estimate) the relative information input from the designer selecting what works vs. input from the environment.
Unfortunately, I'm not entirely sure how one would begin, since it's typically very difficult to figure out what sample space designers are drawing from.
IP: Logged
|
|
|
Printer-friendly view of this topic