Gil’s post With much fear and trepidation, I enter the SZone got somewhat, but interestingly, derailed into a discussion of David Abel’s paper The Capabilities of Chaos and Complexity, which William Murray, quite fairly, challenged those of who expressed skepticism to refute.
Mike Elzinga first brought up the paper here, claiming:
ID/creationists have attempted to turn everything on its head, mischaracterize what physicists and chemists – and biologists as well – know, and then proclaim that it is all “spontaneous molecular chaos” down there, to use David L. Abel’s term.
Hence, “chance and necessity,” another mischaracterization in itself, cannot do the job; therefore “intelligence” and “information.”
And later helpfully posted here a primer on the first equation (Shannon’s Entropy equation), and right now I’m chugging through the paper trying to extract its meaning. I thought I’d open this thread so that I can update as I go, and perhaps ask the mathematicians (and others) to correct my misunderstandings. So this thread is a kind of virtual journal club on that paper.
I’ll post my initial response in the thread.
I haven’t read the paper carefully, but I noticed that in the Acknowledgments, Abel writes
This work was supported by a grant from The Origin of Life Science Foundation, Inc.
But, but, didn’t Abel himself found TOoLSF? In other words, he might as well have written
This work was supported by myself
On page 251, just after Abel gives Shannon’s Entropy equation, but which Abel calls the uncertainty measure (Equation 1), Abel writes:
That equation gives, as I understand it, the average number of bits per item in the string. And, as Mike points out, the flatter the probability distribution, the higher the entropy will be, and the more bits, for given string length [ETA and alphabet size], the string will contain.
But Abel seems to me conflate “random” with “equiprobable”. He says that the most “complex” string (presumably for a given length) is the most “random” string, whereas the most “complex” string, surely, is a string in which there are a large number of possible characters occurring with equiprobable frequency. So for a string length of 1000, the most complex string of integers would be the integers from 1:1000. And it seems to me it would make no difference to the entropy of the string, and therefore to the complexity whether the integers were rank ordered or totally jumbled, as long as they retained a flat distribution.
But Abel then says, in the next two paragraphs:
Is he saying, then, that a string consisting of the integers from 1:1000 in rank order has no complexity in Shannon terms? If so, this seems simply wrong. Shannon complexity/uncertainty doesn’t take into account the order of the items in the string at all, merely the frequency distribution of each item type. If I know the order of the letters in the English alphabet, I can be very confident that the next letter in the string: ABCDEFGHIJKLMNOPQRSTUVWXY will be Z.
But that doesn’t make the entropy of that string any less than the entropy of this one: NROGYATWHXMLBVIFJDQEKUPCS
Both have 25 different item types and in both strings the item types are equally distributed (one of each).
So his idea that complexity (as he seems to define it) and order are at opposite ends of a single dimension is simply wrong. Simple ordered, simple unordered, complex ordered, and complex unordered strings are all perfectly possible, although it will be true of course that if we plot all possible strings by complexity versus order, the highest density will be for complex unordered strings and the lowest density for complex ordered strings.
At least that’s how it seems to me – any comments? Have I missed something important?
Well, there are certainly symptoms of crank present! But cranks are occasionally right, so I’m interested here to check out what he’s saying.
He seems to me to be making the same error as Dembski.
What creationists and ID proponents say about information is mostly nonsense.
Shannon’s theory was a theory of communication, and his measure of information content is a measure of the channel carrying capacity. That channel capacity can be less than the bit capacity, because some of the bits might be parity bits or other redundant information.
As you correctly say, Shannon does not distinuish between an unordered string or an ordered string, unless the ordering is imposed by a protocol that has the effect of limiting channel capacity.
Kolmogorov complexity, sometimes also called Solomonoff complexity or Chaitin complexity, does distinguish between an unordered string and an ordered string. Whether Kolmogorov complexity is useful is probably still controversial, though that never stopped creationists from trying to put it into their dubious arguments.
Getting back to Shannon information, I’ll note that Shannon was concerned with communication between intelligent agents (mainly humans or technology built by humans). The usefulness of a string would depend on the use to which those intelligent agents put it. Abel seems to be trying to talk of usefulness independent of how intelligent agents use it. I’m inclined to think that’s mostly nonsense.
Elizabeth,
Those are indeed legitimate points.
I’ll have much more to say about this when I get home. My time on this computer is limited.
But if anyone wants to look ahead (it doesn’t make much difference in which order one reads the pages of this paper), jump to pages 255 through 257. This is the only place in the paper that Abel actually does any calculation, and it is this calculation and Abel’s assertions that reveal that this entire paper is a non-sequitur.
All of Abel’s misconceptions and misrepresentations as well as those of Dembki and Sewell are revealed in just these few paragraphs.
Everything else in the paper is simply pseudo-philosophical filler and various rephrasing of these fundamental misconceptions.
Look closely at Abel’s example under the figure on page 256. Make sure you follow it. The calculation itself is done correctly. Make sure you understand what Abel is using as the probabilities. Then finish the rest of Section 3
For those who may not know the chemistry and physics, the silliness of it may not be evident on its face. But the first thing any respectable chemist or physicist would ask is, “what does any of this have to do with how atoms and molecules actually interact?”
I’ve used up my time. Gotta go.
I will blog my own impressions as I read through. Here is
Part I
Abel’s opus has all the appearances of a serious scientific paper. Lots of technical terms, hundreds of references, and even a funding acknowledgment. On closer inspection, it turns out to be complete gibberish. And yes, the funding is provided by Abel’s own shell foundation.
The very first sentence of the paper gives a preview of coming attractions.
The reader is left to ponder: WTF? Why would anybody confuse catastrophe with Abel’s theory? What is an “abstract conceptual nonlinear dynamic model?” Why are linear models disqualified? Can a model be non-abstract and/or non-conceptual? Lots of questions.
The introduction continues with a relentless stream of gibberish. Having given a list of items in which “life-origin science is not especially interested,” Abel tells us that its goal is to understand whether “objective chaos, complexity and catastrophe” can generate “Bona fide Formal Organization [4].” If you don’t know what the heck BFFO means, don’t be ashamed: the only instance of this term exists in the very paper we are discussing. Perhaps realizing that, Abel helpfully explains:
Just reflect for a moment on this string of identifiers, taken from different fields of human endeavor and thrown together as a pile of pick-up sticks (Fig. 2a). “Computationally halting” refers to an abstract problem in computer science. Algorithm optimization is an engineering problem, but I am not sure what it means for organization to be “algorithmically optimized.” (Thoughts?) And why does formal organization (“an abstract, conceptual, non physical entity”) have to produce integrated circuits, of all things?
After some ruminations about emergence and complexity, Abel proceeds to formulate his “null hypothesis,” which he will try very hard to disprove, but in doing so (you guessed it) will fail.
“Physicodynamics” is a rarely used term. In Abelspeak it means, presumably, known natural processes such as chemical reactions. I am not a biologist, but I suspect that organisms don’t normally engage in computational halting (unless they are computer scientists) and don’t produce integrated circuts (again, with a few exceptions). Why these are interesting questions remains unclear.
(to be continued)
Part II
There is lots more fun to be had with the introduction, but that is left as an exercise for the reader.
In Section 2, Abel attempts to explain the concept of complexity.
Ref. 1 is to another paper of his own, so it’s hardly “pristine,” but Ref. 183 is a legitimate book on the Kolmogorov complexity. It means, presumably, that the concept of complexity dealt with in this section is Kolmogorov’s. Except that randomness is an odd term in that context.
Let me explain. The Kolmogorov complexity is defined for a single object, in this case a string. It is the minimal length of a program that can generate a given string. No randomness involved.
Randomness is involved in Shannon’s definition of information. Shannon treats messages (bits, words, or sentences) as random variables. They come from a random source with specified probabilities. Shannon’s entropy is a measure of average surprise upon receipt of a message. If only one type of message arrives time after time, there is no surprise. If every message has an equal chance of appearing, the surprise is maximized.
These concepts are quite different. For instance, the number π has a low Kolmogorov complexity: instructions for computing it are not complicated. At the same time, the Shannon information of a stream of its digits (appearing with equal probabilities) is high.
Eq. (1) is Shannon’s measure of information. So Abel conflates the two concepts. He does not understand this subject but pontificates about it as crackpots often do.
(to be continued)
Let me see if I have this right. Under Shannon’s treatment, the information content of a message is its surprisal value, in other words, whatever is conveyed that was not previously known by the recipient? Thus, as you say, any subsequent transmissions of that same message to the same recipient will have no surprisal value and, therefore, presumably no information content? In other words, if exactly the same message can have provide information on the first transmission but not on any others, is it fair to say that this version of information is not so much a property of the message itself but refers more to a process, the change that it effects in the mind of the recipient?
OK, so it is now clear that by “random”, Abel means “equiprobable”, and by a “complex” string, he means a long string with equiprobable elements.
He calls a string “ordered” if either the ordering of the elements means that it can be algorithmically compressed (page 252), which of course makes no difference to the Shannon Entropy, so he is simply wrong here, or if the elements aren’t equiprobable (top of page 257), which does.
So he seems completely muddled regarding a major plank of his argument. According to Abel, the sequence:
1 2 3 4 5 6 7 8 9
which is algorithmically simple (each term is the previous term +1), and which indeed I just actually generated using such an algorithm (such is my dedication)
must be less complex than
9 6 2 1 3 8 7 4 5
which was generated by randomly sampling, without replacement, from the previous string. In both strings, as in my alphabetic example above, each item type is equiprobable, so the Shannon Entropy of the two strings is identical:
3.1699 bits
On the other hand, the string
1 1 1 1 1 1 1 1 1
which I generated simply by omitting “+1” from the algorithm that generated my ascending series above has much lower Entropy than either of the other two. Indeed the Entropy is zero.
But Abel ( I assume) would also say that this string:
2 2 1 2 3 1 2 4 1 2 1 2 2 1 2 2 2 3 1 2 1 1 1 1 2 2 2 1 3 2 1 1 1 2 1 2 2 2 1 3 1 1 2 3 1 3 2 2 2 1 3 1 1 1 3 2 2 1 1 2 1 1 1 2 1 2 2 1 2 2 1 2 2 3 1 1 3 1 2 2 2 2 1 2 1 1 1 2 1 1 4 1 1 1 1 1 1 3 2 3
as this one:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 4 4
In both strings, the probability of each digit is as given on page 256: .46; .4; .12;, .02 respectively, i.e. the probabilities are not equal.
And both strings have the same Shannon Entropy, namely 1.524, as Abel correctly computes.
Fortunately for Abel, this muddle doesn’t matter because he then says that neither “order” nor “complexity” can generate “utility” aka “bona fide functional organisation”, which, interestingly, is different from Dembski’s argument, which is that some strings can have high “compressibility” (one of Abel’s two kinds of “order”) as well as high complexity (high Entropy in a long string), and that it that pair or properties in a pattern that tell us that a pattern is “designed” even if we do not know what the pattern is for.
Dembski is right of course that some patterns can be highly compressible as well as highly complex, but he is wrong that only “designed” patterns can have this pair of properties – maybe keep this for another thread!
So Abel then brings in the idea of “Functional Sequence Complexity”, for which he refers us to work by “Durston and Chiu at the University of Guelph” who “developed a method of measuring what they call functional uncertainty”, for which he disingenuously references a paper (ref 77) by “Durston, K.K.; Chiu, D.K.; Abel, D.L.; Trevor s, J.T.” which is pretty cheeky, I’d say. He repeatedly refers in highly complimentary fashion to this work as being by “Durston and Chiu”, without indicating in the text that he himself is an author on the paper (“Subsequently, Durston and Chiu have developed a theoretically sound method of actually quantifying Functional Sequence Complexity (FSC) [77]”
Another tick on the crank scale chart I’d say. But no matter. I turn to the referenced paper, and it turns out that “FSC” is essentially a measure of how conserved a functional protein is computed as something like Mutual Information, and also gives a specific measure of how conserved an amino acid is at any given site.
Cool. It tells us that similar proteins are similar, and that they are most similar at sites that are most important for the function they seem to serve.
Quick summary so far:
Haven’t reached the punchline yet.
Also the scholarship is terrible. There are 335 references, and not only are some of them to completely irrelevant papers that just happen to contain the phrase referenced (e.g. “Edge of Chaos”) I have no confidence that any of the sources actually say what the author claims they say. Take Monod: Abel writes:
Reference 174 is Monod’s book with the English title “Chance and Necessity”. But in that book, which Abel has clearly not read, Monod says no such thing. The title of his book is a quotation from Democritus: “Everything existing in the Universe is the fruit of chance and of necessity”, and for Monod it is “chance” that governs variance generation (“The initial elementary events which open the way to evolution in the intensely conservative systems called living beings are microscopic, fortuitous, and totally unrelated to whatever may be their effects upon teleonomic functioning.”) but “necessity” that drives natural selection (“Drawn from the realm of pure chance, the accident enteres into that of necessity, of the most implacable certainties. For natural selection operates at the macroscopic level, the level of organisms”). In other words Monod agrees with Democritus.
The reason I am sure that Abel has not read Monod’s book is not simply that he misunderstands Monod’s position re the title, but he totally ignores Monod’s main thesis which is that “function” in biology, refers to what perpetuates the organism and/or the population (what Monod calls teleonomy), which is highly relevant to Abel’s thesis, as far as I can tell. I suggest he reads the book.
William, are you convinced yet that Abel is not the Great White Hope of ID? 😉
Your understanding is about right, although I would have to sharpen it up a bit.
In Shannon’s treatment, information is defined not for a single message, but for a large set of messages. If the same message is repeated over and over then the Shannon information content is zero. If two kinds of messages are repeated with the same frequency, then each message is worth −0.5 log(0.5) − 0.5 log(0.5) = 1 bit of Shannon information. (The logarithms are to base 2.) If the message is “Happy birthday” 99 times of 100 and the rest are “I love you” then the Shannon information is −0.99 log(0.99) − 0.01 log(0.01) = 0.081 bits per message.
One can say that Shannon treats messages as a bulk commodity, sort of like a truck driver who just needs to deliver and unload them. In contrast, Kolmogorov wants to appraise the values of individual messages and treats them as one-of-a-kind objects.
The book by Li and Vitanyi (Ref. 183 in the paper) has some quotes from Shannon and Kolmogorov on p. 603.
Here is a little factoid that will no doubt cheer up ID fans. Discovery Institute maintains a list of Peer-reviewed & peer-edited scientific publications supporting the theory of intelligent design. Currently there are some 50 papers on it. About one quarter of those (13) are authored or coauthored by David L. Abel.
So this virtual journal club entry is not entirely in vain. We are discussing a paper by the most prolific* ID writer. 🙂
*Judged by the number of publications. Dembski probably has the largest page count.
It appears that everyone is picking up on the same problems with Abel’s paper. 🙂
I expect to get home by early evening, and I’ll add some comparisons with physics.
There are some other issues that come up as well, and these fall into the realm of psychology I would assume. For example, why do papers like this capture the imaginations and devotion of those in the ID/creationist community?
More generally, what are crackpots able to pick up on that allows them to write so persuasively for such an audience? Those of us in the science community should probably pay attention to this.
Gotta hit the road.
OK, I have finished the paper. The remainder seems to consists of a string of assertions, AFAIK, supported only by references, not arguments or evidence presented within the paper, most of which are to Abel’s own work.
It is also very badly written, using anthropomorphic phrases that attribute agency to abstract referents, even though core of his point concerns willed agency:
No, “Language” doesn’t “use” any such thing. Language users do.
Indeed it can’t. Nobody claims it does. Intelligent organisms plot and scheme, not “physicality”.
Yes indeed.
In short, Abel constructs (badly) straw men, then asserts (does not demonstrate) that the straw men don’t explain biological function, all the while making rookie mistakes with the basics of information theory.
Oh, and cites irrelevant papers for no other apparent reason than that they use certain phrases, as well as at least one book he either has not read or has radically misunderstood.
Next?
Exactly. The paper’s sections follow the same recipe:
* mention lots of science-y terms from disparate fields of human knowledge (hey, no one can understand all of them),
* cite lots of references (look, all these legitimate people support my statements),
* add some editorial content that does not follow from the above and is just silly if one pauses for a minute to think about it.
This goes on for 45 pages, so one needs great stamina to read through this crap and make sure that it does not contain anything of value. Fortunately, Section 2 is short (just 4 paragraphs), yet entirely representative of the rest of the paper. If you want to get a flavor, go ahead and read it. As I discussed in Part II, it begins with a discussion of “complexity” that conflates the rather distinct concepts of Kolmogorov complexity and Shannon entropy. Having accomplished that, Abel comes to this conclusion:
Indeed, if you pick letters at random, the resulting string will likely have high Kolmogorov complexity. But surely no one measures functionality of a DNA molecule by its Kolmogorov complexity. This is one of the straw men sprinkled liberally throughout the paper.
Not quite. The Shannon entropy is a measure of the compressibility of the symbols of the message based on their frequentist counts relative one another. It really helps to have a visual aid for this so see: Sannon-Fano and Huffman coding for a tree representation of how the bit strings are produced. (Specifically Huffman.) The idea is that you want to represent your alphabet in variable rather than fixed length bit encodings while still being able to uniquely differentiate each symbol from the next. The trees will give a good grasp of where the logarithms come into play.
So there’s three ways to look at it. The second message *is* the symbol and so has no content. Each message is its own alphabet, which is the way winzip and other file compressors approach the problem. Or that you have a broad language wide frequency count for the alphabet, which is how Shannon approached it. In the last the entropy for any message, for itself, may be different from any other.
An oft overlooked issue here is that the proteins are encoded from DNA. In which base pairs not only serve as an ‘information’ encoding but also as a structural component of the DNA itself. So any given site may be conserved because it is a necessary structure for the DNA and/or a necessary structure for the folded protein. It would be good to see more research done on this in the future.
Well, I wasn’t overlooking that (although I’m not aware that protein-coding regions of DNA are constrained by the pressure to preserve the structure of DNA itself, althoutg you could be right), and it makes no difference to my point: that what the FSC measure delivers (and it seems pretty well equivalent to MI) is a measure of similarity between the proteins, and specifically, the measure of similarity at specific sites.
What is that supposed to tell us?
Elizabeth,
There’s a basic misunderstanding at work here about Shannon info theory and Shannon Entropy. Abel isn’t helping, of course, but a couple points to bear in mind:
Shannon entropy is always a measure of a string against a model, as a configuration from a known phase space. That means, that you cannot measure the “entropy of a string” in isolation from a model of the source whence it came. The entropy is dependent on the nature of the source. That means that if you have a truly random source for the string in question, “random” is your model, and governs the entropy calculation for any string drawn from the random source: all successive symbols are equally surprising, and thus have maximum entropy.
Applying this to your post, then, the entropy for:
1 1 1 1 1 1 1 1 1coming from a random source is EXACTLY the same as the entropy for:
9 6 2 1 3 8 7 4 5or
9 6 2 1 3 8 7 4 5when these strings come from that same random source. That means that for a random source, the Shannon entropy for any string from that source will be identical, and the same, for any possible string drawn from that source. This is tru by the definition of ‘random’: every single symbol is equally surprising, and reduces our uncertainty by exactly the same amount as it arrives from the source. If it does not, then by definition, the source is not random.
Alternatively, then, if our source is a generator for only the symbol “1”, then the entropy for the string:
1 1 1 1 1 1 1 1 1Is zero. In this case, we know a priori, (and this is crucial!) that the source in question only emits the symbol “1”, and so we have zero surprise value when each successive “1” symbol arrives from the source. We have reduced our uncertainty by nought when each “1” arrives, as we know ahead of time, what each new symbol must be.
So, please note that the same string has different entropy values, based on the source when whence it came. The entropy of a string is not meaningful without referring to what is known about the source. Entropy, then, is a measure of the relationship between an emitted string of symbols, and the nature of the emitting source.
I’ve more to say on this, but have to run out for a bit. Just hoping to keep things in line with how info theory really works in practice.
eigenstate,
Thanks for that. Yes, I do understand that, I think. I was making the (unsupported) assumption that the frequency distribution of symbols in the message is a representative sample of available symbols – and the reason I did so was because so many ID arguments for intelligent-pattern-detection assume that you don’t have any information about the source other than those in the “message”.
Wasn’t rebutting you, just noting that DNA is the poor cousin of DNA.
But all I take away from bitstring similarity is bitstring similarity. Without context of what the differences cause it doesn’t mean much. For analogy, in computers a single bit flip can be entirely meaningless. Or it can make all the difference in the world, and in completely baffling manners. And that’s in software where there aren’t chaperones to futz with.
For ID to make their case they need to do a deeper dig into just what it is they’re looking at. Of course, that’s the same thing the Darwin camp ought be doing as well for the same reasons, if not different philosophies.
Elizabeth,
I am afraid that eigenstate’s comment might saw some confusion. How come strings 111111111 and 962138745 have the same Shannon entropy? One can answer that the entropy refers to the ensemble’s probabilities, rather than to the individual strings. But at the end of the day, Shannon’s entropy is computed for actual systems, where we are given a stream of messages, and not necessarily a probability distribution.
How do we deal with this in practice? We extract a probability distribution from a long enough sample of messages. The observed frequency of a given message approaches its probability as the number of messages goes to infinity (the law of large numbers). So the frequency of digits in a sufficiently long numerical string should be representative of their actual probability distribution.
In view of this, a long string of ones is indicative of a random source with the probability distribution where p(1) = 1 and p(0) = p(2) = p(3) = … = p(9) = 0. This probability distribution has zero Shannon information per digit.
Strings 111111111 and 962138745 exhibited by eigenstate are not sufficiently long to infer their probability distributions reliably. But I would venture that the former comes from a source with low Shannon entropy. The latter is a bit more tricky: each digit occurs exactly once. This is not a typical string with a uniform distribution.
OK 🙂
?
Well, they need to dig deeper into what it is they are claiming, I would agree.
Not sure what “thing” you are talking about. Can you explain?
Yep, nailed it.
Apologies, both teams need to be tackling the same problems here. Well, I say that, but the Darwin side of things is largely a history issue while the ID side is, whether it intends it or not, a forward going engineering issue. But I’ll still maintain that both camps need dig deeper into structural issues of DNA and the great nasty of protein folding to put the necessary context in place.
? The Darwin side of DNA structure and protein folding is a history issue? I have no idea what this means?
Well, I actually generated all the strings in my post algorithmically!
I mean I actually wrote a little script (in MatLab). I take both yours and eigenstate’s point though, that the strings are two short for anyone to have any confidence that the outputs are a representative sample of what might have been output. After all 11111111111111 might mean something rather important, like I’d fallen asleep briefly on my keyboard, although nnnnnnnnnnnnnnn might be more likely in that instance.
But, yeah.
So, I’m just flabbergasted by this paper. I’m not a scientist by trade, but I am by now comfortable reading scholarly literature on areas of interest, and keep up with scholarly journals of interest in computing (JACM, IEEE journals, etc.)
I never run across a paragraph like this, at the top of page 250:
I read that and think I’m reading an April Fool’s day prank in this journal.
Maybe I’m just not well read enough in the literature. But how does a paragraph like this get by the reviews and the editor?
Or this, from the previous page (249):
Really? Really?
I know IJoMS is one of the “open access” journals, which means authors and publishers pay to have their work published, and the articles are free to the public, but there is obviously no serious scholarly review in place here.
I’m just getting to the parts that are at least nominally concerned with complexity, info theory, biology, chemistry, etc., so I’ll try to contribute something substantial to the discussion from reading that, but…. wow.
It is pretty bad.
I’m checking out The Cybernetic Cut (which I have in fact read, once, but will try harder this time).
Really, David Abel is seriously confused. In The Cybernetic Cut (page 256), he writes:
He has confounded the fitness function with the evolving solution. Evolution of course does have a goal, or rather, evolution follows the path of greatest fecundity. If greatest fecundity involves solving an environmental problem, like being inconspicuous to predators, or being thought cute by some other species, or outputing investment strategies given market inputs, or not breaking your bones when you fall out of a tree, or being attractive to a mate, then that problem may well be solved, not because that is “evolution’s goal”, but because that serves “evolution’s goal” of persisting.
This is as true for a GA as it is for a peke as it is for a bacterial population faced with a new antibiotic as it is for a peacock faced with attracting a mate in the face of competition from other peacocks.
No information is “smuggled in” to GAs that isn’t equally “smuggled in” to evolving populations by the environment they adapt to thrive in.
Elizabeth,
Length is a key factor in measuring complexity according to K-C theory; because algorithmic complexity requires both the “runtime” and the specific code to generate a given output string, for short strings a “simple” string may not possibly be generated by a shorter generating program than a “complex” string.
But here, the length of your strings is not the problem. If you had used strings that were 10,000 chars in length, and having the same constitution (all ‘1’s, a random sequence, etc.), nothing would change.
If you are dealing with a random symbol source for your string, a string of 10,000 ‘1’s is just as surprising (reduces uncertainty) as any other string you might pull from the source. Given a source that randomly generates numbers from 0-9, all generated strings, no matter what their content, will be the same.
It’s definitional; if the entropies are different, then by definition the source if not a random source.
That said, we often (in fact nearly always) have to deal with limited or incomplete information about the source, so we model the source based on statistical analysis of observed output. There is no authoritative model for the English language as a symbol source, but if we analyze large amounts of English text, we can derive a statistical profile of the symbols. Researchers like Ronald Rosenfeld have analyzed terabytes of English text and come up with entropy at approximate 1.23 bits per character, and have determined observed letter frequencies for each character in the language (see here for example freq data).
So models for a source can and have been built without normative descriptions of the source, but these require large samples, way more than what you and I would call a “long string”. These samples would be the statistical basis for what you described as “a representative sample of what might have been output”.
eigenstate,
Thanks. Will read, mark and inwardly digest 🙂
Elizabeth, at the risk of harping on this, let me add one more thing. I should really have responded first to this sentence from you, because it’s the key. Since you used MatLab to write a script to generate your strings, you know, authoritatively, the “phase space” for each source, each script.
For you script generating “1111111111”, you know, with certainty, that the only symbol that can be emitted is “1”. Doesn’t matter how many you emit, a 10 char string or a 10 million char string, there is no reduction in uncertainty in any string that script emits.
Now, for your “random script”, you know that the symbol space is [0-9], because that’s how you wrote it. Since it’s random (OK, likely pseudo-random because we are using conventional computers), in its symbol generation, “1111111111” would be highly surprising, and would reduce our uncertainty with every successive character (but no more surprising than any other random string of [0-9]).
Anyway, I just had the lightbulb moment that your scripts were the key to our joint understanding here. The fact that you wrote them, and have them in hand illustrates how and why the entropy for “”1111111111” changes as the output from your “1”s script to your random script.
Ok, back to the Abel…. whatever-it-is.
eigenstate,
I enjoyed the article on letter frequency. For my itatsi game I computed letter frequency, letter pair frequency, triplets frequency, etc, for six languages and used this to assign fitness values to randomly mutating strings. What I have is an operational definition of what it means for a string to look like a word.
Such an oracle can evolve words and wordlike strings very efficiently. You can compete against the oracle, but you can’t beat it.
I don’t claim it’s like biological evolution, but it does demonstrate that “intelligent” selection is not as powerful as an oracle that can integrate all dimensions of fitness.
Hmmm… I disagree. Rather than evolution, you’re talking about natural selection, for which greater fecundity is a “goal” only in a teleonomical sense.
I skimmed through the paper and picked section 4, “Autopoiesis”. There he introduces the concept of autopoiesis* and claims that it is inadequate to explain OOL, claim he supports with a quote explaining that autopoiesis is intended to describe what life is, not how life came to be. That is true, but I see no indication that anyone in OOL research is confused by it (possibly except Abel himself [if we take seriously the claim he’s an OOL researcher]). After that, he apparently makes no further use of the concept, so I’m left wondering what was the point of the whole section. Does anyone have a clue?
I’m not strong enough to read any further.
* In doing so he incorrectly spells the name of Humberto Maturana, and later “Valari” for Varela.
On Page 255 of his paper, Abel – in referring to a pile of pick-up sticks – says this:
One has to wonder what Abel thinks could come of such a pile of sticks. He certainly must know that they will eventually combine with other elements in the universe.
And then he makes this assertion:
Who does he think is doing this? The implication seems to be that scientists are doing this, and if so, it is a grotesque mischaracterization of scientists and what they think.
So what comes of Abel’s calculation of the “Shannon uncertainty” for, say, a simple system consisting of only two constituents taken from the imagined “primordial soup” in the way Abel suggests in his example on Page 256?
Let’s look at two constituents (A, B) and set the “availability probabilities” at p(A) = ½ and p(B) = ½ and then consider various constituents for A and B. Elizabeth already looked at some cases involving letters and numbers and demonstrated that various arrangements of those numbers and letters had the same “Shannon uncertainty” according to Abel’s calculations.
Consider the following, all of which have exactly the same “Shannon uncertainty” according to Abel’s calculations.
(black marble, white marble), (0 , 1), (X , Y), (electron , proton), (hydrogen , fluorine), (sodium , water), (carbon , hydrogen), (fluorine , silicon dioxide), (hydrochloric acid , zinc), (fox , rabbit).
Clearly things with the same “Shannon uncertainty” according to Abel have very different “capabilities.”
Try another two-constituent case in which p(A) = 1/3 and p(B) = 2/3.
(black marble , white marble), (white marble , black marble), (0 , 1), (1 , 0), (electron , proton), (proton , electron), (hydrogen , fluorine), (fluorine , hydrogen), (oxygen , hydrogen), (hydrogen , oxygen), (hydrogen , sulfur), (sulfur, hydrogen), (carbon , oxygen), (oxygen , carbon), (fox , rabbit), (rabbit , fox).
Chemists will recognize the issues of stoichiometry in these examples.
Abel is projecting a set of preconceived notions and a prescribed set of requirements onto matter and energy and then attempting to leave the impression that “physicodynamic” processes aren’t up to the job. This is supposed to open the way for “information” and “intelligence.”
But the complete argument is a non-sequitur because matter and energy do not behave in the way that Abel implies. Atoms and molecules and their increasingly complex assemblies are not inert things sitting around waiting to be sampled with a uniform random sampling distribution and placed in pre-specified arrangements.
With matter and energy, what you get is what you see. Science takes it apart to find the rules and the dynamics. Those are the rules that apply to matter and energy, not Henry Morris’s, rules, not Abel’s nor Dembski’s, nor Sewell’s, nor any other ID/creationist “theorist’s.”
I’ll do a separate comment on comparing thermodynamic entropy with Shannon entropy.
For those who might be confused by “Shannon entropy” and thermodynamic entropy, here is a comparison.
From the statistical mechanics perspective, entropy is defined as
S = – k ∑p(i) ln p(i)
where k is Boltzmann’s constant, and i goes from 1 to Ω, which is the number of energy microstates consistent with the total energy of the thermodynamic system.
Note that the summation is exactly the same as in the Shannon entropy and is the purported reason that John von Neumann suggested to Claude Shannon that Shannon call his expression entropy because nobody knew what it meant.
As such, the expression for thermodynamic entropy behaves in the same way I described earlier for Shannon entropy. It is maximized when the probabilities of all microstates are equal; and for Ω microstates,
p(i) = 1/Ω for all i.
Then the above expression reduces to
S = k ln Ω,
and this expression is the maximum value the entropy can take on.
Now here is the important difference from Shannon entropy. In the case of thermodynamics, a system in contact with a much larger heat bath at a fixed temperature will have a distribution over energy microstates for which the probabilities may be all different. So the first expression above applies.
When the system is removed from its contact with the heat bath and is isolated, all the probabilities become equal and we get the second expression, and the entropy goes to its maximum.
Here is the critical point. The probabilities become equal for an isolated thermodynamic system because all the microstates interact with each other by exchanging energy. The reason they do this is because matter interacts strongly with matter by various mechanisms such as the exchange of photons, the exchange of phonons (quantized sound waves in solids), or the exchange of other particles.
If the microstates themselves were isolated from interacting with each other, the entropy would not change by drifting up to a maximum.
And this is just the case for things like ones and zeros, or any other inert objects that do not interact with each other. When we see the ID/creationist theorists using entropy when discussing inert objects, that entropy can’t change by “isolating the system.”
There is another important difference with the thermodynamic entropy. Thermodynamic entropy links important state functions of the system to each other. In fact, the temperature, T, of a thermodynamic system is defined buy
1/T = ∂ S/∂ E
where E is the total energy of the system. In words, the reciprocal of the temperature is defined by how the entropy changes with the corresponding change in total energy.
So thermodynamic entropy contains implicitly within it the physical interactions of matter with matter and the physical dynamics of how the number of energy microstates changes with total energy.
Shannon entropy does none of that. Thermodynamic entropy is about the physical universe, and Shannon entropy is simply one of a number of ways to study patterns in the data of one’s choosing.
Elizabeth,
It’s not about what Shannon entropy actually is and how it is actually used; it’s about how Abel abused it.
He was pretty clear at the bottom of Page 256 on how he got his probabilities from his “primordial soup.” He calls the probabilities “prebiotic availability” in one sentence and “hypothetical base-availability probabilities” about two sentences later.
So he appears to be sampling from an imagined, infinite “soup” in which removing a few of the constituents makes a miniscule change in the probabilities for those constituents.
If he were sampling from a relatively small “soup” of constituents, he would have to be sampling with replacement in order to keep the probabilities constant for each of the various constituents.
So Abel is calculating what he calls “Shannon uncertainty” in order to concoct some sort of prescription that “physicodynamic processes” can’t handle. It’s just plain bogus.
Complexity in physics permits more things to happen; but it is the underlying interactions among atoms and molecules and the energy from the energy bath in which they are immersed that do all the work.
I might have included another example of Abel’s calculation by using a two-constituent case in which p(A) = 10^-16 and p(B) = (1 – p(A)).
Then look at (A, B) for (phosphorus, silicon) as compared with (boron, silicon).
In the physics of solid state devices, the first is what is known as N-type silicon, and the second is P-type silicon. Quite different.
Just using Abel’s concoction leads to all sorts of silliness.
I see the UD mele has spilled out of the bar into the alley and has now swelled into a full fledged riot in the viral streets. Apparently my ability to add and subtract integers had in fact not eroded, and I can simply no longer post to UD.
I have been hereby attracted to the mathyness of this post. Now I must look at this paper and see what all the excitement is about.
Welcome to TSZ, junkdnaforlife 🙂
I take the points of all who have pointed out that the the Shannon Entropy of a message doesn’t simply depend on the distribution of symbols in the message but on the distribution of symbols in the population from which it is drawn. Or something.
But am I right (and is Abel simply wrong?) that ordering the symbols in a message (by ranking, for instance) makes no difference to the Entropy?
Remember that Shannon information is defined not for a single message but for a stream of messages. If you reorder symbols in a single message then you change the message. But that’s probably not what you meant.
If you treat a message as a stream of symbols and want to compute their Shannon information then the order of symbols has no impact on their frequencies and hence the calculated Shannon information. In that sense, Abel is wrong.
However, if we are talking about the Kolmogorov complexity of a message then it’s a different story. A message in which symbols are ordered may be described in a shorter way, e.g., “26 As followed by 40 Bs, 37 Cs…, and 25 Zs.”
The problem with Abel is that he conflates the two concepts.
Neil Rickert,
Shannon’s theory had nothing to do with information!
Well, it had little to do with meaning.
Abel rightly makes that point, but AFAICT wrongly implies that symbols arranged in a meaningful order has less Shannon Entropy than the same symbols shuffled randomly.
For this reason he rejects both “complexity” and “order” as measures of “functional sequence complexity”.
But his measure of “functional sequence complexity” (FSC) seems to mean some measure of the mutual information (the similarity, essentially) between DNA or protein sequences that have the same biological function.
Then he just sort of leaves it, and talks about something else.
Well, he both conflates them and says they are incompatible: that a maximally complex string is the opposite of a maximally ordered string (where sometimes ordered seems to mean minimum Kolmogorov complexity, and sometimes means non-equiprobable distributions of symbols).
I think he’s wrong on both counts.
But he rejects both, and plumps for “FSC” which doesn’t seem to get him any forrarder as far as I can tell.
Hey if you people could refute Abel you would- but obviously you can’t.
Your position can’t account for transcription and translation- and that should be a problem.
Well, we can refute his about what his equation 1 indicates because because he’s clearly misunderstood it. We can’t refute much else because it doesn’t even make a lot of sense.
What do you think his argument actually is?
It is, which is why there is a lot of research in that area.
LOL.
It has a great deal to do with information. It is, however, not a theory of meaning.
When I started studying human cognition, I had the same view as you of Shannon’s theory. But, as my investigation has progressed, I have had to reassess that and I now have far greater appreciation for Shannon’s ideas.
Thermodynamic entropy has absolutely nothing to do with order/disorder. I can provide an example if anyone is interested.
The Shannon entropy uses exactly the same mathematical calculation and simply attaches a number to the average of the logarithms of the probabilities, no matter what those probabilities represent and no matter what order. Like any average, order doesn’t matter.
In order to discuss order, one has to have some criterion for determining what one means by order of the entities one is discussing, and then one needs to make use of some knowledge of the permutations of those entities.
I sure would like to see some comment on coding sequences that indicates what advantage a designer has over evolution. It’s one thing to assert (without evidence) that functional sequences are rare and isolated. It’s quite another to assert there is some non-magic way of knowing how to find them.
Human engineers can calculate whether adding a foot or an inch to the length of a cantilever will cause it to break. What is the equivalent body of knowledge for protein design? How does the designer know the effect of a point change?
The corresponding concept in thermodynamics is called an ensemble. This represents all possible configurations of the energy microstates that are consistent with the total energy of the thermodynamic system.
Anyone want to weigh in on FSC?