A working version of FMM’s design detection game is available.
Download and install the applicable version of “Processing”.
https://processing.org/download/?processing
Get the fifthmonarchyman progam code from here, and paste it into the Processing script area.
Sample data here
http://pastebin.com/raw/MjV8RmvW
You need two files in the same folder as the Processing executable.
real.txt and fake.txt
The testing and such starts here
test strings
please explain exactly why you think non-computability can not be verified.
It is mathematically proven that if a close model does not help to describe an object then the function that produced the object is non-computable
peace
have you played the game? have you even tried?
peace
Just answer the question, please. Are the two data sets I acquired from published sources acceptable to you as real?
Yes or no?
When we say a string is “real” all we mean is that it is the actual string you are evaluating. rather than a model or randomized copy (a fake).
So YES any string you wish to evaluate qualifies as a real string.
why is this so hard for you?
peace
That’s not the underlying question. The underling question is how can we — independently of your game — supply non-computable numbers for you to evaluate?
why is that the underlying question?
One point of the game is to help us collectively determine if a string is noncomputable.
If you had another method to determine this you would not need the game
peace
So the claim that you can identify non-computable strings cannot be verified?
You claim to have a method, but you have no way of independently and objectively verifying the identifications.
Thanks
Let me summarize.
The operational definition of a nonrandom noncomputable string is a string identified as such by playing your game.
Is that fair?
There is no alternative, independent way to produce or identify such strings?
It can be verified by comparing my results with the results of other observers.
That is the whole point of making the game shareable.
peace
Think about this you are asking for an alternative independent way to identify an object as designed.
If we has an alternative independent way to identify an object as designed we would not need to work on developing one and there would be no ID controversy
On the other hand what the Game does is allow us to objectively compare and verify our design inferences with other observers with different goals and biases.
It’s this multiple observer check and cross check that makes the game different from other methods and constitutes independent verification.
peace
Rhine at Duke University spent 20 years doing something like this game, looking for ESP. That’s a long time, and a lot of work and a lot of data.
At least he had independently verifiable criteria for correctness or incorrectness.
But, if the test is blind and collects lots of data, the results could be interesting.
I think you have wasted a lot of people’s time by using the terms “real” and “fake” when those cannot be independently verified.
if 10 different observers can repeatedly choose the real string with 100 percent accuracy that would be a pretty good indication that there is a recognizable difference between the two strings.
I would think that would be an independently verifiable criteria for correctness.
Perhaps that is just me
peace
I think you have wasted a lot of people’s time by chasing rabbits instead of
1) helping with definitions
2) trying it for yourself
or
3) waiting till the game is complete
It is beyond simple
a real string is just the string you are evaluating a fake is just the model or copy you have produced
peace
But you are looking for non-computable strings, which are not even operationally definable.
fifthmvonarchyman,
I don’t understand the rules you’ve provided well enough to do so. Please take the strings provided by petrushka and provide step-by-step examples so there is no ambiguity.
how so?
we define a non-computable string as a string that a close model does not help to describe.
That seems like at least the start of an operational definition to me
peace
fifthmonarchyman,
You’ve said that you’ve played the game. It must be complete in some form, correct? If that’s the case then you can provide step-by-step examples using petrushka’s strings.
I can.
Does not mean I feel obligated to drop everything and do it right now.
I already provided a step by step example with OMagain’s string months ago. I don’t feel compelled to do it again at this second,
I rather work on some actual hypothesis testing instead of jumping through your hoops
peace
If after reading the original paper you still don’t know how to play the game
I doubt any amount of additional explanation will ever get you there.
It’s sad but sometimes additional discussion with some folks is just not worth the trouble.
peace
fifthmonarchyman,
I’ve read the Finance Game paper end-to-end several times. Based our interaction ending in this comment I suspect I understand it much better than you.
For example, in that comment I proposed this procedure to test your claim about that paper:
With the exception of using machine learning instead of human volunteers, this is exactly the process described in the paper. This is nothing like the game you’re talking about in this thread.
If you think you have a way of detecting design, let’s see it. Stop being so coy and asking other people to make the effort for you.
Did you ever even once play the game the paper is based on?
I really doubt it. You know the game is freely available don’t you? You can try it out if you want.
This kind of total lack of curiosity is something I have never experienced.
This just confirms to me that there has been no real communication between us in these matters.
Since you have no idea what Ive been talking about so far, I have no confidence that that will change by my jumping through more hoops.
long story short
When I get the time I’ll take a look at petrushka’s strings and i will drop what ever I’m doing and lend a hand to anyone who actually is curious enough to look for themselves.
in the meantime I will continue to wait on the shareable version before I present my argument
peace
fifthmonarchyman,
I listed the steps in the Finance Game paper game. They are nothing like the vague steps you’ve deigned to share with us about your game. If you disagree, show it by comparison with what I wrote rather than cutting out my words and replying without any content of your own.
I spent time reading and understanding the papers you referenced. You can’t even be bothered to provide a demonstration of how you achieve your extraordinary claim.
That’s why science insists on operational definitions.
The game that OMagain posted is just the graphical interface to visualise the strings, yes? (I can’t be sure because it won’t load my submitted string). It doesn’t generate the randomised or ‘model’ strings, correct?
Before we go much further you should be totally clear in your description of how you produce the ‘model that is close’. Present your detailed procedures of randomisation and the GA to get the model ‘close’ to the original. Just saying ‘I randomise’ and ‘I use a GA to get an Rsquared of 0.8’ isn’t detailed enough. If you use Excel, give us the functions you use so we can inspect the procedure in detail. Also post some data sets that comprise of an original, a randomized clone and a model, for us to inspect.
For us to know exactly what is done to these strings is essential to understand what the differences are that will be visualised, and what it might mean when we can spot them or not.
But before you do any of this, please post your OP with your hypothesis laid out in detail. Take your time, and get it right (and don’t forget to specify ‘close’).
This is more important than to get us feeding strings into a black box without really understanding what is being tested.
fG
Yes it’s just my original first ever coding effort. The randomized and model strings are done separately.
randomization works like this, I simply put the real string in a column in excel and then in the next column I put a random number in every corresponding cell then I sort smallest to largest or vice versa
getting a model works like this. I put the randomized string in an excel column then I mutate a portion of the numbers randomly creating a number of copies that are slightly different than the randomized string.
I pick the one with the highest r squared value when pared with the origin string. I place it into the place of the randomized string and repeat the process.
All of this is very crude simple stuff and it’s all open to being modified or changed as the situation arises.
I hope to see more elaborate EA’s for creating models and compare them to see if there is difference in results.
I think that it’s important to have a shareable tool to look at as we work on hypothesis testing.
When we get that I will do my best to put something together. I do hope for some feedback at that time
peace
here is the deal Patrick. I’m concerned at your 75% success criteria and your stipulation that you only look at a limited number of data sets
the “spirit” of the Finance Game paper game is vastly different than your summery.
We are talking about humans general ability to distinguish between actual data and randomized copies,
The reason I continue to ask if you have played the game is that I’m not sure you understand this
peace
I have played the game and failed to see what you claim to see. I’d appreciate it if you would take my datasets and sort them.
I provides a couple of examples of real data and randomized copies.
I think you need to take a look at some commercial games and learn how to write rules. No one understands your game, and when we attempt to play it, you say we aren’t doing it right.
fifthmonarchyman,
Why? The 73% success rate comes directly from the paper (which I note you have again failed to reference) and the paper only looked at eight data sets. Why would you hold a machine learning system to a different standard than the human players?
That’s exactly the kind of nonsensical goalpost moving I am trying to avoid by agreeing on specific details. There is no “spirit” of the process described in the paper — it is exactly as I summarized. If you disagree, show exactly where my summary differs from what is said in the paper by supplying quotes from the paper itself.
You claimed that the results from that paper could not be replicated by software. If you agree that I’ve summarized the process accurately and I can train a software system to meet or exceed the documented human performance, your claim is disproved. If you disagree with that statement, please explain exactly why.
It’s a classification problem. There are a number of machine learning algorithms that are quite good at those.
Your inability to clearly articulate the rules of the game and your refusal thus far to provide a detailed example of playing it suggest that it might not even exist.
The software exists. I posted a screen shot.
What doesn’t exist is a game having clear and unambiguous rules.
Even the description of the game is weird.
A few posts ago FMM said:
What is “actual data”? I keep asking, and what I get is the claim that any number at all will do.
But according to FMM on this thread, some numbers produce a characteristic “W” pattern. And some don’t.
As far as I can tell, data that produces the W is real, and data that doesn’t is not real. Or something. But there is no objective way to confirm this, because humans can see it and analysis programs can’t.
So I am at a loss as to how accuracy could be given a percentage. If you can’t confirm or disconfirm the human perception, it should be 100 percent.
It’s what FMM originally posted.
I have stuff drawing now. I’ll work on the backend shortly.
I don’t understand this. What are the limits on the random numbers that you put in the corresponding cells? Surely this will produce strings that have nothing at all to do with the original? And what with the sorting? Your originals won’t be sorted either, so why do this?
Why not assign to every cell in the second column the number from a randomly chosen cell in the original column, making sure not to sample any cell in the original twice? In that case the random copy will maintain the main statistical properties of the original (range, mean, standard deviation) but destroy any patterns, which I’d think is what you want?
This sounds like the WEASEL program. If you run it long enough, it should get you back to the original. Only thing I don’t understand is why you only mutate a portion of the numbers each generation. What portion, and why not all of them?
Anyway, my takeaway from this is:
– In some cases, you start with an original string that contains a pattern to some degree. The randomisation process will break this, and the GA will try to repair it. Because you stop the GA before the repair is complete, the ‘model’ will not fully contain the original pattern any more. When you compare original and model side by side, you notice this.
I have no problem with this, and have confirmed that this is what is happening in your example string by doing some time series analysis. What I don’t understand is why you think this is a remarkable result? Humans will be able to do this, and I have no doubt, so will neural networks (as Patrick keeps telling you). I don’t think this will be all that controversial.
– In other cases, the original string doesn’t contain a pattern, or only a very weak one. The randomisation/GA process will then produce another string without much of a pattern, and we can’t tell the two apart. No surprise here either. In this case you state that the game can’t say anything meaningful about the original string in terms of it being designed or not, and false negatives are not a problem.
You then seem to jump to the conclusion that a string with a pattern that is clear enough so that we can tell it apart from one where the pattern has been broken, is somehow indicative of ‘design’. You really need to lay out in an OP exactly what that means, and why you think this is the case. I don’t see why this would be a necessary conclusion at all, and I think there will be many natural strings that will come out as ‘designed’.
Now, I predict that you will counter by saying that in your world view everything is designed, but some things display ‘designed-ness’ more than others, hence false negatives. In that case, I find your game useless. A truly ‘reliable design detection tool’ should have three possible outcomes: designed/non-designed/ inconclusive. Even if when running the game in practice there would never actually be an ‘undesigned’ outcome (because your world view happens to be correct), at least the process should allow for such an outcome. This is basically the ‘falsification’ criteria that we need to call a theory scientific. Your game doesn’t have that, because it only has two outcomes: designed/inconclusive, and therefore it only serves to reinforce an a-priori metaphysical bias in the user, and it isn’t scientific.
I think it is far more important to have a clear and detailed hypothesis before we worry about how to best test it.
fG
In essence that is what I do. It’s a simple process. the random numbers in the second column is just the means by which I randomize the original
It is a lot like WEASEL except the target is an R squared of over 80% rather than the original itself.
There really is no correct way to to this. I’ve varied the portion mutated in different ways and put other constraints on the model as well.
The point is I want to produce a model that is close but not identical to the original.
Actually that computers can do this is controversial. This sort of pattern recognition has been suggested as an alternative Turing test.
I for one would be very surprised if Patrick can make good on his little two week hack. If he can do so the entire enterprise is falsified
I think that Patrick is coming to realize the difficulty in this as well. IMO That is the reason for his constant effort to lower the bar of success.
It all boils down to the connection between cognition and design.
It’s my contention that cognition is lossless information integration and that design is the inverse of this process.
more later
thanks for the interaction
peace
The pattern in the string is what the fuss is about.
I hypothesize that the “pattern” contains the distinguishing information in the original string and that this information is absent in models that are close.
That makes what I do when I learn the pattern in a string an integrating function. As such it is mathematically proven to be non-computable.
correct.
Why?
It seems to me that demonstrating that a sting is nonrandom and noncomputable tells you a lot about the process that produced it.
One of the hypothesis that the game will test is my contention that humans can duplicate the patterns we see in a string but algorithms can not unless they target the actual numbers themselves.
If we can separate stings into two piles computable stings and those that come from noncomputable functions I think we will have done something valuable .
peace
fifthmonarchyman,
That is a gross misrepresentation of my attempts to clarify what you would consider success. Given your demonstrated inability to provide clear rules for your game and your refusal to recognize when the claims of a paper you tout are clearly shown to be unsupported, it’s not at all unreasonable to get you to commit to specific details before spending time refuting you.
Please answer the questions I’ve posed in this comment and this comment rather than making cheap, unsubstantiated slurs.
I have said it before: you could run the game a second time with your model as the input, and I predict that an observer will now be able to see the difference between that and a close model of itself. Ergo, your original model will now be flagged as an ‘original’ and you will have to conclude that it is non-computable. Oops – you just computed it in the first round of the game!
See above, I think your process of demonstrating this is flawed.
Have you tried to run the GA until R-squared is 0.95? 0.99? 0.99999? Can you still tell the difference? Is the model computed, or not? Does it target the actual numbers themselves?
There are several If’s here, and you are still a long way away from demonstrating any of them.
fG
What is noncomputable is not the number sequence it’s self but the act of determining a sequence. As I’ve said lots of times and finite sequence can be computed. What we are talking about is not the actual act of producing a string but the act of deciding to do so.
The difference seems obvious to me I’m not sure why you are having difficulty with it.
look at Jenny’s number again 8675309
Randomly mutate one number and you might have Mary’s number 8674309. It’s only one number different but it codes to a completely different address.
Both Mary’s and Jenny’s number are noncomputable in the sense that there is no general algroythym that will produce them.
Jenny’s number is familiar to us it but to Marry and her friends and family 8674309 is just as unique and special.
peace
This is something I need to address moving forward, I have run it to a very high R squared, What we generally see is that the higher I go the more limited in scope but individually obvious the error becomes
for example we might see 8-6-7-5-3-0-9 and the model
8-6-7-5-2.795468794-0-9
Anyway
In the original paper close was defined as one bit of difference but the models were digital. The game is more analog, I would think that a single difference in one number would be enough to distinguish. You are right that more thought needs to be given as to what is close
thanks for the interaction
Sure there is. In pseudocode:
i = 0;
while (1) {
print i;
i = i +1;
}
That will produce
0
1
2
.
.
.
8674307
8674308
8674309 [Mary’s number]
8674310
.
.
.
8675307
8675308
8675309 [Jenny’s number]
8675310
.
.
.
It’s a general “algroythym” and it produces both numbers.
Jesus, fifth.
fifth,
You’ve been stumbling over your own shoelaces for more than six months. This is from June of 2015:
Which is why I chose real genomic data. I’m still waiting for FMM to put real data and scrambled data into two piles.
The difficulty I have is understanding how your game has anything to do with this. All you do is compare two strings and see if you can tell if they are different. What has that got to do with ‘the act of deciding to produce a string’?
No, I won’t look at Jenny’s number again for reasons I already explained to you. Jenny’s number has nothing to do with your game because it doesn’t contain a pattern that can be recognised by an unbiased observer.
Recognising a string you have seen before (semantics) is something totally different than recognising differences between two strings merely on the basis of their internal patterns (syntax). Your game does the latter and should work regardless whether you have seen the string before or not, in other words any unbiased observer will arrive at the same conclusion. Recognising Jenny’s number is impossible unless you have seen it before and know that it represents something (semantics), and will only be possible by biased observers. Very different things indeed.
You keep confusing these things. I’m not sure why you are having difficulty with it.
fG
In any case, you haven’t responded to this particular criticism:
How come this does not totally scupper your game, which supposedly lets you determine if a string is ‘designed’, i.e. non-computable?
fG
Ive done quite a bit of thinking about how we can evaluate the success of your little hack.
What we are looking to determine is if your software performs the same as human beings in the generalized task at hand. I am deeply concerned that the direction you are taking will not lead to a valid determination of this question
Here is my suggestion.
1) A neutral human tester arbitrarily chooses 5 strings to evaluate from any object or process he pleases
2) the tester prepares the 4 assays according to the rules of the game
a) a complexity test
b) randomized test
c) close model test
d) Copy (with human post processing to mimic the pattern in the original ) test
3) The tester submits the assays to your software and 5 human observers
4) The tester then compares the results from each “observer” for each assay
5) if your software’s results can not be distinguished from those of the human observers then it passes the Turing test and my enterprise is falsified
How does that sound?
Please feel free to ask for any clarification you need.
peace
Do you really not see the problem here? Your algroythym needs to know when to halt.
I’m looking for a particular finite number at a particular time not a eternally infinite evolving stream of numbers,
An infinite amorphous cloud of numbers will not connect you to Jenny or Marry
peace
Imagine yourself trying to mimic the pattern in the original string for instance when you do the post processing tweaking we have talked about.
You consciously decide what it takes to actualize the pattern that you have learned. you add or subtract numbers in particular places in the sequence until your constructed string matches the one in your head.
I believe that that is what is happening in the design process.
You start with a mental pattern and then attempt to actualize it physically.
It’s that design process that we recognize when we see pattern expressed in a nonrandom noncomputable string.
Does that make sense?
peace
fifth,
No. According to you, it simply needs to be a “general algroythym that will produce them [Jenny and Mary’s numbers]”.
Lay off the goalpost shifting, fifth. We’re not idiots. We can see it a mile away.
Own up to your mistakes and then see if there is anything you can salvage.