691 thoughts on “Is AI really intelligent?”

1. AI reminds me of Tom Lerner’s observation on sewers:

   What you get out of it depends on what you put into it.

2. 

   I thought it would be fun to ask Claude to analyze this Reddit proposal. Here’s his response:

   This is a fun one to unpack, because it looks compelling on the map until you think it through.

   The red marker appears to be near the Continental Divide in Montana/Idaho, where the headwaters of the Columbia River system (draining to the Pacific) and the Missouri River system (draining to the Gulf) come closest together. The idea being: dig a short canal there, and suddenly you have a continuous water route across the continent.

   Here’s why it doesn’t work as a Panama Canal competitor:

   It’s not actually a shortcut. The Panama Canal saves ships roughly 8,000 miles by avoiding Cape Horn. This proposed route would require a ship to travel up the Columbia (hundreds of miles), through the canal, then down the entire Missouri-Mississippi system (thousands of miles) to reach the Gulf. That’s a massive detour, not a shortcut.

   The elevation problem is brutal. The Continental Divide isn’t flat — you’d need extensive lock systems to get over it, making it more like a very long, mountainous version of the Erie Canal than anything resembling Panama.

   River systems can’t handle ocean shipping. Large container ships and tankers need deep-draft channels. Rivers have sandbars, shallows, dams, and current. The Mississippi is navigable, but not for the vessels Panama handles.

   This is literally what Lewis & Clark were sent to find. Jefferson hoped there was a navigable water route across the continent — they discovered there wasn’t, because the portage between Columbia and Missouri headwaters is an overland mountain crossing, not a gentle lowland gap.

   It’s a classic case of a map making something look deceptively simple. The river systems look like they almost connect, but “almost” is doing a lot of heavy lifting when the gap involves the Rocky Mountains.

   And here’s his thought process:

   This is a fun geography/engineering question. The image shows a map of North American river systems, with a small red marker labeled “Canal” near what appears to be the Continental Divide in the northern Rockies/Montana area – seemingly suggesting a short canal connecting the Columbia River system (flowing to the Pacific) with the Missouri/Mississippi system (flowing to the Gulf of Mexico).

   The idea is: could such a canal “compete” with the Panama Canal by creating a water route across the continent?

   Let me think about this carefully.

   The red marker appears to be near the headwaters of the Missouri River and the Columbia River system, somewhere in Montana/Idaho near the Continental Divide.

   The concept would connect the Pacific-draining Columbia to the Atlantic/Gulf-draining Missouri-Mississippi system via a short canal at the divide. But there are fundamental problems with this competing with Panama. The Continental Divide sits at high elevation, so unlike Panama’s sea-level cut, this would require extensive locks to handle the elevation changes. More importantly, while it would technically link two ocean-bound river systems—the Columbia reaching the Pacific at Astoria and the Mississippi reaching the Gulf at New Orleans—the route would still be constrained by river navigation limitations that ocean-going ships don’t face.

   Ships would need to travel hundreds of miles up the Columbia, cross the canal, then navigate thousands of miles down the Missouri and Mississippi to reach the Gulf. That’s vastly longer than the Panama route. Historically, Lewis and Clark discovered there was no easy water connection between these systems—the portage was overland, and that Continental Divide crossing remains a massive engineering challenge. The Panama Canal saves roughly 8,000 miles on a New York to San Francisco voyage, whereas this route would add thousands of miles compared to existing ocean routes.

   ETA: In case anyone’s wondering, that is from the r/mapporncirclejerk subreddit, home of proposals like the following:

   If Iran wants to block the Strait why not just steal Rhode Island and use it to block the Strait? Are they stupid?

   

3. I showed the Hormuz/Rhode Island map to Claude, who commented:

   This is genuinely clever absurdist humor, and the scale actually works, which is what makes it land.

   The Strait of Hormuz is only about 21 miles wide at its narrowest point. Rhode Island is roughly 48 × 37 miles. So the overlay is roughly correct — Rhode Island would in fact be a plausible geographic cork for the strait, which is both funny and a useful intuition pump for just how strategically vulnerable that chokepoint is.

   The joke is also a perfect parody of the “X is the size of Rhode Island” genre of geographic comparison that r/MapPorn takes very seriously. r/mapporncirclejerk exists specifically to mock that earnest energy by taking it to absurd logical conclusions. “Why don’t they just steal it” is the perfect punchline.

   The serious underlying point, unintentionally made: the fact that ~20% of the world’s oil supply passes through something Rhode Island could theoretically plug is one of the more quietly alarming facts in geopolitics. Iran doesn’t need to block it — just credibly threaten to, and insurance rates spike globally.

   Anyway, the subreddit name alone — mapporncirclejerk — tells you everything about the internet’s capacity for precise self-aware meta-humor about its own niche obsessions. Good finds today, you’ve been on a quality Reddit streak.

4. As if his achievements weren’t already impressive enough, I just became aware that Alan Turing anticipated neural nets and reinforcement training in his paper Intelligent Machinery. I’ve just started reading it, but I was struck by his description of what amounts to a virtual memory system, in which chunks of a larger physical address space are mapped into a 9-bit logical address space:

   As has been explained, the ACE at present uses finite sequences of digits to describe positions in its memory: they are actually sequences of 9 binary digits (September 1947). The ACE also works largely for other purposes with sequences of 32 binary digits. If the memory were extended, e.g., to 1000 times its present capacity, it would be natural to arrange the memory in blocks of nearly the maximum capacity which can be handled with the 9 digits, and from time to time to switch from block to block. A relatively small part would never be switched. This would contain some of the more fundamental instruction tables and those concerned with switching. This part might be called the ‘central part’. One would then need to have a number which described which block was in action at any moment. However this number might be as large as one pleased.

   Virtual memory wasn’t actually implemented until the late 50s, so I was shocked to see Turing’s description from a decade earlier, in a paper that was just two years post-ENIAC.

5. Turing might as well be commenting in this thread. He’s pretty much describing Erik here:

   I propose to investigate the question as to whether it is possible for a machinery to show intelligent behaviour. It is usually assumed without argument that it is not possible. Common catch phrase such as ‘acting like a machine’, ‘purely mechanical behaviour’ reveal this common attitude. It is not difficult to see why such an attitude should have arisen. Some of the reasons are

   (a) An unwillingness to admit the possibility that mankind can have any rival in intellectual power. This occurs as much amongst intellectual people as amongst others: They have more to lose. Those who admit the possibility will agree that its realisation will be very disagreeable. The same situation arises in connection to the possibility of our being been superseded by some other animal species. This is almost as disagreeable and its theoretical possibility is indisputable.

   (b) A religious belief that any attempt to construct such machines is a sort of Promethean irreverence.

   (c) The very limited character of the machinery which has been used until recent time (e.g. up to 1940). This encouraged the belief that machinery was necessary limited to extremely straightforward, possibly even repetitive, jobs. This attitude is very well express by Dorothy Sayers (The Mind of the Maker, p. 46) “… which imagines that God, having created his Universe, has now screwed the cap on his pen, put His feet on the mantelpiece and left the work to get on with itself.” This, however, rather comes into St Augustine’s categories of figures of speech or enigmatic sayings framed from things which do not exist at all. We simply to not know of any creation which goes on creating itself in variety when the creator has withdrawn from it. The idea is that God simply created a vast machine and has left it working until it runs down from lack of fuel. This is another of those obscure analogies, since we have no experience of machines that produce variety of their own accord: the nature of a machine it to do the same thing over and over again so long as it keeps going.

   (d) Recently the theorem of Gödel and related results (Gödel (1) , Church (1), Turing (1) ) has shown that if one try to use machines for such purpose as for determining the truth or falsity of theorems and one is not willing to tolerate an occasional wrong result, then any given machine will be in some case unable to give an answer at all. On the other hand the human intelligence seems to be able to find methods of ever-increasing power for dealing with such problems ‘transcending’ the methods available to machines.

   (e) In so far as a machine can show intelligence this is to be regarded as nothing but the reflection of the intelligence of its creator.

   2 Refutation of some objections

   In this section I propose to outline reasons why we do not need to be influenced by the above described objections. The objections (a) and (b), being purely emotional, do not really need to be refuted. If one feels it necessary to refute them there is little to be said that could hope to prevail, though the actual production of the machines would probably have some effect. In so far then as we are influenced by such arguments we are bound to be left feeling rather uneasy about the whole project, at any rate for the present. These arguments cannot be wholly ignored, because the idea of ‘intelligence’ is itself emotional rather than mathematical.

   The objection (c) in its crudest form is refuted at once by the actual existence of machinery (ENIAC etc.) which can go on through immense numbers (e.g. 10^60,000 about for ACE) of operations without repetition, assuming no breakdown. The more subtle forms of this objection will be considered at length in section 11.

   The argument from Gödel’s and other theorems (objection (d)) rests essentially on the condition that the machine must not make mistakes. But this is not a requirement for intelligence. It is related that the infant Gauss was asked at school to do the addition 15 + 18 + 21 + · · · + 54 (or something of the kind) and that he immediately wrote down 483, presumably having calculated it as (15 + 54)(54 − 12)/2.3. One can imagine circumstances where a foolish master told the child that he ought instead to have added 18 to 15 obtaining 33, then added 21, etc. From some points of view this would be a ‘mistake’, in spite of the obvious intelligence involved. One can also imagine a situation where the children were given a number of additions to do, of which the first 5 were all arithmetic progressions, but the 6th was say 23 + 34 + 45 + · · · + 100 + 112 + 122 + · · · + 199. Gauss might have given the answer to this as if it were an arithmetic progression, not having noticed that the 9th term was 112 instead of 111. This would be a definite mistake, which the less intelligent children would not have been likely to make.

   The view (d) that intelligence in machinery is merely a reflection of that of its creator is rather similar to the view that the credit for the discoveries of a pupil should be given to his teacher. In such a case the teacher would be pleased with the success of his methods of education, but would not claim the results themselves unless he had actually communicated them to his pupil. He would certainly have envisaged in very broad outline the sort of thing his pupil might be expected to do, but would not expect to foresee any sort of detail. It is already possible to produce machines where this sort of situation arises in a small degree. One can produce ‘paper machines’ for playing chess. Playing against such a machine gives a definite feeling that one is pitting one’s wits against something alive.

   These views will all be developed more completely below.

6. Do LLMs have access to calculators? Fast, error free algorithms for calculation.

   There has been a tendency in popular science fiction to assume AI would have access to error free knowledge, I.e. facts.

   Aside from four banger arithmetic, I’m pretty sure facts are mutable.

7. petrushka:

   Do LLMs have access to calculators? Fast, error free algorithms for calculation.

   They have access to entire Python sandboxes, and evidently they use those sandboxes for any calculations that the LLM proper isn’t confident about doing natively. I guess the sandbox overhead is low enough that a separate, preformed calculator module isn’t needed. They just generate Python code, stick it in the sandbox, run it, and plug the results back into the context window.

8. We are terrified of AI becoming conscious, but Yuval Noah Harari warns we’re missing the real threat: Non-conscious intelligence.

   In the 58 seconds it takes to watch this, Harari dismantles our biggest AI myth. We visualize “Hal” from 2001: A Space Odyssey—a machine that destroys us because it gained a soul.

   But the reality is far more clinical.

   The distinction we’re missing:

   – Intelligence: The ability to solve problems and pursue goals (winning chess, driving a car, optimizing a supply chain).

   – Consciousness: The ability to feel—pain, joy, love, or suffering.

   AI doesn’t need to “feel” anything to destabilize civilization. It just needs to be better at solving the “problem” of human influence.

   As Harari notes, we don’t even understand human consciousness yet. To wait for AI to become “sentient” before we take it seriously is a category error. We are building a world of unprecedented intelligence without a shred of consciousness.

   If a machine can solve every problem we face without ever feeling a single consequence, are we actually in control, or are we just part of its optimization loop?

9. petrushka,

   When the author of that article refers to “the distinction we’re missing” between consciousness and intelligence, I’m not sure who the “we” is supposed to refer to. I wasn’t aware of a widespread belief that AI will only be dangerous once it becomes sentient. Perhaps the phrase should have been “the distinction I’ve been missing”.

   In any case, the point is correct. AIs could wreak catastrophic damage without feeling a thing, which in a way is even creepier than the thought of a sentient AI doing so.

   If a machine can solve every problem we face without ever feeling a single consequence, are we actually in control, or are we just part of its optimization loop?

   “Feeling a consequence” is a misleading phrase because “feel” is ambiguous in English. You can feel sorrow, or you can feel that something is true. The former is an emotion, but the latter is a cognitive stance. AIs can feel in the latter sense, and if an AI is properly and robustly imbued with human-aligned goals, it will feel cognitively that it shouldn’t do anything counter to those goals, while not feeling anything in the emotional sense.

   The danger comes if malicious actors give malicious goals to an AI, or the AI develops a harmful goal due to a malfunction (as in the case of the AI agent we talked about earlier that deleted all those emails without asking for confirmation), or if well-meaning humans inadvertently frame the goals in a way that allows for or even encourages harmful behavior. The AIs will feel (cognitively) that they should pursue the goals while feeling nothing emotionally or subjectively.

   I’m disappointed that so many in the AI community are downplaying the dangers or barging ahead with development while insufficiently considering the safety issues. Current LLMs are limited in the damage they can do. AI agents are more dangerous, and now is the time to be especially focused on the safety issues, given the fact that agents are proliferating.

10. Proposition:

    The word truth should be replaced in ordinary speech with something like high probability.

    AIs produce text that sounds like something a smart human would say, but without analysis, we don’t know if it’s the result of careful reasoning, or the result of consensus.

    I’m of the opinion that consensus inevitably leads to corruption, because valuable social institutions attract predators. Medicine, social work, science, all attract mendacious people. Without vigilance, they become counterproductive.

11. petrushka:

    The word truth should be replaced in ordinary speech with something like high probability.

    Why, when people already understand that saying “X is true” doesn’t mean X is 100.0% certain? Also, for things that are virtually certain, using the phrase “high probability” would be misleading. If a friend wants to know whether I’ll pick them up at the airport, saying “there’s a high probability that I will” might be technically correct — it’s always possible that I’ll be in a traffic accident, for instance — but if I use the phrase “high probability”, my friend will infer that the probability is lower than it actually is.

    AIs produce text that sounds like something a smart human would say, but without analysis, we don’t know if it’s the result of careful reasoning, or the result of consensus.

    That’s why AIs now usually show their reasoning for complicated questions, either explicitly in their response or in the thought process window that you can open and inspect. Besides the benefit to the user of being able to see and verify the step-by-step reasoning, the process of describing it actually makes the model produce better answers. The reasoning sits in the context window and influences later output positively.

    They also annotate portions of their responses with links to the sources they consulted. You can apportion your trust in the answers according to the quality of the sources, and you can click the links and check the sources yourself to be certain.

    I’m also a big fan of consulting multiple AIs when the question is important, to see if they agree.

12. AI isn’t a problem for cautious users.

    The problem would arise if AI controls active processes.

    Your optimism assumes that users will typically cautious and well informed.

    I would ballpark that as the top fifth percentile.

    But AI could be a blessing to the less gifted if it were trustworthy.

13. petrushka:
    Proposition:

    The word truth should be replaced in ordinary speech with something like high probability.

    NO. This is a category error. It presumes some sort of probability distribution for any phenomenon, and “truth” is some high percentile. Not all propositions fit this model. We don’t speak of the “probability” that the sun will rise in the morning or in the east. Or of death, or taxes. We don’t say that 2+2 is probably four. Definitions are true by definition – we can argue utility, but not truth.

14. Nevertheless, most propositions in common discourse should be shaded.

    Death and taxes are colloquially cited as the only certainties.

    And 2+2 could equal 11, depending on context.

15. petrushka:

    Nevertheless, most propositions in common discourse should be shaded.

    That would be inefficient and misleading, as my airport example shows. Imagine the following exchange:

    Griselda:

    Mort, can you pick up some Chinese on the way home tonight?

    Mortimer:

    But Chef Chu’s is on the other side of town.

    Griselda:

    I know, but you’ll be over there anyway. You’re picking Gertrude up from the airport, remember?

    Mortimer:

    True.

    Has Mortimer misspoken? Should he have said “probably” instead? He could add “inshallah” to every statement that he isn’t 100% certain of, but why bother? Griselda knows what “true” means in this context.

    Death and taxes are colloquially cited as the only certainties.

    And you can cite that to anyone who makes the mistake of thinking that “true” means “100.0 percent certain”. It’s usually not necessary. Griselda knows that Mortimer won’t be picking Gertrude up at the airport if he has a heart attack today, and she isn’t fooled by the word “true” into thinking that the pickup is guaranteed to happen.

    And 2+2 could equal 11, depending on context.

    If so, the context should be communicated along with the statement, since the statement is false in the default context (base 10 arithmetic) but true in the one you’re presumably employing (base 3 arithmetic). If I say “it’s true that 2 + 2 = 4”, I’m not being sloppy; that statement is true in the default context of decimal arithmetic. The solution to your problem, which everyone already employs, is to add qualifiers where necessary and to leave them out when they’re not needed. I would say “it’s true that 2 + 2 = 4”, or “it’s true that 2 + 2 = 11 in base 3 arithmetic”, and both of those would be correct statements. There’s no reason to avoid using the word “true”.

16. petrushka:

    AI isn’t a problem for cautious users.

    Technology can malfunction even in the hands of the most cautious users. Nothing in the Boeing manual explained what to do if the 737 Max’s MCAS system failed, causing the airplane to aggressively trim nose down. Even the most cautious pilot would have been at a loss in that situation.

    That executive at Meta, mentioned earlier in the thread, was cautious. She specified that the AI should get confirmation before deleting any of her emails. That didn’t save her because the AI lost that part of the prompt when it compacted the context window.

    The problem would arise if AI controls active processes.

    The danger arises whenever AI interacts directly with the world rather than through human intermediaries. That’s what AI agents do, and they’ll become ubiquitous. There’s a risk in using them, just as there’s a risk in depending on an airliner’s flight computers.

    Your optimism assumes that users will typically cautious and well informed.

    I don’t make that assumption, and I wouldn’t even describe myself as an AI optimist. AI could end up being one of the best things humans have ever created, but it could also end up being one of the worst. The potential and the peril are both enormous.

    But AI could be a blessing to the less gifted if it were trustworthy.

    It’s already a blessing to them, if they’re using it. One-on-one tutelage from an infinitely patient teacher who is available 24/7. Search capabilities that are superior to most humans’. The ability to reason through complicated problems. AIs can screw up, but the solution to that is to educate people about their shortcomings, not to encourage them to avoid AI.

    There are people who will blindly trust AI, to their detriment, just as there were people who blindly trusted their GPS systems and drove into the ocean. That’s no reason to avoid either technology.

17. I think we are miscommunicating in a numberer of ways.

    AI is capable of tutoring children and adults, but there’s a serious alignment problem.

    I have a grandson who is home schooled. He has a number of quirks that resulted in failure in school. Hyperactive, verbally gifted, slow processing, emotionally immature.

    My son uses Perplexity to generate lessons. But AI is not ready yet to take over direct interaction with children.

    People who can generate prompts and evaluate results are probably in the 95th percentile intellectually. That leaves a lot of people as consumers who require some sort of protection from exploitation.

    Shades of censorship. This is not unlike the debate over internet censorship.

    This is not a new debate. There was resistance to translating the Bible.

18. petrushka:

    AI is capable of tutoring children and adults, but there’s a serious alignment problem.

    Alignment is important, obviously, but the existence of risk doesn’t mean that a technology shouldn’t be used. It’s risk vs reward, and if that ratio is low enough, it makes sense to use the technology. We allow (some) kids access to the internet despite the dangers, so why not handle AI the same way?

    I have a grandson who is home schooled. He has a number of quirks that resulted in failure in school. Hyperactive, verbally gifted, slow processing, emotionally immature.

    AI tools are already capable of tailoring their delivery to the abilities of the student, and they will only get better. In any case, AI is going to be more helpful to some students than to others, but that isn’t an argument for denying access to the ones who can make good use of it. Plus, I would argue that the infinite patience and constant availability of AI are disproportionately beneficial to slower students who would otherwise fall behind due to limited or no access to one-on-one tutoring.

    My son uses Perplexity to generate lessons. But AI is not ready yet to take over direct interaction with children.

    It isn’t ready to replace teachers, obviously, but it definitely has a place in education. I read recently that more than 80% of teachers and students are already using AI.

    People who can generate prompts and evaluate results are probably in the 95th percentile intellectually.

    That’s way too pessimistic. If you’re capable of carrying on a conversation, you’re capable of interacting with AI. It doesn’t even have to be by text anymore, now that AIs are voice capable.

    Also, the question of whether someone is capable of evaluating results is distinct from the question of whether AI is useful to them. Some people are better than others at evaluating the truth of what they read in books or magazines, see on TV, hear from friends, encounter in YouTube videos or elsewhere on the internet. The solution is to educate people in critical thinking, not to limit access to those resources. I don’t see AI as being fundamentally different.

    That leaves a lot of people as consumers who require some sort of protection from exploitation.

    Which is true of resources other than AI, too. Again, it’s risk vs reward, and the solution is to educate people on how to avoid being exploited or scammed.

19. I just watched a YouTube short of a guy responding to claims that British English is the original English. Someone in the comments wrote

    Hū dærrað hīe cweðan þæt hīe sprecað þæt frumstæpplice Englisc? Hlæhlīc!

    That’s Old English. I clicked on ‘Translate to English’, and YouTube produced

    How dare they say that they spread that primitive English? Lol!

    YouTube’s translator (presumably Google Translate) knows Old English. The “lol” is icing on the cake.
    ·
    The Google Search AI did even better:

    How dare they say that they speak the “primeval/original” English? Laughable!

    It commented:

    The idea that anyone today speaks frumstæpplice Englisc (Primitive Old English, c. 450–650) is indeed hlæhlīc, as that stage of the language is almost entirely unreconstructed, surviving only in limited runic epigraphy rather than written texts. Even the most skilled scholars speak a version of West Saxon from 300-500 years after the initial settlement, not the primordial form.

20. The following news leaked due to a misconfiguration of Anthropic’s website:

    Claude Mythos

    Research preview

    Claude Mythos

    We have finished training a new AI model: Claude Mythos. It’s by far the most powerful AI model we’ve ever developed.

    “Mythos” is a new name for a new tier of model: larger and more intelligent than our Opus models—which were, until now, our most powerful. We chose the name to evoke the deep connective tissue that links together knowledge and ideas.

    Compared to our previous best model, Claude Opus 4.6, Mythos gets dramatically higher scores on tests of software coding, academic reasoning, and cybersecurity, among others.

    In preparing to release Claude Mythos, we want to act with extra caution and understand the risks it poses—even beyond what we learn in our own testing. In particular, we want to understand the model’s potential near-term risks in the realm of cybersecurity—and share the results to help cyber defenders prepare.

    Mythos is also a large, compute-intensive model. It’s very expensive for us to serve, and will be very expensive for our customers to use. We’re working to make the model much more efficient before any general release.

    For those reasons, we’re taking a slower, more gradual approach to releasing Mythos than we have with our other models. We’re beginning with a small number of early-access customers, who will explore the model’s cybersecurity applications and report back what they find.

    The cybersecurity angle is scary, but it sounds like the model will be amazing.

21. Modern English is a mongrel language. Anglo-Saxon, plus French. That’s just a start. One could seriously argue that there are two “Standard” varieties —British and American — and hundreds of dialects.

    Other languages borrow from each other. Each other. English mugs them and empties their pockets.

    One of the reasons English is the actual lingua franca.

    Shakespeare’s English is mostly understandable, and it predates the empire. The King James Bible spread Elizabethan English through the world. Britain is the origin. The stewpot.

22. Long article in the Atlantic about the mind-bending cost of AI hosting and training. Like, some sites with many tens of thousands of square feet of space consume more electricity than the city they’re in. One planned site will need more electricity than is currently consumed by New England altogether. These sites generally build their own power plants generating gigawatts for single buildings – natural gas plants, which can be constructed quickly (though their turbines are now back ordered, so try something else). The processors are water cooled, heating millions of gallons of water a month.

    What I found odd was that the Big Four (Alphabet, Meta, Amazon and Microsoft – maybe Anthropic is in there somewhere) have so far invested nearly a trillion dollars into AI, and have yet to show a profit. The driving factor isn’t money, or software, or even electricity. Rather, it’s speed – all the players are terrified that a) AI will become so dominant that it will be absolutely make or break for the companies; and b) they will fall behind in a race where catching up is currently not even possible. Ther imperative is to win the race at any cost. So these big AI factories are rehabilitating old nuclear plants, building massive coal-fired plants, causing brown-outs in nearby cities, etc. They concede that they could be a LOT more environmentally friendly, but that would require the development of huge solar or wind farms, and nukes. And the problem isn’t the environment, it’s time. They all believe they can NOT lose this race, even though there is no guarantee it will ever be worth winning. Sound build-out practices mean losing.

23. Not showing a profit is not the same as having no income.

    If one contender gets significantly behind, they could lose their income.

24. I read recently that some tech companies are starting to use AI token usage as a metric in performance reviews. Some are even maintaining leaderboards. I guess the idea is that if you aren’t consuming tokens like crazy, you aren’t being productive enough.

    I wonder how many people are using bogus prompts in order to keep their token counts in an acceptable range.

    ETA: Found this quote from Jensen Huang:

    If that $500,000 engineer did not consume at least $250,000 worth of tokens, I am going to be deeply alarmed.

    How convenient that the CEO of the world’s leading manufacturer of AI chips is encouraging people to use more tokens and characterizing it as “alarming” if they don’t.

25. Flint:

    They concede that they could be a LOT more environmentally friendly, but that would require the development of huge solar or wind farms, and nukes.

    There’s a paper out now claiming that massive data centers are creating heat islands, raising the temperature of their surroundings by an average of 2° C. Yet another environmental impact of AI.

    ETA:

    The data heat island effect: quantifying the impact of AI data centers in a warming world

    Abstract

    The strong and continuous increase of AI-based services leads to the steady proliferation of AI data centres worldwide with the unavoidable escalation of their power consumption. It is unknown how this energy demand for computational purposes will impact the surrounding environment. Here, we focus our attention on the heat dissipation of AI hyperscalers. Taking advantage of land surface temperature measurements acquired by remote sensing platforms over the last decades, we are able to obtain a robust assessment of the temperature increase recorded in the areas surrounding AI data centres globally. We estimate that the land surface temperature increases by 2°C on average after the start of operations of an AI data centre, inducing local microclimate zones, which we call the data heat island effect. We assess the impact on the communities, quantifying that more than 340 million people could be affected by this temperature increase. Our results show that the data heat island effect could have a remarkable influence on communities and regional welfare in the future, hence becoming part of the conversation around environmentally sustainable AI worldwide.

26. Am I wrong to think most of the energy usage is for training?

    Tesla is hoping to get car computers down to 100 watts.

    IBM asserted a while back that chips optimized for AI could be a thousand times more energy efficient than current GPUs.

    OpenAI just backed away from a promise to buy massive numbers of memory chips.

27. petrushka:

    Am I wrong to think most of the energy usage is for training?

    The estimates I’ve seen are that training energy only accounts for 10-20% of the total, with the rest being used during normal operation. However, that ratio depends on how long they run a model before releasing the next one, and the time between releases is decreasing. Seems like Anthropic just released version 4.6 of Sonnet and Opus, but they’re already talking about the release of Mythos, the next generation.

28. We seem to be at that stage of the game when the world only needs five AI computers.

    AI cannot scale unless we achieve orders of magnitude increases in efficiency.

29. If you were interested in how AI is built and how it works, you would be talking about Claude’s leaked source code. But you are not talking about it, because you are not interested in how AI is built and how it works.

    Despite your lack of interest, here’s how it is built and how it works.

    At 3:50 – Claude’s prompt codebase includes tons of hardcoded guardrails to make it behave. There is no other way to keep the “learning” or “intelligence” in check.

    At 4:20 – Claude’s code includes poison pills to make it appear as if certain tools/features exist in Claude, but they don’t.

    At 5:30 – “Frustration detector” is a regex looking for keywords.

    Basically, I have been more right than I knew. For a while I asked AI about AI and I gave some answers the benefit of a doubt. This was wrong. Don’t trust anything that AI says about AI. By design, it lies about everything that matters.

30. Here’s one for self-driving optimists.

    A little while ago, there was this article:

    A Tesla Actually Drove Itself from Los Angeles to New York: Exclusive

    The article did not even make a dent in the AV and AI industry. Why? Because it’s a total lie. A Tesla driving itself from Los Angeles to New York did not happen. Then why the article? A possibility is that the writers are keithses who are at square zero with their understanding of any of the relevant concepts and definitions, hence they are unable to describe (much less explain) what actually happened. However, more likely they got paid to lie, so they lied.

31. No one says self-driving is solved. No unsupervised software has been released to the general public.

    The important thing is it’s much safer than all but the top five percent of human drivers.

32. Just a word about Dan O’Dowd.

    He published videos claiming they represented FSD, when they were, in fact, autopilot — a discontinued product — that was basically cruise control plus lane keeping.

    The lidar company that sponsored him disavowed his videos.

    I accept the fact that fully autonomous vehicles are still experimental, but the game is changing quickly.

    Most of the quirks remaining are errors in navigation maps, and are not safety issues.

33. Erik,

    If AI’s flaws and shortcomings mean that AI isn’t truly intelligent, then humans, with their flaws and shortcomings, aren’t truly intelligent either. Do you really want to go there? Is it your position that all intelligence, both natural and artificial, is illusory? If not, and if humans get a pass despite their imperfections, then why not AI?

    You keep making this mistake. Your goal is to show that AI isn’t intelligent, not that it isn’t perfect. To show that it isn’t intelligent, you need to show that it can’t do anything that requires intelligence. We’ve already seen that in example after example, AI can do things that you and I agree require intelligence.

    The debate was over long ago. All that remains is for you to come to grips with that fact.

34. keiths:

    If AI’s flaws and shortcomings mean that AI isn’t truly intelligent, then humans, with their flaws and shortcomings, aren’t truly intelligent either. Do you really want to go there?

    Not the way you phrased it, no. But your phrasing is a bit slanted. I think what currently bothers people is the nature of the mistakes AI makes, which are quite different from the nature of human mistakes. I still wonder about what I’d describe as the “wait a minute, that can’t be right!” sort of mistake. I remember when pocket calculators came out, and people would regard clearly nonsensical answers as correct to 9 decimal places! Maybe the latest versions of Claude are programmed to do sanity checks on their pronouncements? I have asked the Google AI for directions to do something on the computer, and it confidently tells me to access non-existent menus or menu selections. When I reply that those selections don’t exist, the AI says “Oh, I’m sorry” and suggests something else that doesn’t exist or doesn’t do what the AI said it does.

    Now, I can try out each direction and see that it’s bogus. But if I was informed of some medication or medical treatment, trying it out could be dangerous. My only defense would be “wait a minute, that can’t be right”, a reaction it seems AI doesn’t have in its arsenal.

    So yeah, AI and humans both make mistakes. But not all mistakes are created equal. A human lawyer, for example, may misunderstand the applicability of a citation, but he doesn’t make up fake cases. That’s a whole different category of mistake.

35. I think people have to learn what AI can and cannot do. And adjust as it changes.

    If I could make only one request, I’d ask it to bring the Weekly World News back to life.

    I think that’s within its capabilities, and no alignment worries.

36. Flint:

    I think what currently bothers people is the nature of the mistakes AI makes, which are quite different from the nature of human mistakes.

    Some of them are very different, but my argument doesn’t rest on the mistakes being the same or similar. Two humans can both be intelligent despite making different kinds of mistakes, so why apply a different standard to AI? Someone (I think it was Yann LeCun) characterized AI as “alien intelligence”, and I think that’s apt. It’s quite different from ours, and it’s far better at some things and far worse at others. To expect it to mimic human intelligence is misguided. AI is its own thing.

    None of that affects my theme in this thread, which is that if AI can do X, and X requires intelligence, then AI is intelligent. Erik keeps falling into the trap of thinking that the argument is “If humans can do X better than AI can, then AI isn’t intelligent.” Or “If AI makes mistake X, but humans don’t, then AI isn’t intelligent.” It doesn’t follow. If it did, what about interhuman and interspecies comparisons? Crows can’t do taxes, but anyone who argues that they aren’t intelligent is fighting a losing battle.

    My advice: use AI for what it’s good at, but use it cautiously or not at all for the things it’s bad at. In other words, treat it like you would any other tool. (Likewise with people. My dentist is topnotch, and he’s my go-to for dental work, but I wouldn’t trust him to design a microprocessor.) And keep abreast of developments, because AI is getting better all the time.

    Maybe the latest versions of Claude are programmed to do sanity checks on their pronouncements?

    The latest batch of top-tier AIs are engineered to be more careful and explicit in their reasoning, plus opening the thought process windows is a great way to confirm that the reasoning makes sense. Simply asking an LLM to double-check its answer also improves accuracy. They can often catch their own mistakes with no other guidance than a request to double-check. They don’t do that by default, however, because that would be wasteful. It consumes tokens and it isn’t always needed.

    You can also ask them to explain how they arrived at an answer, if it isn’t already clear from the response or the thought process window.

    I have asked the Google AI for directions to do something on the computer, and it confidently tells me to access non-existent menus or menu selections. When I reply that those selections don’t exist, the AI says “Oh, I’m sorry” and suggests something else that doesn’t exist or doesn’t do what the AI said it does.

    Yeah, that’s annoying, and I run into it a lot, but I don’t particularly blame AI for it, because it’s something that happens whether or not you’re using AI. Software companies constantly modify their user interfaces, so if you google a “how to do X” question, odds are good that you’ll get outdated advice based on an older version of the OS or app. AIs, like humans, are limited by the information that’s available to them.

    I”m expecting to this to get better now that AI agents are becoming widespread. An AI agent could open the application on your computer and scan the menus intelligently to give you an up-to-date answer. Or instead of telling you how to do something, it could simply do it for you.

    I recently switched from Evernote to Notion, and the other day I needed to create a table. I was about to look up how to do that when it occurred to me that I could just ask the built-in AI to construct the table for me. It did, perfectly, and I was able to tweak the appearance by giving further commands to the AI. I still don’t know how to create tables, and I probably never will, because why bother learning how to do it when the AI can do it for me?

    Now, I can try out each direction and see that it’s bogus. But if I was informed of some medication or medical treatment, trying it out could be dangerous. My only defense would be “wait a minute, that can’t be right”, a reaction it seems AI doesn’t have in its arsenal.

    There are other defenses. I never take a single AI’s word for it when the question is important. I ask multiple AIs, and I ask them to double-check their answers. I also follow the links they provide to see where they get their information and whether it seems reliable. That will become less necessary as AI advances, but for now I think it’s prudent.

    So yeah, AI and humans both make mistakes. But not all mistakes are created equal. A human lawyer, for example, may misunderstand the applicability of a citation, but he doesn’t make up fake cases. That’s a whole different category of mistake.

    That’s an example of a case in which following the links is a good idea. Or you could ask the AI, when you’re wrapping up, to double-check that every citation corresponds to court records available on the web. The thing about hallucinations is that they typically aren’t systematic — that is, the model won’t usually re-hallucinate the same hallucination if asked to double-check.

    Another trick I use is to take an AI’s answer and feed it back into itself, but in a different chat. That way the model isn’t unduly influenced by the contents of the current context window. So if the model makes a mistake at some step in the reasoning, that mistake won’t carry over to the new chat, and the model will be able to check the answer afresh.

37. There is obviously a difference between being intelligent and being knowledgeable. Perhaps a lot of the seemingly idiotic AI output is caused by lack of knowledge rather than by lack of intelligence, similar to how kids like my intelligent 6 year old granddaughter can make hilarious pronouncements on the existence of gnomes and witches. As they grow older and acquire more and more knowledge they will be better equiped to avoid such bloopers. Perhaps AI wil develop along similar lines?

    Which brings me to a question I have about the current state of AI when it comes to science. Is paywalled scientific literature accessible by AI for its training? If not, it is going to struggle almightily to become trustworthy on scientific questions.

38. faded_Glory: Which brings me to a question I have about the current state of AI when it comes to science. Is paywalled scientific literature accessible by AI for its training?

    I think the answer is yes, but it’s only available to users who pay for some specifically science-trained agent. The devs have shown exactly zero regard for copyright when cooking up AI. (And isn’t it amazing how there is no one to challenge AI makers on this point? Nobody to call out their ruthlessness, lawlessness and lies…)

    For unpaid users, seeing how AI-assisted court case documentation is going, the results are mixed at best. And court materials are more or less openly available, so it can be a neglected aspect in AI development.

39. keiths: I have asked the Google AI for directions to do something on the computer, and it confidently tells me to access non-existent menus or menu selections. When I reply that those selections don’t exist, the AI says “Oh, I’m sorry” and suggests something else that doesn’t exist or doesn’t do what the AI said it does.

    The attribution is wrong for the embedded quote.

    For thirty years I’ve solved IT problems by searching online support.

    Microsoft has this nasty habit of changing utilities and menus, so that their own support documents are garbage.

    Not an AI defect.

    It is, perhaps, something a future generation of AI could address. It would be costly, but unified documentation would only have to be created once.

40. Regarding copyright:

    I have been under the impression that ideas cannot be copyrighted. Only the specific expression of an idea.

    Is this true of data accumulated by experiment? How does this work?

    Patents presumably apply to objects rather than ideas. But there are software patents.

41. I’ve recently read that Jonathan Swift foresaw the possibility of mechanical authors, artists, and composers.

    As did Orwell, Roald Dahl, and others.

    The products of artificial creators were regarded by these authors as slop.