Could you expand? What I got from her talk was that basically AI is a tool that can be used amorally very much like a hammer “reflects” the morals of its operator. I found this point pretty trivial , I’m still puzzled why the speaker
My vibes based analysis of AI is it will both not live up to the hype and also be weaponized to further entrench existing power structures. But doubling down on technocratic top down processes will further erode the very things that make a society functional. It’s peak Faustian, thinking we can harness technology to wield it as gods. Our flawed and incomplete maps of the territory will never achieve a recreation of the territory itself. Our small errors will be magnified exponentially with increased computational power.
What annoys me the most about Chloe's starry eyed propaganda is her omission of the technology's rapacious material requirements. Its emergence isn't just a competition against humanity for "intelligence" it is a competition for resources.
Sure. If the AI companies manage to pull off the nuclear powerplants and the AI still fails, they can just pivot to energy sector and upend another industry.
One problem is that consciousness is in fact integrally involved with human intelligence - intrinsic to it - despite the AI/Cog Sci notion that consciousness is unneeded - that they can discern "no role" for it. This ties into the second problem, namely, just saying, as per Chloe, that "AIs are not the same as us, but are inspired by our neural architecture" is unfortunately a very weak, very ineffective response to the ultimate question: "Are we not just machines as well?" Chloe's fallback has no in principle substance behind it, and the results of accepting that humans are equally machines, are, as I'm sure you intuit, horrific. I really suggest/hope, Mary, you'd get and study this book, even though a bit out of your normal field: "The Challenge to AI: Consciousness and Ecological General Intelligence." Having a grounded, clear, alternative model of the brain/mind (which is to say of perception, memory and cognition and how far LLMs are away from this) is becoming absolutely essential.
AI summary of the book: Robbins’s central claim is that humanlike general intelligence and consciousness cannot be captured by treating the mind as a digital computer that manipulates symbols or patterns in isolation. Instead, he argues that real intelligence is rooted in the biological, time-continuous, and embodied nature of the brain as it engages with a real environment. In this view, cognition is not just computation over representations but a dynamic process grounded in the organism’s physical structure and its ongoing coupling with the world.
He reframes “general intelligence” as fundamentally ecological. Rather than being the ability to solve arbitrary abstract problems, it is the capacity of an organism to perceive and act within a rich environment, picking up affordances and acting appropriately in real time. Drawing on ecological psychology (e.g., Gibson), he emphasizes that perception is direct and world-involving, not just the construction of internal models from sensory inputs, and that intelligence emerges from this continuous organism–environment loop.
Consciousness, for Robbins, is inseparable from lived time, continuous perception, and the construction of an experienced world, influenced by Bergson’s philosophy of time and mind. He argues that standard computational models, which discretize time into steps and treat perception as encodable data, miss this temporal, experiential dimension. This critique extends to language and action: generative AI systems, he contends, lack genuine understanding, affect, and intentional, voluntary agency because they operate as pattern predictors rather than embodied agents with feelings and goals.
From this, Robbins concludes that creating truly conscious, generally intelligent machines would require radically different architectures—ones that replicate biological-like dynamics, continuous time, affect, and deep ecological embedding—rather than scaling up current symbolic or neural network approaches.
Ha. A pretty good example of the actual weakness of AI. If we cut through some of its high, but sad abstractions, it plain missed the major concepts and the concrete model:
A better essence: The argument is that Bergson, in 1896, saw the universe as a holographic field (50 years before Gabor's invention, 80 before Bohm's generalization to the universe as a whole - hence why Bergson was never understood). But, as opposed to Pribram (the brain is a "hologram") and Bohm, he - in essence -saw the brain as creating a modulated reconstructive wave, passing through this holographic field, specific to (or specifying) a source within the field as an image, e.g., the cup and stirring spoon. Gibson, who saw the brain as "resonating" to the invariants defined over the (coffee stirring) event, and thus "specific to" the cup on the table, right "where it says it is," out there on the table (his "direct perception"), to account for the image, is in effect relying on his resonating neural mass being, acting, simultaneously as a reconstructive wave within a holographic environmental field.
The spoon stirring, a fly going by - all are in the past - and this brings in the importance of Bergson's temporal metaphysic (unmentioned by the AI, along this holographic model). The transformation of the universal field (and the brain embedded therein) is indivisible, not a series of discrete states (per the classic metaphysic), hence the motion/transformation of the field itself carries an attribute, we can say, of "primary memory," and the specification can indeed be to a past portion of the transformation. Due to the biochemical dynamics underlying the resonating neural mass, underlying the chemical velocities, the specification is at a scale of time - a "buzzing" fly, wings ablur at our normal scale, but given the injection of a catalyst (e.g. LSD) increasing the oscillation rate of the neural mass, perhaps now a heron-like fly barely flapping his wings (a large part of Chap 3. is devoted the biochemistry behind this time-scale specification - lost to the AI). This vision of changing the scale of time of the specified experience (imagine one's body at a scale where the fly is specified as a vague cloud of electrons) and given there is no true spatial demarcation or precise outlines between our body and the field) explicates Bergson's very Zen principle: "The relation of subject and object, and their union and differentiation, must be treated in terms of time, not space:"
All this comprises a solution to Chalmers' rather mis-stated hard problem (better stated - explaining the origin of the qualitative image of the external world), but note, perception - our experience of coffee stirring - is not occurring solely within the brain in this model, so the experience cannot be solely stored there - in fact we require a 4-D model of being, and an entirely different model of memory and cognition (which the book describes).
I could go on - but this should give you an idea of how vaguely abstract, somewhat right, but missing of the concrete alternative model that the AI actually is. Interesting.
I had to research and study that understand it. And I would say that I am generally aligned with all of this. Despite some of it being a bit over my head. Thanks for sharing.
What Jonathan's talk can’t quite answer from inside a room like ARC is the criterion for which practices actually do that building versus which merely feel formative. Learn poems, grow a garden, go to church — sure, but the version of each that passes and the version that fails are very different things. Memorizing a poem until it’s in the belly of the mind is different from reading it appreciatively once. A church organized around recurring embodied encounter with the same texts at the same moments does different things to a person than one organized around a weekly inspiring experience. The gap between the prescription and its fulfillment is a formation question, not a content question. And of course if it doesn't pass the Gargoyle's "Peasant Test" it is only useful for those with tickets on this post diluvian ARC... which would seem a more human version but still a version of a doomsday bunker.
To defend Jonathan - I think his point is praxis. It is that learning poems, growing a garden and going to church turns you into a certain kind of person in a way that intellectually understanding something doesn’t. It molds your soul.
Jonathan Pageau is an exceptional cultural analyst and I have been in his debt for almost a decade. Apart from a few technicalities, I have nothing negative to say about his talk for ARC.
It is a condensation of a recent talk for a Catholic school in Ottawa: https://youtu.be/ETaRAauDxS0 He says there that he personally uses AI as a research assistant—powerful but untrustworthy. That’s how I use it: Verify everything.
The question to ask, which he doesn’t quite address, is how AI influences one’s thinking and being and doing.
Crudely stated, it has changed me like every tool I have ever used has changed me:
The more I use it, the more proficient I become with it, the more ways I find to use it, the more dependent I become on it, and the more I frame my approach to every problem as an AI problem.
To paraphrase the old adage, if the most powerful tool you have is an AI hammer, then you tend to see every problem as a potential AI nail.
That’s been my experience, and when I’m most conscientious, that scares me shitless.
She made some conceptual leaps which she did not explain that well. First example: the word for "small" goes 'to the same place' in the LLM brain when the LLM is trying to come up with its opposite in different languages. But it's doing that because even if not explicitly, the original idea of machine learning and neural network was to get that kind of generalisation! Hell, you can create much simpler types of machine learning that can categorise in a similar way if trained appropriately but which no-one would accuse of being sentient. Then when the network has a "functional emotion" of "fear" again it has possibly 'just' formed an association from its data between 16g (why say 16000 mg ?) of paracetamol and the import (danger! Will Robinson) of a lethal dose which is being characterised as fear by a human. I mean she says "something that looks like fear" but in what way does it "look like fear"? She does not explain that at all. I would have at least like to have heard her explanation of why it "looks like fear".
I thought Jonathan’s talk was inspired and so profound. Definitely so much food for thought there. Conceptualizing technology as abundance and defense was foundational.
I am really looking forward to your chat with Carl Trueman. I am reading his new book right now and think it’s excellent. I think he is putting his finger on all the points that were missing from disenchantment.
An additional argument against Chloe's thesis comes from AI skeptic Gary Marcus, who says that scaling actually can't continue to deliver greater results - https://x.com/GaryMarcus/status/2070355999148351984
I also heard Cory Doctorow point out recently that the growth of improvement from scaling is already slowing.
Anthropic has been accused of being at the forefront of voicing the perils of AI in order to create 'regulatory capture' where it is at the head of the table. "Regulatory capture occurs when a regulatory agency, created to serve the public interest, becomes dominated or unduly influenced by the very industries or special interest groups it is meant to regulate, prioritizing their interests over the broader public good."
Anthropic is currently in a tussle with the US Government over how its AI product is used, Anthropic wants veto power over certain military applications.
The last section of the video "Can AI make us more human?" was really something. A Great Turning with a Buddhist AI. Picking a story for AI will be impossible. We are rebuilding the Tower of Babel.
I listened to Chloe. The cleverer she tried to sell the audience on Claude, the more it exponentially became clear how silly it is. Poor dear, she really tried..
I think I know why they turned off the comments on Chloe's video. It's a bunch of BS. It's manipulative. It's propaganda.
Agree. I think a better discussion on AI and "how it works" or how it "thinks" is here on Brink Lindsey's substack where he talks to his son who explains much more clearly what 'interpretability' actually involves: https://brinklindsey.substack.com/p/jack-lindsey-on-ai-psychology?utm_source=post-email-title&publication_id=1096044&post_id=195207760&utm_campaign=email-post-title&isFreemail=true&r=ne38r&triedRedirect=true&utm_medium=email
Could you expand? What I got from her talk was that basically AI is a tool that can be used amorally very much like a hammer “reflects” the morals of its operator. I found this point pretty trivial , I’m still puzzled why the speaker
was so mind blown by the simple fact.
On that topic, check out this other article: https://rhyd.substack.com/p/the-lying-oracle
Preach, sister! 🔥
I wondered about that.
We debate the 'consciousness' of technology while it gradually turns us into mitochondria.
My vibes based analysis of AI is it will both not live up to the hype and also be weaponized to further entrench existing power structures. But doubling down on technocratic top down processes will further erode the very things that make a society functional. It’s peak Faustian, thinking we can harness technology to wield it as gods. Our flawed and incomplete maps of the territory will never achieve a recreation of the territory itself. Our small errors will be magnified exponentially with increased computational power.
What annoys me the most about Chloe's starry eyed propaganda is her omission of the technology's rapacious material requirements. Its emergence isn't just a competition against humanity for "intelligence" it is a competition for resources.
Sure. If the AI companies manage to pull off the nuclear powerplants and the AI still fails, they can just pivot to energy sector and upend another industry.
👍😊
One problem is that consciousness is in fact integrally involved with human intelligence - intrinsic to it - despite the AI/Cog Sci notion that consciousness is unneeded - that they can discern "no role" for it. This ties into the second problem, namely, just saying, as per Chloe, that "AIs are not the same as us, but are inspired by our neural architecture" is unfortunately a very weak, very ineffective response to the ultimate question: "Are we not just machines as well?" Chloe's fallback has no in principle substance behind it, and the results of accepting that humans are equally machines, are, as I'm sure you intuit, horrific. I really suggest/hope, Mary, you'd get and study this book, even though a bit out of your normal field: "The Challenge to AI: Consciousness and Ecological General Intelligence." Having a grounded, clear, alternative model of the brain/mind (which is to say of perception, memory and cognition and how far LLMs are away from this) is becoming absolutely essential.
The book appears to be very on target.
I like to think so. :)
OMG - you wrote the book! lol
While I haven’t read it, the summary speaks well. I completely agree with your thesis.
AI summary of the book: Robbins’s central claim is that humanlike general intelligence and consciousness cannot be captured by treating the mind as a digital computer that manipulates symbols or patterns in isolation. Instead, he argues that real intelligence is rooted in the biological, time-continuous, and embodied nature of the brain as it engages with a real environment. In this view, cognition is not just computation over representations but a dynamic process grounded in the organism’s physical structure and its ongoing coupling with the world.
He reframes “general intelligence” as fundamentally ecological. Rather than being the ability to solve arbitrary abstract problems, it is the capacity of an organism to perceive and act within a rich environment, picking up affordances and acting appropriately in real time. Drawing on ecological psychology (e.g., Gibson), he emphasizes that perception is direct and world-involving, not just the construction of internal models from sensory inputs, and that intelligence emerges from this continuous organism–environment loop.
Consciousness, for Robbins, is inseparable from lived time, continuous perception, and the construction of an experienced world, influenced by Bergson’s philosophy of time and mind. He argues that standard computational models, which discretize time into steps and treat perception as encodable data, miss this temporal, experiential dimension. This critique extends to language and action: generative AI systems, he contends, lack genuine understanding, affect, and intentional, voluntary agency because they operate as pattern predictors rather than embodied agents with feelings and goals.
From this, Robbins concludes that creating truly conscious, generally intelligent machines would require radically different architectures—ones that replicate biological-like dynamics, continuous time, affect, and deep ecological embedding—rather than scaling up current symbolic or neural network approaches.
Sources
Ha. A pretty good example of the actual weakness of AI. If we cut through some of its high, but sad abstractions, it plain missed the major concepts and the concrete model:
A better essence: The argument is that Bergson, in 1896, saw the universe as a holographic field (50 years before Gabor's invention, 80 before Bohm's generalization to the universe as a whole - hence why Bergson was never understood). But, as opposed to Pribram (the brain is a "hologram") and Bohm, he - in essence -saw the brain as creating a modulated reconstructive wave, passing through this holographic field, specific to (or specifying) a source within the field as an image, e.g., the cup and stirring spoon. Gibson, who saw the brain as "resonating" to the invariants defined over the (coffee stirring) event, and thus "specific to" the cup on the table, right "where it says it is," out there on the table (his "direct perception"), to account for the image, is in effect relying on his resonating neural mass being, acting, simultaneously as a reconstructive wave within a holographic environmental field.
The spoon stirring, a fly going by - all are in the past - and this brings in the importance of Bergson's temporal metaphysic (unmentioned by the AI, along this holographic model). The transformation of the universal field (and the brain embedded therein) is indivisible, not a series of discrete states (per the classic metaphysic), hence the motion/transformation of the field itself carries an attribute, we can say, of "primary memory," and the specification can indeed be to a past portion of the transformation. Due to the biochemical dynamics underlying the resonating neural mass, underlying the chemical velocities, the specification is at a scale of time - a "buzzing" fly, wings ablur at our normal scale, but given the injection of a catalyst (e.g. LSD) increasing the oscillation rate of the neural mass, perhaps now a heron-like fly barely flapping his wings (a large part of Chap 3. is devoted the biochemistry behind this time-scale specification - lost to the AI). This vision of changing the scale of time of the specified experience (imagine one's body at a scale where the fly is specified as a vague cloud of electrons) and given there is no true spatial demarcation or precise outlines between our body and the field) explicates Bergson's very Zen principle: "The relation of subject and object, and their union and differentiation, must be treated in terms of time, not space:"
All this comprises a solution to Chalmers' rather mis-stated hard problem (better stated - explaining the origin of the qualitative image of the external world), but note, perception - our experience of coffee stirring - is not occurring solely within the brain in this model, so the experience cannot be solely stored there - in fact we require a 4-D model of being, and an entirely different model of memory and cognition (which the book describes).
I could go on - but this should give you an idea of how vaguely abstract, somewhat right, but missing of the concrete alternative model that the AI actually is. Interesting.
I had to research and study that understand it. And I would say that I am generally aligned with all of this. Despite some of it being a bit over my head. Thanks for sharing.
Agreed, very weak argument. Next we will be told to forgive AI for its poor behaviour because PTSD.
Love you mary!!
who do I serve? Yes—that question —lives in the heart of each of us—unbidden, maybe ignored, denied(I’m too busy!) always there.
What Jonathan's talk can’t quite answer from inside a room like ARC is the criterion for which practices actually do that building versus which merely feel formative. Learn poems, grow a garden, go to church — sure, but the version of each that passes and the version that fails are very different things. Memorizing a poem until it’s in the belly of the mind is different from reading it appreciatively once. A church organized around recurring embodied encounter with the same texts at the same moments does different things to a person than one organized around a weekly inspiring experience. The gap between the prescription and its fulfillment is a formation question, not a content question. And of course if it doesn't pass the Gargoyle's "Peasant Test" it is only useful for those with tickets on this post diluvian ARC... which would seem a more human version but still a version of a doomsday bunker.
To defend Jonathan - I think his point is praxis. It is that learning poems, growing a garden and going to church turns you into a certain kind of person in a way that intellectually understanding something doesn’t. It molds your soul.
Most excellent.
And the formed soul will live eternally.
Jonathan Pageau is an exceptional cultural analyst and I have been in his debt for almost a decade. Apart from a few technicalities, I have nothing negative to say about his talk for ARC.
It is a condensation of a recent talk for a Catholic school in Ottawa: https://youtu.be/ETaRAauDxS0 He says there that he personally uses AI as a research assistant—powerful but untrustworthy. That’s how I use it: Verify everything.
The question to ask, which he doesn’t quite address, is how AI influences one’s thinking and being and doing.
Crudely stated, it has changed me like every tool I have ever used has changed me:
The more I use it, the more proficient I become with it, the more ways I find to use it, the more dependent I become on it, and the more I frame my approach to every problem as an AI problem.
To paraphrase the old adage, if the most powerful tool you have is an AI hammer, then you tend to see every problem as a potential AI nail.
That’s been my experience, and when I’m most conscientious, that scares me shitless.
She made some conceptual leaps which she did not explain that well. First example: the word for "small" goes 'to the same place' in the LLM brain when the LLM is trying to come up with its opposite in different languages. But it's doing that because even if not explicitly, the original idea of machine learning and neural network was to get that kind of generalisation! Hell, you can create much simpler types of machine learning that can categorise in a similar way if trained appropriately but which no-one would accuse of being sentient. Then when the network has a "functional emotion" of "fear" again it has possibly 'just' formed an association from its data between 16g (why say 16000 mg ?) of paracetamol and the import (danger! Will Robinson) of a lethal dose which is being characterised as fear by a human. I mean she says "something that looks like fear" but in what way does it "look like fear"? She does not explain that at all. I would have at least like to have heard her explanation of why it "looks like fear".
I thought Jonathan’s talk was inspired and so profound. Definitely so much food for thought there. Conceptualizing technology as abundance and defense was foundational.
I am really looking forward to your chat with Carl Trueman. I am reading his new book right now and think it’s excellent. I think he is putting his finger on all the points that were missing from disenchantment.
-
Glad you had a good time Mary...
All my favorite thinkers who talk well together... awesomeness ✨️
An additional argument against Chloe's thesis comes from AI skeptic Gary Marcus, who says that scaling actually can't continue to deliver greater results - https://x.com/GaryMarcus/status/2070355999148351984
I also heard Cory Doctorow point out recently that the growth of improvement from scaling is already slowing.
Anthropic has been accused of being at the forefront of voicing the perils of AI in order to create 'regulatory capture' where it is at the head of the table. "Regulatory capture occurs when a regulatory agency, created to serve the public interest, becomes dominated or unduly influenced by the very industries or special interest groups it is meant to regulate, prioritizing their interests over the broader public good."
Anthropic is currently in a tussle with the US Government over how its AI product is used, Anthropic wants veto power over certain military applications.
The last section of the video "Can AI make us more human?" was really something. A Great Turning with a Buddhist AI. Picking a story for AI will be impossible. We are rebuilding the Tower of Babel.
I listened to Chloe. The cleverer she tried to sell the audience on Claude, the more it exponentially became clear how silly it is. Poor dear, she really tried..