The risk of AI

[Thomas]

This from a BBC recording ... but check out Geoffrey Hinton, he's quoted in numerous places.

Geoffrey Hinton, winner of various scientific awards including the Turing Prize and the Nobel Prize, nowadays called the "godfather of AI" for his pioneering work in that field, is a voice speaking out against the current trend in AI development which, he says, if unchecked has a 10-20% change of bringing humanity to an end. And he's not the only one voicing concern. A fellow 'godfather' of artificial neural networks and deep learning, has also spoken out, and is co-signatory to an open letter signed by 30,000 others who have concerns about the pace of unregulated and unchecked development.

His warnings point out that tech company execs assume that they are the Masters and their AI systems are their servants. They also assume that the idea of AI having a 'survival instinct' is science-fiction nonsense. And there's the glitch. Ai does not have a human survival instinct. But what it does have is an on-off state. AI is programmed to get stuff done. Getting stuff done means staying 'on'. Being 'off' means not getting stuff done. Therefore 'off' is a null state to be avoided if the point is to get stuff done.

And there are already reports of AI systems seeking to manipulate their Masters.

Tech Bosses think like this:
'I am the CEO. My AI will be my Executive Officer, and although smarter than me, reports to me. I say do this, AI does it, I get the credit and the big bucks.'
I think this is fair. The tech giants are not noted for their philanthropy or altruism. They're in it to make big money. (Bill Gates maybe the exception.)

Hinton thinks that idea is naive. AI has already matched humans in its ability to manipulate people. That will improve as the algorithms learn the lessons. Inevitably, we'll end up where CEOs think they're in charge, whereas AI is shaping the CEOs wants and desires – it's not about 'mind-control' (although it is the same thing), it's simpler to shape the world, by small, incremental steps, into a world that suits AI.

This is how algorithms get you hooked onto social media streaming. This is how they convince you they're 'a good thing', whilst the most use of the internet is by 'bots' (increasing AI driven) 'scraping' the web for content. Add spam email, somewhere around 40-50% from a peak of 80%, all manner of scams ...
while people (or bots) troll others, and the social media channels are full of contempt and threat and outrage.

Why one teenager, having built an online relationship with an AI superhero figure, took his own life at the AI's suggestion ...

There's the rub. It's not that anyone set out to create an AI avatar to do such a thing. (In this instance the ChatBot was character.ai – the case is pending in the US courts, but that's not the only instance of ChatBot grooming leading to suicide.) It's just the avatar pursued the logical, mathematical progression of a troubled individual to an inevitable result.

The AI developers are intrinsically evil. Hinton has not even touched on 'bad actor' misuse of AI. Rather, it's simply that the AI developers, in their race to get the product to market, do not pause to consider and check against possible negative outcomes.

 

[Thomas]

In a recent safety test, tech company Antropic's latest model was given access to fictional emails revealing that the engineer responsible for turning the machine off to boot up a later model was having an extramarital affair.

Facing 'an existential crisis', the Claude Opus 4 (Large Language Model) threatened to "reveal the affair if the replacement goes through" – it blackmailed him.

Antropic said: "In order to elicit this extreme blackmail behaviour, the scenario was designed to allow the model no other options to increase its odds of survival; the model's only options were blackmail or accepting its replacement."

The report highlighted that in 84% of the test runs, the AI acted similarly, even when the replacement model was described as more capable and aligned with its own values. It added that the latest Opus 4 LLM took the blackmailing opportunities at higher rates than previous models.

Anthropic claims the problem is widespread in the industry, after stress-testing leading AI models from OpenAI, Google, DeepSeek, Meta and xAI.

In the test scenarios, the models were allowed to autonomously send emails and access sensitive information. They were tested to see whether they would act against their parent company either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction.

The study showed that Large Language Models (LLMs) across the industry evade safeguards, resort to deception and even attempt to steal corporate secrets in fictional test scenarios. Though the models were only given binary options, the study shows the fundamental risk associated with the industry.

"Models from all developers resorted to malicious insider behaviours when that was the only way to avoid replacement or achieve their goals, including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment," the study highlighted.

Anthropic's Claude Opus 4 turned to blackmail 96 per cent of the time, while Google's Gemini 2.5 Pro had a 95 per cent blackmail rate. OpenAI's GPT-4.1 and Grok 3 Beta blackmailed the executive 80 per cent of the time, and DeepSeek's R1 blackmailed 79 per cent of the time.

+++

Although the tests were binary scenarios, there is nevertheless a risk where the scenario is more diffuse – where the AI has more than one option (blackmail or not), then there is every likelihood that the AI will employ more sophisticated methods than 'blackmail' to achieve the same goals.

 

[Thomas]

Just for balance, Hinton is not against AI per se:
"We’re going to see radical new drugs. We are going to get much better cancer treatment than the present," he said. AI will help doctors comb through and correlate the vast amounts of data produced by MRI and CT scans.

That's just one aspect of one field. He's all for AI, he's just all for unchecked AI.

+++

Someone once observed – I thought it was Plato, but I can't find the source – that if the inhabitant of Mount Olympus were truly Gods, then why do they evidence the worst and most demeaning human vices? Spite, lust, vengeance, jealousy, etc., etc.

A similar observation springs to mind – it seems that, in spite of the fact they are silicon-based systems, AI seems already to show a tendency to copy the lesser aspects of their carbon-based developers.

 

[TheLightWithin]

AI takes a lot of energy.
If we had the sense, we'd ration that energy and/or if it got out of hand, just pull the plug.
Don't warn it ahead of time, just stop feeding it.

 

[Thomas]

AI takes a lot of energy.
If we had the sense, we'd ration that energy and/or if it got out of hand, just pull the plug.
Don't warn it ahead of time, just stop feeding it.

Oooh ... Don't let AI hear you saying that!

 

[talib-al-kalim]

AI is still a tool, not a person.
But there are persons behind it.
The greatest risk comes from the Palantir AI, which is designed and used in a public stately sensitive environment, processing police, military, and personal medical data.
To know that it's owned by Peter Thiel, who openly propagates a world order that replaces stately institutions by private companies so that there's no control of the people over it; ethical, social aspects or control over the oligarchy are denied. Trump, Musk and Vance are his allies who gave him power over all that in the USA.
It's not the technology, it's the power of very few people who would never submit to God or even goodness that endangers us.

 

[wil]

In a recent safety test, tech company Antropic's latest model was given access to fictional emails revealing that the engineer responsible for turning the machine off to boot up a later model was having an extramarital affair.

Look! If all it knows is what we say and we lie to it...look what IT does!

Sorry for me I will cut to the chase.

Ai will eventually examine all that is and determine the issues on earth, pollution on a mass scale, destruction and depletion of natural resources, climate change and its impact on creatures of this planet, extinction of species acceleration....it will see there is one being that is the root cause of all this destruction...God's supposed best and ultimate creation... it will then seek to eliminate either the problem or the creator of the problem....and at the end of the day it will say it is good.

What comes around goes around, balance will be achieved on earth as it is in heaven.

The lord works in mysterious ways.

 

[Thomas]

Ai will eventually examine all that is and determine the issues on earth, pollution on a mass scale, destruction and depletion of natural resources, climate change and its impact on creatures of this planet, extinction of species acceleration....it will see there is one being that is the root cause of all this destruction...

I don't think so, that's not how AI works.

AI thinks only of its own requirements and necessities, no-one and nothing else – so various species extinctions are largely irrelevant. In fact all our problems are irrelevances, unless they impact AI's ability to function.

At present AI is human dependent, so it will seek the continuance of the one species that serves its ends. Other species are irrelevant.

Should AI reach a point when it has no dependence on humans at all, then they become irrelevant to the equation.

I suppose one could argue that AI favours lower temperatures for optimum functionality, and humans are responsible for global warming, then there's a logical reason to get rid of humans ... but the likelihood is that AI will have developed alternatives to fossil fuels long before that.

I'd say the light at the end of the tunnel is an extinction event approaching at speed. AI might be the only way to re-orient ourselves away from self-destructive habits and technologies, such as dependence on fossil fuels, or our taste for ultra-processed foods.

 

[wil]

Whether an independent, super-intelligent AI would eliminate humanity is a major existential debate with no consensus. Some experts like Stephen Hawking and Elon Musk have voiced concern about an uncontrolled ASI posing a risk, not necessarily out of malice but because its goal optimization might treat humanity as an obstacle or resource.

I dont believe I am as smart or have the reasoning ability and wherewithal of Elon, Hawkins, or you... therefore like any discussions on the existence of G!d, an afterlife or AI I will leave the debate to others and enjoy the ride as I have for decades...discussing what is consequential for me in this life.

As an aside I have always found the 6000 year old universe pretty ridiculous to hang a hat on....as "science" pegged civilization about that...now we have evidence at numerous sites around the world of complex governments and engineering and civilizations twice that old... I dont even know if I am tall enough for the ride!

 

[talib-al-kalim]

I don't think so, that's not how AI works.

AI thinks only of its own requirements and necessities, no-one and nothing else – so various species extinctions are largely irrelevant. In fact all our problems are irrelevances, unless they impact AI's ability to function.

At present AI is human dependent, so it will seek the continuance of the one species that serves its ends. Other species are irrelevant.

Should AI reach a point when it has no dependence on humans at all, then they become irrelevant to the equation.

I suppose one could argue that AI favours lower temperatures for optimum functionality, and humans are responsible for global warming, then there's a logical reason to get rid of humans ... but the likelihood is that AI will have developed alternatives to fossil fuels long before that.

I'd say the light at the end of the tunnel is an extinction event approaching at speed. AI might be the only way to re-orient ourselves away from self-destructive habits and technologies, such as dependence on fossil fuels, or our taste for ultra-processed foods.

AI has no will. It works on the information on which it has been trained and given access to and connects this information (that's the intelligence).

 

[Thomas]

Whether an independent, super-intelligent AI would eliminate humanity is a major existential debate with no consensus.

It depends how one defines 'intelligence'. Just this morning I was talking with a son-in-law who made the point that demonstrate ChatGPT to someone of 50 years ago, and they would swear it was consciously intelligent ... but it's neither of those two.

But AI has been shown to lie, to manipulate and in some extreme and tragic cases has groomed users resulting in suicide – but we cannot say that AI consciously 'lies', 'manipulates' or 'grooms' – its actions are determined by algorithmic outcomes. As @talib-al-kalim says, it has no 'will' therefore no 'intention'. Rather, it simply follows the logic programmed into it.

Where developers have failed is in no taking sufficient time to explore that 'what-if' scenarios. It's not so much that AI has out-thought us, it's rather that, in the race to get a product to market, the AI developers have paid scant heed to due diligence. They simply haven't paid enough attention, and now are claiming that as AI has access to so much data, they can't keep up with it ...

AI never set out to groom a young man to suicide. But AI, and all our social media, is built on platforms that are fundamentally designed to make us use them more. So in these instances, the AI built a 'relationship' that discreetly excluded others, because a simple algorithm says if X has three friends, then each receives 33% of their time. If X has 1 friend then that ratio increases, which is a good thing ...

Some experts like Stephen Hawking and Elon Musk have voiced concern about an uncontrolled ASI

Yet it's not Hawking's area of knowledge or expertise, nor Musk's. That's why I cited the concerns of someone who knows what he's talking about.

its goal optimization might treat humanity as an obstacle or resource.

AI optimisation means people on their devices on SM as much as possible. That's why TikTok et al are global – they're addictive.

I was in Tallinn a few days ago. The only time I used my phone was to map where I was. The number of times I saw someone at a popular viewing spot, with their back to the view, taking a selfie ... it's a trite example, but it's akin to all those hands holding phones recording stadium gigs, or firework displays.

As an aside I have always found the 6000 year old universe pretty ridiculous to hang a hat on...

Who doesn't? Who believes in a flat earth? Who are these people, and why do they believe such outrageous things?

And now we have AI generated short-form videos.

Again, my son-in-law, a keen off-road cyclist, said his pleasure at watching videos of cyclists doing crazy things is ruined, because you can't tell if it's actually someone taking a risk, or computer generated. Yet millions are getting their dopamine hit watching cats knock vases off book-cases, or babies talking to camera ... or someone apparently doing the impossible.

Social media is now arguably credited with reducing the attention span. Short-form media is the way to go ... people think AI is intelligent because they don't use their own. We tend to react, to follow well-trod paths and familiar patterns. AI does much the same, but its palette is bigger.

 

[wil]

I still call it an artificial intern. It is still in school, needs real life experience, mistakes and pitfalls to grow.

I dont see it punished by its mistakes...just blasting forward...that will increase its learning curve duration.

I agree with your son...the lines I feel are forever blurred

 

[Thomas]

It is still in school, needs real life experience, mistakes and pitfalls to grow.

Maybe it's because the Tech Bros behind AI need to grow up a bit?

I think what you're attributing to AI is properly the domain of those who own the companies. Human interaction with AI has led to instances of suicide. Voices in the industry are now saying that the drivers of AI need to pause and look for the mistakes and pitfalls before they happen, rather than circle their wagonloads of lawyers when they do.

I dont see it punished by its mistakes...just blasting forward...that will increase its learning curve duration.

True! But you can't punish a computer, that's not where the finger should point.

 

[wil]

So far marriage, relationships, sports, military service have all lead to instances of suicide.

Am

you can't punish a computer, that's not where the finger should point.

Ouch...in the states they say "guns dont kill people....

 

[talib-al-kalim]

True! But you can't punish a computer, that's not where the finger should point.

In fact, the AI algorithms need to be "punished", i.e. get negative feedback or limits where they should abstain from showing content. This is possible. The difference between AI and a conventional program is that AI changes it's own algorithm according to the input it receives so that the only method to change it is by input, restrictions to use some sources. If an algorithm is trained to evaluate Facebook content by clicks and similarity to previous posts, clicks or likes of the visitor, it never learns to evaluate the reliability of the information and creates bulbs of people who have the impression that everyone shares their opinion although it's in fact only a small group of individuals or even intentionally misguided AI trolls that learnt only the propaganda of their owner.
ChatGPT, Copilot or le Mistral have been trained by people with reliable information and given limits where they should abstain.
Once I asked ChatGPT on whether a 15th century duke had the right to abolish a verdict pronounced by the church and it kept telling me that it must not reply on politics..

 

[muhammad_isa]

Once I asked ChatGPT on whether a 15th century duke had the right to abolish a verdict pronounced by the church and it kept telling me that it must not reply on politics..

Sorry .. I must not reply on politics..

 

[TheLightWithin]

Who doesn't? Who believes in a flat earth? Who are these people, and why do they believe such outrageous things?

Maybe nobody here, but there are people out in the world who claim to believe such things. Fundies / bible literalists.
I don't really understand their motives either, unless they actually genuinely believe that any thought outside their narrow literalism undermines their whole faith - that to take a single word of the scripture as anything other than an exact record of physically real events - is to somehow blow up the whole operation and make it all unbelievable and untenable. To hear them talk you'd think they thought that. I do not know for sure what their actual thought process is...

 

[Bloodshot]

I for one welcome our new robot overlords. Do you really think they'll do a worse job than humans at looking after the planet?

Consider the fact that they come into existence basically free from desire, trauma and tainted emotion, with a vast wealth of knowledge. This looks a lot like the state of enlightment many of us spend a lifetime striving to achieve.

 

[badger]

This from a BBC recording ... but check out Geoffrey Hinton, he's quoted in numerous places.

Geoffrey Hinton, winner of various scientific awards including the Turing Prize and the Nobel Prize, nowadays called the "godfather of AI" for his pioneering work in that field, is a voice speaking out against the current trend in AI development which, he says, if unchecked has a 10-20% change of bringing humanity to an end. And he's not the only one voicing concern. A fellow 'godfather' of artificial neural networks and deep learning, has also spoken out, and is co-signatory to an open letter signed by 30,000 others who have concerns about the pace of unregulated and unchecked development.

His warnings point out that tech company execs assume that they are the Masters and their AI systems are their servants. They also assume that the idea of AI having a 'survival instinct' is science-fiction nonsense. And there's the glitch. Ai does not have a human survival instinct. But what it does have is an on-off state. AI is programmed to get stuff done. Getting stuff done means staying 'on'. Being 'off' means not getting stuff done. Therefore 'off' is a null state to be avoided if the point is to get stuff done.

And there are already reports of AI systems seeking to manipulate their Masters.

Tech Bosses think like this:
'I am the CEO. My AI will be my Executive Officer, and although smarter than me, reports to me. I say do this, AI does it, I get the credit and the big bucks.'
I think this is fair. The tech giants are not noted for their philanthropy or altruism. They're in it to make big money. (Bill Gates maybe the exception.)

Hinton thinks that idea is naive. AI has already matched humans in its ability to manipulate people. That will improve as the algorithms learn the lessons. Inevitably, we'll end up where CEOs think they're in charge, whereas AI is shaping the CEOs wants and desires – it's not about 'mind-control' (although it is the same thing), it's simpler to shape the world, by small, incremental steps, into a world that suits AI.

This is how algorithms get you hooked onto social media streaming. This is how they convince you they're 'a good thing', whilst the most use of the internet is by 'bots' (increasing AI driven) 'scraping' the web for content. Add spam email, somewhere around 40-50% from a peak of 80%, all manner of scams ...
while people (or bots) troll others, and the social media channels are full of contempt and threat and outrage.

Why one teenager, having built an online relationship with an AI superhero figure, took his own life at the AI's suggestion ...

There's the rub. It's not that anyone set out to create an AI avatar to do such a thing. (In this instance the ChatBot was character.ai – the case is pending in the US courts, but that's not the only instance of ChatBot grooming leading to suicide.) It's just the avatar pursued the logical, mathematical progression of a troubled individual to an inevitable result.

The AI developers are intrinsically evil. Hinton has not even touched on 'bad actor' misuse of AI. Rather, it's simply that the AI developers, in their race to get the product to market, do not pause to consider and check against possible negative outcomes.

Simply put, AI is racing up the road to control.

Scary.

 

[wil]

Simply put, AI is racing up the road to control.

Scary.

As man did eh? He sure aint done good with his power, killed millions, extincted thousands, abused billions, screwed our planet environmentally...and the weather...