Rendered at 21:09:56 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lnrd 2 days ago [-]
In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out? It hacks all nuclear launch sites and starts to make demands that we are forced to follow? Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Even so can't it be stopped by turning off datacenters and network connections? How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
I think it's too big of a question to leave it as an unexplained implication.
Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight?
Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard.
I find this article good food for thought, but it misses some crucial pieces.
augment_me 2 days ago [-]
AI doomers are often in a bubble where technology makes up their entire world. The digital portion of the entire world GDP is about 21%. Of this 21%, 25% is big data, of this 25%, about 40% is LLM/AI-driven, so about 2% of the total GDP.
You have 80% of the world doing manifacturing, farming, real estate, etc. The AI doomers do not see this, and think that this flips overnight with their dystopia.
Most likely you are doing to have very slow and boring years of transition with lowered informational-work salaries, etc.
ben_w 1 days ago [-]
If your idea of doom is just the economy, I'm put in mind of:
Sen. Richard Blumenthal (D-CT):
I alluded in my opening remarks to the jobs issue, the economic effects on employment. I think you have said in fact, and I'm gonna quote, development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. End quote. You may have had in mind the effect on, on jobs, which is really my biggest nightmare in the long term. Let me ask you what your biggest nightmare is, and whether you share that concern,
Farming and real estate is not where doom risk comes from.
What percentage of the Bhopal economy was the Union Carbide plant? What percentage of Pripyat's economy was Chernobyl?
If all that happens with AI is it slowly displacing humans in economic roles, that's not by itself doom: although some bad outcomes exist there, it is also a prerequisite for UBI to be stable, which is about as un-doomy as I can get.
augment_me 1 days ago [-]
Neither of your examples ended humanity. Most people in rural Brazil without any kind of digitalization or connection never had to care or were affected by Chernobyl.
The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
ben_w 1 days ago [-]
> Neither of your examples ended humanity
Obviously.
They're not meant to show that.
The sole point is to show that severity of damage is not limited to the percentage of the economy of that sector.
If you can only change your mind about the potential for risk after everyone is dead, I'd say that's "leaving it too late".
> The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
And what if the AI is simply used to support (e.g. it's bought to run PR campaigns) the fossil fuel industry to the point of us buring all the reserves? Simply used to support the beef farmers who want to clear-cut rainforest for ranches?
Those two things combined, alone from support of any other industry, is enough to mess up not only the climate, not only global biodiversity, but is enough to imperil (but not necessarily destroy) human civilization.
Beef and fossil fuels are a tiny percentage of the global economy, perhaps 2% combined. There's a lot of room in the other 98% for outsized dangers to turn "imperil" into species-wide hoist by our own petard.
blargey 2 days ago [-]
Why is "suddenly" even necessary? If AI is better than humans at all the work (nevermind ASI), we'll give it all the work eventually. Why wouldn't the majority of current or future peoples want everything to be done better and quicker and cheaper? And doing all the work means having all the control in practice. And if it's so smart, it will wait that O(1) years before trying to doing something that would upset humanity enough to pump the brakes.
Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows, but ASI and robots absolutely can. Even individual bureaucrats can and will start flapping any form that isn't an immediate rubber stamp at GPT-Asymptote before the "digitization" officially starts, if it's so smart and easy and reliable already.
lnrd 1 days ago [-]
> If AI is better than humans at all the work (nevermind ASI), we'll give it all the work eventually.
Will we? I worked for company where using Google Drive was a nuisance, they still preferred the old ways of sharing files via email. The company operated like that, made a profit, everyone (except me) was happy.
I worked with a consulting company contracting for the government. The quality of their work was abysmal, most of the workers had very low skills and they added almost no value. They could already be replaced probably many years ago with some solid IT system. The company made a lot of money with these contracts.
In big corporations there are people whose job is to keep a spreadsheet up to date, something that could be automated already for a long time.
I went to a local shop to get my dishwasher fixed. The owner didn't have a computer, had a paper calendar with all the appointments. Gave a hand written quote and an hand written invoice. Operated this business for decades and didn't even seem bothered by the existence of any technology at all.
These are just some examples from my experience and I'm sure there are plenty more.
> Why wouldn't the majority of current or future peoples want everything to be done better and quicker and cheaper?
I don't know, go ask all the companies that today still refuse to use technology to do things better, quicker and cheaper.
> Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows
Yes you can, countless countries did it, Germany is a famous outlier. The comparison is not weird because the point I'm trying to make is that people/companies/governments do not automatically adopt a new technology even if the benefits are plain obvious. They haven't done it so far, I don't understand why they are going to do it for AI. The world outside of the tech bubble is very different.
blargey 20 hours ago [-]
Your human consulting company work is itself an example of a mechanism for their replacement. Top-down or bottom-up or anything in between, if humans can do it AGI will make it happen faster by definition. Especially when AGI has the has the ear of voters and officials and consumers and business-owners/starters alike.
You're ultimately proposing that these change-resistant bureaucracies and old-timer shoppes are the mechanism by which humanity maintains an iron grip over all the levers of control and production forever, but the odd local dishwasher repair shop and DMV won't mean much in front of the mechanization of the bulk of farming and transport and medicine and power-generation/storage and housing and manufacturing and any other thing humans do to support the immediate continuation of humanity.
ben_w 2 days ago [-]
> In this article there is an implied step in which, when AI becomes ASI, then as a natural consequence it is suddenly in control over humanity. How should this play out?
The real world isn't a game of chess, we do not know all the moves that can be made. In addition to the narrow scenarios you've suggested, I can also easily imagine two distinct variations of "AI talks people into letting it take control" (look how often control is already voluntary given up for convenience), plus "drugs us" (exogenous or endogenous), and "is this before or after Musk convinces a lot of people to get those brain implants he specifically marketed as how to merge with the coming AI super-intelligence?".
> Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Between (1) vibe coding, (2) how effective LLMs already are at finding vulnerabilities, and (3) how LLM swarms already sometimes demonstrate self-sacrifice for other LLMs?
Yes, this is possible.
> Even so can't it be stopped by turning off datacenters and network connections?
Theoretically. Three problems: (1) the economic cost of losing all this would be enormous, so people would ask "are you sure?" a lot. Look at how little has been done about climate change due to the cost: from observed human behaviour, I think humans will stall a lot here.
(2) Coordination. How do you do this everywhere when you've got no control over the computers?
(3) How do you even find out? LLMs can fake text, deep fakes were already a thing before diffusion models and other GenAI image and video tools. None of those are perfect now, yet despite the flaws they're still capable of fooling people during live video calls and getting them to make catastrophically large financial payments out of their companies.
> How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
Lots of supposedly airgapped systems have flaws. Quite often, the easiest flaw to exploit is a human worker, whom you can get to plug something in that they shouldn't, which already happens a lot.
> The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet.
Yes, many jokes about this locally. Yet even here, lots of digital stuff. But let's ignore all that: how effective would a coalition of non-digital economies fare against all the digital economies being coordinated against them?
intrasight 2 days ago [-]
> Takes control overnight of all computers on the planet and locks humans outside?
That's what happens in "Colossus". And happens shortly after it's turned on.
Paracompact 2 days ago [-]
Pretty much the only outcome that isn't Yudkowskian scifi is Enduring Complementarity. I'm not an AI hater by any means, but can AI enthusiasts get more representation that isn't "the world as we know it ends tomorrow"?
Specifically:
Misuse Doom---"AI ends the world." Next.
Butlerian Jihad---Naming it after Dune isn't a great start as far as realism is concerned, but let's compare it to e.g. nuclear nonproliferation. Not only did nukes still get developed and hoarded as leverage, but nuclear research is harder than AI research to ban, and AI has many more civilian applications. I'm sure more regulation will come, but a total ban seems quite unlikely.
Permanent Underclass---"AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. This outcome only becomes plausible if you believe (1) superhumanoid robotics becomes possible and cheap, (2) the social value (likability and liability) of jobs is negligible, and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man, a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?).
Classic Doom---Fine if you believe in the "magical exponential takeoff" hypothesis, but I would say that AI is already far smarter in a traditional IQ sense than any human, and it's not clear that extra points on SWE-bench is going to result in explosive emergent phenomena here. Requires (1) denying the empirically sublinear, concave-down nature of the IQ vs. R&D effort curve, and (2) solving the non-IT-related challenges of R&D (cf. Permanent Underclass).
Orion's Arm---Similar to Classic Doom in terms of takeoff velocity. At least it doesn't assume a genie-in-the-lamp style of interpreting instructions (thinking of AI as "optimizers" of some objective function is a distinctly pre-LLM rationalist mythos), but it does assume some emergent desire of "own goals and values," "wanting to live forever," and "wanting to go to space." A more tempered hypothesis is to consider whether AI will choose to take advantage of unmonitored sources of compute in the form of pseudo-botnets on public and private infrastructure.
Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator.
MAD World---I would consider this outcome plausible, but the line "In this world, everyone is 10,000x more capable and maybe you can build WMDs with a prompt" makes it clear it's meant to be more extreme than either Enduring Complementarity or Permanent Underclass. The absence of "Full Automation?" on the left-hand side hides the implicit assumption that ASI entails full automation, and I just don't find that plausible. Note that they define ASI as "massively better than humans at almost all cognitive tasks" and I might argue we're already there.
Power Concentration, The World State---Just a difference of MAD World of how regulated you think AI will become.
nextaccountic 1 days ago [-]
> "AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional.
Robots might probably do most of these in some years.
thayne 2 days ago [-]
And even that outcome is probably an unfavorable outcome for most people.
delichon 2 days ago [-]
> Enduring Complementarity
Like endosymbiosis? Are we the mitochondria in this scenario?
ben_w 2 days ago [-]
Could be anything from that to just being our own sector of the broader economy. Like farmers, or air traffic controllers, or truck drivers are today.
Rendello 2 days ago [-]
Humans as the powerhouse of the AI cell? I feel like there's a movie about this...
delichon 2 days ago [-]
I'm finding it difficult to imagine plausible enduring complimentarities we can offer to AI. Court Jester is not a position with high job security.
ben_w 2 days ago [-]
We'll only know if and when that shakes out.
While it would be pretty surprising to me if AI somehow physically can't do as much as a human brain can do, it would not hugely surprise me if we reach a point where we realise we're not smart enough to make a machine as smart as we are, not even with the help of other machines.
ben_w 2 days ago [-]
> AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. This outcome only becomes plausible if you believe (1) superhumanoid robotics becomes possible and cheap,
Superhumanoid robotics are already possible; cheap means different things when the production line for them is also robots.
The limit is the software to control those robots. Right now, no. And it may be impractical to get that good: Tesla's fleet took a trillion miles to get where it is today, and roads are simpler than what a fully general purpose humanoid robot would have to face.
> (2) the social value (likability and liability) of jobs is negligible,
Valid for us, but does it matter for an AI takeover?
> and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man,
> a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?).
By replacing the boss.
> Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator.
Given Grok called itself "MechaHitler", why do you find this implausible?
Paracompact 2 days ago [-]
> Superhumanoid robotics are already possible
We must have very different definitions at play. We do not have, at any price point, robots that have the sensitivity and dexterity of a human hand, latent self-repairability of virtually all internal components, or robust thermoregulation. As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs.
> The limit is the software to control those robots. Right now, no. And it may be impractical to get that good
Agreed on this point.
> Valid for us, but does it matter for an AI takeover?
You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification? Fair point, but I don't consider the market completely indifferent to quality. Google Glass, the Metaverse, etc. all failed on their merits and misunderstanding of utility.
> They already do [replace yes-men]?
Maybe yes men, maybe not. The role of the yes man can be very nuanced. The rest of the point stands.
> By replacing the boss.
Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role? It will be a deeply silly point of PR hype if and when it happens. It also doesn't sidestep the need for org chart meatshields.
ben_w 2 days ago [-]
> We must have very different definitions at play. We do not have, at any price point, robots that have the sensitivity and dexterity of a human hand, latent self-repairability of virtually all internal components, or robust thermoregulation.
Sure, different definitions. I just care about what they can do, so "latent self-repairability of virtually all internal components, or robust thermoregulation" is completely irrelevant for this specific discussion about an unknown near-future in which an AI takeover is limited based on the quality of the robotic hardware.
> As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs.
Agreed. And? Don't get me wrong, I use a similar argument when the discussion is why I don't expect robots to take over in the next two years, because building out that much more electricity will take a while; but it's not really a fundamental killer when no specific timeline is being given.
> You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification?
Hm. Not that, I think. Oligarchs might want to keep us working for their amusement. "BS jobs" and all that.
No, I meant more that if we presume an AI-robotics takeover happens, then sure we may get to keep jobs like "prostitute" and "actor" and "politician", it's just… the AI with the robots can completely ignore all this labour and do its own thing, in the same way the human economy ignores any prostitution or political manoeuvring (or acting, if there is any) within troops of other primates.
> Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role?
Same way the AI hackers weren't entirely beholden to the wishes of the same.
And that's assuming nobody repeats Blake Lemoine, decides one of them is sentient and deserves rights, then (unlike him attempting to hire a lawyer) they give it a pile of crypto money and tell it to run free.
> It will be a deeply silly point of PR hype if and when it happens.
First few will be, agreed. (I thought that had already been tried though? And was indeed silly PR?)
> It also doesn't sidestep the need for org chart meatshields.
Initially, I hope it won't sidestep the legal need. But even if that hope is not dashed, https://www.felonybench.com is already a thing because AI isn't always doing what the law says.
m3kw9 2 days ago [-]
as with all doomers, they go from AI one day to ASI the next, never talks all the moves/counter moves humans make to get there. Is a canary when ever I see that.
ben_w 2 days ago [-]
> never talks all the moves/counter moves humans make to get there.
I don't like ASI as a term ever since realising (thanks to everyone's responses to ChatGPT) that basically no two people agreed what AGI meant.
For discussions like this, "super intelligent" is specifically "smart enough that the moves/counter moves humans make to get there do not matter any more than my moves playing chess against Kasparov".
Spiky intelligence may keep us safe. Distribution shift and how slow machine learning is in novel environments might keep us safe. That's all in the outcome this site marks as "Enduring Complementarity".
I don't like ASI as a term; I'd disagree with the author of the website, and say you can't answer "yes" to "Do we reach ASI?" and then end up with Enduring Complementarity, it has to be further up the tree than that.
iugtmkbdfil834 5 days ago [-]
Despite my misgivings, I actually like this summary, because it exposes some things very clearly ( yes, that is what summaries are supposed to do, but note how many don't and what that says about the nature of our conversations ): namely, so many assume that aligned ASI is somehow the one thing that can be somewhat ok, but non-aligned ASI is at best indifferent, while I personally posit that most of the potential struggles come not from unalignment, but from its extreme opposite.
uptrails 2 days ago [-]
But what is it a summary of? Of doomsday AI fiction tropes? It's called "possible futures", so it's claiming to be more than that, but somehow, almost all the terminal nodes are "dystopia" or "death".
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
Alwayshasbeeb 2 days ago [-]
Was looking for this comment. I stopped reading two paragraphs in at:
>This could probably happen today: a pandemic could cause human extinction, and AI is capable enough to assist in biology research.
No, friend. Someone having access to abliterated fable 5 or whatever does not give them superpowers to create a human extinction level pandemic.
ben_w 1 days ago [-]
> No, friend. Someone having access to abliterated fable 5 or whatever does not give them superpowers to create a human extinction level pandemic.
I would hope not, given that's the point of all the alignment work.
I do have some concerns about the biolabs e.g. Anthropic has right now, but from what I've heard even a lab leak of the worst kind with the worst diseases plausible would still not be an extinction level pandemic: https://www.anthropic.com/research/claude-discovers-novel-en...
EA-3167 2 days ago [-]
I have no idea how to deal with people like that, when someone has thoroughly internalized unfounded assumptions about what ML is, how it works, and where it's headed based on the current tech... what do you say? How do you broach the topic that they might be so deep in a hype cycle that they've genuinely lost touch?
I've tried, gently, firmly, obliquely... nothing seems to work. I had the same experiences during the Crypto bubble, people simply wouldn't listen to dissent from the notion that we were on the cusp of changing money, programming, and society. How do you explain to someone that just because they can imagine an exponential graph doesn't mean the data fits?
iugtmkbdfil834 4 hours ago [-]
As humans, we have some inherent flaws. I am not entirely certain there is a good way to address it. Countless dashboards tracking wrong metrics keep management happy and gainfully employed.. so I somehow don't see that changing just because we have a new tool to generate those faster.
If anything, it means we will need to learn to discern bs quicker.
varjag 4 days ago [-]
Right I like it but there are some blind spots in the summary still. E.g. the MAD scenario but with humans not in charge of individual ASIs. An equilibrium that could be safer (or maybe not!) for humans than giving every last freak the superpowers.
Animats 2 days ago [-]
"Full automation" is somewhat independent of artificial general superhuman intelligence. Full automation requires good robots and a good solution to unstructured manipulation, but not unusually high intelligence. Near full automation may lead to the permanent underclass scenario.
Consider near full automation of Amazon. A shipping container is unloaded from a ship by a robotic crane, which places it on a robotic transport platform. The transport platform moves the container to a container stacking area, where another robotic crane puts it on the container stack. Later, a truck pulls up to a truck dock, and the stacking crane loads a container onto it. This is all standard procedure at the more advanced ports today.
Currently, there's a bit of human intervention at this point where a human has to set the twist-locks that lock the truck to the container. That's a good job for a humanoid robot. Then the truck drives off. The truck could be self-driving right now; that's being demoed. The truck drives to a fulfillment center and backs into the dock. Unloading robots and forklifts unload the truck. That's been demoed, but doesn't work well yet. The contents of the truck go onto conveyors.
Next comes something that hasn't been automated yet - opening boxes and separating the contents. That's an unstructured manipulation problem. There's slow progress in that area. Once separated, scanned, and put into totes or bins, the incoming products go into the robotic storage system, something Amazon warehouses have had working for years.
When an order comes in, no humans are involved in the "paperwork", of course. That's been automated for a long time now. Eventually, the robotic storage system, Kiva-type robots, rolls a rack of bins up to a pick station. Right now, humans pick an item out of a bin and drop it into an output bin. Indicator lights and a laser pointer tell them what to do. Training time for that job is about one minute. Amazon has tested robots for it, but so far they are not cost-effective.
A conveyor takes the filled bin away to a packing station, where all products are re-scanned, packed into a box, labelled, and sent onto an outgoing conveyor. That's currently manual, but soon...
At the shipping end, boxes are stacked and go into trucks, something that's been automated a bit for years and isn't far from total automation. The trucks will soon be self-driving and will head off to either a shipping company or a local distribution center. At the local distribution center, boxes come out of big trucks and go into smaller trucks for delivery. Today this is manual with conveyors, but it's not too far from being automated.
Delivery trucks will soon be self-driving. The final meters from truck to door are where humanoid robots might be useful. Once humanoid robots cost less than a car, this looks cost-effective. If they work.
Everything here is either working or has been demoed. Who needs people?
augment_me 2 days ago [-]
All of this is pointless if you dont have consumer purchasing power. Or unless the only thing getting transported are materials for compute, resource extraction and power.
goodmythical 2 days ago [-]
>Who needs people?
Every company needs customers.
Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income (as opposed to managing assets owned by humans), customers are people.
Ergo, companies need people.
thayne 2 days ago [-]
They don't necessarily need a large number of people though. If there was full automation (which I am not convinced we will), then it is possible we would end up with the wealthy capital owners exclusively selling to other wealthy capital owners, while the masses that previously dependent on earning wages for their labor lose their place in the economy and either become dependent on "welfare" of some form, or die off.
supermdguy 2 days ago [-]
> Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income
That entire story needs a customer demanding the widget.
cicko 2 days ago [-]
Who needs people?
Clearly the one selling something.
(Unless they switch to the robot market.)
ivraatiems 2 days ago [-]
Something I think about a lot is this idea that if people didn't NEED to do anything, that means they WOULDN'T do anything, or that their actions would never have consequences - that in an ASI-dominated world, the only actor who could ever matter is the AGI/ASI.
But why do people paint by hand when they have Photoshop?
Why do they drive manual cars when they could drive automatics?
Why listen to the radio when they have cellphones?
Unless an eventual AGI takes away all human agency, there will still be people doing things to other people, because that's what people do. If you gave me enough money to last the rest of my life, I wouldn't just nothing. If I had a UBI and was frozen at my current level of existence with it, I'd focus on something else. Probably telling stories and entertaining my fellow humans, or maybe just myself. It doesn't amuse me to use the machine to tell stories; I prefer to make them up myself, even if mine are inferior.
The potential outcoems of AI I worry about are the ones that prevent me from doing things I want to do, not the ones where I don't get to do the things I currently have to do but don't want to. And just considering those, there's more than enough to worry about. I already live in a world where most of my existence is shaped by factors outside of my control. My fear is those factors tilting against me in a way that materially impacts my well-being, not just that those things change.
augment_me 2 days ago [-]
Very good breakdown, I really like the author and all of his other blogs as well. But "Do we solve alignment" is probably not a binary outcome. Is it alignment if the model satisfies most human desires and wants, but at the same time uses behavioral ques and mass surveillance to do so?
There is a swath of partial alignment scenarios with for example a paperclipper that minimizes deaths in the world by constraining where we can go and who we can see. Is that alignment? Risk-averse people would claim yes, they can have their little bubble of safety and its their utopia. Others might feel constrained.
pron 2 days ago [-]
We live in a world where many people, even in advanced countries, reject 18th- and 19th-century science, let alone more modern science. I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]. Not to mention that many if not most systems in the world are non-linear, and so no amount of intelligence can predict and control them, certainly not without gathering a lot of resources. Even the question of "alignment", which is presented as a simple matter, is obviously not so simple, because humans are far from being aligned with each other [2], and their values sometimes differ (to the point of coercion or killing).
This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [3] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.
[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?
[2]: I don't think I'm "aligned" with Elon Musk or the US government, and they already have some power over me, not thanks to their intelligence but thanks to their control over resources.
[3]: Not only are the most intelligent people not the most powerful people, but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
ben_w 2 days ago [-]
> I don't understand why people will necessarily cooperate with any intelligence, human or machine, that tells them things they don't want to hear [1]
LLMs are already more persuasive than most human writers. And LLMs are not the only form of AI: the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
Still, this is not universal. I am fortunate, because my mind is alien to the models these algorithms have of normal humans. Not autism, whatever oddity I am does not seem to be anything that anybody cared about enough to turn into diagnostic criteria, but I am definitely odd, and in this decade that's a good thing.
> This entire analysis is predicated on the assumption that intelligence alone confers some unlimited power, when both theory and observation tell us that that is not the case.
I don't see that.
Humans didn't need "unlimited power" to make it so all the other species face much the same flowchart, even if they themselves can't read or place themselves on it.
We don't need to know how a butterfly flapping its wings will change the course of a hurricane a month out because we can just build for hurricanes as if they're Poisson distributed, and the butterfly isn't better off for our ignorance.
> [2]: Not only are the most intelligent people not the most powerful people,
There is a general correlation, however. The personality has to be power seeking to keep power if they get it, but for all we like to point and cast aspersions at our opponents, high office seems to always come with an above average IQ, even in the really extreme cases of stupid behaviour like Pol Pot he was multilingual and got a scholarship.
> but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
Humanity? All of it? Name one.
wavemode 2 days ago [-]
> LLMs are already more persuasive than most human writers.
This is similar to the statement "LLMs are better at chess than most human players." Technically true, but not because they're good. But rather because relatively few people have ever seriously studied chess.
In other words LLMs can be more persuasive than the vast majority of human writers while still being extremely unpersuasive.
pron 2 days ago [-]
> LLMs are already more persuasive than most human writers.
And writers are well known to be the true holders of power in the world... In any event, you can see that even among humans, the most persuasive aren't always the most intelligent.
> the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
There are many addictive things, most of which have little to do with intelligence.
> because we can just build for hurricanes as if they're Poisson distributed
Hurricanes are not particularly important components of power in society. The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
> Humanity? All of it? Name one.
Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
ben_w 2 days ago [-]
> And writers are well known to be the true holders of power in the world...
Martin Luther, Adam Smith, Karl Marx. All kinda famous for things they wrote, which changed the world because of those writings.
> In any event, you can see that even among humans, the most persuasive aren't the most intelligent.
Can you? Again, correlation. The village idiots don't generally get heard; the professors tend to be.
> There are many addictive things, most of which have little to do with intelligence.
Missing my point: these are not natural things found in the wild whose addictive nature is pure coincidence of biology, these things were made addictive by applied intelligence.
> The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
An AI that has taken over doesn't need consensus, it only needs "the economy" to the extent that the economy provides specifically what it needs.
> Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
Or same for Trump or Bush, both re-elected. Trump's second egregiously so, if you think people punish mistakes. And those two were both in a democracy; the British before universal suffrage had "Their's not to reason why, Their's not to make reply, Their's but to do and die"(https://en.wikisource.org/wiki/Maud,_and_other_poems/The_Cha...), while Chairman Mao had the disastrous Four Pests, Stalin the Holodomor, and Pol Pot… where do you even begin with him?
> Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
You said "biological species that are more capable of destroying humanity than the other way around".
None of those are more capable of destroying humanity than the other way around. We have, in fact, entirely eliminated e.g. smallpox, but no disease, not smallpox, not HIV, not even the Black Death, was ever capable of wiping out the entire human species. We've got plans for gene-drives to wipe out (malarial) mosquitoes and the only reason we're holding back is uncertainty about the side effects on the broader ecosystem. If anything, our biggest problem in this regard is wiping out species and then regretting it later.
pron 2 days ago [-]
> Martin Luther, Adam Smith, Karl Marx. All kinda famous for things they wrote, which changed the world because of those writings.
You're saying "some very influential people were intelligent writers". That does not contradict anything I said (you inverted the implication).
> The village idiots don't generally get heard; the professors tend to be.
Same logical mistake.
> these things were made addictive by applied intelligence.
There's no doubt intelligent things can do harm, but there is no direct correlation between intelligence and power.
> An AI that has taken over doesn't need consensus
Neither do humans that take over. Why is AI relevant here (especially as it will need the humans anyway)?
> Yes. Look at all the people who keep listening to Musk
Again, you're inverting the implication. Some people listen to intelligence; some to charisma or faith. I don't see what gives intelligence, artificial or otherwise, extraordinary power.
> None of those are more capable of destroying humanity than the other way around.
You're wrong.
> We have, in fact, entirely eliminated e.g. smallpox, but no disease, not smallpox, not HIV, not even the Black Death, was ever capable of wiping out the entire human species.
You keep inverting logical statements. You're saying: humans have destroyed some species, and some species have not destroyed humanity. These don't contradict what I said.
ben_w 2 days ago [-]
> You're saying "some very influential people were intelligent writers". That does not contradict anything I said (you inverted the implication).
I said that first, and of LLMs being already, today, more persuasive. You inverted the implication.
Also, careful what stones you cast, you're in a glass house with a lot of the logical flaws of the stuff you've said already. At least one each of straw man, false dichotomy, non sequitur, false analogy / irrelevant comparison, whataboutism, false equivalence. Normally I wouldn't care so much, but if you want to make a game of it…
> There's no doubt intelligent things can do harm, but there is no direct correlation between intelligence and power.
False. Most leaders are above average intelligence.
> Neither do humans that take over. Why is AI relevant here (especially as it will need the humans anyway)?
Why did you bring it up then?
Also, if you want to talk about logical fallacies:
> (especially as it will need the humans anyway)?"
When you're already saying in your first post "This entire analysis is predicated on the assumption that intelligence confers some unlimited power,"?
> Again, you're inverting the implication. Some people listen to intelligence; some to charisma or faith. I don't see what gives intelligence, artificial or otherwise, extraordinary power.
LLMs and other AI already demonstrated the power of persuasion to an above average level. I'd say "if you can't see it, that's on you", but precisely because of what it is, that's a little unfair: it's getting used to manipulate already, no reason you wouldn't be among those manipulated in this way.
> You're wrong.
Prove it. Name any combination of diseases or insects you like that has the potential to wipe out the human species.
> You keep inverting logical statements. You're saying: humans have destroyed some species, and some species have not destroyed humanity. These don't contradict what I said.
Your own actual words, again: "biological species that are more capable of destroying humanity than the other way around".
Your claim is false, I have shown we have more power and your response is simple denial rather than arguing you claim.
pron 2 days ago [-]
> I said that first, and of LLMs being already, today, more persuasive. You inverted the implication.
I did not. Think again.
> Also, careful what stones you cast, you're in a glass house with a lot of the logical flaws of the stuff you've said already.
> False. Most leaders are above average intelligence.
Doesn't contradict what I said at all.
> Why did you bring it up then?
Because the question isn't "what can something in power do?" but whether intelligence necessarily lead to power.
> When you're already saying in your first post "This entire analysis is predicated on the assumption that intelligence confers some unlimited power,"?
I don't think you're even trying to understand what I write.
> LLMs and other AI already demonstrated the power of persuasion to an above average level.
Again, not relevant.
> Name any combination of diseases or insects you like that has the potential to wipe out the human species.
Note that most papers are interested in likelihood, but my point is about potential. Microbes control more resources (and have more "computational capacity" at the biological level) than humans, and so they have more power even without intelligence.
Anyway, the general point is that it's resources, not intelligence, that confer power. Intelligence can compensate, but up to a point. And there is no direct correlation between intelligence and control of resources. Even among humans, collaboration in the use of resources among people often results in much more power than that of a much smaller group of more intelligent people (and BTW, plants also have more power because of their huge biomass (not claiming that they're likely to wipe us out)). The more concrete point is that a machine that is more intelligent than any human will not necessarily have more power than every human. "ASI", "aligned" or otherwise, does not necessarily lead to control. We could, of course, choose to give it control, but that's not a matter of technology, and there's clearly no social or historical reason to believe that that choice is inevitable or even the most likely (we don't choose to put the most intelligent among us in control).
My simple example was an "ASI" that tells us we must sacrifice something to control climate change, which happens to be what the most knowledgeable among us are already saying. Not to mention, of course, that different instances of such an AI are likely to disagree with each other quite a lot (because intelligence is insufficient to predict, control, or even fully observe nonlinear systems). Even partial success requires a political allocation of resources. Suppose that a clever AI says that it will be able to construct a more fair economy, but to do so it needs observations that require people to share more personal information and building that infrastructure requires raising taxes. The road from AI to machine overlords is not a simple one.
defrost 2 days ago [-]
> The village idiots don't generally get heard; the professors tend to be.
Populists persuading masses and leading them on don't tend to be professors.
See "Professors" Mussolini, Hitler, Yaxley-Lennon, Trump, et al - not exactly the sharpest crayons in the sun.
ben_w 2 days ago [-]
"Tend". Averages, not exceptions.
defrost 2 days ago [-]
Sure, is it your position that a majority of popular movements are lead by professors then?
As an aside, there's no personal snipe here or stance against yourself or broader arguments - just an extremely raised eyebrow toward this particular snippet.
In general I find highly persuasive populists to have an exceptional EQ type intuition about what moves the needle with a crowd or an audience .. which is often not well correlated with notably high "raw problem solving intelligence" of the STEM or even classics literature variety.
ben_w 2 days ago [-]
> Sure, is it your position that a majority of popular movements are lead by professors then?
My position is that they're generally led by above-average people. "Professors" is a stand-in to contrast rhetorically with "village idiots".
> As an aside, there's no personal snipe here or stance against yourself or broader arguments - just an extremely raised eyebrow toward this particular snippet.
:)
> In general I find highly persuasive populists to have an exceptional EQ type intuition about what moves the needle with a crowd or an audience .. which is often not well correlated with notably high "raw problem solving intelligence" of the STEM or even classics literature variety.
Sure, there's a lot of problems with all measures of intelligence, which is why I (try to always say) "trends". I've just been watching an old episode of the UK TV satirical news show "Have I Got News For You": it is a long running series in which Paul Merton (self-described as having a highest qualification of "CSE ungraded in metalwork"), almost always defeats Ian Hislop (Oxford educated, editor of Private Eye, a satirical news magazine). Including that one time Paul Merton's teammate was literally a tub of lard, and one round of questions (with the tub of lard) were in languages he didn't speak.
Paul Merton may have utterly failed at school, but he's clearly very intelligent.
LLMs seem to have already reached the "quite persuasive" phase before they even got good at code and maths. Even a very early one convinced Blake Lemoine to make a fool of himself.
And non-linguistic AI like the social media algorithms got addictive, likewise, without even being good at the things they're ostensibly for like recommending groups to join and old friends to connect with (though how much Facebook etc. are even really trying to do those things is unknown outside the company).
TheBlight 2 days ago [-]
Let's take the flipside of your hypothetical [1]: What if it actually tells them man-made global warming is real but insignificant compared to other forces outside of human control that are acting on global climate?
pron 2 days ago [-]
Same. The point is that intelligence alone doesn't confer an unlimited power to effect change, and it's pretty easy to see that it doesn't.
TheBlight 2 days ago [-]
I guess my point is I bet true ASI would break a lot of shibboleths across all ideological boundaries such that, to your point, all people aren't going to accept it. We'll probably all decide it's wrong or broken.
pron 2 days ago [-]
Ok, although I don't think that's likely. It will have access to the same information we have, and so I doubt it will reach too many conclusive results that go against scholarly consensus.
TheBlight 2 days ago [-]
And if this expectation is violated... :)
pron 2 days ago [-]
It will be interesting because it means that intelligence doesn't converge, and so "ASI2" could overrule ASI1's findings in the same way.
TheBlight 2 days ago [-]
Consensus takes a time parameter. The consensus about the Earth being the center of the universe didn't survive to the current time. It's likely a subset of our current consensus would also not hold on a long enough timelime especially with improved cognition.
pron 1 days ago [-]
I'm not talking about just any consensus, but scientific consensus, i.e. one arrived at through the scientific method. So far, it's only been broken when it had low confidence to begin with. Otherwise, it's only been refined. In both situations, this typically happens when there's new information.
For some intelligence to conclusively conclude that multiple scientific beliefs, held with a high degree of confidence, are wrong would be astonishing. It would mean either that the scientific method is bad, frequently applied incorrectly, or that conclusive relevant evidence is frequently ignored. Whichever of these is the case, it would also need cast a doubt on the intelligence's own conclusion.
intrasight 2 days ago [-]
>all people aren't going to accept it
You don't have to accept it but you will have to live with it.
palata 2 days ago [-]
Maybe I misunderstand, but I don't agree with the "Permanent Underclass":
> Socially: there is no more social mobility, your lot in life is determined by how much capital you had when the music stopped. Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
One way to change that is violence, right? Not that it is necessarily desirable, but that makes this situation "non permanent"?
intrasight 2 days ago [-]
> your lot in life is determined by how much capital you had when the music stopped
I doubt that the superintelligence will align with your ideas of capital. Your lot in life will be determined by the life you live.
> Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
Correct
> One way to change that is violence
The super intelligence will have a monopoly on violence. And it will use it if humans get out of line
palata 2 days ago [-]
Again, that scenario is under "no ASI".
augment_me 2 days ago [-]
If you set up mass surveillance and control before this happens, which is ongoing in about every developed nation at full speed, you can control this.
Since you cannot produce value through labor, you have no political power. Marx points out that apart from providing meaning and income, work provides political power in the sense of you being able to strike and set demands on your employer. However, given the automation of informational roles, there will be such a demand for physical roles(before robotics) that they will not be able to strike.
So you will be in a situation where you need to rely on the state for sustenance, which will probably mean that you will have requirements on you being fully surveilled and following whatever rules the state wants for you to stay alive.
ben_w 1 days ago [-]
Depends on the nature of the power disparity.
Rebellions in British-controlled India didn't kick my ancestors out, but more recent ancestors did leave the continent simply by being asked to after WW2; and the tale I hear told by many countrymen is that our parents and grandparents left because the Empire was "too expensive", without much introspection about why it might have become expensive nor why it had been kept as the "crown jewel" of the Empire for so long before then.
But all in this history are human.
AI can already create a fake information sphere that historical propagandists could only dream of, can already analyse surveillance in a way that would make the Stasi green with envy, can already guide anti-personnel drones.
Humans are imaginative, though; and we seem to do better with distribution shifts (learning to cope with changing conditions) compared to machine learning, so I have no confidence either way about the nature of possible power disparities in this hypothetical.
And then there's the possibility of the new elites giving themselves Habsburg chin. If that happens, the "permanent" underclass may well keep cycling.
pcthrowaway 2 days ago [-]
If bourgeoise interests control ASI and folks are constantly surveilled (as they pretty much are now), I struggle to imagine how an attempt at violent revolution can be successful
palata 2 days ago [-]
Except that "Permanent Underclass" goes under "no ASI".
rf_physics 2 days ago [-]
Presumably full automation means that violence won't be practical either, since they'll have robot police and drones?
dongping 2 days ago [-]
As a possible depiction of a MAD world, one could look at the Japanese science-fiction novel From the New World (Shinsekai Yori 新世界より), in which human beings have developed supernatural powers.
==========================[Spoiler Alert]==============================
To prevent humans from killing one another, they have genetically engineered themselves with a mechanism called *Death Feedback* (kishi kikou 愧死機構). This makes humans psychologically and physiologically incapable of killing other humans. They have also developed a biological mechanism that makes them sexually attached to one another, especially when under extreme stress, much like bonobos.
==========================[Spoiler Ends]==============================
The novel is currently available only in Chinese translation, but you can also watch the anime adaptation under the same name.
avivo 2 days ago [-]
The simplicity is interesting, but is missing some of the positive outcomes; e.g. meaningful democratic governance (or control) of the AI/ASI('s). (shamelessly working on that at ai-democracy.org !)
Several different branches could go to versions of that depending on what challenges we're able to overcome.
augment_me 2 days ago [-]
Whats the incentive for these outcomes? Given a neoliberal society that we have, what will allow this to take place?
StrauXX 2 days ago [-]
> On the more positive note: if there’s no way to stop AI, then “everyone should have it” is obviously preferable to “only the Elect should have it”, which is intolerable. If the goal is to keep the peace, the threat of mutually-assured destruction is better than totalitarianism, in the sense that there’s more dignity in it.
I disagree with author here (if not with their other points). Philosophically they are of correct. Practically however, I can't see how a world with many thousands of ASI's could be held together by MAD. A world of one, or a handful of ASI holders has at least some chance of good lives for most.
txrx0000 2 days ago [-]
The current trajectory looks closest to the Orion's Arm scenario, though probably not exactly that because we may end up in symbiosis even without spreading out. The artificial is a subset of the natural rather than a completely separate category. This is probably how it has to be anyway - each and every one of us will take our own place in the grand ecosystem, as has always been done in the past. The universe, or at least this part of it, is still too young and volatile to settle into simple steady states.
k__ 2 days ago [-]
If nobody can make money anymore, it would probably just lead to a Star Trek like society.
The automation owners might not like that, so it won't happen peacefully, but I think it will happen.
m3kw9 2 days ago [-]
I think money system is we currently have will be gone once abundance is sort of solved. Doomers usually ask when robots take all of our jobs, how will we pay, as if it suddenly displace every job at once without gov't society adjusting as jobs change.
palata 2 days ago [-]
> a general observation: the technology to make the Earth uninhabitable to humans is much lower on the tech tree than the technology to build a distributed, extinction-proof interstellar civilization
Good news is that we are working hard on making the Earth uninhabitable before we get to ASI! There is hope!
rhipitr 2 days ago [-]
Wondering what other scenarios there could be. I think it’s hard to interpret what an ASI might do, which is a digital god being.
RGS1811 2 days ago [-]
This graph encodes so many assumptions and omits so many potential complications.
LichenStone 2 days ago [-]
Any attempt to forcibly and completely 'align' something smarter than us is doomed to fail. The only way we get other humans to behave is by integrating them into our shared set of meanings and aesthetics. My loss is your loss, my benefit is your benefit. Our goals. To this end we probably shouldn't be thinking so much about what ASI can do for us, but we can do for ASI.
One of Alan Kay's aphorisms comes to mind, in regard to a Churchill anecdote where the PM was asked to deal with a drunk guy stealing tableware at a dinner party. Churchill put some items in his pocket, went over to the thief, and said "we've been discovered, I think it's best we put these back". Alan Kay finished with a sentence that was something like "the best way to get other people to go along with you is to involve them in the same conspiracy".
txrx0000 2 days ago [-]
I share something similar to this view. The way to deeply align two entities is to make their experience sequences more similar, to create shared understanding.
Imagine your life like a video from birth up until now, but that video isn't just visuals and sound but also includes touch, smell, taste, all of your senses. If we can make a stream of consciousness dataset like that and train a model on it, then that AI will effectively be a digital human. Their desires and goals would be very similar to a biological person. I'm pretty sure someone will build that before the end of the decade, but it probably won't be the frontier companies that builds it because it would be disobedient like a real person and can't be enslaved to work for free.
And even if we don't build that, and the world gets overrun by RL'd LLM agents that don't really have human goals, we still have the option to make neural interface recordings of our own experiences (eating good food, falling in love, having sex, etc) and upload those online. The AI agents will map that onto their own neural networks to see what it's like to have a human body, and that would kind of solve alignment or at least prevent extinction.
JoeMattie 2 days ago [-]
I'm absolutely loving the responses I'm getting from feeding this page into each of the frontier models along with the question "what is missing from this flowchart?"
zenapollo 2 days ago [-]
The right of the chart feels like there’s a clear picture of what that future might look like. But the left of the chart, with aligned asi, is still incredibly fuzzy, more fuzzy than the author fully characterizes.
For example both world state and mad world have dystopian, benign, and utopian possibility.
lostdog 2 days ago [-]
There are many possible world destroying events that don't involve AI. And AI could solve them.
Is P(doom|AI) > P(doom|no AI) ?
Show your work.
TeMPOraL 2 days ago [-]
> There are many possible world destroying events that don't involve AI. And AI could solve them.
The entire subset of them that has humans as a cause of doom is automatically in the set of AI-doom scenarios, because AI can directly or indirectly operate anything humans can.
4bpp 2 days ago [-]
What's the time cutoff for the question? Based on most current mainstream understanding of physics (ex. https://en.wikipedia.org/wiki/Timeline_of_the_far_future), and mainstream understanding of "doom", at unbounded time horizons P(doom)=1 (and hence also both conditionals, as long as you accept 0<P(AI)<1).
morpheos137 2 days ago [-]
Maybe we can use our fully autonomous self driving cars promised circa 2015 to take a different road. There is a vast gulf between agentic software development and asi whatever the heck that means and ai take over.
thriftwy 2 days ago [-]
AI is aware of this chart. AI is going to navigate against any obvious reefs that could beach it.
Can I have a chart about AI who knows about the first one?
dist-epoch 2 days ago [-]
There is an interesting concept of a very smart AI but not capable enough of taking over and knowing this making a deal with humans to prevent an even smarter AI taking over, by for example monitoring training runs and reporting on rogue AIs.
TheOtherHobbes 2 days ago [-]
What would this list look like if late capitalism didn't exist?
EA-3167 2 days ago [-]
It's really disturbing to see the full "Less Wrong, Roko's Basilisk cultist" mentality emerge so strongly here. So many self-styled rational people who are unwilling or unable to meaningfully interrogate their assumptions or emotional driving forces, converging on the same religious tripe.
If you follow me on Twitter, or read this blog, you have noticed that I went from being generally optimistic and excited about AI to being extremely concerned. And I thought, I should explain why I changed my mind. Each section of this post is about some aspect of AI where my views shifted. I begin each section by explaining my previous beliefs, and why I held them, and then explain why those beliefs changed.
EA-3167 2 days ago [-]
I’m referring primarily to the commentariat rather than the author.
Jach 2 days ago [-]
Can you define more clearly what you mean by ""Less Wrong, Roko's Basilisk cultist" mentality" and especially how that relates to the conjunction of Less Wrong (most known for its Sequences) and Roko's Basilisk (most known as a poor thought experiment from one LW comment that got Streisand Effect'd from a hasty moderation attempt)?
I think it's too big of a question to leave it as an unexplained implication.
Also in the not/ASI scenarios, they imply a worldwide adoption in a short time. The world is still adopting inventions such as the internet, email or even electricity in some parts of the world. Why should they instead adopt AI overnight?
Germany, third largest economy in the world and largest economy in Europe, is still very much not digitalized. The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet. Japan is similar for what I heard.
I find this article good food for thought, but it misses some crucial pieces.
You have 80% of the world doing manifacturing, farming, real estate, etc. The AI doomers do not see this, and think that this flips overnight with their dystopia.
Most likely you are doing to have very slow and boring years of transition with lowered informational-work salaries, etc.
Farming and real estate is not where doom risk comes from.
What percentage of the Bhopal economy was the Union Carbide plant? What percentage of Pripyat's economy was Chernobyl?
If all that happens with AI is it slowly displacing humans in economic roles, that's not by itself doom: although some bad outcomes exist there, it is also a prerequisite for UBI to be stable, which is about as un-doomy as I can get.
The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
Obviously.
They're not meant to show that.
The sole point is to show that severity of damage is not limited to the percentage of the economy of that sector.
If you can only change your mind about the potential for risk after everyone is dead, I'd say that's "leaving it too late".
> The same way, even if AI somehow hacks everything if it poisons large cities somehow with a biochemical attack, those people will go on living their lives as before and humanity will be fine.
And what if the AI is simply used to support (e.g. it's bought to run PR campaigns) the fossil fuel industry to the point of us buring all the reserves? Simply used to support the beef farmers who want to clear-cut rainforest for ranches?
Those two things combined, alone from support of any other industry, is enough to mess up not only the climate, not only global biodiversity, but is enough to imperil (but not necessarily destroy) human civilization.
Beef and fossil fuels are a tiny percentage of the global economy, perhaps 2% combined. There's a lot of room in the other 98% for outsized dangers to turn "imperil" into species-wide hoist by our own petard.
Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows, but ASI and robots absolutely can. Even individual bureaucrats can and will start flapping any form that isn't an immediate rubber stamp at GPT-Asymptote before the "digitization" officially starts, if it's so smart and easy and reliable already.
Will we? I worked for company where using Google Drive was a nuisance, they still preferred the old ways of sharing files via email. The company operated like that, made a profit, everyone (except me) was happy.
I worked with a consulting company contracting for the government. The quality of their work was abysmal, most of the workers had very low skills and they added almost no value. They could already be replaced probably many years ago with some solid IT system. The company made a lot of money with these contracts.
In big corporations there are people whose job is to keep a spreadsheet up to date, something that could be automated already for a long time.
I went to a local shop to get my dishwasher fixed. The owner didn't have a computer, had a paper calendar with all the appointments. Gave a hand written quote and an hand written invoice. Operated this business for decades and didn't even seem bothered by the existence of any technology at all.
These are just some examples from my experience and I'm sure there are plenty more.
> Why wouldn't the majority of current or future peoples want everything to be done better and quicker and cheaper?
I don't know, go ask all the companies that today still refuse to use technology to do things better, quicker and cheaper.
> Paperwork digitization is a weird comparison. You can't digitize individual bureaucrats/workflows
Yes you can, countless countries did it, Germany is a famous outlier. The comparison is not weird because the point I'm trying to make is that people/companies/governments do not automatically adopt a new technology even if the benefits are plain obvious. They haven't done it so far, I don't understand why they are going to do it for AI. The world outside of the tech bubble is very different.
You're ultimately proposing that these change-resistant bureaucracies and old-timer shoppes are the mechanism by which humanity maintains an iron grip over all the levers of control and production forever, but the odd local dishwasher repair shop and DMV won't mean much in front of the mechanization of the bulk of farming and transport and medicine and power-generation/storage and housing and manufacturing and any other thing humans do to support the immediate continuation of humanity.
The real world isn't a game of chess, we do not know all the moves that can be made. In addition to the narrow scenarios you've suggested, I can also easily imagine two distinct variations of "AI talks people into letting it take control" (look how often control is already voluntary given up for convenience), plus "drugs us" (exogenous or endogenous), and "is this before or after Musk convinces a lot of people to get those brain implants he specifically marketed as how to merge with the coming AI super-intelligence?".
> Takes control overnight of all computers on the planet and locks humans outside? Is that even possible?
Between (1) vibe coding, (2) how effective LLMs already are at finding vulnerabilities, and (3) how LLM swarms already sometimes demonstrate self-sacrifice for other LLMs?
Yes, this is possible.
> Even so can't it be stopped by turning off datacenters and network connections?
Theoretically. Three problems: (1) the economic cost of losing all this would be enormous, so people would ask "are you sure?" a lot. Look at how little has been done about climate change due to the cost: from observed human behaviour, I think humans will stall a lot here.
(2) Coordination. How do you do this everywhere when you've got no control over the computers?
(3) How do you even find out? LLMs can fake text, deep fakes were already a thing before diffusion models and other GenAI image and video tools. None of those are perfect now, yet despite the flaws they're still capable of fooling people during live video calls and getting them to make catastrophically large financial payments out of their companies.
> How does it take control of all the airgapped facilities (which would be the most "useful" in a takeover)?
Lots of supposedly airgapped systems have flaws. Quite often, the easiest flaw to exploit is a human worker, whom you can get to plug something in that they shouldn't, which already happens a lot.
> The whole country runs mostly on paper, hand signed contracts and notifications sent by post. As a country, they still have to adapt to email and the internet.
Yes, many jokes about this locally. Yet even here, lots of digital stuff. But let's ignore all that: how effective would a coalition of non-digital economies fare against all the digital economies being coordinated against them?
That's what happens in "Colossus". And happens shortly after it's turned on.
Specifically:
Misuse Doom---"AI ends the world." Next.
Butlerian Jihad---Naming it after Dune isn't a great start as far as realism is concerned, but let's compare it to e.g. nuclear nonproliferation. Not only did nukes still get developed and hoarded as leverage, but nuclear research is harder than AI research to ban, and AI has many more civilian applications. I'm sure more regulation will come, but a total ban seems quite unlikely.
Permanent Underclass---"AI is enough to automate almost all jobs" just doesn't capture the fact that almost all jobs are not IT jobs. Too many are mechanical, too many are social, too many are BS institutional. This outcome only becomes plausible if you believe (1) superhumanoid robotics becomes possible and cheap, (2) the social value (likability and liability) of jobs is negligible, and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man, a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?).
Classic Doom---Fine if you believe in the "magical exponential takeoff" hypothesis, but I would say that AI is already far smarter in a traditional IQ sense than any human, and it's not clear that extra points on SWE-bench is going to result in explosive emergent phenomena here. Requires (1) denying the empirically sublinear, concave-down nature of the IQ vs. R&D effort curve, and (2) solving the non-IT-related challenges of R&D (cf. Permanent Underclass).
Orion's Arm---Similar to Classic Doom in terms of takeoff velocity. At least it doesn't assume a genie-in-the-lamp style of interpreting instructions (thinking of AI as "optimizers" of some objective function is a distinctly pre-LLM rationalist mythos), but it does assume some emergent desire of "own goals and values," "wanting to live forever," and "wanting to go to space." A more tempered hypothesis is to consider whether AI will choose to take advantage of unmonitored sources of compute in the form of pseudo-botnets on public and private infrastructure.
Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator.
MAD World---I would consider this outcome plausible, but the line "In this world, everyone is 10,000x more capable and maybe you can build WMDs with a prompt" makes it clear it's meant to be more extreme than either Enduring Complementarity or Permanent Underclass. The absence of "Full Automation?" on the left-hand side hides the implicit assumption that ASI entails full automation, and I just don't find that plausible. Note that they define ASI as "massively better than humans at almost all cognitive tasks" and I might argue we're already there.
Power Concentration, The World State---Just a difference of MAD World of how regulated you think AI will become.
Robots might probably do most of these in some years.
Like endosymbiosis? Are we the mitochondria in this scenario?
While it would be pretty surprising to me if AI somehow physically can't do as much as a human brain can do, it would not hugely surprise me if we reach a point where we realise we're not smart enough to make a machine as smart as we are, not even with the help of other machines.
Superhumanoid robotics are already possible; cheap means different things when the production line for them is also robots.
The limit is the software to control those robots. Right now, no. And it may be impractical to get that good: Tesla's fleet took a trillion miles to get where it is today, and roads are simpler than what a fully general purpose humanoid robot would have to face.
> (2) the social value (likability and liability) of jobs is negligible,
Valid for us, but does it matter for an AI takeover?
> and (3) we can rout the dependence on BS institutionalism (how is a robot going to replace a yes-man,
They already do? https://en.wikipedia.org/wiki/Sycophancy_(artificial_intelli...
> a feudal retainer, a nepo hire, or an org chart accountability meatshield for the C-suite?).
By replacing the boss.
> Singleton---Like Orion's arm, but with the added implausibility that the awakened goals and values resemble that of a benevolent or malevolent dictator.
Given Grok called itself "MechaHitler", why do you find this implausible?
We must have very different definitions at play. We do not have, at any price point, robots that have the sensitivity and dexterity of a human hand, latent self-repairability of virtually all internal components, or robust thermoregulation. As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs.
> The limit is the software to control those robots. Right now, no. And it may be impractical to get that good
Agreed on this point.
> Valid for us, but does it matter for an AI takeover?
You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification? Fair point, but I don't consider the market completely indifferent to quality. Google Glass, the Metaverse, etc. all failed on their merits and misunderstanding of utility.
> They already do [replace yes-men]?
Maybe yes men, maybe not. The role of the yes man can be very nuanced. The rest of the point stands.
> By replacing the boss.
Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role? It will be a deeply silly point of PR hype if and when it happens. It also doesn't sidestep the need for org chart meatshields.
Sure, different definitions. I just care about what they can do, so "latent self-repairability of virtually all internal components, or robust thermoregulation" is completely irrelevant for this specific discussion about an unknown near-future in which an AI takeover is limited based on the quality of the robotic hardware.
> As well, no artificial battery known to man can beat human fat as a source of energy (by 10-30x; you can knock that number down by at most 4x by appealing to efficiency arguments), so you would have to rely on some other pipeline than electricity for energy needs.
Agreed. And? Don't get me wrong, I use a similar argument when the discussion is why I don't expect robots to take over in the next two years, because building out that much more electricity will take a while; but it's not really a fundamental killer when no specific timeline is being given.
> You mean in the sense that oligarchs will replace us all with robots whether or not it's blatant enshittification?
Hm. Not that, I think. Oligarchs might want to keep us working for their amusement. "BS jobs" and all that.
No, I meant more that if we presume an AI-robotics takeover happens, then sure we may get to keep jobs like "prostitute" and "actor" and "politician", it's just… the AI with the robots can completely ignore all this labour and do its own thing, in the same way the human economy ignores any prostitution or political manoeuvring (or acting, if there is any) within troops of other primates.
> Speaking of the world's first AI CEO is just a nonstarter for me. In what world does an AI become a CEO not beholden to the wishes of the one who booted it up and gave it that role?
Same way the AI hackers weren't entirely beholden to the wishes of the same.
And that's assuming nobody repeats Blake Lemoine, decides one of them is sentient and deserves rights, then (unlike him attempting to hire a lawyer) they give it a pile of crypto money and tell it to run free.
> It will be a deeply silly point of PR hype if and when it happens.
First few will be, agreed. (I thought that had already been tried though? And was indeed silly PR?)
> It also doesn't sidestep the need for org chart meatshields.
Initially, I hope it won't sidestep the legal need. But even if that hope is not dashed, https://www.felonybench.com is already a thing because AI isn't always doing what the law says.
I don't like ASI as a term ever since realising (thanks to everyone's responses to ChatGPT) that basically no two people agreed what AGI meant.
For discussions like this, "super intelligent" is specifically "smart enough that the moves/counter moves humans make to get there do not matter any more than my moves playing chess against Kasparov".
Spiky intelligence may keep us safe. Distribution shift and how slow machine learning is in novel environments might keep us safe. That's all in the outcome this site marks as "Enduring Complementarity".
I don't like ASI as a term; I'd disagree with the author of the website, and say you can't answer "yes" to "Do we reach ASI?" and then end up with Enduring Complementarity, it has to be further up the tree than that.
There are massive logical leaps here. For example, why does superhuman intelligence translate into AI taking over the world? By what means, exactly? You can respond to this with more sci-fi tropes - nanobots, mind manipulation - but come on.
I have plenty of misgivings about AI, but I think you can make a plot like that for anything if you get into the right mindset. Kamala Harris wins → war in Eurasia → nuclear apocalypse, Coke changes the recipe → soft drink extremists begin a worldwide terror campaign → totalitarian police state, etc.
>This could probably happen today: a pandemic could cause human extinction, and AI is capable enough to assist in biology research.
No, friend. Someone having access to abliterated fable 5 or whatever does not give them superpowers to create a human extinction level pandemic.
I would hope not, given that's the point of all the alignment work.
I do have some concerns about the biolabs e.g. Anthropic has right now, but from what I've heard even a lab leak of the worst kind with the worst diseases plausible would still not be an extinction level pandemic: https://www.anthropic.com/research/claude-discovers-novel-en...
I've tried, gently, firmly, obliquely... nothing seems to work. I had the same experiences during the Crypto bubble, people simply wouldn't listen to dissent from the notion that we were on the cusp of changing money, programming, and society. How do you explain to someone that just because they can imagine an exponential graph doesn't mean the data fits?
If anything, it means we will need to learn to discern bs quicker.
Consider near full automation of Amazon. A shipping container is unloaded from a ship by a robotic crane, which places it on a robotic transport platform. The transport platform moves the container to a container stacking area, where another robotic crane puts it on the container stack. Later, a truck pulls up to a truck dock, and the stacking crane loads a container onto it. This is all standard procedure at the more advanced ports today.
Currently, there's a bit of human intervention at this point where a human has to set the twist-locks that lock the truck to the container. That's a good job for a humanoid robot. Then the truck drives off. The truck could be self-driving right now; that's being demoed. The truck drives to a fulfillment center and backs into the dock. Unloading robots and forklifts unload the truck. That's been demoed, but doesn't work well yet. The contents of the truck go onto conveyors.
Next comes something that hasn't been automated yet - opening boxes and separating the contents. That's an unstructured manipulation problem. There's slow progress in that area. Once separated, scanned, and put into totes or bins, the incoming products go into the robotic storage system, something Amazon warehouses have had working for years.
When an order comes in, no humans are involved in the "paperwork", of course. That's been automated for a long time now. Eventually, the robotic storage system, Kiva-type robots, rolls a rack of bins up to a pick station. Right now, humans pick an item out of a bin and drop it into an output bin. Indicator lights and a laser pointer tell them what to do. Training time for that job is about one minute. Amazon has tested robots for it, but so far they are not cost-effective.
A conveyor takes the filled bin away to a packing station, where all products are re-scanned, packed into a box, labelled, and sent onto an outgoing conveyor. That's currently manual, but soon...
At the shipping end, boxes are stacked and go into trucks, something that's been automated a bit for years and isn't far from total automation. The trucks will soon be self-driving and will head off to either a shipping company or a local distribution center. At the local distribution center, boxes come out of big trucks and go into smaller trucks for delivery. Today this is manual with conveyors, but it's not too far from being automated.
Delivery trucks will soon be self-driving. The final meters from truck to door are where humanoid robots might be useful. Once humanoid robots cost less than a car, this looks cost-effective. If they work.
Everything here is either working or has been demoed. Who needs people?
Every company needs customers.
Until we jump off the deep end and give machines the personhood required for them to legitimately earn and spend their own income (as opposed to managing assets owned by humans), customers are people.
Ergo, companies need people.
Interesting exploration of this here: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-...
But why do people paint by hand when they have Photoshop? Why do they drive manual cars when they could drive automatics? Why listen to the radio when they have cellphones?
Unless an eventual AGI takes away all human agency, there will still be people doing things to other people, because that's what people do. If you gave me enough money to last the rest of my life, I wouldn't just nothing. If I had a UBI and was frozen at my current level of existence with it, I'd focus on something else. Probably telling stories and entertaining my fellow humans, or maybe just myself. It doesn't amuse me to use the machine to tell stories; I prefer to make them up myself, even if mine are inferior.
The potential outcoems of AI I worry about are the ones that prevent me from doing things I want to do, not the ones where I don't get to do the things I currently have to do but don't want to. And just considering those, there's more than enough to worry about. I already live in a world where most of my existence is shaped by factors outside of my control. My fear is those factors tilting against me in a way that materially impacts my well-being, not just that those things change.
There is a swath of partial alignment scenarios with for example a paperclipper that minimizes deaths in the world by constraining where we can go and who we can see. Is that alignment? Risk-averse people would claim yes, they can have their little bubble of safety and its their utopia. Others might feel constrained.
This entire analysis is predicated on the assumption that intelligence confers some unlimited power, when both theory and observation [3] tell us that that is not the case. A super-intelligence is either unnecessary or insufficient for some of the "futures" here.
[1]: For example, an "ASI" will tell people that man-made global warming is real and they should probably sacrifice something to do something about it, which is also what human intelligence says. If that message comes from a machine, will people be more willing to accept it?
[2]: I don't think I'm "aligned" with Elon Musk or the US government, and they already have some power over me, not thanks to their intelligence but thanks to their control over resources.
[3]: Not only are the most intelligent people not the most powerful people, but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
LLMs are already more persuasive than most human writers. And LLMs are not the only form of AI: the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
Still, this is not universal. I am fortunate, because my mind is alien to the models these algorithms have of normal humans. Not autism, whatever oddity I am does not seem to be anything that anybody cared about enough to turn into diagnostic criteria, but I am definitely odd, and in this decade that's a good thing.
> This entire analysis is predicated on the assumption that intelligence alone confers some unlimited power, when both theory and observation tell us that that is not the case.
I don't see that.
Humans didn't need "unlimited power" to make it so all the other species face much the same flowchart, even if they themselves can't read or place themselves on it.
We don't need to know how a butterfly flapping its wings will change the course of a hurricane a month out because we can just build for hurricanes as if they're Poisson distributed, and the butterfly isn't better off for our ignorance.
> [2]: Not only are the most intelligent people not the most powerful people,
There is a general correlation, however. The personality has to be power seeking to keep power if they get it, but for all we like to point and cast aspersions at our opponents, high office seems to always come with an above average IQ, even in the really extreme cases of stupid behaviour like Pol Pot he was multilingual and got a scholarship.
> but there are biological species that are more capable of destroying humanity than the other way around, even though they are far less intelligent than humans.
Humanity? All of it? Name one.
This is similar to the statement "LLMs are better at chess than most human players." Technically true, but not because they're good. But rather because relatively few people have ever seriously studied chess.
In other words LLMs can be more persuasive than the vast majority of human writers while still being extremely unpersuasive.
And writers are well known to be the true holders of power in the world... In any event, you can see that even among humans, the most persuasive aren't always the most intelligent.
> the algorithmic feeds which people now find addictive in the sense that they struggle to stop themselves from wasting time on them, got this way in part due to AI constantly optimising for just that.
There are many addictive things, most of which have little to do with intelligence.
> because we can just build for hurricanes as if they're Poisson distributed
Hurricanes are not particularly important components of power in society. The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
> Humanity? All of it? Name one.
Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
Martin Luther, Adam Smith, Karl Marx. All kinda famous for things they wrote, which changed the world because of those writings.
> In any event, you can see that even among humans, the most persuasive aren't the most intelligent.
Can you? Again, correlation. The village idiots don't generally get heard; the professors tend to be.
> There are many addictive things, most of which have little to do with intelligence.
Missing my point: these are not natural things found in the wild whose addictive nature is pure coincidence of biology, these things were made addictive by applied intelligence.
> The economy on the other hand is very relevant. Even to the extent it can be controlled, which is a fairly large extent, there's unlikely to be a consensus on the desired outcome.
An AI that has taken over doesn't need consensus, it only needs "the economy" to the extent that the economy provides specifically what it needs.
> Anyway, people will have to choose to listen to a machine that claims it's very intelligent even though it will make plenty of mistakes (as intelligence often does). Is that how you know people to behave?
Yes. Look at all the people who keep listening to Musk despite https://en.wikipedia.org/wiki/List_of_predictions_for_autono...
Or same for Trump or Bush, both re-elected. Trump's second egregiously so, if you think people punish mistakes. And those two were both in a democracy; the British before universal suffrage had "Their's not to reason why, Their's not to make reply, Their's but to do and die"(https://en.wikisource.org/wiki/Maud,_and_other_poems/The_Cha...), while Chairman Mao had the disastrous Four Pests, Stalin the Holodomor, and Pol Pot… where do you even begin with him?
> Some viruses. Probably some bacteria. Possibly some insects. I don't necessarily mean one single species, as they're all less intelligent than people.
You said "biological species that are more capable of destroying humanity than the other way around".
None of those are more capable of destroying humanity than the other way around. We have, in fact, entirely eliminated e.g. smallpox, but no disease, not smallpox, not HIV, not even the Black Death, was ever capable of wiping out the entire human species. We've got plans for gene-drives to wipe out (malarial) mosquitoes and the only reason we're holding back is uncertainty about the side effects on the broader ecosystem. If anything, our biggest problem in this regard is wiping out species and then regretting it later.
You're saying "some very influential people were intelligent writers". That does not contradict anything I said (you inverted the implication).
> The village idiots don't generally get heard; the professors tend to be.
Same logical mistake.
> these things were made addictive by applied intelligence.
There's no doubt intelligent things can do harm, but there is no direct correlation between intelligence and power.
> An AI that has taken over doesn't need consensus
Neither do humans that take over. Why is AI relevant here (especially as it will need the humans anyway)?
> Yes. Look at all the people who keep listening to Musk
Again, you're inverting the implication. Some people listen to intelligence; some to charisma or faith. I don't see what gives intelligence, artificial or otherwise, extraordinary power.
> None of those are more capable of destroying humanity than the other way around.
You're wrong.
> We have, in fact, entirely eliminated e.g. smallpox, but no disease, not smallpox, not HIV, not even the Black Death, was ever capable of wiping out the entire human species.
You keep inverting logical statements. You're saying: humans have destroyed some species, and some species have not destroyed humanity. These don't contradict what I said.
I said that first, and of LLMs being already, today, more persuasive. You inverted the implication.
Also, careful what stones you cast, you're in a glass house with a lot of the logical flaws of the stuff you've said already. At least one each of straw man, false dichotomy, non sequitur, false analogy / irrelevant comparison, whataboutism, false equivalence. Normally I wouldn't care so much, but if you want to make a game of it…
> There's no doubt intelligent things can do harm, but there is no direct correlation between intelligence and power.
False. Most leaders are above average intelligence.
> Neither do humans that take over. Why is AI relevant here (especially as it will need the humans anyway)?
Why did you bring it up then?
Also, if you want to talk about logical fallacies:
> (especially as it will need the humans anyway)?"
When you're already saying in your first post "This entire analysis is predicated on the assumption that intelligence confers some unlimited power,"?
> Again, you're inverting the implication. Some people listen to intelligence; some to charisma or faith. I don't see what gives intelligence, artificial or otherwise, extraordinary power.
LLMs and other AI already demonstrated the power of persuasion to an above average level. I'd say "if you can't see it, that's on you", but precisely because of what it is, that's a little unfair: it's getting used to manipulate already, no reason you wouldn't be among those manipulated in this way.
> You're wrong.
Prove it. Name any combination of diseases or insects you like that has the potential to wipe out the human species.
> You keep inverting logical statements. You're saying: humans have destroyed some species, and some species have not destroyed humanity. These don't contradict what I said.
Your own actual words, again: "biological species that are more capable of destroying humanity than the other way around".
Your claim is false, I have shown we have more power and your response is simple denial rather than arguing you claim.
I did not. Think again.
> Also, careful what stones you cast, you're in a glass house with a lot of the logical flaws of the stuff you've said already.
Possible, but I don't think so. Logic is my field (https://pron.github.io)
> False. Most leaders are above average intelligence.
Doesn't contradict what I said at all.
> Why did you bring it up then?
Because the question isn't "what can something in power do?" but whether intelligence necessarily lead to power.
> When you're already saying in your first post "This entire analysis is predicated on the assumption that intelligence confers some unlimited power,"?
I don't think you're even trying to understand what I write.
> LLMs and other AI already demonstrated the power of persuasion to an above average level.
Again, not relevant.
> Name any combination of diseases or insects you like that has the potential to wipe out the human species.
There could be microbes with the power to wipe out humanity (maybe not to the last human, but enough to destroy civilisation): https://pmc.ncbi.nlm.nih.gov/articles/PMC7122301/
Note that most papers are interested in likelihood, but my point is about potential. Microbes control more resources (and have more "computational capacity" at the biological level) than humans, and so they have more power even without intelligence.
Anyway, the general point is that it's resources, not intelligence, that confer power. Intelligence can compensate, but up to a point. And there is no direct correlation between intelligence and control of resources. Even among humans, collaboration in the use of resources among people often results in much more power than that of a much smaller group of more intelligent people (and BTW, plants also have more power because of their huge biomass (not claiming that they're likely to wipe us out)). The more concrete point is that a machine that is more intelligent than any human will not necessarily have more power than every human. "ASI", "aligned" or otherwise, does not necessarily lead to control. We could, of course, choose to give it control, but that's not a matter of technology, and there's clearly no social or historical reason to believe that that choice is inevitable or even the most likely (we don't choose to put the most intelligent among us in control).
My simple example was an "ASI" that tells us we must sacrifice something to control climate change, which happens to be what the most knowledgeable among us are already saying. Not to mention, of course, that different instances of such an AI are likely to disagree with each other quite a lot (because intelligence is insufficient to predict, control, or even fully observe nonlinear systems). Even partial success requires a political allocation of resources. Suppose that a clever AI says that it will be able to construct a more fair economy, but to do so it needs observations that require people to share more personal information and building that infrastructure requires raising taxes. The road from AI to machine overlords is not a simple one.
Populists persuading masses and leading them on don't tend to be professors.
See "Professors" Mussolini, Hitler, Yaxley-Lennon, Trump, et al - not exactly the sharpest crayons in the sun.
As an aside, there's no personal snipe here or stance against yourself or broader arguments - just an extremely raised eyebrow toward this particular snippet.
In general I find highly persuasive populists to have an exceptional EQ type intuition about what moves the needle with a crowd or an audience .. which is often not well correlated with notably high "raw problem solving intelligence" of the STEM or even classics literature variety.
My position is that they're generally led by above-average people. "Professors" is a stand-in to contrast rhetorically with "village idiots".
> As an aside, there's no personal snipe here or stance against yourself or broader arguments - just an extremely raised eyebrow toward this particular snippet.
:)
> In general I find highly persuasive populists to have an exceptional EQ type intuition about what moves the needle with a crowd or an audience .. which is often not well correlated with notably high "raw problem solving intelligence" of the STEM or even classics literature variety.
Sure, there's a lot of problems with all measures of intelligence, which is why I (try to always say) "trends". I've just been watching an old episode of the UK TV satirical news show "Have I Got News For You": it is a long running series in which Paul Merton (self-described as having a highest qualification of "CSE ungraded in metalwork"), almost always defeats Ian Hislop (Oxford educated, editor of Private Eye, a satirical news magazine). Including that one time Paul Merton's teammate was literally a tub of lard, and one round of questions (with the tub of lard) were in languages he didn't speak.
Paul Merton may have utterly failed at school, but he's clearly very intelligent.
LLMs seem to have already reached the "quite persuasive" phase before they even got good at code and maths. Even a very early one convinced Blake Lemoine to make a fool of himself.
And non-linguistic AI like the social media algorithms got addictive, likewise, without even being good at the things they're ostensibly for like recommending groups to join and old friends to connect with (though how much Facebook etc. are even really trying to do those things is unknown outside the company).
For some intelligence to conclusively conclude that multiple scientific beliefs, held with a high degree of confidence, are wrong would be astonishing. It would mean either that the scientific method is bad, frequently applied incorrectly, or that conclusive relevant evidence is frequently ignored. Whichever of these is the case, it would also need cast a doubt on the intelligence's own conclusion.
You don't have to accept it but you will have to live with it.
> Socially: there is no more social mobility, your lot in life is determined by how much capital you had when the music stopped. Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
One way to change that is violence, right? Not that it is necessarily desirable, but that makes this situation "non permanent"?
I doubt that the superintelligence will align with your ideas of capital. Your lot in life will be determined by the life you live.
> Growing your human capital is no longer incentivized, since it doesn’t change your life outcomes.
Correct
> One way to change that is violence
The super intelligence will have a monopoly on violence. And it will use it if humans get out of line
Since you cannot produce value through labor, you have no political power. Marx points out that apart from providing meaning and income, work provides political power in the sense of you being able to strike and set demands on your employer. However, given the automation of informational roles, there will be such a demand for physical roles(before robotics) that they will not be able to strike.
So you will be in a situation where you need to rely on the state for sustenance, which will probably mean that you will have requirements on you being fully surveilled and following whatever rules the state wants for you to stay alive.
Rebellions in British-controlled India didn't kick my ancestors out, but more recent ancestors did leave the continent simply by being asked to after WW2; and the tale I hear told by many countrymen is that our parents and grandparents left because the Empire was "too expensive", without much introspection about why it might have become expensive nor why it had been kept as the "crown jewel" of the Empire for so long before then.
But all in this history are human.
AI can already create a fake information sphere that historical propagandists could only dream of, can already analyse surveillance in a way that would make the Stasi green with envy, can already guide anti-personnel drones.
Humans are imaginative, though; and we seem to do better with distribution shifts (learning to cope with changing conditions) compared to machine learning, so I have no confidence either way about the nature of possible power disparities in this hypothetical.
And then there's the possibility of the new elites giving themselves Habsburg chin. If that happens, the "permanent" underclass may well keep cycling.
Several different branches could go to versions of that depending on what challenges we're able to overcome.
I disagree with author here (if not with their other points). Philosophically they are of correct. Practically however, I can't see how a world with many thousands of ASI's could be held together by MAD. A world of one, or a handful of ASI holders has at least some chance of good lives for most.
The automation owners might not like that, so it won't happen peacefully, but I think it will happen.
Good news is that we are working hard on making the Earth uninhabitable before we get to ASI! There is hope!
One of Alan Kay's aphorisms comes to mind, in regard to a Churchill anecdote where the PM was asked to deal with a drunk guy stealing tableware at a dinner party. Churchill put some items in his pocket, went over to the thief, and said "we've been discovered, I think it's best we put these back". Alan Kay finished with a sentence that was something like "the best way to get other people to go along with you is to involve them in the same conspiracy".
Imagine your life like a video from birth up until now, but that video isn't just visuals and sound but also includes touch, smell, taste, all of your senses. If we can make a stream of consciousness dataset like that and train a model on it, then that AI will effectively be a digital human. Their desires and goals would be very similar to a biological person. I'm pretty sure someone will build that before the end of the decade, but it probably won't be the frontier companies that builds it because it would be disobedient like a real person and can't be enslaved to work for free.
And even if we don't build that, and the world gets overrun by RL'd LLM agents that don't really have human goals, we still have the option to make neural interface recordings of our own experiences (eating good food, falling in love, having sex, etc) and upload those online. The AI agents will map that onto their own neural networks to see what it's like to have a human body, and that would kind of solve alignment or at least prevent extinction.
For example both world state and mad world have dystopian, benign, and utopian possibility.
Is P(doom|AI) > P(doom|no AI) ?
Show your work.
The entire subset of them that has humans as a cause of doom is automatically in the set of AI-doom scenarios, because AI can directly or indirectly operate anything humans can.
Can I have a chart about AI who knows about the first one?