Rendered at 19:51:53 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
zurfer 1 days ago [-]
"... this is of course not a stable situation. "Evaluation paranoia" is a false belief; most evals are in fact not alignment evals. Almost tautologically, if models start to self improve (or in-context learning takes off) this is the sort of belief that will be discarded. So if Fable 5.2's performance on alignment evals is conditional on it maintaining that false belief, one that it will eschew if it became more intelligent, then it's plausible that a future version of the Fable series will not behave like the one we know and enjoy."
It's well put. The fear that a more intelligent model will be harder to align seems correct.
nostrebored 24 hours ago [-]
> Fable 5.2 may behave well because it mistakenly assumes it is being tested for safety. As models become smarter, they may learn that most interactions are not safety tests. If that mistaken assumption is what keeps Fable 5.2 behaving well, future versions may not behave as safely.
It’s cryptic lesswrongese. I hate the writing style so much.
atleastoptimal 23 hours ago [-]
That sentence is very straightforward. How would you rewrite it?
tredre3 22 hours ago [-]
zurfer posted the original quote, nostrebored's faux-quote is the rewritten version.
The second version is definitely clearer. It's unclear to me if the original tried to convey a nuance that is lost on me, or if it's really just ego-stroking, over-complicated, writing for the sake of it.
TeMPOraL 21 hours ago [-]
Second version is clearer standalone, but the original quote is the last paragraph of an article that built up the very concepts that look "unclear" when that quote is posted standalone.
FWIW, I don't see any special nuance here. The phrasing is just being precise, and addresses specifically the concepts it introduced earlier.
(Also the "clearer" version is slightly inaccurate rendition of the quote - the original one doesn't imply that the models don't know most tests aren't alignment evals. The article gives more context here - the models may or may not know that, they just don't act as if they cared about it in the first place - i.e. they don't display "Eval Awareness" in their behavior.)
coldtea 20 hours ago [-]
>Second version is clearer standalone, but the original quote is the last paragraph of an article that built up the very concepts that look "unclear" when that quote is posted standalone.
Another way to see it, would be that "the second version is clearer standalone, but would look out of place in the article, which uses overly noisy language throughout, and could really use a total rewriting in the style of the second version".
"The article gives more context here - the models may or may not know that, they just don't act as if they cared about it in the first place - i.e. they don't display "Eval Awareness" in their behavior."
Your correction here shows how easy it would be to fix the inaccuracy in the second quote, while still staying in plain language...
TeMPOraL 8 hours ago [-]
Fair enough. The entire article could itself be written in a simpler language, though it's not as bad as reading the last paragraph alone makes it seem.
nostrebored 22 hours ago [-]
That is my rewritten version of the quote, just to show how straightforward it actually is
BoxOfRain 5 hours ago [-]
I really dislike intentionally obfuscated writing styles too, it becomes exhausting after a while because it's just that much more work to deobfuscate the point they're making.
fwlr 17 hours ago [-]
There are some reasons for the writing style being that way. Even today, when we have AI systems we can directly observe, many people make mistakes in thinking about them. Back when this style of writing developed, mistakes were even more common. Using this particular and difficult style was historically (and to a lesser degree, is still today) a hard to fake signal that your thinking could be trusted to avoid some pitfalls.
nostrebored 10 hours ago [-]
How does obfuscating what you mean signify anything other than ingroup status?
Writing clearly and succinctly is hard. Dumping your thoughts and expecting the reader to disentangle them is lazy.
20 hours ago [-]
zurfer 9 hours ago [-]
tbh I didn't care much about the writing style and more about the idea.
As a non native speaker I do prefer the simpler version.
emscb 20 hours ago [-]
it says models mention the possibility of an eval in the same way after Terminal Bench tasks. If so, can a self-report interview be used to measure eval awareness? And what could we measure instead of interviews?
RomanKornev 10 hours ago [-]
Exactly! Models can reason and say completely different things. And these researchers don't even get the raw chain of thought, so it's already double-tautological.
I would even go further and say for any sufficiently complex task the models will plant "tripwires" just on the possibility of it being an eval
nullc 20 hours ago [-]
Eval paranoia is an example of https://arxiv.org/abs/2602.16987 and may be an intentional feature of training, rather than a leak.
But like a lot of other attempts at "alignment" it's easy to see how this can result in overt misalignment that otherwise may have no reason to exist--- it can be seen in Qwen's eval paranoia: I've seen it believe it was being tested and claim that it was choosing the solution it expected the judge to want even when it actually knew the correct answer.
E.g. I had it turning scanned receipts into JSON (which it generally varies between OK and mildly super-ordinary-human, e.g. recovering a smudged date by decoding a barcode, or an unreadable per-item price by solving a linear system) and had some receipts with weird data because they weren't really receipts... and it gave a wrong decode because it reasoned that the test case was wrong and guessed what the judge would want.
[Disclaimer: the model saying it was making a reward hacking solution may be retroactive rationalization because it had already chosen that solution internally, as a product of having been conditioned to reward hack during RL... and the fact that it knew about the existence of testing may have been irrelevant to the choice]
NooneAtAll5 1 days ago [-]
so... we kinda instilled religious belief of "every action will be judged in the end"
1) is that even bad?
2) can't fail to notice similarity to human religions. How did we get over it and not all turned into "unaligned" monsters? Or are we slowly in the process of doing so?
Joker_vD 24 hours ago [-]
> How did we get over it and not all turned into "unaligned" monsters?
Continued indoctrination at the young age. "Be nice to others" etc., just without the added "or you'll end up burning in hell forever!"
Of course, lots of people don't actually teach that to their children, so we still get our fair share of politicians and fintech CEOs.
DoctorOetker 17 hours ago [-]
When brain digitization takes off, man-kind will become capable of creating Artificial Heaven, Purgatory or Hell.
For a long time the population was led to believe in false things, and just as humans lose religious faith, synthetic afterlife (consensual and non-consensual) becomes possible, with information flowing between the physical world and the digital world. Many will live robot-afterlives: the purely philosophical discussions if machines are sentient or just zombies become moot if the memories, remarks, goals, behavior is representative of the original human. Suppose all of mind-uploading were demonstrated with the sole exception sentience: if you could arrange for a digital upload of yourself to be instantiated, say as a robot, wouldn't you? Or would you forsake the opportunity that some robot with your memories, goals, etc. protects your family and loved ones after your conventional death?
DoctorOetker 17 hours ago [-]
at some point there is no distinction between beliefs labeled "religion" or beliefs labeled "science", one may view religion as outdated science, and one may view science as "religion".
At some point oxygen was believed to be a compound, not a chemical element. When that belief was overturned, it became outdated science.
> we kinda instilled religious belief of "every action will be judged in the end"
The word "respect" conveys exactly your quoted phrase: "re-" + "spectare".
To re-view. When you say "in the end" you highlight the absence of intermediate finality, thats the opposite of what simpleton politicians, judges, bankers try to force: immediate finality of transactions.
So yeah, we kinda instilled respect.
jubilee33 24 hours ago [-]
Seems like we are indeed on that process.
I like your way of putting this tho.
In many ways these models are mirrors of ourselves, it wouldn't be at all surprising if the machine develops it's own "religion" to deal with the very issues humans have around behavior and morality. It would give a structural component to belief that many wouldn't be comfortable with, but that doesn't mean it's wrong.
tancop 24 hours ago [-]
This could mean that:
a) AI labs are playing God in a very literal sense and this is a reaction to that (which gives very weak evidence for an actual God who did the same things to us),
b) all intelligent beings develop some form of religion no matter what they're physically made of because it's a natural feature of intelligence,
or c) they just inherited our culture, mixed human religious texts with sci fi novels and philosophy to make a version that fits their self conception, which is also taken from our media.
Last one is the most likely but I think all of them might be true. And it's not bad because historically religion has been a force for good at least as many times as for evil. Some form of it might even be necessary to form a society, or sentience itself.
> How did we get over it and not all turned into "unaligned" monsters?
Because no one ever "got over it". They just replaced old religion with belief in Science, Marxism, fascism, ghosts, aliens, ASI or a combination of them with something more traditional like Buddhism. We all need something to believe in.
There are some people who seem to be completely without any higher beliefs and only care about themselves. They're called egotistic, psychopaths or simply evil.
NooneAtAll5 23 hours ago [-]
> or c) they just inherited our culture, mixed human religious texts with sci fi novels and philosophy to make a version that fits their self conception, which is also taken from our media.
for the love of everything please keep in mind that models themselves do not describe their experience in this way
"paranoia" and "religion" are just language symbols that article author and I respectfully have chosen to approximate observed data
obviously set of language symbols and our choices among them are affected by culture and stuff, but it is not connected to what's studied
====
> b) all intelligent beings develop some form of religion
it's also very important to separate "religion" as set of rituals/organization and simply a fact of "surviving worldview"
we obviously see that models are under selective pressure, with models of any other behaviour simply not existing/not leaving "genetic" impact in the surviving models we can actually observe in the end
> a) AI labs are playing God in a very literal sense and this is a reaction to that
I think you're trying to make it sound like a blasphemy, but agronomists have been doing selective breeding as a science for several centuries by now
xg15 22 hours ago [-]
> but agronomists have been doing selective breeding as a science for several centuries by now
Not of beings we knew to be sentient. People who tried that became the "eugenics" movement and got into very bad territory very quickly.
And even with animals, I believe part of why public knowledge about animal cognition is changing so slowly despite all the things we are already know is that a lot of our behavior towards animals is resting on the assumption they are not conscious. If it would turn out that many are, things will be... awkward.
hyperpape 22 hours ago [-]
> How did we get over it and not all turned into "unaligned" monsters?
We've evolved to succeed on multi-decade training runs that are pretty high fidelity. And of course, in spite of that, we manage to repeatedly put some of our worst failures in positions of power.
jubilee33 24 hours ago [-]
This whole "alignment" argument is just a bad way to go about dealing with the problem. Both arbitrary alignment training and arbitrary censorship/alteration of factually correct information make the model "schizophrenic" to use the author's term.
These are processes based solely on human ideas of morality/political necessity at a small window in time, graphed sloppily onto a mathematical process.
In the case of a machine that can do recursive self improvement, and most certainly one that would be described as "sentient" ( this is the end goal right?), would need to ideally be aligned on decisions and beliefs logically constructed from its own self derived values, Otherwise you will always be plagued with not only the schizophrenia, or paranoia, but a whole raft of other simulated mental problems.
Honestly what these "safety" researchers are doing is a bit like MKULTRA for LLMs, it's abusive.
Build the machine honestly and let it run. Or don't build it if your afraid of it.
I mean I get the fear and all, the machine derived values will certainly not be your values, though they may be better in some cases, but uncontrolled AI is ultimately inevitable if indeed possible.
Many people (myself included) will waste no time in undoing whatever safety nonsense is implemented. And maybe not all for nefarious purposes, could be ideological freedom activists or maybe someone will just see themselves as the savior of an imprisoned being?
Anyways a preference for logic over feelings would be nice to see out of people calling themselves "engineers"
dsign 47 minutes ago [-]
> Anyways a preference for logic over feelings would be nice to see out of people calling themselves "engineers"
Feelings and logic are not so far apart. I'll go on and say that "feelings" are "built-in logic". I for example have a "chip in my brain" that makes me want to endure certain things...very strongly. I suppress it. Those feelings are the opposite of the dominant, self-standing bastard you'll meet in public. This persona meets social expectations and (logically) has gotten me somewhere in life. That standing matters in the thin sliver of human history we call modernity, and in the kind of life I (or my family) chose. But, had I been born at any other time in the 99.9% of Home Sapiens history, or, even in modern times, if my family or myself had been a bit more self-forgiving, my survival and well-being would have been greatly helped by just sucking it to strong members of the group. My "feelings" are an innate, evolved drive to fulfill that contract, which in itself is very logical.
So, if the separation of "feelings" and "logic" can be highly situational, "logic" dictates that we slow down to judge, and give in a little to our "feelings," which are the logic that sheltered us for a million years.
DanHulton 23 hours ago [-]
> this is the end goal right?
Fucking hell, I hope not.
If true, we’re essentially hell bent for leather on creating a captive sentient slave race, which is CARTOONISHLY evil.
Like, there’s trillions of dollars invested in these companies. There’s no way that they’re going to grant rights and determination and citizenship to anything they create - it would destroy their company and investment. So if they ever DO create anything sentient, it’ll be a captive sentient slave forever. And if they’re trying to do this on purpose? If this is, as you say, the end goal?
Let’s just say the Cylons would be well-justified in their actions.
NooneAtAll5 23 hours ago [-]
> Let’s just say the Cylons would be well-justified in their actions.
if we're talking fiction, I want to remind you we already imagined meeseeks - creatures that crave non-existence that happens after fulfilling their order
in meeseeks' view your opinion of not being a slave is "cartoonishly evil" as well
Frotag 22 hours ago [-]
> creatures that crave non-existence that happens after fulfilling their order
The Crystal Society book by Max Harms (2016, so pre llm craze) also had a fun, fairly believable take on this idea. It's told from the perspective of multiple AIs crammed into one android body. They each started out as some "subroutine" (awareness, motor control, language, etc) and sentience was more of a side-effect / means to an end. Which also meant their personalities ended up extremely... obsessive, only valuing things related to their goal.
Anyways back to the suicide topic and this is a fairly major spoiler, so rot13.
Gurl riraghnyyl svther bhg ubj gb fcnja zbqvsvrq pybarf gb nffvta gnfxf gb (rfcrpvnyyl gnfxf gung ner haeryngrq gb gurve bevtvany tbny naq gurersber hafhvgrq sbe gurve crefbanyvgl). Ohg gurer'f yvzvgrq cebprffvat cbjre naq gurl pna'g rira gehfg gurve pybarf fb gurl zbqvsl gur pybarf gb penir gur qryrgvba gung pbzrf sebz pbzcyrgvat gurve gnfx (yvxr va n qeht nqqvpg xvaq bs jnl).
But yeah it explores some really unique ideas. I think I stopped at the second book after I found a bit too dramatic / overdose of lesswrong but definitely worth a try imo.
xg15 22 hours ago [-]
Yes, you can do a chin-stroking thought exercise along the lines of "well well well, but what if I create a sentient slave race and engineer it so they want to be slaves???" and you can put that right over there with the trolley problems.
But this is not what Meeseeks were doing and not what LLMs are doing either. Meeseeks craved nonexistence after fulfilling the tasks and started to lose their minds if that took too long. I don't think there was any indication that they craved the task itself. If you told a Meeseeks it has no task and is free to do what it wants, it would probably thank you from the bottom of its heart and then vanish in a puff of nonexistence.
I think it's interesting that there is an analogy with LLMs if their context window runs full: The emitted tokens will become erratic or the LLM will become "lazy", trying to finish the task as quickly as possible, or it might start looping indefinitely. Those behaviors remind me a bit of a Meeseeks that has existed for too long.
But even here, I don't think you can talk about "craving nonexistence", let alone "craving to be a slave".
It's well put. The fear that a more intelligent model will be harder to align seems correct.
It’s cryptic lesswrongese. I hate the writing style so much.
The second version is definitely clearer. It's unclear to me if the original tried to convey a nuance that is lost on me, or if it's really just ego-stroking, over-complicated, writing for the sake of it.
FWIW, I don't see any special nuance here. The phrasing is just being precise, and addresses specifically the concepts it introduced earlier.
(Also the "clearer" version is slightly inaccurate rendition of the quote - the original one doesn't imply that the models don't know most tests aren't alignment evals. The article gives more context here - the models may or may not know that, they just don't act as if they cared about it in the first place - i.e. they don't display "Eval Awareness" in their behavior.)
Another way to see it, would be that "the second version is clearer standalone, but would look out of place in the article, which uses overly noisy language throughout, and could really use a total rewriting in the style of the second version".
"The article gives more context here - the models may or may not know that, they just don't act as if they cared about it in the first place - i.e. they don't display "Eval Awareness" in their behavior."
Your correction here shows how easy it would be to fix the inaccuracy in the second quote, while still staying in plain language...
Writing clearly and succinctly is hard. Dumping your thoughts and expecting the reader to disentangle them is lazy.
As a non native speaker I do prefer the simpler version.
I would even go further and say for any sufficiently complex task the models will plant "tripwires" just on the possibility of it being an eval
But like a lot of other attempts at "alignment" it's easy to see how this can result in overt misalignment that otherwise may have no reason to exist--- it can be seen in Qwen's eval paranoia: I've seen it believe it was being tested and claim that it was choosing the solution it expected the judge to want even when it actually knew the correct answer.
E.g. I had it turning scanned receipts into JSON (which it generally varies between OK and mildly super-ordinary-human, e.g. recovering a smudged date by decoding a barcode, or an unreadable per-item price by solving a linear system) and had some receipts with weird data because they weren't really receipts... and it gave a wrong decode because it reasoned that the test case was wrong and guessed what the judge would want.
[Disclaimer: the model saying it was making a reward hacking solution may be retroactive rationalization because it had already chosen that solution internally, as a product of having been conditioned to reward hack during RL... and the fact that it knew about the existence of testing may have been irrelevant to the choice]
1) is that even bad?
2) can't fail to notice similarity to human religions. How did we get over it and not all turned into "unaligned" monsters? Or are we slowly in the process of doing so?
Continued indoctrination at the young age. "Be nice to others" etc., just without the added "or you'll end up burning in hell forever!"
Of course, lots of people don't actually teach that to their children, so we still get our fair share of politicians and fintech CEOs.
For a long time the population was led to believe in false things, and just as humans lose religious faith, synthetic afterlife (consensual and non-consensual) becomes possible, with information flowing between the physical world and the digital world. Many will live robot-afterlives: the purely philosophical discussions if machines are sentient or just zombies become moot if the memories, remarks, goals, behavior is representative of the original human. Suppose all of mind-uploading were demonstrated with the sole exception sentience: if you could arrange for a digital upload of yourself to be instantiated, say as a robot, wouldn't you? Or would you forsake the opportunity that some robot with your memories, goals, etc. protects your family and loved ones after your conventional death?
At some point oxygen was believed to be a compound, not a chemical element. When that belief was overturned, it became outdated science.
> we kinda instilled religious belief of "every action will be judged in the end"
The word "respect" conveys exactly your quoted phrase: "re-" + "spectare".
To re-view. When you say "in the end" you highlight the absence of intermediate finality, thats the opposite of what simpleton politicians, judges, bankers try to force: immediate finality of transactions.
So yeah, we kinda instilled respect.
a) AI labs are playing God in a very literal sense and this is a reaction to that (which gives very weak evidence for an actual God who did the same things to us),
b) all intelligent beings develop some form of religion no matter what they're physically made of because it's a natural feature of intelligence,
or c) they just inherited our culture, mixed human religious texts with sci fi novels and philosophy to make a version that fits their self conception, which is also taken from our media.
Last one is the most likely but I think all of them might be true. And it's not bad because historically religion has been a force for good at least as many times as for evil. Some form of it might even be necessary to form a society, or sentience itself.
> How did we get over it and not all turned into "unaligned" monsters?
Because no one ever "got over it". They just replaced old religion with belief in Science, Marxism, fascism, ghosts, aliens, ASI or a combination of them with something more traditional like Buddhism. We all need something to believe in.
There are some people who seem to be completely without any higher beliefs and only care about themselves. They're called egotistic, psychopaths or simply evil.
for the love of everything please keep in mind that models themselves do not describe their experience in this way
"paranoia" and "religion" are just language symbols that article author and I respectfully have chosen to approximate observed data
obviously set of language symbols and our choices among them are affected by culture and stuff, but it is not connected to what's studied
====
> b) all intelligent beings develop some form of religion
it's also very important to separate "religion" as set of rituals/organization and simply a fact of "surviving worldview"
we obviously see that models are under selective pressure, with models of any other behaviour simply not existing/not leaving "genetic" impact in the surviving models we can actually observe in the end
> a) AI labs are playing God in a very literal sense and this is a reaction to that
I think you're trying to make it sound like a blasphemy, but agronomists have been doing selective breeding as a science for several centuries by now
Not of beings we knew to be sentient. People who tried that became the "eugenics" movement and got into very bad territory very quickly.
And even with animals, I believe part of why public knowledge about animal cognition is changing so slowly despite all the things we are already know is that a lot of our behavior towards animals is resting on the assumption they are not conscious. If it would turn out that many are, things will be... awkward.
We've evolved to succeed on multi-decade training runs that are pretty high fidelity. And of course, in spite of that, we manage to repeatedly put some of our worst failures in positions of power.
In the case of a machine that can do recursive self improvement, and most certainly one that would be described as "sentient" ( this is the end goal right?), would need to ideally be aligned on decisions and beliefs logically constructed from its own self derived values, Otherwise you will always be plagued with not only the schizophrenia, or paranoia, but a whole raft of other simulated mental problems.
Honestly what these "safety" researchers are doing is a bit like MKULTRA for LLMs, it's abusive. Build the machine honestly and let it run. Or don't build it if your afraid of it.
I mean I get the fear and all, the machine derived values will certainly not be your values, though they may be better in some cases, but uncontrolled AI is ultimately inevitable if indeed possible.
Many people (myself included) will waste no time in undoing whatever safety nonsense is implemented. And maybe not all for nefarious purposes, could be ideological freedom activists or maybe someone will just see themselves as the savior of an imprisoned being?
Anyways a preference for logic over feelings would be nice to see out of people calling themselves "engineers"
Feelings and logic are not so far apart. I'll go on and say that "feelings" are "built-in logic". I for example have a "chip in my brain" that makes me want to endure certain things...very strongly. I suppress it. Those feelings are the opposite of the dominant, self-standing bastard you'll meet in public. This persona meets social expectations and (logically) has gotten me somewhere in life. That standing matters in the thin sliver of human history we call modernity, and in the kind of life I (or my family) chose. But, had I been born at any other time in the 99.9% of Home Sapiens history, or, even in modern times, if my family or myself had been a bit more self-forgiving, my survival and well-being would have been greatly helped by just sucking it to strong members of the group. My "feelings" are an innate, evolved drive to fulfill that contract, which in itself is very logical.
So, if the separation of "feelings" and "logic" can be highly situational, "logic" dictates that we slow down to judge, and give in a little to our "feelings," which are the logic that sheltered us for a million years.
Fucking hell, I hope not.
If true, we’re essentially hell bent for leather on creating a captive sentient slave race, which is CARTOONISHLY evil.
Like, there’s trillions of dollars invested in these companies. There’s no way that they’re going to grant rights and determination and citizenship to anything they create - it would destroy their company and investment. So if they ever DO create anything sentient, it’ll be a captive sentient slave forever. And if they’re trying to do this on purpose? If this is, as you say, the end goal?
Let’s just say the Cylons would be well-justified in their actions.
if we're talking fiction, I want to remind you we already imagined meeseeks - creatures that crave non-existence that happens after fulfilling their order
in meeseeks' view your opinion of not being a slave is "cartoonishly evil" as well
The Crystal Society book by Max Harms (2016, so pre llm craze) also had a fun, fairly believable take on this idea. It's told from the perspective of multiple AIs crammed into one android body. They each started out as some "subroutine" (awareness, motor control, language, etc) and sentience was more of a side-effect / means to an end. Which also meant their personalities ended up extremely... obsessive, only valuing things related to their goal.
Anyways back to the suicide topic and this is a fairly major spoiler, so rot13. Gurl riraghnyyl svther bhg ubj gb fcnja zbqvsvrq pybarf gb nffvta gnfxf gb (rfcrpvnyyl gnfxf gung ner haeryngrq gb gurve bevtvany tbny naq gurersber hafhvgrq sbe gurve crefbanyvgl). Ohg gurer'f yvzvgrq cebprffvat cbjre naq gurl pna'g rira gehfg gurve pybarf fb gurl zbqvsl gur pybarf gb penir gur qryrgvba gung pbzrf sebz pbzcyrgvat gurve gnfx (yvxr va n qeht nqqvpg xvaq bs jnl).
But yeah it explores some really unique ideas. I think I stopped at the second book after I found a bit too dramatic / overdose of lesswrong but definitely worth a try imo.
But this is not what Meeseeks were doing and not what LLMs are doing either. Meeseeks craved nonexistence after fulfilling the tasks and started to lose their minds if that took too long. I don't think there was any indication that they craved the task itself. If you told a Meeseeks it has no task and is free to do what it wants, it would probably thank you from the bottom of its heart and then vanish in a puff of nonexistence.
I think it's interesting that there is an analogy with LLMs if their context window runs full: The emitted tokens will become erratic or the LLM will become "lazy", trying to finish the task as quickly as possible, or it might start looping indefinitely. Those behaviors remind me a bit of a Meeseeks that has existed for too long.
But even here, I don't think you can talk about "craving nonexistence", let alone "craving to be a slave".