sethev
The challenge here is in trying to decide whether LLMs are
intelligent or have a mind. Famously, the criteria for
intelligence seem to slip with each advancement in
technology. But going back to Turing, his test was
actually more carefully phrased than we remember: he said
that when machines could pass the test, the question of
whether they are intelligent would become moot. That seems
to be what we're actually seeing: if people can't tell the
difference, it kind of won't matter whether they're "truly
intelligent" or not.
|
> tomrod
I'm a bit more prosaic. I think if we engineered ways
for LLMs to begin conversations, rather than just
respond, we'd be more open to the concept of their
intelligence. Without perceived "will" to do things,
they operate as a next-gen search engine or
encyclopedia.
|
> > 10xDev
I believe this is alignment working as intended.
|
rayiner
> LLMs are not brains and do not meaningfully share any of
the mechanisms that animals or people use to reason or
thinkLLMs are a type of neural network. We know that's how
the human brain works, at least directionally. It's going
to be very upsetting to a lot of people when we figure out
that the brain is just a neural network. Akin to when we
found out that humans and apes evolved from a common
ancestor.Which I don't understand-most of the people
having this cognitive dissonance presumably do not have a
theological worldview. And there's not exactly a direct
theological conflict here anyway. Nothing in any major
religion I'm aware of ascribes any supernatural
explanation to cognition. It's a biological computational
process, just like using ATP to power muscle fibers to
move your limbs is a biological mechanical process.
|
rahidz
Man I remember back when a psychic conned me by solving
the Navier-Stokes problem.Also
>July 4th, 2023
|
> krupan
The LLM did not solve it. It's not intelligent. Humans
did, using a statistics-based computational tool (the
LLM). We don't even know all the details of how the
tool was used, we haven't been allowed to use the
exact tool they used ourselves, we don't know much it
really cost in dollars, energy, or time, etc. etc.
|
> > simianwords
Sometimes I think this amount of skepticism is not
necessary. Leaving the pedantic (its not AI its
humans who solved) arguments, its pretty clear
that LLMs are able to help solve things. A lot of
erdos problems were solved. Millenium problems as
well. Cyphers broken. Skepticism is fine but
there's a point at which it just looks like cope.
|
> > claytongulick
And it may have trained on a NYU professor's work.
|
> trescenzi
LLMs can be supremely useful but also not intelligent.
It might seem like a pointless distinction but the way
we talk about these models matters because it impacts
how we interact with and understand their outputs.For
example if there's a strongly held belief that models
are independent intelligent entities we're more likely
to lay blame upon them instead of their user. It's
important for the safety discussion too. If they are a
new class of life then safety is going to focus on
making sure they don't do bad things. If we instead
see them as statistical models we will instead try to
make sure people don't misuse them.This distinction is
even more important today when some of the most
powerful people are looking to absolve their crimes by
passing them off on their LLMs.
|
> Angostura
But were you successfully conned into believing an AI
had solved the Navier-Stokes problem?
|
> > tescreal
What is the consensus among experts?
|
> > > grey-area
That OpenAI stole the discovery from them.
|
> > > > daishi55
That is very much not the consensus of
experts.
|
> lukewarm707
this seems like non-sequitur if you mean that solving
NS is 'intelligence'.ai is not conscious. you can
solve NS without thinking. the psychic con aspect is
anthropomorphising the model. the same phenomenon is
present in ELIZA, clever hans, the chinese room.it's a
significant problem.a non-zero number of researchers
at anthropic are in some form of ai psychosis. an
example of that is ethics employees asking claude
about its feelings and ethical concerns in order to
make the claude constitution more amenable to the
"welfare" of claude.they are asking claude how claude
feels and then modifying claude according to how
claude feels.constitution1-claude is trained on
constitution1.
constitution1-claude edits constitution1.
constitution2-claude is trained on constitution2.
constitution2-claude edits constitution2.claude's
emotions are a closed system. there is no external
truth to improve against, no metric to verify about
claude's emotions. there can be no novelty or
reduction in entropy from signal processing in a
closed system. no truth can arise. this is model
collapse. it is like photocopying the same thing over
and over. from the cognitive error of anthropomorphism
anthropic is causing ethical collapse.
|
> inopinatus
The more compelling inversion is whether the likes of
John Nash, Richard Feynman, John Conway etc could've
had a lucrative second career as conmen.
|
> > mistercow
Feynman for sure could have. In fact, I think he
sort of did, although I'm not sure he meant to.
Multiple generations of nerds now have taken books
of his anecdotes varying in plausibility and
obvious exaggeration as some sort of weird physics
cult of personality gospel. But I don't think he
was really setting out to curate his legacy so
much as he was a good storyteller and he liked to
entertain.But could he have conned people on
purpose? Absolutely.
|
> > atemerev
von Neumann could. The others, I doubt it.
|
> rusk
Did an AI do that on its own though? I heard it was
human mathematicians using a sophisticated machine as
a tool.
|
> > meowface
All of the latest big proofs were driven by
professional human mathematicians steering and
priming the models, yes.All of the best AI-made
software projects are also driven by experienced
human software developers steering and priming the
models. Does that mean the projects "aren't made
by AI"?No, it just means AI is not quite good
enough yet to fully replace humans, and, so,
unsurprisingly, the best results will be obtained
from people who are already great at a field and
who take the time to squeeze as much force
multiplication out of LLMs as possible. The AI is
still doing well over 95% of the significant work.
|
> > > TomGarden
While I agree with you, I think we also have
to concede that this is not how these
accomplishments have been presented. I'd argue
most people I've seen talk about this online
are unaware of the mathematicians steering the
models.
|
> > > agileAlligator
https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-b
bea-8deb42cfed...Terrence Tao's conversation
with ChatGPT is very illuminating.HN
Discussion:
https://news.ycombinator.com/item?id=49010345
|
> > > basch
"Good enough to replace humans" isn't
necessarily the benchmark.The question is is a
computer with a human stronger than a computer
without a human. At what point does the hybrid
go from being stronger, to the human getting
in the way, or steering the computer in more
wrong directions that right ones, or the human
not being able to keep up. Does the human add
enough extra randomness to be of value for a
while, even as a minor co-processor.
|
> > broast
The way frontier models work, that loop will get
compressed to a one-shot within a version or two
|
> > mstank
It sounds like LLMs were pretty useful to them...
|
> > > ModernMech
So was Lean. Did Lean solve it?
|
> > > > zamadatix
Nothing is solved in isolation but credit
usually goes to wherever the new work in
the paper comes from instead of the whole
mountain of previous mathematics or
existing tools used. The most relevant of
those get referenced and then this
reference tree builds a tree of collective
base work needed across history.
|
vhantz
Building up strawmen against LLMs will only make the
hypers look more reasonable. A language model responds to
inouts exactly as a model of anything else would. That is
enough to explain all the "intelligence" without believing
a model is somehow a "new kind of mind". Those who think
LLMs are intelligent don't know enough about models. And
those who think they are useless don't know enough about
models.
|
> 10xDev
So once we have online learning in LLMs, what do you
think you will have that makes you intelligent but not
LLMs? Better learning efficiency? That will be
improved as well.I think we need to start moving on
from the term LLMs because it clearly confuses people
since they started modelling more than just language.
|
daishi55
> many people are convinced that language models, or
specifically chat-based language models, are intelligent.
But there isn't any mechanism inherent in large language
models (LLMs) that would seem to enable thisSeems like we
can just stop reading here right? The author seems to have
made up their mind that this very open question is closed,
or at least they are not really interested in the question
at all. Not sure why I would continue reading a blog based
on this premise.Edit: oh I see, written in 2023. Well, I
wonder if the author has updated their attitude towards
this question? Indeed that would be the most interesting
thing to know.
|
> azakai
It's even worse than that. Of course there is a
mechanism in LLMs that could explain intelligence.
That is the entire point of neural networks, from the
1950's! They were designed from the start as a model
of brain computation.Neural networks are not literally
brains - just computational models - but if you are
not a dualist, then computation is what the human
brain does. Modeling that computation can explain
something about intelligence.Specifically: when
scientists look inside a human brain, it seems it does
its work using large numbers of highly-interconnected
but simple units. The neural network model of brain
computation begins there and tries to produce
intelligent behavior. If it succeeds, then perhaps the
model is right.And it has succeeded: after 75 years,
neural networks produce complex behavior that is
arguably intelligent. Nobel Prizes were awarded. This
does not prove the neural network model of
intelligence is accurate, but it is a significant
point in its favor, at least.The author seems entirely
unaware of any of this.
|
ungreased0675
It does seem like LLMs share the language of psychics.The
author is also correct that LLM evangelicals and believers
in the occult speak about it similarly.
|
scotty79
> The intelligence illusion is in the mind of the user and
not in the LLM itself.I think that might be the solution
to consciousness illusion.The consciousness might be
purely in the mind of someone that believes some other
entity to be conscious.
|
mstank
This is a strange newsletter post. It's harping against AI
but seems AI-written itself.Its ultimate conclusion:"I've
come to the conclusion that a language model is almost
always the wrong tool for the job.I strongly advise
against integrating an LLM or chatbot into your product,
website, or organisational processes."Seems so obviously
biased that I can only understand it with the context that
the writer is trying to sell their book for €35
|
> mstank
Just realized this was written in 2023. Can't believe
how wrong we were about AI back then.
|
> > JoshTriplett
Can't believe how right we were, too.
|
> ryuuseijin
The post makes sense for 2023. Today, not so
much.Also, definitely not AI written if the date is
accurate.
|
aetherspawn
Stay the hell away from spooky stuff like psychics and
tarot readers - the more you don't believe in it, the
better.But better again that you do believe, and know that
these are not harmless fun, but that there are dark and
hidden and evil things in this world to stay away from.
|
> besterman23
Stop trying to pump your Tarot reader IPO
|
_superposition_
I predict a lot of butt hurt tokens in the comments...
|
krupan
People are missing the point. We should all be much more
skeptical, much more careful about how we evaluate the
claims made by the people selling these products because
we as humans are super vulnerable to the types of scams
the author of this article describes. We all know someone
that has fallen for a scam, been "cured" of a "disease" by
someone whose just selling sugar pills, but we are blind
to our own weakness for similar schemes. Surely we aren't
that easily duped! Yet hacker news is now full of comments
from people confidently predicting what is coming right
around the corner, revering the Frontier Models, and
defending every claim from OpenAI and Anthropic about how
amazing their proprietary closed source secret sauce
fueled product is.
|
atemerev
All of this is accurate (and useful). But yeah, psychics
do not prove theorems from frontier math and build working
complex software.
|
> cmiles74
In my opinion, this article is talking about people
who get into long conversations and become convinced
the model is "intelligent", maybe even "AGI". This
seems to be a trap people are falling into, even in
2026.
|
> > krupan
It's definitely talking about that, but it's also
talking about all of us:- fawning over how amazing
these tools are- believing everything OpenAI and
Anthropic say about how powerful and dangerous
their product is- minimizing the amount of human
effort and involvement in every "AI" achievement
|
> > atemerev
Initial definitions of AGI (as of 2014) are long
surpassed. Then, AGI was defined as something that
is general enough to work in many fields and
orient themselves. It did not include being
smarter than humans or even having comparable
intelligence to humans. By original standards, we
have AGI already for some time.Current AGI
definitions are intentionally vague.
|
> krupan
Neither do LLMs. Humans using them do. Similar to how
cops solve crimes using psychics.
|
> igortg
On one side I agree that LLMs and Agents are not
intelligent, but they present an illusion of
intelligence given the shear amount of data they can
process and act upon.But that cannot be used to
discredit the fact that these are incredibly powerful
tools that can get out of control and cause great
damage.
|
> > atemerev
What is "intelligence" then. I think the
universalism of LLMs is now evident enough that
they can be considered "intelligent", even in
slightly different realization compared to
humans.As of "great damage", I doubt it. They do
not have self-preservation instinct (all
"worrying" experiments are the attempts to
initiate something resembling self-preservation
from human initiative). The driving part is
external - the query loop can always be turned
off. So yes, a dangerous tool that can be
exploited by humans (including governments,
especially governments - which is why I am
skeptical to government regulation proposals,
particularly looking at what passes as governments
in this era). But there are no inherent dangers
from their own agency, as there is none.
|
> > > alansaber
"Slightly different"- no, a completely
different realisation of intelligence to
humans. LLMs can definitely do cool stuff
though.
|
> Madmallard
Neither does LLMs. Not without a monumental amount of
human oversight and expertise.Everyone not in a
particular field asking LLM about said field is
rolling bad dice.
|
> > atemerev
I have published a LLM-assisted proof of a
long-standing problem in convex geometry (70
years) while not being a specialist in convex
geometry. I am a scientist and I have education
and scientific experience in computer science and
systems biology, but I was never a mathematician.
So, well, you can definitely work across fields at
least.
|