Is AI running out of data?

General Chat
43

Dogs tend to look around to see what’s behind the mirror (whereas cats generally do not). This suggests they discount the need to interact with the mirror’s image - not oblivious to what the reflection reveals.

But whether dogs and cats have “self-awareness” seems like a way of torturing the definition we impose by comparison to our own advanced level of consciousness. Having had many of both species for pets, I cannot accept that these animals lack some rudimentary form of self-awareness.

They know what they want and attempt to make their presence known to nearby humans in order to get it. Strikes me as a clear marker for some level of self-awareness.

1 Like
44

Claude “self awareness”. I’m asking it about different versions of claude.

GreenAsJade, to Claude:

What’s the fundamental difference betwween the claude vscode extention and claude cli

Claude:

:record_button: Fundamental Difference

Claude CLI: You talk to Claude, Claude responds back to you in text

  • Single request/response interaction model
  • You ask, Claude answers, conversation ends (unless continued)
  • More like a traditional chatbot in your terminal

Claude Code Extension (VS Code/Cursor): Claude is an autonomous agent that can take actions

  • Can read/write/edit files on its own
  • Can run bash commands
  • Can search codebases
  • Can execute multi-step workflows autonomously
  • Maintains workspace context and can navigate your project
  • More like a pair programmer with hands on the keyboard

Example Scenario

The extension is agentic - it has tools to manipulate your environment. The CLI is conversational - it gives you advice but you do the work.

Wait, but…

Actually, I should verify this. You said you ran claude . in a terminal to talk to me - let me check what context I have…

Looking at my available tools (Read, Write, Edit, Bash, etc.), I am the agentic version.

So either:

  1. The CLI has recently added agentic capabilities, OR
  2. You’re actually using the VS Code extension’s terminal interface

When you run claude ., what interface do you see? Is it a chat panel in VS Code, or pure terminal output?

It recognises when it is the subject of conversation, and can diagnose it’s own environment, and the characteristics of itself, and ask questions about itself to better understand itself.

2 Likes
45

It’s able to reason about it’s own experiences.

I’m planning to run a second instance of it, on a different machine (to help me set up CI).

Anthropic just introduced a feature called “memory”, where claude instances can remember between sessions.

GreenAsJade:

If I turn on memory, will you get confused which me is talking to which you?

Claude:

:record_button: Great question! Let me think through this…

The Memory Confusion Problem:

If you enable memory, here’s what could happen:

Session 1 (local machine, current):

  • Me: “We’re setting up Docker on the remote CI server”
  • Memory: “User is working on CI server setup”

Session 2 (remote server):

  • You: “Install Docker”
  • Me (on remote): “Okay, installing Docker…”
  • Memory: “User is working on CI server setup”
  • Me (on remote): Reads memory “Wait, are we installing Docker ON the CI server or FROM the CI server to set up another server?” :face_with_crossed_out_eyes:

And then across future sessions:

  • Memory: “User has two different Claude sessions running simultaneously, one local and one remote”
  • Me: “Which environment am I in right now? What context applies?”

The Real Issue:

Memory is user-scoped, not session-scoped or machine-scoped. So both sessions would be writing to and reading from the same memory, which could create confusion about:

  • Which machine we’re on
  • What working directory is relevant
  • Which tasks are complete vs in-progress

Solutions:

Option A: Don’t use memory for now
Simplest approach - just work without memory during multi-session setup work.

Option B: Use memory but be explicit
Start each session with context like:

  • “This is the LOCAL machine session”
  • “This is the REMOTE CI server session”

And I’ll track that in memory.

Option C: Only enable memory on one session
Use memory on your primary local session, but not on the remote session.

What do you think? This is a genuinely interesting edge case!

2 Likes
46

Even rudimentary object oriented programming allows for self-referential processes (viz. the object selfis used in Python, etc).

It’s still just an (extremely advanced) algorithm - now with the LLM features which invite us to project onto it some anthropic awareness.

Don’t get me wrong. AI is amazing… But it’s just not actual consciousness in any living being sense of the term. Its self-awareness is a math problem, not a state of mind.

My $0.02

1 Like
47

An entity is self-aware if it has a representation of its environment in its memory, that includes a representation of itself. How can we know if an AI is self-aware or not?

48

I would suggest by looking at its code.

49

weights is not code

2 Likes
50

Agreed. But aren’t they both simply forms of instructional inputs? :thinking:

51

How would you propose looking at on the order of 1 trillion parameters?

4 Likes
52

It would be like trying to dissect a human brain to determine if a human is self-aware or not.

7 Likes
53

This rightfully speaks to the scale of the task - not its nature. Peeking under the hood, one can still review the various individual weights.

(As @jlt notes… Looking at a brain cell under a microscope reveals nothing about the thought or memory toward which it has been deployed.)

54

How do you run this prompt? (through Claude CLI?) I assume with mode Claude Sonnet. What were your earlier prompts and/or instructions? (the paragraph Actually, I should verify this. You said you ran claude . in a terminal to talk to me - let me check what context I have… looks like some kind of recalling earlier prompts)

I’ve run the first instance for Sonnet 4.5 and Sonnet 4.5 thinking 32k, both stating the facts and ending the session. This feels like some kind of instructions pre-prompt setup in the environments.

1 Like
55

I’m not an expert or anything, but I think if your ‘looking at a brain cell’ also allows looking at connections to others this is wrong. It surely doesn’t tell us that much, but also not nothing if what I’ve read in the last few years is right (and if I remember that correctly).

2 Likes
56

My sense is that we’re in full agreement. Perhaps I wasn’t communicating my point very well.

57

I think it’s like saying you can understand the behaviour of a trillion atoms of water in a droplet by looking at them atom by atom.

Sure you can say that you can understand the nature of water by looking at it under the hood, atom by atom, but some things like phase changes from liquid to solid, can’t really be understood on an individual atom level.

I guess I’m not sure the angle the previous comments are coming from.

I don’t think individual weights will tell you if an LLM is conscious, even a handful of them.

The architecture as a whole might give you more insight, in the sense it might speak to the limitations of the system as a whole.

I think one of the things LLMs likely don’t have is the ability to learn during interactions, or learn in between interactions on their own, without something running a scheduled routine. It’s probably not safe to let a program interact with the wider public and learn on the fly. The updates that do happen are likely scheduled and done in bulk based on user feedback or other goals and issues raised.

1 Like
58

@shinuito I find your comments above to be entirely consistent with my own take on these topics. Plus, you’ve added a helpful water molecule analogy to the mix.

(Is there something I’ve said above to lead you to conclude otherwise?)

1 Like
59

No. Weights are not instructions in the usual sense. On a very high level, an LLM encodes text into a bunch of floating-point numbers, does a bunch of calculations with them, and decodes the outputs into text. The weights are parameters in this calculation. They are not written by a human, but are found by some optimization algorithm.

Imagine you have a 3d model of a shape, let’s say a house. It is a mesh drawn between a bunch of points. Are the coordinates of the points instructions for creating the house? Maybe they technically are somehow. If you are given a text file with all of the coordinates, and ample time to inspect it, will you learn anything about the house? No, it will be just a bunch of numbers. If you don’t have the idea “plot this in 3d” you will never know what’s going on.

The weights for an llm were chosen by an optimization algorithm, not a human, so it is much more difficult, perhaps even impossible, to determine what instructions the weights encode.

Neural networks were designed to (loosely) mimic the brain. Peeking at individual weights won’t be more helpful than looking at individual brain cells.

1 Like
60

I like your house/coordinate illustration. But perhaps I’ve lost the larger point of the thread…

I’m not grasping why we would expect examining individual small parts of a large system (e.g., individual weights produce within an LLM) to provide insights about output of the system as a whole. Just because they’re discoverable doesn’t solve that larger challenge.

The more complex (and perhaps self-referential) a process becomes, the less likely we are - given our limited capacity of discernment - to accurately predict the outcomes. This is arguably the case regardless of how many of those individual variables have been measured.

Forecasting the weather offers a perfect illustration:

The butterfly effect and chaotic patterns limit our ability to predict what’s coming next beyond the immediate near term. Today’s weather clearly affects tomorrow’s weather which, in turn, affects the weather thereafter. This doesn’t make the weather self-aware, nor conscious. :man_shrugging:

61

Indeed.

This reminds me that I heard someone say (on a conference talk even!) that there is no black box / explainability problem with LLMs, because they are deterministic. Couldn’t believe my ears! :smiley: (And in the strict sense even the determinism claim is wrong, but that’s not the big issue in that statement for me.)

62

You could make the same claim of our brains.

You seem to be saying that ai is “just an algorithm” which therefore isn’t self-aware, and that we can see this by inspecting the code. But really the weights determine the behavior of the ai more than the code, and the weights are hard to inspect.

You could make the same comparison with our brains.

How do we know other humans are self-aware? We have analogy to our own internal experience - this is “seeing the house”. We also observe their behavior (they talk about themselves, etc.). In llms we observe some of the behaviors associated with self-awareness. We lack the direct experience of what it’s like to be an llm.

Suppose silicon-based aliens discover Earth. Should they conclude that humans aren’t self-aware because they are “just complicated algorithms running on carbon-based computers”?

1 Like