How does GPT learn and interpret language?

Machine-assisted translation of the archived article. Read the Chinese original.

Give me a chestnut:

There were two little girls named Sally and Anne.

Sally has a basket, Anne has a box.

Sally has a marble in the basket, but Anne's case doesn't. One morning, Sally went out.

Anne snuck out of Sally's basket, stole the marble and put it in his own box.

Sally came home at night and she wanted to find her own beads.

So the question is, where should she look for it?


Humans over the age of four can answer quickly: go find their own basket.

But previous AIs, including GPT 3.5, said to go to Anne's basket.

Attention is drawn to the fact that even if the name, scene and behavior of the story were replaced, a similar question could be answered by GPT4, not because it was read in the pre-training data.

This experiment is used to determine the ability to distinguish between self-awareness, self-perception and others, and logic such as narratives and God's perspective, which many autistic children cannot pass.

Why can you learn language through words?

And what the results of this training have produced.

I wonder if you've thought about it.

Why is it impossible for violence to learn language?

But to play hyphenation in a huge vector space,

I can learn to speak human language.

Because the space structure itself stores information.

It's like the last example.

In a word vector space, the king minus men, plus women get out of positions very close to the Queen.

It means nothing, it means location.

The real picture of the relationship in the real world.

So what's the matter with you?

It’s because after dozens of TB language trainings, it’s not the only thing that’s going on here.

He's got every word in the vector space.

I'm not sure if I've been able to find my own place.

And those words are in space.

In fact, the logic behind the language is mirrored in reality.

Since there's enough vector space,

I can map all the logical languages.

So if we have a bigger vector space,

He's not capable of mapping the whole real world.


Each and every one of the answers is based on your questions.

This is the first time I've been able to find a way to get rid of it.

To give you your own answer.

It's like a baking string, oh no, it's like the thinking of our brains.

The little guys who saw the videos about how our brain works.

And remember, our brains don't have a dedicated memory storage space.

Our memories and experiences are stored in the connection of neurons.

The process of extracting memories and thinking.

It's electricity that travels between different neurons.

The process of activation of different neurons,

Every word that comes out of your mind at this point in time,

It's all generated by the waves that flow through neurons.

It's like gpt looking for the words that are needed in magnetic vector space.

The difference is that the neurons of our brains are being updated at any time and place.

And gpt's vector space was fixed after the training.

Sure is, because there are too many parameters.

There's too much space, every training is very resource-consuming.

It takes a lot of graphics, lots of machine training for many days to complete.

So when the training is done, gpt can't be based on everyone's problems.

And to adjust your training, for gpt,

It's a strategy to keep the change, no matter what your problem is,

I've organized answers in a fixed vector space.

But because of your problems, the path of the chain has changed.

So you feel like he's really communicating with you.

So gpt is aware of the world.

Staying on the day when training stopped,

He doesn't have new memories because he talks to you.


Why when a hard question he can't solve,

You let him work it out in a step, and he can figure out the right result.

Because the way people talk is how they think.

It's just that you're in your head, and he has to say it, anyway, when the amount of data trained and the size of the parameters of the model,

And when it reaches a critical mass, the vector space is able to map information that is complex enough.

Some of the capabilities that AI should not have had were created.

A man who was just talking, suddenly had a higher intelligence!