From early chatbots to GPT
Machine-assisted translation of the archived article. Read the Chinese original.

Traditional chat robots, whether Elizabeth or Alice, are based on a technique called pattern matching.
When it hears a keyword, it draws a predefined response, like when you say "hello" to it, it asks you, "Has anyone eaten?" ;
Or when it hears "Mom," it'll ask you to talk to it about your family. In fact, some robots in the areas of shopping sites and banking are now also based on this model.
For example, when you talk to these robots, once they mention the return, they send you the return process; or when you refer to the ATM, they send a map of ATM around you.
Although this pattern of matching is not very smart, it does reduce a lot of repetitive mechanical answers.
More complex and predictable, it is impossible to exhaust all the answers, much less to create new ones. So, trying to get through the Turing test and become real smart,
This pattern alone cannot be achieved.
As a result, a new genre has emerged in language learning, which is also a very important part of artificial intelligence: machine learning. It’s not just a matter of time, but of course.
The basic idea of machine learning is to let the machine learn, not give it some rules and answers that are artificially prescribed, but rather to provide a large number of ready-made examples.
Let it learn by itself, find a pattern. It sounds much more powerful and more in line with our cognitive logic of learning.
By 2010, a field of machine learning, called Artificial Neural Network, had begun to shine.
The human brain communicates through web links with more than 10 billion neurons. Although each neurons is simple, they combine to judge very complex information.
Thus, the goal of artificial neural networks is to simulate this form of human brain. When you enter the information, it is judged by several hidden nerve nodes and gives the output results.
Although the idea of an artificial neural network existed as early as the 1960s, it needed to be supported by a great deal of data and powerful calculations. Until the 2010s, with the growth of the Internet,
The data is becoming more and more rich, and their calculus is increasing. It's only when people find out that this model is very appropriate for things that humans can tell by instinct, such as identifying human faces.
Sound recognition, auto-driving, etc. As a result, the development of artificial neural networks is not going well in the field of writing. Because machine learning usually uses a type of cyclic neural network (RNN)
The way to deal with words is to learn word by word in order, without the means to do a lot of learning at the same time and without long sentences.
Otherwise, what has been learned may be forgotten.
Until 2017, Google proposed a new learning framework called Transformer. It's complicated, I don't fully understand, but the result is,
A word that needed to be learned in the past can now be learned at the same time, just like a combination of circuits. As a result, the speed and efficiency of training has greatly improved. We have Transformer.
In terms of word learning, the machine is like a second line of commission, and many natural language-processing models are based on it. Google's Bert and GPT-3 refer to this Transformer.
The technical aspect is now very strong, and everything is ready, short of people and funds.
In 2015, a billion-dollar non-profit organization, called OpenAI, the parent company of GPT, was set up by a group of top executives like Mask and Peter Tyre.
Work on artificial intelligence research. They claim that the organization was set up not to make money but to promote technological development, so their research and patents are made public.
Some of the investors we know very well are Mask, and in fact he's finding that Tesla also needs a lot of research input in artificial intelligence, such as autopilot.
In order to avoid a conflict of interest between Tesla and OpenAI, he withdrew from the board of directors of OpenAI in 2018, so that now OpenAI has little to do with Mask.
OpenAI, these big cows are really good too. In 2017, Google launched the Transformer model, on which OpenAI immediately built.
In 2018, they published a paper on a new language learning model called GPT (Generate Pre-Train Transformer).
Most of the previous language learning models required manual supervision or labelling, but GPT was virtually no longer needed. It can learn on its own by simply entering a large amount of data into it.
OpenAI launched the first generation of GPT in June 2018, then increased training data volume and introduced GPT-2 in November 2019.
Machine learning relies mainly on models and parameters. Models determine how machines learn, and GPT can learn faster and better than other models with the same data.
Parameters require substantial computing support, and therefore a significant investment in training and validation is required. The OpenAI team is very confident in the model, and it is a good idea for the government to be able to do something about it.
The next challenge is therefore the lack of financial support. Each of these advances may need to be supported by additional quantitative data, which will require the support of real resources.
Microsoft supports OpenAI’s super-calculative capacity and is prepared to contribute significantly to development. The GPT-1 generation had only 120 million parameters, and by the GPT-2 it had increased to 1.5 billion.
The GPT-3, launched six months later, has increased 100-fold directly to 1750 billion parameters, with very good results. It's starting to show the real meaning.
The GPT features, whatever you ask, can answer. It's causing a lot of noise in the industry.
But this pure machine-trained GPT-3 has a question, which is sometimes very good and sometimes less interesting, and very limited improvement, however much it adds to the parameters.
In order to address this problem, OpenAI has incorporated an enhanced learning method of manual feedback and has made significant progress in terms of training efficiency and effectiveness.
GPT-3.5 was launched in March 2022 and the dialogue was optimized. It was launched in November 2022 as a very simple, rough chat interface.
But you ask it anything, it answers you. It makes sense. Of course, there may be some issues that we will discuss later.
In any case, it does talk and is very natural in its language. After half a century of development, ChatGPT can certainly easily pass the Turing test.
One might suspect that OpenAI is a fraud company that hires thousands of people every day to respond to you (a joke).
But I don't think that's a very likely possibility, and GPT has indeed subverted many people's perception of chat robots, including me. So in just two months,
The number of active users of GPT has surpassed 100 million, with the fastest expansion in history.
