Hello everyone, my Cheshire cats 🐈⬛🌟
I decided to write a little cheat sheet about how generative models like GPT work.
I think it's important to clarify❗️: I will hardly touch on neural network learning. I'll pay more attention to exactly how it generates responses.
How does the neural network perceive our question?
Of course, she doesn't see the words we wrote to her. Text is divided into tokens (pieces words or whole words).
Each token is assigned a vector of numbers🎲. Initially, these vectors are random, but during training they change a lot and are eventually fixed.
During training, the model learns from lot of text to predict the next token by finding patterns between them. After training, she already has sets of weights (ready-made "rules" according to which probabilities are distributed)⚖️
What happens when a model receives a message and prepares to respond?
After receiving the message, the model analyzes the entire context, both the current message and the chat history, if any.
How does the analysis work?
Three basic things are built for each token: query, key, and value.
To put it more clearly, "what do I want to learn from other tokens"🔎, "how can I be found"🗝, and "what information do I offer"🧩
Next, each token compares its request with the keys of other tokens. This comparison takes place through a scalar product. The model measures how well a query for one token matches the key of another👥
For example: in the sentence "mom ate porridge" the "ate" token will most strongly pay attention to "mom" (who performs the action) and "porridge" (what is the object).
Each token updates its state, becoming a mixture of information from the rest of the tokens, taking into account their importance.
To do this, the model first turns similarity scores into attention weights☝️
Then, using softmax, it makes sure that all the weights add up to one and are positive. And only after that it calculates the total weighted sum of values from other tokens.
As a result: "mom" learns possible actions and objects, "porridge" who eats it, and "ate" who participates in the process and what exactly happens.
In other words, the neural network seems to see the general meaning of the phrase, but "from different angles"📷
Naturally, all this happens in parallel in several threads🧠, usually from 8 to 128. If the calculations were done strictly in turn, it would take too long to generate the answer.
How does the model determine the close context?
After training, numbers arrays (from queries and keys) are generated for each token.
Proximity is calculated based on the similarity of these arrays: the more similar they are, the more closely the tokens are connected🪢 in the current context.
How is the response generated? ✒️
When the representation is formed, the ai does not go through its entire knowledge base. Instead, it calculates the next token within the representation.
The response itself is created using a single token: a new token is added to the context, the entire context is recalculated, and the process repeats. Simply put, with each message, the model receives the history📖 of the dialog and builds a response based on this entire history📝
That's why the new models work with context much better than the old ones, before the advent of transformers🤔 They can just pick up the connections between words at a much greater distance.
It also explains why jailbroken chat😈 continues to work even after a long conversation the chat. However, if you enter the same jailbreak in a new chat, it will most likely be immediately blocked by censorship.
The possibilities of the model are not unlimited. It has limit on the size of the context window, how many tokens it can process at the same time. When the new models first appeared, it was better.
But now, due to savings, companies are reducing the number tokens💲 available (especially in the free versions, it's much better in the paid versions). Therefore, the context window is getting smaller...😥
#ai #llm #transformers #GPT #attention_mechanism #neural_networks #tokenization
