GPT Explanation: The GPT Model (Learning Notes) The Earlier You Know, the Better

In this rapidly developing The Internet Age, everyoneNew innovations and breakthroughs are emerging every day. Now, let's talkoneTalking about recent developments in the internet industryoneSome hot topics, see what amazing things are happening.

GPT model (learning notes) GPT model Gererate Pre Training Model is essentially unsupervised learning. The number of upper layers of the transformer foundation increased to one2 layers does not make much contribution to the model, which proves that the big model and Big data set are effective.

Dataset: Books Corpus (7000 books, 800 million words, 5GB text) 8 GPU trainingoneLast month's paper: Radford et al. "Improving Language Undersatting by Generative Pre-Training.

yesheightThe text token vector is the word token vector matrix is given by the position matrix1Unsupervised tokens, which is also the conversion of Conditional probability maximum likelihood maximum likelihood estimation of Language Model into Loss Function. Contribution 2 * * * *.

Method: (1) Transformer for Pre training_ Block fixed (2) replace last1Layer linear layer classifier. The previous classifier layer was a prediction word, and the vector was very large. If we classify documents into 100 categories, it would be 100 dimensions.

(3) Given1A dataset D with a Label can be considered as a critical entry loss1The activation layer of the layer is the linear output layer, and here is the final layer1Single word or multiple words, please adjust (4) the final oneLoss function pre training loss (ML maximum likelihood estimation), Supervised learning loss, this is multi task learning.

GPT model (learning notes) GPT model Gererate Pre Training Model is essentially unsupervised learning. The number of upper layers of the transformer foundation increased to 12 layers does not make much contribution to the model, which proves that the big model and Big data set are effective.

Dataset: Books Corpus (7000 books, 800 million words, 5GB text) 8 GPU training1Last month's paper: Radford et al. 'Improving Language Undersatting by Generative Pre Training'.

yesheightWentokeN vector is a word token n vector matrix is a positional matrix given1Unsupervised tokens, which is also the conversion of Conditional probability maximum likelihood maximum likelihood estimation of Language Model into Loss Function. Contribution 2 * * * *.

Method: (1) Transformer for Pre training_ Block fixed (2) replace last1Layer linear layer classifier. The previous classifier layer was a prediction word, and the vector was very large. If we classify documents into 100 categories, it would be 100 dimensions.

(3) Given1A dataset D with a Label can be considered as a critical entry loss1The activation layer of the layer is the linear output layer, and here is the final layer1Single word or multiple words, please adjust (4) the final loss function pre trained lOss (ML maximum likelihood estimation), the loss of Supervised learning, is called multi task learning.

GPT model (learning notes) GPT model Gererate Pre Training Model is essentially unsupervised learning. The number of upper layers of the transformer foundation increased to 12 layers does not make much contribution to the model, which proves that the big model and Big data set are effective.

Dataset: Books Corpus (7000 books, 800 million words, 5GB text) 8 GPU training1Last month's paper: Radford et al. 'Improving Language Undersatting by Generative Pre Training'.

yesheightThe text token vector is a word token vector matrix is a bitGiven matrix1Unsupervised tokens, which is also the conversion of Conditional probability maximum likelihood maximum likelihood estimation of Language Model into Loss Function. Contribution 2 * * * *.

Method: (1) Transformer for Pre training_ Block fixed (2) replace last1Layer linear layer classifier. The previous classifier layer was a prediction word, and the vector was very large. If we classify documents into 100 categories, it would be 100 dimensions.

(3) Given1A dataset D with a Label can be considered as a critical entry loss1The activation layer of the layer is the linear output layer, and here is the final layer1Single word or multiple words, please adjust (4) the final loss function pre trained loss (ML maximum likelihood estimation) and supervise itThe loss of learning is called multitasking learning.

In this poetic moment, I infuse my emotions into every1In a word. I hope your heart will surge after reading it1Silk warmth. Remember to follow and like your favorite friends! “