markusheimerl/gpt

来源：https://github.com/markusheimerl/gpt

gpt

一个生成式预训练Transformer的实现

该项目使用Transformer架构实现了一个自回归序列模型。模型处理字节序列（8位令牌），学习根据先前上下文预测下一个字节。虽然本实现训练文本数据，但架构本身与内容无关。它可以建模任何字节流，包括但不限于DNA/RNA序列、压缩数据、图像、音频、视频或可执行二进制文件。

架构首先使用令牌嵌入层将每个字节转换为连续向量表示。

模型核心是一个多层Transformer，处理嵌入后的序列。每个Transformer层由两个主要组件组成：因果自注意力机制和前馈网络，两者都通过残差连接包裹。因果注意力确保每个位置的预测只能依赖之前的位置，这对于自回归生成至关重要。注意力机制计算查询、键和值的投影，对查询和键应用旋转位置编码以编码相对位置，使用因果掩码计算缩放点积注意力，并将结果投影回原始维度。前馈网络应用两个线性变换，中间带有swish激活函数——这是一种平滑、非单调的函数，将输入与其sigmoid输出相乘。

经过所有Transformer层处理后，一个线性投影将最终隐藏状态映射到词汇表（所有256个可能的字节值）上的logits。这些logits通过softmax函数转换为概率，模型通过交叉熵损失训练，以最大化正确下一个字节的概率。

训练过程使用AdamW优化器，它通过将权重衰减与基于梯度的更新解耦来增强标准Adam优化器。AdamW维护梯度和平方梯度的指数移动平均，并利用它们为每个参数单独调整学习率。权重衰减作为L2正则化，鼓励模型使用较小的权重，从而提高泛化能力。

本实现使用BLAS（基本线性代数子程序）进行高效的矩阵运算，使模型能够在现代硬件上高效训练。

如何运行

Ubuntu

sudo apt update
sudo apt install -y clang make time libopenblas-dev nvidia-cuda-toolkit git curl
git clone https://github.com/markusheimerl/gpt && cd gpt/
make data
make run -j 6
make infer

示例输出

提示词为 "Once upon a time, there was a"：

markus@thinkpad:~/gpt$ make infer
Loaded: d_model=512  hidden=1024  layers=16  vocab=256  seq_len=1024
Generating 995 tokens (T=0.70, seed=1779612639)
Once upon a time, there was a compassionate little girl named Lily. She loved to play with her friends in the park. One day, she saw a small bird on a tree. The bird was sad because its wing was hurt.
Lily asked the bird, "Do you have any hurt wing?" The bird said, "Yes, I don't have any hurt wing." Lily wanted to help the bird, so she tried to make it feel better. But the bird was too big and her hurt wing still did not want to hurt Lily's wing.
Lily had an idea. She found a long stick and brought it to the bird. The bird said, "Thank you, Lily! You saved me!" The bird felt better and thanked Lily. They played together in the park all day. They were very happy and became best friends.
<|endoftext|>
Once upon a time, there was a little girl named Mia. Mia loved to study with her toys. She had a big box full of toys in her room. One day, Mia found a new toy. The toy was a small doll. The doll had a pretty dress and smiled a little.
Mia took the doll outside to play. She studied hard and felt the dress on her f
markus@thinkpad:~/gpt$ make infer
Loaded: d_model=512  hidden=1024  layers=16  vocab=256  seq_len=1024
Generating 995 tokens (T=0.70, seed=1779612665)
Once upon a time, there was a little boy named Tim. Tim had a big tree in his yard. He loved to run and play in the tree. One day, he saw a perfect bird in his yard. The bird was sad because it could not find its mom.
Tim wanted to help the bird. He kneeled down and looked all around. He saw a little girl named Sue. Sue was playing with a ball. Tim asked her, "How can I be like your bird?" Sue smiled and said, "You can be my friend."
Tim helped the bird get close to Sue. Sue was so happy and thanked Tim. They became good friends and played together in the tree. The bird sang a song and they all lived happily ever after.
<|endoftext|>

Once upon a time, there was a little boy. He was very careful as he walked around a park. One day, he saw an unusual thing called a rabbit. The rabbit hopped over to the thing and asked the other animals if they had seen it. The other animals thought it was a funny sight.
The rabbit and the other animals were very curious. They asked the other animal, "What do you think is a fun
markus@thinkpad:~/gpt$