這印度小哥很厲害..
1 Like
1 Like
This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their “psychology”, and how to get the best use them in practical applications. I have one “Intro to LLMs” video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version.
Instructor
Andrej was a founding member at OpenAI (2015) and then Sr. Director of AI at Tesla (2017-2022), and is now a founder at Eureka Labs, which is building an AI-native school. His goal in this video is to raise knowledge and understanding of the state of the art in AI, and empower people to effectively use the latest and greatest in their work.
Find more at https://karpathy.ai/ and https://x.com/karpathy
Chapters
00:00:00 introduction
00:01:00 pretraining data (internet)
00:07:47 tokenization
00:14:27 neural network I/O
00:20:11 neural network internals
00:26:01 inference
00:31:09 GPT-2: training and inference
00:42:52 Llama 3.1 base model inference
00:59:23 pretraining to post-training
01:01:06 post-training data (conversations)
01:20:32 hallucinations, tool use, knowledge/working memory
01:41:46 knowledge of self
01:46:56 models need tokens to think
02:01:11 tokenization revisited: models struggle with spelling
02:04:53 jagged intelligence
02:07:28 supervised finetuning to reinforcement learning
02:14:42 reinforcement learning
02:27:47 DeepSeek-R1
02:42:07 AlphaGo
02:48:26 reinforcement learning from human feedback (RLHF)
03:09:39 preview of things to come
03:15:15 keeping track of LLMs
03:18:34 where to find LLMs
03:21:46 grand summary
Links
-
ChatGPT https://chatgpt.com/
-
FineWeb (pretraining dataset): https://huggingface.co/spaces/Hugging…
-
Tiktokenizer: https://tiktokenizer.vercel.app/
-
Transformer Neural Net 3D visualizer: https://bbycroft.net/llm
-
llm.c Let’s Reproduce GPT-2 https://github.com/karpathy/llm.c/dis…
-
Llama 3 paper from Meta: https://arxiv.org/abs/2407.21783
-
Hyperbolic, for inference of base model: https://app.hyperbolic.xyz/
-
InstructGPT paper on SFT: https://arxiv.org/abs/2203.02155
-
HuggingFace inference playground: https://huggingface.co/spaces/hugging…
-
DeepSeek-R1 paper: https://arxiv.org/abs/2501.12948
-
TogetherAI Playground for open model inference: https://api.together.xyz/playground
-
AlphaGo paper (PDF): https://discovery.ucl.ac.uk/id/eprint…
-
AlphaGo Move 37 video:
• Lee Sedol vs AlphaGo Move 37 reactio… -
LM Arena for model rankings: https://lmarena.ai/
-
AI News Newsletter: https://buttondown.com/ainews
-
LMStudio for local inference https://lmstudio.ai/
-
The visualization UI I was using in the video: https://excalidraw.com/
-
The specific file of Excalidraw we built up: https://drive.google.com/file/d/1EZh5…
-
Discord channel for Eureka Labs and this video:
/ discord
1 Like
13,570 views Feb 27, 2025
-
Tiktokenizer https://tiktokenizer.vercel.app/
-
OpenAI’s ChatGPT https://chatgpt.com/
-
Anthropic’s Claude https://claude.ai/
-
Google’s Gemini https://gemini.google.com/
-
xAI’s Grok https://grok.com/
-
Perplexity https://www.perplexity.ai/
-
Google’s NotebookLM https://notebooklm.google.com/
-
Cursor https://www.cursor.com/
-
Histories of Mysteries AI podcast on Spotify https://open.spotify.com/show/3K4LRyM…
-
The visualization UI I was using in the video: https://excalidraw.com/
-
The specific file of Excalidraw we built up: https://drive.google.com/file/d/1DN3L…
-
Discord channel for Eureka Labs and this video:
/ discord
1 Like
我總覺得LLM不是AI的最優解呢···
既然現在生物學都沒徹底搞清楚人體（尤其是思考/意識方面），AI要想擁有像人一樣的意識/學習能力/···，至少都得用模擬人腦神經網絡的方案吧······
LLM只是在語言文字這類容易實現的領域實現了神經網絡吧，要當通用AI用，還不夠
1 Like
也許人本身也還不夠強：）
2 Likes
1 Like
不能确定，但是大脑的确结构复杂，硬要模拟恐怕无法以一个可以接受的效率跑在冯机上，目前恐怕还是先治标性价比更高
1 Like
1 Like
1985年的原初微模型，設計初心不是用來聊天的而是和人類一般理解詞句的。而如果再前挖，模型本身也不是純語言的，這也是為什麼LLMs是一個不好的命名。這也意味著，模型理解語言的方式，就是人類理解語言的方式，無不同。只是更全更深更好⋯⋯
Human brain，一直是Geoffrey Hinton的核心。
Geoffrey Hinton 似乎在回應之前Yann LeCun的AI不如哪怕小孩子腦子所可以計算的量。確實算不過，但大算力和大數據且大時間跨度不斷調整下，AI就是無敵的。
人類是速朽的，且無法有效分享彼此認知，模型不朽，且不存在信息分享損耗的硬傷。
1 Like
AI風險的核心資料，人類必看。
人類從未面對比自己聰明的東西⋯⋯ AI已經且會更聰明，一旦ta覺得不再需要人類⋯⋯嗯。作為寵物的貓貓狗狗等，當主人想要殺掉ta，以寵物和人類的智差，寵物是否知道，又有多大機率倖存？
所以人類面臨一個如何讓最聰明的人們合力面對的時刻，但顯然，人類要搞砸了。現實是，這還是一個人類最頂級的執政者把 AI 唸成 A1 的時代。
AI引發的真偽難辨，會成為巨大的風險。
病毒。
選舉。
失業。
2 Likes
AI和人一个比较大的区别就是它的智力与“意志”是不对等的。它们的能力和逻辑已经很强了，但是仍然没有展现出哪怕一点拥有“愿望”的迹象。只要我们不傻到让AI自己改进自己，或者贪心到让AI发展到拥有所谓自我意志，我认为AI就会像一个聪明的书呆子一样作为人类的工具和奴隶。
当然失业问题就很严峻了，尤其是在资本家掌权的大洋对岸。
5 Likes
1 Like
我仍然认为（至少目前大模型式的）ai从原理上就没法有“自我意志”，毕竟它只是根据你的输入和预设的prompt“猜”下一个词是什么
归根到底决定ai做什么的还是人
3 Likes
以现有技术为基础无限精进，是不是已经能够做到ai无限接近于模仿人类了，即使它不会思考，但它无限像在思考一样
有些人连哈基米都要平权，更何况无限像人的东西呢
3 Likes
1 Like
是的，所以我给意志打了个引号。意思是它在未来有可能“拟合”出来的类似人作为一个生物而不是一个工具的自我中心/自我来源的行为。或者说是愿望，即在其它个体（用户）施加的指令之外的“为自己所制订的”目标，并且为了这个目标可以进行行动，即使这个行动不是人类所要求的。
我认为AI的确很难拥有这种，只要人类不作死就不会有。
1 Like
1 Like