While reading papers on Transformer and post-Transformer architectures, I wanted to build intuition on their trade-offs and failure modes.. Inspired by Kaparthy’s nano-GPT videos, I built vibejam - a minimal GPT trained on 9 years of my personal journal entries.
What is vibejam?
While models like ChatGPT and Claude write quickly, they struggle to capture my personal writing style. When I ask them to help me write something, it usually turns into a very AI-sounding style. I wanted to train a GPT that writes like me. I have been keeping a light journal (1 long google doc per year documenting my goals, reflections daily to weekly and annual reflections) since 2017. So I used 9 years of journal entries (2017-2025) as training material to train a GPT from scratch. I then extend the model to take in post-transformer architectures, using RWKV and MoE as a start, to see how they perform versus transformers.
Here are a few lessons I learned from building this project:
1. More failures are system bugs, not model bugs
Implementing GPT from scratch (Layer 1) taught me data pipelines matter more than attention math. I dealt with tokenizer mismatches, prompt format leakage, and shape errors; none of these are related to attention mechanisms. Getting the system right is necessary before architectural experiments matter. And most of the time, it takes longer than one thinks!
2. Decouple System from Architecture
From Layer 1 (a nano-GPT style attention model), I wanted to extend the project to swap in other post-transformer architectures (Layer 2 & 3). I built a minimal model interface (BaseLM + factory pattern), which made swapping architectures smooth. This minimizes rewriting scripts and made the code much easier to read and extend.
3. RWKV introduces new tradeoffs between retrieval and efficiency
What RWKV is (simplified): Receptance Weighted Key Value (RWKV) replaces attention with recurrence. Instead of looking back at all previous tokens (which is expensive), it maintains a compressed ‘memory state’ that gets updated as new tokens arrive. I learned that recurrence is sensitive to time constants. The decay parameter controls how quickly the model “forgets” past tokens. When I first initialized the decay parameter, the memory half-life was 0.9 tokens, making it effectively memoryless. After fixing to target ~16 tokens, the model starts to actually use context.
This taught me that compressed state isn’t just “attention but cheaper”. It is a fundamentally different tradeoff: attention can retrieve exact past tokens when needed, while recurrence compresses history into a fixed-size state - you gain efficiency but lose precision.
4. MoE is more for Scaling
What MoE is (simplified): Mixture of Experts (MoE) replaces the standard feed-forward layer with multiple ‘expert’ networks. For each token, a gating mechanism routes it to the top-k most relevant experts. The idea is to increase model capacity without proportionally increasing compute - only the active experts run.
What surprised me was that MoE didn’t really improve quality at a small scale. It just added more knobs to tune. The routing dynamics (which experts get used, whether they specialize or collapse to mostly one expert) became a separate debugging challenge. What I learned is that MoE is fundamentally about scaling: it’s useful when you need more capacity than you can afford to run densely, not as a way to make small models better.
5. Evaluation is A Must, Not An Afterthought
I learned to treat evaluation as infrastructure, not an afterthought. Fixed prompts, fixed random seeds (so sampling is repeatable), and deterministic decoding (same temperature / top-k settings) allowed me to compare architectures objectively. Without these scaffolding, I would have been chasing sampling noise instead of architectural differences.
6. Post-Transformers: Different Problems, Different Failures
Post-Transformer architectures try to solve Transformers’ quadratic scaling cost - attention over all previous tokens gets expensive fast - and memory inefficiency. But they introduce new failure modes: RWKV’s recurrence can collapse if decay isn’t initialized carefully, and MoE routing can degenerate such that all tokens use only one expert.
Learning through Implementation
I learned a lot, and had a lot of fun building vibejam as a side project. Small, readable implementations beat reproducing papers when it comes to developing intuition.
vibejam is intended to be an ongoing playground, where I’m curious to 1) keep adding more architectures to compare results with transformer and others, 2) explore what personalized, expert-knowledge curated, or private GPTs look like as a product for individuals.