PyTorch from scratchBPE · Transformer · LoRAHinglish (Hindi-English code-mixed)

mini-gpt: a Hinglish GPT built from scratch, against a LoRA-tuned open LLM

A 5.3M-parameter GPT written from scratch in PyTorch and pre-trained on 98M tokens of Hinglish tweets, compared with classic NLP baselines and with Qwen3.5-0.8B (zero-shot, few-shot and LoRA fine-tuned) on the same SemEval-2020 SentiMix sentiment test set: 3,000 tweets, macro F1.

Code on GitHub →

0.714

best test macro F1: Qwen3.5-0.8B + LoRA (10.8M trained params)

0.679

mini-GPT from scratch (5.3M params), pre-trained then fine-tuned

+3.9

macro F1 points from pre-training (same model, random init vs pre-trained)

0.415

Qwen3.5-0.8B zero-shot: big models still need task labels

Results

Every model on the same test set. Mean ± std over 3 training seeds (or 3 draws of the few-shot examples); the 95% interval is a paired bootstrap over the test tweets for one representative run.

Bar chart of test macro F1 for all models: Qwen3.5-0.8B with LoRA is highest at 0.714, followed by TF-IDF at 0.696 and the pre-trained mini-GPT at 0.679; zero-shot Qwen is at 0.415.

Model	Stage	Test macro F1	95% CI	Test accuracy
Qwen3.5-0.8B + LoRA	3	0.714 ± 0.007	[0.699, 0.730]	0.710
TF-IDF + LogReg	1	0.696	[0.680, 0.712]	0.691
mini-GPT (pre-trained, fine-tuned)	2	0.679 ± 0.002	[0.663, 0.696]	0.676
BiGRU (word vocab)	1	0.653 ± 0.002		0.651
mini-GPT (random init, fine-tuned)	2	0.640 ± 0.006		0.636
BiLSTM (word vocab)	1	0.639 ± 0.017		0.638
Qwen3.5-0.8B 3-shot	3	0.508 ± 0.026	[0.490, 0.522]	0.565
Qwen3.5-0.8B zero-shot	3	0.415		0.511
Majority class	1	0.179		0.367

LoRA vs TF-IDF: +0.018 macro F1, 95% CI [+0.002, +0.035], a real but small win. TF-IDF vs the mini-GPT is not significant at 95%. Few-shot Qwen (0.508) almost never answers "neutral"; LoRA fixes that.

Explore the predictions

Pick a tweet to see each model's probabilities. The first six were written for this page; the rest are SentiMix test tweets where the models disagree (tweets with strong profanity skipped). Predictions were computed with the exact demo models and stored in this page; nothing runs live.

The mini-GPT

Tokenizer: byte-level BPE (8,192 tokens) written from scratch and trained on the corpus: 3.74 bytes/token, no unknown tokens.
Model: decoder-only transformer, 4 pre-norm blocks, d=256, 4 heads, context 128, weight-tied LM head.
Pre-training: 98M tokens of L3Cube-HingCorpus tweets, 12k steps, AdamW + warmup/cosine, 40 min on a GTX 1650 (fp32: fp16 was measured 4.6× slower on this GPU). Validation perplexity 8,659 → 150.
Fine-tuning: classification head on the closing token; lr 3e-5, 3 epochs, 3 seeds.

Training and validation loss falling from 9.07 to about 5.0 over 98 million tokens, with train and validation curves close together. — Train and validation loss stay together: the model is capacity-limited, not overfitting.

Sample generations (temperature 0.7, top-k 50)

Prompt	mini-GPT continuations

What the attention heads learned

Heatmaps of per-head attention statistics: layer 2 head 3 puts 57% of its weight on the previous token; several layer-0 heads put 24 to 40% on the first token. — Layer 2 head 3 is a previous-token head (57% of its attention on token i-1); layer-0 heads park weight on the first token (an "attention sink").

Attention weights of all 16 heads for the sentence 'bhai aaj ka match bahut accha tha'. — "bhai aaj ka match bahut accha tha": in layers 2-3, "bahut", "accha" and "tha" attend to "match".

What didn't work, and what it taught

A word-level BiLSTM overfit within 3 epochs and lost to TF-IDF: 10.5% of validation words were unknown to its vocabulary.
Fine-tuning the mini-GPT for 8 epochs overfit after epoch 1; short runs with the learning rate decaying over the whole run added ~1 F1.
Contextual calibration made few-shot Qwen worse: its content-free inputs ("N/A") are genuinely neutral, so dividing out that "bias" removed the neutral class.
fp16 mixed precision was 4.6× slower than fp32 on a GPU without tensor cores. Measure before assuming.
Three times the best hyperparameter sat at the edge of the search range; extending the range changed or confirmed the result each time.

Run it yourself

The full live demo (classify with all three models, generate text, attention maps) runs locally from the GitHub repo:

pip install -r requirements.txt && pip install -e .
python -m minigpt.export_app
python app/app.py