| --- |
| library_name: transformers |
| tags: |
| - mdlm |
| - diffusion |
| license: apache-2.0 |
| datasets: |
| - turkish-nlp-suite/InstrucTurca |
| language: |
| - tr |
| base_model: |
| - diffutron/DiffutronLM-0.3B-Alpaca |
| pipeline_tag: text-generation |
| --- |
| |
| # DiffutronLM-0.3B-Instruct |
|
|
| **Diffutron** is a parameter-efficient, Masked Diffusion Language Model (MDLM) specifically designed for the Turkish language. Unlike standard autoregressive models that generate text one token at a time, Diffutron generates text by iteratively refining sequences in parallel, allowing for simultaneous consideration of the entire sentence context. |
|
|
| Despite its compact size of 307 million parameters, `DiffutronLM-0.3B-Instruct` achieves highly competitive performance against much larger, multi-billion-parameter autoregressive baselines on Turkish NLP benchmarks. |
|
|
| ## 📌 Model Details |
|
|
| * **Model Type:** Masked Diffusion Language Model (MDLM) |
| * **Base Architecture:** `jhu-clsp/mmBERT-base` (Multilingual Encoder) |
| * **Language:** Turkish |
| * **Parameter Count:** 307M (0.3B) |
| * **Context Length:** 256 tokens (Instruct), 512 tokens (Base) |
| * **Training Libraries:** `dllm`, PyTorch |
|
|
| ## 🚀 Architecture & Approach |
|
|
| Diffutron departs from traditional next-token prediction. It treats text generation as a discrete diffusion process: |
| 1. **Forward Process:** Clean text is gradually corrupted into a sequence of `<mask>` tokens. |
| 2. **Reverse Process:** The model learns to denoise the sequence globally, attending to visible context bi-directionally to predict the original tokens. |
|
|
| This non-autoregressive paradigm compresses linguistic knowledge efficiently, allowing this 0.3B model to punch significantly above its weight class. |
|
|
| ## 📚 Training Pipeline |
|
|
| The model was developed through a resource-efficient, multi-stage training pipeline: |
|
|
| ### 1. Continual Pre-training (CPT) |
| To adapt the multilingual backbone to Turkish without catastrophic forgetting, we employed a high-rank LoRA strategy (r=256, α=256) targeting all linear modules (Attention and MLP). |
| * **Data:** ~2 million sequences sourced from Havadis (news), Temiz-OSCAR (web), and Turkish Wikipedia. |
| * **Result:** Perplexity on the Bilkent Turkish Writings Dataset dropped significantly from 3.42 (base) to 2.75. |
|
|
| ### 2. Progressive Instruction-Tuning (SFT) |
| To unlock generative instruction-following capabilities, we utilized a two-stage supervised fine-tuning approach: |
| * **Stage 1 (General Adaptation):** Trained on `metunlp/LlamaTurk-Instruction-Set` for 20 epochs to establish fundamental instruction-following behaviors. |
| * **Stage 2 (Complex Specialization):** Trained on the nuanced `turkish-nlp-suite/InstrucTurca` dataset for 8 epochs with an increased batch size, enhancing the model's ability to handle intricate, domain-specific Turkish commands. |
|
|
| ## 📊 Evaluation Results |
|
|
| The model was evaluated on a representative subset of the **CETVEL Benchmark Suite**. DiffutronLM-0.3B (2nd Stage) demonstrates remarkable parameter efficiency, outperforming models up to 7x its size (e.g., Kumru-2B and TURNA-1.1B) on average scores. |
|
|
| | Benchmark | Diffutron-1st-Stage (0.3B) | Diffutron-2nd-Stage (0.3B) | TURNA (1.1B) | Kumru (2B) | Kanarya (2B) | Llama-3.2 (3B) | Trendyol (7B) | Aya-101 (13B) | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | |
| | **Belebele_TR** | 22.22 | 27.00 | 22.56 | 29.00 | 28.11 | **55.78** | 36.22 | 22.89 | |
| | **EXAMS_TR** | 25.95 | 27.74 | 23.66 | **30.03** | **30.03** | 26.21 | 28.50 | 22.90 | |
| | **IronyTR** | 50.67 | **52.00** | 48.33 | 51.00 | 50.00 | 50.17 | 50.00 | **52.17** | |
| | **News_Cat** | 23.20 | 32.40 | 32.80 | 26.40 | 66.80 | 64.00 | **81.20** | 20.00 | |
| | **MNLI_TR** | 33.29 | 32.81 | 34.94 | **36.42** | 33.40 | 34.76 | 35.19 | 27.90 | |
| | **STS_TR** | 17.77 | **18.78** | 14.21 | 11.75 | 12.91 | 12.91 | 15.52 | 16.97 | |
| | **XCOPA_TR** | 53.80 | 52.00 | 55.80 | 54.00 | **64.20** | 54.60 | 61.00 | 59.60 | |
| | **Average** | 32.41 | **34.68** | 33.19 | 34.09 | 40.78 | 42.63 | **43.95** | 31.78 | |
|
|
| ## 💻 Usage |
|
|
| Because Diffutron is a Masked Diffusion Language Model, it requires inference strategies distinct from standard causal generation. We recommend using the `dllm` library or custom generation loops tailored for discrete diffusion. |
|
|
| ### Generation Parameters Used in Paper: |
| * **Longer Context:** Steps: 128, Temp: 0.1, Block Length: 32, Repetition Penalty: 1.2 |
| * **Shorter Context:** Steps: 64, Remask: `low_conf`, Stochastic: `False`, CFG: 0.0 |
|
|
| ## ⚠️ Limitations |
|
|
| * **Multilingual Backbone:** Built upon a multilingual encoder rather than a native Turkish foundation model. |
| * **Context Window:** Restricted to a 256-token context window for generation, limiting its use in long-form summarization or document-level generation. |
| * **Data Nuances:** Instruction datasets rely heavily on translations or synthetic data, which may occasionally miss subtle cultural contexts. |
|
|
| ## 📝 Citation |
|
|
| If you use Diffutron in your research, please cite our preprint: |
|
|
| ```bibtex |
| @misc{diffutron2026, |
| author = {Kocabay, Şuayp Talha and Akkuş, Talha Rüzgar}, |
| title = {Diffutron: A Masked Diffusion Language Model for Turkish Language}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/collections/diffutron/diffutronlm}} |
| } |
| ``` |
