Experimental Turkish GPT models trained from scratch
Kayra is a small-scale GPT-based language model trained from scratch for Turkish language processing research. This project includes three different models:
| Parameter | Range | Default | Description |
|---|---|---|---|
| Temperature | 0.1 - 2.0 | 0.7 | Controls output randomness. Lower values produce more consistent results, higher values produce more creative results. |
| Max Tokens | 10 - 512 | 150 | Maximum number of tokens (word pieces) to generate. |
| Top K | 1 - 100 | 50 | Only the top K most likely tokens are considered at each step. Lower K = more focused, higher K = more diverse. |
| Top P (Nucleus) | 0.1 - 1.0 | 0.9 | Selects from tokens whose cumulative probability exceeds P. Used together with Temperature. |
| Repetition Penalty | 1.0 - 2.0 | 1.3 | Penalizes repeated tokens. Higher values reduce repetition but may break fluency. |
| Stream Mode | On/Off | On | When on, the response is shown word by word; when off, it appears all at once. |
High temperature and top-k produce more original and unexpected outputs. Ideal for story writing and brainstorming.
temp=1.2, top_k=80, top_p=0.95, rep=1.1
Default settings. Suitable for most use cases.
temp=0.7, top_k=50, top_p=0.9, rep=1.3
Low temperature for more consistent and predictable outputs. For factual questions.
temp=0.3, top_k=20, top_p=0.8, rep=1.5
Maximum randomness. For wild and unexpected results. Use with caution!
temp=1.8, top_k=100, top_p=1.0, rep=1.0
| Feature | kayra-1 (Stable) | kayra-1-exp (Experimental) | kayra-1-thinking-exp (CoT) |
|---|---|---|---|
| Type | Instruction-tuned | Pretrained Only | Chain-of-Thought fine-tune |
| Usage | Question-Answer format | Text completion | <|think|> reasoning format |
| Stability | Stable | Experimental | Experimental |
| Prompt Format | ### Q: ... ### A: | Plain text | ### Question: ... <|think|> |
| Recommended Usage | General testing and demo | Research and analysis | Reasoning tests |
sixfingerdev/kayra-thinking-data
4,191 Turkish reasoning examples from 4 sources: duxx (226), efe (350), Gemma-4 (32), ThinkingData-200K-Turkish (~3,583)
| Model Type | Decoder-only Transformer (GPT-style) |
| Number of Layers | 10 |
| Hidden Size | 640 |
| Attention Heads | 10 |
| FFN Size | 2560 |
| Vocabulary | 32,000 BPE tokens |
| Context Length | 512 tokens |
| Total Parameters | ~85 million |
| Source | Document Count | Description |
|---|---|---|
| Wikipedia TR | ~170,000 | Turkish Wikipedia articles |
| mC4 Turkish | ~330,000 | Common Crawl web documents |
| Total | ~500,000 | Deduplicated with MinHash LSH |
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"sixfingerdev/kayra-1-exp",
trust_remote_code=True # IMPORTANT!
)
tokenizer = AutoTokenizer.from_pretrained("sixfingerdev/kayra-1-exp")
prompt = "Türkiye'nin başkenti"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
inputs.input_ids,
max_new_tokens=100,
temperature=0.8,
top_k=50,
top_p=0.9,
repetition_penalty=1.2,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License: MIT License - Free for commercial and academic use.
@misc{kayra2024hallucination,
title={Why Small Turkish GPTs Hallucinate Facts: An Experimental 85M Model},
author={sixfingerdev},
year={2024},
publisher={HuggingFace},
howpublished={\url{https://huggingface.co/sixfingerdev/kayra-1-exp}},
note={Research on loss-factuality divergence in low-resource language models}
}