The Post-Trainer's Guide for Multilingual Synthetic Data Generation

High-quality training data is often scarce for less-resourced languages, and collecting it from native speakers demands substantial human effort, creating a bottleneck for language-specific model development. To alleviate this challenge, synthetic data generation has gained traction as a promising solution: a typically larger teacher model generates training examples, which are then used to finetune a smaller student model to replicate the knowledge of the teacher.

In this work, we study the key design decisions in the synthetic data pipeline: which teacher model to use, which data generation method to apply, and how these choices play out across languages. Based on our findings, we recommend a recipe for generating multilingual synthetic data.

Our evaluation framework

We study three synthetic data generation approaches commonly found in the literature.^{1 1} We conduct a brief review of synthetic data approaches in our paper (Appendix A).

Swipe to see all three methods ▸

The output is a set of question-answer pairs in the target language, quite similar to the conversations we have when talking to an AI assistant. These pairs are commonly used for supervised finetuning (SFT), which involves training a student model to mimic the teacher's outputs.

Synthetic data pipeline. We then formalize the post-training synthetic data generation pipeline.

To evaluate a synthetic data recipe, we measure two things: (1) the intrinsic quality of the synthetic dataset itself, and (2) the extrinsic performance of a student model trained on that data across a diverse set of multilingual tasks. We find it important to consider both, since synthetic datasets enter the data commons and get reused, while student models are deployed in real-world applications. To do so, we introduce the Polyglot Score (PG-Score), which combines both aspects into a single number:

Experiments & Findings

We test 10 teacher models on 6 typologically diverse languages (Arabic, Czech, German, Spanish, Indonesian, and Japanese), chosen for their variation in resource availability, script, and language family. In total, we generate over 1.4M SFT instances and train 240 student models.

Which teacher model excels in multilingual data generation?

Which qualities of the dataset predict downstream performance?

SetupIn order to identify which qualities of the generated data drive student performance, we run principal component analysis (PCA) on the six intrinsic data-quality metrics, then fit a linear model that uses the resulting principal components to predict downstream student performance.

ResultsThe first four components explain over 93.3% of the variance in the intrinsic metrics,³ and those components predict student performance with R² = 0.664 from a simple linear model. In other words, the qualities of the generated data carry most of the signal about how well a student will learn.

What are some heuristics for choosing a teacher model?

In order to distill some heuristics for choosing a teacher model, we test different aspects such as model family, data generation method, and amount of SFT examples. We note that these observations are conditional on our setup—the tasks and languages we evaluate on—and may not hold in every setting.

Model family. Each teacher belongs to a model family (Gemma, Llama, Aya, and so on), and models within a family tend to share tokenization schemes and pretraining data. We test whether this shared lineage makes a family a useful signal for pairing a teacher with a student.

Data generation method. The three methods make different assumptions: Generate depends on the quality of the few-shot seed examples, Translate transfers from widely-available English prompts, and Respond reuses existing prompts in the target language. These assumptions may favor different languages depending on how much data they already have.

SetupFor each method in isolation, we generate 10k samples per teacher–language pair, finetune an OLMo 3 7B student, and compute PG-Score. We test three teachers spanning high to low PG-Score (Gemma 3 27B, Aya Expanse 32B, Llama 3.1 70B) across German, Indonesian, and Arabic.

Table 3. PG-Score across the three data generation methods. We highlight the best method for each teacher and language.

ResultsFor a high-resource language like German, the Generate method yields the highest PG-Score, while for less-resourced languages like Arabic and Indonesian, the Respond or Translate methods are more effective. Generate leans on the few-shot seed examples, which are typically of higher quality in high-resource languages.

Amount of SFT examples. Generating synthetic data is not free, so we ask how many SFT examples are actually needed to reach stable student performance before the gains level off.

Discussion: Towards a recipe for multilingual synthetic data generation

Our results provide actionable insights for selecting and effectively using teacher models in multilingual synthetic data generation. We distill them into a recipe of four decisions:

• Teacher model. Model scale does not significantly predict teacher effectiveness. What matters instead is the quality of the generated data: prompt diversity, response fluency, and length collectively capture over 93% of the variance in intrinsic data quality and predict student performance, offering practitioners a cheaper alternative to full student training runs for screening teacher candidates.
• Data generation method. There are language-dependent considerations for data generation. For high-resource languages like German, where seed data quality is high, the Generate method performs best. For less-resourced languages like Arabic and Indonesian, methods that leverage existing prompts (Respond) or transfer from English (Translate) can yield substantial gains over a uniform mix of methods. For truly low-resource languages, we recommend combining synthetic data generation with targeted data collection.
• Synthetic data scale. Around 10k synthetic examples per language is sufficient to achieve strong student performance, and generating more mostly adds cost — though saturation points may depend on task diversity.
• Student base model. When the optimal teacher is unknown, matching model families offers a reliable heuristic for teacher selection. Gemma teachers paired with Gemma students, and Llama teachers with Llama students, outperform a mismatched baseline by at least 20%.

Next, we put this recipe into practice on a held-out language, applying the same four decisions.

Our recipe in practice: Tagalog case study

SetupTo put the recipe to the test, we apply it to a held-out language that was not part of our evaluation: Tagalog, a mid-resource language and the standardized form of Filipino, the national language of the Philippines.

• Data: We collect Filipino seed data from publicly available SFT datasets such as WildChat and the Aya Collection, and include English data from Tülu 3 for the Translate method.
• Evaluation: We evaluate on FilBench (Miranda et al., 2025), a Filipino-centric benchmark spanning classical NLP, cultural knowledge, reading comprehension, and generation.
• Baselines: We compare our recipe against a public-data baseline (10k Tagalog pairs sampled from public datasets, non-synthetic) and a GPT-4o-mini baseline (10k instances synthesized by an off-the-shelf GPT-4o-mini teacher, chosen ad hoc for its perceived strength).

ExperimentWe perform a series of synthetic data interventions using the recipe and recommendations we learned, adding each one on top of the last. Step through them to watch the FilBench score climb.

Figure 7. Student model performance on a held-out language, Tagalog, across several synthetic data interventions. Given a held-out language (Tagalog) and an evaluation benchmark (FilBench), we apply data interventions based on our recommendations on creating a multilingual synthetic data recipe. The interventions are additive.

Conclusion

We conduct a comprehensive evaluation of state-of-the-art LMs as multilingual teachers for synthetic data generation by assessing both intrinsic data quality and extrinsic student model performance. We find several properties that contribute to teacher effectiveness outside of model size or benchmark performance, such as prompt–response diversity, fluency, and language representation. Finally, we outline practical recommendations for creating a multilingual synthetic data generation recipe. We hope our findings guide future work on developing inclusive language technologies through high-quality synthetic data.

Citation

@misc{miranda2026polyglotteachersevaluatinglanguage,
  title={{P}olyglot {T}eachers: {E}valuating {L}anguage {M}odels for {M}ultilingual {S}ynthetic {D}ata {G}eneration},
  author={Lester James V. Miranda and Ivan Vuli\'{c} and Anna Korhonen},
  year={2026},
  eprint={2604.11290},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2604.11290},
}