Training Azerbaijani Models Is Now an Operational Problem

AWS shows how Azercell approached Azerbaijani LLM training on SageMaker AI. The real lesson is tokenizer evidence, artifact control, and security operation

2026-05-29 GIGATAP Team #security
#AI#SageMaker#LLM

Training Azerbaijani Models Is Now an Operational Problem

Source: AWS Machine Learning Blog — https://aws.amazon.com/blogs/machine-learning/training-azerbaijani-language-models-on-amazon-sagemaker-ai/

AWS published a case study on Azercell Telecom LLC training Azerbaijani language models on Amazon SageMaker AI. The useful part is not that another company wants a chatbot. It is the shape of the work: tokenizer design, continued pre-training, supervised fine-tuning, and cloud job orchestration treated as one production path.

Azercell, described by AWS as Azerbaijan’s leading telecommunications provider, worked with the AWS Generative AI Innovation Center on an Azerbaijani LLM for telecom use cases and a customer-facing chatbot. The source says the collaboration ran for six weeks and produced a framework on Amazon SageMaker AI using open source tools including PyTorch, Hugging Face Transformers, and Liger Kernels.

AWS reports three headline efficiency gains: 23% higher training throughput, 58% lower peak GPU memory use through kernel-level optimizations, and a 2× improvement in tokens per word from a custom tokenizer. Those numbers matter because low-resource and morphologically rich languages often lose practical model capacity before the model even starts reasoning. Bad tokenization burns the context window.

What changed#

The AWS post describes a three-stage training framework for Azerbaijani rather than a single fine-tuning job.

First came tokenizer development. Azerbaijani is morphologically rich, so one word can carry meaning through suffixes that English might spread across several words. English-optimized tokenizers tend to fragment those forms into more subword tokens. AWS gives the example of Azerbaijani words being split in ways that reduce how much actual language content fits inside a fixed model context window.

The team evaluated three tokenizer approaches: baseline English-optimized tokenizers, vocabulary extension, and custom monolingual tokenizers. The custom monolingual tokenizer performed best in the source material. AWS says it reduced the average number of tokens per Azerbaijani word from 3.22 with the baseline Llama 3.2 tokenizer to 1.59. That is the reported 2× gain.

The tokenizer was trained on Azerbaijani text using Byte-Level Byte-Pair Encoding. That choice matters because byte-level training avoids needing a hand-built alphabet definition for Azerbaijani-specific characters. AWS says the team tested vocabulary sizes from 50k to 100k and selected 100k tokens.

Second came continued pre-training. The goal was to adapt a foundation model to Azerbaijani using distributed training and Liger Kernel optimizations inside Amazon SageMaker AI training jobs. AWS notes that distributed training was not required for the 1B-scale proof of concept, but would become important if Azercell scales to larger models.

Third came supervised fine-tuning with LoRA. Continued pre-training teaches the model to predict Azerbaijani tokens. It does not, by itself, make a useful assistant. LoRA was used as a parameter-efficient fine-tuning method to turn the adapted model toward conversational behavior.

The training jobs were launched from Amazon SageMaker Unified Studio and ran as SageMaker AI training jobs pointing to custom scripts. AWS says each job provisions fresh EC2 instances and terminates when complete, with training data and model artifacts stored in Amazon S3, metrics tracked through TensorBoard in SageMaker AI, and system metrics captured through CloudWatch.

Why training Azerbaijani changes the risk surface#

Training Azerbaijani models is not only a language coverage story. It changes what a telecom operator can automate, what customer data may enter model workflows, and which technical artifacts become security-relevant.

A customer-facing chatbot in a telecom setting can sit close to account workflows, support histories, billing questions, identity checks, device issues, and service complaints. The AWS post does not say what production data Azercell used or will use, and readers should not infer that private customer data was included in training. But the intended use case is enough to raise the operational question: where does language adaptation stop, and where does customer-specific processing begin?

The tokenizer itself becomes part of the system’s behavior. A poor tokenizer makes Azerbaijani expensive and lossy inside the model context window. A better tokenizer improves capacity, but it also becomes an artifact that must be versioned, tested, and tied to the model that expects it. If teams swap tokenizers casually, evaluation results can become misleading. AWS appears to treat the tokenizer as a first-stage artifact feeding later stages, which is the right operational shape.

The open source security angle is also real. The framework builds on PyTorch, Hugging Face Transformers, and Liger Kernels. That is normal modern AI engineering. It also means the training path inherits dependency risk, model artifact risk, container risk, and update risk. For teams using this as a template, open source security is not a separate checklist after the model works. It is part of the training pipeline.

GigaTap has covered the same pattern in software supply chain work: security artifacts only matter when they become operational. See also: OpenSSF’s April signal: make security artifacts operational and Open Source Security Needs More Than Code.

What to check before copying this approach#

The practical lesson is not “use SageMaker and a custom tokenizer.” It is to check whether your language, data, and operating model justify the extra moving parts.

Start with tokenizer evidence. For a low-resource or morphologically complex language, measure fertility score or an equivalent token-per-word metric against your target corpus. If the baseline tokenizer already behaves well enough, a custom tokenizer may add lifecycle cost without enough gain. If it fragments core vocabulary badly, tokenization can become the cheapest capacity win.

Then check quality with a metric that survives vocabulary differences. AWS used Bits-Per-Byte instead of perplexity when comparing models after continued pre-training, because BPB normalizes prediction quality at the byte level. That is a useful guardrail. Token efficiency alone can flatter a tokenizer if it hides a modeling-quality loss.

Security operations should treat each stage as an artifact boundary:

  • tokenizer training data and tokenizer version
  • base model and continued pre-training checkpoints
  • LoRA adapters and supervised fine-tuning data
  • training scripts and containers
  • S3 buckets holding data and artifacts
  • IAM permissions for SageMaker, EC2, S3, CloudWatch, and related services
  • logs and metrics that may expose dataset names, prompts, paths, or operational details

None of this is exotic. That is why it gets missed. AI training infrastructure often fails through ordinary cloud mistakes: broad permissions, unclear artifact ownership, weak dataset lineage, and logs that outlive their intended use.

Privacy risk deserves a separate check. If the model is headed toward customer support, define what data is allowed into continued pre-training, what is allowed into supervised fine-tuning, and what is only available at inference time under access controls. These are different risk classes. Mixing them makes later audits painful.

For teams building on open source components, pin dependencies, track licenses, record model and dataset provenance, and make rebuilds testable. A training run that cannot be reproduced is not production-ready just because it finished successfully. The same discipline applies to package coverage and test claims: the useful question is what the checks actually cover, not whether the dashboard looks complete. See: 100% package test coverage is the point, not the slogan.

What not to overclaim#

The AWS post is a vendor case study. It gives useful architecture and efficiency details, but it is not an independent benchmark of Azerbaijani LLM quality across all tasks.

The reported 23% throughput gain and 58% lower peak GPU memory use are tied to the described setup and kernel-level optimizations. They should not be read as universal SageMaker numbers. Hardware, model size, batch configuration, sequence length, and implementation details can move those results.

The 2× token-per-word improvement is more portable as a concept than as a guaranteed result. Azerbaijani morphology makes the tokenizer problem visible, and the reported baseline-to-custom improvement is substantial. Another language, corpus, or tokenizer baseline may produce a different gap.

The post also does not establish that a production chatbot is safe, compliant, or ready for every telecom workflow. Continued pre-training and LoRA fine-tuning are model adaptation steps. They do not replace red-team testing, privacy review, abuse handling, customer-data controls, or monitoring after deployment.

The strongest takeaway is narrower and more useful: for languages underserved by default model tooling, training quality starts before training. Tokenization, artifact lineage, cloud job boundaries, and dependency control shape the final system. Treat those as security operations work from day one, not cleanup after the demo.