A 3.35B-parameter model and a LangGraph agent walk into a support queue. What follows is a surprisingly competitive multilingual pipeline.
When every ticket is a language problem
Enterprise support does not happen in one language. A French engineer files a bug report with a full stack trace. A Turkish operations team submits a ticket about a failing backup schedule, complete with cron expressions and log excerpts. A Japanese developer describes a memory leak, referencing heap dumps and JVM flags.
Before anyone can triage, classify, or respond to these tickets, someone (or something) needs to understand them. And understanding a multilingual technical support ticket is harder than it sounds. You need to preserve error codes, API endpoints, SQL fragments, and the causal reasoning that makes a bug report actionable. Translate "le serveur renvoie une erreur 504" wrong and you lose the status code. Paraphrase a stack trace and you lose the debugging path entirely.
The standard approach is to route everything through a large model API, typically 30B+ parameters on a cloud endpoint. It works. It is also slow (10 to 30 seconds per request), expensive at scale, and creates a hard dependency on external infrastructure.
We wanted to see if we could build a complete multilingual support agent powered primarily by a 3.35B-parameter model. Language detection, translation, verification, classification, all running fast enough for real-time triage.
Enter Tiny Aya
When Cohere Labs released Tiny Aya in February 2026, it caught our attention. Most multilingual models at this scale treat language coverage as a checkbox: they support many languages, but performance degrades sharply outside English and a few high-resource languages. Tiny Aya does something different.
Rather than spreading thin, Tiny Aya covers 67 languages with instruction-tuned variants that specialize in different linguistic regions:
- TinyAya-Global: the generalist, balanced across all 67 languages.
- TinyAya-Earth: Africa and West Asia (Arabic, Turkish, Swahili, Amharic, Hausa, etc.)
- TinyAya-Fire: South Asia (Hindi, Bengali, Tamil, Urdu, etc.)
- TinyAya-Water: Asia-Pacific and Europe (French, Japanese, Chinese, Portuguese, Spanish, Korean, etc.)
This structure immediately suggested an architecture. Instead of picking one model and hoping it works everywhere, we could run the global model as a baseline while simultaneously running the specialized variant for the detected language. Two translations, one verification step, best output wins.
Worth noting: Tiny Aya's tokenizer was designed to reduce fragmentation across scripts. Where other multilingual tokenizers produce long token sequences for non-Latin scripts (slower inference, more memory), Tiny Aya gets significantly fewer tokens per sentence. That directly means faster inference and lower memory usage.
The agent architecture
We built the agent as a LangGraph workflow. Each node does one job and passes state forward. Here is the full pipeline:
START
├── fetch_ticket ──────────┐
│ ├── detect_language ──┬── translate_global ──┐
└── fetch_categories ──────┘ └── translate_sub ─────┤
├── verify_translation ── classify ── update_ticket
│ │
│ END
Eight nodes, three parallelization points, two model families.
Fetching context
The pipeline starts with two parallel operations: fetch_ticket retrieves the ticket from Jira, and fetch_categories queries the project for available category labels. Both use MCP tools to interact with Jira.
async def fetch_ticket(state: TicketState) -> FetchTicketResult:
tools = await load_mcp_tools()
tool = get_tool("jira_get_issue", tools)
result = await tool.ainvoke({"issue_key": state["ticket_key"]})
# Parse summary + description from the response
...MCP (Model Context Protocol) is an open standard for connecting agents to external tools. Instead of writing a Jira API client, we declare the tools we need and use them as standard LangChain tools. Agent code stays focused on business logic.
Language detection
detect_language uses TinyAya-Global to identify the source language, then resolves it to the appropriate regional variant:
_global_model = create_tiny_aya(TinyAyaVariant.GLOBAL)
def detect_language(state: TicketState) -> DetectResult:
response = _global_model.invoke(
[HumanMessage(content=f"What language is this? Answer in English: {state['ticket_text']}")]
)
# Match response against supported languages
language = next(
(lang for lang in _SUPPORTED_LANGUAGES if lang.lower() in raw.lower()),
None,
)
variant = get_variant_for_language(language)
return DetectResult(detected_language=language, sub_variant=variant)The routing map:
LANGUAGE_TO_VARIANT = {
"French": "water", "Arabic": "earth", "Hindi": "fire",
"Turkish": "earth", "Japanese": "water", "Chinese": "water",
"Portuguese": "water", "Spanish": "water", "Swahili": "earth",
"English": "global",
}Dual translation
After language detection, the pipeline forks into two parallel translation paths:
translate_global runs the ticket through TinyAya-Global. Broadest training data, most consistently formatted output across languages.
translate_sub runs the same ticket through the detected regional variant (Earth, Fire, or Water). Deeper linguistic grounding for its target languages, potentially better at capturing nuances.
Both run concurrently with the same system prompt:
_SYSTEM_PROMPT = (
"You are a professional translator. "
"Translate the following text to English. "
"Output ONLY the English translation, nothing else. "
"Do not add notes, explanations, or commentary."
)Cohere Labs built the regional variants for "stronger linguistic grounding and cultural nuance" while the global model provides a "consistent multilingual backbone." We wanted to see if combining both would beat either alone.
Verification
verify_translation receives both outputs and uses TinyAya-Global as arbiter. It picks the better translation, with a bias toward the regional variant when they conflict:
_USER_PROMPT = (
"Below are two English translations of the same text.\n\n"
"Translation A:\n{global_translation}\n\n"
"Translation B:\n{sub_translation}\n\n"
"Pick the better translation. If they conflict, pick Translation B. "
"Write ONLY the final English translation below, no explanation:\n\n"
"Final translation:"
)The bias toward Translation B reflects the idea that when two models disagree, the specialist's interpretation is more likely correct. As we will see in the eval section, that turned out to be more nuanced than expected.
Classification and update
Once we have a verified English translation, the pipeline leaves Tiny Aya's domain. classify uses Gemini 3 Flash to assign a category and priority (Critical/High/Medium/Low). We use Gemini here because classification is a reasoning task over English text, not a multilingual problem.
update_ticket writes everything back to Jira: detected language, variant used, verified translation, category, and priority. The original ticket is preserved, with the analysis appended.
The state
The pipeline communicates through a typed state dictionary:
class TicketState(TypedDict, total=False):
ticket_key: str
ticket_text: str
detected_language: str
global_translation: str
sub_translation: str
verified_translation: str
sub_variant: str
category: str
priority: str
categories: list[str]
updated: boolEach node reads what it needs and writes its results back. LangGraph handles routing, parallelism, and state merging. total=False means nodes only set the fields they are responsible for.
From notebook to production
Building a LangGraph agent that works locally is one thing. Running it as a production service with observability, access control, tool management, and a serving layer is another. Most teams spend more engineering time on this infrastructure than on the agent logic.
MCP tools and access control
Our agent talks to Jira: fetching tickets, looking up categories, writing back results. Without a standard protocol, you write a REST client, handle auth, parse responses, manage connections: a few hundred lines of integration code, minimum. An MCP server (mcp-atlassian, over STDIO transport) gives the agent jira_get_issue, jira_search, and jira_update_issue. Restrict which MCP tools each agent can reach as soon as several agents run with different permission levels.
For a support ticket agent processing customer data, input/output guardrails (PII detection, prompt-injection defenses, topic restrictions, allowlists/blocklists) are not optional, and neither are SSO and RBAC: the internal support team gets full access, external integrations only submit tickets without seeing raw translations.
Observability with Langfuse
Building the agent was the fun part. Figuring out why a ticket took 30 seconds instead of 3, or why a translation came back empty, was the hard part.
We went with Langfuse because it traces actual prompt/completion pairs, token counts, and cost per model call, not just spans and durations. Every node execution, model call, and tool invocation gets traced. You can drill into any span to see exact prompts, raw responses, token counts, and latency.
Some things we caught through traces:
- First translation call for each variant was much slower than subsequent calls. Turns out the HuggingFace Inference API was loading the model on first request. We started caching model instances.
- A ticket got a surprisingly low eval score. Tracing back through the spans showed the issue was in verification, not translation. The global model was mangling the sub-variant's output during comparison.
- The verification step was sometimes the slowest node, not the translation. The model was "deliberating" between two similar translations instead of quickly picking one.
Langfuse also gives aggregate metrics across executions: latency per node, error rates, token usage over time, cost breakdowns by model. When you are processing hundreds of tickets a day, that is how you know things are working.
Evaluating translation quality
Building the agent was the easy part. We needed to know how Tiny Aya actually performs on technical content compared to models many times its size, so we built a systematic evaluation.
The dataset
40 technical support tickets across 10 languages: French, Spanish, Portuguese, Arabic, Turkish, Hindi, Japanese, Chinese, Swahili, and English (4 per language). Each ticket includes the kind of technical detail you would find in a real support queue:
- Stack traces and error logs: Python tracebacks, Java exceptions, Node.js errors with file paths and line numbers that must be preserved exactly.
- SQL queries and database details:
SELECTwithJOINclauses, index names,VACUUM FULL, specific table/column identifiers. - API endpoints and configuration: REST paths like
/api/v2/batch/process, env vars, config keys, HTTP status codes. - Numbers that must survive: error codes, port numbers, memory sizes (3GB, 12GB), thresholds (100 requests), durations (24-hour window).
- Causal reasoning: "When X happens, Y fails because Z." Break this chain and the ticket becomes useless.
The tickets were written in each language by native-level prompts, not machine-translated from English. Machine-translated test data would test the model's ability to reverse its own output, which is not what we wanted.
The judge
We used Gemini 3 Flash as an LLM judge through deepeval's GEval metric. Each translation scored on a 0 to 10 rubric (normalized to 0 to 1) with six criteria:
- Numbers: are all numbers preserved exactly?
- Technical identifiers: API endpoints, file paths, config keys, SQL snippets preserved verbatim?
- Completeness: any sentences or details missing?
- Factual accuracy: does the translation introduce incorrect information?
- Fluency: would a native English support engineer find it confusing?
- Coherence: logical structure maintained (numbered steps, cause-effect)?
We paired these with explicit score anchors using deepeval's Rubric class:
- 0 to 2: Unintelligible, wrong language, or missing most content
- 3 to 4: General topic captured but major errors
- 5 to 6: Understandable and mostly complete, but several errors
- 7 to 8: Accurate and fluent with only minor issues
- 9: Near-perfect
- 10: Flawless
This mattered. In early iterations, we only used evaluation steps with penalty-based scoring and every model scored nearly perfectly. The rubric gave the judge calibrated expectations for what each score level actually looks like.
The competitors
We evaluated Tiny Aya against models from 4B to 32B parameters, all through the Hugging Face Inference API:
| Model | Parameters |
|---|---|
| TinyAya-Global | 3.35B |
| TinyAya-Routed | 3.35B (regional variants) |
| Gemma3n-E4B | ~4B |
| Qwen3-4B | 4B |
| Qwen3-8B | 8B |
| Qwen3-32B | 32B |
| Qwen3.5-27B | 27B |
"TinyAya-Routed" is our routing pipeline that sends each ticket to the appropriate regional variant based on detected language. This lets us compare global-only against specialized routing directly.
Each model translated all 36 non-English tickets. We parallelized with pytest-xdist across models and ThreadPoolExecutor within each model's ticket set (3 threads, staying within HuggingFace rate limits).
The results
| Model | Params | Avg Score | Avg Latency |
|---|---|---|---|
| Qwen3.5-27B | 27B | 0.981 | 26.2s |
| Gemma3n-E4B | ~4B | 0.972 | 4.5s |
| Qwen3-32B | 32B | 0.953 | 2.1s |
| Qwen3-8B | 8B | 0.950 | 4.3s |
| Qwen3-4B | 4B | 0.944 | 2.6s |
| TinyAya-Global | 3.35B | 0.922 | 1.7s |
| TinyAya-Routed | 3.35B | 0.858 | 2.9s |
TinyAya-Global scores 0.922, within 3 to 6% of models up to 10x its size, at the fastest inference speed in the comparison. At 1.7 seconds per ticket, that is roughly 2,000 tickets per hour. Qwen3.5-27B scores 0.981 but at 26.2 seconds per ticket, so about 137 per hour.
The efficiency gap is even wider:
| Model | Score/Second | Score/Param |
|---|---|---|
| TinyAya-Global | 0.546 | 0.275 |
| Qwen3-32B | 0.455 | 0.030 |
| Qwen3-4B | 0.362 | 0.236 |
| Qwen3-8B | 0.222 | 0.119 |
| Gemma3n-E4B | 0.214 | 0.243 |
| Qwen3.5-27B | 0.037 | 0.036 |
Highest score-per-second and score-per-parameter of any model tested. Nothing in the comparison is both faster and better.
Performance by language
| Language | TinyAya-Global | Qwen3.5-27B | Gemma3n-E4B |
|---|---|---|---|
| Arabic | 0.97 | 1.00 | 0.97 |
| French | 0.95 | 1.00 | 0.97 |
| Spanish | 0.97 | 0.97 | 1.00 |
| Hindi | 0.95 | 0.97 | 1.00 |
| Portuguese | 0.95 | 0.95 | 0.93 |
| Turkish | 0.90 | 1.00 | 0.95 |
| Japanese | 0.90 | 0.95 | 0.97 |
| Swahili | 0.88 | 0.97 | 1.00 |
| Chinese | 0.82 | 1.00 | 0.95 |
Strongest on Arabic (0.97), Spanish (0.97), French (0.95), Hindi (0.95), and Portuguese (0.95), all within 1 to 5% of the best model. The gap widens for Turkish (0.90), Japanese (0.90), Swahili (0.88), and Chinese (0.82). Chinese is the clearest weak point, 0.18 behind the best.
The consistency is the interesting part. Score distribution is tightly clustered. No catastrophic failures, no degenerate outputs. Every ticket produced a usable translation. For a production system, that consistency matters more than occasional peak scores.
Global vs routed
One of the more surprising findings. We expected routing to the regional variant would outperform the global model on its home languages. It mostly did not.
| Language | Variant | Global | Routed | Diff |
|---|---|---|---|---|
| Chinese | water | 0.82 | 0.90 | +0.08 |
| Hindi | fire | 0.95 | 0.95 | 0.00 |
| Swahili | earth | 0.88 | 0.88 | 0.00 |
| Arabic | earth | 0.97 | 0.95 | -0.03 |
| French | water | 0.95 | 0.93 | -0.02 |
| Spanish | water | 0.97 | 0.90 | -0.07 |
| Turkish | earth | 0.90 | 0.78 | -0.12 |
| Portuguese | water | 0.95 | 0.75 | -0.20 |
| Japanese | water | 0.90 | 0.70 | -0.20 |
Global scored 0.922 versus 0.858 for routed. The global model was more consistent, the routed approach more variable.
This comes down to task fit. The regional variants are designed for deeper linguistic grounding and cultural nuance, which matters for open-ended generation and culturally sensitive content. Our task is structured technical translation with strict output formatting. The global model's broader instruction-following training gave it an edge here.
The exception was Chinese, where routing to Water improved the score from 0.82 to 0.90. That suggests a middle ground: use global as default, route only the languages where it demonstrably underperforms. That is exactly what our dual-translation architecture enables.
Both configurations use a 3.35B model. The fact that we are debating routing strategies between variants of a model this small, with scores in the 0.85 to 0.92 range, says something about where small multilingual models are at right now.
What we learned
Small models can compete on real tasks. TinyAya-Global at 3.35B is within 3 to 6% of models 8 to 10x its size on technical translation. The gap is real but narrow. And the 15x speed advantage over larger models changes what is possible: 2,000 tickets/hour versus 137.
Consistency beats peak performance. TinyAya-Global never catastrophically failed across 36 translations. Every output was usable. The routed approach occasionally scored higher but also dropped as low as 0.10. In production, the system that consistently scores 0.92 is more useful than the one that alternates between 0.95 and 0.50.
Eval design is harder than it looks. Our first run gave every model near-perfect scores (rubric was too lenient). Our second run penalized harmless model preamble as "hallucination," scoring good translations at 0.05. It took several iterations to get scores that actually differentiated model quality. Spot-checking would never have revealed the routing underperformance, the Chinese exception, or the failure modes of each variant.
Infrastructure is undifferentiated work. Time should go to agent logic and eval design, not to building a serving layer or debugging Jira auth. An MCP server replaces hundreds of lines of API client code.
Model families are a design tool. Tiny Aya's structure (global backbone plus regional specialists) directly shaped our architecture. The dual translation step exists because the model family makes it possible. Routing underperformed global in our eval, but the architecture is sound. With different tasks (open-ended generation, culturally sensitive content), the regional variants' deeper grounding could make the difference.
Try it yourself
The code, eval suite, and results are all open source:
If you run this eval on your own data and get different results, we would be curious to hear about it.
