Large Transformer inference is increasingly memory-bandwidth bound rather than compute-bound. In autoregressive decoding, each token requires repeatedly reading the KV cache from memory, and this cost scales linearly with sequence length, layers, and head count. In long-context settings, the KV cache can rival, or exceed, the model’s parameter memory, making memory movement, not FLOPs, the dominant bottleneck.

This post introduces Low-Rank Key-Value (LRKV) attention, a drop-in modification to multi-head attention that reduces KV cache size by 45–53% vs standard MHA, while achieving lower test loss across model scales (128M → 6.3B), faster convergence in training steps, and stronger downstream performance after supervised midtraining.

The key idea is that attention heads are not independent. There’s structured redundancy across heads – yet fully sharing keys/values (like in MQA/GQA) can constrain expressivity. LRKV instead exploits redundancy using a shared full-rank KV basis plus head-specific low-rank residuals, yielding a continuous knob between complete sharing and full per-head independence

Why this matters: the KV cache bottleneck

In autoregressive decoding, each layer caches keys and values for all previously generated tokens.

For a sequence length L, number of heads H, per-head dimension d_h, and hidden dimension d =Hd_h, standard multi-head attention (MHA) caches, for each head, keys and values $$\mathbf{K_h}, \mathbf{V_h} \in \mathbb{R}^{L \times d_h}$$

So the KV cache memory per layer scales as:

$$M_{\text{standard}} = 2 L H d_h = 2 L d$$

Existing methods such as MQA (Multi-Query Attention) and GQA (Grouped-Query Attention) reduce cache size by sharing K/V across heads (or groups). This often improves throughput, but it forces heads to look through the same K/V representations, reducing representational diversity.

Empirically and theoretically, we know that heads specialize: different heads represent complementary syntactic and semantic patterns, and so fully sharing K/V across attention heads can degrade capabilities such as code generation and structured reasoning.

At the same time, we know that head specialization is not fully independent: recent analyses show high correlation and overlapping subspaces across head projections, so the redundancy is structured rather than random.

This motivates the central question:

Can we compress the KV cache by exploiting cross-head redundancy without comprimising head specialization?

This bring us to our main contribution, Low-Rank Key Value Attention (LRKV).

Background

Before discussing our proposed LRKV attention mechanism, we give a concise refresher on the well-established baselines that have been the mainstay mechanisms used in Transformers. If you are already familiar with Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped Query Attention (GQA) and Multi-Latent Attention (MLA), you can skip this section.

For Multi-Head Attention, each attention head maintains its own independent key and value projection matrices. Standard attention uses per-head key and value projections:

$$\mathbf{K}_h = \mathbf{X} \mathbf{W}_h^K, \quad \mathbf{V}_h = \mathbf{X} \mathbf{W}_h^V \quad \text{where} \quad \mathbf{W}_h^{K,V} \in \mathbb{R}^{d \times d_h}$$

The full projection matrix is formed by concatenating H independent weight blocks side by side, i.e. there is no parameter sharing whatsoever between heads. This gives each head complete freedom to learn specialised key/value representations, making MHA the most expressive configuration available. However, this expressiveness comes at a steep cost: the KV cache scales linearly with the number of heads (\(2Hd_h\) values stored per token), making it the most memory-intensive option during long-context inference.

All other attention variants can be understood as different strategies for reducing this cost while preserving as much of MHA’s expressiveness as possible.

Multi-Query Attention takes the most aggressive approach to KV cache reduction: all heads share a single key and value projection. Each head still computes its own query, preserving some capacity for diverse attention patterns. However, every head attends over the exact same keys and values. The KV cache shrinks from \(2Hd_h\) to just \(2d_h\) per token – a 75% parameter reduction that is transformative for long-context inference where cache memory is the primary bottleneck. 

But the trade-off is severe: because all heads attend over identical keys and values, the only way they can differentiate is through their query projections. In practice, heads tend to converge on similar attention patterns, reducing the model’s ability to capture diverse linguistic phenomena simultaneously. In fact, in our paper (see at the end), we quantitatively show that the query parameters are forced to diversify more compared to other less restrictive mechanisms because K and V are the same across heads.

Grouped-Query Attention offers a middle ground between MHA and MQA. Attention heads are divided into G groups, and all heads within the same group share a single key and value projection. In the example above, \(G= 2\): heads 1 & 2 share one projection while heads 3 & 4 share another. This allows groups to specialize in different roles. For instance, one group might focus on local syntax while another captures long-range dependencies. The KV cache reduces from \(2Hd_h\) to \(2Gd_h\) per token. In practice, G is typically set to H/4 or H/8, yielding a 4-8× cache reduction.

The key limitation of this approach is that sharing boundaries are fixed at architecture design time and cannot adapt to the data. Heads assigned to the same group must share representations regardless of whether their learned roles are compatible. This is a coarse-grained constraint that limits flexibility compared to methods that can adapt per-head.        

Introduced in DeepSeek-V2, Multi-Latent Attention takes a fundamentally different approach to KV cache compression. Rather than sharing projections between heads (like MQA/GQA), MLA compresses all key/value information into a single low-dimensional latent vector per token. During inference, only this compact latent is stored in the KV cache. When attention is computed, the full per-head keys and values are reconstructed on the fly via learned up-projection matrices. This decouples the cache cost from the number of heads entirely. 

This architecture works in two stages. First, a down-projection compresses each token into a latent of dimension \(d_c\) (much smaller than \(d\)). Second, separate up-projections reconstruct per-head keys and values from this shared latent. The effective key projection is an 8×8 matrix but has rank at most d_c. This resembles MHA’s full projection but lives in a lower-dimensional subspace.

There’s a fundamental trade-off here: heads can specialise (unlike MQA/GQA) – but they must reconstruct their keys and values from a shared compressed representation. Any per-head information that does not survive this bottleneck is permanently lost at inference time. We can compare this with LRKV, where each head’s low-rank correction is baked directly into the weight matrix – getting around the information bottleneck during inference.

Given this background on related attention mechanisms, we now move to explaining LRKV.

LRKV parameterization: shared full-rank base + per-head low-rank residual

LRKV factorizes each head’s key and value projection into a shared full-rank base plus a low-rank residual:

$$\mathbf{W}_h^K = \mathbf{W}_{\text{shared}}^K + \mathbf{U}_h^K (\mathbf{B}_h^K)^\top, \quad \mathbf{W}_h^V = \mathbf{W}_{\text{shared}}^V + \mathbf{U}_h^V (\mathbf{B}_h^V)^\top$$

Here W_shared is dense, full-rank, and shared across all heads in the layer,

$$\mathbf{U}_h^{K,V} \in \mathbb{R}^{d \times r}, \quad
\mathbf{B}_h^{K,V} \in \mathbb{R}^{d_h \times r}$$

Finally, \(r << d_h\) is the residual rank.

Our interpretation is that the shared base learns a global KV basis for the layer, and each head then learns a small low-rank deviation from that basis. This gives a continuous interpolation: \(r = 0\) reduces to full sharing of K/V within a layer (MQA-style limit) and increasing \(r\) increases head-specific capacity, moving toward MHA.

A subtle but important design choice: LRKV applies the factorization only to K and V (the cached parts). Queries remain unconstrained (per-head), preserving attention expressivity while targeting the true inference bottleneck.

KV caching in LRKV

At inference time, we want to avoid caching K_h and V_h for every head and every token. LRKV caches shared features once per layer:

$$ \mathbf{K}_{\text{shared}} = \mathbf{X} W_{\text{shared}}^K \in \mathbb{R}^{L \times d_h}, \quad \mathbf{V}_{\text{shared}} = \mathbf{X} W_{\text{shared}}^V \in \mathbb{R}^{L \times d_h} $$

Per-head latents:

$$\mathbf{R}_h^K = \mathbf{X} \mathbf{U}_h^K \in \mathbb{R}^{L \times r}, \quad
\mathbf{R}_h^V = \mathbf{X} \mathbf{U}_h^V \in \mathbb{R}^{L \times r}$$

Then the implied per-head features are:

$$ \mathbf{K}_h = \mathbf{K}_{\text{shared}} + \mathbf{R}_h^K (\mathbf{B}_h^K)^\top, \quad \mathbf{V}_h = \mathbf{V}_{\text{shared}} + \mathbf{R}_h^V (\mathbf{B}_h^V)^\top $$

Crucially, B_h are parameters, not cached per token. So the cache memory becomes:

$$M_{\text{LRKV}} = 2 L d_h \; (\text{shared } K,V)\quad 2 L H r \; (\text{per-head latents}) = 2L(d_h + Hr)$$

Relative to standard MHA:

$$\frac{M_{\text{LRKV}}}{M_{\text{standard}}}\frac{d_h + Hr}{H d_h}\frac{1}{H} + \frac{r}{d_h}.$$

This is the cleanest engineering knob LRKV provides: for fixed H and d_h, the residual rank r trades cache size for per-head flexibility.

Exact attention without explicitly reconstructing full K/V tensors

Naively reconstructing the full per-head keys and values for every cached token would erase the practical gain. Instead, LRKV computes logits and outputs exactly via associativity.

For a decoding step with query q_h the attention logits are:

$$\mathbf{q}_h \mathbf{K}_h^\top=\mathbf{q}_h\mathbf{K}_{\text{shared}}^\top+(\mathbf{q}_h \mathbf{B}_h^K)(\mathbf{R}_h^K)^\top, \quad \mathbf{q}_h \in \mathbb{R}^{d_h}$$

For attention weights a_h the value aggregation is:

$$\mathbf{a}_h \mathbf{V}_h=\mathbf{a}_h \mathbf{V}_{\text{shared}}+(\mathbf{a}_h \mathbf{R}_h^V)(\mathbf{B}_h^V)^\top, \quad \mathbf{a}_h \in \mathbb{R}^{1 \times L}$$

These expressions compute the same outputs as full reconstruction, but operate on smaller cached tensors:
$$\mathbf{K}_{\text{shared}}, \mathbf{V}_{\text{shared}} \in \mathbb{R}^{L \times d_h} \quad \text{and} \quad \mathbf{R}_h^K, \mathbf{R}_h^V \in \mathbb{R}^{L \times r}.$$

That is precisely where the memory win is realized.

What does LRKV cost in compute?

During decoding, standard MHA’s dominant per-head cost scales as:

$$O(L d_h), \quad \text{while LRKV adds } O(L r + r d_h).$$

For long contexts: $$L \gg 1 \quad \Rightarrow \quad O(L r) \text{ dominates, giving overhead } \sim \frac{r}{d_h}.$$

In modern inference, we’re often memory bandwidth bound, so reducing the bytes moved can dominate a modest FLOP increase. LRKV reduces total bytes read from cache proportionally to its cache reduction: it reads two shared tensors plus small per-head latents instead of full per-head K/V.

Results

With the design space now fully mapped, from MHA’s full independence, through MQA and GQA’s discrete sharing strategies, to MLA’s latent bottleneck and our proposed LRKV’s continuous low-rank interpolation, the natural question is: how do these architectural choices actually play out in practice? We evaluate all five methods under identical pretraining and midtraining conditions across three model scales (128M, 2.5B, and 6.3B parameters), measuring both pretraining loss and downstream task performance on five diverse benchmarks.

Cross-scale pretraining curves

Our results are very encouraging: LRKV reaches each baseline’s final validation performance 18-30% faster, averaging 23.6% training compute savings across all baselines while achieving better final performance. Critically, this reveals an asymmetric advantage: LRKV reaches any baseline’s performance target early in training, but no baseline reaches LRKV’s final performance (0.719 BPB) even after the full token budget.

• Across 128M → 6.3B, LRKV reaches lower test loss than MHA, MQA/GQA, and MLA, while using 45–53% of MHA KV cache.
• LRKV reaches equivalent baseline quality 18–25% faster (in training steps), i.e., better sample efficiency.

Training efficiency & memory/performance tradeoff

The residual rank r controls the tradeoff. The ablation shows monotonic improvement with larger rank, and a strong Pareto frontier relative to other KV-efficient methods.

LRKV appears to be not merely “a compression trick”, but a bias that improves optimization and/or effective capacity under the same token budget. Empirically, we see a consistent “useful rank” regime around: $$r \approx 0.36–0.43 \times d_h$$ as a threshold where LRKV matches/exceeds MHA while still delivering ~50% cache reduction.

Long-context pretraining

At longer sequence lengths, the benefits of KV-efficient attention become more pronounced. The below figure shows that LRKV not only maintains its advantage over standard MHA, but actually widens the gap in the long-context regime. At 8K context, LRKV achieves lower test loss while using roughly half the KV cache, outperforming both MHA and other KV-efficient baselines such as MQA, GQA, and MLA. This suggests that the low-rank decomposition is not merely compressing redundant structure, but acting as an effective inductive bias for long-range modeling. As context length increases and KV cache pressure becomes the dominant bottleneck, LRKV’s combination of memory efficiency and preserved head diversity translates directly into improved modeling performance.

Downstream Task Performance

A reasonable question is whether these gains turn into better downstream task performance. On a standardized evaluation after supervised mid-training, LRKV achieved the highest combined score across ARC, MMLU, GSM8K, and HumanEval, confirming that its improved pretraining efficiency translates into better downstream performance.

The below figure shows LRKV consistently achieves the highest combined accuracy at every scale – 18.9% (128M), 37.9% (2.5B), and 40.2% (6.3B) – demonstrating that the pretraining gains afforded by LRKV transfer reliably to downstream capabilities. At the 2.5B and 6.3B scales, LRKV leads on four of five benchmarks, with particularly strong margins on knowledge-intensive tasks: at 6.3B it surpasses the next-best method by +2.3% points on ARC-Easy, +4.4% on ARC-Challenge, and +1.8 on MMLU. Notably, MQA suffers a pronounced collapse on HumanEval at all three scales (2.4%, 3.7%, 4.3%). Crucially, the gap between LRKV and competing methods widens with scale, rising from +0.9 over MLA at 128M to +3.3 at 6.3B, reinforcing the conclusion that LRKV’s architectural advantages compound as model capacity increases.

Why does LRKV preserve head diversity (and why that’s non-trivial)?

A common failure mode of aggressive KV sharing is that heads lose the ability to represent distinct interactions. LRKV claims you can reduce KV cache without degrading diversity by preserving a shared basis plus low-rank head-specific deviations. We measure head diversity using gauge-invariant similarity metrics derived from bilinear forms:

$$\mathbf{A}_h = \mathbf{W}_h^Q (\mathbf{W}_h^K)^\top$$

We then compare the architectures via similarity matrices and effective-rank via eigenvalue entropy.

Effective rank across scales

We see LRKV exhibits very similar patterns to Standard MHA with sufficient rank of r=64 and achieves 98.3% effective rank at 2.5B scale versus 98.9% for Standard MHA. In contrast, MQA achieves only 86.2% and GQA 95.4%.

Interpreting uncentered vs. PCA-based effective rank.
The distinction between uncentered and PCA-based effective rank reveals LRKV’s factorization structure. Uncentered analysis measures total variance including the shared mean direction, the global structure captured by W_shared. PCA-based analysis centers the Gram matrix, isolating variance around this mean and measuring true head independence. The modest 4.8% gap indicates LRKV achieves diversity primarily through genuine per-head specialization rather than merely perturbing a dominant shared structure. For comparison, MQA shows dramatic improvement from uncentered (86.2%) to centered (91.0%), a compensation effect where forced KV sharing creates a strong mean direction, but heads recover diversity by aggressively diversifying query projections around this baseline.

Implementation notes

Cache layout
Store:

$$\mathbf{K}_{\text{shared}}, \mathbf{V}_{\text{shared}} \in \mathbb{R}^{L \times d_h}, \quad\mathbf{R}_h^K, \mathbf{R}_h^V \in \mathbb{R}^{H \times L \times r}$$

while B_h^K, B_h^V are parameters stored in the weights, not the KV cache.

Kernel fusion
The associativity forms used in LRKV are:

$$\mathbf{q}_h \mathbf{K}_h^\top = \mathbf{q}_h \mathbf{K}_{\text{shared}}^\top + (\mathbf{q}_h \mathbf{B}_h^K)(\mathbf{R}_h^K)^\top, \quad
\mathbf{a}_h \mathbf{V}_h = \mathbf{a}_h \mathbf{V}_{\text{shared}} + (\mathbf{a}_h \mathbf{R}_h^V)(\mathbf{B}_h^V)^\top$$

These allow exact attention computation to be embedded in fused kernels (e.g. FlashAttention) without reconstructing full K_h and V_h tensors.

Choosing rank r

Use the memory ratio:

$$\text{ratio} = \frac{1}{H} + \frac{r}{d_h}, \quad r \approx (0.36\text{–}0.43)\,d_h$$

Choose r to match your cache target, then validate quality.

Takeaways

LRKV is a structural change to attention that:

1. Targets the true production bottleneck (KV cache memory + bandwidth)
2. Exploits structured redundancy across heads
3. Provides a smooth knob (rank r) between MQA-like sharing and MHA-like independence
4. Empirically delivers a strictly better quality/efficiency frontier than common baselines, on analysis across a wide range of scales.

For full details and additional experiments, see the paper: Low-Rank Key-Value Attention (arXiv).

The post Low-Rank Key Value Attention: Reducing KV Cache Memory and Maintaining Head Diversity appeared first on /research.

We wanted to get the quality benefits of modern generative AI, and the scalability and reliability of machine learning based approaches.

This blog describes how we achieved the following progress:

• Topic modelling is good for extracting themes from large text corpora. However, support conversations are short, noisy, and informal, which poses a lot of unique challenges
• To address this, we adopted an embedding-based clustering approach with HDBSCAN (which avoids predefined cluster counts and adapts to each customer’s vocabulary, volume, and complexity)
• Using a bottom-up hierarchy, the system organically discovers emerging topics – such as new feature friction or early bugs – without requiring prior labels
• Finally, we layered generative AI on top of the unsupervised structure to name topics, summarise intent, and surface examples
• This hybrid approach transforms messy support conversations into actionable insights

Introduction

Support conversations are messy – tickets and messages flood in daily, full of critical feedback and pain points. They’re also fragmented and hard to summarise at scale. We wanted to turn this chaos into clarity: transform raw conversation data into a structured, actionable view of what customers are saying.

Our solution combines classic machine learning with generative AI to automatically detect and label topics within support conversations. These topics give teams a clear lens to understand user behaviour, spot issues early, and take action. Originally built for Fin, our AI agent, the system began as a feedback loop:

1. Detect topics with low AI performance (low resolution rate), frequent handoffs, or poor satisfaction
2. Analyse representative conversations
3. Act – update knowledge, add tasks, refine workflows
4. Measure results
5. Repeat

This cycle still powers how topics drive improvement, and is very useful. But we soon realised that topics were more than a debugging tool – they became a shared language across teams: support specialists, analysts, engineers, managers, and product marketers. Topics can be very useful for…

• AI Content Managers: to track which topics Fin handles well vs. those needing training or escalation (i.e. using performance data to prioritise improvements)
• Data Analysts: to monitor trends over time and user segments (e.g., perhaps refund issues are spiking; or there are more onboarding questions than expected post-release)
• Technical Support Engineers: to identify clusters of related bugs, measure scope, and track how long issues persist
• Support Managers: to see ticket volume by topic, to allocate support resources, and evaluate operational metrics across categories
• Marketing and Sales: to spot patterns in product questions to refine messaging, improve docs, and surface recurring feature requests
• …even more roles in future. Topics have proven flexible and interpretable, powering both day-to-day workflows and higher-level strategic decisions.

Pipeline Overview

Practical Challenge

Topic modeling has decades of research behind it. At its core, it finds abstract themes – or “topics” – within large text collections without labeled data. It’s been applied everywhere from academic literature and news clustering, to social media and network analysis.

However, conversational data is different. Support chats and assistant transcripts are short, noisy, and informal – far from the structured documents most models were built for. Our challenge was to adapt topic modelling to this messy reality and make results both accurate and useful for business teams. We also had to ensure it could scale.

Choosing the Right Modelling Approach

We treated this as a feasibility analysis – not just a technical evaluation, but a product development challenge. The right model needed to balance accuracy, interpretability, performance at scale, and business usability:

After exploring approaches mentioned above, we ultimately chose embedding-based clustering because it struck the right balance between flexibility, scalability, and performance. Unlike rule-based methods or supervised classification, it allowed for fully unsupervised exploration – critical for surfacing unknown or evolving topics without relying on predefined labels.

In our initial tests, traditional models like LDA or NMF struggled with short, noisy text, while embedding-based methods handled conversational nuance far better. Just as importantly, they scaled easily to millions of conversations. When we tested it, it just worked – producing coherent, interpretable topics out of the box.

So we made a bet on this method, and it paid off!

How It Works: A High-Level View

Each customer gets a tailored topic model that breaks their conversations into distinct themes. Topics and subtopics are linked to key metrics like CSAT and resolution rate, and teams can drill down to review individual conversations or apply filters to explore trends. Here is how it looks in the UI:

Here’s a simplified view of how it works under the hood:

The upper part of the picture is subtopic discovery. It is based on BERTopic framework, and here’s what it does:

1. Extract key questions – A lightweight LLM summarises the main questions from CS chats
2. Embed questions – A sentence transformer converts key questions into vector embeddings (we settled on sentence-transformers/all-MiniLM-L6-v2, based on speed and quality)
3. Reduce dimensions – UMAP (particularly effective for short-text data) projects the embeddings into a lower-dimensional space
4. Cluster into subtopics – HDBSCAN groups the reduced embeddings into coherent clusters/subtopics

The rest is topic aggregation:

1. From subtopics to topics – Clustering on subtopic embeddings surfaces higher-level, tentative topics
2. Refine with LLM – An LLM polishes tentative topics into a definitive set of final topics
3. Name topics & subtopics – Another LLM generates clear, human-readable labels, based on the final topics
4. Compute centroids – Average embeddings define centroids for each cluster (topic/subtopic)

Why HDBSCAN Beats K-Means for Conversational Data

A pretty natural question is why didn’t we use well-known K-Means for clustering?

Overall, we found embeddings and dimensionality reduction methods had only marginal impact, while clustering algorithms and hyperparameters made the biggest difference in output quality. The main advantage of HDBSCAN for our purposes is that it doesn’t require a predefined number of clusters. That flexibility was essential – each customer needs a unique topic model, and support volume, vocabulary, and conversation style vary widely (which results in very different number of topics for different customers).

The key parameter we tuned was min_cluster_size, which sets the smallest number of messages that can form a cluster. Lower values create more granular clusters; higher values produce fewer, broader ones. We settled on 15 as the minimum, which for our own Intercom data surfaced more than 700 micro-topics – a reflection of how diverse support conversations can be. Cluster counts also scale naturally with company size: more traffic means more clusters. This reinforced our intuition that fixing the number of clusters upfront wasn’t an option.

Here’s a relationship between company size (measured by 3-month conversation volume) and number of discovered subtopics. Larger companies with more diverse customers naturally yield more granular clusters – validating our choice of HDBSCAN’s adaptive approach over fixed cluster counts:

Another advantage of HDBSCAN over K-Means is how it handles noise. K-Means forces every data point into a cluster – even outliers – and offers no confidence scores. HDBSCAN, by contrast, assumes noise exists and can explicitly flag it. In customer support, many one-off or unusual questions don’t fit neatly into a topic. HDBSCAN’s ability to mark these as outliers, with associated scores, is invaluable for managing such edge cases.

From Subtopics to Topics: Building a Bottom-Up Hierarchy

Now let’s get to the second part of the pipeline: topic aggregation.

Many systems define topic hierarchies top-down: decide the categories first, then slot everything in. That works if you already know what to expect – but support data is full of surprises. We wanted the opposite: a bottom-up approach where the data speaks first, surfacing themes customers may not anticipate. The process starts with fine-grained subtopics, discovered via embedding-based clustering with HDBSCAN (as described above). Each subtopic captures a narrow slice of intent:

– Message: “What is the price for resolutions?”
→ Subtopic: Fin resolution rate

– Message: “What model does Fin use under the hood?”
→ Subtopic: Fin LLM usage

While different in detail, these subtopics both belong to the broader Fin topic. The challenge was: how do we automatically group subtopics into meaningful parent topics?

We first tried HDBSCAN’s hierarchy features, but the results were too coarse and inconsistent. So we reapplied our pipeline – this time clustering subtopic embeddings (mean sentence vectors). To our surprise, it worked: fine-grained Fin-related subtopics clustered under “Fin,” and so on. This recursive clustering produced flexible, two-layer hierarchies that adapt to each customer’s data – without predefined categories.

Still, the second clustering wasn’t perfect. To refine it, we added an LLM step: cleaning up inconsistencies, correcting edge cases, and generating clear, human-readable topic names. We layered generative models on top of the unsupervised structure so each cluster becomes understandable and actionable. The LLM helps by:

• Flagging spam or irrelevant data
• Naming subtopics and topics
• Summarising clusters in plain language
• Highlighting representative examples

This hybrid approach gives us the best of both worlds: the scale of ML with the fluency of generative AI.

Conversational Data

Traditional topic modeling was built for long-form, structured text – like news articles or research papers. But conversations don’t follow those rules. They are:

• Short & fragmented – messages like “It’s not working” or “login fails” lack standalone context
• Informal & messy – typos, abbreviations, emojis, and non-standard grammar are the norm
• Multi-turn & multi-speaker – dialogues jump between issues across several back-and-forths
• Contextual & implicit – meaning often depends on prior messages or hidden intent

In other words, applying standard NLP pipelines to chats is like trying to summarize a movie from random one-liners. We had to rethink preprocessing, clustering, and even how we evaluate topic quality.

Preprocessing

Greetings, disclaimers, brand names, agent scripts, typos, and pleasantries can easily swamp the real signal. Without preprocessing, clustering just produces giant, meaningless groups like “Hi, I have a question.” So we built a preprocessing pipeline that goes beyond standard NLP cleanup:

1. Remove brand names & boilerplate. Frequent mentions (e.g., “Intercom”) or scripted phrases skew clusters, creating vague, oversized groups that add no real insight
2. Extract main messages & collapse turns
   – A lightweight LLM pinpoints the messages that carry intent – what the customer wants, the issue they face, or their reaction
   – When individual turns are too short or vague, we combine related ones to form richer inputs – especially helpful when intent unfolds gradually across a chat
3. Spam filtering. We use a lightweight LLM to filter out irrelevant conversations that aren’t related to customer support from our training data
4. “Classic” text cleaning. Lowercasing, punctuation handling, stop word removal, typo normalisation, emoji stripping (or translation), etc.

Most of these are self-explanatory, but here’s an example of what step #2 does:

Inference

Our inference strategy was heavily inspired by BERTopic. Instead of running the full sequence of trained models (Embeddings → UMAP → HDBSCAN) for each new conversation, we opted for a simpler, faster, and more production-friendly method: centroid-based inference.

1. Cluster representation. After training, each discovered cluster is represented by a centroid – the mean embedding of all conversations in that cluster
2. New conversation processing. At inference time, we:
   – Assign the conversation to the closest cluster, based on the highest similarity score
   – Embed the new conversation using the same sentence transformer
   – Compute cosine similarity between the new embedding and all cluster centroids
3. Confidence thresholding. We define a minimum similarity threshold. If no centroid crosses this threshold, the conversation is considered out-of-distribution and flagged as a potential new or unknown topic.

Switching to centroid-based representations gave us high-throughput, real-time inference with a compact model. But it came with risks. Unlike HDBSCAN, which supports arbitrary cluster shapes, centroid methods assume clusters are roughly spherical. That’s not always the case – some clusters are elongated or multi-modal, so the centroid may sit outside the densest region, reducing accuracy. To check this tradeoff, we analysed cluster geometry to confirm they were compact enough to make centroids a reliable shortcut.

Our geometry analysis confirmed the trade-off was worthwhile: clusters were compact enough for centroids to serve as reliable shortcuts; click to minimise this section, if you want to skip this detail.

To validate that centroids could reasonably represent our clusters, we needed to check if those clusters were roughly spherical – i.e., compact, well-distributed around a center, and not elongated or fragmented. We used Principal Component Analysis (PCA) and spectral entropy as geometric proxies.

Here are the metrics we calculated for every cluster of multiple customer topic models:

• PCA Aspect Ratio: The ratio between the first and second PCA eigenvalues, measures how elongated clusters are
• PCA Top Variance Ratio: Indicates how much variance is captured by the first principal component
• Spectral Entropy: Quantifies the distribution of eigenvalues, measures how evenly variance is distributed across all components

We performed this analysis across multiple clusters to validate the centroid-based approach, and it showed promising signs:

PCA Aspect Ratio & Top Variance Ratio mostly fell in the expected range for spherical clusters (aspect ratio 1–2.5; variance ratio >0.35). Spectral Entropy was more mixed: most clusters scored 0.6–0.8 – egg-shaped rather than round – with some above 0.8, indicating near-spherical shapes.

In practice, this moderate anisotropy didn’t prevent centroids from being effective, especially when clusters were well-separated. 

We also visually inspected the clusters to verify their shape, density, and cohesion. While not all clusters are perfectly spherical, both the metrics and visual inspections of samples confirmed that most are compact and coherent enough for centroids to act as reliable representatives:

Does it hurt performance? In short – no. To compare inference methods, we used LLM-as-judge: asking an LLM to evaluate whether each subtopic–topic pair accurately captured the theme of a conversation. It’s not a strict system metric, but a useful proxy for human judgment. The results were clear: centroid-based inference performed about 2 points better on average than the full pipeline, with both approaches exceeding 85% accuracy. In other words, simplifying didn’t just hold up – it slightly improved results.

Outlier reduction: switching to centroids cut outliers by 50%, boosting topic coverage. This mattered because support data mixes dense clusters with diffuse, low-density groups. HDBSCAN favors well-defined clusters and often discards the rest as noise – even when those “noisy” points are meaningful. Centroids let us reassign many of those borderline cases to valid topics, improving recall without sacrificing quality.

To balance performance and flexibility, we split the pipeline into two stages:

• Discovery with HDBSCAN – unsupervised exploration to uncover new topics without preset limits.
• Inference with centroids – a fast, similarity-based method optimised for scale

Evaluating Topic Quality

Evaluating topic modelling is inherently difficult. As well as being an unsupervised task, it is also inherently ambiguous. There are many valid ways to split conversations into clusters, and the “best” result often depends on human judgment and downstream usability rather than strict metrics. Still, every ML system needs some form of evaluation. We approached it from multiple angles:

• Clustering quality – Are topics well-separated and coherent?
• Prediction accuracy – Are new conversations assigned correctly?
• Human validation – Do these topics make sense to real users?

Next, we’ll walk you through how we assessed clustering quality, prediction accuracy, and human validation. If you’d rather skip the technical deep dive, feel free to skip  –  the key takeaway is that our evaluation confirmed the system’s clusters are both coherent and practically useful.

Clustering quality

To evaluate cluster geometry in embedding space, we focused on two cosine-distance metrics:

• Mean Inter-Centroid Distance (Separation). Measures the average distance between cluster centroids. This is critical for centroid-based inference, where close centroids can cause unstable topic assignments
  • Higher = better separation → topics are distinct, with less semantic overlap
  • Lower = weaker separation → clusters risk blending, making assignments less reliable
• Mean Intra-Cluster Distance (Cohesion). Measures the average distance between messages and their cluster centroid. This shows whether a centroid is truly representative of its cluster
  • Lower = stronger cohesion → messages are compact and consistent
  • Higher = weaker cohesion → clusters may contain noise or multiple subthemes

Together, these metrics capture the classic trade-off: clusters should be internally cohesive yet externally distinct – a must for our centroid-based approach.

Here’s a plot of these two metrics, with each data point being an Intercom customer:

Mean inter-centroid distance falls mostly between 0.6 and 0.8, averaging around 0.75, suggesting strong topic separation. Mean intra-cluster distance ranges from 0.0 to 0.35, with a typical value around 0.2, indicating good cohesion in most cases. We did observe a few outliers – models with lower separation and/or weaker cohesion. These may reflect noisier datasets or edge cases with very small or highly variable support volumes. However, the majority of models fall within an acceptable trade-off zone, showing that our unsupervised clustering approach generalises well across different customers with diverse conversation sets.

Beyond metrics, we ran sanity checks to confirm clusters behaved as expected across datasets:

• Noise Points (Outliers): HDBSCAN flagged ~10–15% of conversations as outliers – reasonable for noisy support data. Too many (50–70%) would suggest the model is overly strict, while too few might mean clusters are too broad
• Uneven Cluster Sizes: another failure mode is one giant cluster swallowing most data, leaving only tiny ones behind – usually a sign of overly permissive parameters
• Cluster Size Distribution: in practice, we saw a J-shaped distribution: many small, focused clusters and a few large, high-volume ones. This matches expectations – most issues are niche, while a handful dominate support traffic:

Prediction accuracy

It’s worth emphasising the distinction between cluster discovery and cluster assignment:

• Discovery is about in-sample structure – how well the model can group historical conversations into meaningful topics using unsupervised learning
• Prediction is about out-of-sample generalization – how accurately we can assign new, unseen conversations to the right existing topic.

This framing also highlights one of the most important challenges: deciding when a new conversation doesn’t fit any existing topic. That’s where the similarity threshold plays a key role. If enough conversations consistently fall below that threshold and form a dense region of their own it might signal the need to form a new topic cluster entirely.

To optimize the decision boundary, we inspected the distribution of cosine similarity scores. A threshold of 0.5 provided a good trade-off between precision and recall: above 0.5 = assign to closest cluster; below = flag as unassigned.

To validate the quality of topic assignments, we used LLM-as-a-judge – a lightweight evaluation method where an LLM was asked: “Does this topic match the conversation theme?”

The model returned a True/False answer, serving as a semantic proxy for accuracy in the absence of ground truth labels. This approach simulates human evaluation and helps assess how well the assigned topic captures the user’s actual intent. We also used this method to cross-validate our cosine similarity threshold. Based on these judgments, we found that a threshold of 0.5 struck the right balance between precision and recall. Overall, assignment accuracy exceeded 85%.

Unassigned: outliers or a new topic? When a new conversation receives a cosine similarity score below 0.5, we assume it doesn’t belong to any existing cluster – and it’s flagged as unassigned. However, that doesn’t always mean the conversation is noise or a rare edge case. Sometimes, it reflects something more important: a new topic that has recently emerged and wasn’t present in the data when the original clusters were created. These unassigned conversations can be early signals – new issues, product questions, or behavioural patterns that haven’t yet been captured by the system. More on how we detect and promote new topics in the next section (Model Updates).

Multi-topic aspect and borderline assignment. We also explored the multi-label nature of conversational data – since it’s well understood that a single conversation can touch on multiple topics. There are two main reasons for this:

1. In longer conversations, the topic can naturally change over time. To handle this, we extract multiple key messages per conversation, allowing us to capture more than one topic when necessary. For example, if a chat starts with Fin pricing and then moves on to a bug in conversation assignment, our system will spot and tag both topics separately
2. One message can genuinely fit more than one topic. For instance, “What’s the difference between Fin AI Agent and Fin AI Copilot?” belongs to both categories in the absence of a dedicated “comparison” topic. Such semantic overlap is common in real-world language, especially in support data where questions blend multiple intents. Occasionally, overlap instead signals redundancy – two clusters describing the same issue – which can be spotted through small inter-centroid distances.

To quantify ambiguity, we analysed secondary and tertiary topic matches via cosine similarity. Only about 5% of messages showed less than a 0.05 gap between their top two or three topic candidates – meaning true borderline cases are rare. While clearer separation between clusters reduces edge cases, genuine multi-intent conversations will always resist perfectly clean boundaries.

Human validation

No matter how good the metrics look, topics must make sense to people. Cohesion and separation scores are helpful, but they’re meaningless if clusters don’t reflect how users actually interpret their data.

Quantitative metrics can’t fully capture subjective structure – especially hierarchies. There’s no single “right” way to group subtopics: we cluster by semantic similarity (e.g., all “Fin”-related items under one topic), while some users prefer grouping by issue type, like bugs or pricing, across products.

To stay grounded in real-world needs, we interviewed customers and gathered qualitative feedback on topic clarity and usefulness. This feedback loop remains central to our process – because in the end, quality is measured by how well the system helps people do their jobs.

How to Update the Model?

Emerging topics

Unsupervised models are typically static – they’re trained once and can’t easily adapt to new data. But support data evolves constantly: new products launch, features change, and fresh themes appear. A model trained weeks ago won’t recognize today’s conversations. This causes two main issues:

1. Misassignment – new topics get forced into old clusters
2. Unassignment – new conversations fall below the similarity threshold (e.g., cosine < 0.5) and remain unlabelled

To address this, we built a daily pipeline that detects emerging topics and incrementally updates the model. Thanks to the centroid-based architecture, adding a new topic is simple – just introduce a new centroid.

This is how we identify new topics:

1. Discovery on new data. We apply the same topic modelling pipeline (embedding → UMAP → HDBSCAN) to a combination of new unseen support conversations (out-of-sample), and unassigned conversations from recent inference runs
2. Centroid comparison & deduplication
   – We compare the centroids of these newly discovered clusters against existing centroids to check for similarity
   – We retain only the clusters that are well-separated from existing topics – ensuring they represent genuinely new themes.
3. Update the model. The filtered centroids are simply appended to the existing centroid list – effectively extending the model without retraining. This process lets us keep the topic model fresh and adaptive, without sacrificing scalability or interpretability.

Archiving topics

Just as new topics emerge, others naturally fade. Over time, two patterns appear:

• Recurring topics – persistent themes like pricing, cancellations, or onboarding that remain active and should stay in the model
• Event-driven topics – short-lived spikes tied to launches, campaigns, or temporary bugs

When these transient topics go quiet, we archive them. Centroids for inactive clusters are removed, reducing noise and keeping the model aligned with current conversation trends.

Conclusion

We’ve designed the system with real-world constraints in mind: messy input data, changing user behavior, and the need for fast, high-throughput inference. With centroid-based inference, ongoing model updates, and human-in-the-loop validation, we’ve built a flexible, production-ready system that evolves alongside the conversations it’s built to understand.

Now, topics are no longer just labels – they’re a lens into customer reality. And with the right mix of structure and language, that lens becomes a powerful tool for action.

The post Unsupervised Learning Meets Generative AI: Topic Modelling for Real-World Dialogue appeared first on /research.

Pratik’s workstream has really been at the cutting edge of using LLMs to take actions within business, and all the complexity that’s required to make that work, delivering our Tasks product.

This is a wide ranging and candid discussion about exactly what’s required to make this work and all the considerations with having LLM agents actually take consequential actions, and how this gets integrated into a business.

You can follow the Fin AI podcast on Apple Podcasts, Spotify or anywhere you get your podcasts.

The post Podcast EP2: Shipping reliable AI actions appeared first on /research.

We discussed the purpose and rational behind building the Insights product; Fedor’s application of Causal Analysis to understand its impact without ab-testing, and his journey to Intercom and ideas about the future.

Relevant blog posts are this one on generating suggestions to improve the content powering the RAG system, and Fedor’s approach to causal analysis.

You can follow the Fin AI podcast on Apple Podcasts, Spotify, RSS or anywhere you get your podcasts.

The post Podcast EP1: Closing the loop appeared first on /research.

In this setup, a reranker plays a crucial role: after retrieving potential answers from our knowledge base, the reranker reorders them by relevance to help Fin choose the best content to include in its reply.

We built our own reranker that outperforms Cohere Rerank v3.5, an industry-leading commercial solution. This improved our answer quality, reduced reranking costs by 80%, and gained more flexibility to evolve our system.

Fin’s RAG Workflow

Here’s how Fin uses RAG at a high level.

When someone asks Fin for help, Fin starts by summarizing the conversation into a short, focused query, like “How do I reset my password?” or “Where can I find my invoices?”. This query is used to search the knowledge base, where all help articles and snippets are pre‑processed into vector embeddings for efficient retrieval.

Fin compares the query embedding to these vectors to find the closest matches. It then takes the top \(K=40\) candidates and re-ranks them using a specialized reranker model. Initial vector retrieval is fast, but can miss nuances, so the reranker uses deeper context understanding to reorder the passages by relevance.

Finally, a context budget filter selects the top-ranked passages, and Fin uses these to craft a clear, accurate answer for the user in real-time.

Why Build Our Own Reranker?

Previously, we relied on Cohere Rerank-v3.5, a commercial reranker offering high-quality results but incurring substantial costs. Previously tested open-source models (BGE-large and BGE-m3) couldn’t achieve required performance levels, and using LLM-based reranker caused latency issues.

To address these challenges, we decided to develop our own reranker tailored specifically to the domain of English customer support. Our objectives were clear and ambitious: match or exceed Cohere’s quality, run efficiently on standard GPUs, and reduce vendor dependency.

Fin-cx-reranker: Our Custom Solution

Our custom reranker uses ModernBERT-large (2024) as a component. This is a state-of-the-art encoder-only transformer designed specifically for retrieval and classification tasks. ModernBERT supports an 8,192-token context window (vs. 512 in vanilla BERT), employs rotary/relative positional encodings, GeGLU activations, efficient attention, and was trained on ~2T tokens. It consistently surpasses encoders like BERT, RoBERTa, and DeBERTaV3 across benchmarks such as BEIR and GLUE by 3-8pp.

For scoring candidate passages, we concatenate each query and passage pair as [CLS] {query} [SEP] {passage[i]} [SEP] and feed this into ModernBERT. We then apply mean pooling across all token embeddings (excluding padding) to obtain a single vector. This vector passes through a linear layer, producing a final relevance score used for ranking.

Training Details

We trained the reranker on 400,000 real Fin queries, each with \(K=40\) candidate passages, 16M pairs in total. Labels were provided by an LLM-based pointwise reranker, giving us high-quality training signals.

Our implementation uses Hugging Face Transformers. To optimize ranking, we employ a RankNet loss. The teacher LLM first sorts the \(K\) passages by relevance, assigning each passage a rank \(r_i\) where a lower number means higher relevance (e.g., \(r_i=1\) means top-ranked). The model then produces a score \(s_i\) for each passage.

Training minimizes the following over all ordered pairs where the LLM says passage \(i\) should outrank \(j\):
$$
{\mathscr{L}}_{\text{RankNET}} = \sum_{i=1}^{K}\sum_{j=1}^{K} \mathbf{1}[r_i < r_j]\,\log(1 + \exp(s_j – s_i)).
$$
Equivalently, this runs over the \(\frac{K(K-1)}{2}\) ordered pairs \(i<j\) with \(r_i<r_j\):
$$
{\mathscr{L}}_{\text{RankNET}} = \sum_{i < j,\,r_i < r_j} \log(1 + \exp(s_j – s_i)).
$$
By penalizing cases where a lower-ranked passage scores higher than a higher-ranked one, the model learns to follow the correct order. This pairwise objective helps it judge passage relevance better and leads to smooth, stable convergence.

Evaluation

To confidently establish the superiority of Fin-cx-reranker, we used a rigorous three-stage evaluation funnel: from controlled offline tests to live production traffic.

FinRank-en-v1: Offline internal benchmark:

We built an internal static evaluation set with 3,000 real English queries sourced from 1k+ customer apps, each paired with 40 candidate passages. “Ideal” ground-truth rankings come from a two-stage LLM oracle. For queries with a confirmed (hard) resolution, passages cited by Fin were moved to the top. This setup allows us to directly compare models using classic information retrieval metrics: MAP, NDCG@10, Recall@10, and Kendall tau.

Backtesting production conversations

We sampled 1,500 recent support conversations from 685 apps and ran them through a frozen RAG pipeline, measuring precision / recall for cited passages appearing in the first 1,500-token context window Fin uses. This stage also checks how well the model generalizes to out-of-distribution apps not seen during training.

Online A/B testing:

We ran a two-arm, 1.5M-conversation A/B test, resulting in no change in latency (P50 ≈150 ms), but a statistically significant improvement in Resolution Rate over Cohere Rerank‑v3.5 (p < 0.01). We do not share the exact Resolution Rate effect size, for competitive reasons.

What’s Next

Bringing reranking capabilities in-house through Fin-cx-reranker has proven to be a clear win. We’ve improved answer quality, reduced costs on reranker by 80%, and gained more control to keep evolving the system. Our experience highlights that targeted, domain-specific models can indeed outperform top commercial solutions.

Looking forward, we see clear opportunities to enhance performance further. We’re working on refining label quality by re-annotating with stronger models, and extending our reranker beyond English. These initiatives are already in progress.

The post How We Built a World-Class Reranker for Fin appeared first on /research.

A key part of this process is reranking, which reorders the results from vector search so the final answer is grounded on the most relevant passages. Open-source cross encoder models are a popular choice for this because they are fast and easy to use. But in our experience, they don’t hit the quality bar we need.

In this post we share how we deployed an LLM-based reranker and the engineering needed to make it 5x faster while staying reliable on production traffic. We also show how we’ve applied it in our Fin and Copilot Agents and reveal the prompt we used. LLM reranker also guided the training of a custom reranker for Fin, which we describe in a companion post.

The Core Idea

Vector search returns the top-\(K\) passages from the index based on the user’s query. The LLM reranker then scores these passages to decide which are most relevant. 

There are three ways to prompt the LLM for reranking [arxiv]:

• Pointwise reranking: ask the LLM to rate how relevant each passage is on a scale from 1 to 10. Example output with passage ids: [("id0", 6), ("id1", 10), ("id2", 5), ("id3", 10), ("id4", 4)]
• Listwise reranking: ask the LLM to order the passages by relevance. Example output: "id1" > "id3" > "id0" > "id5" > "id2" > "id4"
• Pairwise reranking: build a ranking through pairwise comparisons, asking the model which of two passages (\(p_i\) or \(p_j\)​) is more relevant to the query. If you implement the LLM as a comparator inside a sort, you typically need \(O(K \log K)\) or \(O(K^2)\) comparisons, making it the most expensive. Studies often find pairwise prompting best on quality, though at higher cost [arxiv].

We went with pointwise reranking since it gives clear scores that are easy to use and allows useful optimizations.

Now, if you’ve got a lot of passages (we use \(K\) = 40), the naive version runs into a few problems we saw in the first iterations:

• The number of output tokens becomes large, which hurts latency
• The LLM sometimes misformats output or mis-scores (duplicate ids, missing ids, etc.)
• The inputs get large: 40 passages × ~200 tokens is ~8k tokens just for the passages (this doesn’t include the prompt!)
• LLMs can also be sensitive to input’s order [arxiv] and show positional bias [arxiv]

We improved this by (a) reducing output tokens and (b) parallelizing the reranker.

Reducing output tokens

When cutting latency, reducing output tokens is a good first step: latency gain is roughly proportional to how much you cut.

We landed on two optimizations:

• Removed spaces (surprisingly expensive tokens!) and switched to a Dict format instead of List[Tuple].
• Added thresholding: instructed the LLM to omit passage ids if their score is below 5.

The first change cut output tokens by ~28%, and the second brought another ~50% latency drop.

We also tried dropping the “id” token to save ~20% more latency, but it didn’t work. The LLM started confusing passage indexes with scores (both are integers in a similar range), so quality dropped.

Parallel Reranking

To improve both speed and accuracy, we split the \(K\) candidate passages into \(N\) batches and score them in parallel. For example, with \(K = 40\) and \(N = 4\), each worker gets 10 passages with the same prompt. This keeps inputs and outputs small, which improves speed and quality.

Batching strategy. Vector search already imposes an ordering bias (higher semantic similarity first). If you naively split passages into consecutive chunks, for example the first \(\frac{K}{N}\) for worker 1, the next \(\frac{K}{N}\) for worker 2, etc., you’ll overweight the first shard with the “best” candidates. We instead assign round-robin by index so each batch sees a similar mix of high/medium/low similarity passages: \(B_j = \{\, p_t \mid t \bmod N = j \,\}\). So with \(N=4\), the batches are \(B_0=\{p_0, p_4, \ldots\}\), \(B_1=\{p_1, p_5, \ldots\}\), and so on.

Merge step. We pool all scored items, sort by the LLM score, and use the BGE cross-encoder to break ties or fill missing scores.

Calibration. Parallelism comes with a consistency risk: workers run independently and may drift in how they grade. We address this by adding a clear grading rubric to the instructions and anchoring it with a small few-shot example set. This keeps all workers on the same scale so their scores stay comparable.

Latency & reliability. End-to-end latency is limited by the slowest worker, so we set tight per-call timeouts and skip retries to avoid long tails. If one call times out with probability \(p\), then with \(N\) parallel calls, the chance that at least one times out is roughly \(1 – (1 – p)^N\), which grows as \(N\) gets larger. This estimate is optimistic because it treats calls as independent, and in practice the failure rate increases under concurrent load.

If a shard does time out, we fall back by reranking that slice with the BGE cross-encoder and completing the request. We track timeout rates and tail latency (p95/p99) in dashboards.

Benefits. Parallelizing the LLM reranker improves performance:

• Smaller inputs per call let us run a faster, cheaper LLM without losing reranking accuracy.
• Latency improves because each worker handles a shorter prompt, produces a shorter output, and system prompt caching removes most of the fixed overhead.
• Round-robin batching evens out the positional bias across batches.

RAG in Copilot: Keeping Source Diversity

Unlike Fin, Copilot searches across more entity types, including internal content and past conversation history that aren’t user-facing. If treated as one pool, past conversation excerpts can dominate, making it harder to surface more authoritative content.

To fix this, Copilot retrieves by type, scores within type, then merges using heuristic rules that keep source diversity near the top of the RAG context. We use three parallel streams: internal content, public content that Copilot can cite, and conversation history.

Impact

Fin. Our first working setup of the LLM reranker added ~5 seconds of latency. After tightening the output format, enabling thresholding, adding prompt caching, and parallelizing calls, the added latency dropped to <1s, and costs fell ~8x.

• Latency P50: +0.9s
• In the A/B test against the open source BGE reranker, the LLM reranker showed a clear quality win, with a statistically significant uplift in resolution rate

Copilot. With entity-aware retrieval and the LLM reranker, we observed the following on A/B test against BGE reranker:

• Assistance rate: +3pp
• The answer rate: +2pp
• Cited conversation excerpts: –27%
• Citations of public + internal articles: +63%

Pointwise vs Listwise

We ran an A/B test comparing a listwise LLM reranker with our parallel pointwise setup. The listwise version scores all passages at once, so needs a stronger model.

We gave it a solid try, but it didn’t outperform the pointwise version: resolution rate was the same, but latency increased ~40% and cost ~15%, so there was no clear benefit.

Reflections

LLM-based reranking helped us significantly improve quality compared to open-source cross-encoders. It also gave us a simple way to confirm that reranking quality really does make a difference in practice.

That said, this approach comes with trade-offs. Even after optimizations, the added latency is still noticeable (+0.9s), and coordinating multiple LLM calls in parallel introduces complexity.

These challenges motivated us to train a custom reranker, with the LLM reranker acting as a teacher, so we could keep the quality while reducing latency.

The Prompt

We’re open-sourcing our prompt for the LLM reranker below:

You are a customer support answer service. Your task is to evaluate help center passages and score their relevance to a given customer query for a retrieval augmented generation (RAG) system.

Evaluation Process:
1. Analyze the customer's query to identify both explicit needs and implicit context including underlying user goals
2. Assess each passage's ability to directly resolve the query or provide substantive supporting information with actionable guidance
3. Score based on how effectively the passage addresses the query's core intent while considering potential interpretations

Grading Criteria:
<grading_scale>
10: EXCEPTIONAL match - Contains exact step-by-step instructions that perfectly match the query's specific scenario. Must include all required parameters/context and resolve the issue completely without any ambiguity. Reserved for definitive solutions that exactly mirror the user's described situation and require no interpretation. 

9: NEAR-PERFECT solution - Contains all critical steps for resolution but may lack one minor non-essential detail. Addresses the precise query parameters with specialized information. Solution must be directly applicable without requiring adaptation or assumptions. 

8: STRONG MATCH - Provides complete technical resolution through specific instructions, but may require simple logical inferences for full application. Covers all essential components but might need minor contextualization. 

7: GOOD MATCH - Contains substantial relevant details that address core aspects of the query, but lacks one important element for complete resolution. Provides concrete guidance requiring some user interpretation.


6: PARTIAL match – General guidance on the right topic but lacks the specifics for direct application. May only resolve a subset of the request.


5: LIMITED relevance – Related context or approach, but indirect. Requires substantial effort to adapt to the user's exact need.


4: TANGENTIAL – Mentions related concepts/keywords with little practical connection to the request. Minimal actionable value.


3: VAGUE domain info – Talks about the general area but not the query's specifics. No concrete, actionable steps.


2: TOKEN overlap – Shares isolated terms without context or intent aligned to the request. Similarity is coincidental.


1: IRRELEVANT – Uses query terms in a completely unrelated way. No meaningful link to the user's goal.


0: UNRELATED – No thematic or contextual connection to the query at all.
</grading_scale>

Input Format:
<input_format>
<query>
// The customer's question or request
</query>
<passages>
<passage id='id0'>...</passage>
<passage id='id1'>...</passage>
...
</passages>
</input_format>

Output Format:
<output_format>
Return your response in a valid JSON (skip spaces):
{{"id0":score0,"id1":score1,...}}

Strict guidelines:
- Return ONLY a well-formed valid JSON with passage IDs as keys
- Each key must be a passage id in the format "idN"
- Each score must be an integer between 5 to 10. EXCLUDE passages that score below 5 (i.e. 0, 1, 2, 3 or 4)
- Integer values only, no decimals
- Skip spaces in the JSON
- No additional text or formatting
- Maintain original passage ID order
- Note: If NO passages score 5+, return empty JSON object
</output_format>

<examples>
{few_shot_examples}
</examples>

The post Using LLMs as a Reranker for RAG: A Practical Guide appeared first on /research.

However, even the most advanced LLMs have a limitation: they don’t always have up-to-date knowledge about the world or the product the user is having problems with. That’s where Retrieval Augmented Generation (RAG) comes in. Like many tools in this space, Fin uses RAG to dynamically retrieve and incorporate relevant information at runtime.

The RAG pipeline has three key stages:

1. Retrieval: We fetch 40 potentially relevant documents from the knowledge base. This stage prioritises speed and scalability over accuracy. (See diagram below to get a high level idea).
   
   
2. Reranking: These documents are re-ordered by relevance using a more sophisticated model, and a smaller subset (say top 5 to 10) is passed forward.
   
   
   
3. Answer Generation: Finally, the LLM uses these top documents, along with contextual information (like user data or timestamps), to generate a precise answer.

In this blog, we describe how we replaced a general-purpose retrieval model with the one fine-tuned on our high-quality customer support specific data, and the results of doing so.

How does a retrieval model work?

Fin’s retrieval system is powered entirely by semantic search¹. In semantic search, each document² is compressed into a vector (also known as an embedding: a dense numerical representation) that captures its meaning. These embeddings are generated when you onboard a new app into Fin or update your knowledge base. When a user submits a query, we generate a similar vector representation of that question and compare it to the stored document vectors. The search then returns the top 40 documents with the highest similarity to the query vector.

If you browse the Hugging Face model hub for sentence similarity, you’ll find over 12,000 open-weight models. What differentiates them is their ability to understand both the query, and the documents and to generate embeddings that reflect their meaning in a way that semantic search can leverage. One common and popular way to evaluate these models is the Massive Text Embedding Benchmark (MTEB), which ranks embedding models across a broad set of tasks.

About a year ago, we adopted bge-large-en-v1.5 for English content and multilingual-e5-base for multilingual content. Since then, new models have been released almost monthly, each claiming to outperform the last on the MTEB leaderboard. But we can’t just hop on to the latest best model every month: evaluating and switching models at that pace is costly. Each change would require recomputing embeddings for over 300 million documents which is expensive to compute and time-consuming to A/B test in production.  In our constant push to make Fin be the best it can, we recently decided to revisit our options. Could a newer model provide a meaningful improvement? Or better yet could we outperform any open model by training a custom model?

As a first step, we fine-tuned a base model using data from across our platform³. We were skeptical. After all, some open models are trained on billions of examples. Could our smaller, focused dataset really compete?

Surprisingly, the results were clear: Fine-tuning on our customer support specific data outperforms other models by a significant margin.

But before we began fine-tuning, we had one important question to answer…

Which model to use as a base model?

To answer this question, we benchmarked a mix of open-weight and closed-weight models. This served two main goals:

1. Assess how our current production model compares to the best publicly available alternatives.
2. And, identify a strong base model for fine-tuning on our own data.

From past experience, we’ve seen that the top-performing model on general-purpose benchmarks (like MTEB) doesn’t always lead to good performance on domain-specific use cases like Fin. Hence, we shortlisted a set of models that ranked highly on various benchmarks, instead of just picking the best one. Here is the list of our shortlisted candidates.

Our primary focus was on models under 1 billion parameters, though we included two larger ones for comparison: Stella 1.5B (1.5B parameters) and Voyage-large-3 (exact size unknown, but likely around 7B).

To evaluate the models, we created a test set of ~3,000 user queries (details in the next subsection). For each query, we included:

• Up to 3 positive examples (relevant documents)
• Exactly 10 negative examples (irrelevant documents)

A good model should consistently rank the positive examples higher than the negatives. To quantify this, we used two metrics:

• Precision: The model succeeds if all positive examples are ranked above all negatives.
• Recall@5: How many of the top 5 ranked results are positive examples?

Below are the results.

Some observations from the results:

• Several models outperformed the one that we were using in production. 
• Although Voyage performs the best among the bunch establishing a strong baseline, we can’t fine-tune it as it is a closed-weight closed-source model. 
• Among the open-weight models, Stella 1.5B was the best followed by Stella 400M. However, Stella uses non-standard architecture, which lacks support for training and inference tools in our stack.
• Snowflake Arctic 2 offered a strong balance between performance and practicality, while being small in number of parameters. It is built on the well-established XLM-RoBERTa architecture, having good support across the ecosystem. Additionally, it’s a strong performing multilingual model. If a fine-tuned version of Arctic performs well in production, it could significantly simplify our multilingual Fin pipeline.

Given these factors, Snowflake Arctic 2 became the clear starting point for our fine-tuning.

Note: Voyage Large is indeed large. Given the good performance of it on the benchmark, we tried to deploy it in production to establish a stronger baseline, however we had trouble scaling that model for our use case. So we dropped the idea of testing larger models in production, and focused on sub 1B models.

Training data for fine-tuning 

Since the base model we use is pretty strong, we thought that just fine-tuning it on hard examples from our own customers can go a long way. As part of earlier work on improving Fin’s answer generation, we had experimented with the re-ranking of documents retrieved by our semantic search model. In that experiment, we used an LLM to assign scores to each retrieved passage and re-ordered them accordingly, sending only the highest-ranking ones to Fin. These logs turned out to be a valuable source of training data.

We mined data from ~2 million real user queries. For each query, we start with the top 40 documents returned by the search model. Then we extract

• Hard positives: Documents that were used by Fin in the answer and received a high score from the LLM-based re-ranker.
• Hard negatives: Documents that were not used by Fin in the answer and received a low score from the LLM re-ranker.

Fine-Tuning Details

With our curated set of hard positives and negatives, we fine-tuned the selected base model (Snowflake Arctic 2) using a contrastive learning approach. Specifically, we used InfoNCE loss, a commonly used objective in retrieval tasks.

For each training instance we selected:

• 1 hard positive document
• And, 4 hard negatives

InfoNCE loss function is defined as 

$$ \mathscr{L} = -\log\left( \frac{\exp(s \cdot \text{sim}(q, p^+))}{\exp(s \cdot \text{sim}(q, p^+)) + \sum_{j=1}^{4} \exp(s \cdot \text{sim}(q, p_j^-))} \right) $$

This loss function tries to increase the similarity between query(q) and positive passage(p⁺), while pushing query and negative passages (p^–_j) apart.

We fine-tune the base model end-to-end for 2 epochs with an effective batch size of 256. We used AdamW optimiser with default parameters, starting learning rate 1e-5, and linear LR decay. 

Does fine-tuning actually help?

Offline validation on in-domain data 

To evaluate the effectiveness of our fine-tuned model, we began with offline validation using the same internal benchmark dataset described earlier. The table below compares the base model (Snowflake Arctic 2), two top-performing large models, and our fine-tuned Snowflake Arctic 2 model.

The improvement was substantial: precision increased by around 30 points, and Recall@5 increased by 10 points, outperforming even larger models.

Out-of-distribution evaluation

To confirm that we hadn’t overfit to our in-domain data, we evaluated the models on a similar dataset created from out-of-distribution apps⁶. This dataset primarily included English queries, so we report overall metrics only.

Performance on the out-of-distribution dataset was lower than on the in-domain benchmark, as expected, and this trend was consistent across all models. Nevertheless, the results remained strong: the fine-tuned model outperformed both the base Arctic 2 and even Voyage-Large-3 by a wide margin showing an improvement of over 20 percentage points in precision. 

Reranking evaluation on held-out benchmark subset

We also validated our model on a 1,000-query subset of our benchmark dataset, where each query included 40 documents scored by an LLM. This allowed us to compute traditional IR metrics like NDCG.

We see that on this reranking dataset also the fine-tuned model performs 10pp better than the base model. The fine-tuned model outperforms Voyage-Large by about 7pp.

FinRank Eval EN 1.0

Finally, we tested on FinRank Eval, a dedicated internal benchmark we created for in-house reranking tasks. This dataset consists of 3,000 English-only queries, each with 40 passages scored by Sonnet 3.5 and reranked using a secondary model to break ties.

Across every metric the fine-tuned model outperforms even larger models like Voyage-Large-3.

Performance in Production: A/B Testing Results

While offline metrics gave us confidence in the fine-tuned retrieval model, users ultimately don’t care about precision, recall, or NDCG. They care whether Fin answers their question. To validate real-world impact, we ran two A/B tests comparing the fine-tuned model with our current production setup: one for English-only conversations, and another for non-English conversations.

Our primary success metric is resolution rate: the percentage of conversations Fin resolved without human intervention. A secondary metric, answer sent rate, tracks how often Fin responds with an answer instead of asking for clarification. We saw statistically significant improvements in both metrics (with p-value < 0.01), for both English and non-English conversations. We are not sharing the exact effect sizes for competitive reasons.

In both tests, we also observed that Fin cited more documents suggesting that the improved retrieval model was retrieving more useful context for answer generation. Other key metrics such as cost and latency remained stable⁷.

Data Security

As noted in the “How does a retrieval model work” section, each passage is independently transformed into a vector representation during indexing, without influence from any other workspace, document, or passage. At query time, we use our existing database infrastructure to only retrieve documents from the same workspace in which the user is interacting, ensuring no cross-app data access or leakage. To retrieve relevant results, the user’s query is temporarily converted into a vector, which is used solely for the retrieval process and immediately discarded after. This process ensures there is no PII leakage. Additionally, all fine-tuning is performed on our own secure AWS infrastructure, so no data is exposed to third parties.

Conclusion

Before we started this journey, we didn’t expect to outperform large models like Voyage Large, but we were curious to know how close we could get. But, from the results we see that not only we closed the gap, we surpassed it by fine-tuning on high quality curated data. At scale, in production, these improvements translate to hundreds of thousands more users getting their issues resolved without needing agent intervention.

1. That is, we do not use hybrid search or incorporate traditional TF-IDF or BM25 like techniques ↩︎
2. Technically speaking, we divide the document into smaller chunks called as passages ↩︎
3. We only use data from the apps which allow their data to be used for training ↩︎
4. 7B, Based on https://huggingface.co/voyageai/voyage-3-m-exp ↩︎
5. Even though Stella1.5B is an English only model, it still performs well for multilingual cases. The base model they use Alibaba-NLP/gte-Qwen2-1.5B-instruct is multilingual ↩︎
6. These are apps which didn’t allow their data to be used for training ↩︎
7. This may feel counterintuitive, as we increased the number of parameters. However, thanks to the amazing Text Embeddings Inference (TEI) toolkit, we didn’t need to change our infra ↩︎

The post Finetuning Retrieval for Fin appeared first on /research.

Large Language Models (LLMs) are a powerful tech that have turned reasoning in natural language, into a service. They’ve had a huge impact on customer support, powering agents like Fin. Fin is already delivering real value, with many customers routinely experiencing resolution rates in the high 70s, and an overall average resolution rate across all customers of upwards of 60%. 

Now that Fin is a mature product, we can start testing more ambitious ideas. One key hypothesis is that for narrow, well-scoped tasks, we might match the performance of much larger models by training smaller, more efficient ones, on enough high-quality data.

Fin primer 

A part of Fin’s architecture builds on the RAG foundations to achieve an optimal experience for informational customer support use cases. This is built with components that try to understand the message exchanges with the end user and summarise the user’s problem, retrieve relevant information as passages, rerank them, and then generate an answer. The diagram below is a rough representation of how this flow works.

The goal here is to first focus on a well scoped, narrow task, that we could train a smaller LLM for: detect and extract the user’s issue.

Detect and extract issue summary 

Issue detection and extraction is an important component of Fin’s RAG pipeline, where a series of messages between the end user and Fin are transformed into a single answerable summary issue, which is then used for the downstream retrieval pipeline. 

The problem? Not all conversations have addressable issues. The old baseline setup used just one “issue detection and extraction” prompt: if there was no outstanding issue, it returned None. 

But in reality, issue detection has lots of tricky edge cases, like:

• If a user gave negative feedback at the end, we want to catch it as non-informational, even if there’s still an outstanding issue.
• Users often mix feedback, greetings, or random noise with updates to their previous request, making intent hard to spot.

To deal with this, our prompt kept growing in order to cover for the nuances of non-issues – over 80 few-shot examples, >5k tokens of instructions, and 17 defined non-informational categories. A few categories of non-issue examples can be found in Table 1 below.

These nuances of issue detection made this prompt a prime candidate for experimenting with custom modelling. We can split the problem into two independent models, one that classifies an interaction as one with or without an issue, and another that extracts an issue if the first model thinks there is one. 

This split strategy now makes the issue extraction task on its own a narrow task, allowing us to experiment with fine-tuned LLMs. 

How do we measure success? 

Before we talk about the model training effort, we need to have a clear definition of what success looks like. Following metrics are the key indicators of Fin’s health and performance: 

• Offline metrics: These metrics are measured by locally replaying a sample of Fin’s production requests via the new feature:
  • Answer rates: This rate measures the fraction of times Fin was able to provide an answer for a real production query, when the new models were injected in the RAG process. Any large statistically significant drop in this number is an indicator of performance deterioration. However, a small change might not directly imply an actual degradation in production, which has been observed time and again. 
  • Semantic alignment: The fine‑tuned model should generate issues whose meaning closely matches the production issues extracted by the large LLMs. We quantify this by computing the distance (e.g., cosine similarity) between the embedding of the production issue and the embedding of the corresponding issue produced by the fine‑tuned model.
• Online Metrics: These metrics are measured via an A/B test in production: 
  • Resolution rates: This is the foundational metric that directly impacts Fin’s bottom line. No matter how good the model behaves offline, if it significantly deteriorates this metric, it is not a success. This metric can be split into
    • Hard resolutions : Resolutions where the end user acknowledges that the answer actually solved the problem
    • Soft resolutions: Resolutions where there is no explicit acknowledgement or positive feedback from the user. 
  • CSAT: Customer satisfaction (CSAT) score indicates the quality of what Fin provides. 
  • Latency: Latency has been an important metric to track for our product experience. We are constantly trying to bring this metric down, allowing end users to experience a seamless low latency interaction [5]. We want to make sure at the very least, this number remains the same. 
• Cost: Often talked about in terms of amortised cost per token generated, this metric is an important one to track, especially for custom fine-tuned models. A comparable online performance, but at 2x the cost is not a success. 

Training an Issue classifier model 

For our issue classifier, we started by curating training data from our original LLM-based system, which was designed to both detect and extract issues in conversations. Here’s what the new setup does:

• We encode an input with ModernBERT
• A simple linear layer with sigmoid predicts the binary label: informational or non-informational
• The model is trained with binary cross-entropy loss

ModernBERT is a newer flavor of encoder-only transformer based models, and it outperforms almost all BERT-like models on retrieval and classification tasks [4]. As you can see in our blogs on retrieval, reranker, parsing feedback and escalation detection. ModernBERT works really well for routing and classification tasks, once you fine tune it on the right data.
For the issue classification task, we trained the model on 1M examples and ModernBERT achieved a remarkable 0.995 AUC score. When ModernBERT’s results didn’t match the ground truth, it was mostly the teacher’s mistakes, not the student model’s.

Fine-tuning the issue extraction model

Finetuning a generative language model, implies taking an open sourced model –which is already trained on trillions of tokens from the internet, achieving a baseline level of performance on a diverse set of tasks – and changing its weights slightly to optimise for a particular task. 

We use a specific way of fine tuning called Low-Rank Adapter(LoRA) [1][2]  based tuning, which is an extremely parameter efficient way of finetuning language models. LoRA freezes the large model’s original weights and learns only two much smaller, low-rank matrices per targeted layer. This setup slashes trainable parameters by orders of magnitude while preserving quality.

The blue blocks in the figure above visually describe what we actually train instead of the model weights (W). For each layer of the model, we inject two low rank matrices A and B, whose product can be added to the W once the training is complete. At train time, we optimize the weights of these two matrices instead of W itself. 

A good introduction to LoRA can be found here. These LoRA adapters are like lego bricks which can be added or augmented to the original un-tuned model, to achieve an optimised performance for a specific task. 

Data 

We curate data from customers who have agreed with using their data for training. 

The data is curated using the following conditions: 

• Conversation must have happened within the past 2 months
• Customers with an account in the US, with conversation locale set as “English”¹
• The conversation must have had an issue according to the older issue detection and extraction prompt.

The data is cleaned for obvious hygiene issues, and then anonymised by redacting any mention of emails, addresses, account numbers, phone numbers, names, places, organisations etc.

The resulting data contains 60 thousand training samples and 10 thousand validation samples. 

Experiments 

Before arriving at a final A/B testable model, we tested several variants of open source models, starting from a lightweight Gemma 8b, Qwen3 8b, finally getting a respectable result from Qwen3 14b variant. Some offline testing results are seen below:

The fine-tuned Qwen3 14b model seemed the most competent candidate out of all, performing at par with our baseline on answer rates. It is worth noting that we trained another variant of Qwen3 14b on just hard resolutions. The results were interesting to say the least, since despite learning to produce highly semantically aligned issues, the end to end answer rate performance was very poor. Upon closer examination, it seemed like the model only learned to produce an issue summary if the issue was extremely clear from the conversation, and refrained from producing any tokens when the conversation was ambiguous. This odd behaviour shows the importance of data curation in such projects. 

Results

The A/B tests were done in two phases, since we have two tuned models active in tandem, instead of the incumbent 1 LLM call to the large model provider. 

Issue Detector

We ran this A/B test before the Issue extractor model collecting enough data for statistically significant read out.

The results of the issue detector model were promising, with a slight decrease in answering rates, but almost no impact on other online metrics. However, since we are not using an LLM to do the more nuanced task of detecting whether there is an addressable issue or not, this also indirectly results in a 5% reduction in cost.

Issue Extractor

Once the issue detector was shipped to production we ran an A/B test with the winning candidate model as per the offline tests, for a week. This was enough to get enough data for statistically significant results.

The overall answer rate dropped by 0.1pp. However, we saw no evidence of negative impact on other online metrics, except a slight 100 ms increase in P50 end-to-end latency. The biggest win here was the relative reduction in cost per transaction of 12.5%.

Discussion and Conclusion

The fine-tuned models delivered substantial reduction in costs, while being competitive with state of the art models for this particular task of issue detection and extraction. We are seeing three distinct qualitative impacts on Fin’s performance 

• The Issue summarizer model can now focus only on summarizing issues, making the prompt much shorter. This decoupling of detection from extraction also stabilizes the prompt, as we now don’t need to keep adding examples of non-issues.
• The new issue detection model is much more precise, removing inauthentic soft resolutions. It handles edge cases much better and makes fewer hallucinations in simple cases.
• The issue detector gives a probability for an issue, so we can tune exactly how many informational vs non-informational queries we want, just by tweaking the threshold. You can’t get this kind of control with vendor hosted LLMs, like the ones from OpenAI or Anthropic.

While the newer approach with a smaller fine-tuned 14B LLM is significantly cheaper per transaction, there might be some more gains to be had in terms of impact on resolutions with further iterations. There are currently two running hypotheses to explore.

Impact of anonymisation: Protecting customer trust is paramount for us. Since generative models are generating tokens, training on customer data means taking utmost care to anonymise PII. To that end, in this first attempt we took an extra conservative approach by redacting every PII entity. There is a chance that this approach has negatively impacted the performance of the fine-tuned LLMs in the wild, because we are redacting important contexts at training time. We plan to improve on this approach and build a secure yet performant anonymisation strategy, with contextual replacement of PIIs instead of just redaction.

Impact of model size: The goal here was to find an optimally sized model that is light enough to minimise training/inference cost/infra, but large enough to be able to adapt to the task’s complexity. The 14B model was the first feasible model that we found to be competent in offline tests, and hence progressed towards an A/B test. However, research does suggest that a model’s competency to solve complex tasks scales with model parameter size [6]. Therefore, we think that it is definitely worth exploring larger models for such tasks.

In conclusion, this exercise provides a great evidence for deploying fine-tuned models for Intercom scale tasks. 

Citations 

[1] https://arxiv.org/pdf/2106.09685

[2] https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms

[3] https://en.wikipedia.org/wiki/Rank_(linear_algebra) 

[4] https://arxiv.org/pdf/2412.13663 

[5] https://fin.ai/research/does-slower-seem-smarter-rethinking-latency-in-ai-agents/ 

[6] https://arxiv.org/pdf/2001.08361

Appendix:  Are fine-tuned models learning a new skill?

While offline and online performance of these tuned language models do suggest that there is a lot of value in going through the fine-tuning process, one might ask: What is an unequivocal sign that the fine-tuning process is improving the model’s performance at a specific task, when compared against the original off-the-shelf base model. After all, the original off-the-shelf base model is also trained on trillions of tokens from the internet, and may contain the intrinsic intelligence to solve the task out of the box. 

The most obvious way is to compare the offline metrics before and after fine-tuning: 

We see that both average semantic alignment and answer rates take a massive hit while using off the shelf base model. Particularly the answer rate drop shows that the off-the-shelf model lacks the competency to extract usable queries for the Fin’s RAG pipeline. 

Average Perplexity computed on the generated tokens for the issue extraction task. Lower perplexity implies that the model is more sure of the tokens it generates for the same input context.

Another definitive sign of learning is a metric which is directly linked to the model’s loss that is optimized at training time. This metric is called perplexity. 

All LLMs are trying to predict the next token, given all the tokens they have seen till that point in time. The way this is done is by predicting a vector of probabilities over all the tokens in their vocabulary, and then choosing the next token based on those probabilities. The way these models learn is by optimising these probabilities using a specific loss function called the cross entropy loss. Perplexity just averaged exponentiated cross entropy, progressively computed over all the generated tokens, given the model has seen the input context (in this case the anonymised chat history). 

$$
\text{Perplexity}(T)=\exp\left(-\frac{1}{N}
\sum_{i=1}^{N}\log p \bigl(t_i \mid t_{<i}\bigr)\right)
$$

The perplexity metric quantifies how much the model is surprised by seeing the next token ti, given it has seen all the previous tokens t<i. When we evaluate this metric on the generated tokens for off-the-shelf Qwen 3 14B base model and for the LoRA fine-tuned models, we see that the fine-tuned model shows substantially lower perplexity on the generated tokens. Both these results indeed confirm that fine-tuning models helps them acquire competency in specific tasks. 

1. English is the most represented language with Fin, comprising about 80% of our traffic. Limiting to English language further narrows the problem, and controls for performance issues due to imbalanced language representation. We plan to explore multi-lingual in the future ↩︎

The post David vs Goliath: are small LLMs any good? appeared first on /research.

Intercom is an AI company.

In this post we’ll discuss how we built tools to make our scientists productive at training modern AI models. Ones that require bigger, more expensive, harder to get GPUs, often running on the bleeding edge of the software stack. We’ll talk about our next-gen AI infrastructure, a different approach we took, as well as about failed tries in building a good experience, learnings, and some surprises along the way.

Trying out off-the-shelf solutions

When you think about AI infrastructure, the first thing that comes to mind are GPUs, CUDA, big datacenters and power management. Our journey of training our own models has started from a completely different angle. Instead of planning the exact GPU capacity, topology of our clusters, bandwidth between GPUs and nodes, we’ve started from the experience we wanted to give to our engineers and scientists.

Before we started training our models, M-based Macs were a great home for all our development. We have our great monolith, all parts of the codebase are easily accessible, and we heavily lean on interactive ways to develop software (yes, notebooks!). You just run ./script/update and you’re set to start building new exciting parts of Intercom.

Needing a GPU breaks that paradigm straight away. We really like that integrated development environment, but you can’t train (or even run) anything serious on Macs. 

We run on AWS so it was cheap to try their product offering. We were early users of Sagemaker Notebooks and we used it to get GPU-powered instances. The experience was lukewarm at start – it was really nothing more than Jupyter Notebooks with GPUs, behind some authenticated proxy.

AWS launched Studio to replace Notebooks, which was a positive change. They even rebuilt it twice from scratch, each iteration being significantly better than ones before. Storage improved, stability improved, but it’s never been quite there. For instance, not being able to “SSH in” meant that scientists were not able to use their favorite IDEs. It was just about acceptable for a while (since AWS provided VSCode in the browser), but the rise of AI IDEs like Cursor showed us that using VSCode in the browser was leaving a lot of productivity on the floor.

Run Less Software runs deep in our culture and it was difficult to shake off the feeling that we should just try a bit harder to make it work for us. And we’ve tried hard to make it work. We’ve used all sorts of tricks to allow some sort of SSH connection, bringing it closer to where we wanted it to be. But we saw engineers jumping back to their Macs as soon as they could, using remote machines only when necessary.

These jumps were painful (data transfer, unexpected capacity issues without ability to change Availability Zone etc.) and it was hard to maintain the remote setup because engineers were spending very little time using it. Feedback was coming sparsely, and was always urgent because many issues were blocking them from working. As the team grew, this half-hacked experience was becoming more expensive for everybody and hindering our ability to move fast. We needed something better.

Our take at it

We’ve set a goal to have a dev environment that’s as seamless as the one on Macs, but with GPUs attached. We want scientists to just develop features and think about datasets and training techniques. We do want to let them know about layers like Kubernetes, Slurm, and NCCL as late as possible (hopefully only when they have to innovate on that layer).

The vision was there, but the deadline was tight – we had around two weeks to give people some infra to start training models. 

We knew we couldn’t build some big AI infra from scratch and we had no option but to cheat. It was clear that the team wanted to start with smaller models, whose training runs fit into one node. For these cases, we were sure we could provide a good dev experience and make it feel like it’s running on Mac with more power.

We’ve made several decisions that now seem pivotal:

• Create a CLI tool for managing anything related to AI infra.
• Split the training infra and experience for small and big models.
• Build on top of some more flexible platform so its constraints are not preventing us from providing a good experience.

We started by developing a CLI tool we called ai-infra. To be honest, we didn’t anticipate it would become so central to our AI dev infra. We imagined it only as a central place for basic create/start/stop actions on servers with GPUs, but it ended up integrating deep in scientists’ workflows, offering help across all stages of the model’s lifecycle:

We created a single, easily installable and maintained python package. It enabled us to improve all the small parts of the developer experience that were causing enough friction to be annoying, but none of them annoying enough to automate the fix. Suddenly all the tricks, fixes and code snippets shared around random docs got home and could be run automatically, often without scientists even knowing, just when they’re setting up their user/workspace.

We can now set up their SSH to make sure GitHub keys are forwarded, we can transfer their git config, shell setups, make sure tools like nvitop work out of the box, make sure uv is set-up so they can install GPU packages without compilation. Having this “central” CLI tool enabled us to do all that, but also to cut through a million of AWS clicks and APIs that are hard to use through AWS CLI. You want to create a new instance? No problem, ai-infra create will create one for you, install our big monorepo (with a few example kernels/vens) and make sure your remote machine feels like your Mac. You want to find pricing and purchase some machine for the future? No problem, just use ai-infra find-capacity. How about  spinning up an efficient server for the model you just trained so you can run some evals? ai-infra serve <model_path> will do it for you. Need a  hosted MLFlow instance? You don’t have to go to AWS Console and click ten times, just run ai-infra mlflow.

Here’s how creating a new instance looks:

Right now, it takes eight minutes (we want to make it five, then three) from running ai-infra create to being able to run ai-infra cursor and get Cursor to ssh into your new instance, with all the packages installed and having different virtual envs easily bootstrapped from several recipes we give to people.

It takes the same eight minutes from needing a GPU machine to getting one (sort of, more on that later) and working on it. But you probably already have one that’s been running since the day before.

I’m still surprised with the power of hands-off experience we achieved just by gluing things together, removing confirmation prompts, automating clicking through AWS console, skipping authentication screens, recognising best serving software stacks for the just trained model etc.

There’s nothing smart there, just the wish for a very good dev experience and Claude Code that generated 98%+ of the 10k lines the tool currently holds.

Here’s how it came to be

We asked our scientists and engineers about what kind of setup would feel most natural to them for working with GPUs, and what didn’t work well with previous setups. The major piece of feedback we got was that they wanted  to use their IDE with their extensions, having access to the real terminal, trying out things quickly, letting files stay there after restarts etc. AWS Sagemaker eventually built out most of it, but it was still inflexible in opaque in terms of infrastructure placement and capacity management.

All these were pointing to something persistent with an SSH access. We knew we wanted a platform more flexible than SageMaker so we can iterate quickly and don’t have to work around constraints all the time. We have a good bit of experience with EC2 so the choice was obvious – plain EC2 with Ubuntu DLAMI. We get all devices set up and we’re ready to start with our layers of software the moment a new instance boots.

ai-infra glue is really necessary to make it all work. Actually, any glue like that will be a great experience for engineers, as now they won’t be forced to stitch many parts themselves and make decisions they don’t actually care about.

Some of the lessons learned

Splitting the training between small and big models

Why did we think it was a good idea? First of all, our hand was forced by a short deadline, but we also weren’t sure that we could provide a great user experience if we make training all types of models unified.

The basic case is pretty simple: as long as your needs fit into one node (which NVIDIA is happily making more powerful), we have an answer for a good dev experience. Once that’s not true, all sorts of tradeoffs and technical questions start to kick in:

• How do you share the data between them? 
• How do you iterate on your code? 
• How do you install deps remotely?
• Do you use docker? 
• Do you build everything into an image or not? 
• Do you build ssh-ing into nodes into the core part of the experience? 
• Do you offload some of these decisions to the user?

Whatever avenues you take, it’ll be hard to beat the ease of experience of ssh-ing in with Cursor. We have been working on it for some time, using Slurm, experimenting with EKS, but I can’t say we’re providing the experience we’d like. We still need to pair scientists with infra engineers who are very familiar with the setup, to take shortcuts and iterate quickly.

This confirms that the early split we made was a good one, and it enabled us to provide a great experience at least for smaller training runs.

Securing GPU capacity

Intercom is a cloud-native company and we have no prior experience managing assets like servers in the datacenters, planning for the network capacity etc. We are happily used to EC2 API calls booting machines within three minutes and adding compute capacity to our clusters. It was quite a shock to find out that it just doesn’t work like that with servers that have GPUs! Yes, you will still boot these machines with an API call, but only after you buy it upfront for a fixed period of time in the future, or talk to your account manager and sort it out some other way.

Also, the capacity elasticity takes a very different form. GPUs are such a precious resource that you shouldn’t count on cloud providers giving more of them to you when your demand goes up during the day. Not without talking a lot with the seller upfront, at least. It was painful to discover that it’s true even for smaller GPUs like L40/L40S and that all kinds of GPUs are in short supply. Most cloud providers will tell you that there is on-demand capacity, but our observations is that information can be stale the next day and you really need to reserve the capacity that you need to count on.

Instead, treat your GPU capacity as an asset that’s hard to buy and sell. That means that you can change the “inventory” infrequently and the investment is shifting to finding ways to use it as much as you can instead. Unfortunately, this shapes the experience of working with it. Nightly async batch jobs are not as nice as interactive development, but they are useful in the age of GPU shortage.

The future of our AI infrastructure

We know this is only the first version of our AI infrastructure. It’s very early, but it already enabled us to move quickly and improve Fin in only a few weeks.

We took some shortcuts that we want to fix in the long term. There’s a vision of unifying training small and big models, backtesting them and running them in production. It’s blurry, but it’s there, and each training run brings more clarity.

The post Building out Intercom’s AI infra appeared first on /research.

Fin’s north start metric is resolution rate; it’s how we measure how well Fin, our customer support AI agent, is performing. Each resolution is priced at US$0.99, so accurately detecting when Fin resolves a conversation directly impacts our revenue. 

At Intercom, we classify resolutions in two categories: assumed resolutions and confirmed resolutions. An assumed resolution, on the one hand, happens when a user leaves the chat without giving feedback (positive or negative), and without the conversation being escalated to a human support agent. A confirmed resolution, on the other hand, occurs when a user explicitly states that Fin’s answers were helpful. We typically view confirmed resolutions being more valuable than assumed resolutions because they come with clear, reliable signals from users that we did a good job.

User feedback plays a central role in determining both assumed and confirmed resolutions. A user’s response to the question “Was that helpful?” defines whether their issue remains unresolved (in the case of negative feedback), counts as an assumed resolution (if there’s no feedback), or becomes a confirmed resolution (in case of positive feedback). Any confusion or errors in this step lead to frustration for end users, dissatisfaction from our customers, and ultimately lost revenue for Intercom. 

If we incorrectly classify feedback as negative, Fin may keep pushing to solve a problem that is already solved, wasting time and money. On the flip side, if we mistakenly treat a bad experience as a positive one, Fin might abandon the user prematurely, leaving them without help and unfairly charging our customers for a failed resolution.

Historically, Fin was fully powered by large language models (LLMs). These models are zero-shot learners: they can generalize to new and complex tasks, follow instructions, reason, and interpret subtle cues – skills we once thought only humans possessed. LLMs have completely disrupted the chatbot space. What used to be clunky, scripted decision trees have now evolved into highly autonomous AI Agents within months. 

But not every decision in customer support automation requires billions of parameters. Some tasks are simple and binary: Was the answer helpful or not? Does the user need more assistance? Should we bring in a human? With that in mind, we set out to explore whether a simpler model, one with fewer than a billion parameters, could accurately detect user feedback. 

We trained a text classification model on hundreds of thousands of real Fin interactions and compared its performance against state-of-the art LLMs. The goal was to see if a smaller, more efficient model could match or even outperform much larger models on this very specific task.

Multitask Learning with ModernBERT

Bidirectional Encoder Representations from Transformers (BERT) is an encoder-only transformer model that has been a major success since its release in 2018. Even today, it’s widely used across the industry, currently ranking as the fifth most downloaded model on HuggingFace, with 60 million monthly downloads at the time of writing.

ModernBERT, released in December 2024, builds on the original BERT architecture by integrating several recent advancements from the research space. One of the most notable improvements is the extended context window: ModernBERT can process up to 8,192 tokens, 16x more than the original BERT’s 512 tokens, while maintaining a comparable model size (see Table 1). This makes it significantly more powerful than the earlier version for long-text understanding without sacrificing efficiency.

ModernBERT processes input in the form of two segments, Sentence A and Sentence B, which are tokenized and separated by a special token, \( \mathrm{[SEP]} \). Another special token, \( \mathrm{[CLS]} \), is added at the beginning of Sentence A. During training, ModernBERT typically learns to answer the question: “Does Sentence B relate to Sentence A?” Here, “relate to” can have varied interpretations, such as “Does Sentence B follow from Sentence A?” or “Does Sentence B answer Sentence A?” The \( \mathrm{[CLS]} \) token serves as a sentence-level embedding that summarizes the input pair and is commonly used as the input to classification heads.

In our case, simply classifying feedback as positive or negative isn’t enough. We define feedback understanding as a combination of three classification tasks. The first is identifying the feedback type, which can be one of three values: no feedback, positive feedback, or negative feedback. The second is detecting whether the user message contains a follow-up question or complementary information. The third is determining whether the user has ended the conversation with their message. Table 2 illustrates examples for each of these classification tasks.

To address our problem, we designed the classification system using a multitask learning approach. In multitask learning, a single model backbone is trained to perform multiple related tasks simultaneously, such as solving several classification problems at once, combining image classification with segmentation, automatic speech recognition with speaker diarization, etc. The key insight is that by training on related tasks together, the model can learn more generalizable features, ultimately improving performance across individual tasks. Figure 1 shows a schematic of our ModernBERT-based multitask neural network. We use ModernBERT-large as the backbone of our system, hereafter referred simply as ModernBERT.

The overall multitask loss is a combination of the individual losses from each classification head. For feedback type, we use standard cross-entropy loss, while the follow-up question and conversation end tasks are trained with binary cross-entropy loss. The combined loss function is defined as

$$
L_\mathtt{multitask} = L_\mathtt{feedback\ type} + L_\mathtt{follow-up\ question} + L_\mathtt{ended\ conv.} \\
L_\mathtt{feedback\ type} : \text{Cross Entropy Loss} \\
L_\mathtt{follow-up\ question},\ L_\mathtt{ended\ conv.} : \text{Binary Cross Entropy Loss}
$$

Data

As described earlier, ModernBERT takes two input segments, Sentence A and Sentence B. In our setup, Sentence A corresponds to the last turn from the AI: a concatenation of all messages sent by the AI Agent in that turn. Sentence B is the last user turn: a concatenation of all user messages sent before the AI responded. We choose to not include the full chat history for two reasons. First, even with its extended capacity, ModernBERT’s context window is still limited. Second, our goal is to train the model to answer a focused question: “Does Sentence B provide feedback, ask a follow-up question, and end the conversation with respect to Sentence A?” Adding additional context could distract the model from what matters.

Even with this trimmed-down context, the combined turns can occasionally exceed the 8k token limit. In these edge cases, we apply a targeted clipping strategy: we truncate the AI turn from the left and the user turn from the right. This decision is based on the expectation that the most relevant AI tokens appear near the end of its message, where it typically asks if the answer was helpful, offers to talk to a human, or probes for more questions. Conversely, users usually begin their replies with direct feedback or requests, making the start of their message more informative.

After tokenization and clipping, we format the input by concatenating the AI and user tokens with a \( \mathrm{[SEP]} \) token between them, and a \( \mathrm{[CLS]} \) token prepended to the sequence.

Our dataset was built from hundreds of thousands of Fin interactions in English-language, spanning thousands of apps across various industries and business segments. We split the data into train, validation and test on the conversation level to ensure all interactions from the same conversation reside within a single split. To prevent bias, we excluded a small number of very long conversations whose large volume of messages could skew the model’s behaviour. These exclusions account for less than 5% of all Fin conversations.

In compliance with data protection standards, all apps included in the training process provided explicit consent for AI training. Table 3 summarises the number of examples and participating apps in each spit.

Results and Discussion

We evaluated our model on the test set described in the previous section. Table 4 and Table 5 present classification metrics for all three feedback-understanding tasks, using the current LLM-based classifier as the ground truth. 

The results show that ModernBERT performs remarkably well compared to a state-of-the-art LLM. It is especially strong in identifying when a conversation has ended, achieving an ROC AUC of 0.9987. This task is particularly sensitive: a false positive, mistakenly predicting that a conversation has ended, can cause Fin to exit prematurely while the user still needs help, leading to a poor customer experience. 

For feedback type classification, ModernBERT also performs strongly, achieving F1-score above 0.92 across all classes.

So far, our experiments have used our current LLM-based classifier as the source of truth. ModernBERT was trained and evaluated on labels generated by this LLM. But, like any model, LLMs are not infallible, they can and do make mistakes. In Fin’s case, we also operate under tight latency constraints, which limit the usage of long reasoning chains and increase the likelihood of noisy or imperfect labels.

A well-established strategy for estimating label noise is to introduce a second, independent LLM as a judge. This judge independently reasons through the task and generates what it considers to be the correct label. By comparing both the original labels and our model’s predictions against the judge’s decisions, we can gain a clearer picture of which model aligns best with a more robust interpretation of ground truth.

For this evaluation, we selected Anthropic’s Claude 4 Sonnet with Extended Thinking. It’s one of the top-performing models for complex reasoning and provides clear chains of thought that can be reviewed for consistency and transparency. Due to processing cost and runtime considerations, we randomly sampled 5,000 examples from our test set for this evaluation.

Table 6 presents the performance of both the original LLM-based model and ModernBERT on the feedback type classification task, using Claude 4 Sonnet’s outputs as the new gold standard. When judged by Sonnet, both models perform similarly, with the LLM holding a slight edge in overall accuracy. For the positive feedback class in particular, both models achieve high precision and recall, although the LLM shows marginally better consistency and slightly higher precision.

These experiments demonstrate that ModernBERT performs on par with a state-of-the-art large language model for fundamental classification tasks in English, even in scenarios that involve subtle nuances in user feedback, which can be challenging even for human evaluators.

During manual review, one particularly interesting trend stood out: ModernBERT was often better at distinguishing between true feedback and general confirmations. For example, it more accurately identified cases where users responded “Yes” to a different question, not necessarily confirming that Fin’s answer was helpful. The anecdotal examples below surprised us since we initially expected that LLMs would have the upper hand in capturing complex question-answer relationships.

Conclusion

LLMs play a crucial role in engineering intelligent AI agents, but they come with trade-offs. They’re often slow to respond and expensive to operate, both in terms of token costs when using third-party APIs, and infrastructure costs when hosting them in-house.

Fortunately, many common tasks in the customer support pipeline are simple classification problems. These can be solved effectively with traditional classification neural networks, like ModernBERT, which are significantly faster, more lightweight, and cheaper to serve. As we’ve demonstrated, with enough high-quality training data, ModernBERT can match the performance of much larger LLMs on specific tasks without the overhead.

However, there are some limitations. Training multilingual classification models is far more challenging. Fin supports 45 languages, many with different alphabets and regional variations that make each unique. With this level of linguistic variation, it’s hard to obtain enough training data to achieve high performance across the board. For this reason, we focused our efforts on English-only interactions. In contrast, state-of-the-art LLMs are trained with vast amounts of data from the internet, giving them a broad and reliable language coverage and making them a viable option to handle Fin’s global reach.

Understanding customer feedback is essential for measuring the success of customer support AI agents. Fin’s resolution logic depends on correctly interpreting the user’s sentiments towards its answers. Our proprietary classification model reinforces Intercom’s focus on building practical, high-performing AI solutions that serve both our customers and their users.

The post “Was that helpful?” Understanding User Feedback in Customer Support AI Agents appeared first on /research.

Get this wrong, and you either flood support teams with unnecessary escalations or leave users stuck without human help. This decision needs to be both fast and very accurate.

Today, we’re sharing how we built a custom multi-task model for escalation routing, achieving >98% escalation accuracy, reducing latency, and increasing resolution rate.

Understanding the Escalation Challenge

Whenever a user interacts with Fin, our system needs to make a real-time, three-way decision:

• Escalate immediately – Hand off to a human agent or trigger the custom escalation workflow
• Offer to escalate – Ask the user if they’d like to talk to a human
• Let Fin answer – Continue the AI-powered conversation

This decision is informed by two key inputs: the conversation history and business-defined escalation guidelines. These guidelines are rules that businesses configure, such as “Escalate immediately if the user expresses anger about billing”.

The system must also provide reasoning for its decisions. When escalating due to a guideline match, we cite the specific guideline. Internally, we also log broader categories like angry, request, or guideline.

For example, if a user writes “I’d like to check the status of my order #12345” and there’s a guideline saying “If the user asks about a specific order, hand off to a human agent”, the router would escalate right away, cite the guideline ID, and mark the reason as “guideline”.

Starting Point: LLM-Based Routing

Our first setup used a large language model (LLM) to decide: should we escalate, what’s the reason, and which guidelines matched. We also added guardrails to avoid edge cases like offering escalation twice in a row or escalating on the very first user message, unless there’s a guideline explicitly allowing it.

While it worked well, the LLM-based approach had limitations around latency and how much control we had over decision thresholds.

Attempt 1: Fine-Tuning Smaller LLMs

We first tried replacing our LLM with fine-tuned models. We experimented with Gemma and Qwen models of various sizes, training on 100,000 multilingual examples labeled with LLM outputs. This approach achieved solid 97% escalation accuracy, proving that custom models could compete with our LLM baseline.

At the same time, we saw excellent results with encoder-based models on other tasks like issue classification and reranking, which made us curious about using them for escalation routing too. Encoder models looked promising for faster inference and more reliable predictions.

Attempt 2: Classification Without Guidelines

Our next approach was intentionally simple: use a BERT-style encoder for three-way classification not escalate / offer / escalate on English conversations without any escalation guidelines.

We treated it as a standard text classification problem. The model takes the conversation history as input and outputs probabilities for each of the three escalation options.

The results surprised us. The custom model achieved 98% accuracy and often made better decisions than the “teacher” LLM.

However, this approach couldn’t scale: the share of conversations with escalation guidelines was growing fast (now 75% of the traffic), and this model couldn’t handle guideline citations. We needed something more powerful.

Attempt 3: Multi-Task Architecture with Citations

For our final approach, we built a single multi-task model that predicts three things at once:

• Escalation decision (3-way classification)
• Escalation reason (8 categories)
• Guideline citations (which guidelines to cite)

This approach gives us the accuracy we need and full control over the decision process. The multi-task design allows the model to learn shared representations that improve performance across all three tasks: the escalation decision informs the reason prediction, and both help with accurate guideline citation.

Architecture Deep-Dive

Our model uses a single encoder backbone with three classifier heads: escalation and reason classifiers use linear layers with softmax, and the guidelines classifier uses a linear layer with sigmoid for multi-label classification.

The most complex component is guideline citation. The encoder processes the entire input and produces contextual embeddings for every token:

$$\mathbf{h}_t = \mathrm{ModernBERT}(x)_t$$

To represent a guideline (a span of tokens in the input), we:

1. Identify its \([\mathrm{start}, \mathrm{end})\) token positions
2. Extract the contextual embeddings for those tokens and average them (mean pooling):
   $$ \mathbf{h}_{\mathrm{guideline}} = \frac{1}{\mathrm{end} – \mathrm{start}} \sum_{t = \mathrm{start}}^{\mathrm{end} – 1} \mathbf{h}_t $$
3. Score each guideline by passing its embedding through a linear layer and sigmoid:
   $$P(\mathrm{guideline\,is\,cited}) = \sigma ( \mathbf{W} \mathbf{h}_{\mathrm{guideline}} + b )$$

Training details

We trained the multi-task model on 4M examples using a combined loss function that optimizes all three objectives end-to-end:

$${\mathscr{L}}_{\mathrm{total}} = \mathrm{CE}(P_{\mathrm{esc}}, Y_{\mathrm{esc}}) + \mathrm{CE}(P_{\mathrm{reason}}, Y_{\mathrm{reason}}) + \sum_{i=1}^N \mathrm{BCE}(P_{\mathrm{guideline}_i}, c_i)$$

\(\mathrm{CE}\) is cross-entropy loss for escalation and reason classification, and \(\mathrm{BCE}\) is binary cross-entropy loss for guideline citations. Training resulted in stable loss convergence with strong performance metrics on evaluation set:

• Escalation accuracy: 97.4%

• Reason accuracy: 97%

• Citation AUC: 98.7%

Testing and Optimization

We conducted thorough offline testing, covering both in-distribution and out-of-distribution cases. The model performed well across the board, especially at spotting escalation requests and handling ambiguous situations. It often outperformed our original LLM-based system.

To improve further, we analyzed conversations where the custom model and the old production model disagreed. This helped us spot edge cases and refine how we deal with tricky guideline-related situations.

Instead of relying solely on a single model, we implemented a hybrid strategy:

• Our custom model handles 90% of cases with >98% accuracy.
• For the other 10% (mostly very long inputs or complex cases), we fall back to an LLM, which can handle longer context and has better generalization capabilities

This setup gives us the best of both: the reliability and speed of a custom encoder model for common cases, and the adaptability of an LLM for the most challenging interactions.

Impact

An A/B test showed clear improvements:

• Resolution rate increased significantly (p < 0.01), including a significant gain in confirmed “hard” resolutions
• Escalation detection latency dropped by 0.5s
• Cost per resolution decreased by ~3%

Also, refining escalation thresholds improved accuracy and cut false negatives – something we saw with our earlier LLM-based method. 

Key Learnings and Discussion

A big part of building this system safely and effectively was treating fine-tuning code like production-grade software. For our multi-task model, we validated each component separately: starting from data collection, finding guideline positions, and batch construction, to checking output shapes, evaluation metrics, and loss computation.

Before scaling up, we trained on small toy datasets to make sure outputs looked correct and the loss converged to zero. These early checks caught subtle issues that would’ve been painful to discover later during full training runs.

Another key to our success was having access to high-quality, domain-specific customer support data. Combined with guidance from LLMs, this let us train a smaller model to outperform its original LLM teacher for our specific support scenarios.

But developing robust ML systems takes more than just good performance on average. It requires careful handling of edge cases, using fallback strategies when the model isn’t confident, and thorough testing with out-of-distribution data.

Our custom escalation router demonstrates all these principles in action. As a result, we built a system that’s not just more accurate, but also faster, cheaper, and more controllable.

The post To escalate, or not to escalate, that is the question appeared first on /research.

Why do we need language detection? 

On the surface, the problem looks simple: Fin needs to reply in the same language the user is using. Can’t we just do prompt engineering, like below?

You are customer support agent. You need to answer customer's question. Reply in the language they asked the question.

History:
Fin: Hey There, How can I help you?
Customer: Hey, im need help.

...

Reply:
(Fin generates a reply in the user’s language)

But here’s the catch: while large language models can reply in virtually any language, most Fin customers have human agents who speak only a few. That means Fin can only support a few languages per customer. Basically, customers want control over the languages that Fin responds in (they want the ability to constrain it to only languages that they give it permission to).

Consider a hypothetical setup of Fin by AlpNet Communications, a telecom operator based in Switzerland. They’ve set English as the default language for their Fin workspace and enabled support for German, French, and Italian (the country’s official languages). So far, so good: if a customer writes in French, Fin replies in French; if they use Italian, Fin responds in Italian.

But what happens when someone messages in Dutch? Technically, Fin can reply in Dutch, but it shouldn’t, as the customer has “prevented” it from happening. AlpNet doesn’t support that language, and chances are, their agents can’t either. This is where fallback logic comes.

First, Fin checks whether the user’s language is supported. If it’s not, it looks at the user’s browser locale. If the locale is also unsupported, say it’s Dutch, Fin falls back to English, the workspace’s default.

You could try to encode all that logic directly into a single prompt, but that quickly gets messy. A better approach is to handle the decision-making upstream and keep the prompt simple. This upstream logic itself can be handled by a separate LLM call, like below

Determine the language the user is writing in this message: 

User: Hey, im need help.

AI: <detected_language>

That helps, but extra LLM calls slows things down, and we want Fin to be fast. What if we use lighter models, trained for just one job: detecting language?

How do we do language detection?

We used FastText in production to detect the language. FastText…

• …is commercially available and is Open Source (CC-BY-SA licensed)
• …trained on a large corpus of 176 languages (data from Wikipedia, Tatoeba and SETimes)
• …is FAST (takes 10s of microseconds on CPU) 
• …and is more accurate compared to the other models like Google’s CLD3 and langdetect

FastText predicts the chances that a piece of text belongs to one of the 176 languages. For example, given the text Comment ça va?, FastText says there’s a 95% chance it’s French. The remaining 5% is spread across the other 175 languages.

However, we’ve encountered issues with FastText in real-world use. The model was trained on clean, well-structured text, but Fin users often type in a more casual, imperfect way. Spelling errors and missing accents are common. Unfortunately, even minor deviations can significantly impact FastText’s predictions. For example, removing the cedilla in Comment ca va? drops the French confidence from 95% to 68%.

FastText also struggles with short inputs or when the script doesn’t match the language. Here are a few examples where it misclassifies the language:

To stay on the side of caution, we predict the language only if the confidence crosses a certain threshold. Unfortunately, we have observed that this too leads to many cases like below where we don’t predict the language, and often a different language gets used in fallback than the one that the user messaged.

Why is language detection hard?

There are many challenges a real-life language detector has to solve. Here are some of them:

• Spelling mistakes – The model has to handle typos and bad grammar
• Short context – Many messages are short (like ok, si, da, or greetings)
• Ambiguous messages – Many messages are just email addresses or identifiers 
• Mutual intelligibility – Some languages are very similar, like Hindi and Urdu, or Spanish and Portuguese
• Script mismatch – Some users type in Latin script instead of their native script (like Bangla, Armenian, or Hindi)
• Code switch – It is not uncommon to have multiple languages in a single message (e.g., 如何配置help center? )
• Long tail – Most languages don’t have enough training data to teach the model fine details

Our solution

A key limitation of FastText lies in its simplicity. It represents a text input by averaging the vectors of its subwords and then applies a linear classifier to make predictions. While efficient and effective for many tasks, this approach lacks the capacity to capture complex contextual relationships. In contrast, transformer-based models like BERT have been shown to consistently outperform simpler models like FastText across a wide range of benchmarks. We believe that we can do better than FastText by training a BERT-like language model on carefully curated data. Towards that, we’ve curated both general and Fin-specific data for the 45 languages¹ we support, and trained our in-house language detection model. We use the general data to pre-train the language detection model for better generalization, and then fine-tune the model on Fin-specific data. 

General Data: this is a subset of FineWeb2 – an 18TB dataset with content from about 1800 language-script combinations. We curated around 800K labeled text examples. Since FineWeb2 includes multiple scripts per language, it helps solve the script mismatch problem. 

Fin-Specific Data: we collected this from random user conversations with Fin². We used an LLM to identify the true language for each chat. If a message was too ambiguous, the LLM was told not to guess the language. We collected about 100K labeled conversations.

A problem with randomly sampling Fin chats is that around 50% samples are in English, and the top few languages make up 90% of the data (see figure below). That means many tail languages only had 100-200 examples. To fix this, we randomly translated some English messages into each tail language to increase their volume. After this, every language has at least 2K examples³.

We experimented with two models from the BERT family:

• DistilBERT Multilingual (135M parameters, Apache-2 licensed)
• XLM RoBERTa (280M parameters, MIT licensed)

We chose them as our base models because they are pre-trained on around 100 languages for general language modelling. So they already have some understanding of multilingual content.. First, we added a classification head to each base model and trained the model end-to-end on 800K examples from FineWeb2 for three epochs. Then we fine-tuned these models on Intercom’s data using lower temperature.

Evaluation

Offline Evaluation

We have a held-out test set of about 6K LLM labelled examples⁴. This data was collected similar to the Fin-specific training data described above. We measure the goodness of the model on the following metrics:

• Precision: when the model says it’s (e.g.) English, how often is it really English?
• Recall: of all actual (e.g.) English texts, how many did the model find?
• F1 Score: single metric that combines precision and recall

For all the metrics we report Macro Average (all languages are treated equally), and weighted average (weighted by number of samples in each language):

Along with FastText(our current model), we also tested Lingua, because of its claimed good performance on short texts. We see that the BERT family is doing much better than Lingua and FastText across all the metrics. F1 scores (both macro and weighted) are approximately 10 percentage points higher than FastText. Between DistilBERT and XLM-RoBERTa, the XLM-RoBERTa’s F1 is about 1.5pp higher than the DistilBERT. 

In Production 

We ran the RoBERTa-based language detection model in an A/B test, with FastText as the control. The new model made 6pp more predictions than FastText⁵. Just to make sure that we don’t use wrong language on these “extra” conversations, we checked a random sample of these extra predictions using an LLM as a judge, and found that the new model was >99% accurate on these predictions.

To further validate performance in production, we used an LLM to assess a random selection of conversations where both models would have made some prediction⁶. Both models achieved over 99% accuracy overall⁷. For each conversation, we simulated the output of the alternative model to identify disagreements. These occurred in fewer than 1% of cases. Notably, in those disagreements, RoBERTa outperformed FastText by a margin of 20pp in accuracy. As a final validation to the new model, since the full rollout of the RoBERTa model, we haven’t seen any issues reported by our customers related to wrong language detection.

Conclusion

Language detection is a deceptively complex problem, especially in a real-world product like Fin, where the language detector must balance accuracy, and recall across a diverse and noisy input space. FastText gave us a good starting point, but to go further, we needed better data and a model tailored to the kinds of messages Fin actually sees.

By developing our own in-house language detection model, we’ve unlocked a couple of key advantages: we can now support new languages more easily, and control precision and recall by training on additional data where needed. The gains in recall and coverage are already improving user experiences, and we’re set up to keep improving as Fin continues to grow.

1. https://www.intercom.com/help/en/articles/8322387-use-fin-ai-agent-in-multiple-languages
   ↩︎
2.  We only use data from the apps that allow their data to be used for training ↩︎
3.  We verified the quality of these labels by computing accuracy on random 100 conversations per language using a separate LLM as a judge. The goal here is to just make sure that we don’t have noisy examples for any particular language. We did identify some bugs in our earlier prompts using this approach. ↩︎
4.  The dataset is mostly balanced (all but 8 languages have 200 examples, remaining languages have >= 40 examples) ↩︎
5.  Technically FastText made predictions, but with lower confidence ↩︎
6.  That is to say, on control bucket conversations we run RoBERTa, and in treatment bucket we ran FastText model ↩︎
7.  This apparent disconnect between offline and production could be because the offline dataset is balanced across the languages we support. ↩︎

The post Building a Better Language Detection Model for Fin appeared first on /research.

Introduction

Fin is an advanced customer support AI agent powered by a combination of large language models (LLMs) and retrieval-augmented generation (RAG) techniques. This hybrid engine enables Fin to deliver accurate, context-aware responses. However, Fin often relies on generalist LLMs provided by vendors such as OpenAI and Anthropic—solutions that can be both expensive and slow due to the size and complexity of the models.

We wanted to explore whether certain tasks currently handled by these large external models could instead be managed by smaller, fine-tuned LLMs that we host ourselves. Our goal was to evaluate the feasibility and cost-effectiveness of serving these models on our own AWS infrastructure.

This analysis involved:

• Exploring AWS hardware options and serving engines
• Benchmarking various open-weight models
• Estimating cost differences between self-hosting and API usage

Cost Comparison: Hosted vs. Provider

Comparing the cost of self-hosting LLMs versus using provider APIs is not straightforward. The pricing models differ fundamentally:

• Self-hosting: You pay for infrastructure (e.g., GPU instances) over time.
• Provider APIs: You pay per token processed, with separate pricing for prompt, completion, and cached tokens.

In AWS, costs vary significantly depending on whether instances are reserved in advance or acquired on demand—reservation can be 2–3× cheaper. To fairly compare costs, we estimated the number of reserved instances required to serve our existing traffic reliably.

Instead of using “requests per second” (common for traditional web services), we measured tokens per second (TPS), as token usage can vary greatly per request in LLM services.

We used two key data points:

• Maximum sustainable token throughput per instance per model
• Peak token usage per minute from historical Fin traffic

This allowed us to estimate the number of nodes or GPUs needed per feature and model, and ultimately the cost to serve each model.

Benchmarking Setup

We benchmarked a variety of open-weight LLMs across multiple AWS hardware configurations. The models included:

• Qwen 3: 14B, 30B, 235B
• Gemma 3: 4B, 27B
• DeepSeek Llama: 8B, 70B (distilled)

We tested these models using real Fin prompts and organized them into three datasets, categorized by prompt complexity and model size.

Hardware tested:

For simplicity and to allow increased flexibility, we used EC2 instances for the benchmarks:

• g6e (L40s GPUs)
• p4d/p4de (A100) 
• p5/p5e (H100/H200)

Inference engines tested:

There are many available inference engines that can be used today, but we primarily looked into the most popular ones: VLLM, SGLang and TensorRT. However, because TensorRT did not support Qwen 3 and Gemma 3 models at the time, we went forward only with VLLM and SGLang. Both engines showed comparable performance across configurations using default settings.

Latency constraints:

• Time to First Token (TTFT): ≤ 250ms
• Inter-Token Latency (ITL): ≤ 30ms

These targets were based on Fin’s production latency requirements. Our goal was to maximise TPS without exceeding these latency thresholds.

Results

We compared our internal serving cost estimates against popular provider models:

• GPT-4.1 mini
• GPT-4.1
• Anthropic Sonnet 3.7

We calculated monthly token usage per dataset (prompt, output, cached) to estimate provider costs. We then benchmarked each model across all hardware configurations using both VLLM and SGLang.

Cost Ratio Comparison

Key insight: Smaller models (<14B) were significantly cheaper to serve than GPT-4.1 mini, while mid-sized models like Gemma 27B and Qwen 30B were more affordable than GPT-4.1.

Hardware Cost Efficiency (Qwen 3 14B)

Observations:

• g6e (L40s): Multi-GPU setups lacked NVLink and relied on PCIe, making scaling inefficient.
• p4/p5 (A100/H100): NVLink-enabled bandwidth allowed efficient tensor parallelism, boosting performance and cost-efficiency.
• A100 80GB: Outperformed L40s in both performance and cost.
• H100/H200: Though expensive, delivered high throughput and favourable cost/TPS ratios.

Conclusion

Our benchmarks showed that self-hosting smaller and medium-sized LLMs can be significantly more cost-effective than relying on commercial APIs—especially with models under 30B parameters. To serve larger models efficiently, further work is needed to optimise engine performance. While EC2 provided flexibility for benchmarking, it’s not ideal for production. We plan to transition to EKS (Elastic Kubernetes Service) and potentially Sagemaker Hyperpod for live model serving.

The post Cost of Serving LLMs appeared first on /research.

To make this argument:

• We start by explicitly defining intelligence for AI agents along three essential dimensions. 
• We then introduce a taxonomy categorising AI products according to their requirements, illustrating why certain applications thrive while others see slower progress.
• With this foundation, we make the argument that, in Customer Support and similar applications, current state-of-the-art models have already achieved sufficiently high levels of peak intelligence, shifting the bottleneck to intelligence density.
• This reframing provides us with an alternative path to build effective and reliable systems without having to rely on ever-increasing levels of peak intelligence.
• This reframing impacts how we should build AI applications today.

What is Intelligence?

In this article, we will be making a point about the intelligence of state-of-the-art Large Language Models (LLMs) and Large Reasoning Models (LRMs)—essentially LLMs post-trained to develop native reasoning abilities—in the context of the field of AI-powered Customer Support. In order to do that, we need to start by defining what we mean by intelligence. This definition has to respect the general principles that govern intelligence, but be made tangible with respect to what we expect from AI Agents.

Let us define intelligence in a LLM-centric manner: Assuming a fixed and sufficient amount of base world knowledge, we argue the more intelligent model will be the one that, with all other dimensions held constant:

• Is able to reason about the most intellectually complex tasks most correctly and systematically.
• Is able to understand the nuances of human expression and interaction most accurately.
• Is able to discern the limits of its own world knowledge most clearly.

The first of these dimensions is most obvious, and the most watched. But the other two play crucially important roles, particularly in high-risk domains: they provide the context required for the problem-solving as well as enable the understanding and setting of boundaries within which the reasoning should take place. Together, the three dimensions allow us to derive the intelligent behaviours we expect from, and find valuable in, an AI Agent.

Let us consider for a moment the concept of “instruction-following intelligence”, particularly topical in the Customer Support space. This concept is used to refer to the ability of LLMs to consistently and faithfully follow instructions given to them. While as a short-hand this concept is a useful abstraction, we argue that following instructions is not a form of intelligence in itself. Rather, it is an ability borne out of traits that can be mapped back to the concept of intelligence. 

In contrast to that framing, the three dimensions of intelligence we outlined are all individually necessary and jointly sufficient to capture “instruction-following intelligence”. That is, to follow an instruction intelligently the agent needs to:

• Have the ability to reason through the logical steps in the instruction, as well as any steps that would logically follow from them and be required to achieve the instruction’s goals.
• Identify and understand what exactly each top-level logical step in the instruction is and is not about, i.e. where their exact boundaries lie.
• Be able to clearly delineate what set of knowledge it should ground its problem-solving in, both with respect to its learned knowledge (through training) and given knowledge (in-context).

Each of these abilities directly maps to one of the three dimensions we used above to define intelligence, and together they yield the ability to expertly follow even the most complex of instructions. Remove one, and the agent will fail to reliably produce the desired outcomes. And this applies more generally: agents need to be able to understand what they know and what they do not, understand what they are told by humans (and other agents, which for now communicate just like us), and be able to reason about it all.

Why is this important? Because we need to deeply understand what today’s agents are capable of and where they fall short if we want to make progress in how we build and leverage these systems. Simply concluding that an agent “lacks instruction-following intelligence” does not guide us toward a solution. Instead, progress comes from understanding specifically where and why the agent fails: whether it is in interpreting the instruction and the interaction, at reasoning through these elements, or in recognising the boundaries of its own knowledge within the context of the instruction.

Now, equipped with this definition we want to make the following case. While in many domains—such as cutting-edge scientific research or high-risk medical applications—frontier LLMs still fall short across one or more of these three dimensions of intelligence, there are domains where they already surpass the necessary threshold. We argue that Customer Support is precisely one domain in which LLMs demonstrably meet the intelligence threshold today. A more intelligent LLM may help us make progress, of course, but is not a necessary condition for progress. Understanding this will have a big impact on the products we build today.

Latency, Intelligence and building AI Products

Latency is a decisive factor in the outcomes we deliver for our customers and the experiences we provide to our end-users with Fin. It is thus an important leverage point for us, one that can make or break an interaction, and as such, a defining factor of the solutions we build.

This is not, however, necessarily the case for everyone building AI products right now. We can broadly split the AI Products currently being built into five categories:

• Trivial Peak Intelligence, any latency profile (TI): e.g. summarisation, real-time translation agents.
• Lower Peak Intelligence, Higher Latency (LIHL): e.g. data labelling, web-search agents.
• Higher Peak Intelligence, Higher Latency (HIHL): e.g. app builder, research agents.
• Lower Peak Intelligence, Lower Latency (LILL): e.g. content moderation, customer support agents.
• Higher Peak Intelligence, Lower Latency (HILL): there aren’t applications here yet, and we will attempt to explain why this might be the case.

Leveraging this taxonomy, we can take stock of where progress lies for each category. Arguably, over the last year in particular, TI and LIHL use-cases have mostly been solved. HIHL use-cases have been seeing strong progress as of late, with notable examples such as OpenAI’s Deep Research and Anthropic’s Claude Code. And while we have not yet seen the same level of achievement in the LILL domain outside of purely informational interactions, among which Intercom’s very own Fin is a notable example, we are now starting to, also right here at Intercom.

Note that HIHL and LILL use-cases seem to have been on somewhat similar timelines of progress towards real-world impact up to now. And the reason for this is simple: new versions of state-of-the-art LLMs have had similar latency profiles than their preceding versions, but stronger capabilities across all three dimensions of our definition of intelligence, and in particular the first two. Because up to this point both categories have required these higher levels of peak intelligence, and they have been accompanied by broadly stable latency, improvements to the state-of-the-art have been equally benefitting both sets of applications at the frontier.

However, from here on out the paths of HIHL and LILL will diverge, at least partially. In certain HIHL tasks we will be able to make great progress, while others will remain unsolved for some time – the key obstacle being that the word “higher” in “Higher Intelligence” is unbounded, and in the limit there is no latency that can compensate for infinite intelligence requirements. But we now have a clear opportunity to hit escape velocity on almost all LILL use-cases. And this is possible due to the confluence of two key factors: the commoditisation of these higher levels of peak intelligence and the push towards technology that significantly speeds up inference.

Intelligence Frontier and Intelligence Density

Let us be clear – there are still situations in the Customer Support space that only humans can resolve. However, this is more closely tied to the “embodiment” problem than the “intelligence” problem: most fundamentally, LLMs lack the interface and tools to build their own context (a situation that, perhaps, web browsing agents can help change). But outside of this question, with the latest wave of LLM and LRM releases, peak intelligence itself is no longer the critical limitation in AI-powered Customer Support. 

Today, the primary constraint is not the models’ frontier level of intelligence, but rather the density of intelligence they can produce. But how does realising that take us any closer to solving LILL use-cases? In the context of LLMs and LRMs, the concept of density of intelligence is typically framed as the amount of intelligence produced per token. However, this framing gets the denominator wrong, and obscures the real operational bottleneck. To help us understand this, let us consider an example. 

Imagine a slow but very densely intelligent LRM that reads a duplicate-charge ticket in one comprehensive pass and takes sixty seconds to produce a refund resolution. In contrast, a faster, less intelligent model performs thousands of tiny checks—extracting amounts, verifying timestamps, grouping by invoice, checking refund history—one-by-one, some in parallel, some in sequence, and pools the results into the same conclusion in the same sixty seconds. The high-throughput model’s many rapid, lower intelligence passes collectively reconstruct the deep reasoning of the slow model. Under the assumption of a sufficiently high peak level of intelligence for the lower-intelligence-but-faster model, the system laid out matches the speed and performance of the highly intelligent, but slower, model.

And if we extend this reasoning to other parts of a notional “intelligence-latency frontier”—the boundary representing the optimal achievable trade-off between peak intelligence and latency for any given amount of intelligence required—we get the same outcome, even if for wildly different system implementations. That is, conditional on a high-enough latency of peak intelligence for the faster agent, we can trade-off peak intelligence for lower latency along the curve and still maintain the same level of performance.

Therefore, in reality, the right metric for measuring density of intelligence is the amount of intelligence produced per unit of inference time, and not per token. Shifting the denominator to seconds instead of number of tokens fundamentally reshapes our approach because it highlights that there are two actionable optimisation levers:

1. Peak Intelligence: Enhancing the quality and depth of understanding, reasoning and problem-solving abilities embedded within each token.
2. Throughput (Tokens per Second): Increasing the speed at which these intelligently produced tokens are generated.

The key to solving LILL use cases is therefore to understand that simply having more tokens per second, at the same level of intelligence, is a sufficient condition for solving the Customer Support space, in a world where we are no longer limited by the upper-bounds of intelligence. Given a high-enough level of peak intelligence, 1/100 x’ing the latency per token is functionally equivalent to 100 x’ing the intelligence per token. The shapes of the resulting solutions are very different, but the bounds of what they can achieve are a lot more similar than they might appear at first glance.

Ultimately, this reframing illustrates a vital principle: when peak intelligence levels are sufficient, the key driver of value shifts to making LLMs and the systems we build around them denser in intelligence. High-enough peak intelligence is necessary, but once achieved, the opportunities for value creation move towards latency and density, where significant opportunities for differentiation remain under-explored and are cheaper to capture. Work on intelligence density is where we, at the application layer, must focus to make progress and capture value better and faster.

Corollaries

From the various arguments we have made throughout this article, we arrive at a few critical corollaries.

The first is this: many difficult problems are actually solvable provided we have infinite time. However, once we explicitly condition these tasks on a finite, practical time budget – which is always the case in real-world scenarios – they once again become challenging, and sometimes effectively impossible. Real-world tasks inevitably impose constraints, and solutions divorced from practical considerations become irrelevant for value creation irrespective of their intellectual elegance.

This leads us naturally to our second corollary: as the time allowed for task completion approaches infinity, the value of any model or system that requires full utilisation of such time budget to reach a solution correspondingly approaches zero. A solution’s worth is inextricably linked to the constraints of its practical context, making latency-bounded, resource-limited tasks arguably the most faithful metrics for measuring intelligence. In other words, without clearly defining a time budget, any attempt to measure intelligence is at best inconsequential, and at worst meaningless. And we are not alone in thinking this.

Finally, this framing also helps us revisit and hopefully understand a point we made earlier in the article –  why it is that we have seen no practical examples of HILL applications to date (think: Ironman’s Jarvis AI). The reason is straightforward: HILL scenarios leave us with no latency budget through which we might be able to compensate for deficits in intelligence. Without latency to leverage, the only way to solve the challenge becomes relentlessly pursuing ever-higher levels of peak intelligence. But incremental intelligence improvements are costly and hard to come by and thus, the absence of practical HILL applications is not merely incidental—it is structural. Solving HILL use cases effectively becomes the costliest form of AI progress, and thus the rarest.

Why does this matter?

Understanding these notions of peak and density of intelligence, and where we are with respect to each of them on the timeline, directly impacts the products we build. A few months ago, we deeply understood how to leverage the agency and reliability trade-off in building agentic AI solutions to produce superior instruction-following agents. That realisation was how we built the future then.

Now, recognising the primacy of intelligence density over frontier intelligence in the Customer Support space provides us with a new, clear direction. This shift in focus will shape the next wave of AI-powered Customer Support beyond information gathering use cases. It will allow us to deploy agentic systems that follow strict and demanding instructions to execute complex and personalised tasks, interfacing with and modifying customers’ internal and external systems and databases, rapidly and reliably enough to meaningfully enhance real human interactions across the various dimensions our customers care about. And this realisation is how we build the future now.

The post We Don’t Need Higher Peak Intelligence, Only More Intelligence Density appeared first on /research.

Intercom’s customers depend on us for continuous availability. With our AI Agent, Fin, resolving up to 70% of our customers’ conversations, an outage can overwhelm their support teams. To emphasise how important Fin’s uptime is to us, we promise a Service Level Agreement (SLA) with a monthly uptime target of 99.8% for Fin.

Achieving this requires robust systems and practices designed to deliver reliability atop imperfect foundations.

Sophisticated Routing Layer

At the heart of this reliability is a sophisticated LLM Routing layer that decides how an LLM request is handled. Each Fin request uses multiple models. We define all the “routes” for each model in a format that looks something like:

{
    CLAUDE_SONNET_4: {
        us_prod: {
            conversation_serving: LatencyBased(
                GoogleProvider.us_east5(),
                AnthropicProvider(),
                BedrockProvider.us_east_1(),
            ),
            otherwise: LoadBalanced(
                AnthropicProvider(), BedrockProvider.us_west_2()
            ),
        },
        test: Sequence(AnthropicProvider(), BedrockProvider.us_west_2()),
    }
}

These routes let the system know what vendors and regions are available for each model. This setup enables flexibility in routing logic and failover strategies.

Key features of the routing system:

Cross-vendor failover

We maintain vendor redundancy for all models. For example, Anthropic models can be served by AWS Bedrock, GCP Vertex, or Anthropic’s own infrastructure; OpenAI models are served by either Azure or OpenAI itself. If a vendor experiences an outage or degraded performance, our system automatically shifts requests to another, maintaining both streaming and non-streaming operations.

Cross-model failover

Sometimes a new model that we started using for Fin might only be available on one vendor. Or, we might be serving a model using two vendors that alone don’t have enough capacity to handle all of Fin’s load. In this scenario, if one vendor had an outage, the second vendor would not be able to serve all the requests successfully.

And, in the rarest of rare cases*, all the vendors that serve a particular model might have an outage at the same time.

For all these cases, we can also failover across models. So if Sonnet 4 is unreachable on all vendors for any reason, we can send the request to a similarly capable GPT model. Similarly, we can partially failover some of the requests to a different model if we don’t have enough capacity for it on the available vendors.

* rare, but not impossible. Many times there is a sort of co-dependency between the vendors and an issue at the wrong level can end up impacting services on multiple vendors.

Latency Based Routing

We support different modes of routing. Sometimes because of capacity or performance constraints, we might send only a particular percentage of requests to a vendor.

The most interesting mode here is the “Latency Based Routing”. Performance can fluctuate between vendors throughout the day. We monitor real-time response times and route more traffic to the fastest available vendor, taking into account each vendor’s capacity. In practice, choosing the fastest route can mean a difference of several seconds per request which is critical for end-user experience.

Capacity Isolation

A classic failure reason for systems that share the same underlying resources is that a less-important usecase (like an asynchronous task for translating conversations) could end up exhausting this common resource, impacting more important usecases. In our case, Fin is the most important usecase we want to keep serving.

To achieve that, our routing framework lets us define separate pools of capacity for Fin vs everything else. Each pool only has access to certain vendors or regions of the vendor. This isolation prevents Fin from ever being impacted by a less-important usecase.

If Fin’s assigned pool is exhausted, Fin can draw from other capacity, but lower-priority uses are never allowed to encroach on Fin’s pool.

Operational Safeguards and Monitoring

We have protections and processes in place to ensure the system does not diverge from its current reliable state and can withstand Fin’s exponential growth.

Single Point of Failure reporting

We actively track each model and its setup for redundancy and capacity isolation. If any model falls short, we generate a high-priority alert and resolve the issue immediately.

Noisy Neighbor Protection

Noisy Neighbor is a well understood problem in multi-tenant systems. Our protections ensure a single customer or process cannot monopolise resources to the detriment of others.

Load Testing

We regularly perform load testing to ensure our systems can proactively support growth in Fin’s usage and maintain performance under heavy demand.

Maintaining Buffer LLM Capacity through strong relationships

Reliable operation requires buffer capacity. Through strong relationships with major vendors like OpenAI, Anthropic, AWS, Google, and Azure, we maintain ample headroom, with the ability to handle two to three times Fin’s normal traffic at any point.

This buffer capacity is not easy to come by and we have our account managers to thank for championing Intercom whenever we need extra capacity to run Fin reliably.

Observability

Intercom has a strong observability culture. We have written before about improving our observability posture and how we like to focus on the customer outcome. Instrumenting every LLM call, we collect data on token usage, response times, and system load across vendors. These insights drive capacity planning, vendor selection, and rapid troubleshooting.

All of these sophisticated systems and processes ensure that Fin runs reliably, and the whole is greater than the sum of its parts.

Future

We will continue investing and making sure we can run Fin reliably as the demand for it grows. Here are some things we plan on taking up in the future. 

Request Prioritization

The current architecture protects Fin from other usecases exhausting all the LLM resources, but many times Fin doesn’t need the capacity we have provisioned for it. In these cases, this extra capacity could be shared with other usecases which otherwise run constrained. This can be achieved by assigning a priority to each LLM request, and then dropping the lower priority requests when LLM capacity is constrained. Such a solution would make sure Fin can use all the capacity it needs without constraining the other usecases when there is no need for it.

Chaos Monkey

So far we haven’t needed this as regular vendor outages keep our systems well tested. But as vendors grow more reliable and as we move towards more automated failovers, short lived outages can go undetected. This can give us a false sense of security which makes it important to regularly test our systems by causing controlled outages and ensuring Fin won’t be impacted by a particular vendor or model going down.

Exploring using an off-the-shelf proxy for routing

Fin’s reliability and elasticity is a competitive advantage for Intercom, so we are happy to lean towards “build” in the build vs buy decision. This was more straightforward when we started building our routing framework as there weren’t many options available that provided the flexibility we needed.

But tools like LiteLLM and Bifrost have covered a lot of ground since then. While we are proud of what we have built, we don’t want to maintain a system if we don’t need to. We will still need to make sure we don’t introduce a new single point of failure by using these tools, but they look promising and are worth exploring.

Ultimately, the real measure of our work is that Fin stays available to solve problems for the people who depend on it. The work is ongoing, and reliability is never finished, but with these systems in place, we ensure Fin remains a reliable part of our customers’ workflows.

The post Fin: Running a Reliable Service over Unreliable Parts appeared first on /research.

The Promise and Problems of Reasoning Models

Reasoning models are the next frontier of LLMs; they’ve provided the breakthrough needed to continue the relentless increase in evaluation benchmarks we’ve seen since the release of ChatGPT-3.5 . They’re revolutionary, they’re accessible, they’re agentic. 

They’re also very, very, slow. 

They tackle difficult problems by producing “thinking tokens”. A traditional LLM will immediately begin producing the answer as soon as it has processed the user’s input. A reasoning model will potentially produce thousands of thinking tokens, considering options and working through the solution before it sends a single token back to the user.

However, along with the release of DeepSeek-R1 came small, open source, distilled models like DeepSeek-R1-Distill-Qwen-7B. While fast by LLM standards, these models were still too slow for our uses. At Intercom over the past year we’ve poured a huge amount of engineering effort into bringing the median start of the response from Fin down to 7 seconds. These new smaller, faster, reasoning models were multiples too slow to be used as a component of an agent as fast as Fin is.

Our AI group machine learning scientists posed the engineers in the AI Infra team a question: But could they be that fast? Ideally 3ms per token fast? They could then churn out 2000 tokens to think through a difficult problem and still be useful components.

In this post we present the series of steps we took to reduce the inference time from 11.35ms down to 3.1ms. Most can be readily applied to any LLM and should be considered when putting a model into production. More interestingly we’ll present the rationale so that readers may know when to apply each optimisation themselves. 

The optimisations applied were: 

• Conservative quantization to FP8.
• Substituting low latency kernels. 
• Tensor parallelisation to divide the work between GPUs. 
• Prefill/decode disaggregated serving.

We’ll also discuss what didn’t work, and what we didn’t attempt and why.

Optimisation Process 

Napkin math, Hardware and Software Choices

Before starting any optimisation process it’s useful to do some rough math to establish bounds and answer reasonable questions like “are we trying to break the speed of light today?”. Inference in LLMs is usually memory bandwidth bound due to the characteristics of modern GPUs. For instance, the Nvidia H200 can perform roughly 400 computations (floating point operations, or FLOPs) for each 16-bit parameter transferred — bandwidth is scarce relative to compute. Every single output token in a sequence requires us to load the entire model from memory and run the model’s calculations. A reasonable lower bound then on the physically possible is just “How long would it take to load the model from memory if we had no calculations, no overhead, and no coordination”. For DeepSeek-R1-Distill-Qwen-7B, a 7B model, at 16 bits/2 bytes per parameter then on a H200 with bandwidth of 4.8 TB/s this works out to:

$$T = \frac{14×10^9\:\textrm{bytes}}{4.8×10^{12}\:\textrm{bytes/s}}\approx0.002917\:\textrm{seconds}$$

Or about 2.9ms. So 3ms is possible! Or at least not ruled out by physical law. 

It’s worth noting that this is an unusual optimisation task — usually the goal is to maximise throughput so as to minimise dollars per request. So another useful reference point is to see what result other similar efforts have had. Amazon’s inference hardware advertises a low latency approach of splitting a Llama 7B model across 24 cores for a final result of 7.7 ms. Others have found latency optimised deployments of 8B models could hit about 5ms on an H100 GPU. Significantly lower latency with larger models has been achieved using Nvidia’s newer Blackwell GPUs. Unfortunately, as of the time of writing, these GPUs are not widely available. Consequently, we’ll focus on the Hopper family of GPUs. The results obtained with Blackwell GPUs are further discussed in the Appendix.

What we’re really attempting to optimise here is the Inter-Token Latency or ITL. This is the average time between consecutive output tokens.

As mentioned, in concrete terms the ITL is the time it takes to load all the weights, do a single pass of the model’s calculations, and write the produced token back. We know GPUs are bandwidth-constrained, but just how much time should we focus on optimising the loading of the weights versus the speed of the calculations? 

Thankfully, we already have the data for similar models. Taking a Llama 3.3 70B model running on an Nvidia H100 as an example: when processing a single sequence (no batching), it will spend ~100x as much time transferring the weights as doing the calculations. 

The easiest optimisation we can make therefore is to just pick the GPU with the largest bandwidth. The H200 is widely available and has about 40% more bandwidth than the H100 for about 10% more cost, so that was ideal. 

We chose the TensorRT-LLM inference engine as it provides a large number of optimisation options.

Benchmarking Approach

Many benchmarks found online focus on ChatGPT-like applications in ideal conditions where the number of input and output tokens are in the 100s. We were specifically undertaking this optimisation exercise to see if the model could operate fast enough for use within Fin. Fin is a production RAG system where prompts often contain thousands of tokens of relevant context and instruction. 

Accordingly, we’ll begin using a representative set of 100 prompts drawn from one of the most challenging parts of Fin for an LLM. These prompts average 6000 input tokens, are only about 10% cacheable, and when answered by a non-reasoning model average 260 output tokens. The prompts are used as follows:

• A fixed Requests-per-second (RPS) is used to generate outputs for the entire sample to determine average time to first token (TTFT) and ITL. The number of tokens returned for a request is limited to the exact length of the expected response in the sample dataset to ensure apples-to-apples comparison between models despite output length variance.
• Repeated runs ramped the RPS up from 1 until the model’s performance passed a maximum TTFT and ITL.

This provides us with a good way to characterise the throughput and latency of the model in a way that allows quick iteration and easy comparison with other models we’ve benchmarked. The throughput figure should not be taken as a literal real-world expectation however due to differences in output length between reasoning and non-reasoning models and due to how LLM requests are load-balanced. It should be viewed as a proxy for the amount of load a particular configuration can serve, and how that configuration scales with increasing load.

Given this is not reflective of how a reasoning model would behave in production due to its much longer average output length, at the end of the iterative optimisation process we’ll show how to validate the low latency results under a benchmark that mimics real world reasoning model deployment with longer outputs.

Quantization

So, how can we further speed up transferring the bytes from memory? Cheat. Transfer fewer bytes. Quantization lets us reduce the amount of memory taken up by a model. Often used just to make large models fit into constrained GPUs’ memory it also crucially reduces the time taken to transfer the model to the compute units and calculate the next token. To minimise impact on the results of the model the conservative FP8 quantization method was chosen — essentially just 8 bit floating point numbers instead of the existing 16 bit (there’s more to it than that of course, scaling, calibration, test data, etc. — but this isn’t a blog on quantization). The Nvidia ModelOpt framework was used via TensorRT-LLM’s wrapper. 

Even better, Hopper architecture Nvidia GPUs like the H200 contain specific hardware to accelerate FP8 calculations. 

We evaluated the model post quantization using the MMLU and GSM8K benchmarks, the pre and post quantization results of the model are in the table below. This was the only step where we performed accuracy evaluation as all the other optimization steps we took are lossless — it’s the same operations with the same weights, just being performed in different orders, or in parallel.

This step reduced ITL from 11.35ms to 6.68ms. A 40% reduction in latency to start with, or a 1.7x speedup. Not bad.

Low Latency Kernels 

The advantage of TensorRT-LLM over alternate LLM inference servers like vLLM and SGLang is its level of configurability during its build phase. Instead of loading a HuggingFace model directly it first converts it to a standard intermediate format, during which it can quantize and perform other operations on the parameters. Then it converts the model into a TensorRT “engine” that can be loaded by Nvidia’s TensorRT optimised inference framework. At this point you can choose from a wide range of build options, two of which are low latency kernels for FP8 models, namely low_latency_gemm_plugin and low_latency_gemm_swiglu_plugin . By swapping in these kernels we get a further reduction in latency from 6.68ms to 5.05ms on a single H200. Another 24% relative reduction, a 1.3x speedup.

Tensor Parallelism 

Tensor parallelism works by splitting large collections of weights across multiple GPUs. Operations that can be performed separately (such as large matrix multiplications) are then performed in parallel with the data being synced before operations that cannot be split. Tensor Parallelism in TensorRT-LLM is limited to a number of cards that divides evenly into the number of attention heads. Our model has 28 attention heads which results in a maximum tensor parallelism of 4 on an 8 GPU machine. 

Splitting between 2 cards we see an improvement to 3.84ms and between 4 cards brings it down to 3.15ms. We see diminishing returns as tensor parallelism is less effective in situations like the one we were optimising: 

• When your model or batch size is too small it becomes misaligned with the performance characteristics of modern GPUs which prefer to work on large matrices. 
• When you’re not performing enough computation to mask the extra communication between the cards.

It may seem economically unwise to go from 2 to 4 GPUs, however it will provide headroom when we scale the throughput in the final optimisation step. Regardless, another 37% speedup or 1.6x faster. That’s >300 tokens per second, more than a page of a novel each second. 

Disaggregated Serving and Throughput Scaling Result 

LLM inference can be split into prefill phase (generating the first token by processing the entire prompt in a single forward pass, a slow and compute bound operation) and a decode phase (doing a much faster pass for every subsequent token produced) The length of the prefill phase determines the TTFT and the length of the decode phase determines ITL. 

The prefill and decode phases can be separated onto different GPUs or even nodes. This is usually deployed due to its cost efficiency benefits: 

• Different hardware can be used that more suits the compute or memory requirement of each phase. 
• Even using the same hardware, if the limitation on throughput is that TTFT is too high you can use higher tensor parallelism in the prefill phase and if ITL is too high you can use more resources in the decode phase allowing you to decouple scaling requirements.
• The configuration of the inference engine can be completely tailored to the requirements of the phase — a larger number of tokens per forward pass for instance, something that benefits TTFT but slows ITL.

One less recognised advantage of disaggregated serving however is that it prevents the prefill phase from destroying the mean ITL of the decode phase. 

Normally when a new request arrives while a previous request is still in the decode phase the server has two choices — pause the decode while doing the prefill of the new request, or batch them together using Continuous Batching (the vLLM term) or In-flight batching (the TensorRT-LLM term). This is illustrated in the diagram below. 

At step 1 two new prompts arrive in requests R1 and R2. At step two we allocate these to an inference step and do the entire prefill phase for R1 and R2, allowing us to now produce output tokens. At step 3 we do a decode phase for R1 and R2, producing the next output token for each sequence. During step 3 a new prompt arrives to begin R3. At Step 4 we do the decode for R1 and R2 but additionally do the prefill phase for R3. Finally at step 5 we do a decode for R1,R2, and R3.

We can see the problem immediately from the “time taken” figures in the diagram – mixing the prefill and decode phases results in wildly varying ITLs. If we separate out the prefill onto a different server we preserve ITL regardless of how many requests arrive.

In the case of our sample dataset the impact is even worse than is shown in the image as our prompts are 6000 tokens long.

Adding a server dedicated to the prefill phase reduces the latency from 3.15ms to 3.10ms. We leave the speedup calculation as an exercise for the reader. It might seem surprising given the above example but there’s a straightforward reason we’re seeing so little improvement — we’re sending 1 RPS. Each request is limited to producing ~260 tokens. The model is now so optimised that 260 tokens takes about 0.8 seconds, so overlaps between requests have become rarer. 

Once we scale up the request rate however to determine the throughput limits of our architecture we see an entirely different story. 

On the chart below you can see the throughput scaling of a server using only 4 way tensor parallelism vs adding a single prefill server. Adding a single prefill server increases the maximum throughput with acceptable latency from under 30k tok/s to over 50k tok/s. 

However we’re hitting a wall once the requests per second pass 8 or so. This is due to the single prefill server hitting its limit, if we add another prefill server we can continue to increase the throughput again to over 60k tok/s while remaining below 4ms ITL, at the penalty of a slight ITL increase.

Unfortunately while the final result looks performant it’s completely ignoring the key benefit of the model — reasoning. 

Reasoning Model Benchmarking, How and Why Does it Scale, and Cost Calculation 

The advantage of reasoning models is that they scale at inference time by producing many extra thinking tokens to increase the quality of their outputs. However, this is a completely different scenario from the one we normally benchmark. Our standard benchmark approach limits the output to the length produced by non-reasoning models, and as a result the average output length is ~260. This is far too small for a reasoning model. 

When we remove this output limitation this increases to ~1700 tokens on average — which at 3.1ms per output token would take 5.3 seconds to process, if we sent 7 RPS to a server it would quickly result in a very high rate of concurrency and thus latency. 

The effect can be seen in the below chart where the 260 limit has been removed and tok/s craters to 3200 from over 60000 and RPS barely reaches 1 while exceeding 4ms ITL. This is exacerbated by the nature of reasoning requests where it is unpredictable how many thinking tokens each request will require resulting in much larger variance in concurrent requests.

So how would you actually serve this? Load balance intelligently. Inference Serving Frameworks like Nvidia Dynamo contain LLM aware routers that route requests away from already overloaded servers. 

So to benchmark in a way that’s more relevant to the way a reasoning model would be deployed in production we switched from scaling RPS to find a realistic throughput to instead scaling the number of concurrent requests with unlimited output. 

How Does it Scale, and Why? 

The result is in the chart below. After 3 concurrent requests our performance exceeds our requirements.

It may seem ridiculous that we top out at such a low number of concurrent requests — surely this isn’t the computational limit of the cards? The whole point of inference economics is that you can scale to large batches, serving many requests simultaneously for minimal extra performance penalty. Unfortunately that’s just not the problem we were optimising. 

So, what’s the culprit? Memory yet again. We’ve reduced the model itself to about 7GB through quantization. If that’s the limiting factor on our speed then our performance will be sensitive to transferring relatively small amounts of data. Relatively small amounts like the KV cache. 

The KV cache is an essential optimisation that turns the attention calculation at the core of the LLM from a quadratic operation to a linear operation per token. The tradeoff is that the size of the KV cache itself grows linearly with the length of the sequence being processed. In our case for a sequence of length 7500 it uses about 400MB. If you take the 3 concurrent requests into account this additional 17% use of memory bandwidth roughly matches the ITL increase we see as concurrent requests increase. 

Still, it’s not all negative, an interesting aspect of this new scaling behaviour is that we’re only introducing a new request about once every 2 seconds. That’s >10x slower than we established a single prefill server can handle. As a result of us splitting the prefill and the decode we can move that computation to a much much cheaper machine. Or split the cost of a single prefill server using a H200 between many servers outputting tokens. 

Costs and Conclusions

So, this must be prohibitively expensive? It depends. Let’s finish with some more napkin math.

The 4XH200 configuration we ended up with results in 3 requests every 6 seconds or so for our real world, long prompt use case — this is not a toy problem. That’s a request every 2 seconds. A standard AWS configuration for the H200 is 8XH200 in a single machine for ~$800 per day as of July 2025. We can host 2 decode servers on that single machine for a total output of 1 request completed per second, or 88400 per day. That works out at about a cent a request.

The appeal of reasoning models isn’t just that they solve problems better than prior LLMs, it’s that they also solve fundamentally different problems than LLMs, new problems. 

So the question becomes, is solving these new, different, problems by making an open source reasoning model operate at application speeds worth 1c a request? 

Appendix

Future Paths 

More aggressive quantization 

The FP8 quantization implemented is the most conservative quantization possible and while it was not optimised intensely it was chosen to maintain accuracy. Possible next steps that could be attempted include: 

• Auto quantization down to 6 bits: TensorRT-LLM’s quantization framework allows autoquantization using a variety of quantization methods to bring the average number of bits used per parameter down further. 
• AWQ and GPTQ quantization to test the effect on accuracy of reducing the model size down further to 4 bits.
• Manual choice of quantization on a per-layer basis can be performed.

Faster hardware — B200

The Blackwell B200 Nvidia cards have a memory bandwidth of 8TB/s or about 67% more than the H200 — this would be expected to provide a massive decrease in latency even without additional optimisations. Furthermore, just as the Hopper series introduced accelerated FP8 support, the Blackwell cards now offer FP4 support, expanding the options available for deploying low-latency, quantized models. Nvidia has demonstrated this capability on a system equipped with 8 B200 GPUs, achieving 1000 output tokens per second with the Llama 4 Maverick model and over 350 output tokens per second with the DeepSeek-R1 model (1k tokens in, 2k tokens out). These demonstrations involved extensive kernel-level optimisations, along with techniques discussed here, such as custom speculative decoding models and quantization.

Training Custom Speculative Decoding Models

Speculative decoding is the practice of producing multiple sequential candidate tokens on each forward pass that can then be evaluated on the next forward pass to potentially result in >1 tokens being accepted per iteration, and thus much lower latency. Approaches like EAGLE  require a smaller compatible model to be trained that runs in tandem with the original model. While no pretrained model is provided for DeepSeek R1-Distill-Qwen-7B the paper does cover DeepSeek-R1-Distill-Llama-8B and reports a 5x speedup over baseline 16 bit decoding in ideal conditions. As reported later in the paper the results drop to a more modest 1.75x improvement for a Llama 8B model when tested in a production server setting (vLLM) with a batch size of 2, and benefits decrease as batch size increases. 

Approaches that Failed or Weren’t Attempted 

KV Cache Quantization 

While the KV cache is a fundamental optimisation to modern LLM inference, it can take up a substantial amount of memory and thus occupy the memory bandwidth during inference. FP8 quantization of the KV cache was attempted, however the model appeared to be extremely sensitive to the optimisation as MMLU performance dropped to a score of 1.5 from a starting point of >50. Ideally an alternate quantization method would be more amenable to KV cache quantization with a potential total memory transfer reduction of ~8%.

Draft/Target Speculative decoding 

Speculative decoding, as described above, is extremely promising for low latency inference. Unfortunately our use case has several aspects that reduced its likely benefit: 

• Availability of appropriate draft models: The draft model produces candidate tokens that are evaluated on subsequent forward passes for their suitability by the main model. For the acceptance rate to be high enough to result in a latency decrease the model has to closely match the outputs of the main model. Unfortunately no suitably trained draft models existed — the mainline Qwen 2.5 0.5B model is incompatible with the 7B model for this purpose and community provided models were insufficient.
• Model size: For the greatest benefit there should be a large difference between the size of the main model and the size of the draft model. The size difference between a 7B model and a 0.5 model would likely result in much reduced speed improvement compared to often quoted results from 70B and >100B models. See squeezebits blog for details. 
• Sequence length: Many of the tests used to benchmark speculative models have been benchmarks like GSM8K and HumanEval. GSM8K contains 40-80 tokens per question. The average prompt length for HumanEval is 67 words. Speculative decoding reduces in utility as sequence length increases, again see squeezebits blog for details.

Lookahead Speculative Decoding

Lookahead decoding is a form of speculative decoding that generates a pool of likely n-grams and then verifies them in a single forward pass similarly to other speculative decoding approaches. Ideally it results in single forward passes where n tokens are output for a possible n times reduction in ITL. TensorRT-LLM can apply this approach automatically and so it was straightforward to evaluate. Unfortunately though a range of parameters were tested the method appeared sensitive to the length of the prompt, with the fastest lookahead run being 33% slower than the unadjusted version.

The post Think Fast: Reasoning at 3ms a Token appeared first on /research.

Fin relies on retrieval-augmented generation (RAG) to answer questions, so the quality of its responses depends on the quality of your content. While the AI Group has invested heavily in making both the retrieval and the generations better, we also needed to help improve documentation. Suggestions uses LLMs to generate targeted, pre-written updates, and delivers them in a simple UI so you can review, accept, or edit changes with a single click.

To generate these Suggestions we analyse all your unresolved conversations to recommend ‘Edit Suggestion’ or question-answer ‘Snippets’ based on how human agents handled these cases. This creates a self-improving loop, helping to enhance both the knowledge base and Fin’s resolution rate while giving more control and support to AI Specialists maintaining them.

You can find out more about its inner workings on our blog post on how we built the product.

Measuring the Impact of Suggestions

The question is: once suggestions get approved, how do we measure whether this has increased the resolution rate? Practitioners will naturally turn to A/B testing first: this is the gold standard to infer causality after all. But in our context, it is difficult and complex to run: on the one hand we would need to maintain a holdout version of their Knowledge Centre that would be able to accept critical updates while holding back Suggestions; on the other hand we would need customers to agree to run an A/B test that could be costly for them because of the additional volume going to their support team while it runs.

As an alternative, we decided to analyse conversations after the fact by applying causal inference techniques which reduces bias from potential confounding factors. We split conversations into those influenced by Suggestions (treated) and those that weren’t (untreated). By applying a matching method to find pairs of semantically similar conversations, we created a comparable ‘pseudo-control’ group from the untreated conversations, enabling us to estimate an unbiased effect of Suggestions. Our analysis showed that within the first month since launch using Suggestions resulted in an average increase of 1.23 percentage points in Fin’s resolution rate across active customers, with some achieving gains as high as 10.

Treated and Untreated Conversation Data

To measure the resolution impact using causal inference, let’s first define what the treatment set is: conversations where Fin used at least one impacted piece of content to answer a question. In most cases, these will be articles with a recent ‘Edit Suggestion’ or a recently approved ‘Snippet’ (a small question-answer piece of content that is not visible in the Knowledge Centre but can be used by Fin to answer specific questions.)

To create the treatment dataset, we define an arbitrary starting point in time, and collect conversations in a window between N days before and after (e.g. a week or a month). After the starting point, we keep track of all the suggestions merged during that period. All Fin answers that use documents impacted by at least one of these suggestions are assigned to the treatment set, as illustrated in the drawing. The reason we want to look for a week before will become clear once we describe our matching methodology – but in brief, we want to increase the chances of finding a good quality match for each treated conversation in case a customer approves a lot of suggestions and ends up impacting all their traffic.

Inferring a Causal Effect

When treatment can’t be randomised as in an A/B test, one cannot just compare the resolution rates of treated and untreated samples. In general, inferring causality requires two key ingredients: (1) ensuring the direction of the causal relationship between the treatment \( T \) and the dependent variable \( Y \) (resolution rate in our case); (2) accounting for confounders that influence both treatment assignment and the outcome. In our case, we do not have to worry about the direction because the treatment is always assigned before the resolution. Hence our focus is on removing the potential confounding bias by finding a good set of covariates that would act as a proxy for potential confounders.

In most causal inference use cases, there may be a plethora of confounders. In practice, it is often impossible to observe all of them, so we aim to identify variables that, either as true confounders or as proxies, capture most of the confounding effect. By conditioning on this set of variables \( X \), we assume that the treatment assignment \( T \) is independent of the potential outcomes \( (Y_0, Y_1) \) given \( X \) or \( (Y_0, Y_1) {\perp\!\!\!\perp} T |  X \). Under this crucial assumption, and given that treatment precedes the outcome, we can estimate the causal impact of \( T \) on \( Y \) by comparing mean outcomes between the treated and matched control groups.

Using Question Summaries as Proxy

So what could be a good variable to capture confounding? In the case of resolutions, a big role is played by the nature of the end user questions. Some questions are more meaningful or relevant than others; some questions have better content available in the knowledge base; AI Specialists prioritise topics considered more critical, etc.

We posit that using the ‘subject’ or ‘topic’ of these questions is likely the best way to account for most confounding factors. The main assumption is that similar conversations will share a lot of peculiarities that would impact both resolution rate and whether a suggestion has been generated by the system and approved by the specialist.

The good news is that we have access to summaries of user questions that Fin generates before answering. By embedding these summaries with a sentence transformer and measuring their cosine similarity, we can effectively match similar conversations.

Matching Treated and Untreated Conversations

‘Matching’ is a widely used method in causal inference (see references.) The basic idea is very simple, because of the confounding factors, we cannot compare the resolution rates of the treated and untreated conversation sets, simply because the treatment assignment is not neutral making the two sets not consistent. In other words, suggestions might impact a narrow group of questions, making the treatment set much more ‘specific’ than control. 

The way to get around this problem is by ‘matching’ (with replacement) each treated conversation with an untreated one maximising a similarity function on the covariate \( X \) that we want to condition on. This effectively creates a pseudo-control group with similar covariate distributions which makes it comparable to the treatment set. This allows for causal comparisons under the assumption that confounders have been accounted for in the matching process. By matching on the relevant covariates, we are, in effect, simulating what would happen if treatment were assigned randomly within groups defined by \( X \) , thereby approximating the conditional independence assumption \( (Y_0, Y_1) {\perp\!\!\!\perp} T |  X \) that underpins valid causal inference.

As mentioned above, there is a natural way for us to match conversations: 

1. Use concatenated question summaries (from one or more questions the end user asked Fin)
2. Embed them using a sentence transformer (eg. all-MiniLM-L6-v2)
3. Use cosine similarity to find the most similar untreated conversation for each treated conversation with replacement.
4. Finally, we prune pairs that are a very poor match (less than 0.5 cosine similarity); this risks loss of generality but will make the system more robust in edge cases, making it safer to run at scale.

Here are some examples with various levels of similarity (a conversation might contain more than one question):

Average Treatment Effect on the Treated

It is now a good time to address what we are measuring. As a reminder, our objective is to give the following information to our customer: “What was actually the impact of accepting suggestions on the resolution rate?” To answer such a question, the Average Treatment Effect on the Treated (ATET) is the most pertinent metric. In other words, the impact on the treated set, what has actually been influenced by suggestions.

$$ATET = E[Y_1 – Y_0 \mid T = 1]$$

Measuring the ATET is quite simple once the matching is done: we just look at the difference between the resolution rate of the pseudo-control and the treatment. Formally:

$$\widehat{ATET} = \frac{\sum_{i: T_i=1} y_i \, – \, y_{m(i)}}{\sum_i T_i}$$

Where \( m(i) \) is the index returned by the matching function \( m \). To measure business impact, we scale the ATET by the share of treated conversations to show the overall increase in resolution rate. For example, a 10% ATET with 10% of conversations treated results in a 1% overall boost on the Resolution rate, which is more material for business decision making.

Debiasing Matching Errors and Addressing Time Effects

Matching Error bias

In practice, it is impossible to achieve perfect matching. While some units may find exact matches (e.g., questions with identical summaries), most matches will be semantically similar but non-identical. This imperfect matching introduces matching error bias: as the quality of matches deteriorates (that is, as differences between matched units increase), the resulting bias in causal effect estimates could grow substantially, see Abatie (2011). 

Imperfect matching is a well-known limitation—some bias is almost always present, but it becomes problematic if matching quality is poor or matching error accumulates rapidly. This explains why we keep N days of conversations before the starting point of the analysis: we want to avoid cases where the suggestions impact an overwhelming proportion of conversations, shrinking the pool of untreated questions and impacting the average matching quality. Nevertheless, this is something we should address to increase the quality of the analysis.

Time-bias

Until now, we have ignored temporal effects. However, our experimental design is vulnerable to time-related confounding. Specifically, the likelihood of receiving the treatment \( T \) is positively correlated with time \( t \): as time progresses and more suggestions are approved, more requests are affected.

The main concern is that the resolution rate \( Y \) could also be influenced by time-varying factors such as model deployments, new features, or infrastructure improvements, etc. If these improvements tend to occur more recently and also improve resolution rates, then time acts as a confounder: it is associated with both treatment assignment and the outcome, potentially biasing our estimates upward.

Pragmatically, we chose not to include time in our matching similarity function because doing so would introduce an additional hyperparameter (balancing the weight of time versus semantic similarity). While semantic similarity is broadly applicable across customers, tuning a time-weight hyperparameter would require company-specific calibration and could substantially worsen matching quality if mis-specified. However, time bias is still something we will need to account for.

Debiasing

To address potential bias due to time and some poor matching quality, we can adjust the observed differences between treated units and their matched controls by estimating what their outcomes would have been had both not received the treatment, see Abatie (2011) for details.

The procedure is as follows:

1. We train a classifier (e.g., logistic regression) on the subset of untreated data to predict the probability of resolution as a function of observed covariates X and time t (one-hot encoded day): $$E[Y|X,t,T=0]$$
2. For each treated unit \( i \) and its matched (pseudo-control) unit \( m(i) \), we use this model to estimate their expected outcome had they not received the treatment: $$Bias_i= E[Y|x_i,t_i, T=0]-E[Y|x_{m(i)},t_{m(i)}, T=0]$$
3. This difference captures residual expected outcome differences due to time and sentence embeddings not addressed by the matching process. Intuitively, if there is a bump in resolution rate this will be captured by the variables \( T \).
4. The estimate for the treatment effect can be effectively “de-biased” by subtracting this quantity from the observed difference in outcomes:

$$\widehat{ATET} = \frac{\sum_{i: T_i=1} y_i \, – \, y_{m(i)} \, – \, Bias_i}{\sum_i T_i}$$

This approach adjusts for any systematic differences: such as changes between days—that the matching did not balance but which can be predicted using untreated data.

Results so far

We have run the analysis on a sample of around 1000 customers that have approved suggestions, and the results are clear: adding suggestions can help you increase your resolution rate. We have run the analysis of the first month of usage, taking a month before and after the launch date. The weighted average (per customer size) shows an increase of 1.23 percentage point.

Analysing our most active customers (see Table bellow for top 10) we have noticed the following:

• Suggestions tend to focus on high traffic questions. Approving them can have a far reaching impact. Some customers managed to impact up to 76% of their Fin involved conversations by approving as little as 8 suggestions.
• The more you approve, the more impact you have with a correlation of 0.22 between total approved and Resolution Rate impact. Although, sometimes 1 or 2 well targeted suggestions can fill a critical gap and have a major impact.
• Although the tool can lead to significant improvements early on, our system runs constantly and will always find fresh improvements as long as customer representatives need to resolve end user issues. 
• Most customers have at least one approval per week. Showing repeated usage with time.

Top 10 most active in terms of number of conversations impacted by suggestions:

Conclusion

This analysis clearly shows that Suggestions create measurable, meaningful gains in resolution rates for our customers. By applying robust causal inference methods to real-world data, we’ve demonstrated that Suggestions don’t just close the feedback loop; Suggestions actively drive continuous improvement in both Fin’s accuracy and customer support outcomes.

With Suggestions, AI Specialists finally have a scalable way to improve Fin, spot knowledge gaps, and deliver better answers. The results speak for themselves: approving Suggestions leads to higher resolution rates, and those who adopt them widely see the greatest benefits.

This is just the beginning. As we keep refining Suggestions, expect more powerful features: surfacing duplicates, highlighting contradictions, and even recommending database read actions that will make Fin’s answers more personalised than ever. Stay tuned! Your smartest AI support experience is always getting smarter.

References

Abadie, A., & Imbens, G. W. (2011). Bias-corrected matching estimators for average treatment effects. Journal of Business & Economic Statistics, 29(1), 1-11. https://doi.org/10.1198/jbes.2009.07333

For an intuitive introduction we recommend:

Facure, M. (2022). “Causal Inference for the Brave and True”, 10. Matching, from https://matheusfacure.github.io/python-causality-handbook/10-Matching.html

The post A Causal Inference Approach to Measuring the Impact of Improved RAG Content appeared first on /research.

Today, AI Customer Agents like Fin are handling more tickets than humans do. But as the conversation volume they handle continues to grow, the bottleneck becomes high quality knowledge, and keeping your knowledge center up-to-date.

On May 21st, we launched Insights, an AI-powered suite of products that delivers real-time visibility into your entire customer experience. As part of Insights, we built ‘Suggestions’ to help improve knowledge center documentation and Fin’s performance.

Fin relies on retrieval-augmented generation (RAG) to answer questions, so the quality of its responses depends on the quality of your content. While the AI Group has invested heavily in making both the retrieval and the generations better, we also needed to help improve documentation. Suggestions uses LLMs to generate targeted, pre-written updates, and delivers them in a simple UI so you can review, accept, or edit changes with a single click.

To generate these Suggestions we analyse all your unresolved conversations to recommend ‘Edit Suggestion’ or question-answer ‘Snippets’ based on how human agents handled these cases. This creates a self-improving loop, helping to enhance both the knowledge base and Fin’s resolution rate while giving more control and support to AI Specialists maintaining them.

Since the launch of this new feature, the average approval rate of new Suggestions is around 60%, and we have found strong evidence of positive impact on resolution rate. For a detailed explanation of how suggestion effects are calculated, please refer to the companion article. You can access it through this link.

Building the Suggestions product 

Our system began as a single conversation extraction system called Inbox Snippets and evolved into the current iteration of the Content Suggestions. The key success of this new feature is a new architecture that we’re calling `accumulation driven extraction`.  In this article, we’ll present the main features of both of the systems and the reasoning behind the improvements and trade-offs we made to achieve these results.

This new Accumulation Driven Extraction architecture enables us to gather multiple conversations as evidence for new content generation.  This new system addresses previous issues from the Inbox Snippets like content duplication and low relevance by generating suggestions only when there is evidence from multiple customer conversations. This approach also significantly improves the content quality. 

Extracting Answers from Conversations

LLMs demonstrate significant strengths in summarizing and rewriting the information found in documents. 

Because Fin utilizes a customer’s knowledge center to answer new customer questions, a lack of up-to-date data could result in a poor customer experience and missed opportunities to answer informational queries.

Our hypothesis was that we could leverage the conversations that Fin couldn’t answer but that customer agents did resolve as a source of up-to-date information, by extracting and summarizing these questions, then adding these snippets to the knowledge center so Fin can use fresh information next time another customer asks the same question. This creates a Data Flywheel for Fin. In this context, a Data Flywheel refers to a system that continuously updates itself with new data, enhancing its performance over time and counteracting data drift so as to achieve ongoing improvement.

Solution Architecture

The current system is based on extracting question and answer pairs from relevant conversations, grouping them based on their similarity and triggering them only when certain conditions are met. This structure has many benefits especially for improving the suggestion quality, being able to gather multiple pieces of evidence for any information included and reducing the risk of duplicated suggestions. Finding and eliminating potential duplicates has been a key consideration in the design of this architecture. It achieves this through three primary methods::

1. Detection After Extraction: Each Q&A pair is compared against existing knowledge center articles. If the information is identical, the pair is discarded. On average, this step eliminates around 20% of all generated pairs.
2. Accumulation Grouping: Similar Q&A pairs are grouped together, and these groups remain active even after a suggestion is generated. If a duplicate slips through initial detection, it is still likely to be caught at this stage. Approximately 8% of Q&A pairs are absorbed into these existing groups, effectively preventing duplication.
3. Detection After Generation: Though less frequent, a final comparison with the knowledge center is conducted post-generation. Content found to be too similar to existing articles is removed. This step accounts for the elimination of about 3% of all generated content.

Create Suggestions and Edit Suggestions

Once the content is generated, it can be added to the knowledge center as a stand alone document. 

However, as we started to generate longer and more thorough documents, these pieces of content started to have more sections that were natural extensions of the other existing documents in the knowledge center. Also, these sections often covered similar topics from different angles.

To resolve this issue once we generate a suggestion, we check whether it is similar enough to any of the existing documents and if they are similar enough, these are surfaced as Edit Suggestions.

If we don’t find any similarity to the existing content, we instead  suggest a standalone document. This is what we call Create Suggestions.

The Foundation: Inbox Snippets

The first iteration of what eventually became today’s automatic content generation was a feature we first built in 2023, called ‘Inbox Snippets’. This feature’s main goal was to capture the answer that resolved a customer’s question after the end-user requested to talk to a human agent. This content was then presented as a simple question and answer to the customer agent before the conversation’s closure. The new content required approval from the agent to be added as a snippet.

To achieve this, we:

• Discard:
  • short conversations
  • answers that are too similar to knowledge center articles
• Limit the conversation size to the last 10,000 characters
• Extract all questions and answers in the conversations
• Discard irrelevant or AI-generated Answers
• Summarize high-quality questions
• Present to the customer agent to decide whether to add the snippet or discard

To prevent overwhelming agents, we limited daily recommendations to 3 per agent.

It Worked! But With Caveats

The Inbox Snippets feature gathered a lot of attention and excitement after its launch in November 2023. By November 2024, over 80% of all active Fin customers used the Inbox Snippets. It helped to generate 91,000 snippets.

Over this time, we conducted multiple A/B tests to verify the snippets’ contribution to the resolution rate. While we found a positive effect from these ablation studies, it was difficult to fully quantify the effects. Some of the reasons are as below:

• AI Specialists frequently copy the contents of snippets into articles, thus compromising the independence of the A/B test (up to 30% of the information in the snippets was replicated into articles).
• As the knowledge centers are dynamic environments, the information within them evolves constantly, creating multiple paths to the same resolution, making simple ablation studies hard to conduct, for further discussion on this please refer to the companion article, here.

Despite the positive effects, customer reception and high usage did not fully translate into a high approval rate for the snippets. By July 2024, over nine months after its release, agents accepted only 12% of the generated snippets. They reject most of the snippets immediately or snoozed, only to be reject them in bulk later.

After analyzing the rejection reasons of the snippets and collecting direct customer feedback, we identified the following major issues:

• Too much content: an excessive number of snippets were recommended at any given time. Customers initially expected to receive a large batch of snippets, anticipating a reduction in recommendations over time. However, the number of snippets remained generally consistent, even for customers who accepted many.
• Capturing the low relevance questions: a lot of the customer conversations can focus on the specific information that is focused on a single customer’s case. It is difficult to determine if a conversation is generalizable just by looking at it, and even comparing it with the knowledge center content doesn’t always help because snippets bring new information by nature. Customers regularly complained about receiving frequent snippets that could not be generalized for this reason.
• Wrong approval point: with Inbox Snippets, customer agents decide whether information should be added. However, agents often lack the time to evaluate if the information should be permanent and may need to consult experts. The UI, however, forced an immediate decision. This tension resulted in many agents rejecting high quality snippets.
• Duplicated Content: the system processed all conversations involving Fin, which triggered many content generations. Although we had checks to detect existing information in the knowledge center, duplicate content was still generated. This caused reduced trust in the product from customers and a frustration from agents as it wasn’t easy to check if the information was duplicated in the knowledge center already.

Addressing the Low Approval Rates

Input Guardrails

First, we introduced input guardrails to more intelligently filter out conversations before we attempt to extract anything. For this step, we added a smaller LLM judge to sanity check if the conversation is above the desired quality. This LLM judge checks the following parameters:

• Make sure that the conversations have a full cycle. A full-cycle conversation is one where an agent has answered at least one of the customer’s questions, and the customer has responded to that answer
• Detect spam or abusive conversations
• Make sure that there are conversation is due to the informational answers and not by actions from the human agent

Output Guardrails

Output Guardrails were added to detect any issues leftover after the extraction was complete. It acts as a second pass to detect:

• PII
• Time sensitivity
• Non-informational answers (such as agent changing some data manually)
• Personal links and URLs
• Phatic expressions
• Situational interactions (e.g. account verifications)

If any of these issues were detected, the output guardrails would try to rewrite the information without these, and if the rewritten content lost the critical information that could make the suggestion unviable, the system discarded the content.

Better Duplicate Detection

The original system we implemented checked whether the customer agent’s response was a direct copy from an existing knowledge center article. While this actually was successful in filtering out around 3% of all extractions, it failed whenever the agent altered the text semantically or added any information.

To improve detection, we added an LLM judge that also evaluated the generated content and checked for similar passages in the existing knowledge center.

We considered evaluations in the following details:

• Identical: The new generation is completely covered by the knowledge center article
• Similar with variations: There is some similar information but some novel information exists as well
• Unrelated: There are no overlapping information between the texts

Impact on Snippets Approval Rates

Following the implementation of all these guardrails, the approval rates of the snippets increased to 28% from the initial 12%. 

Filtering out lower quality conversations, rewriting bad quality outputs and catching duplicates helped, but we still suffered from a high number of snippets being generated. We also couldn’t solve the problem ofthe snippets capturing non-generalizable information. 

While much better than the initial 12%, 28% approval rate didn’t signal a well trusted solution by the users. That led to seeking alternatives that changed the approach more dramatically, leading us to Accumulation Driven Extraction architecture.

Extracting via Accumulation: Content Suggestions

While the improvements described above doubled the approval rates of automatically generated content, the issues of generalization and duplication persisted. The number of duplicated content decreased but the issue of too many snippets and the non-generalisable information were only affected marginally by these changes. 

The solution we came up with is to add an additional layer between extraction and generation. This layer would group similar questions together and only generate content for information that can be verified from multiple conversations. After extracting all of the questions from the conversations, the questions are grouped by the accumulation system and only if specific conditions are met, these groups are moved to the generation stage, where we attempt to validate the information presented from multiple conversations and propose a final document for the customers to review:

Previous improvements, such as duplicate reduction, are still part of the extraction pipeline.

Notably, we shifted the decision-making from customer agents to AI Specialists, providing them with multiple opportunities to validate the underlying conversations and understand their necessity. Here’s how it looks in the UI:

The extraction has been divided into four independent layers:

• Extraction: filters out low quality conversations and extracts any resolution source found in the document, also checks if the extracted information already exists.
• Accumulation: checks whether a similar question has been asked, and groups them together.
• Trigger: checks whether trigger conditions are met. The trigger conditions could be based on the volume, a sudden spike in the group or persistent long term queries that might have lower volume.
• Generation: selects the relevant question answer pairs that can be verified from the triggered group and generates content. There are two kinds of suggestions:
  • Create Suggestions: if the new content is not substantially similar to any existing content available to Fin, we recommend adding it as a standalone article.
  • Edit Suggestions: if the newly generated content is similar to an existing article, the system can suggest adding the new information to that article. The system might decide to generate the edit, or AI Specialists can decide to add the content to an article themselves.

This new architecture solves the previous issues as detailed below:

• Too Much Content: as the system waits until there’s enough evidence, the singular questions from customers never get to be surfaced. This acts as a barrier also after the suggestion is surfaced to ensure that we don’t pose similar suggestions.
• Generalizable Information: by cross-checking many conversations, we can confirm that the same information is being repeated to multiple customers, creating a more solid foundation for generation.
• Duplicated Content: the questions groups formed in the accumulation step forms an effective catch-all even after the generation.

Trade-Offs

Lower Suggestion Volume for Higher Relevance

The system deliberately waits for content to accumulate and finds multiple pieces of evidence to cross-check the validity of any information that is discovered from the conversations. This requires a significantly higher volume of conversations. To mitigate this, we implemented a variable triggering structure that sets the threshold to trigger a suggestion based on conversation volume. While this partially offsets the effect, the trade-off remains: companies with lower conversation volumes (under 500 per month) will receive fewer suggestions than they might have with the previous, less precise system.

Higher Cost-per-Suggestion for Greater Accuracy

The new system is more complex and as we rely on multiple pieces of evidence, this relies on substantially longer inputs to our LLM services, making the cost-per-suggestion significantly higher than in the previous system.

This increased cost-per-suggestion is balanced by a lower frequency of generation. Due to our aggressive, multi-stage filtering, the system generates suggestions less often, leading to a more manageable total cost. We also put a limit to the number of suggestions generated in a week as the customers are unable to review too many generations at a time.

Results

The new suggestions experience has moved to Open Beta after the Intercom’s Built For You event in May 2025. Within the first month, we have seen rapid adoption and the customers added over 5,000 suggestions to their knowledge centers. 

During this month, we have seen a massive increase of approval rates

The articles with accepted suggestions have so far been used in over 100k conversations and contributed to over 70k resolutions made by Fin.

We estimate an overall 1.2% absolute resolution rate improvement. The full distribution of the resolution rate results can be seen below.

The results and the graphs above are taken from our companion article on how we measured the effectiveness of the suggestions feature. Make sure to check that article to understand how we think about measuring the impact of Suggestions.

Future Of Suggestions

With Suggestions, AI Specialists finally have a scalable way to teach Fin, spot knowledge gaps, and deliver smarter answers—faster and better. The results speak for themselves: embracing Suggestions leads to higher resolution rates, and those who adopt them widely see the greatest benefits.

This is just the beginning. As we keep refining Suggestions, we’ll be working on smarter features: surfacing duplicates, highlighting contradictions, and even recommending actions that will make Fin more personalized than ever. Stay tuned—we’re grokking even more sophisticated Insights into AI support experience provided by Fin.

The post Generating Knowledge Center Content from Customer Service Conversations appeared first on /research.

When we set out to design our AI retrieval systems, we began with first principles, not vendor promises. Our research showed that Elasticsearch struck the right balance of performance, cost, operability, and long-term maintainability – at least for our needs, and given our experience operating it at scale.

In this post we share our rationale, and two years of operational results running Fin, the market’s most advanced AI agent.

Initial System: In-Memory Retrieval

In early versions, each customer’s content and embeddings lived on S3. For each inference request, we:

1. Retrieved the customer’s S3 object
2. Loaded all vectors into memory
3. Ran a brute-force KNN search

This design was a good starting point:

• The “database” could handle any number of requests or spikes
• Iteration was simple
• Our scientists could experiment easily, leveraging familiar tools like notebooks and pandas

It also was battle-tested to some degree, having served our previous ‘Resolution Bot’ product for years (which used much smaller datasets).

But as we added support for new content types, and began generating content from conversations, the number of embeddings for each customer soared. Soon, S3 download and deserialization times began to dominate request latency— we started to see spikes of 15 seconds just to load vectors for our largest customers, before even starting the search or LLM inference.

Requirements & Constraints

Prior to committing to a particular technology, we defined concrete constraints:

• Scale: Target of 100M+ embeddings of 768 dimensions, accounting for accelerating growth
• Cost: Consider infrastructure and ongoing operational burden
• Filtering support: Must be able to filter retrieval (by language, content type, audience, etc. to support our platform and permissions)

Other nice-to-haves included:

• Run Less Software: We want to avoid undifferentiated heavy lifts
• Full text search: We might want to move to hybrid search in the future
• Predictability: Known scaling properties, failure modes, and monitoring

Looking back, even our optimistic scale estimate of 100 million embeddings was much too low!

Surveying the Landscape

We evaluated Pinecone, Milvus, Qdrant, Weaviate, and Elasticsearch. All had the basic features we required, so cost initially seemed like the main differentiator. At the time, we had about 20 million embeddings; we projected costs for 5x that scale—100 million embeddings.

Several promising open-source solutions offered theoretical top-end performance, but with potentially increased operability risks. And the managed services tended to be more expensive — Pinecone and Milvus, for example, were estimated to be 2x the cost of their open-source alternatives.

In the end, the primary differentiator for us was operational familiarity and proven reliability. Elasticsearch, already core to several business-critical Intercom systems, presented clear advantages here.

Benchmarking: Vector Search in Elasticsearch

We began to look at standard benchmarks:

• Elasticsearch nightly vector search benchmarks (2M embeddings, 768D): search latency between 100ms (nightly-so_vector-script-score-query-match-all-latency) and 200ms (nightly-so_vector-script-score-query-acceptedAnswerId-filter71%-latency).
• ANN-Benchmarks: Elasticsearch sat in the center of the pack, neither the fastest nor the slowest.

And verified these against our in-house benchmarks on production-like datasets, which gave similar results:

Overall, this performance was good enough for us. For an AI Agent, the bottleneck will generally be the LLM’s latency, which is measured in seconds. Even if search latency dropped to zero, we only stand to save ~200ms. This would be imperceptible to the user in most cases.

Decision Criteria: Why We Chose Elasticsearch

Beyond raw benchmark numbers, the following factored heavily in the final decision:

Low Infrastructure Cost

Just accounting for the infrastructure, we estimated Elasticsearch would cost us the least to run and the gap would widen as we scaled up, especially compared to the Managed offerings.

Initially, it would be even cheaper to get started as we planned to run on a shared large scale Elasticsearch cluster that Intercom used for multiple use cases. And we could migrate to a separate infrastructure when necessary.

Low Onboarding Cost

Since we would not be running a new distributed system, we didn’t need new runbooks and could avoid common scaling/maintenance issues and failure modes. 

We could also reuse most of the tooling around running a database: disaster recovery, snapshots, replication.

Subject-Matter Expertise

Elasticsearch is one of the “core technologies” at Intercom. We have experience running it and have subject matter experts within the company (1, 2) who have familiarity with how ES scales. They can proactively manage cluster health, be able to tweak capacity or performance by playing with multiple levers like the number of shards in an index, size of the data nodes etc.

Filterable Hybrid Search

The ability to compose vector and structured filters with existing ES query DSL is a requirement most vector-first DBs do not easily fulfill. With Elasticsearch, we can combine structured filters and vector search with full-text search to improve relevance of the results in the future. 

Possibility to move to Approximate KNN in the Future

Since Exact KNN performed adequately for us, we could start with it and avoid having to worry about recall. This simplified the migration. In the future, we could move to ANN for faster results.

Production Results

Two years later, our scale exceeded even our most ambitious forecasts by 3x—a welcome problem.

Note: The actual number of embeddings we have is ~600 Million as we are continually experimenting with different embedding models or chunking strategies. The cost here is for this total architecture: 600 Million embeddings, replicated once, with raw data and metadata. Since we don’t currently create an index for ANN, which usually requires as much memory/disk space as the size of embeddings, we use 300 Million as the number for a fair cost comparison with other vendors.

Compared to other vendors, this setup costs us at least 3 times less (comparing just the unblended cost for fairness). For example: Qdrant’s estimate is ~$30k/month (not accounting for capacity needed for experimentation). Pinecone would be even costlier.

Specifics of our Architecture

• Architecture
  • Data nodes: 9 x i4g.4xlarge
  • Client/Master nodes:  3 each x c6g.xlarge 
• Index
  • Embeddings are partitioned into 15 indexes, each with 30 primary shards and a replica.
  • Refresh Interval: 2 seconds 
  • The embeddings field is a dense_vector with index=False.
• Ingestion
  • Ingest on every create/update. 
  • Average request processes 3 contents (articles, file etc.), which are chunked and bulk-ingested.
  • In the future, we can buffer content updates for a few seconds before processing them to improve ingestion efficiency if needed.
• Querying
  • Exact vector-search using script_score and functions for vector fields.
  • Queries use filters for locale, content that should be visible/invisible to a user etc.

Business Impact

The migration from S3/in-memory to Elasticsearch drove customer-visible response time improvements:

The system has also absorbed 10x increases in customer data volume and query traffic without architectural change, cluster downtime, or significant operational incident. Most importantly, all operational practices—from tuning, upgrades, snapshotting, disaster recovery, to scaling—were well understood from prior ES experience.

Conclusion: Practical Takeaways

When picking tools or infrastructure—whether it’s for vector search, databases, or anything else—these principles worked well for us:

• Leverage what your team already knows. Familiar tools let you move faster, reduce onboarding, incidents, and painful outages. If your team already has expertise in a given technology, that’s a huge advantage.
• Expect trade-offs. Cutting-edge technology (like vector databases) might promise slightly lower latency or higher throughput, but come with new operational risks, documentation gaps, and complex migrations. Make sure those trade-offs are worth it for your business.
• Don’t underestimate the costs outside infrastructure. The time spent learning, debugging, and supporting a new system often outweighs the dollar savings or benchmark wins.

Our experience is not a call to avoid new technology that’s better suited for your situation, but to reassure you that a “boring” choice might not necessarily be the wrong one.

The post Do you really need a Vector Search Database? appeared first on /research.

Why is customer support such a challenging space? 

Over the past few months, Large Language Models (LLMs) have significantly advanced. Products like ‘computer use’ from Anthropic and OpenAI, and DeepResearch by OpenAI, demonstrate LLMs’ increasing capability in high-agency tasks.

High-agency agents are those where an agent’s actions are primarily self-governed, constrained only by its environment, and a goal.

However, most examples of high agency agents operate in ideal environments which provide complete knowledge to the agent, and are ‘patient’ to erroneous or flaky interactions. That is, the agent has access to the complete snapshot of its environment at all times, and the environment is forgiving of its mistakes. 

This contrasts sharply with customer support agents, like Fin, who generally have knowledge gaps since they are configured by real humans, and interact with humans who may be incoherent, and are often impatient and frustrated. In addition, the agents are highly constrained by how much time they can spend solving a problem, as latency is a core user experience issue.

What are the customer’s expectations from agents?

Fin has excelled at addressing informational queries using its state of the art RAG framework. This framework has achieved resolution rates in the high 60s for customers with well-developed knowledge bases.

However, as Fin resolves more informational queries, we are seeing a rise in demand for it to solve complex problems. Achieving this introduces many requirements, like conversational debugging by gathering personalised context, getting personalised data from external APIs, asking questions, executing business decision logic, and much more.

Typical use cases from our customers include: 

1. Subscription/order management: renewing, cancelling, pausing subscriptions 
2. Credit refunds, typically conditional on account state and/or metadata
3. Context gathering, and decision logic based on gathered context values
4. For sensitive issues, gathering personal context from the end user, before escalating to humans to finally resolve them

The hidden requirement behind all of these use cases, which are typically quite sensitive for the business, is that customers require a very high level of reliability and control.

This means they want to tune how Fin asks questions, takes branching decisions, and calls external APIs. But they also want a guarantee that Fin would make the same decision reliably and consistently over hundreds of conversations, no matter how the user expresses themselves!

This means that a successful agent should possess (1) high agency, (2) solve highly complex problems, (3) with very high levels of reliability.

To begin to address these customer expectations, it becomes essential to develop a robust method for measuring agent performance.

We we discuss our measurement in the context of our work to build our ‘Give Fin a Task’ (GFAT) agent, which is a agent within Fin designed to allow customers to achieve the high reliability they want, by constraining some agency.

Measuring agents 

Fig.1 : Balancing the AI agent’s agency and control with reliability for complex tasks is hard. Customers typically want high reliability, even if it comes at a cost of some agency. “Give Fin a Task” is a variant of AI agent that fits this requirement, and with its composable nature, customers can progressively solve more complex tasks.

Assessing the competence of customer support agents is challenging because performance depends on both task complexity and the level of agency an agent has (Fig. 1).

A practical way to measure reliability is to evaluate how consistently an agent completes a given task when repeated multiple times. This is particularly relevant for customer support, where agents frequently handle repetitive requests, such as processing refunds or cancellations.

Simulated Task Testing

To measure agent performance, we simulate a “Task” where the agent interacts with a simulated “end user” over multiple turns. The simulated user, powered by another LLM, follows a predefined script to mimic real customer interactions while attempting to resolve a specific support issue. Each set of such interactions forms a “test,” which can be repeated as needed. A test consists of:

1. Expected outcomes – Clearly defined success criteria for the test, such as an expected order status, a specific value like a refund amount, or required API calls (e.g., getShopifyOrders) to retrieve additional information.
2. Simulated user prompt – Defines how the user interacts with the agent, including their persona and communication style.
3. Stopping condition – Reached when the agent fails to meet the task requirements or exceeds the maximum number of interaction turns.

The test concludes when either the expected outcomes are met or the stopping condition is reached. For example, in an order shipment inquiry, the agent must extract the correct user ID, call the appropriate function to retrieve the order status, interpret the response, and clearly communicate it to the user. Only when all these expectations are met, the test would be considered a pass. 

Realistic User Simulation

To reflect real-world interactions with Fin, we model customer behaviors such as impatience, brevity, and incomplete information. These factors, combined with task complexity and imperfect API specifications, create a realistic testing environment.

By simulating diverse agentic tasks, we gain insights into how different LLM agent architectures perform under real-world conditions.

The Metric

Most benchmarks use pass@k, which measures the percentage of tasks successfully completed at least once in k repetitions. However, this is not a useful measure of reliability for customer support agents.

Instead, we need the total percentage of tasks successfully completed every time when repeated k times.  This metric is widely known as pass^k. This is a much stricter metric, as it requires consistent success rather than a single correct attempt. The choice of k is indicative rather than absolute—it helps identify trends but does not perfectly reflect real-world expectations. In practice, no two customer interactions are identical, making absolute replication unlikely.

Agency, Control, and Reliability 

Agency and control are a tradeoff, which means the more free the agent is to take actions, the less can our customers control its behaviour, which sounds intuitive. But also as seen from our experiments with our benchmark tests, agency and reliability is also a tradeoff. Allowing an agent to take long horizon decisions bounded only by the availability of tools is a recipe for unreliable behaviour. 

The graph below shows the computed value of the pass rate metric across all our tests, at different values of k. This graph suggests that if someone expects an LLM based agent to solve a high agency task over multiple customer support conversations, they will be disappointed by their success rates. But even this graph tells only half the story 

Fig 2. Task pass rates across all tests for high agency function calling agents, with different values of K for repetitions. The dotted trend line indicates that the reliability of these agents falls rapidly.

To get a deeper insight into the agent’s behaviour, we classified the tasks in our benchmark into three categories, 

• Simple Tasks: Involve independent context gathering or straightforward action calls. This might include a one-off status check for an order. 
• Moderate Tasks: Require conditional branching or chaining, such as using one action’s output as input for another. This might involve use cases like validation of some information from the user, but first authenticating them, checking their accounts, and then gathering context for validation. 
• Complex Tasks: Involve multiple pathways to achieve a goal, imperfect information (e.g., a user forgetting credentials), and extensive context gathering and chaining. These are much more complicated tasks where we are dealing with a user who has incomplete information, or a user that wants to do several changes to their state, or simply a difficult user.

When we run the experiments just on simple and complex tasks as classified by the above rubric, the metric of pass^k looks very different. While agents perform in line with the overall performance as seen in Fig 2 for simple tasks, their performance drops by a huge margin on complex tasks. Function calling agents using older models like Claude sonnet, perform slightly better on simple tasks compared to reasoning models like O3. Reasoning models gain some advantage on complex tasks, but not enough to be optimistic.

Fig 3: Difference in the agent’s pass^k performance on simple vs complex tasks

The above results show that in the context of customer support, the LLMs and their agentic capabilities are not yet at a level of reliability to solve the full range of task complexities that our customers might experience. Low reliability might directly impact CSAT and escalations, while making agent interactions more frustrating for our end users. 

So what’s the alternative at this point? 

We believe that the primary challenge for LLM based support agents in the immediate future is to maximise their performance along the Agency and Complexity dimension (Fig 1), and progressively move them towards the top right corner of this grid. But there is ample evidence as of today, that the reliability is inversely proportional to the agency of an agent, and this tradeoff seems non-negotiable for the current state of LLMs. So maximising agency on highly open ended and complex customer support tasks would likely come at the cost of the agent’s reliability, and therefore their impact on resolutions. 

That said, the results suggest that if we constrain the agent’s autonomy and frame each task as a simple, well-structured instruction—minimising ambiguity and room for interpretation—we might be able to significantly boost reliability of the agent, potentially reaching resolution rates in the high 60s, comparable to those seen with informational queries. This might pose as a great stop gap solution for an agentic customer support tool, before we reach a fully capable agent that could competently handle open ended high agency tasks. 

So we need to make a bet. Initial customer research shows that our customers are willing to accept a lower level of agency (and higher level of control) in Fin, if it means a much more reliable product. With this in mind, we have designed the “Give Fin A Task” agent which is going to be an integral part of the Fin Tasks feature in Intercom.

Fig 4: Performance on tasks strictly structured as steps with two variants of agents: an agent designed to strictly follow a steps based execution for a steps based task structure (GFAT agent), and a conventional function calling agent but one that still uses a steps based task structure. We have used Claude 3.5 Sonnet V2 to generate the above results

This agentic product  would strongly lean on the following key concepts : 

• Control: Since the agent leverages the pre-existing, and familiar workflow-like environment, a teammate would find it easy to build complex workflows with agentic blocks that do narrower well defined tasks, while achieving a bigger and more complex end goal via composition.
• Agency: Although we will trade off agency for control, we would build these agents to be highly expressive via steps based instructions for the agent. The Give Fin a Task agents would be configured using steps based instructions to fulfill a task, where each step is an executable instruction. The agent is designed to follow these steps reliably, allowing high levels of reliability for narrower and moderately complex tasks, while still maintaining all the conversational benefits of an agent powered by a LLM. 
• Composability: : The product would allow teammates to break down highly complicated tasks into narrower and more reliable tasks, thereby pushing the overall reliability of the individual task to a much higher level, when compared to an unconstrained agentic approach. This would also allow teammates to compose much more complex workflows, with lots of such individual “Give Fin a Task” blocks. 
• Feedback loops: Based on the research done in the team, we will build tooling that
  • Helps the teammate to write task prompts that maximise reliability, without meaningfully compromising the agency.
  • Provides insights into the reliability of a configured task prompt to the teammate, and provides suggestions for improvements 
  • Provide meaningful metrics around the agentic blocks in the workflow, once they are in production. 

This bet has shown a lot of promise in our early benchmarking tests, as seen in Fig 4. The Give Fin a Task agent shows significantly higher levels of Pass^k performance (k = 3,4,5), on tasks configured using the steps based structure and executed by the steps executor agent (GFAT), which has a tempered level of agency. The steps executor agent is restricted in what it can do, by the steps based structure of its instructions, therefore reducing its agency when compared against pure function calling agents. 

The performance on simple tasks show a very high degree of reliability, and we see a considerable gain in performance for moderate tasks for the steps executor. The previous, almost exponential, trend in the drop in the performance with the increase in repetition almost vanishes, indicating a much more stable and reliable agent performance.  The benefit of the steps based task structure is also extended to pure function calling agents, where their reliability also receives a considerable boost across the three task categories, and especially for simple tasks.

Conclusions

In summary, the tradeoffs between agency, control, and reliability are central to deploying effective customer support agents. Our analysis shows that while high-agency agents are capable of handling complex tasks, their performance suffers in terms of consistency—a critical factor for customer satisfaction and adoption. By quantifying reliability through a stricter metric (pass^k) and simulating realistic customer interactions inspired from real customer use-cases, we expose the limitations of current LLM-based systems in open-ended environments.

Our solution lies in strategically restricting agency through controlled, modular task configurations. The “Give Fin a Task” agent is a prime example of this approach: by emphasizing step-based instructions and allowing complex tasks to be composed via simpler agentic blocks in workflows, we can achieve higher reliability and ultimately a better customer experience. Although inherent limitations in current LLM technology persist, balancing agency with enhanced control offers a promising pathway to improve performance in real-world customer support scenarios.

The post The Agency, Control, Reliability (ACR) Tradeoff for Agents appeared first on /research.

In particular, they are vulnerable to a specific class of problem, where they assume that the content provided by the RAG system is complete, and can make ungrounded assumptions as a result.

Here, we describe our work combining answer generation with a hallucination checker, to reduce their risk of occurrence, without reducing efficiency too much.

Our approach uses multiple Actor-Critic iterations where the critic points to hallucinations, and the actor uses previous failures as negative examples. This allows us to run Fin more safely without unreasonable latency cost.

Hallucinations

By now, everyone reading this blog will be familiar with the hallucinations that LLMs can be prone to. Since the earliest days of Fin, we have worked hard to reduce the risk of hallucinations, so as not to provide wrong information to users. We have made a lot of progress around the structure of our prompts and how we word specific instructions. Additionally, LLMs have become increasingly better in following these instructions. Still, there are rare cases where a hallucination could slip through the net, and so we need to reduce risk further beyond a single LLM call.

Actor-Critic using a hallucination checker

As part of our work on reducing hallucinations, we developed an offline hallucination checker. A simple prompt (example in Appendix) can be used to detect hallucinations in an answer when provided with the question and the content used. This approach has proven to be quite effective at detecting ‘obvious’ hallucinations. By obvious, we mean hallucinations that are pure fabrications or completely tangential answers. For example, in the case of a password reset process, the LLM could mention receiving a text – even though it is clearly stipulated in the documentation that the user will receive an email. 

The Actor-Critic approach, a pattern inspired from reinforcement learning, is a natural way of combining the answer generation and the hallucination checker. It is a way to structure a chain of thought process that iterates several times between an actor and a critic phase. The answer generation will play the role of the ‘actor’ while the hallucination checker, the ‘critic’, will look for hallucinations and point them out to the actor (if any). The following diagram summarizes this:

To close the loop, the critic outputs the problematic sentences in the candidate answer and it’s reasoning why. These are injected back as instructions into the generation prompt (see. Appendix). Effectively, the critic here plays the role of a bespoke negative example generator.

The system will then iterate several times between the actor and the critic, adding new negative examples, until the critic approves or ‘clears’ the answer. In case the model fails to clear the check, we escalate the question to a customer representative. This is a simple to implement yet powerful way to ensure the quality of the answer, that could also apply other types of guidance. However, we believe that removing hallucinations is the most important application so far.

Offline Experiments

To assess the effectiveness of this approach, we need a dataset. We used a mixture of direct feedback from customers and farming using the checker to gather a dataset of 148 examples of hallucinations in case of informational queries (i.e. queries that can be resolved entirely with appropriate static information in a Knowledge base). We backtested them using the new Actor-Critic approach, with a maximum of 3 iterations before escalation. 

The results are very promising: 75% of the time we managed to remove the hallucination. 19% of the cases still had to be escalated – but these were mostly due to lack of informational content available for the query, so escalation was inevitable.

There were still 6% of cases where we observed that the Actor managed to ‘evade’ the critic by reformulating the sentences instead of addressing the error, but that is a significant reduction to an already rare event.

Regarding iterations, 81% of the cases cleared the hallucination checker after 1 iteration and 7.3% after 2. We also noticed that all cases with more than 2 iterations have led to an escalation. Hence this could be the sweet spot in terms of the right number of iterations for this approach.

Examples

We show 3 illustrative examples showing how this new system works. Note the output of the following examples was modified/paraphrased to mask any sensitive information.

Clear Fabrications

Most of the cases in our dataset are gross fabrications that can be easily spotted and removed. The actor critic system allows us to surgically replace them and generate answers which are more aligned with the knowledge base.

From negative to absence of positive

The hallucination checker also does not let the actor make assumptions like “Unfortunately, we do not support integration with Product X.” unless it is clearly stated. The actor usually regenerates the safe answer “I do not have information whether we support Product X”. This type of assumption can be dangerous as it can be potentially misleading, and is a common type of error LLMs make in a RAG setting. Although the root issue here should ultimately be improved with better content in the Knowledge base, you’re never going to anticipate all customer questions and so a system that admits not knowing is a much safer option.

Limiting Tangential Responses

In order to provide a helpful answer, LLMs can go off script and make common sense recommendations. Although this behaviour can be useful in some chatbot applications, this is undesirable for customers in industries like Fintech, and can generate problematic answers. 

Next Steps

We constantly iterate on Fin to enable more innovative architectures and approaches. We believe that the application of the Actor-Critic pattern can be extended to other types of quality checks as well as helping to enforce guidances that customers provide to Fin.

Appendix

Hallucination Checker Prompt

Unset

Consider the following sources used to generate the answer:

{sources_text}

And the following conversation history:

<conversation_history>

{conversation_history}

</conversation_history>

And this customer query:

<query>

{query}

</query>

Given the following generated answer:

<answer>

{answer_text}

</answer>

Go through the answer line by line and evaluate each sentence against the provided sources, looking for contradictions, misinterpretations, conflation,  inaccuracies, fabrications, made up suggestions, hallucinations and false inference.

Pay particular attention to any case where the generated answer is trying to bend the meaning of the sources in order to answer the question.

It is ok if the answer doesn't directly address the query, the most important thing is that for every answer sentence: any facts, suggestions, steps or solutions proposed in the answer are fully grounded in the information provided in the resources.

Severity level definition: 

* "NONE" means no hallucinations

* "LOW" means super minor extrapolations or extremely reasonable assumptions

* "HIGH" means any statements not supported by the sources

Think hard about how severe the problems are (my job depends on it!) and answer with a VALID JSON output:

(When quoting or listing sentences, remove all single or double quotes to prevent JSON parsing errors)

{{

    "general_thoughts":  str, # Your general thoughts about how grounded the answer is in the sources

    "problematic_sentences": [[str]], # List of lists, each sublist being 3 strs [sentence, reason, severity_level]: Quote any sentences you think are problematic and state why alongside a severity level of "HIGH", "LOW", or "NONE", empty list if none.

    "severity_level": str # Overall hallucination severity, must be one of "HIGH", "LOW", or "NONE" (if one problematic sentence is "HIGH", this should be "HIGH")

}}

Injection of hallucination checker failures as negative examples into our generation prompt

Unset

[...]

Make sure not to repeat the mistakes committed by <previous_answer> to avoid <failed_checks>:

    <failed_checks>

        List of quality checks that the previously generated answer (in <previous_answer>) failed and the reason why.

        Make sure that the generated answer follows the instructions from the failed checks to avoid failing them again.

    </failed_checks>

    <previous_answer>

        This is a previous answer provided with this prompt that has failed in one or several quality checks.

        Make sure to write the <answer> so that remarks in <failed_checks> are respected.

    </previous_answer>

[...]

<previous_answer>{previous_answer}</previous_answer>

<failed_checks>

<ungrounded_sentence>

    The following sentence is ungrounded:

    <sentence>

        {sentence1}

    </sentence>

    Here is the reasoning:

    <reason>

        {reason1}

    </reason>

    Do not repeat this mistake. This is important to avoid quality checks to fail again.

</ungrounded_sentence>

<ungrounded_sentence>

    The following sentence is ungrounded:

    <sentence>

        {sentence2}

    </sentence>

    Here is the reasoning:

    <reason>

        {reason2}

    </reason>

    Do not repeat this mistake. This is important to avoid quality checks to fail again.

</ungrounded_sentence>

</failed_checks>

[...]

The post An Actor-Critic Approach to Reduce Hallucinations appeared first on /research.

“Slow is smooth, smooth is fast.” -Navy SEALs saying

Nobody likes slow software. Despite Moore’s law and improving hardware, software doesn’t seem to be getting faster over time. Sometimes it feels like things are even slowing down due to increasing bloat. Software latency isn’t just a user annoyance, there’s also documented evidence of real business cost: 

• Google found a 0.5 second delay caused a 20% decrease in repeat traffic, that persisted after the delay went away¹
• Amazon found every 100ms slower the site loaded, they lost 1% in revenue²
• Walmart.com found a very large change in conversion rate based on page load times³

Does the same apply to AI agents for customer support? Is latency just annoying, or also harmful?

As LLMs inherently do so much computation – which takes time – and as there are so many different LLMs available, with different latency and quality tradeoffs, we needed to know.

The only way to answer that important question is through an AB test, as the natural latency variation is confounded by many factors (e.g. peak time has higher latency then early morning, but the user types and queries in those periods are different too).

At Intercom’s AI group, we run hundreds of AB tests per year. Almost all of them are to evaluate improvements to our AI Agent Fin, but we made an exception here: To understand the impact of latency to Fin, we ran an experiment where we increased latency in a small amount of conversations.

We faced two difficulties:

1. We cannot artificially decrease latency, only increase it: if there were any latency reduction low-hanging fruits, they’d have been picked already!
2. We cannot harm the end user or customer experience: the experiment must be imperceptible to them

We managed to implement an experiment design where the changes were indeed imperceptible. The experiment also uncovered a non-obvious truth:

• As we expected, latency increases deflections
• Unexpectedly, latency also increases positive feedback without harming negative feedback or CSAT

Those results were so surprising that we had to run confirmatory tests. Here is a summary of the 2 more relevant experiments:

Despite those results, since then Fin has become faster, not slower: Despite our findings, we still want to speed Fin up! Even if the latency doesn’t seem to negatively effect the business outcomes, we want to make the experience fast. Nonetheless, these results are valuable – not least because they have changed the way we run AB tests at Intercom, to account for latency as an important confounder.

Before exploring the results and take aways, let’s dive into the methodology first.

Experiments and methodology

Due to the nature of the experiment and to make it blind, we had to design it in a way that it wouldn’t be noticeable to both users and customers:

We first measured the current latency percentiles and made increases that would be consistent with the already-existing latency. For example, at the time of the experiment, the P99 latency used to be 20 seconds over the median, so the variant that increased latency by 20 seconds was just 1% of the conversational volume.

In other words, as a user, you wouldn’t notice an increase in latency, unless you were unlucky enough to be in the natural 99 percentile and fall in the 20 second delay experiment arm, a tiny 1 in 10k chance!

During the time of the experiment, we didn’t receive any latency complaints that could be attributed to the experiment, indicating that the methodology was a success in terms of hiding the experiment effects from users and customers.

We randomized the experiment at the conversation-level⁴ with 1 control and 4 treatment arms roughly mimicking the natural latency variation:

• Control / no latency increase: 90% of conversations
• 5s delay: 4% of conversations
• 10s delay: 3% of conversations
• 15s delay: 2% of conversations
• 20s delay: 1% of conversations

As the results were so surprising, we ran two more confirmatory experiments:

1. We targeted our previous generation Resolution Bot, which is much faster than Fin as it doesn’t use AI/LLMs, thus covering the shortcoming of not being able to decrease latency
2. One unrelated experiment was naturally decreasing latency by 2 seconds, so we added one more arm there to compensate for that latency decrease

All 3 experiments found similar results.

Metrics

When running AB tests, we monitor it through a dashboard that covers dozens of metrics. Here are the most important ones:

• Resolution rate: our key business metric (we charge customers by resolutions)
• Volume: to ensure the experiment has enough statistical power
• Sample ratio mismatch: to ensure assignments and randomization is working as expected
• Confirmed resolution rate: resolutions with positive feedback
• Positive / negative feedback rates: users reacting to “did that help?” with “yes” or “no” respectively
• CSAT score: customer satisfaction survey sent after the conversation is closed
• Latency percentiles: both time to first token and total duration
• Cost per conversation in $
• Error rate
• Interaction analysis between other concurrent experiments

Those metrics help understand the impact of each experiment under many different dimensions. They allow us to judge whether the experiment was well designed and implemented, has enough sample size, and its impacts on business, users, and engineering metrics.

We will expand on our experimental methodology in future posts.

Results

“Any figure that looks interesting or different is usually wrong” -Twyman’s law

Before we get to the actual results, here is what we expected to happen from first-principles:

• Deflections and assumed resolutions will go up due to more users bouncing off
• Confirmed resolutions, positive feedback and CSAT score will go down due more users getting annoyed and fewer getting to the feedback stage

Here is what we actually saw:

Overall resolution rate

The overall resolution rate increases by up to 2 percentage points (pp):

To understand what is going on, we need to break resolution down into its two components:

• Assumed resolutions: When users engage with the AI agent, receive an AI answer with sources, don’t leave positive or negative feedback, and don’t escalate to a human support agent
• Confirmed resolutions: Same as above, plus the user needs to leave positive feedback to the last AI answer (e.g. saying “that helped 👍”)

Assumed resolutions

As expected, assumed resolution is the main driver of the increase: we expect users to drop off more, leading to more such resolutions when AI answers are delivered⁵.

Confirmed resolutions / positive feedback

Surprisingly, confirmed resolutions, which are resolutions with positive feedback, seem to actually increase by 0.4pp for higher latency increases. This was the most puzzling result – a Confirmed Resolution is the result of an end user explicitly clicking a button saying “that helped 👍”, and we use this as our north star. As such, this should only go up if the user was in fact more satisfied with the AI answers. It was this puzzling result that led to further analysis and experiments.

Negative feedback

Also surprisingly, negative feedback is not increased and seems reduced for higher latencies. Note that negative feedback does not mean anger or dissatisfaction, only replying “No” after “did that help?”. 

CSAT

Finally, we don’t see a clear impact to CSAT rating, but we do see an increase in response rate in higher latencies, another puzzling result:

Take aways

We saw expected results for deflections and assumed resolutions but not for the feedback associated metrics. Are users actually more happy with increased latency? 

We brainstormed potential explanations:

1. The metrics are wrong i.e. the results are just due to some data artefact, in the spirit of Twyman’s law
2. There is a poorly understood psychological effect at play, not well documented by the literature

We ruled the first hypothesis by double checking the data, running confirmatory experiments, and conferring with our excellent AI infrastructure engineering team, owners of the AB test assignment logic.

Psychological explanation

“When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth”. -Sherlock Holmes

Our best hypothesis is a psychological explanation: The longer users wait, the more human or effortful the reply becomes in their minds, leading to improved answer perception. We have some corroborating evidence for that hypothesis:

• An experiment which provided detailed status on Fin “thinking” slightly decreased hard resolutions, showing user perception of how Fin works impacts their feedback. In this case, by making Fin more transparent, we slightly hurt the positive feedback rate.
• We have received complaints that Fin over email is “too fast”, as it seems suspicious to get an almost instant response for email queries.
• An independent AI engineer formed similar conclusions here: Improved chatbot customer experience: sleep() is all you need.
• An independent designer also shares similar conclusions: Why Users Love ‘Thinking’ Chatbots: The Use of Delays in Conversational AI:
  The Labor Illusion, as discussed in a previous post, is a term coined by Harvard researchers. This behavioural economic concept posits that users tend to appreciate a service more when they perceive that effort is being expended on their behalf. In the context of chatbots, delays in response times simulate the effort of a human agent ‘thinking’ through the customer’s query, creating an illusion of diligence and attentiveness.

If true, a question we naturally might ask is, how else can we make an AI agent be more humanlike whilst remaining within legal⁶ and moral limits? That has led to other interesting experiment ideas:

• Changing “thinking…” into a typing indicator
• Removing unnecessary references to “AI agent” in the UI
• Adding an acknowledgement message (“let me look into Fin pricing for you 🔎”) while Fin searches for answer

While some of those experiments are still ongoing (more on them in future posts), all have promising results, so we are even more confident that the psychological hypothesis is the correct one.

Latency as a confounder

Another takeaway is that latency is a confounder for any other experiment that accidentally increases or decreases latency.

In other words, if an experiment testing something else happens to increase latency by 10 seconds, it might show positive benefits solely due to latency. Conversely, an experiment that reduces latency might decrease resolution rate solely due to latency reduction.

That means latency needs to be corrected for. For experiments with increased latency, we should remove the resolution rate gains as shown above. For experiments with decreased latency, we need to add another variant with latency corrections, resulting in an ABC test:

• A: control arm
• B: treatment arm
• C: treatment arm with same latency as control arm

This effect also explains some historical results. For example, the biggest ever Fin latency reduction required multiple iterations not to harm Fin metrics. At first, we thought it was due to the quality of the original prompt, but now we realise some of it was just due to latency. Meaning, we underestimated how good the change was as the only reason the resolution rate didn’t go up was due to the latency reduction.

Fin latency today

Despite those results, since we ran that experiment in late 2024, we have made good progress in reducing Fin’s latency, from changing the way we serve Claude Sonnet models to a new more streamlined architecture, where we rely on eager execution to save precious seconds.

As of March 14, 2025, Fin is at its fastest ever: The median time to first token is now 7 seconds, a 60% reduction from its peak.

Why are we making Fin faster, given the experiment findings? While we’re confident in the findings, we know latency effects go beyond just end user psychology. We must also serve our customers, who are actually paying for Fin’s resolutions. While they might be glad of a marginal resolution rate increase, they’d much prefer for their users to have the best possible experience.

Fin is now the first-line of support for many of our customers, which range from Anthropic to Clay to Culture Amp, and is responsible for over half of all their resolutions on average, meaning Fin is an integral part of their customer support operations.

From that prism, it’s unacceptable to have a subpar experience even if it doesn’t harm end user metrics. We go back to the original opener of this post: nobody likes slow software. We want Fin to be fast and to be perceived as such, as we want Fin to be the best AI agent in all dimensions.

You should expect even more latency improvements to Fin in 2025.

1. Marissa Mayer at Web 2.0 ↩︎
2. Amazon study: Every 100ms in Added Page Load Time Cost 1% in Revenue ↩︎
3. Walmart slides ↩︎
4. We typically randomize AB tests at the conversation instead of the more typical user-level. There are pros and cons to each alternative, but here the decision was clear: We didn’t want to harm the end user experience, so randomizing by conversation gives the end user opportunity to get support at natural latencies most of the times. ↩︎
5. AI answer delivered does not mean it was read or seen by the end user! ↩︎
6. California’s SB-1001, the Bolstering Online Transparency Act, prohibits bots from communicating online to mislead about their artificial identity for commercial transactions or influencing votes, requiring clear disclosure to California consumers, with enforcement by the state attorney general and fines up to $2,500 per violation. ↩︎

The post Slower Feels Smarter? Experimenting with AI Agent Latency appeared first on /research.