How a Small Model Learned to Do a Large Model's Job at Docusign's Scale

Docusign AI team members who contributed to this blog include Minh Phan, senior applied scientist, and Ramachandra Kota, senior director of applied science.

This is the second post in a 4-part series on how Docusign processes agreements at scale. Read part one on our runtime architecture.

Running a frontier model on every document looks good in a demo. However, that is not a prudent or viable strategy at Docusign’s 1M+ documents/day scale. The per-call cost and latency that are negligible at prototype scale become the dominant line item in the infrastructure budget the moment it is at enterprise scale in production, and accuracy requirements don't get a discount just because the economics got harder.

This is the question we had to answer: could we get a fast and inexpensive model to do the work of a large, expensive one without having to compromise on accuracy and quality or invest in significant resources for building and labeling datasets?

The answer was a teacher-student distillation: use a frontier model to generate high-quality labels at scale (the teacher), then train a smaller model on those pseudo-labels instead of on scarce ground truth (the student). The rest of this post is about what it actually took to make that work.

Training data without compromising privacy

Docusign strictly separates data used for real-time inference from data used to train AI models. We never train on raw, identifiable customer data: Docusign draws on a large repository of public and privately consented data, and before any of it is used for training, it passes through our Compliance Fabric, where it's anonymized and scrubbed of sensitive information.

Building datasets

For model training and evaluation, a smart sampling pipeline has been developed to ensure that the selected documents are diverse, representative, and meaningful. The sampling employs three key strategies: diversity signals, user-feedback signals, and error-pattern signals.

First, the sampled dataset is designed to cover all agreement and extraction types, ensuring that no category is missing or underrepresented while also ensuring diversity across documents and sources. Second, user feedback is incorporated into the sampling process such that documents whose AI extractions need corrections are more likely to be selected. Last, the pipeline deliberately oversamples known failure case modes, so that the sampled training data targets the gaps that actually exist rather than the ones that happen to be easy to find.

Defining the extraction before the model can learn it

Distillation only works if the teacher model has a clear, consistent target to aim at. That sounds obvious until the task is defining something like "effective date" precisely enough for a model — or a person — to apply it the same way across thousands of structurally different documents.

The dictionary answer for “effective date” is straightforward: the date an agreement becomes legally binding and enforceable. But real documents complicate that immediately: is a lease's commencement date the effective date, or is it the signing date? What's the effective date for a document that never uses that phrase and instead refers to a dependency based on an "execution date" or a " signed date"? Across 50+ extraction types, every one of them has some version of such ambiguity.

We turned to our own legal subject matter experts (SMEs) for assistance. Docusign’s legal SMEs wrote the guidelines that resolved these cases, and those guidelines became the backbone of both the teacher prompts and the evaluation dataset. The evaluation set went through independent triple annotation, then a panel review to settle disagreements and lock in consensus. We also cross-checked human annotations against LLM predictions, which caught inconsistencies and missed annotations that slip past manual review. As we keep expanding and improving the AI extractions, we regularly revisit and revise the expert guidelines with our legal SMEs.

The teacher-student approach

Standard fine-tuning assumes a large set of labeled ground truth to train against. Building one by hand at that scale across 50+ extraction types wasn't the approach we wanted to reach for first. Instead, we use a frontier reasoning model to generate the labels at scale, then train a smaller model on those pseudo-labels.

The process runs in three steps:

1. Prompt engineering for the teacher. For each of the 50+ extraction types we handle, we built an individual tightly optimized prompt for a frontier reasoning model (the teacher), chosen for its reasoning ability and instruction-following on complex legal text. These prompts aren't simple instructions. They encode the extraction guidelines from the legal SME work above, reasoning frameworks for ambiguous cases, and curated few-shot examples addressing common error patterns.

2. Generating training data. The teacher runs these expert prompts across a corpus of unlabeled, anonymized documents. Its predictions become pseudo-labels; not ground truth in the traditional sense, but a high-quality approximation of it.

3. Fine-tuning the student. These pseudo-labels train an efficient model (the student) using unified prompts that extract multiple entity types in a single call, replacing the teacher’s per-type specialized prompts. As a result, the student accomplishes in a fraction of the calls what required dozens of specialized passes from the teacher.

A significant portion of the effort lies in that first step. The teacher's prompt quality sets the ceiling for everything downstream. Any gap, ambiguity, or bad assumption baked into it propagates into the pseudo-labels and then into the student. Getting the teacher instructions right took multiple rounds of analysis, with SMEs and engineers reviewing teacher outputs, refining guidelines, and re-testing edge cases. We treat it as core modeling work, not setup.

The student's simpler, unified prompt is a deliberate tradeoff in the other direction. It doesn't carry the same depth of task-specific instruction the teacher needed. That's by design, as fewer calls and fewer tokens per call means lower latency and lower cost, and the student doesn't need that depth because it learned patterns from the teacher's labels rather than reasoning through detailed guidelines from scratch.

The models in production today have been built through this process. We've kept iterating on both teacher and student since launch; the current production models are the seventh generation, and our focus now is reinforcement fine-tuning with newer reasoning models.

What was the bottleneck: evaluation infrastructure

None of the above works without a way to actually compare model versions, prompt changes, and fine-tuning runs against each other quickly. Early on, that meant MLFlow for experiment tracking and Google Sheets for everything else, such as error analysis, SME reviews , and dataset versioning.

The shift to LLMs changed where the effort goes: we spend less time on model implementation and far more on prompt tuning and training iterations. That makes speed of iteration the real constraint. We needed to try a change, evaluate it quickly, and keep or discard it on the evidence, but MLFlow plus spreadsheets couldn't keep that loop tight. Seeing how a single prompt edit moved performance meant stitching results together by hand.

We moved to a single unified platform for logging, error analysis, dataset management, and SME annotation. Docusign engineers and legal SMEs can now compare F1, precision, and recall side by side, and its UI surfaces how performance moves with every change. Its feedback on regression cases does double duty, pointing us at what to fix and acting as a quality gate against silent regressions. That shortened the loop between "we think this prompt change helped" and actually knowing.

Results

There are two comparisons worth separating because they answer different questions.

Against the teacher it learned from, the fine-tuned student lags by only about 1 percentage point of F1 score – the expected, and marginal loss from distillation – while running at roughly a hundredth of the teacher's cost and a quarter of its latency.

Against the model it actually replaced in production (a non-fine-tuned frontier model), it wasn't a tradeoff at all. The new student model runs at roughly a twentieth of the cost, at about eight times the throughput, with a 7-percentage-point accuracy gain over the previous larger model, while cutting inference time about in half. It also let us retire the custom post-processing step that used to patch the non-fine-tuned model's predictions, removing an entire component from the pipeline.

The models in production today have been built through this process. We've kept iterating on both teacher and student since launch; the current production models are the seventh generation, and our focus now is reinforcement fine-tuning with newer reasoning models.

Fine-tuning through our platform partners

Our distillation efforts benefited from our partnerships with hyperscalers to leverage frontier models and fine-tuning capabilities. We leveraged Microsoft Azure Foundry for fine-tuning OpenAI models – which we co-presented with Microsoft at Ignite last year opens in a new tab – and are working on Gemini fine-tuning with the team at Google. 

Up next

The models in this post extract a fixed set of fields that Docusign defined as broadly useful. But the extraction types that matter most to an enterprise are often specific to its own business. 

Stay tuned for the third post in this series, which will cover how we extended the platform to let enterprises define their own custom extractions, applied at the same scale and accuracy as the standard ones.