Meet the Engineer: Shreeya Dhakal

Summary•5 min read

Shreeya Dhakal spends her days evaluating Docusign's AI agents, and her nights building NLP tools for Nepali, her native language and a low-resource one she's worked to bring into the computational world since college. In this installment of Meet the Engineer, the Applied Scientist talks about debugging an AI agent that gave different answers to the same question, and what "good enough to ship" actually means in a multi-tool agentic system. She also explains how building tools for her own language taught her to notice when a system is quietly failing, long before the aggregate metrics would show it.

Meet the Engineer is a recurring series on the Docusign Engineering blog where we sit down with the people building our platform. The clearest way to understand what it's actually like to work here is hearing our engineers talk about the problems they're solving and how they think.

What are you working on right now, and what's the hardest part of it?

I’m currently working on the evaluation of Docusign chat agents. These agents are already very good at handling different agreement-related tasks, but there are still critical nuances they can fail to capture. My job is to help understand what our chat agent is good at and what needs to be improved. I also help define what “good” even means in a multi-tool agentic system that deals with long, binding agreements. Evaluation also means understanding the paths the agent takes to answer a question or serve a customer request, and finding ways to improve latency, cost, and reliability along the way.

Outside of work, what's something you're into that ends up shaping how you think about your job?

Outside of work, I build and evaluate NLP tools for the Nepali language. I’ve been doing it since college. Nepali is my native language so it’s important for me to continue working on improving the computational future of the language. It’s a low-resource language with very few people working on it, which is why I dedicate my free time to building resources for it and writing about it at icodeformybhasa.com opens in a new tab.

“Internationalization is often an afterthought in most software products … Because of my work with Nepali, I've learned to check the outputs myself, be suspicious of aggregate metrics, and notice if a system is quietly failing for a subset of users.”

My work with Nepali has shown me that internationalization is often an afterthought in most software products. While I'm building and evaluating AI pipelines here at Docusign, I pay close attention to whether our models capture the nuances in different languages and markets correctly. Because of my work with Nepali, I've learned to check the outputs myself, be suspicious of aggregate metrics, and notice if a system is quietly failing for a subset of users.

Shreeya's first hardware electronics project, a small device that can help folks learn Nepali words.

What keeps you busy and motivated when you’re not at work?

When I’m not working, I spend most of my time outdoors. Since my work requires me to be indoors, I look forward to being outdoors on the weekends. In summer, I love hiking to alpine lakes, waterfalls, and mountains, and in winter, I'm out on the slopes skiing. I live in the Pacific Northwest; the Cascades remind me of the Himalayas so there’s a certain joy I find in these mountains. The weekends really charge me up for the week of work after.

At Marten Lake in the Alpine Lakes Wilderness east of Seattle, WA.

How would you describe the engineering culture at Docusign to someone who hasn't worked here?

We have a very collaborative and open engineering culture at Docusign, particularly in the AI team. It’s very easy to reach out to other scientists and engineers for reviews, suggestions, and collaboration. We keep an open roster of every research project the team is working on, and we meet regularly to review proposals together and share input.

Walk us through a technical decision on this project you'd make differently if you started over.

Our initial focus was on a single agent, so scaling the evaluation became challenging once we moved to multi-agent scenarios. The primary challenge was document sourcing and data annotation for offline evaluation. If we were to start over, I would definitely invest equal time and effort across all the agents we’re building.

What's the most interesting problem you've solved that customers will never see or know about?

Failures in agentic systems can be very interesting and hard at the same time. Early in the project, I'd occasionally run into issues while evaluating our chat agent that I couldn't reproduce in a subsequent run. The input would be the same, but the tool call path would change, and that would change the output. Our team iteratively updated the agent and tool definition such that there is a more reliable tool-call path that the agent now follows, resulting in low variability and high reproducibility in results.

“We have a very strict definition of what quality means … We iteratively evaluate and build the models until the acceptance criteria in terms of quality and speed are met. We then work with a subset of our customers to understand how useful the solution is and whether they are satisfied with the quality and performance before we GA.”

How do you decide when something's good enough to ship versus needs more work?

We have a very strict definition of what quality means. For each project, we work with our stakeholders to define the scope of a feature release, then work with our legal and data experts to define detailed guidelines for how we annotate and evaluate the models. We iteratively evaluate and build the models until the acceptance criteria in terms of quality and speed are met. We then work with a subset of our customers to understand how useful the solution is and whether they’re satisfied with the quality and performance before we GA. What ships vs. what needs more work depends on offline evaluation metrics and early customer satisfaction scores.

What made you choose engineering, or what keeps you in it?

I love language and building language technologies. I’m fascinated by how words are formed, how they change meaning depending on where they are in the sentence. I’m also deeply interested in ML models and how we can effectively generate natural language text today. As I've worked with different languages over the years, the problems I've encountered have each been unique, and finding solutions to them has always been fun.

Interested in working at Docusign? Our engineering team is hiring across machine learning, site reliability, search, and cloud. See open Engineering roles →