All workNow · NYU Langone Radiology

RadAssistant

An agentic assistant for radiology researchers, built inside 3D Slicer, where every number it reports has to come from a tool.

NYU Langone Radiology2026, in useLangGraphTool callingHITLGitHub
Requestplain EnglishPlannerLangGraph50+ imaging toolsDICOM, radiomics, moreVerifierdeterministicAnswerevery number sourcedProvenance ledgertool output per figureHuman approvalconsequential stepsModel APIinside NYU LangonePatient data stays inside the hospital network.

The problem

Radiology research runs on long chains of imaging steps: segmentation, radiomics, registration, DICOM handling. Each step is a menu in 3D Slicer, and a study repeats them across hundreds of scans.

A language model can plan those chains from a plain-English request. The hard part is trust. In a clinical setting, a model that confidently reports a number no tool produced is worse than no model at all.

How it works

  • LangGraph plan-and-execute agents call 50 validated imaging tools through native tool calling. This is the third version of the system: a native loop first, then hand-rolled orchestration, then LangGraph.
  • A deterministic verification layer checks every tool output. It replaced asking the model to check its own work.
  • A provenance ledger records where every reported measurement came from. It caught the model inventing a measurement in a live session, before a researcher read it.
  • Consequential actions wait for a person to approve them. Outgoing text is checked for patient identifiers, and a diagnosis guard keeps clinical conclusions out of answers.
  • The model layer is swappable. The build the lab runs calls NYU Langone's internal model API, so requests never leave the network.

Who uses it

The lab's radiologist, day to day. Their routine tasks became reusable skills the agent can run, which is how the tool grew from a demo into something the lab depends on.

What I'd point to

The ledger. Moving verification out of the model and into deterministic code is the design decision everything else depends on, and it is the part I would build again first.

Status

In use at the lab and still changing. The code is public on GitHub. The current build calls NYU Langone's internal model API, so no patient data leaves the network.

Next