NVIDIA Builds Medical AI Apps Using Autonomous Agents

NVIDIA developers recently demonstrated how an AI coding agent can build a real-time endoscopic tool segmentation application using the NVIDIA Holoscan platform and its HoloHub repository. Guided by human-defined goals, the agent—running Codex with GPT-5.6 sol max mode—iteratively constructed and optimized the software. The project reused the MONAI endoscopic tool segmentation model, known as monai_endoscopic_tool_seg, to process video frames.

The development progressed through three phases. First, the agent generated a baseline application in 40 minutes, consuming 11 million tokens. Next, the agent spent 20 minutes implementing a benchmarking suite with three execution modes: visual, smoke (a 60-frame headless run), and benchmark (a 300-frame offscreen run). In the final phase, the agent spent 30 minutes optimizing latency by reusing HoloViz input specifications and asynchronously queuing GPU-to-host telemetry in two pinned buffers. These optimizations increased rendered throughput by 50.5 percent, jumping from 204.0 to 306.9 frames per second. Meanwhile, mean application-path latency fell 33.6 percent from 4.891 milliseconds to 3.247 milliseconds, and ninety-fifth percentile latency dropped 27.4 percent from 6.273 milliseconds to 4.554 milliseconds.

An ablation study on August 1 using Codex version 0.146.0 with GPT-5.6 Sol at maximum reasoning effort highlighted the value of specialized tools. When provided with the Holoscan command-line interface, development skills, and documentation, the agent completed the task in 40 minutes using 11 million tokens. Without the skills, the agent took 65 minutes and consumed 20 million tokens, relying on generic tools. When denied both CLI guidance and skills, the agent finished in 40 minutes using 15 million tokens, but the resulting application was 2.6 times slower because it ignored optimized Holoscan operators like TensorRT inference.

For edge AI practitioners, this workflow establishes a collaborative loop where developers offload repetitive scaffolding, testing, and optimization to autonomous agents. By providing agents with structured command-line tools and domain-specific skills, engineers can reduce token costs and development times while ensuring the generated code remains highly optimized for specialized hardware.