introduction
Ultrasound is one of the most widely used medical imaging modalities due to its safety, real-time capabilities, portability, and low cost. For decades, ultrasound images have been formed using hand-designed reconstruction pipelines. This pipeline compresses a wealth of raw sensor measurements into a final image while simplifying assumptions about physics, such as the constant speed of sound across the human body.
In the age of AI and foundational models, natural questions arise. Can we go beyond traditional beamforming pipelines to learn directly from raw ultrasound sensor data and take advantage of information typically discarded during reconstruction? If so, what new capabilities will it unlock?
Researchers from NVIDIA and Siemens Healthineers collaborated to find answers to these questions. As a result of this work, we will release a reconstructed model called NV-Raw2Insights-US.
Raw2Insights
Essentially, ultrasound is sound, not an image. What clinicians ultimately see on the screen is a reconstructed image built from millions of tiny echoes returned by the body. But in that reconstruction process, much of the original signal, the richness of how sound actually travels through tissue, is simplified or lost.
Our approach starts much earlier. Rather than working from finished images, NV-Raw2Insights-US learns directly from the raw signals captured by the ultrasound probe. This is the best representation of how sound actually interacts with the body. This allows the model to “listen” more carefully and understand how each patient uniquely shapes sound waves. Our vision is to enable end-to-end AI for ultrasound imaging, and this is the first step toward that vision. This class of models is called Raw2Insights.
This first Raw2Insights application estimates the speed of sound for adaptive image focusing. The result is a system that can generate personalized sound velocity maps for each patient and use them to modify images in real time. What once required complex and time-consuming calculations is now performed in a single AI pass. This is a move from raw ultrasound channel data to actionable insights – AI systems that not only process ultrasound images but actively understand and adapt to the physical properties of individual patients.
introduction
Raw ultrasound channel data is typically not easily accessible on clinical-grade ultrasound scanners due to its wide bandwidth. Holoscan Sensor Bridge (HSB) is an open source FPGA IP developed by NVIDIA that enables high-bandwidth, low-latency data transfer to GPUs via (RDMA over Converged Ethernet). The Altera Agilex-7 FPGA Development Kit, when combined with the NVIDIA Holoscan Sensor Bridge, enables raw ultrasound channel data streaming from the DisplayPort output of the ACUSON Sequoia ultrasound scanner. We call this technology Data over DisplayPort. NVIDIA HSB then packetizes the data and sends it over Ethernet to NVIDIA IGX for data collection and AI inference. It shows how modern high-performance computing power can be integrated with existing scanner architectures using high-bandwidth DisplayPort output.
We will deploy NV-Raw2Insights-US using NVIDIA Holoscan, an edge AI sensor processing platform designed for high-performance real-time workloads on systems such as NVIDIA IGX Thor and NVIDIA DGX Spark.
Once the data is stored in GPU memory, NV-Raw2Insights-US performs fast inference on Blackwell class GPUs to generate patient-specific sound velocity estimates. This estimate is streamed back to the ultrasound scanner, allowing for improved focus in the live imaging stream.
System functions
This demo architecture provides flexibility across both development and deployment.
Software-only integration: Software-only changes using Data over DisplayPort enable NVIDIA acceleration of existing medical devices.
Software-defined ultrasound: This software-defined approach allows for continuous improvement through software updates.
Modular expansion: Seamlessly integrate new AI models using raw ultrasound channel data already stored in GPU memory.
prospect of the end
Create a scalable path to AI-native imaging by moving ultrasound intelligence from traditional algorithms to the AI-powered Raw2Insights pipeline. By learning directly from raw ultrasound channel data rather than reconstructed images, NV-Raw2Insights-US reduces errors introduced by traditional assumptions and effectively adapts image processing to each patient.
This architecture not only improves the clarity of today’s images, but also establishes a modular foundation for the next generation of AI-powered diagnostic systems. You can start developing based on NV-Raw2Insights-US here (GitHub / Model Weights / Dataset).
References
“Ultrasonic autofocus: Optimizing common midpoint phase error with differentiable beamforming,” IEEE Transactions on Medical Imaging, Vol. 45, Issue 2, February 2026. https://ieeexplore.ieee.org/document/11154013 “Investigation of pulse-echo sound velocity estimation in breast ultrasound using deep learning”, arXiv:2302.03064, 2023. https://arxiv.org/abs/2302.03064 NVIDIA Holoscan SDK documentation, https://developer.nvidia.com/holoscan-sdk
understand
This project was carried out in close collaboration with Siemens-Healthineers. We are grateful for the support of Siemens-Healthineers, including the direct collaboration of Ismayil Guracar and Rickard Loftman from the AI & Advanced Platforms group.
This technology is under research and development and has not been approved or sold in the United States or any other country. Future availability cannot be guaranteed.
