The robot, having watched surgical videos, behaves with the skill of a human doctor

A robot trained for the first time using videos of experienced surgeons performed the same surgical procedures as skillfully as human doctors.

The successful use of imitation learning to train surgical robots eliminates the need to program robots with every single movement required during a medical procedure and brings the field of robotic surgery closer to true autonomy, where robots could perform complex surgeries without human assistance.

The findings, led by researchers at Johns Hopkins University, will be presented this week at the Robot Learning Conference in Munich, a top event for robotics and machine learning.

“It’s really magical to have this model. All we do is feed it camera input and it can predict the robot movements required for the operation,” said lead author Axel Krieger, assistant professor in JHU’s Department of Mechanical Engineering. “We believe this represents a significant step forward toward a new frontier in medical robotics.”

The team, which included researchers from Stanford University, used imitation learning to teach the da Vinci Surgical System robot to perform three basic tasks required during surgical procedures: manipulating a needle, lifting body tissue and suturing. In each case, the robot trained on the team’s model performed the same surgical procedures as skillfully as human doctors.

The model combined imitation learning with the same machine learning architecture that underlies ChatGPT. However, while ChatGPT works with words and text, this “robot” model speaks with kinematics, a language that breaks down the angles of robot movement into mathematics.

The researchers fed their model hundreds of videos captured by wrist cameras attached to the arms of da Vinci robots during surgical procedures. These videos, recorded by surgeons around the world, are used for post-operative analysis and then archived. Nearly 7,000 da Vinci robots are in use worldwide and more than 50,000 surgeons are trained on the system, creating a large archive of data that robots can “mimic.”

Although the da Vinci system is widely used, researchers say it is notoriously inaccurate. But the team found a way to make the faulty inputs work. The key was to train the model to perform relative movements rather than absolute actions, which are inaccurate.

“All we need is an image input and then this AI system finds the right action,” said lead author Ji Woong “Brian” Kim, a postdoctoral fellow at Johns Hopkins. “We find that even with a few hundred demos, the model is able to learn the procedure and generalize to new environments it has not yet encountered.”

Krieger added, “The model is so good at learning things we didn’t teach him. For example, if it drops the needle, it will automatically pick it up and continue. I didn’t teach him that.”

The model could be used to quickly teach a robot to perform any type of surgical procedure, the researchers said. The team is now using imitation learning to teach a robot to perform not just small surgical tasks, but a complete operation.

Before this advancement, programming a robot to perform even a simple part of an operation required manually coding each step. Someone could spend a decade modeling sewing, Krieger said. And that’s suturing for just one type of operation.

“It’s very limiting,” Krieger said. “The new thing here is that we only need to collect replicas of different procedures and we can teach a robot to learn it within a few days. This will allow us to achieve the goal of autonomy more quickly while reducing medical errors and performing more precise surgical procedures.”

Johns Hopkins authors include graduate student Samuel Schmidgall; Associate Research Engineer Anton Deguet; and associate professor of mechanical engineering Marin Kobilarov. The Stanford University authors are graduate student Tony Z. Zhao and assistant professor Chelsea Finn.