The robot learns to perform surgical tasks expertly by watching videos

Performing surgery takes years of intensive research and a steady hand, but robots may find it easier to take on the task with today’s AI technology.

Researchers at Johns Hopkins University (JHU) and Stanford University have taught a robotic surgical system to perform a range of surgical tasks as competently as human doctors simply by training it on videos of these procedures.

The team used a da Vinci surgical system for this study. This is a robotic system typically controlled remotely by a surgeon, whose arms manipulate instruments for tasks such as dissection, suction, and cutting and closing vessels. Systems like these allow surgeons much better control, precision and a closer look at patients on the operating table. The cost of the latest version is estimated to be over $2 million. This does not include accessories, sterilization equipment and training.

Using a machine learning method known as imitation learning, the team trained a da Vinci surgical system to independently perform three surgical tasks: manipulating a needle, lifting body tissue, and suturing. Take a look.

Not only did the surgical system perform these as well as a human could, it also learned to correct its own mistakes. “For example, if it drops the needle, it will automatically pick it up and continue. I didn’t teach him that,” said Axel Krieger, an assistant professor at JHU who co-authored a paper on the team’s findings presented at this week’s robot learning conference.

The researchers trained an AI model by combining imitation learning with the machine learning architecture on which popular chatbots like ChatGPT are based. However, while these chatbots are designed to work with text, this model spits out kinematics – a language used to describe movements using mathematical elements such as numbers and equations – to control the surgical system’s arms.

Intuitive

The model was trained using hundreds of videos captured by wrist cameras attached to the arms of da Vinci robots during surgical procedures.

The team believes their model could teach a robot to perform any type of surgical procedure quickly and much more easily than the traditional method, which involves hand-coding every step needed to control a surgical robot’s actions.

Johns Hopkins University / Stanford University

According to Krieger, this could help automated surgery become a reality sooner than we could previously imagine. “The new thing here is that we just need to collect replicas of different procedures and we can teach a robot to learn it in a few days,” he said. “It allows us to achieve the goal of autonomy more quickly while reducing medical errors and achieving more precise surgery.”

This could be one of the biggest breakthroughs in the world of robotic surgery in recent years. There are some automated devices for use in complex surgeries, such as Corindus’ CorPath system for cardiovascular procedures. However, their abilities are typically limited to specific steps of the operations they assist with.

Additionally, Krieger pointed out that coding each step for a robotic system can be very slow. “Someone could spend a decade modeling sewing,” he said. “And that’s suturing for just one type of surgery.”

Krieger also previously worked on a different approach to automating surgical tasks. In 2022, his research team at JHU developed the Smart Tissue Autonomous Robot, or STAR for short. Guided by a three-dimensional endoscope based on structural light and a machine learning-based tracking algorithm, the robot intricately sewed together two ends of a pig intestine without human intervention.

The JHU researchers are now working on training a robot to perform a complete operation using their imitation learning method. It will likely be years before robots completely take over the work of surgeons, but innovations like these could make complex treatments safer and more accessible for patients around the world.

Source: Johns Hopkins University