Advances in AI Surgery with Dr. Hashimoto

In an excellent review Dr. Hashimoto wrote several years ago, he defines artificial intelligence, or AI, as the study of algorithms that give machines the ability to reason and perform cognitive functions. Dr. Hashimoto also notes that AI is a parent field that encompasses subfields, including machine learning, and its subdomain, computer vision, both of which have applications in surgery. In this episode, he discusses each of these in turn and provides insight on how machine learning can be applied to surgery.

Transcript

Podcast transcript (PDF)

Melanie Cole (Host): Welcome to the podcast series from the specialists at Penn Medicine. I'm Melanie Cole. And joining me today is Dr. Daniel Hashimoto. He's the Director at the Penn Computer Assisted Surgery and Outcomes Laboratory, and he's an Assistant Professor of Surgery at the Hospital of the University of Pennsylvania. And he's here today to highlight AI advances in surgery and offer insight on how machine learning can be applied to surgery.

Dr. Hashimoto, it's a pleasure to have you join us today. This is a really interesting topic. In an excellent review that you wrote several years ago, you define artificial intelligence or AI as the study of algorithms that give machines the ability to reason and perform cognitive functions. You also note that AI is a parent field that encompasses subfields, including machine learning and its subdomain computer vision, both of which have applications in surgery. I'd like you to start by telling us what exactly is machine learning and how is it being applied to surgery?

Dr. Daniel Hashimoto: Well, thanks very much, Melanie. It's a privilege to be here with you today on this podcast. And you're right, you know, AI is this parent field of study that includes machine learning, which I consider to be a tool within the overall field of artificial intelligence. And machine learning, I think there's lots of different definitions for it, but a straightforward one that I think about is machine learning as the study of the algorithms and the processes that learn from data or that adapt to data. And so, it's a little bit more specific than thinking about artificial intelligence as a broader category of lots of different types of approaches and applications that can exist.

And machine learning as a tool can be applied to all sorts of different data, particularly in surgery. When you think about some of the numerical data that we might be getting from our laboratory values or from our vital signs, it can be applied to the text data that you can get from the electronic medical record. Or in the case of my lab's research, it can be applied to the images and the videos that are obtained during the delivery of procedural care. So, it's a very versatile tool that can be used to study a lot of different things.

Melanie Cole (Host): This whole topic is so fascinating. But before we get into really some of the benefits like risk prediction and computer vision, can you tell us a little bit about public perception or stigma, things that the public might not trust when we hear machine learning and artificial intelligence? You know, there's been a whole lot about that. Speak about how you are working on overcoming that public perception.

Dr. Daniel Hashimoto: Absolutely. And I think particularly in the news recently with what's happening with large language models, these generative artificial intelligences that are creating, for example, images or music or conversations, it's raising some concerns around what types of data are these artificial intelligence being trained on? Is that data being ethically acquired? Is it being correctly labeled? And is it representative of the task at hand?

In particular, when you think about medicine, you want to make sure that any artificial intelligence tool that might be being applied to a patient would've been trained on data that would include the patient in question. I think, furthermore, you got to wonder, can you trust these algorithms? Can we peer into the black box, so to speak, and try to figure out just why exactly an algorithm may recommend X or Y diagnosis or X or Y therapy for a given patient?

And some of the steps that we're taking within our laboratory is to partner with a lot of the fantastic engineering groups that are at Penn Engineering that are thinking about trustworthy artificial intelligence. So, asking the serious questions around what does that training data look like? Is it representative of the patient population of interest? Is it really thinking about, for example, biases that can be built into the data? Is it further thinking about, well, how can we peer into the inner workings of some of these machine learning algorithms to try to better understand how it's coming to its predictions and its considerations for what it might bring up as a recommendation?

And then, I think most importantly, we're also trying to better understand how do patients feel about these technologies and that if these technologies are going to be used on patients or with patients, in what manner do patients want these algorithms used? I think that patients should be included in that conversation and help us, as providers and as researchers, better understand how these technologies need to be built into the workflow of taking care of patients.

Melanie Cole (Host): Excellent points. And you mentioned preoperative risk prediction, for example. I'd like you to speak about some of the applications you see it used for imaging, risk assessment, ECG monitoring, and in what ways is Penn Medicine innovating in the AI space?

Dr. Daniel Hashimoto: Absolutely. I think a lot of the places where using machine learning can come in handy is to think about the complex nature of the different data streams that we have coming into the hospital every second, every millisecond even, during the course of a patient's encounter. In every single interaction with any sort of device that we might be using, whether it's the EKG, as you mentioned, or potentially an EEG or maybe even a CT scan, an ultrasound, an MRI, these are all sources of data that are just gathering just gigabytes, terabytes of information about patients, and the relationships between that data are so complex that's very difficult for us as humans to really try to put that all together and different types of machine learning algorithms can help us integrate those to better understand, for example, which patients might be at higher risk to have a readmission to the hospital.

And if we can identify those patients earlier on in their hospitalization, can we take steps as clinicians to try to mitigate that risk, to try to identify perhaps the right series of medications or the right sequence of outpatient care, or maybe even a disposition to a rehab facility, and then to a subsequent step-down, a nursing facility before they go home in a manner that's going to allow us to try to minimize the risk to the patient?

From our perspective, furthermore, I think the other thing that we're trying to think about is trying to really think about novel uses of data. So even some types of algorithms that aren't FDA approved, we're not using them clinically, but we are investigating them in the laboratory to understand how can we predict risks based on, for example, if we're analyzing the video feed of an operation, can we try to pick up on what we consider to be sub-threshold events? Little things that happen during the course of an operation that in an individual patient may not make that much of a difference, but when you're talking about thousands, tens of thousands of patients, maybe something results in one complication. And if we can pick up on that by using these machine learning tools, that's one patient that's going to be saved from a potential complication. And that's the type of work that we're really trying to do to try to almost peer into the future in some way to try to see if we can prevent significant morbidity and mortality.

Melanie Cole (Host): That's what it's really all about. What an exciting time in this field of medicine. Dr. Hashimoto, when we talk about and think about computer vision and its role in intraoperative video analysis, tell us a little bit about why this is necessary and how the integration of AI into the interpretation of video during surgery really can aid surgical practice.

Dr. Daniel Hashimoto: Well, I think if you look back historically, we've been fairly good about collecting data from the preoperative side of care. So, that's when you think about plugging in things like ASA status or thinking about somebody's preoperative creatinine or their preoperative hemoglobin to understand where they are from a surgical risk perspective.

Post-operatively, we've been very good at collecting data about, for example, did this person have an infection? Did this person have to come to the emergency department? Did they need a re-operation? But we've had this hole in our data collection and our understanding of the surgical process when it comes to what's happening inside of the operating room. And what we know is that we've had to rely very heavily on operative notes. And there have been very large studies that have been done in, for example, the Netherlands, that have looked at the accuracy of operative notes, and they have found that 27% of operative notes are either incomplete or, worse, incorrect entirely.

And so, that leaves us with a hole to understand, well, if a patient has a complication, why did that happen? We don't want to just throw our hands up in the air and say, "Well, that was unavoidable. Hope it doesn't happen again." What we really want to do is understand, "Well, what happened during that course of that operation that may have contributed to this complication?" It may have been unavoidable. And it may have been something that's not necessarily an error, but just a part of the operation. But if we as surgeons can better understand what are these small things that occur during the course of the operation that we can learn from and that we can use to get better, that's what we want to try to achieve. And that's where I think computer vision really plays a role.

There is a ton of data that is being generated during the course of a surgical operation, and most operations are not video recorded. They're just operations that happen and all that data just disappears into the ether. When we capture the video, it allows us to look at it in an anonymous manner, because you don't actually need to record the identity of the patient or anything like that, what's happening during the course of the operation. And we can use these algorithms to pick up on pixel-level differences in the video to see if those differences correlate to any outcome that happened afterwards. That could be a complication or it could be a good outcome. We want to understand what leads to a favorable result for a patient as well. And I think that's where these algorithms come in really handy because the level of resolution that you can get is much sharper than we can get with our own eyes. We have biological limitations as human beings. We're very good at things, but sometimes it can be helpful to augment our ability to perceive visual events by using the technology that we have available to us today.

Melanie Cole (Host): Wow. So interesting. And thank you for telling us how you envision this translating to patient care and how you're translating these findings into potential treatment strategies that add to the promising approaches being developed. So, Dr. Hashimoto, before we wrap up, I'd like you to expand on your own efforts in AI and its application to surgery, and you're the director of the Penn Computer Assisted Surgery and Outcomes Laboratory at the Perelman School of Medicine. Tell us a little bit about the laboratory's mission, what you hope to achieve in the years ahead, and how you think this will change medicine for patients and providers.

Dr. Daniel Hashimoto: Absolutely. Well, in our lab, which we call the PCASO Lab for short, we're really a multidisciplinary group. Our laboratory is home to surgeons, to computer scientists, to statisticians, to bio-engineers, and we bring those different groups of expertise together because what we know is that this type of advance and this type of move toward the future can't be done in isolation. We really need the expertise of lots of different fields of study to come together in creative ways to try to make surgery safer and more accessible.

One of our main goals is to try to augment the ability of surgeons to make good clinical decisions by taking advantage of these advanced technologies and offering expert clinical surgeons even more data to make them see things that they couldn't see with their own eyes to help them integrate complex data points that couldn't be perceived without the help of a computer. It's almost like think about designing a GPS for surgery. Some of the algorithms that we've been able to create with our collaborators that are around the world allowed the machine to actually highlight, for example, the optimal plane of dissection, and to also mark areas in the anatomy that would be considered what we call no-go zones or unsafe areas that could result in a severe complication.

And it's about trying to bring those elements together, tracking where the instruments are, tracking how they interact with tissues; understanding how, for example, inflamed tissue or malignant tissue, necrotic tissue can all potentially change the way a surgeon may want to approach their operation and to tie that into data from prior operations. So, it's learning from hundreds, thousands of operations that it has been able to analyze previously to inform the current operation and to augment that decision-making capability with the goal of resulting in a safer surgery and a better outcome for the patient.

Melanie Cole (Host): Great information. What an exciting time, Dr. Hashimoto. Please join us again and update us as you're learning more about this fascinating topic. Thank you so much. To refer your patient to Penn Medicine, please call our 24/7 provider-only line at 877-937-7366 or you can submit your referral via our secure online referral form by visiting our website at pennmedicine.org/referyourpatient. That concludes this episode from the specialists at Penn Medicine. Please always remember to subscribe, rate, and review this podcast and all the other Penn Medicine podcasts. I'm Melanie Cole.