VLOG: NeuroOp Guru: Can AI distinguish between papilledema and optic disc drusen using color fundus photos?

Andrew Lee, MD, and Andrew Carey, MD, sit down on another episode of the NeuroOp Guru to discuss deep learning systems and whether they can distinguish between optic disc drusen and true papilledema

Video Transcript

Editor's note - This transcript has been edited for clarity.

Andy Lee, MD:

Hello and welcome to another edition of the NeuroOp Guru. I'm here with my good friend from Johns Hopkins, Drew Carey. Hi. Dr Carey.

Drew Carey, MD:

Hi Dr Lee.

Andy Lee, MD:

And today we're going to be talking about deep learning systems and whether they can distinguish between optic disc drusen and true papilledema, just using color fundus photograph. So Drew. Why is this important? Can't just people do this?

Drew Carey, MD:

Well, I think a very skilled neuro-ophthalmologist is pretty good at distinguishing between optic disc drusen and papilledema, but there are some cases where it's a real challenge. Particularly with buried disc drusen and younger patients, especially in little kids, if they're not going to sit still and cooperate with the exam. If we're lucky, you can get an OCT that can be helpful, but a lot of eye exams are not done by skilled neuro ophthalmologists. We are a limited quantity with a limited ability to see a lot of patients. And if we can find way to help our colleagues distinguish this, it can help to reduce a lot of unnecessary anxiety, unnecessary testing, and to help us focus on the patients who really need us.

Andy Lee, MD:

So what'd they do in this study?

Drew Carey, MD:

So this was a very large retrospective study with a lot of centers. They had 30 centers in this study group that sent in photos for the training data set, and they sent in patients who had papilledema and patients with confirmed disc drusen using validated criteria. And then so they trained an AI system on these photos. And they said, This is papilledema, this is drusen. And you, AI system, figure out how to tell the difference between the two of them. And then they validated that training system on some of the internal photos. But because you don't know–did the machine just memorize them, or did it actually figure out the difference? You also want to test it on an external system, or external training set that the deep learning system has never seen. And so they did that. They got seven additional centers to send in 426 photos on top of the 4000 that it used for training, and then it evaluated those photos with the known diagnosis to see, could it tell the difference between papilledema and disc drusen.

Andy Lee, MD:

Maybe you could just walk us through this sensitivity and specificity graph here

Drew Carey, MD:

Absolutely, so this is a receiver operator curve, and on the bottom axis is one minus specificity. So at zero is perfect specificity. On the y-axis is sensitivity, and so one at the top is perfect sensitivity. And for a lot of diagnostic tests, as you shift your cut-off points–so the the AI system, it doesn't actually say drusen versus papilledema. It gives you a probability score. And it can say this is 98% probable drusen and 2% probable papilledema. And you, as the clinician or the creator of the AI system, has to figure out what you're comfortable with. And depending on what probability score you set, as your cut off, you're going to shift your sensitivity and specificity. And a lot of times for tests that will be trying to rule out a super scary thing like papilledema that might be related to a brain tumor or vision-threatening condition causing elevated intracranial pressure, you're going to maybe err on the side of sensitivity, because you don't want to miss something bad. And a perfect score would go all the way up and then across. And this is really quite very good for the area under the curve. This fits really well. And what it does show is, though, for buried disc drusen, it was a little bit more of a challenge, so there's a little bit loss of sensitivity and specificity, as you do buried disc drusen, compared to all of papilledema and then buried disc drusen compared to mild papilledema, which I think is what we see as neuro-ophthalmologists, we're not perfect and and sometimes you look at that nerve and you say, it could be a normal variant, it could be buried drusen, it could be mild papilledema. And we're going to need some ancillary testing, whether it's an OCT or an MRI scan to help us figure that out.

Andy Lee, MD:

And how does this compare to people?

Drew Carey, MD:

They did not compare this to people in the study. Of course, in the landmark BONSAI trial for papilledema versus not papilledema study, they did compare that to well respected and very good neuro ophthalmologists. In this study, they did not compare this to people. This was just training on an external data set.

Andy Lee, MD:

And is this commercially available?

Drew Carey, MD:

This is not commercially available. This is the first results that they have reviewed from this. None of the BONSAI algorithms that they've put together have made it through FDA regulatory training. FDA approval and available on commercially available devices–that is the hope one day. That either the AI package would be built into the camera system so you take your picture and it gives you an answer, or you could upload your color photo to some sort of online platform where it could then run it through a cloud based AI system and give you an answer. You know, in the US, the regulatory system is not as lightning fast as our AI algorithms are at processing the photos. And so that is a future aspect that we have to look forward to.

Andy Lee, MD:

So what do you think the take-home message for our audience is today?

Drew Carey, MD:

I think the take-home message is, this is really promising technology, and we can expect that we're going to be using AI more and more in the clinic, and it's going to be really useful. You can think of it as your your AI neuro-ophthalmologist on the computer. If you're a comprehensive doctor or any ophthalmologist, and you say that this looks funny, we're going to have more tools at hand that based on this data, look like they're going to be really reliable.

Andy Lee, MD:

Yeah, I think that AI is probably not going to replace neuro ophthalmology, but maybe the neuro ophthalmologists who are using AI might replace regular neuro ophthalmology. But I thank you again Dr Carey, for your time, and that concludes yet another edition of the NeuroOp Guru.