Reducing Diagnostic Errors: Peer Review & Decision Support

Introduction: Stepping Back to the Big Picture

I'm going to step back and look at the big picture of how and where we can try to reduce error and why we should reduce error.

I'm going to briefly touch on some of the themes from yesterday about how our practices are still too variable for us to be able to manage error appropriately.

I'm going to go into what I think are some practical solutions on how to reduce error across that value chain, which I've talked about and again today.

I'm also going to talk about managing peer review in the traditional sense of what we should be doing on a day-to-day or at least weekly basis in our own organizations about evaluating our work product, particularly around the radiologists.

The Iceberg of Hidden Errors

I showed you this yesterday. This is what we don't see or don't know below the iceberg.

This is what we really have to start getting into to find a way to look into all of this error that we routinely perform, partly for understandable reasons, as I said yesterday.

I think we can actually do a lot to reduce or at least elevate this iceberg such that we can see more of what we do and therefore understand it and deal with it and improve it.

Error is Inherent in Human Practices

Slide from yesterday. Error is inherent because we're human.

I just want to reiterate the quote. It's not enough to do your best. You must know what to do and then do your best.

I introduced the concept of the value chain where we really have to look at every link in this chain to improve our practices because as I said yesterday, if you order the wrong test at this process, then everything's wrong. The whole process is waste from there on out.

If you can be the best radiologist in the world, but if you are evaluating a test which was never indicated, then that's a huge error.

The Core Concept of Value in Radiology

One thing I want to introduce briefly here, and it's another whole talk in itself, is if you actually step back and think of what are we actually doing on a day-to-day basis at work? What are we there for? Why are we radiologists? Why are we subspecialty radiologists, et cetera.

I think at the end of the day, it all comes back to this concept of value.

If you ask most of us what we think of how and where we create value, we'll probably say I'm very good at CT or MRI devise the protocols. I'm a national expert in this or that, et cetera.

I actually think we really need to think deeper than that. What's at the core value. This sort of gets back to that value chain.

Frankly, if we are not attendant to the whole value chain, then we're just a cog in the wheel in that process. If we are not doing it, who is looking after the rest of that value chain?

Because we may be the best interpreter. But if the image was taken inappropriately, the wrong protocol was used, the wrong dose was used, then I'm not sure how much value we are really producing.

Ultimately we're here to provide actionable information. We're in the information business. I got that term from Dita Anzman from UCLA. He wrote about this.

Our ultimate product is the report. But more than that, not just any report, it's a report that is concise and has data in there which is actionable.

What I mean by that is that the referring physician can take that data and make an informed decision on the next best test, step treatment, et cetera, for a patient.

If you put a report out, and Mike and I have been talking about this, and we see this in our practices, four centimeter left renal mass suspicious for malignancy, recommend urology consult. That's not particularly actionable.

That then goes to the urologist who's going to come back to us and say, tell me more about that. You say it may be malignant. Well, okay, I could biopsy it, but what's the chance of malignancy? Might I have to operate? Give me some sense of what you actually think that is.

The way to deliver an actionable report is ultimately to extract the maximum amount of diagnostic information out of the data set, which will now increasingly include collateral clinical information off the electronic medical record, which I talked about yesterday.

No longer will it be acceptable just to look at the image and the history.

I heard someone today, I think it was Nancy. She just left. This isn't a Nancy thing because it happens all the time. And I say the thing too. Nancy said, if we are lucky to get the history, we might be able to think of X, Y, and Z in the report.

Nowadays, we won't have that excuse because all the data's in the electronic medical record. Granted, it may be hard to get it out, particularly if it's a standalone system and it's not integrated in the PACS and the RIS, but it is coming.

How do we get that data out? As Jim Th his coin, he's written about this, we're very good at putting data into systems, but how do we get it out of systems and create knowledge from that? That can inform our practices and improve our services.

The core function is an information business, which is the rapid delivery of actionable data that are precise and personalized.

Personalized Medicine and Outcomes

The other dynamic going on, which we're all too familiar with, another whole talk, is the personalized or precision medicine agenda.

If you think you can offer today a report which is sort of generic, even if it's somewhat customized, we are going to have to think in the new domain of it has to be customized to the individual patient given their biomarker idiosyncrasies.

It won't just be a generic abdominal report. It will incorporate all that collateral clinical data and customize to an individual patient.

All of which of course goes to impact the diagnosis therapy. What we're getting into now ultimately is outcomes. This is the big driver, as I said.

Really we need to focus on outcomes, which is about value, and outcomes by the way is really gets back to this whole concept of value and which is really what we should be focusing on.

Challenges Due to Variability

The problem is, as I said yesterday, how do we do this? Because our practices are so variable and it's variable at every step in the work process and what we do, and frankly, it's variable in medicine as a whole.

Other industries have done a much better job than we are.

That variable practice along the workflow leads to variable actionable data. In fact, it's highly questionable. The reports are often imprecise and impersonal.

We have to rethink this value chain, how we deliver through the value chain.

Examples of Variability

If we just go through that value chain briefly, again, this is data here. I've managed to get the slides with the circles on it from yesterday in the southeast of the country, there's a 50% variation compared to the northeast and northwest in terms of imaging utilization.

Something's up there and someone's wrong. I'm not saying the southeast is right or wrong or vice versa. I have a hunch, but the answer's probably somewhere in the middle. That's unacceptable.

If you look at the protocols, and I've showed this yesterday, we have too many different protocols, too much variation. It's just simply not possible to manage all these protocols consistently.

You have a radiologist who comes in and says, I want it done this way or that way according to his or her preference.

If you look at reporting, we're all over the map. Do you use the EMR or do you not? Do you use voice recognition or standard recognition? Are we subspecialize or generalized radiologists? What's our turnaround times? How do we manage recommendations? What our communication protocols?

I didn't show this slide yesterday, it's work from out of MGH. Looked at 24 radiologists who interpreted 25,000 CT examinations. Pretty robust number of studies here.

We have fellows on the left, this is out of my division, fellows on the left, junior radiologists, which I think were less than five years in this study, intermediate were between five and I think 15 years. Senior radiologists above 15 years.

Look at the difference in recommendations between that group of individuals. 45% recommendation rate from clinical fellows versus about 23% recommendation rate from senior radiologists.

What's going on here? There's probably one could segue off for about 15, 20 minutes to talk about this, why senior radiologists are making less recommendations, but presumably they have more experience. They do and are more comfortable because they've had that experience.

Whereas clinical fellows and junior radiologists are still in that sort of gray zone uncertainty.

I showed this yesterday with recommendation rates for lumbar spine MRI, ranging from 33% again from our organization basically for the same dataset compared to 7%. For some radiologists, 7% is probably too low. I'm not saying that the 7% person is appropriate. They may actually be under recommending after lumbar spine MRI.

Then of course we get into the different types of reporting. Where's the actionable data in the free text compared to the customized standardized reports.

Analogy to Garage Mechanics and Second Opinions

Now I'm going to say something and I know I'm being filmed here, so this can go. It's interesting when you're filmed. I did a talk about teleradiology and the ups and downs of teleradiology and pretty much made a case for the fact that we are devaluing ourself by outsourcing imaging at night.

In other words, if we're so important between the hours of eight and six, what happens after six when we're happy for it to go to Australia and all the rest of it. It's a bit of a mixed message to everyone.

I said that and then I thought that was the end of it. Sure enough, about four weeks later, I get an email from someone from some other country who happened to pick it up on the net and it started criticizing me. This one may get me some criticisms too. No offense, but I think we are a little bit like this if we're honest with ourselves.

Here's the problem. What's the similarity between this guy and this guy? I think we're pretty similar.

If you take your car to a garage, and they look under the hood, et cetera, they look and they think, it could be this, it could be that. We'll try this. We try that.

Granted, if it's a tire and you can see it and it needs changing, it's easy, maybe like a skin lesion, you could see it. But frankly, I don't think we are that different sometimes to garage mechanics.

You can go to five different garages and get five different opinions for why your engine's not working.

We know the whole process that goes on in medicine in terms of second opinions. In fact, I recommend to many of my friends and colleagues if they come to me seeking advice as to what they should do.

We're all in this business, of course, we have friends and colleagues who come to us saying, do you know someone? It's in your hospital, who can help me out here, there, and everywhere? It's usually outside of my domain and radiology. It might be a surgeon or a neurologist or something.

I don't profess to know much about neurology at all. But the bottom line is I think they're being smart. They're getting second opinions.

It may not be with my organization, I don't have a particular issue with that, that they can go somewhere else. But if you have a complex disease, I think it's highly advisable to actually go and get different opinions.

You may get a very different opinion from one organization to another. This issue is germane to medicine as a whole.

Partly of course 'cause every patient's unique. We can't as much as we should be into standardization, it's very hard to do that given that every patient presents with their own unique findings, let alone the psychological overlay that the patient's present with.

Handling Overwhelming Data and Best Practices

How do we handle this overwhelming data? There are too many recommendations and too many best practices out there for us to consistently adhere to them. Frankly, I think it's impossible. To give ourselves a bit of a break.

I know I'm on the soapbox banging the drum here saying we're not doing a good job. But on the other hand, how do we keep up? In imaging, how do we keep up with the protocols, the dose, 256 scanners, spectral scanning?

I mean, we do now in our organization, I remember when I started as a fellow in 1992, we did the number of images in an MRI is now equivalent to the number of pulse sequences we do. In fact, we do more pulse sequences probably than we did images.

We probably did a T1 and a T2, maybe something else, maybe a bit of gadolinium or something. We might've been dealing with about 20 to 30 images.

We deal with about 50 pulse sequences. If you add them all in with the DWI, the ADCs, the fat sat, the Dixon suppression technique, the scouts and the T2s, I can't even keep up with them.

In fact, one of my colleagues said to me on Wednesday, an MR colleague said, I only look at the T2s, the T1s and the post gad may be a fat sat, I don't, I ignore the rest. Now, I'm not saying that he's right in doing that, but we just cannot consistently follow what we say we should be doing.

I'm not even sure we should be doing all those pulse sequences, by the way.

Here's an interesting one. It takes 17 years before a seminal best practice is introduced routinely across this country in most, if not all, practices from the original seminal article.

You take propranolol or aspirin, for instance, in myocardial infarction, it takes 17 years to pervade healthcare organizations. That actually is kind of reflective of who we are and how we practice.

Think about the airline industry. That lithium battery thing, which I showed yesterday, they got on top of it straight away. They shut the planes down, they grounded all the planes, and within a hundred days, they're back up and running with new batteries.

Granted, we are not the airline industry and we're far more complex. I don't know, a jet airliner to me is pretty complex in trying to manage that thing. But I will propose we are more complex than a jet airliner.

Moving to Practical Solutions

How are we doing in all this process? I showed this slide yesterday. We tend to think of peer review and reducing error just basically the end that we see as radiologists, which is on the screen and our diagnostic ability to diagnose and communicate disease.

I shared with you that Johnny Resco has this data that with frankly we do an appalling job even in peer review at that end, let alone all the rest of it. This is from 2012.

2012 is well after everyone started to wake up about the need to do peer review, particularly because the joint commission and others say that you have to do it.

Let's move away from the problem and let's see what we can do in the next half an hour or so to actually try and fix some of this.

Granted, what I'm going to be talking about is still only a few organizations are able to do well. The tools I'm going to show you now are available, and I'm not here to promote any products.

You can Google this stuff to be able to allow you to deliver the best practices, reduce error, and allow us to get back to this ultimate product or actionable report.

Scheduling and Imaging Appropriateness

Let's talk about scheduling first of all, and imaging appropriateness. It's all about safety and efficiency.

Imaging appropriateness. If we order the inappropriate CT, that's all kinds of costs. There's costs around anxiety, there's financial costs, there's radiation costs, quality and safety costs, et cetera. It's all over the map.

It adds nothing to the healthcare of that patient, except you might find an incidental finding which you then go and chase and order an MRI and ultimately may even biopsy it. In the worst case is you create a complication.

The utilization should be evidence-based using appropriateness criteria. The ACR appropriateness criteria have been out there for 20 years or so, which are updated.

How many of us have actually when was the last time we actually looked at the ACR appropriateness criteria? I know we're looking at bits and pieces, but how many of you can confidently say in your organization, in your department, you're consistently following the appropriate criteria set by the ACR, which by the way are set by experts in the field nationwide, many of whom you've heard and seen. Some are even probably in this room who have set these guidelines.

In fact, I think Jen yesterday you said you were on Jen Bennett on one of the ultrasound guidance appropriateness criteria.

This stuff isn't fluff. It's not there just to put on the wall and look at it and say that looks nice and not use it. The people have put hard work into this stuff. They've looked at the evidence. Granted, some of the evidence is hard to get, but they've used the best evidence available and they developed these algorithms.

Why for us to be able to image the right test to the right patient at the right time.

The problem of course is it's just a guideline. It's not mandated. It's not really embedded efficiently into our workflow.

It's all very well having the appropriate criteria. But if we have to go to a computer to look it up for every case or a piece of paper, et cetera, and you've got 50 radiologists in your department, and you've got 600 referring physicians, it's just you cannot manage it. It's just not possible.

Frankly, the only real way for us to consistently adhere to these best practices ultimately will be through computer based decision support algorithms which incorporate the ACR appropriateness criteria and is an effective way of managing utilization.

The way we really manage utilization now is we've outsourced it to the RBMs and to the referring physicians have to get pre certifications, which of course as we know drives them crazy 'cause they have to make this phone call which is a redundant process because we are not able to manage that process.

I'm saying part of the reason we can't manage it because it's too complex, it's too difficult, it's too frustrating. It's unreasonable to expect everyone to pick up the phone every time you see an inappropriate request come through.

May take you half an hour to get to the referring physician. They may say to you, in the worst case, look, I know better than you buddy. Please just do it. It's fraught with difficulties.

To get to this precision imaging or the right test, the right time, the right patient, it's going to involve computer algorithms.

I'm just going to go through our clinical decision support system. This is proprietary, it's in-house, it's not available. But there is a product out there which is available. I think there's two right now which are available, and which is being embedded into a number of larger organizations nationwide.

Just to show you a flavor of what we do, this is every single imaging test now in our organization is ordered through this way, including the ER, including interventional radiology, ultrasound, plain film, CT, whatever you want. You have to go through this system.

For it to work, of course, it has to be seamless, it has to be easy to use. The referring physicians have had to be informed about the whole process, and they were certainly incorporated into the implementation phase, and we sought their opinions when it was being built.

You put the unit name or number of the patient in, it links. First of all, you put the physician in, it links to the address and phone of the patient, name, date of birth policy, it's all secure, password protected, et cetera. Everything's logged by the way.

That's a key point here, because it's logged. You can data mine and you can monitor performance adherence and compare one physician to another within the same division from the same specialty and across different or other organizations.

In this case, we're going to go through the tree and order a lumbar spine MRI. It's automatically specifies the CPT code which is appropriate for billing. It's all embedded into the system.

It asks you certain key data safety and lab work. Does the patient have claustrophobia? Does the patient have any internal devices that might preclude them?

Think about that. The efficiency of this process in trying to get it right is at this point, we will determine whether the patient should show up for MRI or not.

Far worse the patient to arrive at the MRI desk. Oh, you have hardware. Sorry, we can't do your test today. We'll have to reschedule. Another test.

At this point, there's the safety checks are done so that the appropriate algorithms or management can be put in place, duplicate exams. It will find out across the system whether there's been a duplicate exam when, and whether this test is really indicated and needed across the multiple partners institutions.

It will inform about the BUN and creatine, et cetera. This is all embedded into the ordering process.

I can already hear people saying, referring physicians don't want to do this. This takes too long. They're having to do all the work. Someone's got to do this work. They're the ones with all the most of the information.

Ultimately, of course, this should automatically be dumped into it as we get better at getting this knowledge out of electronic medical record systems.

But on the whole, in fact almost exclusively in our organization, this has been accepted as the way to go because it leads to better outcomes and better clinical care.

We're going to put in back pain for six weeks for this particular patient. This is following the ACR recommendations. Remember? They put in back pain six weeks, and this is what happens.

It specifies the ICD code. Boom, you'll get a score of one and a score of one means that you should not proceed with the test. It gives you alternatives here whether you should be doing an x-ray which I think is three up there or a CT which is a one, a one or a two as well.

In other words, this test is not indicated. Now you can proceed with the exam, but you have to explain why you're proceeding with the exam. For certain examinations, you're going to have to call the radiologist to justify it.

By the way, we are logging it, we're tracking it. If you're going to try and game the system and just go through this on every test, you'll get a phone call from the utilization management person in the organization just to say, why are you doing this?

By the way, this whole process with most of the payers has obviated the need for pre certifications because this is the pre-certification. They've accepted this as appropriate tool to be able to manage these expensive tests which we're ordering.

This test you won't be able to get. It's inappropriate. I'm not an MSK radiologist, but I think even I know that back pain for six weeks without any reflex changes, weakness, and all the rest of it is not indicated.

If you add osteoporosis, if the patient has osteoporosis, then it bumps to a three if they're reflex changes. Now we've got some problems in the spinal cord. That jumps to a nine.

Embedded within these algorithms are all the possible indications are in there such that it will choose the right test for your patient at the right time.

This is published in 2009. This was CT, it's not really an outcome. Outcome of course is what the patient feels like, whether they actually got better or not. But it's an outcome certainly from an imaging perspective.

We were able to reduce the growth in CT through this process.

You might say, that's what it was going to be anyway because of the radiation, because of the utilization management programs 'cause of the DRA because of bundling and all this kind of stuff. But this actually sort of came in. This is 2005, I think. Yeah, right? This trend happened before of that, it was really 2007, 2008, where most of us in our practices started to see a big hit in CT. This predated that trend.

Protocols and Dose Management

That's scheduling. What about protocol? I'm just going to spend a minute or two on protocols. Frankly, we're all over the map with protocols too.

This is about peer reviewing at every step of the way. I'm not sure peer review is the right word for every step of the way, but I think we need to think of ourselves as peer reviewing every step of the way.

I talked about radiation dose, which is all over the map. We really have to take this dose situation absolutely seriously.

I know we say we take it seriously if you ask us. But who is actually dealing with it on a day-to-day basis in your organization? How accountable are they? How do you know if they're benchmarking?

It's all very well saying I've reduced the radiation dose in my organization from let's say 15 millisieverts to 10 millisieverts from the doppler pelvic CT. I've done a great job. I've got it down by five millisieverts. That is good. Not complaining about it, but some organizations out there are getting it down to five or six for real stone protocols.

We are getting it down to two or three with the spectral scanning machines that are coming out. The goal is to actually get to sub millisievert scanning.

This gets back to you don't know what you don't know. If you don't know what other organizations are doing, how do you know whether you are doing the right thing or not? Whether you actually are implementing best practices.

Granted, it's tough. Mike and I had a conversation with Todd over lunch, and how many of us are our value is seen in terms of the number of RVU units we produce? That's an interesting concept of value, isn't it?

From a patient's point of view. If you ask what, where, how we create value from a patient's point of view. It's all about did you make the right diagnosis from my CT scan, for instance? Not how many did you read?

But that is how we're measured within our often with our own organizations, which will change probably with this volume to value dynamic.

We are all over the map with protocols with dose. I don't have an easy answer for this except to say we need to take it seriously.

I think ultimately decision support systems will take a part in this. I think one thing I should say is make sure that they're standardized.

The same organization I was in last week has MRI and CT protocols, but every examination is approved by a radiologist. I think it was 60% of the time they changed the protocol.

I don't know how you manage an organization like that in terms of because then the technologist said, who's on today? Okay, it's Dr. Joe. Okay, we better ask Dr. Joe 'cause he gets upset if we use the standard protocol for what is a given set of standard circumstances here, we better find him. Where is he? Let's call him up. Dr. Joe, what do you want, yeah, I do want to change it. It's not a way to manage your practice.

Procedure End: Efficiency and Asset Management

Down here on the procedure end in terms of how we know whether we're managing our equipment appropriately? How many of us have actually gone in and looked at equipment utilization, staffing mixes?

It's all around efficiency management, asset management. Lean is popular. I'm not saying it's the only way to do it, but it is one tool to use to see whether you are operating as efficiently as you can and whether you can have the opportunity.

I can bet you in every organization there's always the opportunity to remove waste, even if you've done lean, because machines keep changing. Protocols keep changing, as I've talked about.

You need to flow chart the operations. Remember that inpatients and outpatients are completely different businesses.

In fact, many patients are scanned outpatients are scanned on the same machines as inpatients. That's highly inefficient way to operate a business because of the different types of businesses.

Another whole talk on that in the sense that inpatients will always trump an outpatient who now has to wait maybe two or three hours inpatients who can get in and out. Outpatients you can't.

This is just briefly a study we did. I looked at the modality management on we felt there were about 36 tasks that technologists had to perform.

If you multitask, in other words, you had two or even three technologists. The third technologist could be a tech aide to support these tasks. You can work in parallel as opposed to in series.

You can get up to 10 examinations per hour. I'm including a chest and an abdomen as two examinations with two to three as opposed to three examinations in hour with one.

If you want to look at the dollar revenue perspective, and this may change as we get into ACO type payments, then trust me, we're still going to be driven on efficiency and productivity, even if the payments are going to be driven in a different way.

Interpretation: Using All Available Information

To the big one that I think most of us are interested in here and most of us are here at this conference for, but I did want to spend some time on the rest of this value chain because again if we're not addressing all of these issues, you can be the best radiologist in the world, but it's going to be meaningless 'cause you've done the wrong test or the wrong protocol.

I've already mentioned I think it's essential to use all the information possible. It's very difficult without a seamless electronic medical record.

Our electronic medical record, like probably some of yours, is now embedded into the PACS and the RIS such that we can seamlessly, well, not as seamless as we'd like, but relatively seamlessly extract the information.

Of course, prior reports is a no brainer. But what about the lab work? What about the surgery?

We are looking for instance Mike was talking about was it internal hernia or something at bowel ischemia, bowel ischemia, et cetera. Has the patient recently had surgery?

If I knew that they've recently had surgery, it might actually trigger me 'cause these cases were missed with internal hernias. It might actually trigger me to think that this could be an internal hernia as a result of the recent operation they had such that I can now call the referring physician and we can prevent the bowel ischemia.

You get the point, whether it's pathology, other biomarker data, and I mentioned yesterday about the increasing use of genomic data to drive practices.

I don't know whether in your organizations, when you go to clinical realms, something really struck me about six months ago, and which I'm now trying to change the way we think at least in my division, hopefully in our department, is when I go to the clinical oncology conferences and it's a GI conference, they're talking a different language.

They're off in the KRAS BRAF or FOLFIRINOX domain. They're talking way beyond what we are talking about. It's pancreatic cancer and it cases the superior mesenteric artery, I'm done.

But they are often this domain and we have to really keep up with these folks because unless we keep up with them and talk their language, we're going to be back to where we were in the old days, which is just some sort of adjunct in the process.

If we really want to deliver this actionable report which we're talking about, we have to talk to the same language in this example with an oncologist. Of course it could be neurologists, dermatologists, whoever you want to talk about.

We need to be in their space walking step by step in this process.

Synthesize the data into meaningful report needs to be concise, clear, critical, standardized language, standardized ontology. We can't do appropriate data mining if you're using that.

If you're looking for mass and we use a tumor, how do we know how many patients have a tumor versus a mass, hyperlinks to key imaging findings. Of course the recommendations.

This is what our data mining tool if you put in I don't know what's to happen to the R but if you put for instance put in cancer like a Google in EMR, this is on the desktop, on the PACS, it'll come up with every case of cancer. Name of word of cancer throughout the electronic medical record.

You can put in for instance SLE or scleroderma, if you have a thought, and I saw a lee yesterday showed a case of sarcoidosis. I wonder if this patient has sarcoidosis. Boom, it'll tell you there.

Whether they got sarcoidosis or not, if it's known in the electronic medical record, ultimately where this is going, and we're working on this, is when you bring up a CT, and I'll use my area again in GI Oncology, when you bring that CT up, what is going to happen?

'Cause this is already happening in the ER. When you present to the ER at MGH, you get 70 data points before you've even seen the patient, their prior history, their drugs they're on, allergies. A whole slew of data.

When we go to pull up say a GI CT for someone who's got colon cancer, we are going to have their path, their genomic data, their surgery, their white count, all the key clinical information that we will need to provide.

Why getting back to this actionable report, prior history and everything, not just the prior report, but all that data that's necessary. If you really step back and think about it, that is going to transform the way we practice.

I think because we are going to become clinical radiologists through this use.

Managing Recommendations with Decision Support

I showed this yesterday, recommendations all over the place. PET CT was the most common test for this. How are we going to stop this variation in recommendations?

I would propose to you, and I wrote this about two years ago now, that the only real effective way to do this in a standardized consistent manner is to use decision support systems.

I'm going to show you what we have piloted in our organization. We've got about five or six RSNA presentations on this. We think it's potentially a game changer in the way to be able to manage our recommendations and make them more consistent.

Here for instance is the adrenal algorithm out of the JACR appropriateness for recommendation criteria, but it's all over the map. If you go to the chest desk in our hospital, even within my own division, people will recommend different tests with the same patient as I showed yesterday.

How do we incorporate this into a standardized uniform output for the SA every time giving the same recommendation for the same context of patient?

I already said it's just not possible to remember all of these best practices. I mentioned to Jim Bennett yesterday, when I get some issue in the pelvis with an adnexal lesion, I frankly just can't remember all the most appropriate recommendations.

At the moment, I have to go and ask some of my colleagues for a ovarian cyst that's five centimeters with a septation, what do I do with it kind of thing.

You probably all know what to do with it, and I do know what to do with it, but you get my point.

We've developed recommendation algorithms for the liver, kidney, adrenal, pancreas, and adnexal. The two that have actually and that most of those are going to be implemented in November.

I think this is the only way to adhere to best practices.

On the voice recognition screen, there's an app if you will, a little icon. Up this comes. This is the earlier version, adrenal pulmonary nodule. This is for abdominal or chest radiologists if it happens to be an adrenal nodule.

We click that this comes up, you put the data in, series image number size, put in some of the diagnostic features. What's its density? Is it hypodense, less than 10 Hounsfield units, et cetera? Has there been a history of malignancy? Has it changed in size, et cetera?

Let's do this. We're going to put in a 16 millimeter lesion, no diagnostic feature 'cause it was a contrast enhanced CT stable for six months. The patient does have a history malignancy.

As we are putting this data in, it's formulating the standardized text on the left. When you're ready, so there we go. It's formulating that data, boom, it dumps it right into the voice recognition, standardized report, same language, same text, every radiologist, same recommendation.

We even put links to at least references as to why some of the indications are made. For instance in our organization, the endocrinologists like we ask for endocrine testing for an incidental adrenal lesion because of the risk that it's hyper functioning.

As radiologists, we don't tend to buy into this, but they are clinical colleagues and they say for instance up to 10% of hypertension they believe it's due to subclinical hyperaldosteronism, for instance.

That data gets reported into a standardized template. This is our generic template. We're changing this now by the way to be standardized and customized to disease.

Such if you are evaluating pancreatic cancer, you will be needing requiring to evaluate the SMA SMV, the celiac axis, the lymphadenopathy, any liver metastases, et cetera.

It's not just your old generic CT report anymore. It's customized to the different disease. Of course the recommendations are always in the same place.

The referring physicians go immediately to where the act and impressions to where the actionable data is.

Peer Review in the Conventional Sense

In the last six or seven minutes, talk about peer review in the conventional term that we are familiar with.

Why should we do it? It should be self-evident. None of us really can claim to be good practitioners if we are not concerned about our errors. An error abounds as I've said.

However, despite the fact that we know it's going on, much resistance is out there from our colleagues in terms of how to manage it and how to look at it.

What is it? It should be a structured process, fair and transparent. Opportunities for error reduction in self-improvement.

We should be able to benchmark against other colleagues and other ideally other organizations. It's required for the joint commission and for ACR accreditation, it's used for OPPE and FPPE.

It's becoming part of our work practice. The attributes are in the conventional sense of interpretation, reporting and communication.

Let's not just not think it's about interpretation, it's about the reporting and the communication. Ideally it should be unbiased, random.

The problem is often it's not random. I've heard of people say in its worst sort of form is someone's out to get someone. I'm fortunately I don't work in an organization like that, but it happens.

You can always find out if you want to. You can always ding one of your colleagues if you want to go out and look for it, you'll find it 'cause we all make mistakes.

We should incorporate both our opinions and others. It should be consistent methodology, timely, continuous, ideally not disrupt the workflow, proactive versus reactive.

What I mean by that is it should be at the point of care as opposed to six months previously 'cause what's the point in a way from a patient's perspective, if you find out the mistake six months down the line, the horse is bolted.

Resistance to Peer Review

Here's some of the core resistance. It's too time consuming. Not everyone buys into it. I think we're all familiar with this dynamic questionable value.

Does it actually really change the way I do things? Because look, we're only looking at one or 2% of cases. We are actually not looking at 98% of the cases. How much does it really change?

This is a big one. Of course it's not reimbursed. Remember how we valued around our RVU productivity? That's how we incentivized. In this outcomes world this is going to be a much bigger driver, even though we're not reimbursed today, perhaps we will be by our organizations.

It's seen as punitive, too subjective, and frankly I don't blame people if given the choice most of us won't do it.

I'm not sure I would be doing regular peer review had it not been mandated by my organization. I'm sorry to say that, I'm a little bit ashamed to say that, but you know, we're all in this situation together.

Incorporating Peer Review into Workflow

How does one incorporate into the workflow? Ultimately it has to come. Like anything in life frankly is from leadership. You've got to have the vision.

You've got to have the leaders in your organizations to say this is the right thing to do. Bring the teams together, here's the teams to find out the best way to do it for your organization and implement it and follow it and evaluate it and iterate accordingly to improve the system.

It's all about culture. Leaders set the culture in the organization, and we've got to get into a culture of quality and safety. Frankly that means peer review and not just every six months.

Not really M and M meetings, yeah that's partly peer review. But if it's just M and M meetings, you're only seeing the most egregious mistakes. Even some of them aren't brought up to M and Ms because they're egregious.

How do we do it? It depends on the type, whether it's a diagnostic error or communication error. This is sort of the M and M or case review. It's reactive bias and often subjective.

I've said proactive is preferable. We are looking and I propose we should look at imaging appropriateness, the protocols, the report format and language, and the communication as well.

I'll show you that's what we do in our peer review process every day. In my division, briefly, RadPeer, there's been a lot of criticism about it, but that's the main one out there.

People think I've done RadPeer I'm good to go. All the rest of it, I'm not so sure. A radiologist comes across a discrepant case maybe six months later as it's described. It through triggers a review event.

You score it, it's then ordered by a committee. The results go to the ACR. It's all monitored by the ACR, the data share with the individuals.

But here's some of the problems. Selection bias, case bias, mix, reactive error detection, too late, and only assesses interpretive error. There are many other areas. As of course as I've shown you.

These are the scoring system. These are the ones that trigger a response diagnosis that should be made most of the time or almost every time there's subcategories whether it's clinically significant or not.

Here are the problems. It's not routinely random, it's not fully integrated into the workflow. The original radiologist is known. That's often a problem.

Review could take months after the event. The original radiologist may actually be right. Who reconciles that? The communication with the physician's too late. The horse is bolted. Doesn't go into the report. The report doesn't change necessarily.

RADPEER here is known to have a number of problems. What are the expected error rates? About 3% of all cases are the ones where there's a miss. That that wouldn't be expected to necessarily get the right diagnosis.

In the three and four category it's about 1%. In other words you should have found it and it's probably significant, but it goes up to about 5% in CT.

Remember that paper I alluded to yesterday which from our own organization that about 30% of the time we're making intra and inter observer errors with abdominal CT.

Proactive Peer Review: The Grapevine Process

In the last minute or two, I'll show you what we do. It's proactive, real time. It's done on the cases from the same day. 2% of cases are reviewed.

All participants that day on the clinical set schedule or in the room, it's we built a CT that is not punitive. We don't actually know, except the person who read it probably knows whose actually case it is.

It evaluates the whole chain. The appropriateness, the communication, the text, the language, it's consensus versus no consensus. We actually have a paper performance on this.

It's called Grapevine. I've blocked out the names we put in. Who is who are we going to evaluate? For instance these three radiologists.

This brings up just random number of cases that 2% of their cases of the from the day could be a CT, an ultrasound, whatever you want to choose.

We then look at the text, we evaluate the report, we see the images. We go scroll through the images, we look at the text at the same time we see whether language is appropriate, whether recommendation is appropriate, whether communication was appropriate.

We put a consensus or no consensus. This is done across the department in every division in our organization every day. I believe it's every day I have.

It may be IR may be an outlier there because of their workload. We our goal is to do a 2%. We're scored on this. If you're falling behind we want to know why you're falling behind on the peer review process.

We also by the way evaluate the technologists such that and this is an icon on the desktop on the PACS. If we feel that inappropriate examination was done, the wrong protocol, the wrong slice thickness, we feed back to the technologists in this way.

What the problem was, clip posterior lung or lateral, for instance. This is another QA peer review process if you will.

Conclusion: Reducing Error Through a Systems Approach

Finally, how do we reduce error? The best approach of course is reduce it in the first place. My mantra here is a systems approach to this value chain going through every aspect.

Decision support for scheduling, standardize the protocols, lean for the procedures, standardized templates for reporting and clinical decision support, electronic feedback mechanisms for critical and important findings, alerts such that we all find a way to add value in this actionable report.

Peer review every aspect of the work product, yes it's not directly reimbursed, but we have to do it. We should do it.

The way I'll ideally is real time evaluation of cases, random consensus. The original radiologist should be present so that they see their error at that time transparent process.

Ultimately this leadership that drives this culture of safety and quality.