WHY CALIBRATION IS CRITICAL WITH AUTOMATED TESTING
Recently, Grason-Stadler hosted a discussion between Dr. James W. Hall III and Dr. Robert Margolis about the current audiology landscape. They discussed the patient-provider gap that exists and the urgent need for a solution. In this transcribed excerpt of the interview, they address the importance of calibrated and validated hearing test results and how that affects automated testing.
James Hall: I’m Jay Hall. I’m an audiologist.
Robert Margolis: I’m Bob Margolis. I’m an audiologist.
JH: You know, I have a question, Bob. There are other alternatives for getting a hearing test without going to an audiologist - maybe an online option, or maybe an app, you know, for a smartphone. There are a number of them actually. Why something like the ATMAS? Why not just find any available way to get a hearing test? How would you respond to that?
RM: There are two problems with these tests - you know, they started out as telephone tests, and then they were internet tests, and cell phone tests. The main problem is calibration. We are very careful about calibration in our field, so that we know that an audiogram done in one place on a calibrated audiometer by a licensed audiologist, is the same as they would get at another place. And that’s because we have standards for calibration, and we follow the standards. That’s really critical. When you have a test that’s performed either over the internet or over a telephone, the problem is calibration. That system hasn’t been calibrated in the manner that audiometers are calibrated. That is solvable, but it hasn’t been solved yet. But the main issue right now, is that they are not calibrated, and so they are not valid for purposes of, for example, diagnosis or programming a hearing aid. They would not be suitable. They are screening tests, and useful as screening tests; but they’re not to the standard that would be required for using them the way we use audiograms in a clinical setting. The second issue is, the procedures have not been validated. That’s not a simple thing to do. We worked years, funded by grants from the National Institutes of Health and from the VA, to validate the method and demonstrate that the method provides results that are equivalent to results that would be obtained by an experienced licensed audiologist.
JH: In a sound treated environment?
RM: Right. There’s no validation of those tests. There have been a number of papers published recently that show that it is possible, but if you let the patient use whatever, earphones they have laying around the house, well, they’re not going to be calibrated.
JH: Right.
RM: But if you can control the hardware, which is inexpensive now, then it is possible to perform calibrated tests that way. But the calibration issue has to be resolved, and the procedure has to be validated.
JH: I would assume the quality of the results is unknown with a method that doesn’t document the quality, like the AMTAS.
RM: Right.
JH: I think the main point to convey to the public or whomever might be using it, is the difference between a diagnostic test and a screening test. It’s fine for screening, but not yet ready for diagnostic purposes.
RM: Right.
JH: We’re extolling the virtues of automated audiometry, and my first reaction is, well, gosh, this is a no-brainer. Who wouldn’t want to adapt this immediately and adopt it? But is there any resistance - are there any negative impressions of audiologists about automated technology that you’ve encountered as you’ve developed this technology and launched it?
RM: Yes, you know, I got into this completely naïve. There was a great deal of pushback. I think that really has subsided a lot, for a number of reasons.
JH: These are practicing audiologists…
RM: Right.
JH: …practicing audiologists were pushing back?
RM: Right. One reason that resistance to automation is diminishing is that every audiologist I know is very busy. They’re learning that there are ways that they can manage their time more efficiently and provide service to more people. And when I do presentations on this, I always start with the data that shows that if you look at the need for audiograms and the capacity of the field to provide them, there is a wide gap.
JH: You can’t argue with that.
RM: You can’t argue with it. And you should not be concerned that this is going to take anybody’s job away.
JH: No, that’s for sure.
RM: The gap is so enormous and this is just one way to narrow that gap. But the important thing to me was quality control. When you take the audiologist out of the process, what you are losing is not the ability to turn tones on and off. You’re losing the expertise of the audiologist in identifying problems, and then making corrections to ensure that you have an accurate result.
JH: Yeah, there is a lot of judgment - I mean, that’s what makes a good audiologist.
RM: Exactly.
JH: The decision-making - it’s all learned, it’s unconscious. You don’t even realize you’re doing it.
RM: But I thought that the cues that the experienced audiologist use to make those judgments, could be written those down and a computer could track them better than a human could. And then you could build into the system the kind of quality control that experienced audiologists bring to the process. When we were developing the quality indicators for AMTAS, one of the things we looked at was age, because we grew up with this folklore that older people can’t do well in testing hearing. Well, that turned out to be not the case. We did, our first trial was in three different centers. One was a VA hospital, where there were many older patients; and age was not predictive of accuracy at all. The older people are perfectly capable of doing this test. And in fact, they like it a lot; and as you say, the quality indicators pick up those where there are problems.
JH: Okay. Of course, we’ll also detect those who are willfully trying to falsify hearing loss. I’m involved as a consultant in a major multi-site study. There are ten sites. Eight of them are VA centers; the other two are military facilities. And the population of participants in this study are veterans with mild TBI. I’m involved in just a small part of a larger investigation of the consequences of mild traumatic brain injury. The audiometric study involves collecting pure tone air and bone conduction thresholds from the veterans, and also three components of the Scan-3 auditory processing screening test battery. All of the pure tone audiometry and the Scan-3 tests are administered in typical rooms, not sound booths. The data is not being collected in the VA audiology service itself. We are finding that a non-audiologist can very quickly develop the necessary skills as the technician who sets the veterans up for audiometric testing. We actually had the non-audiologists videotaped while testing some of the participants, just to see how it would all work. The veteran participants read the instructions, and then follow through with the pure tone audiometry task. My role is to monitor quality of the data. We’re picking up some pure tone hearing asymmetries using the AMTAS asymmetry indicators, and also other indicators of the quality of the data. The AMTAS system does work in that kind of setting, and is providing data that simply - there would be no other way to get the data, because audiologists are not available for the research project… that would be prohibitive in terms of cost, and the VA audiologists don’t have the time.
RM: You know, the VA now has many community-based clinics.
JH: Right.
RM: They have many more of those now than they have hospitals. And those clinics for the most part have no audiology services.
JH: Right.
RM: And so we set up AMTAS systems in these community clinics around the Nashville area, your old hometown.
JH: That’s right.
RM: And then the clinic results were sent back to the hospital, so that it was a store and forward telehealth system. And it worked very well, and this is another example of getting hearing testing in places where it’s needed, but not necessarily in an audiologist’s office.
JH: Well, that solves so many problems. First of all, a veteran who may have transportation problems doesn’t have to go to the big center; plus it takes the pressure off the audiologist in the big medical center to perform other functions that they really need to. That’s a wonderful idea, and that - I hope that it’s expanded nationwide, because there are so many rural areas where the VA is a long distance off. Presumably most of these people just won’t get to the VA for their hearing assessment as they should.
RM: Right.
JH: With this study and with the assessment of any veterans, including those with mild TBI, that’s very useful. There are at least two clear advantages. One is the very systematic definition of asymmetry. While lecturing to audiologists and students about diagnostic audiology, I often ask: “Well, when do you refer a patient to an otolaryngologist with concerns that perhaps their problem might need medical or surgical treatment?” And of course, immediately people say “Oh, when there’s an asymmetric hearing loss.” But then of course, when you ask, well, what’s an asymmetry, you’ll get as many answers as there are people in the audience.
RM: Right.
JH: The AMTAS automated calculation of asymmetry alone was very attractive to me. It’s a systematic, documented definition for an asymmetry that’s then immediately displayed on the AMTAS screen, and printed on the audiogram report.
RM: Right. And that’s part of the – we call it the AMCLASS system, which is the automated audiogram classification system. When we were doing hearing evaluation, I’ll bet the first sentence in your report was, “The results indicate that the patient has a mild to moderate sensory-neural hearing loss, symmetrical sensory-neural hearing loss.”
JH: Yeah.
RM: Well, there are no standard definitions of any of those terms, mild to moderate, sensory-neural, symmetrical or asymmetrical. So we wanted to standardize those terms. And so you could, with symmetry for example, you could make up a perfectly reasonable definition of symmetry or asymmetry. You could say it has to be greater than 10 dB difference at two or more frequencies, and a thousand different reasonable definitions. Instead of doing that, our approach was to have a panel of experts, all of whom have been testing hearing and interpreting audiograms for many years, and ask them…
JH: They’re the judges.
RM: …what’s an asymmetrical hearing loss. So our definition of asymmetrical hearing loss is a hearing loss that the experts say is an asymmetrical hearing loss.
JH: Well, that makes sense because the system is supposed to be representing how an experienced audiologist would function, I mean, to take their thinking and put it into the algorithm.
RM: We establish our rules for the severity of the hearing loss – mild, moderate, severe, profound; for the configuration of the hearing loss – flat, sloping, rising, trough; for site of lesion – conductive, sensory-neural or mixed; and symmetry. And then we presented a whole lot of audiograms to our panel of experts, and we ask them, is this conductive or sensory-neural? Is this symmetrical or asymmetrical? We get the results from this panel of experts, and then we tailored our definitions to agree with the experts. So the definition of asymmetry is the one that has the highest agreement with what the experts say is an asymmetric hearing loss. It’s a little awkward when somebody says, “So what’s your definition of…?”
JM: Yeah, you can’t cite specific frequencies….
RM: Right. I have to tell them that an asymmetrical hearing loss is a hearing loss that the experts say is asymmetrical.
JH: But of course, audiologists can’t really argue with that, because they’re the experts.
RM: That’s right. So it’s validated against experts.
JH: The standard of care, and what audiologists would typically do.
RM: Right.
JH: You know, you hear of this complaint or concern – “Well, I don’t want to be replaced by this automated device.” I actually remember in the mid-1990s, as OAEs and OAE devices became more commonplace and introduced into clinics, audiologists were worried about being replaced. I guess my response to those concerns is: if you can be replaced by an audiometer, are you really providing valuable services? Simply performing pure tone audiometry, and even fitting hearing aids, is not what we’re all about. It’s about improving people’s quality of life, helping them to live independently, to communicate effectively, to enjoy this world that we live in – not to perform hearing tests – not even to diagnose hearing loss. It’s the ultimate outcome and benefit for the patient that we should be focusing on.
RM: I’m sure somewhere there’s an audiologist that makes their living doing audiograms all day long. That person might be replaced by AMTAS. It would be too bad for that person, but that’s not what our profession is aspiring to be. That’s not why we became a doctoral-level profession, to be able to do a technical test, a technical procedure that can be automated. It’s not what professionals should be doing.
JH: And no other profession that I can think of, no other health profession certainly in the United States, spends that much time on a simple technical task that could be done some other way. So why should we be an exception? We shouldn’t be.
Visit https://www.Grason-Stadler.com/ClosingTheGap to view the first part of the four-part interview series with Dr. James W. Hall III and Dr. Robert Margolis, and to register for access to the entire series.