Again, for those who are not familiar with the term, Press-Ganey, it is a survey that administrators of hospitals use as gospel for who is the best hospital and is all about customer satisfaction.
But there are flaws in this Press-Ganey. What are the flaws? Oh, too many to list here (let me count the ways...). I'll just hit the highlights:
- First is that Press-Ganey has a complete monopoly. They are accountable to no one, despite their less-than-scientific method of procuring data. If I practiced medicine in the same "Evidenced Based" manner that the Press-Ganey does, I'd lose my license.
- While the (from now on abbreviation PG will be used) PG is supposed to take a random sample of patients a physician sees, it is biased in the sense that the surveys are voluntary and only motivated and healthy-enough to fill out a survey in the first place.
- Plus, take the ER for example. My "random" sample is based on patients who are discharged from the hospital only. No admitted patients fill these out. As an "Emergency" physician, the admitted patients represent the true emergencies. I'm not saying all discharged patients aren't supposed to be there, but I'd call them easily addressed "Urgencies."
- Worse, the odds are that the more often a patient shows up to the ER, the more likely they are to eventually get a survey. Do I need to spell out what kinds of patients show up to the ER frequently?
- I am graded by the patients that I had to spend the least amount of time with since I focus my attention on the true Emergencies first and foremost.
- And while the PG should have the same standards for every hospital - since they compare every hospital and form a grade for each hospital/provider, they don't. Each hospital interprets the selection of patients differently - I know bc in my career I've already worked at five different hospitals. One hospital excluded patients with "drug seeking behavior," whereas my current hospital does not.
- The grades are on a curve. In other words, hypothetically, if you think that on a five point scale 5 is excellent and 3 is average, and all doctors surveyed had an average of four, that would mean all docs are above average. But all docs don't get a PG score of above average. They can't since it is done on a curve. You could give above average care and get a score in the bottom 25% (an "F" if you were in school). WHAT??? Yes. That is true. And that is exactly what happens. Good doctors are getting penalized even when the satifaction scores are above average.
I started a new job in January, and from the get-go had great PG scores. Of course, the sampling was low, so nothing was statistically significant. Once I got one unhappy patient, it killed my score. So 4 excellent scores and 1 bad one = Very bad PG score. But despite that, my scores were still in the top 80%. And stayed that way until June.
All of a sudden, in June my scores dropped to 1%. I naturally assumed I must have gotten a bunch of really bad reviews. Bad luck? Bad day for me? I didn't know. We have a binder that has the recent PG reviews for the department, and looking in there, all the reviews were 5's. The highest. I didn't get it.
The inevitable happened. My administrator saw my scores and insisted I be rehabilitated. My medical director had a "talk" with me. But to his credit, he promised to pull these surveys so that I could find out what was said so I might know what to improve. And he assigned someone to the four hour, painstaking task, of searching for all of the surveys and matching it up with the charts. And you know what they found (drumroll.......)?
These charts didn't belong to me! They were the PA's patient, or another physician (obvious when the pt is criticizing the tall male MD when I am a petite female doc). In fact my scores were almost exclusively 5's. Putting me at the top of the scale, not the bottom. Yet I still get that scarlet letter score pasted up all over the ED that makes me look like a mean uncaring doctor.
My director apologized to me, but it is not his fault. It is the fault of the administrators who are so desperate to have some standard with which to grade themselves, they choose not to use scientific rigor to evaluate that very imperfect standard that they use. And we doctors are the ones who suffer.
Posts
I also cringe and HATE the P-G word as a nurse. They focus on them and they reflect nothing accurate. I was recently an ICU patient and got to fill one out, it was silly had no reflection on the true quality or lack of care I had. So I hate them more now.
ReplyDeleteGreat story and expose of the BS that physicians have to maneuver around to practice medicine. Like you, I have found that these surveys seem to find the most unhappy patients, or at least those patients are the ones willing to fill them out. Nothing truly statistical or scientific about them but the admin types love them.
ReplyDeleteThese are like internet reviews bcs they tend to be a self selective group. Accordingly, people with intense feelings give ratings which are either 1-2 or 4-5 on the 5 scale. They obviously are not science based but everybody like to "profile" with number scores, FICO, P/G, CLUE, etc.
ReplyDeleteNice post - very different take than I am used to. I'm interested in your reaction to the counter points those of us non-clinicians often tout:
ReplyDeletecolor me an misanthropic administrator, but is there *anything* to be learned from patient satisfaction scores? I'm willing to grant that their survey practices may be far from perfect, and will add that most people don't think an ER experience is pleasurable, regardless of outcome. But, don't patient sat scores have some value in the big picture of how patients feel about their treatment? Hospitals the employ Studer, or the Ritz think so, and there is strong evidence to correlate higher scores to higher market share.
In regards to not surveying admitted patients (CMS level 5s and 6s) - perhaps that makes sense, those admitted patients are there for life saving treatment. I'm willing to guess that many are not even conscious during their time in the ER. So isn't taking a survey of the less emergent cases a more accurate representation of what basically amounts to customer service? In other words, no one cares about how they are treated with the paddles come out, the do care when its a stubbed toe or ear ache, and those cases matter too.
Finally, and just playing devils advocate here, what about the other organizations doing similar work - PRC and Rand come to mind. Perhaps you could persuade your administration to try another sampling agency and compare the results.
"Hospitals the employ Studer, or the Ritz think so, and there is strong evidence to correlate higher scores to higher market share."
DeleteThis must be the administrator in you coming out. Market share has nothing to do with good care, the drug dealer on the corner has good market share too! We have a patient who has over a hundred ED visits in two years. I'm sure he's taken at least a few surveys. Do you really think someone who comes in for drugs that many times is representative of the average patient experience?
When I received PG surveys after my various hospitalizations in 2007, it was glaringly obvious how incompetently the surveys were administered.
ReplyDeleteFor one thing, the surveys arrived weeks later. I was in and out of the hospital week by week and each one had no indication of which hospitalization I was to review.
For another, it was so far after the event that all my memories were gone.
When a competent professional is truly interested in harvesting truly useful information they follow proven principles like ensuring the responder knows what they're responding to, and capturing the information right in the moment. Real-time feedback is infinitely more valuable than after-the-fact mailings. (And that's not to mention the issues you cite.)
To me this whole subject exemplifies how the healthcare industry is pretty much in its infancy when it comes to understanding data. For data to be worth anything, you have to ensure that it arose in a way that accurately represents what you're trying to measure.
This final remark by e-patient dave bears repeating: "For data to be worth anything, you have to ensure that it arose in a way that accurately represents what you're trying to measure."
ReplyDeleteRe: devils advocate nick dawson, Admitted patients are levels 4,5 and critical care. Most of my admitted patients are awake and fully conscious during their stay. At some point they often receive drugs to ease their pain. Certainly, they can assess all the aspects of a visit. Do you want to exclude pts who were intubated during their stay? Sure I have no problem with that. Otherwise the admitted patients are definitely the better determinant of what I do.
To assess customer service, first you have to define "who is my customer?" For me in the ER, my customer is the emergency. Anyone else is an urgency and can wait. Do I have a responsibility to be cordial to the urgent patients? Sure. And for that reason, I think ER docs should have some discharged patients in the sampling. But it shouldn't be the sole determinant of the grade.
I don't know about other sampling agencies bc all my hospitals have ever used was the PG. I can't address those others.
But as an administrator, you need to recognize (and please see my previous blogs about the PG) that an overuse of patient satisfaction erodes the physician ability to look beyond what a patient wants to give them (or deny them) what they need. Patients who come in looking for an antibiotic prescription when they don't need it, and don't get one won't be happy. But sometimes I need to do that not just for the patient, but for public health at large. Patients don't always see the big picture, and even the nicest, most charming doc will have a poor score if the pt waited 4 hours in the waiting room to leave "empty-handed."
good and fair response - thanks for the insight. I see your point about the emergency being the customer. I do believe that an ER should meet all needs for all people, but understand that A) its not a widely shared view and B) regardless, the emergency comes before urgency and even mundane. Maybe we are measuring the ER incorrectly.
ReplyDeleteWould measuring outcomes alone be an appropriate metric?
Re: Nick Dawson thank you for the thoughtful discussion.
ReplyDeleteThe problem with pure outcomes metrics is that they depend on self-reporting of hospitals on these issues - and that is variable. Perhaps if an independent company or the government did random sampling of patients to see outcomes it would be more accurate.
If you will refer to my prev blog "How to Choose a Hospital" I think that more than just outcomes data should be used to select a hospital. I believe that patients should have other factors as mentioned there: Bedsores developed in hospital? Nosocomial infection rate. Nurse-to-patient ratio. Number of patients boarded in the ER. Etc...Plus, I would be hesitant to use data about cardiac arrest/codes in hosp as some hosp have sicker pt populations than others - not sure how to standardize. But I do think it is fair to judge all hospitals with 24 hour cardiac catheterizations on their cardiac outcome data, since they are more fairly standardized.
Regarding the hospital as a whole, and the ER as a unit within that, I think that the most important measure of a hospital's efficiency and management is how many patients present to the ER vs. How many get seen (& don't LWOS). In other words how fast can pts get in, get out, or get up to a bed. Ironically, this usually has less to do with the doc as with the whole system.
For instance, Radiologist delays in reading scans make patients stay longer, as do delays in drawing blood, picking up blood, running the tests, getting the urine out of the patient, and getting a bed to be free and available and getting that pt up to that bed. Also, requirements that ER docs do full contrast CTs on r/o Appy creates huge unnecessary delays as all the studies show you don't need full contrast for all of that. Most radiology departments require it (3-5 hour delay right there). How many nurses for each pt. ER beds are partly an issue, but how you use those beds is more relevant. Many efficieny studies show that having one dedicated tech per nurse improves nurse efficiency and costs less than hiring another nurse.
So if the ER efficiency is failing for throughput, and LWOS, then it is less an indicator of the ER doc function and more an indicator of the failure of the overall hospital system to allow proper function.
A measure of the individual doc is more a measure of how many patients per hour a doc sees compared with overall throughput times. If throughput is slow and doc sees few pts, not docs fault. But if throughput is good and doc doesn't see many patients, doc is too slow. And if doc sees >2.5 pts per hour, doc is seeing too many pts and ER is understaffed.
Based on some prodding by @arthurwlane here are some of my suggestions to fix the Press-Ganey (excerpted from twitter):
ReplyDeleteBasically do the opposite of what is done: standardize the process. Do not grade on a curve. Increase study power w/more pts surveyed...make it so PG can't grade u until you have enough surveys to be statistically significant. Do it like a scientist would do study.
For ER docs, include inpatient and d/c'd pts 50/50. And admin should do random samplng of docs surveys to see if assigned correctly. Also, admin should make sure to show docs all the surveys good & bad so docs have opportunity to learn...exclude pts with 2 visit avg/mo.
On the Press-Ganey: For docs w/bad review should be a question "Is there anything that MD could change that would make experience better for you next time?" This would reveal much more info to the Doc and administration.
I also think, the way to choose a hospital is not by the PG scores (do pts even know what those are?) but as was in my blog "How to choose a hospital" the REAL reasons to pick a hospital. Pts who get good care tend to be satisfied. http://bit.ly/GaLgj
As a note, I work for Press Ganey. I can tell you clearly that their focus is on money and not on improvement. This company works on a 40% gross margin on all products. The earned over $90 million last year alone on the mailing of surveys. It is hard for an ED to EVER get enough back to really be able to have valid and reliable data. National response rate averages for EDs is about 10%. Realistically, in most areas, it runs 6-8%. Most facilities simply cannot afford to get a valid sample.
ReplyDeleteSomething to keep in mind when looking at the data. Benchmarking is effective to see where you stand on the grading curve. On the other hand, it is something you cannot control. Your percentile ranking falls into the "area of concern". Your mean score falls into the "area of control". You can only work on your own facility, you cannot effect the outcomes of another. Administration needs to understand this basic fact and tie performance to that which can be controlled and effected. It is demoralizing to staff when they do not make their percentile rank goal. Take two facilities and say one has more buy in at the top and more resources to make things happen. Both have the save level of committed staff members. Logically, the greater resource facility will improve faster with all else remaining equal. Hence, the other facility see a lowered percentile ranking in comparison, even with the same amount of effort. Be aware of the benchmark, but focus on you!
Let's be completely honest here, ALL survey companies fall into this boat. They have used statistical manipulation to pad their pockets and hold healthcare hostage. In the end, data is data. This is another commodity product. With Press Ganey you pay for a 23 year old college grad in South Bend that has little to no healthcare experience. What they provide cannot be justified at the price point you pay. For a hospital that has in excess of 100 beds the ED contract price is over $8000. Want to get the patient's comment? Add another $4100+ As far as mailing, to get the bare minimum for a report, add another $15-$18,000. Suppose you had 10 ED docs that you wanted breakout data. Hold onto your seats, but it would cost over $54,000 just for the mailed surveys! Over $66,000 a year just for data. Do yourself a favor, hire another nurse and find a cheaper provider.
This company is changing by the day and its reputations is far from what it was. You can do better and you all deserve better!
Anonymous, thank you for speaking out. The problem is that my CEO goes to a Hosp admin conference and P-G is all the rage. How do you convince otherwise? Personally I think the government should ban P-G type flawed surveys as it leads to the increased costs that they are desperately trying to tamp down.
ReplyDeleteI think I would start from a point that all administrators understand, that being profitability and ROI. Take a look at the average facility that has 3-4 service lines with PG. They will be spending in the $65-$100K range depending on their sampling percentage. Look at that expenditure in terms of value. You get to run endless reports on eCompass which administration will never use sans the willingness to recommend question. You are assigned a so-called "consultant". The demographic for these is 23-28 year old college grads who have no healthcare experience outside of PG. Over 70% of them have never stepped foot into a healthcare setting. Their consulting is limited to what they are fed by PG's statistical experts. None of these consultants are active in ACHE or have even picked up a copy of Modern Healthcare, H&HN, or any other trade journal. Are your response rates low? They will suggest using posters to make patients aware of the survey. They will tell you to implement scripting but have no idea how to make this practice effective. They have no idea how to relate nursing issues and limitations to the ancillary services that are also effecting the results. If you spend in excess of $100K, you get to have one of these consultants on site for a day or two per year to give a canned powerpoint presentation. What value is that providing? Take a look at your own "sales contact" who has a background in running restaurants and his site visits (rare!) are consistently rated by his clients as one of the lowest in the entire company. Yep, those surveys effect PG employees as well. I would direct your administrator to this year's PG National Client Conference. Represented will be their highest level of consulting. If you are a big enough facility, you may have the chance to have this individual come and assist. His background? Well, after a year or so of doing the sports reporting for a small station in WY, he moved on to bigger things like folding sweaters at the mall in South Bend, IN. Now, considered one of their most effective consultants. Amazing that in one year he has earned those qualifications. Of the 600+ employees, less than 1% have a clinical background.
ReplyDeleteAs an ED doc, you have challenges. When has a PG employee ever been on site and walked your ED? Has anyone ever given you any meaningful insight on reducing boarded patients? Anything about the need for centralized transport or a dedicated bed manager to ensure patients are directed to the right nursing areas? How about ways to facilitate hand offs and shift change? My guess is no. A big benchmark is nice, but what does it really mean if you do not get the tools to succeed. The ED is the front door to the hospital. It deserves a priority. A less expensive provider would free up funds to bring in some actual consulting that could really do something in the eyes of the patients. PG will never be that. If you really want to discredit them, facilitate a presentation for administration on their Patient Flow product. PG is not sold one of these yet and would be happy to come on site. They will then tell you about the mystical power of queuing technology and how streamlining your OR (yes! OR, not ED) will eliminate you patient flow issues. How will this be accomplished, by introducing blocked schedules for the surgeons. The cost for this amazing product, high six to seven figures! That alone should shed some light on what these people are really about.
Lead with the costs, the facts, the value, and give PG enough rope to hang themselves. Best of luck!
I also work ED and get an occasional not regarding my scores being low basesd on a sample size of 2. I am wondering, can I run a CL add to buy unsigned PG reports from people and fill them out and send them in? Rather than giving meal tickets, movie passes or free cab rides I want to just bypass the middleman. Its a flawed system, whynot use a flawed way to give admin what they want?
ReplyDeleteAnonymous: I understand your frustrations, but poor professional ethics on the part of the doctor is certainly a fireable offense, and could possibly be illegal in this case. And really, I expect more from physicians ethics-wise.
ReplyDeleteif the info is flawed and not corrected. what duty exists to make sure the data are correct before using it to determine reimbursements or to make it public. i see a liability issue here
ReplyDeleteWell, good for all of you PG is no longer the biggest thorn in your side. CMS and HCAHPS are. The HCAHPS rating & comparrison data process is equally as flawed. I hope that you all over the last couple of years have used your collective energies to petition the HCAHPS process (which is now spreading to Outpatient, Home Health and Private Practice). HCAHPS is mandatory and for some reason physicians and clinicians have let that process slip right by them and now we are all stuck with that survey.
ReplyDeleteAlso let me add, I've been working in the are of change management and organizational development in healthcare for 20 years and in 5 different organizations. Regardless of the sampling methodology, the end results of the patient perception reported via survey has typically been an accurate depiction of the experience and quality of care that the hospital provided. There was no denying where our problem areas were. But for some reason, we'd rather challenge the data just so that we can avoid improving.
ReplyDeleteFor the those physicians particularly interested in their patient's feedback, why don't they call inpatients/dispo'd patients and ask them about their experience. I guarantee that the results in general will be the same (if not worse) than what was stratified via the Press Ganey (or any other vendor) surveys.
~Anonymous 1140 (I am the same writer of the anonymous post directly above this)
For completely unscientific data such as the PG, there is still some value - that is for overall trends. Basically, the PG is like a poll. The trends are useful, but the day to day movements are just noise and subject to a million things that can alter them one way or the other. But if the trend is in one direction or another, it is useful for showing changes in processes, such as adding EMR, doc in triage, etc...See how the PG trend line is before and after.
ReplyDeleteWhat is infuriating to me is having impossibly small sample sizes of biased patients (unhappy ones tend to report, and frequent flyers of people who misuse the ER would be more likely to fill these out) and have numbers that vary so that one month you are at 92% and next month at 20% and somehow be expected to find anything that would be improved. If a docs numbers are consistently in the 20s when others are consistently in the 80s, then there is certainly something that can be learned. However, the administrators like giving you numbers but not data. What did the patients say? What did their chart say? Were there any special circumstances? How can this instance be a learning experience for this and other doctors/nurses? Numbers are not enough. And relying on unscientific numbers as a basis for so many things, is at best unfair, and at worst abhorrent.
I work in inpatient psychiatry and our patients fill out the surveys before D/C, whereas medical patients receive surveys by mail. I have seen staff "helping" patients fill out their surveys (per management directions). To me this is coercive, especially for patients who may be suspicious that D/C could be deferred (yet again!) if they are less than complimentary of staff.
ReplyDeleteIn addition, the surveys make no effort to discover whether patients were court-ordered admissions or voluntary. I'm guessing that people who are locked in the hospital against their will may find the "amenities" unsatisfactory (regardless of staff efforts). I imagine that if it was my perception that I was imprisoned, I'd have little positive to say.
Where I work you don't dare question the utilty of P-G.
It is sad to see the health care INDUSTRY become ever more business driven, and less
about improving human health.
My group now has a huge bonus tied to our patient sat scores. and what do you know: our independent review of the surveys show that some patients received surveys for doctors they never saw, some patients receive more than one survey, some patients received surveys for the doctor that did not discharge them(as they were supposed to) Only 25% of the surveys were properly assigned. We are willing to seek legal counsel now to move forward and recoup monies potentially owed to us unless we get satisfactory reimbursement from administration. Press Ganey is the biggest scam since Bernie Madoff.
ReplyDeleteI went through thirteen years of rigourous medical training. I have been in practice for 12 years. I have been board certified and recertified for four times each.I should look like a stalwart right!!!
ReplyDelete1)The CMS and hospital coders teach me how to dictate
2)The hospital tells me based on evidence based what antibiotics i should administer the patients
3)Pharmacy wants to decide wehn patients swithc from IV t o po
4) Press ganey and its hospitalrepresentative tells me how to behave with the patient, how to charm them with my answers, how much time I need to spend etc, etc.They do not have any clue about the patient flow
5) Then comes EHR where you are basically a typist.
EHR and press ganey are scams, trust me they are. They are enterprises out to suck money from the heatlh care.
I told my hospital administrators that they need to getoff their AC offices, wander aroung hospital they will collect more realistic data, rather than send it to a remote office who do not have any clue.Use the money to hire nurses or other option related to direct patient care.Now I know why americans hate big government. Dont blame them
Have to make this anonymous, don't want any repercussions. We had a survey come back as a 1 percentile for our unit. We decided to call the person who left the all 1 responses. Guess what? It was not the patient listed. The person who answered the phone said that he had filled it out for his companion. Turns out the person was in the ER and then transferred to another hospital, never even got to our unit. Then when we found the right number for the patient who was actually on our floor, he said that he actually had nothing but high praise for the care that he received. he wants Press Ganey to change his scores to reflect the high grades. We hope that they will at least delete the all 1's, and also hope that they will put all 5's for this guy. So frustrating as we are evaluated based on these scores. We wind up having to police the PG results.
ReplyDeleteAs an employee I hate the PG we are required to do on management. Supposedly anonymous. Small small dept and my boss then wants us all to discuss her low scores. Very uncomfortable.
ReplyDeleteIlene, you're hell on wheels. Just discovered your blog!
ReplyDeleteSteve Pitts (remember me?) It's wonderful that you can get not just grumpy ER docs but also anonymous PG employees and hospital admin folks in the same discussion group! One thing I've always wondered: How much does PG pay Thomas Lee, their chief med officer? Why would a smart and influential guy like him take that kind of job?
Steve! Hi. I do remember of course. Thank you for your kind words. I am amazed at how many people find this post. As for your question, I do not know exactly how much he is paid, but it is quite a lot. He might just have sold out, I guess. If you, the most scientific EBM evaluator I know, ran PG, I know it would be legit science. You of all people understand what it takes to create valuable statistics. If Thomas Lee knows, he lets his underlings run amok.
DeleteThis comment has been removed by the author.
ReplyDeleteAs a patient and a nurse, I can say that these surveys are not being administered correctly. After scheduled surgery, coma, icu, and a discharge to a re-hab facility, I received a survey in the mail. An entire month AFTER being unconscious for most of my stay. I'm alive, that is all I can say. Can I rate my quality of care? No. I was completely drugged to the gills (amen). My survey would be totally worthless if I bothered to fill it out. And I must say, that nowhere on the form does it express the exigency of a reply.
ReplyDeleteSame thing happened to me. I have been at my job for 9 months. Shortly after I started my percentile was low, as the sampling was low and I'm sure there were one or two bad surveys from drug seekers. The more surveys I got, the better my percentile became. About a month ago, I got recognized for being in the 99th percentile and got a customer service award. This week I went to work and was given my spreadsheet and all my numbers were in the red, with one individualized question being in the 1st percentile. I'm sorry, I call bullshit. There is NO way I went from the 99th overall to the 25th in two weeks. I have about 30 surveys and there is NOOO way that I have that many horrible ones. I only know of one or two. I've seen most of my surveys and they are usually all 5's. My one or two bad ones I'm sure are drug seekers, but they can't all be drug seekers! I'm blaming it on administration. I'm trying to get an answer-still working on it. I will be as relentless at this as PG is at crushing my spirit.
ReplyDeleteI received a questionnaire from Press-Ganey just now. I created a Facebook post inviting friends to send lots of made-up responses. I also sent an image of the Facebook post with the responses I received directly to Press-Ganey, and also to the medical organization that apparently hired them. This link is to the image of my Facebook Post:
ReplyDeletehttps://drive.google.com/open?id=0B7pM9PGzaXuuelpaOWJBRXpCSGM
This comment has been removed by a blog administrator.
ReplyDelete