On Ethics and Integrity in Cognitive Interview Practice

On Ethics and Integrity in Cognitive Interview Practice
Paul BeattyNational Center for Health StatisticsUSA
Questionnaire evaluation as benign enterprise
Even an incomplete evaluation effort offers more benefit than noneRisk to participants seem minimal, especially when advance knowledge of the topic is givenInstitutional Review Boards (IRBs)– in the U.S. at least, a constant presence reviewing all procedures to minimize risks to participants (with enforcement power)
Collaboration with third parties
One of the key reasons for collaborating with outside institutions is that they can put us in touch with difficult-to-reach participantsDry cleaners—generally suspicious of “health researchers”; contact through an association was more effectiveBlood donors– especially those with specific donating patterns or historiesHIV-positives– some study requirements called for anonymity requiring third-party help
When collaborators helpwith recruiting
Many collaborators are quite helpful, but some could be using such opportunities to push through an agendaDry cleaners saw us a potential source of regulation rather than a benign research enterpriseAfter many questions, offered hands-on assistance– resulting participants might have reflected skewed experiences (e.g. very clean operations)Attempts at influencing our respondent pool would have reflected misunderstanding of our goals, but could have affected our outcomes
Other risks from recruitment with collaborator assistance
Questionnaire testing with blood donors; required assistance from blood centers to identify people with specific donating patternsThe process:We identified characteristics of interestBlood center had direct contact with participant and set up appointment time at their centerWe have no specific info from center records, and the center gets no data from our interviewThey get a copy of our report, with no individuals identified
Potential loss of confidentiality
Collaborator was alarmed to find in our report that one participant may have been donating blood inappropriatelyThey asked us to identify or contact this person (not allowed by our IRB)Although the individual was not identifiable to the public, she might have been identifiable to the collaboratorUltimately contact became unnecessaryBut, this was seriously alarming; further actions could have constituted a breach in confidentiality
Collaborators with a strong stake in the outcome of cognitive testing
Evaluation of screening questionnaires (e.g. for blood donors)– results used to:Help refine the instrumentsWhen positive, results can be part of a case for approval to regulating agencies that approve these instrumentsWhen collaborators are full partners (but respect our independence) this can be productive and usefulBut on one occasion with a different sponsor, we discovered problems with the instrumentThe collaborator did not agree with findings and considered them potentially damagingThey took several steps to influence the report content
Objectionable content
Several examples of participants’ dishonest reporting about screenable behaviorsTranscriptions of interview segments, offering evidence and explanation:The money was useful!Some participants did not respect the relevance of questions about certain risk behaviors, and didn’t feel obligated to answer them honestly“I didn’t catch anything in jail.”
Collaborator reaction
Clearly NOT what they expectedArgument: this is not relevant to comprehension, which is what we asked you to exploreThe sponsor:Re-wrote sections of the report and asked us to accept these as alternatives.Asked us to delete discussion of donor motivation…or at least, severely cut it back with various caveats and qualificationsAsked us to delete other details deemed irrelevant
Our concerns
This actually speaks directly to matters of comprehension and interpretationMisunderstanding of intent, purpose, and need of informationNo need to admit sex with someone from Africa– “not everyone in Africa has AIDS”Ownership and control: This is not a work for hire. Even though sponsor paid for some of the testing, we work independently and in the public interestWe reserve the right to determine which findings are relevant, although we are open to changes and deletions based upon methodological discussion
Negotiation over content
We did not accept their verbatim re-writes of our reportWe agreed to replace transcriptions with summary statements…with some reservationsWe did generally add caveats about limitations of our findings as suggested, within reason (e.g. allowing that our participants could be non-representative)We modified language to reduce speculative statements (“a lot of verbiage…could be tuned out by less attentive donors”)Generally resisted requests to drop sections or findings that were considered “not relevant” (e.g. the prison employee)
Concerns for the future
If they had known then what they know now, could they have manipulated the outcome?They had considerable control over recruiting–undoubtedly, they could have fed participants to us leading to certain conclusionsWe must be wary of that possibility when using outside collaborators who have a stake in our findings
Other considerations:emotional content
Perhaps the most pervasive concern; how serious of an issue is it?On the one hand, no greater risk than everyday conversationOn the other, we go intoincreaseddepth of discussion over typical surveys, while oftenretainingsurvey detachment. Can that exacerbate discomfort?
Why do people agree to talk about these issues with us
Some serious issues covered: severe disability, major oral health problems, blindness and other irreversible lossesReasons for participation“I hope it helps someone”Motivation for themselves to take actionSeeking information, or sometimes explicitly seeking help
Creating new respondent concerns?
Testing attitudinal questions:Do you believe that immunizationsalways, often, sometimes, rarely or neverhave serious side effects?Do you believe that immunizationsalways, often, sometimes, rarely or neverare dangerous?The possibility had never occurred to many participants. They wanted to know if this was a real possibility.Could it have affected behavior?
In conclusion
Collaborations have benefits and risks, especially when involved in recruiting for studies on potentially sensitive issuesWorthwhile to find out what sponsors expect to get out of testing in advance, and issues of ownership/controlConsider respondent motivations and expectations, and potential impact of questions on their well-being