In early 2023, I read the article “Gender bias in postgraduate year one pharmacy letters of recommendation” published by in the Journal of the American College of Clinical Pharmacy. The analysis contained methodological flaws, and it presented conclusions that were not derived from the evidence presented. Over the ensuing weeks, I returned several times to the printed manuscript, which I’d shoved in the back of my locker at the hospital. I wanted to throw it away and forget about it, but I could not bring myself to do so. Each time I opened my locker, I was met with an uneasy feeling that, although its scope and application were narrowly applied to letters of recommendation, the publication of such a manuscript in the journal of my professional society might have broader implications.

 

In a 1941 essay, C.S. Lewis coined the term Bulvarism to describe a method of fallacious reasoning he felt was so commonly employed that it deserved a biography of its imaginary inventor Ezekiel Bulver, a 5-year-old who realized “the great truth that refutation is no necessary part of argument. Assume that your opponent is wrong, and the world will be at your feet. Attempt to prove that he is wrong or (worse still) try to find out whether he is wrong or right, and the national dynamism of our age will thrust you to the wall. That is how Bulver became one of the makers of the Twentieth Century.” Those familiar with logical fallacies will recognize this as a sophisticated variation of the ad hominem.

 

Although Lewis’ essay contained the subtitle “The Foundation of 20th Century Thought,” Bulvarism is alive and well in the 21st century. Efforts that claim to use the scientific method to identify unconscious bias seem to be a particularly prominent example. Consider the methods employed by Sahlstrom et al to investigate gender bias. The analysis relied on word counting software to categorize and compare word choice within 7,529 letters of recommendation written on behalf of 2,383 applicants (of which 71.5 percent were female). Words in the domains of positive emotion, social process and prosocial behavior were observed more often in letters for female applicants. Female applicants were more likely (23.2 percent vs. 18.9 percent, p = 0.027) to receive an interview offer, but gender was not associated with interview offer in a regression model. Data on applicant matching and performance during or after residency were not included.

 

What implications can be drawn about the presence of unconscious bias from these data points? The answer provided in the article is a need for “conversations regarding implicit biases, including gender, ethnic, and racial biases, in order to promote equitable PGY1 pharmacy recruitment processes.” The truth is that no valid conclusions about unconscious bias can be made using the data, and the attempt to do so is an example of Bulvarism.

 

Lewis offers a simple thought experiment that illustrates the central problem. “Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is ‘wishful thinking.’ You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself.” The point according to Lewis is that “you must show that [emphasis original] a man is wrong before you start explaining why he is wrong.”

 

At the end, the article attempts to draw conclusions about the presence of bias [a clever rebrand of “wishful thinking”] in the author’s assessment and description of the candidate. Stated clearly, the claim of gender bias in a letter of recommendation is a claim that the author has wrongly presented information about a candidate due to prejudiced beliefs about members of that gender. However, the only way to determine if the author’s assessment of a candidate was wrong would be to work through the sum of all the merits of a particular candidate’s performance and personality as actually observed by that author. No matter how robust it may be, an analysis of the aggregate differences in word choice provides no information about the accuracy of an individual author’s assessment. Instead of acknowledging this fact, the authors simply assume that the language used on behalf of male or female applicants should be the same. The possibility that letter writers could in fact be responding to actual differences between the temperament, work habits and sociability of men and women has been ruled out a priori without mention or justification.

 

Importantly, nothing in the arguments presented here should be taken as a claim that individuals do not have the capacity for unconscious bias or commit the error of making incorrect inferences based on prior experience leading to unfair treatment of individual applicants. Observation of a fallacy in an argument does not prove its negation true. The presence of discriminatory treatment of individual applicants remains an open question subject to investigation by alternative methods. Ultimately, the problem with Bulvarism in unconscious bias research is that it presents a grossly over-simplified analysis of a complex problem possibly leading to well-meaning but incorrect policy prescriptions that fail to achieve their intended effect.

 

What is the appropriate response to Bulvarism in peer-reviewed journals? Lewis reminds us that “you can only find out the rights and wrongs by reasoning—never by being rude about your opponent’s psychology.” I suggest CMDA members be aware of attempts to Bulvarize, so we can avoid this pitfall ourselves and be able to engage in respectful correction when it occurs, especially in print. This may mean being willing to speak up during conversations and conference proceedings, write a letter to the editor or volunteer as a peer reviewer.

 

In my case, the editor was willing to publish my critique of the article alongside a response from the authors. I subsequently identified a total of 13 reports relying on the same linguistic model to draw conclusions about unconscious bias across a variety of medical specialties, including six in The Journal of Surgical Education. If a CMDA member wants to collaborate on another letter or editorial on the subject I would be interested.

 

Take Home Points

1. Choose a journal relevant to your professional affiliations and commit to read regularly.
2. Be alert to common flaws and fallacies.
3. Discuss your observations with your peers and trainees.
4. Write a letter to the editor.
5. Sign up as a peer reviewer.

 

Warning: Being salt and light requires humility. Hold your own work to higher standards than what you are reviewing. Ask colleagues whom you suspect might think differently with you on the issue to review what you have written. Sending a draft to four peers across the ideological spectrum identified flaws in my argument and started lasting conversations about intellectual integrity and academic freedom with all three individuals who responded.

 

---

About the Author

Stephen Perona, PharmD, BCEMP, BCPS, is a CMDA member and critical care pharmacist in Tucson, Arizona.