In a recent post I argued that we behavior analysts are a contradictory bunch.
On the one hand, we say we can help people become happier, healthier, safer, and more productive. We proclaim that we hold the key to existential threats like climate change. We believe passionately that our science can save the world.
On the other hand, we seem to work actively to prevent that world from seeing the power of our science. We communicate about our work primarily to other behavior analysts. We invest much of our discipline’s intellectual capital in problems that affect a relative few. We conduct research using methods that many others find strange, and we talk about our science and its applications using vocabulary that other people don’t understand.
And somehow we are surprised when the world does not beat a path to our discipline’s door.
The goal of the present series of posts, “Greatest Hits of Dissemination Impact,” is to spur a discussion about what it takes to get behavior analysis in the public eye in a constructive way. The modus operandi of this series is, when possible, to accentuate the positive: While much of behavior analysis may remain in the shadows, some of our work does get noticed. If we want to understand what it takes to get noticed, we could make a worse start than by cataloguing what does get noticed.
Dissemination Impact
So, how to measure this kind of attention? When a scholarly article is mentioned in print by other scholars, we call that citation impact and treat it as a useful (if incomplete) measure of an article’s influence or importance. When a scholarly article is mentioned outside of scholarly sources, this suggests attention from non-scientists, that is, stakeholders in the everyday world where solutions are needed and, ideally, implemented. Such mentions have been proposed as a useful (if incomplete) measure of dissemination impact. A burgeoning literature tracking the dissemination impact of articles in various disciplines has emerged in recent years though so far this development appears to have been little noticed within behavior analysis.
Because different audiences are involved, dissemination impact is functionally distinct from citation impact. In my experience of examining behavior analysis articles, dissemination impact and citation impact correlate only weakly, which is consistent with findings in a number of other disciplines (e.g., see here and here). Therefore it is impossible to anticipate an article’s dissemination impact from metrics that normally interest scientists, such as the scholarly reputation or citation impact factor of the publishing journal.
Identifying Our Discipline’s Most-Noticed Articles
To determine which 2022 behavior analysis articles had the most dissemination impact, I used the Altmetric Explorer app, one of a suite of tools available from Altmetric.com, which tracks the mentions scholarly publications receive in a variety of sources other than scholarly citations. Some examples include news media, social media, Wikipedia pages, blogs, patent filings, and policy documents. By the way, there are other providers of altmetric data, but in my opinion the Altmetric.com database is the most comprehensive and easiest to work with.
On January 1, 2023, I searched for articles published between 1/1/22 and 12/31/22 in each of 15 journals that feature primarily behavior analysis content. A few other behavior analysis journals did not appear in the data base, and presumably received no relevant mentions.
For each article it tracks, the Altmetric.com database aggregates various types of mentions into an Altmetric Attention Score, which ranges upward from 1 depending on the number of mentions. You can do a deep dive into the database to see precisely what kinds of mentions an article receives, from what sources, and elsewhere (e.g., here and here) I have suggested some ways these details can be informative to behavior analysts. But for now let’s stick to the big picture by focusing on the 2022 articles that earned the highest Altmetric Attention Scores.
This table shows the 10 most-mentioned articles (for a really deep dive, download this list of the Top 100 articles). For context, note that many published articles (I estimate 30% to 40%) receive none of the mentions that comprise the Attention Score. Among the millions of articles that have received mentions, an Attention Score of 2 is the approximate median, and 10 is around the 75th percentile. The Top 10 behavior analysis articles all rank at the 95th percentile or above. What follows are three observations about what we might learn from articles like these.
Lesson 1: Focus on the Low-Hanging Fruit of Altmetric Mentions
The Top 10 articles got most of their attention through either news coverage or social media mentions. This was true for 89 of the top 118 articles, and overall in the Altmetric.com database news coverage and social media mentions are the two highest-prevalence types of mentions. They are also among the quickest mentions to appear after publication (typically, an article will receive half of its total attention in social media or the news within a few days or weeks of publication).
Other kinds of mentions can be important, but more elusive. For instance, a discipline concerned with saving the world probably ought to monitor its impact upon public policy — not much happens at scale without the cooperation of policy makers. The Altmetric.com database includes mentions of journal articles in policy documents, and I hope to talk about policy impact in a future post. For now note that this is a tricky topic, for at least two reasons: Policy mentions of an article can take many years after publication to appear, and only a tiny percentage of journal articles ever receive policy mentions.
My present point, therefore, is that if you want feedback on how much public attention your article is attracting, the place to start is by focusing on news reports and social media. For this reason, future posts will feature a deep dive into the attention profile of behavior analysis articles in these two domains.
Lesson 2: Article Format Can Matter
By my count, nearly 40% of the behavior analysis articles with the highest Attention Scores took the big-picture perspective of review, discussion, and conceptual articles (as opposed to the more fine-grained, detailed-focused approach of primary empirical reports). I don’t know exactly what percentage of behavior analysis articles overall are review/discussion/conceptual, but my impression is that it’s a lot less than 40%. For instance, these kinds of articles comprised about 15% of Volume 55 of Journal of Applied Behavior Analysis.
Thus it appears that “big-picture” articles are over-represented in the Top 100 of dissemination impact, and the reason why seems like common sense. As a teacher of smart, but not necessarily behavior-analysis-savvy undergraduates, I can attest that, for non-experts, our empirical papers are a difficult slog. They are written by experts, for experts, and they tend to do little to invite others in. From my days as an Associate Editor at Journal of the Experimental Analysis of Behavior (JEAB), for instance, I can remember publishing empirical papers that were intellectually rigorous, methodologically elegant, and theoretically groundbreaking — but of obvious interest to maybe 30 people in the whole world. Here I stress obvious interest, because the general importance of those papers was a forest lost amongst trees of specialized details.
By contrast, review/discussion/conceptual articles tend to aim for a broader audience. They often place greater emphasis on narrative, meaning a story line that motivates, holds attention, and recommends action with regard to the topic under discussion (for some behavioral analyses of narrative, see here, here, here, here, and here). This matters in light of who, other than behavior analysts, is most likely to make public reference to behavior analysis articles.
Consider the challenge of getting behavior analysis mentioned in news articles. It’s highly unlikely that science journalists are trained in behavior analysis per se, and in fact there is no guarantee that most journalists have good scientific training of any type. In this regard, gatekeepers of the news may have more in common with my smart undergraduates than with the authors of behavior analysis articles. It simply makes sense for them to gravitate toward what they can understand, and narrative pieces often explain better than rigidly-structured primary empirical reports.
Regarding social media, most mentions of scholarly articles come from “X,” the social media platform formerly known as Twitter. Based on my examination of the Altmetric.com database, it appears that the large majority of X users who mention scholarly articles self-identify as members of the public (rather than as scientists or professional practitioners). Consistent with this, only about 1/3 of X users have a college education. Once again, it’s logical that non-expert readers will gravitate toward what they can understand.
To be sure, primary empirical papers are good at what they are for, sharing technical information with specialists. The Top 100 articles may be telling us, however, that it’s unreasonable to ask such papers to serve as a vehicle for dissemination. What’s fascinating to me is that other kinds of papers may be better at dissemination than even their authors suspect. I imagine that most behavior analysis authors think of their review/discussion/conceptual articles as reaching other behavior analysts. But the Top 100 list suggests that at least some of these papers also succeed at (as I put it in a recent post) breaking out of our “behavior analysis ghetto” to make contact with the interests of everyday people. For those who are concerned with dissemination, this sounds like a word to the wise.
Lesson 3: Autism Only Takes Us So Far
Perhaps the most noteworthy aspect of the Top 100 list was foreshadowed in a previous post: There may be a disconnect between what behavior analysts like to emphasize and what people in the everyday world care about. The pie chart shows how often articles from each of 14 journals appeared in the Top 100 list (a few articles from a 15th journal, The Analysis of Verbal Behavior, received mentions, but none made the Top 100 list). A key to journal title abbreviations appears at the end of this post. As preface to the chart, it has been pointed out that an awful lot of behavior analysis has aimed at helping persons with autism and developmental disabilities — important matters to be sure, but of direct concern to a small percentage of the population. What I find striking in the chart is that two of the top three most represented journals, Journal of Contextual Behavioral Science and JEAB, have little to do with autism and developmental disabilities. Only 4 of 116 contain the word “autism” in their titles. Many of the articles focus clearly on something other than autism (e.g., cancer survivors, adolescent stress, compassion, brain injury), while others focus on topics that are not unique to autism (therapy animals, telehealth, research ethics, reinforcer assessment).
Skinner taught us that organisms are always right — for instance, if a reinforcer assessment shows that a child likes red Skittles candies but not green ones, we don’t persist in forcing the green ones on her. I think the organisms whose behaviors shape Altmetric Attention Scores are telling us something important: They have limited enthusiasm for autism. Yet, in effect, many of our journals persist in forcing a lot of autism on them.
Another thing that Skinner taught us is that when something interesting comes along, an astute individual drops everything else to pursue it. Well, here’s something interesting: Attention Scores suggest that when our work corresponds to the values and interests of everyday people, and when it is explained in a way those people can understand, they may reward us with their attention. For behavior analysts who hope for the world to embrace the solutions they create, that sounds like a pretty critical preliminary achievement.
Key to Journals in Pie Chart:
BAP = Behavior Analysis in Practice
Beh Int = Behavioral Interventions
Beh Mod = Behavior Modification
BSI = Behavior and Social Issues
EJOBA = European Journal of Behavior Analysis
ETC = Education and Treatment of Children
JABA = journal of Applied Behavior Analysis
JCBS = Journal of Contextual Behavioral Science
JEAB = Journal of the Experimental Analysis of Behavior
JOBE = Journal of Behavioral Education
JOBM = Journal of Organizational Behavior Management
JPBI = Journal of Positive Behavioral Interventions
POBS = Perspectives on Behavior Science
TPR = The Psychological Record