Author note: If you have thoughts on this topic that you’d like to share with readers, let me know (tscritc@ilstu.edu) and I’ll get you pointed in the right direction.
The figure reproduced below, from the website of the Wing Institute for Evidence-Based Education, compares the effects on student progress of regular skills assessment versus a bunch of politically-popular educational interventions. Spoiler alert: The formative assessment strategy, which mirrors the practice-plus-feedback strategy of just about all behavioral instruction, dominates. But what also becomes clear from glancing at the figure is that the alternatives — what society thinks of most often when trying to improve education (e.g., allocating more money, high-stakes standardized testing, charter schools, etc.) — rarely have anything directly to do with instruction.
I kept thinking about this figure when reviewing the entries to our recent “Put Butts in the Bleachers” contest aimed at generating potential improvements to professional conferences. There were a lot of creative suggestions, but many were tangential to the business around which professional meetings are organized: presentations. At first I was perplexed that behavior analysts could so casually overlook the core purpose of presentations. Then I realized that we rarely actually discuss the purpose of presentations, and now I’m wondering what that purpose is.
On the surface, presentations would seem to be for informing or educating, much like the academic lecture. At least that’s the tale we tell our employers when requesting travel funds to attend meetings. We and our employers justify funding on the basis that they’re supporting “professional development,” which strongly suggests an expectation that hearing presentations will change us in valuable ways. This is the same tale, by the way, that credentialing bodies tell when requiring ongoing continuing education: Presumably it makes you a better professional. So let’s run with this notion and cast a critical eye at how well the typical presentation might fulfill an education mission.
The first concern that comes to mind is that presentations are almost always “tell-a-thons.” Someone at the podium talks, while I sit in the peanut gallery and listen. Over 100 years ago Thorndike is reputed to have said that telling isn’t teaching. The great instructional designer Bob Mager‘s variant on that acorn was, “If telling were teaching, we’d all be so smart we couldn’t stand it.” It might be said that the whole basis of behavioral systems of instruction is to minimize telling and maximize learner active responding with feedback. Readers of this blog will be well familiar with many examples, including Skinner’s programmed instruction, Keller’s Personalized System of Instruction, Direct Instruction, Precision Teaching, and Interteaching. In all of these cases, there’s very little telling and a lot of learner responding.
This raises the question: If the purpose of professional presentations is to educate, and to educate requires a lot of learner active responding, how often do professional presentations really take up this gauntlet? I suggest writing to James Diller (dillerj@easternct.edu) and Adam Brewer (brewera@wcsu.edu) for a preprint of their (currently unpublished) paper in which they ask precisely this question about Continuing Education sessions: How well do those sessions incorporate insights from our best work on behavioral instruction? As you might guess if you’ve sat through a few of these sessions, the conclusion is far from encouraging.
I’ve been attending conferences since 1980, and I tried to think of sessions I’ve attended that felt like an approximation of good behavioral instruction. I could come up with only one: At a meeting of the Mid-American Association for behavior Analysis, Janet Twyman demonstrated various digital apps for promoting active responding in group instruction settings (e.g., Kahoot and Acadly). Janet used those apps liberally in her talk to hammer home her points. [By the way, multiple published studies document empirically that Janet is someone to emulate when presenting; see here (Table 4) and here (Figure 6).]
Having come up with only this example, I loosened my criteria and tried to simply remember specific sessions I’ve attended over the years. Informatively, but maybe unsurprisingly, I could remember fewer than a dozen. Now, of course it’s possible I was changed in useful ways by sessions I can no longer consciously recall — cognitive psychologists would call this a failure of “source memory.” Or maybe I’m just losing it in old age. But for the sake of argument let’s take this at face value and say that whatever happened in most of those presentations is long forgotten. When a student or client spends considerable time receiving services that don’t change behavior, we call that unethical (according to ABAI and most ethics codes, people have a “right to effective treatment”). By that token, if professional presentations are for educating, where’s my right to effective education?
Pleased don’t misconstrue my message. I have personally given a lot of presentations over the years, and I’m guessing most people in my audiences don’t remember most of them either. Although I work very hard preparing my talks, I haven’t ever built a presentation around behavioral systems of instruction, so apparently, like everyone else, I have tacitly bought into the tell-a-thon. I speak from experience, therefore, when I say that we all seem rather content doing business at our professional meetings pretty much as it was done over 400 years ago, when “tell-a-thons” were first invented.
But since we know so much about how to do good instruction, I have to wonder: Why are we so complacent?
Hypothesis 1: We Actually Believe That Telling is Teaching
Maybe, despite our collective expertise with respect to instruction, we have naively embraced the myth that telling is teaching. Yet if that were true, you’d think we, as an obsessively empirical discipline, would have produced good evidence that bears on the notion that presentations change behavior.
Based on a quick, and probably incomplete, literature search, I couldn’t find a single behavior analytic study linking professional public speaking to meaningful listener behavior change (better professional functioning). I’m not saying that studies have looked for, and failed to find, this relationship. I’m saying that I couldn’t find a single instance where someone looked for the relationship.
Here’s what the behavior analysis literature on public speaking does show. A fair number of studies have sought to reduce speaker discomfort during presenting; in these cases, the listener was scarcely considered. A smaller number of studies have aimed to teach speakers some useful presenting behaviors. Consider an early study by Fawcett and Miller (1975). Target behaviors included such things as occupying a specific portion of the stage, smiling, and establishing eye contact with the audience. Once these behaviors were trained up, audience ratings of speaker performances became more positive. This is a sort of consumer satisfaction measure. But regardless of how listeners may have felt about presentations, it remains unclear whether listeners actually learned anything or, if they did, whether what they learned changed their subsequent professional behavior in meaningful ways.
Hypothesis 2: Motivation
Maybe presentations are less about instruction and more about motivation. Skinner talked a lot about the potential motivation properties of verbal behavior, as in this comment from Science and Human Behavior:
In a “pep talk” a coach may take advantage of the fact that players exert themselves more aggressively against their opponents if they have been made angry. The skilled cross-examiner may use the same procedure to lead a witness to emit verbal responses which would otherwise be withheld. Soldiers and civilian populations are aroused to aggressive action with stories of atrocities, reminders of present or past injuries, and so on. (pp. 169–170)
Perhaps presentations are meant less to educate than to rouse us to action. Keller, in his famous “Goodbye Teacher” article suggested as much about academic lectures. If that is true, we still, to my knowledge, we appear to lack any hard evidence that presentations function in this way.
[As an aside, I’ll confess that more than once during someone else’s presentation I’ve found myself daydreaming about new studies I could do. Did the speaker inspire me to do this? Or was the talk simply so unengaging that I fell back into high probability behaviors that could be emitted while the talk was droning on? Difficult to say.]
Hypothesis 3: Let Us Entertain You
Maybe, despite what we tell our employers and credentialing institutions, the purpose of professional presentations is just entertainment. Maybe they exist simply because we think they’re fun to listen to. Largely consistent with this assumption are the presenter target behaviors that have been enhanced with behavioral interventions. They focus more on stage presence than instructional effectiveness . The same may be said of expert advice about public speaking. Consider this slightly condensed list of recommendations from presentation pro Pat Friman:
- Model your style after people who are good presenters
- Prepare and practice
- Check out the room ahead of time
- Make a memorable entrance to the stage
- Talk a bit about yourself at the start of the presentation
- Dress impressively
- Stand up straight and smile
- Modulate your inflection to sound interesting
- Use slides only to supplement your remarks
- Tell stories (people love stories)
- Say something important
- Don’t go on too long
Everything in Pat’s list is vital to an enjoyable presentation, but only one item seems directly relevant to an educational function — and, consistent with our tell-a-thon traditions, that item focuses on what the speaker does (“say something important”), not on how to assure listener behavior change.
I can’t resist saying that I’m pretty skeptical about this one. I’ve sat through far too many presentations that were not entertaining, and in fact there’s hard evidence that lectures are not necessarily reinforcers (as true via Zoom as in person, apparently). I would therefore require some serious persuading to buy the notion that presentations exist mainly for entertainment value.
Hypothesis 4: Trojan Horse
But maybe the entertainment is real… it’s just not the real point. Maybe a good song and dance is necessary to get “butts in the bleachers” (see recent posts on this theme), after which other things assure listener learning. That is, maybe this is a bit of a self-control hack. We know from experience that people won’t do enough “professional development” on its own merits, because that involves work in the short run and benefits only after some delay. So we dress up the experience in a way that adds short-term reinforcers.
Nice hypothesis, except for the fact that so many talks are so bad.
Also, if we believed the Trojan Horse approach led to behavior change, wouldn’t we have documented that by now? I’m rejecting this one.
Two Cynical Propositions
Allow me to summarize thus far. I’m not buying it that professional presentations are simply good entertainment, or even good learning in entertainment’s clothing. And there seems to be little evidence that we view presentations as educational — that is, in either instructional or motivational fashion they mobilize professionally useful behavior. If we viewed presentations this way, we would, presumably, mount efforts to evaluate effectiveness that are proportional to our investment in the talk-a-thon format. Let’s put that statement into context. Assume that an average of 200 spoken presentations take place at each annual ABAI conference (probably a conservative estimate). By this count, the association’s 49 meetings have hosted close to 10,000 presentations. The fact that data-obsessed behavior analysts have generated so few data on the educational effects of presentations would suggest to a skeptic that we may not expect there to be any.
And yet here we are, entering ABAI’s 50th year of hosting an annual meeting, without having had a clear discussion of what professional presentations are actually for.
Since we’re now having that discussion, allow me to advance two rather cynical hypotheses that would seem capable of accounting for the fact that we rely so heavily on educationally-questionable presentations.
- Maybe we understand that presentations are a lousy way to do instruction, but we deem them “good enough” under the circumstances, for two reasons. First, maybe we assume that professional audiences, with their advanced degrees and years of experience surviving lecture-format instruction, are simply more resistant to the pitfalls of “telling” than your average listener. Second, maybe we know good instructional programming is difficult and time-consuming to create and evaluate. For instance, consider Dan Fienup’s doctoral dissertation on equivalence-based instruction, which I supervised. This project yielded a one-hour lesson, but took two years to develop and properly implement. There is simply no way speakers will volunteer to present if we hold them to this kind of standard. And without speakers there are no professional meetings. Without professional meetings a key form of “professional development” dries up. So we accept mediocrity because it’s manageable.
- Maybe presentations are rather pointless from an educational standpoint, but “good enough” in other ways. Here we must ask who benefits from our current system of talk-a-thons… and the answer, it turns out, is pretty much everyone. Speakers face limited response cost in piecing together talks. Thus, with relatively little effort (at least when compared to the toils of instructional design), they are able to gain a bit of professional notoriety and receive a bit of credit from the home institution for making scholarly contributions. They may also get their travel to a cool city subsidized. Unlike in behavioral instruction, they are not held accountable for instigating behavior change (good thing, because how would you track what audience members do once back in their respective jobs?). All of this, presumably, assures a supply of speakers on which organizations capitalize to fill out their programs. Once convened in a common location, both speakers and listeners profit from opportunities for informal professional networking outside of the formal sessions, and they get to have fun in a cool city. Let’s also not overlook that professional meetings, in many cases, are money makers for sponsoring organizations, particularly where continuing education credits are involved. And by requiring continuing education experiences, credentialing agencies are able to put forward a good public face about demanding ongoing professional development from practitioners. Because there is no accountability associated with the traditional professional presentation, credentialed practitioners can satisfy professional development expectations without the rigors of really changing their repertoires. And so on. In short, presentations may be lousy educationally, but they facilitate a host of other valuable (or maybe valued?) functions.
If either of the preceding hypotheses approximates the truth, then it’s worth asking why we continue to schedule presentations in more or less the same format that British Royal Society did in the 1600s. Think about it: If presentations as originally conceived do not fulfill a demonstrable educational function, then let’s stop pretending. What’s stopping us from innovating… from developing other kinds of non-instructional programming that are less dull? That was more or less the spirit of our recent contest.
Postscript: What Should Public Speaking Research Look Like?
If we’re to better understand the function of professional presentations, then we’re going to need more research on public speaking and its effects on listeners.
After the contest ended, I talked about this with one of the judges, Matthew Laske of the University of Kansas, who along with his mentor Florence DiGennarro Reed has done research on public speaking. Matt agreed that most of the contest entries didn’t address educational outcomes, and that this probably reflects our collectively unclear conception of what presentation are for. In general, then, we need a clearer specification of what presentations are mean to accomplish. Matt suggested, and I agree, that a good first step would be to consult work from other disciplines that have paid more attention to this topic. In the end, to do research we need some agreement about meaningful dependent measures. According to Matt:
I can think of three ways to measure the outcome of a conference talk. Ranked from worst to best are (1) audience perceptions of the speaker’s effectiveness, (2) audience verbal behavior after the talk (i.e., can they tact the relevant information from the talk), and (c) audience behavior change post-conference (i.e., when they get back to their daily life/job does their behavior change because of the talk).
I think the gold standard would be the behavior change post-conference. At best, we have audience ratings/perceptions at the moment. After each talk I give, I include a QR code that takes participants to a survey asking them, “What was the most valuable thing you learned from this presentation?” I use this as a proxy for (2) to see if they can tact the critical point of my talk. I did not structure or deliver my talk well if their responses did not align with what I intended. It also gives me insight into other things participants might have found valuable. Perhaps I could also ask, “What are you going to do differently because of the presentation?” or something like that. Perhaps get some more verbal behavior that might prompt (3).
Matt’s comments reminded me of a 1970 report by Ogden Lindsley. At a behavior analysis conference, he has his student, Eric Haughton, to record content data on the content (loosely defined as attention to various behavioral concepts) of the formal presentations and the unstructured discussion that followed each of them. We can think of this as a measure of control over audience verbal behavior, or an approximation of Matt’s Measure #2 above. Below are some sample results. For most of the speakers, outcomes were like that for Ted Ayllon, in that speaker and discussion emphases were positively correlated. In other words, what commentators talked about mirrored what the speaker talked about. In other cases, however, there were notable discrepancies. Ivar Lovaas talked about reinforcement more than punishment, but the audience only wanted to talk about punishment. Gerald Patterson talked about. punishment a fair amount, but his audience only wanted to discuss reinforcement. Lovaas talked a lot about behavioral programs, but his audience ignored this; the converse was true for Patterson. These discrepancies serve to illustrate that we don’t know much about how presenting affects subsequence listener verbal behavior.
