Measuring learning impacts: who benefits from science centres?
Image : Smithsonian Museum of Natural History. Credit: RadioFan (Wikimedia). Licensed under CC-BY-SA 4.0.
The tangled strings of learning
In their article, “The 95% Solution”, John Falk & Lynn Dierking described the traditional view of science educators, where ‘formal education’ (i.e. schools) formed the core source of learning. But that view is changing, and the importance of informal education (i.e. everything that you learn outside of the classroom) is becoming more recognized. Thomas la Belle describes learning as a process where formal and informal education are “interacting modes of emphasis”, in which people are “engaged in learning experiences at all times, from planned... to unplanned…” Put simply: learning is one big tangled ball of string - a complex myriad of experiences that occur over time, each becoming interwoven and ultimately forming what we know about a topic. Let’s say I asked you to define an earthquake. Probably a simple question, but then what if I asked you: where did you learn that? For some, such as myself, it gets a bit trickier. Was it from that old geology book I bought as a kid from the school fair? The earthquake exhibit at Te Papa? During my geography teacher’s sleep-inducing monologues? Or perhaps on one of my numerous late-night Wikipedia binges?
Get my point? But of course, we aren’t true scientists unless we back up our claims with cold, hard evidence. So, if we want to say that institutions such as museums and science centres are making an impact on a community, we must prove it. And that’s sometimes easier said than done. Just as it’s difficult to untangle a ball of string, it can be a challenge to pull part learning experiences and see how one component has impacted the learning of a community. But such information can be highly valuable, as not only can it provide an indicator of how effective an institution is on learning generally but can also reveal how different demographics of people within the community are responding.
Unravelling the strings: an outside-in approach
A study by John Falk & Mark Needham set out to find how the California Science Center, which was redeveloped in 1998, impacted the Los Angeles community’s attitudes and learning about science. But to tackle the problem of ‘unraveling’ the threads of learning, they opted for a unique methodology. Termed by the authors as an “outside-in” approach, instead of interviewing people as they entered and left the Center, random samples from the wider LA community were interviewed in two periods - in 2000, just after the Center reopened, and then a second time in 2009. So instead of struggling to untangle the ball of learning, they measured how it changed over time as more and more people visited the California Science Center (45% of respondents in 2009 reported visiting the Center, whereas only 23% said the same in 2000). In other words, rather than pulling the string out, they simply looked at what the ball of learning looked out before the string got interwoven into the minds of the LA community.
A key aspect of their research was the use of a ‘marker’. As I hopefully made clear in my introduction, it’s not always easy to remember where you learnt something! When the California Science Center reopened in 1998, it contained an exhibition that explored the concept of homeostasis. Respondents in 1998 and 2009 were asked about the topic - whether they had heard of it, and to provide an “acceptable definition” of it. By doing so, they felt that this would enable them to accurately measure the contribution of the Center on learning, as opposed to asking people where they learnt something.
What they found
Falk & Needham found that the California Science Center was perceived quite positively by the LA public by 2009. Respondents were likely to agree that visiting the Center improved their attitudes towards science, as well as increasing their knowledge about science and technology. In particular, parents reported a significant impact on their children after visiting, believing that it positively impacted their understanding (87%), appreciation (80%), and interest (78%) in science and technology. And when it came to the homeostasis marker, while the number of people familiar with the time did not change over time, the number of people who could give an “acceptable” definition doubled between 2000 and 2009!
But here’s where it gets interesting. Lower income respondents felt greater positive shifts in their curiosity and attitudes towards science after visiting. And what I consider the most significant result: particularly for most low-income and minority families, visiting a Science Center made them believe that their child would have a greater chance of success in life. I think these results serve to highlight the importance of public institutions for those of lower socioeconomic backgrounds. But despite this seemingly positive finding, there was an interesting contrast when looking at visitation rates. Those same demographics (along with those with less education) were also less likely to visit the Science Center compared to other income and ethnicity groups - though this relationship was relatively weak. So, while they seemed to reap the most benefit from the Center, a smaller proportion of them are seemingly getting the opportunity to get those benefits.
But why? Falk & Needham believed that the relationship between visitation and income was “not surprising”, since the money you have on-hand determines how much “leisure” you participate in. The California Science Center is a free-entry institution, but it still costs families money to travel there, pay for parking, buying food there… there’s still a cost. However, a study by Emily Dawson, whose study involved interviewing low-income minority individuals in London, hints towards other factors. Some respondents did cite the so-called “cost of a ‘free’ visit” as a discouraging factor. But the interviews also suggested that institutions expected a level of both language skills and scientific knowledge for visitors to effectively participate. For those who are socioeconomically disadvantaged, it creates a perception that places like science centres are, as Dawson puts it, “not designed for us”. While the two studies don’t 100% agree with each other - in Dawson’s study, visiting museums & science centres only reinforced their negative perceptions of such places - it nonetheless provides some food-for-thought about how places like the California Science Center could have stronger impacts on their communities.
An effective approach?
While credit must be given to Falk & Needham’s novel approach, I don’t think that some of their findings are entirely conclusive. In particular, the homeostasis “marker”: with twice as many people in 2009 being able to give an “acceptable definition” of homeostasis, the authors used this in their justification of the positive impact of the Center on community science knowledge. But only 2% recalled hearing about homeostasis from the Center. Certainly, that 2% would be an under-representation of the actual contribution of the California Science Center, yet there’s a lot of uncertainty remaining. How have the other ‘strings’ of knowledge changed in that time - other museums & centres, the formal education system, technological advances and access…? We don’t know, and I feel it would be naïve to extrapolate. The study also reported a lot of positive impacts on children. But I feel somewhat cautious about these conclusions, because these impacts weren’t self-reported by the children themselves; they were reported by the parents. Not to undervalue what the parents perceive, but could this potentially create biases within the results, something that Falk & Needham do acknowledge.
Despite these pitfalls, this is certainly a very useful study in showing the importance of science institutions for surrounding communities; if anything, showing a novel methodology for measuring it. Where most studies tend to rely on inside-out approaches, I think Falk & Needham’s methods present an opportunity to obtain new perspectives and ways to unravel the complex, intertwined way of learning. Not to mention that their findings in regard to how various demographics interact and respond to these “informal education institutions” sheds some light on areas for institutions to improve their potential for their local communities.
What do you think? Do you feel outside-in approaches to measuring the success of an institution is as effective as typical inside-out interviews? How could institutions work about the fact that lower-income and minority groups perceive these institutions as having positive impacts, but are at the same time discouraged to visit said places?