While participatory projects do not require fundamentally different evaluation techniques from other types of projects, there are four considerations that make participatory projects unique when it comes to their assessment:

1. Participatory projects are about both process and product. Participatory projects require people to do something for them to work, which means evaluation must focus on participant behavior and the impact of participatory actions. It is not useful to merely catalog the participatory platforms institutions offer—the number of comment boards, participatory exhibit elements, or dialogue programs provided. Evaluators must measure what participants do and describe what happens as a result of participation. Participatory outcomes may be external, like increased incidence of conversation among visitors, and internal, such as development of new skills or enhanced relationships.
2. Participatory projects are not just for participants. It is important to define goals and assess outcomes not only for participants, but for staff members and non-participating audiences as well. For each project, you should be able to articulate goals for the participants who actively collaborate with the institution, for the staff members who manage the process, and for the audience that consumes the participatory product.
3. Participatory projects often benefit from incremental and adaptive measurement techniques. Many participatory projects are process-based. If you are going to work with community members for three years to design a new program, it’s not useful to wait until the end of the three years to evaluate the overall project. Incremental assessment can help complex projects stay aligned to their ultimate goals while making the project work for everyone involved.
4. Sometimes, it is beneficial to make the evaluative process participatory in itself. When projects are co-designed by institutions with community members, it makes sense to involve those participants in the development and implementation of project evaluations. This is particularly true for co-creative and hosted projects in which participants have a high level of responsibility for the direction of the project.

Evaluating Impact

Evaluating the impact of participatory projects requires three steps:[1]

1. Stating your goals
2. Defining behaviors and outcomes that reflect those goals
3. Measuring or assessing the incidence and impact of the outcomes via observable indicators

Goals drive outcomes that are measured via indicators. These three steps are not unique to evaluating participatory projects, but participatory projects frequently involve goals and outcomes that are different from those used to evaluate traditional museum projects.

Recall the wide-ranging indicators the Wing Luke Asian Museum uses to evaluate the extent to which it achieves its community mission (see page 286). When it comes to participants, the staff assesses the extent to which “people contribute artifacts and stories to our exhibits.”[2] With regard to audience members, staff track whether “constituents are comfortable providing both positive and negative feedback” and “community members return time and time again.” Institutionally, they evaluate employees’ “relationship skills” and the extent to which “young people rise to leadership.” And with regard to broader institutional impact, they even look at the extent to which “community responsive exhibits become more widespread in museums.” These outcomes and indicators may be atypical, but they are all measurable. For example, the metric around both positive and negative visitor comments is one that reflects their specific interest in supporting dialogue, not just receiving compliments. Many museums review comments from visitors, but few judge their success by the presence of negative as well as positive ones.

Step 1: Articulating Participatory Goals

The first step to evaluating participatory projects is to agree on a clear list of goals. Particularly when it comes to new and unfamiliar projects, staff members may have different ideas about what success looks like. One person may focus on sustained engagement with the institution over time, whereas another might prioritize visitor creativity. Clear participatory goals can help everyone share the same vision for the project or the institution.

This simple diagram helps MLS staff members evaluate potential programs against institutional goals.

Goals for engagement do not have to be specific to individual projects; they can also be generalized to participatory efforts throughout the institution. For example, at the Museum of Life and Science (MLS), Beck Tench created a honeycomb diagram to display the seven core goals MLS was trying to achieve across their forays into social participation: to educate, give a sense of place, establish transparency, promote science as a way of knowing, foster dialogue, build relationships, and encourage sharing. This diagram gave staff members at MLS a shared language for contextualizing the goals they might apply to participatory projects.

The diagram became a planning tool. For proposed social media experiments, Tench and other MLS staff members shaded the cells of the honeycomb to identify which goals they felt that project would target. This helped them prioritize their ideas and be aware of potential imbalances in their offerings. Later, staff teams used the diagrams to reflect on the extent to which the goals they expected to achieve were met in implementation.

The honeycomb diagram is a simple framework that MLS staff members can apply against a range of projects at different points in their planning and implementation. It is accessible to team members of all levels of evaluation experience and expertise, including visitors and community participants. This makes it easy for everyone to use and understand.

Step 2: Defining Participatory Outcomes

Shared goals provide a common vocabulary to help staff members talk about their aspirations. Outcomes are the behaviors that the staff perceives as indicative of goals being met. Outcomes and outputs are two different things. For example, consider the participatory goal for an institution to become a “safe space for difficult conversations.” The starting point for many museums with this goal would be to host exhibits or programs on provocative topics likely to stir up “difficult conversations.” But offering an exhibition about AIDS or a panel discussion about racism neither ensures dialogue nor the perception of a safe space. An exhibition is an output, but it does not guarantee the desired outcomes.

What are the outcomes associated with a “safe space for difficult conversations?” Such a space would:

• Attract and welcome people with differing points of view on contentious issues
• Provide explicit opportunities for dialogue among participants on tough issues
• Facilitate dialogue in a way that makes participants feel confident and comfortable expressing themselves
• Make people feel both challenged and supported by the experience

Each of these is an outcome that can be measured. For example, imagine a comment board in an exhibition intended to be a “safe space for difficult conversations.” Staff members could code:

• The range of divergent perspectives demonstrated in the comments
• The tone of the comments (personal vs. abstract, respectful vs. destructive)
• The extent to which visitors responded to each other’s comments

Researchers could also conduct follow-up interviews to ask visitors whether they contributed to the comment board, how they felt about the comments on display, their level of comfort in sharing their own beliefs, the extent to which their perspective was altered by the experience of expressing their own ideas and/or reading others, and whether they would seek out such an experience in the future.

Developing Meaningful Measurement Tools

Once staff members know what goals they are trying to achieve and what outcomes reflect those goals, they can develop evaluative tools to assess the incidence of the outcomes. This is often the most challenging part of evaluation design, and it requires thinking creatively about what behaviors or indicators are associated with desired outcomes.

The New Economics Foundation has defined four qualities of effective evaluative indicators: action-oriented, important, measurable, and simple.[3] Imagine developing indicators to reflect this goal for an educational program: “visitors will have deeper relationships with the staff.” What measurable indicators would provide valuable information about relationships between staff members and visitors? How could assessment help staff members develop new strategies or practices to improve incidence of the outcome in future projects?

To assess this outcome, you might consider the following indicators:

• Whether staff members and participants could identify each other by name at different stages during and after the program
• The volume and type of correspondence (email, phone, social networks) among participants and staff outside of program time
• Whether staff members and participants stayed in touch at fixed time intervals after the program was over

These indicators might help determine what kinds of behaviors are most effective at promoting “deeper relationships” and encourage the staff to adjust their actions accordingly. For example, some educators (like the volleyball teacher described on page 39) find it particularly valuable to learn students’ names and begin using them from the first session. These kinds of discoveries can help staff members focus on what’s important to achieve their goals.

CASE STUDY: Engaging Front-Line Staff as Researchers at the Museum of Science and Industry Tampa

When indicators are simple to measure and act on, staff members and participants at all levels of research expertise can get involved in evaluation. For example, at the Museum of Science and Industry in Tampa (MOSI), front-line staff members were engaged as researchers in the multi-year REFLECTS project. The goal of REFLECTS was to train educators to be able to scaffold family visitor experiences in ways that would encourage “active engagement” (as opposed to passive disinterest). To make it possible for front-line educators to effectively recognize when visitors were and were not actively engaged in their experience, the research team developed a list of eleven visitor behaviors that they felt indicated active engagement, including: visitors making comments about the exhibit, asking and answering each other’s questions, making connections to prior experiences, and encouraging each other’s behaviors.

Educators made video and audio recordings of their interactions with families and then went back later to code the recordings for those eleven indicators. The REFLECTS team didn’t judge the content of the cues (i.e. whether a visitor asked a personal question or a science-focused one), just tallied their incidence. And then educators headed back out on the floor to adjust their behavior and try again.

At the 2009 Association of Science and Technology Centers conference, MOSI staff members showed video of themselves engaging with visitors before and after working in the REFLECTS program, and the difference was impressive. The educators didn’t communicate more or better content in the “after” videos. Instead, they did a better job supporting visitors’ personal connection to the exhibits, rather than trying, often unsuccessfully, to coerce visitors into engagement.

The REFLECTS research served three audiences: visitors, front-line staff, and the institution. For visitors, it improved the quality of staff interactions. For the front-line staff, it provided empowerment and professional development opportunities. For the institution, it made front-line interactions with visitors more effective. The primary researcher, Judith Lombana, had observed that museums spend a lot of time engaging with visitors in ways that do not improve engagement or learning. This is a business problem. As she put it: “waste occurs with activities or resources that some particular guest does not want.” By finding action-oriented, important, simple measures for active engagement, the REFLECTS team was able to create an evaluation strategy that succeeded on multiple levels.

Evaluation Questions Specific to Participation

Because participatory practices are still fairly new to cultural institutions, there are few comprehensive examples of evaluation techniques and instruments specific to participatory projects. Traditional evaluation techniques, like observation, tracking, surveys, interviews, and longitudinal studies are all useful tools for assessing participatory projects. However, because participatory projects often involve behaviors that are not part of the traditional visitor experience, it’s important to make sure that evaluation instruments will capture and measure the distinct experiences offered.

If your institution’s standard summative evaluation of an exhibition or program is about the extent to which visitors have learned specific content elements, switching to an evaluative tool that allows the staff to assess the extent to which visitors have exercised creative, dialogic, or collaborative functions can be quite a leap.

To design successful evaluative tools for participatory projects:

• Review the specific skills and values that participatory experiences support (see page 211) to determine which kinds of indicators might reflect your project’s goals
• Take a 360 degree approach, looking at goals, outcomes, and indicators for staff members, participants, and non-participating visitors
• Consult with participants and project staff members to find out what outcomes and indicators they think are most important to measure

Here are some specialized questions to consider that pertain specifically to the participant, staff, and visitor experience of participatory projects.

Questions about participants:

• If participation is voluntary, what is the profile of visitors who choose to participate actively? What is the profile of visitors who choose not to participate?
• If there are many forms of voluntary participation, can you identify the differences among visitors who choose to create, to critique, to collect, and to spectate?
• How does the number or type of model content affect visitors’ inclination to participate?
• Do participants describe their relationship to the institution and/or to staff in ways that are distinct from the ways other visitors describe their relationship?
• Do participants demonstrate new levels of ownership, trust, and/or understanding of institutions and their processes during or after participation?
• Do participants demonstrate new skills, attitudes, behaviors, and/or values during or after participation?
• Do participants seek out more opportunities to engage with the institution or to engage in participatory projects?

Questions about staff:

• How do participatory processes affect staff members’ self-confidence and sense of value to the institution?
• Do staff members demonstrate new skills, attitudes, behaviors, and/or values during or after participation?
• Do staff members describe their relationships to colleagues and or visitors as altered by participation?
• Do staff members describe their roles differently during or after participation?
• How do staff members perceive the products of participation?
• Do staff members seek out more opportunities to engage in participatory projects?

Questions about non-participating visitors who watch or consume the products of participation (exhibits, programs, publications):

• Do visitors describe products created via participatory processes differently from those created via traditional processes? Do they express comparative opinions about these products?
• If participation is open and voluntary, do visitors understand the opportunity to participate?
• Why do visitors choose not to participate? What would make them interested in doing so?

Because participation is diverse, no single set of questions or evaluative technique is automatically best suited to its study. There are researchers in motivational psychology, community development, civic engagement, and human-computer interaction whose work can inform participatory projects in museums.[4] By partnering with researchers from other fields, museum evaluators can join participatory, collaborative learning communities to the mutual benefit of all parties.

CASE STUDY: Studying the Conversations on Science Buzz

Imagine a project that invites visitors to engage in dialogue around institutional content. How would you study and measure their discussions to determine whether users were just chatting or really engaging around the content of interest?

In 2007, the Institute of Museum and Library Services (IMLS) funded a research project called Take Two to address this question. Take Two brought together researchers in the fields of rhetoric, museum studies, and science education to describe the impact of a participatory project called Science Buzz that invited visitors to engage in dialogue on the Web about science.

Science Buzz is an award-winning online social network managed by the Science Museum of Minnesota.[5] It is a multi-author community website and exhibit that invites museum staff members and outside participants to write, share, and comment on articles related to contemporary science news and issues. Science Buzz also includes physical museum kiosks located in several science centers throughout the US, but the Take Two study focused on the online discourse.

Science Buzz is a complicated beast. From 2006 to 2008, staff members and visitors posted and commented on over 1,500 topics, and the blog enjoyed high traffic from an international audience. While the museum had conducted internal formative evaluation on the design and use of Science Buzz,[6] the staff was interested in conducting research on how users interacted with each other on the website and what impact it had on their learning.

That’s where Take Two came in. Because Science Buzz is a dialogue project, it made sense to work with researchers from the field of rhetoric. Jeff Grabill, a Michigan State University professor who focuses on how people use writing in digital environments, led the research. The Take Two team focused their study on four questions:

1. What is the nature of the community that interacts through Science Buzz?
2. What is the nature of the on-line interaction?
3. Do these on-line interactions support knowledge building for this user community?
4. Do on-line interactions support inquiry, learning, and change within the museum – i.e., what is the impact on museum practice?[7]

The first two questions are descriptive and focus on better understanding the user profile and the dialogic ways that people engage with each other on the website. The last two are about impact outcomes both for participants and for the staff. Since the researchers were examining historic blog posts, they did not have access to non-participating audience members. They did not study the impact on those who consume the content on Science Buzz but do not contribute content to the site themselves.

To evaluate the knowledge building impact of Science Buzz, the researchers coded individual statements in blog posts and comments for twenty percent of posts with fifteen comments or more, grouping them into four categories: “building an argument,” “exploring new ideas,” “building a writer’s identity,” and “building a community identity.” Staff members associated each statement with one of these four categories using a comprehensive set of descriptive indicators (see table).

The Take Two research team coded comments on Science Buzz based on the incidence of these indicators.

By coding individual statements, researchers were able to spot patterns in argumentation used on the site that represented different forms of individual and or interpersonal knowledge building. For the representative sample used, the researchers found the following overall distribution of statement types:

• Building an argument – 60%
• Building a writer’s identity – 25%
• Building community identity – 11.4%
• Exploring new ideas – 1.8%

This data demonstrated that Science Buzz users were definitely using the blog to make arguments about science, but not necessarily to construct knowledge communally. For this reason, the Take Two team shifted its research in the third year of the study, away from “co-construction of knowledge” and toward a broader examination of “learning.”

They used the National Academies of Science’s 2009 Learning Strands in Informal Environments (LSIE) report as the basis for the development of new indicators with which to code Science Buzz conversations.[8] The LSIE report presented six “strands” of science learning, including elements like identity-building, argumentation, and reflection that were clearly visible in Science Buzz discourse. As of January 2010, this research is still ongoing. Dr. Kirsten Ellenbogen, Director of Evaluation and Research in Learning at the Science Museum of Minnesota, commented:

The LSIE report stated that science argumentation is rare in museum exhibits, and it suggested that informal environments were a long way from providing the necessary instruction to support scientific argumentation.[9] But Science Buzz is a natural dialogue setting, and we felt like we were seeing scientific argumentation and debate happening all over the site. The Take Two research gave us evidence to support that.

By partnering with researchers in the field of rhetoric, the Science Buzz team was better able to understand and describe the nature and potential impact of conversation on the website. The research also revealed new questions for study both on Science Buzz and other online dialogue sites. One of the findings of Take Two was that identity-building statements are often intertwined with scientific arguments, and it is important to understand who a person is as well as what they say. This may sound obvious, but as Grabill noted, rhetoriticians frequently separate what they consider rational statements from affective or “identity work” statements, which are considered less important to argumentation. While the first phase of the research focused entirely on types of statements made in discrete, anonymized comments, the second phase included examination of the particular role of staff participants in promoting learning. In future research of Science Buzz, it’s possible to go even further, examining how individual users’ interactions with the site over time impact their learning, self-concept, and contribution to the community.

Part of the challenge of the Take Two project was simply developing the analytic tools to study a familiar question (science knowledge-building) in a new environment (online social network). The team focused on mission-driven questions, found reasonable tools to answer those questions, rigorously applied those tools, and published the results. Hopefully many future teams will approach research on visitor participation with a comparable level of rigor, creativity, and interest in sharing lessons learned with the field.

Incremental and Adaptive Participatory Techniques

While formal evaluation is typically separated into discrete stages—front-end, formative, remedial, summative—participatory projects often benefit from more iterative approaches to assessment. One of the positive aspects of participatory projects is that they don’t have to be fully designed before launch. Participatory projects are often released bit by bit, evolving in response to what participants do. This can be messy and can involve changing design techniques or experimental strategies along the way, which, as noted in the stories of The Tech Virtual and Wikipedia Loves Art, can either alleviate or increase participant frustration. While changes may be frustrating and confusing, they are often essential to keep an experimental project going in the right direction.

Adaptive evaluation techniques are particularly natural and common to the Web for two reasons. First, collecting data about user behavior is fairly easy. There are many free analytical tools that allow Web managers to capture real-time statistics about who visits which pages and how they use them. These tools automate data collection, so staff members can focus on working with the results rather than generating them.

Second, most Web designers, particularly those working on social websites, expect their work to evolve over time. Most Web 2.0 sites are in “perpetual beta,” which means that they are released before completion and remain a work-in-progress—sometimes for years. This allows designers to be responsive to observed user behaviors, altering the platform to encourage certain actions and minimize or eliminate others.

Adaptive evaluation can help designers and managers see where they are and aren’t hitting their goals and adjust their efforts accordingly. For example, the Powerhouse Museum children’s website features a popular section called “Make & Do” which offers resources for family craft activities.[10] Each craft activity takes about two weeks of staff time to prepare, so the team decided to use Web analytics to determine which activities were most popular and use those as a guide for what to offer next.

In the site’s first two years, Web analytics showed that the most popular craft activity by far was Make a King’s Crown, which provides templates and instructions to cut out your own royal headgear. At first, the staff responded by producing similar online craft resources, providing templates for wizards’ hats, jester hats, samurai helmets, and masks. Then, the Web team dug a little deeper into the Web metrics and realized that the vast majority of the visitors to the King’s Crown page were based outside Australia. When they looked at the geographic detail for the Make & Do section, they found that gardening and Easter-related activities were far more popular with Australian and Sydney residents than various forms of hat making. Because the Powerhouse is a majority state-funded museum (and because the children’s website was intended to primarily provide pre- and post-visit experiences), their priority is local audiences. They decided to redirect future craft resources away from headgear and towards activities that were of more value and interest to Australians from New South Wales.[11]

Adaptive evaluation can be applied to physical venues and visitor experiences, but it’s not easy. Cultural institutions tend to lack both the Web’s automated analytic tools and flexible growth pattern. If the educational program and exhibition schedule is set months or even years in advance, it is unlikely that staff members will be able to shift gears based on visitor input. This is one of the reasons that remedial evaluation can be so painful in museums; even if staff members would like to make changes to improve the visitor experience based on observed problems, it’s hard to find the money or time to do so. Many staff members can also get focused on “seeing it through” and may be concerned about tainting data by making changes in mid-stream. Encouraging adaptive evaluation requires cultivating a culture of experimentation.

When it comes to physical platforms for participation, it’s not necessary to design the perfect system behind the scenes. Release it unfinished, see what visitors do, and adjust accordingly. If projects are truly designed to “get better the more people use them,” then there’s a built-in expectation that they will grow and change over their public life. Staff members can be part of that growth and change through adaptive evaluation.

Continual evaluation can also provide a useful feedback loop that generates new design insights that can ultimately lead to better visitor experiences. Consider the humble comment board. If there is one in your institution, consider changing the material provided for visitors to write on or with and see how it affects the content and volume of comments produced. Change the prompt questions, add a mechanism by which visitors can easily write “response cards” to each other, or experiment with different strategies for staff curation/comment management. By making small changes to the project during operation, you can quickly see what kinds of approaches impact the experience and how.

Involving Participants in Evaluation

One of the underlying values of participatory projects is respect for participants’ opinions and input. Because project ownership is often shared among staff members and participants, it makes sense to integrate participants into evaluation as well as design and implementation. This doesn’t mean evaluating participant experiences, which is part of most institutional evaluative strategies. It means working with participants to plan, execute, and distribute the evaluation of the project.

Adaptive projects often include casual ways for participants to offer their feedback on projects at any time, either by communicating with project staff members or sharing their thoughts in community meetings or forums. Particularly when participants feel invested in a project, either as contributors or as full co-creative partners, they want to do what they can to help improve and sustain the project’s growth. Participants may notice indicators that are not readily apparent to project staff, and they can offer valuable input on the most effective ways to measure and collect data related to their experiences.

For example, the St. Louis Science Center’s Learning Places project, in which teenagers designed science exhibits for local community centers, was evaluated in two ways: by an external evaluator and by internal video interviews with the teenage participants.[12] The external evaluation focused on the teens’ understanding and retention of science, technology, engineering, and mathematics concepts, whereas the participants’ interviews focused on the project’s impact on teens’ educational and career choices. These different evaluative techniques reflected two different sets of measurement goals and priorities. While the funder (the National Science Foundation) was interested in how Learning Places promoted science learning, the teen participants and the project staff were interested in how the program impacted personal and professional development.

Involving community members in the design and implementation of evaluative techniques is not easy. Participatory evaluation requires additional resources. For many institutions, it may be too expensive to sustain participatory partnerships through the research stage, or participants may not be able to continue their involvement beyond the co-design project.

Participatory evaluation is also challenging because it requires a radical shift in thinking about the audience for the research and how it will be used. At least in the museum field, researchers and practitioners are still working out how visitor studies can be most useful to actually impact how institutions function. It’s an even further step to suggest that not only should visitors’ reactions and experiences partly guide professional practice, but that their goals should drive research just as much as institutional goals. This can be particularly challenging when working with an outside funder with specific research expectations that may not be relevant to the goals and interests of participants.

How can you decide whether to involve participants in evaluation? Just as there are multiple models for ways to engage community members in participatory projects, there are several ways to involve them in evaluation. The LITMUS project in South London separated evaluation of community projects into three basic models: top-down, cooperative, and bottom-up.[13] Top-down evaluation is a traditional assessment strategy in which senior managers or external evaluators plan and manage evaluation. External evaluators also lead cooperative evaluation, but in this model, evaluators serve as guides, working with participants and project staff to develop assessment techniques and to collect and analyze data. In bottom-up models, external evaluators still facilitate the evaluation process, but their work is directed by participants and project staff to address their interests rather than institutionally driven measures of success.

Choosing the most effective way to engage participants in evaluation depends on several factors, including:

• Participant motivation. Are participants interested and willing to participate in evaluation?
• Participant availability. Are participants able to continue to contribute to the evaluation of projects after they are completed? Can the institution compensate them for their time in some way?
• Participant ability. Are participants sufficiently skilled and trusted in the community to help lead a fair evaluation of the project? Are the assessment indicators simple enough for amateur evaluators to measure them?
• Relevance. Are the evaluative goals and measures relevant to participants’ experience? Can they gain something from participating in evaluation? Will they be able to take new action based on their involvement?
• Transparency. Is the institution willing to open up evaluative processes to outside involvement? Will participants be able to distribute and use the results of the evaluation for their own purposes?

If the answer to many of these questions is “yes,” it might be appropriate to pursue cooperative or bottom-up evaluation instead of a traditional study. If the answer is mostly “no,” staff members can improve the potential for participatory evaluation in the future by improving the incidence of these indicators. For example, staff members might make traditional internal evaluations available for public use to enhance transparency, or they might work with participants to develop some questions for evaluation without including them in the entire process.

While it can be complex to execute, participatory evaluation encourages staff members to design measurement techniques that are actually useful—tools that they can use to improve the work for next time. When participants are invested in acting as researchers, they hold staff members accountable to the findings. Especially in long-standing partnerships—for example, consultative advisory boards—both the staff and community members should feel that research is helping enhance the project overall. Otherwise, why spend all that time on evaluation? In this way, participatory techniques can help make evaluation more beneficial to how cultural institutions function—not just in participatory projects, but across the board.

***

We’ve now looked at a range of techniques for planning, implementing, and evaluating participatory projects. In Chapter 11, we’ll focus inward and look at how institutional culture can impact which kinds of projects are most likely to succeed at different organizations. Sustaining participation isn’t just a matter of motivating visitors; it also requires developing management strategies that help staff members feel supported and enthusiastic about being involved.

Chapter 10 Notes

I have been most greatly influenced in writing this book by three professional heroes: Elaine Heumann Gurian, Kathleen McLean, and John Falk. If this book excites you, I highly recommend that you “go to the source” and read their books as well. I am honored to consider Elaine, Kathy, and John as mentors and friends, and I would not have written this book without their inspiring example.

This book was not only inspired by colleagues; it was also directly improved by their active involvement in its development. If you think peer review in the museum field is dead or languishing, I encourage you to open your work processes to the eyes and opinions of your colleagues. I wrote this book publicly on a wiki site and invited colleagues and enthusiastic readers of the Museum 2.0 blog to add their comments, insights, and examples along the way. Their contributions, particularly during the editing stage, were invaluable. They pointed out what was valuable and not, shared new examples, and gave me the courage to make major changes to the content, organization, and tone of the book.

Several people contributed many hours of their attention towards improving this book. Some of these editors are colleagues whom I solicited directly, but most are museum professionals and interested folks with whom I had no prior relationship. In no particular order, I’d like to thank the content reviewers (some of whom have provided bios): Conxa Rodà, Sarah Barton, Mark Kille, Barbara Oliver, Bruce Wyman, Cath Styles, Susan Spero, Chris Castle, Claire Antrobus, David Kelly-Hedrick, Ed Rodley, Georgina Goodlander, Linda Norris, Kevin Von Appen, Darcie Fohrman, Maria Mortati, Haz Said, Jody Crago, Jonah Holland, Kerrick Lucker, Kristin Lang, Daniel Spock, Eric Siegel, Lauri Berkenkamp, Rebekah Sobel, Andrea Bandelli, Louise Govier, Lynn Bethke, John Falk, Peter Linett, Ruth Cuadra, Maureen Doyle, Marc Van Bree, Patricia Sabine, Heidi Glatfelter, Susan Edwards, Jane Severs, Phillippa Pitts, Jana Hill, Mariana Salgado, Melissa Gula, Robert Connolly (and his museum practices students), Becky Menlove, Mia Ridge, and Michael Skelly.

Copy-editing was also a collaborative volunteer effort. Thank you to: Dave Mayfield, James Neal, Buster Ratliff, Lizz Wilkinson, Tikka Wilson, Jody Crago, Erin Andrews, Lisa Worley, Monica Freeman, Matthew Andress, Barbara Berry, Kaia Landon, Rhonda Newton, Jonathan Kuhr, Lynn Bethke, Susan Edwards, and L. Corwin Christie for making all the little punctuation and grammatical fixes that make this book read smoothly.

Thanks to Robin Sloan and Scott Simon for their editorial efforts to improve the book overall. Thanks also to Sibley Simon and Sarina Simon, who generously responded to many out-of-the-blue requests for feedback. Great thanks to Jennifer Rae Atkins, who designed the beautiful front and back covers, and to Karen Braiser, who formatted many of the Web-based images for inclusion.

This book was heavily informed by conversations and interviews with professionals across many institutions. Thank you to Evelyn Orantes, Shelley Bernstein, Wendy Erd, Tsivia Cohen, Kris Morrissey, Jeff Grabill, Kirsten Ellenbogen, Kelli Nowinsky, David Chesebrough, Stephanie Ratcliffe, Jane McGonigal, William Cary, Jamee Telford, Barbara Henry, Kathleen McLean, Kevin Von Appen, Sabrina Greupner, Vishnu Ramcharan, Robert Stein, Chris Alexander, Bridget Conley-Zilkic, David Klevan, Nancy Zinn, Jackie Copeland, Josh Greenberg, Jessica Pigza, Lori Fogarty, Beck Tench, Jeff Stern, and countless others who have generously provided information, images, and inspiration for this book.

I take full responsibility for all errors and omissions in this book, and I encourage you to share new case studies, comments, and questions on this site in the Discuss section. Please share your feedback. It improves everyone’s experience with the content and it opens up the opportunity for multi-directional conversation.

Finally, thank you for considering these ideas and, hopefully, integrating them into your work. I can’t wait to hear where you take it.

Drawing by Jennifer Rae Atkins

Nina Simon is an independent experience designer with expertise in participatory design, gaming, and social technology. She is the principal of Museum 2.0, a design firm that works with museums, libraries, and cultural institutions worldwide to create dynamic, audience-driven exhibitions and educational programs.

In addition to design work, Nina lectures and gives workshops on visitor participation. She is an adjunct professor of social technology in the University of Washington Museology program. Nina authors the Museum 2.0 blog, which also appears as a column in Museum magazine.

Previously, Nina served as Curator at The Tech Museum of Innovation in San Jose, CA, and was the Experience Development Specialist at the International Spy Museum in Washington, D.C.

Nina lives in Santa Cruz, California.

HOWEVER, not all of the images included in the book (and on this site) are licensed under Creative Commons. Wherever relevant, I have included the license and ownership information in the captions of images. Please respect these fine peoples’ copyright in your use of this material.

Cite as: Simon, Nina. The Participatory Museum. Santa Cruz: Museum 2.0, 2010.

Please contact me with any questions or proposals for reuse and adaptation of this book.

• none so far – did you find one? If so, go here.

Bugs in the printed book:

• Chapter 1, page 32, last sentence: “motivate” should be “motivated.”
• Chapter 2, pages 61-62. Tim Spalding’s name is misspelled. “Spaulding” should be “Spalding.”
• Chapter 3, page 103, caption on Human Library image: “LIbrarians” should be “Librarians.”
• Chapter 7, Tech Museum case study: “three challenged arose” should be “three challenges arose.”
• Did you find one? If so, go here.

Bugs in the ebook:

• Chapter 2, Librarything discussion. Tim Spalding’s name is misspelled. “Spaulding” should be “Spalding.”
• Chapter 3, caption on Human Library image: “LIbrarians” should be “Librarians.”
• Chapter 7, Tech Museum case study: “three challenged arose” should be “three challenges arose.”

• Did you find one? If so, go here.

Jump to review by: Seb Chan, Kathleen McLean, Daniel Spock, Eric Siegel, Leslie Bedford, Elizabeth Merritt, Bruce Wyman, Harry White, Maria Mortati, Mandi Zanski, Claire Antrobus, Robert Connolly

Seb Chan, Head of Digital, Social and Emerging Technologies, Powerhouse Museum, Australia:

The Participatory Museum is essential reading for all museum educators, designers, curators, program developers and forward thinking directors. Through a well chosen and wide range of international case studies, Nina teases out the implications and opportunities for museums in engaging with a public increasingly familiar with new modes of interaction resulting from nearly two decades of the public web. Refreshingly, she avoids the technical aspects and focuses on how all types of museum can reach out to new audiences and avoid alienating existing ones whose expectations have now changed. She neatly balances the practical ideas for immediate implementation with their broader strategic thinking, making The Participatory Museum *the* go-to recipe book for the coming years.

Kathleen McLean, designer of participatory museum experiences and author of Planning for People in Museum Exhibitions:

This book is an extraordinary resource. Nina has assembled the collective wisdom of the field, and has given it her own brilliant spin. She shows us all how to walk the talk. Her book will make you want to go right out and start experimenting with participatory projects.

Dan Spock, Director, Minnesota History Center:

In readable and engaging prose, Simon provides a multiplicity of practical, real-world examples and strategies for eliciting and enhancing public participation in ways that deliver real value to museum-goers and museums alike. This book will prove essential for any museum seeking to affirm its connection to the public in new ways relevant to the times in which we live.

Eric Siegel, Director and Chief Content Officer, New York Hall of Science:

Nina’s confident, enthusiastic, pragmatic tone overcomes any concern that her ideas are just the latest buzzwords, and fosters a sense of possibility and engagement for museum professionals. As I read this book, there were about 20 times when I thought “we should try this!” The Participatory Museum has the resonance of a manifesto and the potential to make a transformative impact on museum practice and visitors’ experience in museums in the coming decades.

Leslie Bedford, Director, Leadership in Museum Education graduate program, Bank Street College:

Simon is an excellent teacher; she gently leads her readers through what is likely to be new and intimidating territory. She articulates a useful set of intelligent principles, grounded in research and theory, of the sort that promotes reflective and effective practice…  This is the convincing marriage of theory and practice that good graduate programs and professional development efforts espouse and will welcome to their reading lists.

Elizabeth Merritt, Founding Director, Center for the Future of Museums:

Nina Simon’s new book offers practical advice on how museums can become forums for discussion of the crucially important issues of our time. Drawing on lessons from sources as diverse as the publishing industry, casinos and Nike, as well as from innovative museums around the world, this essential guide helps museums break down barriers to cultivating new audiences. I predict that in the future this book will be a classic work of museology.

Bruce Wyman, Director of Technology, Denver Art Museum:

In a blur, I imagine Nina racing into the workroom imploring us all with, “Come, Watson, come! The game is afoot.” The tone and enthusiasm of this book engages and empowers a new generation of museum professionals to reinterpret how we interact with our visitors. Included throughout are thoughtful case studies, interviews, and personal experiences which guide us along a new path of participatory design. This book intrigues, delights, and kept me coming back for more.

Harry White, Science Centre consultant, Cardiff, UK:

This is a field report from the cutting edge of modern museology. Nina sets out the case for participation in a clearly structured distillation of her wide experiences, including lots of case studies which are backed up with more detail on the linked website.

I read the book on holiday and came back fired up with new ideas to try. Can’t recommend it highly enough, if you’re in the business, just add it to your basket now, you won’t regret it.

Maria Mortati, Exhibit Designer and founder of the SF Mobile Museum:

After years of providing insightful, tactical and creative posts on museums, web 2.0 and participation, Nina has put together a clear case for fostering visitor participation. She provides compelling case studies from around the world in museums, libraries, and other industries. All done with her signature smart and fun approach.

If you want to understand the world of participatory design and it’s possibilities for your institution, this is an excellent resource.

Mandi Zanski, Exhibit Design graduate student, Fashion Institute of Technlogy:

As a student of Exhibition Design at FIT in NYC, I found Nina’s book to be a must read for anyone who is interested in creating meaningful and personal cultural experiences. The book serves as a guide for understanding exactly what visitor participation can be, can look like, and how it can be evaluated. The organization of information and the numerous case studies used help the reader to digest the otherwise complex task of empowering individuals in cultural institutions. Everyone is talking about “interactive, social” exhibit environments, but Nina Simon actually describes ways that this can be achieved successfully. An inspiring read that I will refer to for the rest of my career!

Claire Antrobus, Independent Curator and Arts Manager, York, UK:

If you want to know what participation could offer your museum, art gallery, library or cultural organisation then this book is for you. It’s packed full of inspiring examples and practical advice as well as a very clear framework for helping arts professionals understand what participation offers, and which approaches and models might work best for your institution.

Participation is a hot topic, but also one which is poorly understood and feared in some quarters. This book unpicks some of the myths around participation and provides a very clear and simple framework for understanding the benefits and models of participation. It also makes a strong case for why participation is important and how it’s helping museums and arts organisations become more relevant and sustainable.

The main audience will be those working in the design and delivery of museum displays and exhibitions, but the book also has much to offer those involved in managing and leading cultural organisations – with a clear focus on management implications (and useful chapters on evaluation and sustaining participation). It’s well-written and easy to navigate – with great indexing and references for anyone interested in following up the ideas.

Essential reading for anyone interested in increasing participation in museums and arts organisations: from students, to specialists in design and learning, curators, managers and leaders of institutions.

The book combines inspirational ideas with down to earth examples from many different types and scales on institutions and makes you just want to just start doing it!

Robert Connolly, Director of the Nash Museum and Associate Professor of Anthropology, University of Memphis:

As an avid reader of Nina Simon’s Museum 2.0 blog I was quite pleased when she announced she was writing a book. Now in hand, my initial read of The Participatory Museum lives up to my expectations. Ms. Simon is clearly on the cutting edge in the practical, hands-on, applied end of Museum work. I particularly enjoy her outside the box thinking that is firmly grounded in practice. The volume is an excellent resource to kick start creative thinking from conceptualizing through to implementing and evaluating visitor participation appropriate for the monster and miniature museums alike. The Participatory Museum is a welcome addition as an assigned text for my course offerings in Museum Studies.

Books and Journals

Falk, John. Identity and the Museum Visitor Experience. Walnut Creek: Left Coast Press, 2009.

Falk, John and Lynn Dierking. The Museum Experience. Washington D.C.: Whalesback Books, 1992.

Falk, John and Beverly Sheppard. Thriving in the Knowledge Age: New Business Models for Museums and Other Cultural Institutions. Walnut Creek: AltaMira Press, 2006.

Hein, George. Learning in the Museum. London: Routledge, 1998.

Heumann Gurian, Elaine. Civilizing the Museum. London: Routledge, 2006.

Humphrey, Thomas and Josh Gutwill. Fostering Active Prolonged Engagement: the Art of Creating APE Exhibits. San Francisco: Exploratorium, 2005.

Illich, Ivan. Deschooling Society. New York: Harper and Row, 1971.

Jenkins, Henry. Convergence Culture: Where Old and New Media Collide. New York: New York University Press, 2006.

Jennings, Gretchen, ed. “Visitor-Generated Content and Design.” Exhibitionist 28(2).

Koke, Judy and Marjorie Schwarzer, ed. “Civic Discourse: Let’s Talk.” Museums & Social Issues 2(2).

Li, Charlene and Josh Bernhoff. Groundswell: Winning in a World Transformed by Social Technologies. Boston: Harvard Business School Publishing, 2008.

McLean, Kathleen. Planning for People in Museum Exhibitions. Washington D.C.: Association of Science and Technology Centers, 1993.

McLean, Kathleen and Wendy Pollock, ed. Visitor Voices in Museum Exhibitions. Washington D.C. Association of Science and Technology Centers, 2007.

Norman, Donald. The Design of Everyday Things. New York: Doubleday, 1990.

Shirky, Clay. Here Comes Everybody: The Power of Organizing without Organizations. New York: Penguin Press, 2008.

Stanton, Philip, ed. Repensar el Picasso. Barcelona: Agpograf, 2009. [PDF]

Surowiecki, James. The Wisdom of Crowds. New York: Anchor Books, 2005.

Zorich, Diane, Gunter Waibel, and Ricky Erway. Beyond the Silos of the LAMs: Collaboration Among Libraries, Archives, and Museums. Dublin, Ohio: Online Computer Library Center, 2008. [PDF]

Magazine, Newspaper, and Journal Articles

Allen, Sue and Josh Gutwill (2004). “Designing with multiple interactives: Five common pitfalls.” Curator 47(2): 199-212. [PDF]

Anderson, Maxwell. “Prescriptions for Art Museums in the Decade Ahead.” Curator, The Museum Journal, Issue 50.1, January 2007. [read]

Droege, Sam. “Just because you paid them doesn’t mean their data are better.” Ithaca, NY: Citizen Science Toolkit Conference Proceedings, June 2007. [PDF]

Filippini-Fantoni, S. and J. Bowen. “Bookmarking In Museums: Extending The Museum Experience Beyond the Visit?” In J. Trant and D. Bearman (eds.). Museums and the Web 2007: Proceedings, Toronto: Archives & Museum Informatics, March 1, 2007. [read]

Grabill, J.T. et al. “Take Two: A Study of the Co-Creation of Knowledge on Museum 2.0 Sites.” In J. Trant and D. Bearman (eds). Museums and the Web 2009: Proceedings. Toronto: Archives & Museum Informatics. March 31, 2009. [read]

Jenkins, Henry, Xiaochang Li,  Ana Domb Krauskopf,  and Joshua Green. “If It Doesn’t Spread It’s Dead.” Convergence Culture Consortium: March 2009. [read]

Nielsen, Jakob. “Participation Inequality: Encouraging More Users to Contribute.” Alertbox, October 9, 2006. [read]

Sandberg, Jared. “Farming? Running? It Doesn’t Sound Like a Vacation to Me.” Wall Street Journal, July 18, 2006. [read]

Tsybulskaya, Dina and Jeff Camhi. “Accessing and Incorporating Visitors’ Entrance Narratives in Guided Museum Tours.” Curator 52, no. 1 (2009): 81–100.

von Appen, et al. “WeTube: Getting Physical with a Virtual Community at the Ontario Science Centre.” In J. Trant and D. Bearman (eds). Museums and the Web 2009: Proceedings. Toronto: Archives & Museum Informatics, March, 2009. [read]

Weinberg, Carl. “The Discomfort Zone: Reenacting Slavery at Connor Prairie.” OAH Magazine of History 23, no. 2 (2009). [read]

Wolf, Gary. “Want to Remember Everything You’ll Ever Learn? Surrender to This Algorithm.” Wired, April 21, 2008. [read]

Manuals and Handbooks

Abergel, Ronni, Antje Rothemund, Gavan Titley, and Peter Wootsch. The Living Library Organiser’s Guide. Budapest: Council of Europe, 2005. [read]

Chinn, Cassie. The Wing Luke Asian Museum Community-Based Exhibition Model. Seattle: Wing Luke Asian Museum, 2006. [read]

IDEO. Human-Centered Design Toolkit. San Francisco: IDEO, 2008. [read]

Institute for Museum and Library Services. Museums, Libraries, and 21st Century Skills. Washington, DC: IMLS, July 2009. [PDF]

InterAct. Evaluating participatory, deliberative, and co-operative ways of working. Brighton: InterAct, 2001. [PDF]

Evaluation Reports and Impact Studies

Adams, Marianne and Jill Stein. Formative Evaluation Report for the LACMALab nano Exhibition. Annapolis, MD: Institute for Learning Innovation, September 2004. [PDF]

Biriotti, Maurice. An evaluation of the impact and outcomes for guides employed by Dialogue in the Dark. London, England: SHM, April 2007. [PDF]

Center for Advancement of Informal Science Education. Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. Washington D.C.: Center for Advancement of Informal Science Education, 2009. [PDF]

Cohen, Orna. Dialogue in the Dark: What are its consequences and how can they be proved? Hamburg, Germany: Orna and co., March 2006. [PDF]

Ellenbogen, Kirsten, Beth Janetzki, and Murphy Pizza. Summative Evaluation Report for Open House: If Walls Could Talk. St. Paul, MN: Science Museum of Minnesota, December 2006. [PDF]

Ellenbogen, Kirsten, et al. Formative Evaluation of Science Buzz 2005-6. St. Paul, MN: Science Museum of Minnesota, June 2006. [PDF]

Heritage Lottery Fund. A Catalyst for Change: The Social Impact of the Open Museum. London: RCMG, 2002. [PDF]

Istanbul Bilgi University Youth Studies Department. Assessment Results for the Living Library. Istanbul: August, 2007. [DOC]

Ma, Joyce. Formative Evaluation of Daisy: Eliciting Richer Conversations at a Chatbot. San Francisco, CA: Exploratorium, October, 2007. [PDF]

National Endowment for the Arts. 2008 Survey of Public Participation in the Arts. Washington D.C.: National Endowment for the Arts, 2009. [PDF]

Pine, Kylie, Erin Milbeck, Jaisa Halls, Julie Dougherty, and Alex Curio. Advice Exhibit Evaluation. Seattle, WA: University of Washington, June 2009. [DOC]

Ziebarth, E. K. Appreciating Art: A Study of Comparisons, an exercise in looking: at the Hirshhorn Museum and Sculpture Garden. Washington, DC: Smithsonian Institution, 1992.