• slidebg1
    KNOWLEDGE BROKERS
    An innovative game-based workshop for public sector professionals.
    Learn how to support decision-makers with research results
6

hours of an engaging learning experience

12

rounds of decision-making

5

knowledge brokering skills to master

3

mini-lectures combined wtih simulation



Loading the player...


How useful it is

Understand how the game of knowledge brokering can help your organization.

In a complex world, knowledge is needed to run effective public policies. However, getting the message across from studies to decision-makers is not a straightforward process.
Knowledge brokers equip decision-makers with useful knowledge. These are public professionals who perform the role of intermediaries, steering the flow of knowledge between knowledge producers - experts and knowledge users - decision-makers. Brokering requires specific skills.
Our training teaches public professionals the main skills of knowledge brokering:
  • Understanding the knowledge needs of different policy actors
  • Acquiring credible knowledge from research
  • Feeding knowledge effectively to users
  • Building an evidence-based foundation for public interventions
  • Managing an evaluation unit - time and human resources
Apart from gains in the five skills of knowledge brokering our game provides unique benefits for the personnel of public organizations.
For the personnel of analytical and evaluation units our game based-workshop:
  • Broadens their spectrum of research designs
  • Teaches an integrated approach to applied research - from need assessment, through design of the study, to communication and dissemination of results
  • Provides a holistic understanding of the place and role of expertise and analyses in the decision-making process
  • Increases their understanding of colleagues from management units - their needs, the conditions in which they operate and the dynamics of cooperation with them.
For the personnel of units managing public interventions (projects and programs) our game based-workshop:
  • Provides quick and effective learning of "the nuts and bolts" of applied research.
  • Raises their awareness of the importance of credible knowledge in the decision-making process.
  • Strengthens their cooperation with colleagues from analytical and evaluation units by enhancing mutual understanding.
Our workshop has been primarily designed for public professionals.
However students from advanced programs of public policy and public management can also benefit substantially from using the game towards the end of their curriculum.
The workshop can form a part of courses on public policy making, policy analysis, program evaluation, knowledge management, as well as courses on applied research methodology.
Students playing the game in the classroom:
  • Gain five useful skills of knowledge brokering in the public sector.
  • Test in practice their theoretical knowledge on applied research.
  • Gain insight into the dynamics of decision-making in public policies and the role of research findings in that process
  • Experience the real-life dynamics of working in a government analytical unit.
The most effective way of learning for an adult professional is learning by doing. Mistakes are an inevitable part of this process. In the public sector we can't afford mistakes that would cost taxpayers money. How then can we learn safely? We can do so through serious gaming and simulation.
Our game-based method offers the following advantages over traditional lectures:
  • It provides six hours of highly engaging learning experience for professionals.
  • It enables players to gain new professional knowledge on knowledge brokering, including approaches to research design and dissemination of study results.
  • It allows players to test new knowledge in practice and get immediate feedback.
  • It facilitates exchange of ideas, reflection and discussion on solutions for the practice of evidence-use in public policy.
  • It encourages teamwork and integrates practitioners from different institutions of government.

How it works

See what and how you will learn in our game-based workshop

During the game-based workshop you will learn the five key skills of an effective knowledge broker in public policy-making. These are:
  • Understanding the knowledge needs of different policy actors - key politicians, senior civil servants and project managers
  • Acquiring credible knowledge by matching optimal research designs to the evaluation questions and types of interventions that need to be studied
  • Feeding knowledge effectively to users by choosing the dissemination methods preferred by different decision-making actors
  • Building an evidence-based foundation for public interventions by combining the results of different studies into a concise policy argument
  • Managing an evaluation unit with limited time and human resources
The game-based training is designed for a group of up to 30 participants, as a one-day session (6.5 hours including breaks).
Time Content
10 min. Introduction to the workshop aim
30 min. Explanation of the rules of the game
20 min. Training round 1 and clarification
30 min. Rounds 2-3
10 min. Break
10 min. Presentation of partial results
30 min. First debriefing session: (1) mini-lecture on knowledge needs and research designs, (2) Q&A, (3) group internal deliberation
40 min. Rounds 4-6
30 min. Break
10 min. Presentation of partial results
20 min. Second debriefing session: (1) mini-lecture on types of users and feeding methods, (2) Q&A, (3) group internal deliberation
40 min. Rounds 7-10
15 min. Break
15 min. Presentation of final results and choice of the winners
30 min. Final debriefing session & take away points for real-life practice
30 min. Evaluation of the game – the learning process itself
The training method consists of three integrated elements: (1) the board-game sessions, (2) mini-lectures and (3) debriefings.
The Game. This allows participants to experience the real life of a knowledge broker and to test their own brokering skills in a safe environment. The game has been specially designed based on extensive research into the real life practices of evaluation units in Poland and the United States.
Mini-lectures. These provide participants with concepts on public policy and social research that are crucial for effective knowledge brokering. They include: stages of the policy cycle, research questions and research designs, policy actors and knowledge dissemination strategies.
Debriefings. These transfer the game experience back into the real world. Carefully animated sessions allow players to reflect on their strategies within the game. On that basis they draw lessons for practice in their organizations.
Participants of the workshop are divided into 6 groups. Each group manages an evaluation unit in a region for twelve rounds (1 round represents 1 month in real life). Their mission is to help decision-makers with expertise in implementing four different types of socio-economic projects. Each project is at a different stage of development and faces different challenges.
With each turn knowledge needs appear for each project. They can relate to diagnostic issues, the mechanisms or effects of the implemented or planned solutions, explanation of project problems or technical issues. Knowledge needs take the form of concrete questions.
Players have to react to knowledge needs by contracting out studies with an appropriate research design, targeting the key users of the study and choosing methods for feeding knowledge to users. Players have a wide range of options to choose from.
Players’ choices are determined by the available resources – i.e. the number of staff in their units and time required to complete each task. Players can be proactive and invest their resources in networking (to discover knowledge needs in advance) or archive searching (to find already existing studies).
After each turn, groups of players that have completed reports receive detailed feedback, in the form of a card with iconographics. It informs players how well they performed in terms of: knowledge timing, relevance, credibility and accessibility. They are also informed whether a policy actor made a decision based on delivered knowledge or other premises (e.g. political rationale).
Groups of players compete with each other. The winning team is the team with the highest score in terms of reports used by policy actors.

How reliable it is

Read about the research that provided the scientific basis for our workshop.

It all started with the knowledge needs expressed by professionals from evaluation units in Polish government. After ten years of building an evaluation system they noticed that the majority of evaluations produced were of little use in practice.
Polish civil servants started to discuss this problem in different forums with academic researchers and evaluators. Eventually we arrived at three questions:
1. Why does this problem occur?
2. What can we do to address it?
3. How can we learn a new approach?
Karol Olejniczak - a researcher from EUROREG - University of Warsaw, a practicing evaluator and a director of the Academy of Evaluation, teamed up with his colleagues from academia, government offices and research companies in the quest to find answers. To make a long story short we find out the following.
1. Why does this problem occur? The problem of the know-do gap is ubiquitous across all policy fields. Research results are only one of the factors in the complex process of policy decision-making. Utilization of studies can be increased when the configuration of factors triggers a positive response from knowledge users - decision-makers.
2. What can we do to address it? A promising solution is to turn evaluation units into knowledge brokers. Brokers serve as an intermediary between the worlds of research and policy-making practice.
3. How can we learn a new approach? To become knowledge brokers public professionals need to master a set of specific skills. The best way to learn them is through practice. A game-based workshop provides space for safe experimentation and blame-free learning by doing.
For the detailed findings of our research regarding each of the three questions - see the next sections of this webpage.
For more information about the Polish evaluation system see: Olejniczak, K. (2013). Mechanisms Shaping Evaluation System – A Case Study of Poland 1999-2010, Europe-Asia Studies, 65(8), 1642-1666 (file available on request from the author).
We explored the complex dynamics of the role of research in public policy and identified the key factors responsible for evaluation utilization.
We found out the following.
(1) Research evidence is only one of many factors influencing the decision-making process. Other factors are political rationality, organizational dynamics, as well as the characteristics and reasoning processes of the knowledge users.
(2) The chances of evidence use are increased by a combination of four factors:
  • Timing, that is, knowledge arrives at the right moment of the policy cycle, when users need it
  • Relevance of the delivered knowledge to the information needs of decision-makers
  • Credibility of knowledge determined by the quality of methodological rigor of the particular study
  • Accessibility of the knowledge, that is, the form of presentation and channel of delivery that matches the preferences of the targeted policy actor
(3) The research impact is higher when findings from single studies are combined into a stream of knowledge.

We established our findings based on the following sources: [click here] Nutley S. et al. (2007) book on how research can inform public policies [click here] Ferry, M. & Olejniczak, K. (2008) research on the use of evaluation in the management of EU programmes [click here] Two EGO s.c. benchmarking studies (2008 and 2010) on the performance of the network of evaluation units within the system of EU Cohesion Policy in Poland [click here] Prewitt, K. et al. (2014) report for US National Research Council on using science and evidence in public policy [click here] Findings from the MUS project (2014) on organizational learning in public administration [click here] The PhD study by Tomasz Kupiec (2015) on the utilization of evaluation findings in the strategic management of regional government [in press] A body of studies on evaluation utilization published in two leading journals: American Journal of Evaluation[click here] and Evaluation [click here]
The second stage of our quest was practical. We set out to find a solution for the know-do gap experienced by evaluation units working across government.
We found out the following:
(1) Recent studies on research utilization in public policies points to "knowledge brokering" as a most promising strategy. Evidence comes from different policy sectors: health, education, the environment, business and innovation.
(2) Knowledge Brokers are units in government that steer the flow of knowledge between the worlds of science, politics and the public interest. They help decision makers in acquiring, translating into practice and using existing knowledge for better planning and implementation of public interventions.
(3) The better the quality of brokers' activities and the stronger the evidence-base they present, the higher are the chances of the positive influence and eventual success of public interventions.
(4) Effective brokers need to perform five groups of activities: (a) understand the knowledge needs of different policy actors; (b) acquire knowledge, (c) feed it to users, (d) build an evidence-based culture in government and (e) manage own resources.
(5) The quality of the services of a knowledge broker is best measured by: the timing of delivered knowledge within the policy cycle, its relevance for knowledge users, the methodological quality of the studies, and accessibility of their form.
Our findings were based on:
A rapid-fire review on evaluation utilization, knowledge transfer and evidence use in policies (115 articles from Web of Science, 20 books on evidence use in public policies, literature on evaluation use).
A comprehensive survey in Polish evaluation units (n=57 evaluation units, response rate = 80%) and interviews with leading evaluation units (n=6)
A systematic literature review (SCOPUS database) covering 931 articles on knowledge brokering from peer-reviewed journals across 10 disciplines, followed by in-depth full content analysis.
Interviews conducted with evaluation units in a variety of organizations, spanning a number of US federal agencies and international organizations (n=5).
Focus groups, discussions and commentaries with Polish public sector practitioners and with international experts (European Evaluation Society - Dublin 2014 Conference, and American Evaluation Association - Denver 2014 Conference).

The full results are available in English, in paper by Olejniczak, K., Raimondo, E. & Kupiec, T. (2014). [click here] A summary presentation is available here: [here]
The empirical findings from the first two stages of our research allowed us to build conceptual models of: (a) the factors determining research use and (b) the key actions of knowledge brokers. This was good material for an academic lecture but... not for teaching professionals!
Research clearly indicates that adult professionals learn best through experiential learning - a combination of experience, reflection and experimentation (see: classic work by David Kolb [click here])

Thus we have chosen experiential learning in the form of serious gaming. Games are effective learning tools because they: (1) involve participants, (2) allow learning-by-doing in a safe environment, (3) combine hands-on experience with immediate feedback and more reflective discussion [for more information see the journal Simulation and Gaming available [here] and book by Kapp, K. (2012)[click here]
For details about game design see the paper by Olejniczak, K., Kupiec T., and Widawski I. (2015) on ISAGA 2015, available here [click here]

Who trusted us

Find out about the organizations that got immersed in our learning experience.

In July 2017 we met with directors of regional units of Polish National Statistical Office to explore new ways of reaching policy makers with research. Regional units play crucial role in supporting Marshall offices of 16 Polish regions in running effective EU Cohesion Policy programs. The game-based workshop helped us developing new strategies of feeding high quality analytical work into decision-making process.

This June 2017 we visited London City to assist the work of EBRD evaluation unit. We used the game to build the shared understanding among staff of different EBRD units about the utility of evaluation findings in their work. We also have been helping EBRD to incorporate in their current results framework the behavioral aspects of decision-making. The new, human-centered design of the projects should further increase effectiveness of new investments.

In June 2017 our workshop brought together key staff responsible for strategic planning and implementation of Cohesion Policy in Hungary. The initiative led by Prime Minister Office of Hungary aimed at building effective, evidence-informed cooperation networks and synergies in the complex system of Cohesion Policy implementation.

In April we played a game with MPA students at Trachtenberg School of Public Policy & Public Administration, The George Washington University. Thanks to the workshop students could experience real life challenges faced by the personnel of government evaluation units.

In November 2016 we have opened with our game session an elite study program developed by EUROREG – University of Warsaw and Ministry of Economic Development. Mission of the Academy is to educate Polish civil servants to be knowledge brokers who improve our public polices based on research evidence. Game was used as an introduction to the study program. Following the game experience students will have a whole year to explore in-depth the different aspects of knowledge brokers craft.

This summer, as in previous years, The University of Warsaw has used our game-based workshop at the Academy of Regional Development. This is the 11th edition of the post-graduate studies for public managers from the Polish regions responsible for development policies. They have learnt about role of research-based evidences in running public development programs.

In February 2017, the Ministry of Agriculture and Rural Development of Poland used our game as part of the project „Assessment of monitoring and evaluation system for Rural Development Program 2014-2020”. The workshop brought together leaders of local action groups from all around Poland. They were able to network, exchange experiences, and develop shared understanding of how research results can help improving local communities.

In February 2017, we visited Cracow. Polish National Science Centre has been working with EGO s.c. on developing an evaluation system for NSC scientific grant delivery activities. Thanks to the workshop NSC staff was able to develop ideas on the architecture of the evaluation and monitoring system that could support grant decision-making.

In November 2016 The National Coordination Authority’s Evaluation Unit in the Czech Ministry of Regional Development organized international conference on Evaluating European Structural and Investment Funds. We run the workshop with conference participants – professionals specialized in evaluating Cohesion Policy. We were finding strategies for making evaluation more timely, accurate and useful.

In June 2016, we conducted a workshop with the Marshall Office in the Silesian Voivodeship. The goal was to bring together staff of evaluation units and their colleagues from strategic and management units. Together they were learning how they could cooperate in order to improve the effectiveness of public programs based on research insight.

In May 2016, we were invited by the Marshall Office of the Kuyavian-Pomeranian Voivodeship to conduct a one-day workshop with the evaluation unit in Torun. During the sessions, players and experts shared their experiences and knowledge about the best evaluation and brokering practices that could be implemented in every-day work.

In March 2016 we were invited by PHAC to conduct in Ottawa a one-and-a-half day workshop with public policy professionals from the Office of Audit and Evaluation. On the first day, we explored dynamics of effective knowledge brokering. During the creative session on the second day, we designed a sample regulation using behavioral insight and discussed implications for evaluation business resulting from this new approach.

On 24 of November 2015 we played a workshop with group of public servants from City of Warsaw. City of Warsaw consists of over 35 central departments and number of offices in 18 quartiers of Warsaw. They deliver services to citizens and manage the growth of metropolis. In such a complex structure knowledge management is a challenge. The goal of our workshop was to build a network of experienced civil servants who would serve as knowledge brokers, transferring information and experiences between city offices and gaining knowledge from myriad of social partners that operate in our metropolis. During the workshop participants mastered skills of effective knowledge brokers. They network with each other and they also come up with number of ideas for evidence-based initiatives aiming at the improvement of city services.

The American Evaluation Association had chosen Knowledge Brokers for a professional development workshop at their annual conference: „Exemplary Evaluations in a multicultural world”. The conference was held in Chicago, from 9 till 14 of November 2015. Over five thousands participants came to share their experience in evaluation of public programmes and policies. Use of evaluation findings by decision-makers has been always a challenge for the community of evaluators. Thus, our workshops aimed at exchanging experiences of effective knowledge brokering strategies that increase evaluation use. Participants were experienced evaluators and professional staff of evaluation units from all around the world. During the dynamic full day session we learnt about credible research designs and methods of feeding evaluation findings into policy practice. A surprising finding that emerged from our discussions was that challenges we face in evaluation field are similar across the globe. Also ways of dealing with challenge of evidence use are similar across different cultural and organizational contexts. For more information about conference see: [click here] If you are interested in workshop follow-up materials and access to reader – please contact us at: k.olejniczak@evaluation.pl

On 24 of October 2015 we conducted a workshop with Marshall Office of Sub-Carpathian Region. Office has been preparing for the implementation of regional development program, worth almost 2,5 billion EUR. They decided that management of such a huge investment scheme should be evidence-based.

The goal of the workshop was to bring together analysts from different departments and develop with them knowledge-management system and list of possible research topics for the program. During the workshop participants designed research and learn the principles of knowledge management in public policies. An unexpected discovery for analysts who traditionally work alone was the value of communication and team work.

Our game-based workshop was chosen as an exemplary workshop for the opening of the 11th Polish Evaluation Conference, Cracow, September 27, 2015. The conference is a joint undertaking of the Polish Ministry of Infrastructure and Development and the Polish Agency of Enterprise Development. It brings together practitioners of applied public policy research from academia, research consultancies, ministries and Polish regions. A key challenge discussed this year on number of sessions was use of research results for the better implementation of Cohesion Policy programmes (investment package worth 77 billions EUR for years 2014-2020) Our workshop brought together practitioners from Polish ministries, regional government and academia. Together we have been testing effective knowledge brokering strategies for the better implementation of Cohesion Policy interventions. For more information about conference see:http://www.konferencja-ewaluacja.pl/2015/

The presentation and research paper on our game won first prize at the 46th Conference of International Simulation and Gaming Association, held this July in Kyoto.
To learn more about ISAGA click here (link)
To see the presentation click here (link)
To read the award-winning paper click here (link)

About us

The game-based training on knowledge brokering is a joint educational undertaking of a group of international experts on public policy and game design.

KAROL OLEJNICZAK is an assistant professor of management at EUROREG - University of Warsaw, Poland. His research interests focuses on evaluation of public interventions and organizational learning. Karol is a practicing evaluator. He conduced studies for Polish government and European Commission. In 2014, for his extensive applied research he has received from Polish Government a title of the Evaluator of the Year. Karol has 15 years of experience in teaching evaluation. He has been giving lectures in Poland, USA, Japan and Belgium. Since 2008 Karol is the head of the Academy of Evaluation, an elite study program for Polish civil servants.

LUKASZ KOZAK is a graduate of Institute of Sociology at Warsaw University, analytics and statistics specialist, passionate board gamer and technology geek. He is game designer, mechanics developer and algorithm specialist. His ambition is to create tools combining analogue experience with IT solutions. During the game-based workshop Łukasz operates algorithm that provides participants with real-time feedback.

IGOR WIDAWSKI is a head game designer at Pracownia Gier Szkoleniowych. Sociology graduate, he has studied at the University of Warsaw and School of Business & Economics in Maastricht. His games have been already played by thousands of players within corporations, NGOs, public administration and universities. He has designed simulations related to such subjects as project management, negotiations, brand building, public participation, sustainable energy, change management and knowledge brokering. Igor is the chief game designer at Competence Game: an innovative on a global scale competence diagnosis tool in the online environment. During game-based workshop he is game master coordinating game flow.
EGO s.c. - Evaluation for Government Organizations. We help the public administration to implement public policies that are effective in stimulating socio-economic development and improving the lives of citizens. We are a Poland-based research company specialized in the evaluation of public interventions - regulations, programmes and development projects. We have the privilege of working with a number of public sector institutions: the Chancellery of the Prime Minister of Poland, the Ministry of Infrastructure and Development, the Ministry of Culture, the Ministry of Foreign Affairs, the Polish Agency of Enterprise Development, the marshal offices of Polish regions, the Polish Academy of Science and the European Commission. Our approach is well grounded in robust social research methodology. In our projects we always innovate and seek the next frontier of useful research approaches - system thinking, complexity theories, games and simulation, behavioral economics. But at the end of the day, we communicate our findings in the plain language of practice. For more information visit us at: www.evaluation.pl
PGS Pracownia Gier Szkoleniowych Sp. z o.o. Our mission is to trigger change in teams and organizations through the use of games and simulations. Founded in 2004, we were Poland’s first provider of development programs based on management games and business simulations. In 2011 we created Experience Corner – the first online store in Poland to provide games and interactive tools for trainers, consultants and academics. We incorporate management games and business simulation methods into the design of modern training, educational and integration experiences, supporting organizations in their change processes and skills development programs. With over 50 games and simulations illustrating contemporary organizational challenges, we offer the widest range in Poland. Our approach is built on know-how and expertise inspired through our membership in the International Simulation and Gaming Association. Our design adheres to international standards and proven organizational and management theories underpin our training and debriefing models.
This game was developed over a two-year period. We would like to thank all those involved at different stages of the undertaking.
We would like to thank Igor Widawski, Łukasz Kozak and Jakub Wiśniewski for their great work on the game design that includes game mechanisms, algorithms and... the magic fun factor!
We give our thanks to a number of colleagues from the Polish National Evaluation Unit and the Polish Agency of Enterprise Development who inspired us to research evaluation use in public policy. Our special thanks go to Agnieszka Haber and Piotr Strzęboszewski.
We express our appreciation to Estelle Raimondo (George Washington University) and Tomasz Kupiec (EGO s.c.) who coauthored an empirical study on knowledge brokering.
We extend our thanks to the group of participants from ISAGA Summer School Delft 2014 who contributed to the initial design of the game – Stephan van Ljperen, Wouter van der Horst, Marijn Knulst, Stefania Munaretto, Nicole Plasschaert.
Out thanks also go to a number of Polish civil servants and colleagues who participated in the testing of the game and provided us with most valuable feedback. Our special thanks go to: Izabela Grabowska, Agnieszka Haber, Katarzyna Hermann, Jakub Jąkalski, Tomasz Kupiec, Zuzanna Popis, Maciej Szałaj, Paweł Śliwowski, Rafał Trzciński and Dominika Wojtowicz.
We would like to acknowledge the great work of Janek Witkowski who is responsible for the board-game aesthetics, the team from RDS Group for their web page and Tomasz Zuchniewicz for the overall consistency of the graphic design.
Last but not least, we would like to thank our students from EUROREG - University of Warsaw for sharing with us the fun of a serious game.
Karol Olejniczak
and Bartosz Ledzion
EGO s.c.
Jagoda-Gandziarowka Ziołecka
and Joanna Średnicka
PGS sp. z o.o.
Karol Olejniczak
E-mail: k.olejniczak@evaluation.pl
Mobile: +48 696 41 22 82