Skip to main content

Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective

Abstract

Community-based obesity prevention programmes are considered an important strategy to curb the obesity epidemic. The JOGG (Youth At a Healthy Weight) approach is a large-scale community-based programme for childhood obesity prevention in the Netherlands that has been implemented over the past ten years. Practice-based development of the programme, both at the national and local level, increasingly poses challenges for its evaluation. One considerable challenge is the increasing acknowledgement of the complexity in the JOGG-approach, characterized by (a) objectives that vary locally, (b) adaptions to the programme over time in response to a community’s shifting needs, challenges and opportunities, and (c) emergent outcomes and non-linear causality.

We propose an evaluation framework that highlights elements of the complex local practice, including the local programme theory, implementation, adaption, the influence of context and feedback loops and intended as well as emergent and unintended outcomes. By studying each of these elements in practice, we hope to learn about principles that guide effective obesity prevention across contexts. The results of the proposed evaluation will inform both practice and research.

Considering complexity in evaluation is a relatively new challenge in public health and therefore an emergent research area. The proposed framework for complex evaluations allows to retrospectively evaluate a programme that was implemented and developed in practice, and enables us to learn from practice-based experiences. Following the ISBNPA Dare2Share initiative, we kindly invite other researchers in the field to share their ideas and experiences regarding integration of complexity in evaluation.

Background

Worldwide, obesity is considered a pressing health concern, with immense consequences for population health and economy [1]. Over the past decades obesity and overweight prevalence has continued to rise [2, 3]. In order to stop this increasing trend, a large variety of programmes have been introduced to promote healthy dietary and physical activity behaviours and consequently prevent obesity [4,5,6,7,8].

The strategy of obesity prevention programmes in the population strongly evolved over the past decades. Early strategies were aimed at individual behaviour change [9,10,11]. Informed by socio-ecological frameworks of behaviour, later strategies of obesity prevention acknowledged the importance of environmental factors as drivers of unhealthy behaviour [12, 13]. Consequently, many prevention programmes implemented multiple interventions aimed at both individual and environmental drivers of obesity in multiple settings in which people live (e.g. neighbourhoods and schools). Community-based obesity prevention (CBOP) programmes [14] became a popular strategy targeting drivers of unhealthy behaviour at the local level, in which local stakeholders play an important role. Interventions in CBOP programmes are developed with and implemented by community stakeholders.

In response to the growing awareness of the complexity of obesity prevention [15], the design of CBOP programmes became increasingly complexity-oriented over the past years [16]. In this perspective, complexity refers to the context in which a programme is implemented, rather than (only) a property of the programme itself [17,18,19,20]. This context is considered a complex system of interdependent and constantly evolving elements. Within this complexity perspective it is acknowledged that the outcomes and impact of a programme are likely to vary between contexts and emerge as the programme interacts with the context. This means that even apparently simple programmes and interventions can result in varying outcomes in different local contexts [17]. The extent to which this complexity perspective is integrated within CBOP programmes ranges from dynamic programme adaptions in response the community’s changing needs, possibilities and challenges, to mapping and leveraging the characteristics of the context in order to change the whole obesity driving system [5].

Appropriate study designs for the evaluation of complex programmes are needed. Until recently, evaluation designs for CBOP programmes were commonly (quasi-) experimental designs that solely focus on the effect of the programme or specific programme components on health behaviours or obesity [21]. Experimental evaluation designs pose several challenges for complex CBOP programmes [20, 22,23,24], including a strong focus on predetermined and linear causal outcomes and processes, and to a lesser extent on programme development over time and context. As a consequence, much is still unknown about how CBOP programmes impact communities and obesity prevalence, and which mechanisms underly success or failure of such programmes [5, 7, 21].

In order to address these challenges and support future development of CBOP programmes, an evaluation approach is needed that considers complexity in the evaluation [20, 24, 25]. This approach requires novel types of research questions, a shift in thinking about how a programme contributes to behaviour change and obesity prevalence, and different (application of) research methods to study process and impact. So far, evaluation designs that consider complexity in its research questions and approach have been relatively uncommon in public health evaluation [20, 26, 27]. Some alternative approaches that do provide insight in working mechanisms of complex programmes and the interaction with local context, have been developed in other disciplines, including realist evaluation [28], developmental evaluation [29] and systems approaches [30, 31]. To the best of our knowledge, their use in the evaluation of CBOP programmes is still limited [21, 26].

In this paper we use a practical example of a CBOP programme to explain why alternative approaches to evaluation are sometimes more suitable than commonly used effect evaluations. In addition, we present and share our evaluation approach that considers the complexity of CBOP and actively invite other researchers to deliberate on this. By doing this we aim to contribute to the debate on how alternative evaluation approaches and methods can be applied in the evaluation of complex public health programmes.

We developed the evaluation approach presented in this paper for the evaluation of a national CBOP programme in the Netherlands, the JOGG (Youth At a Healthy Weight) approach, which has been implemented and developed in practice over the past ten years. Evaluation of this programme aims at gaining a better understanding of how the programme successfully contributes to obesity prevention in practice.

Community-based obesity prevention: the JOGG-approach

Background and core components of the JOGG-approach

The JOGG-approach was based on the French EPODE programme, which had showed promising results for CBOP [32, 33]. EPODE describes four core components that should be implemented in collaboration with community stakeholders: i) generating political commitment, ii) establishing public–private partnerships, iii) using social marketing techniques and iv) monitoring and evaluation [32, 34]. Each local JOGG-community uses these core components to support community engagement and collaboratively create a more ‘healthy environment’, thereby stimulating children’s healthy lifestyle in order to prevent childhood overweight and obesity. Community stakeholders are to be involved even they do not have a specific objective in public health (e.g. spatial planning sectors in local government or restaurants) with the premise that health should be considered in all policies. See van Koperen et al. [34] for the full logic model of the EPODE programme. The JOGG-organisation added the component ‘connecting prevention and care’, to encourage JOGG-communities to link interventions and stakeholders involved in the care for overweight children to overweight prevention efforts in the community.

Much is still unknown about how and for whom these EPODE-based or similar CBOP programmes work. Recent studies show mixed results [7, 8, 35,36,37], which may be due to the large heterogeneity between the programmes, the diversity of contexts in which they were implemented, and heterogeneity in evaluation designs and research methods. The core components of EPODE were at the time developed based on existing theories [38] and practice [34]. Few studies have since then focussed on further studying the core components or causal links between the core components and intended outcomes [21, 39].

Organisation structure: a national JOGG-organisation and local JOGG-communities

The JOGG-approach is planned and implemented both at the national and local level, which enables influencing both macro-environmental structures (e.g. policies within the national sports sector or food sector) as well as local environmental settings (e.g. in schools and local government). The national JOGG organisation, funded by the Dutch government, creates awareness about childhood overweight and a healthy lifestyle, develops partnerships, lobbies and implements actions at the national level. In addition, the national organisation is responsible for the strategic development of the local JOGG-approach.

At the local level a JOGG-team, consisting of a local JOGG-manager and policy officer, coordinates the development and implementation of the JOGG-approach. A local action plan is developed that fits the local context, challenges and possibilities (for example local demographics or political environment). This action plan is developed in collaboration with community stakeholders (e.g. schools, welfare organisations, sports clubs or private partners).

JOGG-communities are supported by the national organisation. Support structure includes personal support from JOGG advisors and coaches, ready-made communication and campaign materials, road maps and knowledge exchange in an online portal, training sessions and network meetings. The national organisation does not provide financial support, the local JOGG-team is responsible for mobilizing their own (financial) resources.

Development of the JOGG-approach: 10 years in practice

The JOGG-approach was first implemented in 2010 in the Netherlands as part of a national healthy weight agreement, a partnership between government, private and public partners, initiated by the ministry of Public Health, Welfare and Sport. To date, JOGG is an independent organisation and more than 170 Dutch municipalities are implementing the JOGG-approach.

Since the start in 2010, the national JOGG-organisation has continued to develop the JOGG-approach based on (1) practice-based experience in different communities, (2) knowledge exchange within the EPODE international network, and (3) collaboration with scientific partners. Over the years, the JOGG-approach developed from a multi-component programme into a CBOP programme, that builds on a socio-ecological model of health, community capacity building and recognizes the complexity of the context in which it is implemented. These changes reflect the development of CBOP programmes overall.

From targeting children individually to targeting children’s environment

Over time, the national JOGG-organisation gradually shifted focus from targeting the individual child to targeting children’s environment. The first years, the JOGG-approach mainly included implementing standardized social marketing materials (e.g. campaigns, events and activities that raised awareness and knowledge about health behaviours) in order to accomplish behavioural changes at the individual level. Later, the national JOGG-organisation started focussing on identifying and targeting environmental properties that drive children’s health behaviours at the local level such as schools food and drink policies and neighbourhood playgrounds. JOGG removed the core component ‘using social marketing techniques’ from their programme logic model. Creating the healthy environment became the core aim of the approach.

To date, the JOGG-organisation defines seven settings in children’s environment: family, school and child care, neighbourhood, sports clubs, leisure time settings (e.g. playgrounds or parks), work and media. The national JOGG-organisation recognises different levels of influence within these settings (see Fig. 1), which correspond with existing ecological models for health promotion [40]. The first level is the individual level: children’s behaviour, awareness, knowledge, attitudes and skills. Children’s direct social connections, for example family, peers and professionals (e.g. teachers) form the second level. The third level includes the formal and informal organisations which are present in children’s environment (e.g. schools, health services, voluntary agencies or religious organisations) and their policies and practices. The fourth level, the community level, is the community structure in which these organisations operate. For example the role of each organisation in the community, resource distribution, relations and collaborations between these organisations. This structure influences practices within the organisations, for example whether children at risk for obesity are identified and referred to fitting care. Lastly, the community and its organisations operate in a wider context of local policies and laws, the fifth level. Practices and processes on each level are inherently related and are ideally considered collectively when targeting children’s environment.

Fig. 1
figure 1

Settings and levels in the healthy environment targeted by the JOGG-approach

From collaboration in partnerships to community capacity building

From the first years onwards, stakeholders in the community (e.g. schools or food retailers) were involved in the implementation of activities, interventions and campaigns planned by the local JOGG-team [41]. Over the past five years, the JOGG-approach focus shifted to more structural community involvement through community capacity building, as shared ownership was added as a new core component in the approach. Community capacity building includes creating awareness and shared ownership of a problem in the community and providing stakeholders with support, skills and resources to develop solutions and take action together [42,43,44]. By building capacity among enthusiastic leaders within local stakeholder organisations in children’s environment, enthusiasm and ownership is spread, and thereby capacity for action can diffuse through whole organisations. Ideally, community stakeholders do not solely support or implement actions, they initiate, develop and manage actions from or within their own organisation, based on their insights and experiences. This ensures that the approach fits the local needs and challenges and that actions are sustained. Implementation of the JOGG-approach by the JOGG-team changed from implementing activities and interventions to community capacity building: creating a network of community stakeholders, encouraging and facilitating them to take action and maintaining an overview of the network and its (collaborative) actions to ensure coherence and contribution to the overall goals of the local approach.

A growing recognition of complexity

The EPODE logic model on which the JOGG-approach was based [34] illustrates a linear theory of behaviour change. Over time, the JOGG-approach integrated a growing recognition of complexity: it appeared to be not only tailored to the local context in the design phase of the local programme, it also adapted over time. The local JOGG-team is increasingly encouraged to be responsive to changing needs, opportunities, challenges and (unintended) outcomes and to adapt the approach as needed [41]. As a consequence the local JOGG-approach is continuously changing and new objectives and actions emerge over time.

In line with this, the complex causality between the JOGG-approach, the context in which it is implemented and changes they aim to bring about throughout different settings and levels of children’s environment is recognised by the national JOGG-organisation [45]. The stakeholders and organisations within these settings are not passive or isolated recipients of the JOGG-teams efforts. They exist within a wider context and interact with each other. Consequently, the causal pathway from planning and implementation to individual-level effects consists of many, unpredictable and sometimes long mutually reinforcing and non-linear cause-effect chains, that may differ across contexts.

Evaluating the JOGG-approach

An appropriate evaluation approach

As is the case with many other CBOP programmes [5, 7, 21], evaluations of the JOGG-approach have so far consisted of summative and formative evaluations, with research questions mainly directed at monitoring implemented activities, evaluating whether programme components were implemented as intended [46, 47] and evaluating whether the programme worked to accomplish the predefined outcomes, specifically on overweight and obesity prevalence [48,49,50]. Such evaluation approaches are less appropriate for the current evaluation, which aims to gain a better understanding of how the programme works in practice. First of all, because these evaluation approaches produce limited information on how the programme works. The main focus on predetermined outcomes and programme components hardly leaves room for building an understanding of why a local programme’s implementation is or is not successful and learning from practice-based programme development.

Second, past evaluations have done little justice to the inherent complexity of the JOGG-approach. Specifically, programme development at the local level and necessary adaptions during implementation were not acknowledged in evaluation. An evaluation approach and corresponding research questions that values adaptive implementation as a key component of the JOGG-approach and enables learning from local successes and failures is more suitable.

Further, the focus on solely predetermined outcomes and programme components is not sufficient. Due to the adaptive nature of the programme and varying contexts in which it is implemented, it may focus on issues or objectives that are no longer relevant and overlook unanticipated programme developments or outcomes. A broader perspective is required to consider all potential consequences of the JOGG-approach, on different levels in the community, including emerging or unintended outcomes and non-linear causal relations.

Evaluation framework

As a foundation for the evaluation of the JOGG-approach we use the conceptual model of Jolley [23]. This model addresses the challenges of evaluating complex community-based health promotion, including the role of evaluation to support future development of a program, by building on theory based evaluation [51, 52], complexity theory [18, 53] and developmental evaluation [29]. The model provides a framework in which the JOGG-approach is recognised as a complex and adaptive process, regardless of the complexity of the programme itself (Fig. 2).

Fig. 2
figure 2

Evaluation framework of the JOGG-approach, adapted from Jolley [23]

The planning phase in the model specifies the overarching JOGG programme as intended by the national JOGG-organisation, which is based on scientific evidence and practice-based knowledge. In each JOGG-community (local practice) the national programme theory is tailored to the local context and resources and is then implemented. By ‘implementation’ in this framework we mean the implementation strategies [54], all methods or techniques, that are employed by the JOGG-team to bring the programme theory into practice, in order to reach intended outcomes. This is considered a continuous and adaptive process, influenced by external factors and feedback loops. Evaluation supports both local and national developments of the JOGG-approach. Rapid feedback to stakeholders supports the adaptive implementation process at the local level. In accordance with Jolley’s model, the evaluation aims to capture perspectives of all stakeholders involved in the local approach (e.g. policy officers, private partners, professionals, target groups). Their experience and knowledge is considered a valuable source of information about the implementation process and outcomes of the local JOGG-approach. Evaluation results from the local level will feed back to the national level to support further programme development through practice-based knowledge. Compared to Jolley’s original model we added the ‘validation’ arrow from practice-based knowledge to evidence. When sufficiently validated, lessons from practice fill the ‘evidence’ bucket on which programme theories of CBOP programmes are based. Evaluation studies across different context contribute to the validation of practice-based knowledge [23, 25, 29].

Research questions, design and methods

Research questions, design and methods for the evaluation (Table 1) are derived from the components of the model in Fig. 2 and its underlying theories. The evaluation of the JOGG-approach will consist of two phases. Phase 1 will focus on collecting knowledge from local practice. The second phase of the evaluation will then focus on validating practice-based knowledge.

Table 1 Research questions and methods for the evaluation of the JOGG-approach

Phase 1 practice-based knowledge

Following the planning and implementation process in Fig. 2, the first research questions in phase 1 of the evaluation focus on the feasibility and validity of the JOGG-approach’s programme theory at both national and local level. Research questions include whether the national and (explicit or implicit) local programme theories are feasible, logical, likely contribute to childhood obesity prevention, based on the best available evidence and knowledge and what context and resource factors influenced the local translation of the national programme theory. The answers to these questions will provide insight in whether national programme theory successfully guided programme planning in local practice and whether a local programme is likely to yield results.

After this first step we address what lessons can be learned from local implementation and adaption and how they were influenced by external influences and feedback. Research questions appropriate for this step focus on what internal and external factors influenced the implementation process and in what way the local programme was adapted in response to emergent issues. The answers to these research questions will help us to unravel what (combination of) elements of the adaptive implementation process (e.g. adjustments to the approach, responses to internal or external factors) contribute to successful implementation and eventually outcomes in the community.

In order to gain insight in the impact of the JOGG-approach on the ‘healthy environment’ in the community, an essential aim in the national programme theory, the research questions focus on identifying outcomes on different levels in the healthy environment (policy, community, organisational). Intended outcomes as well as unintended and emergent outcomes are taken into account. Intended outcomes are the goals as defined in the national and local programme theory. Emergent outcomes are outcomes that emerge in response to external factors or during the implementation process, for example when new goals are developed through collaboration with stakeholders [55]. Unintended outcomes can for example occur when the response to a programme depends on the context [18]. Since the JOGG-approach aims for structural impact in the community, we specifically focus on structural outcomes of the JOGG-approach.

In the last step in phase 1 of the evaluation we focus on the extraction of practice-based principles for effectiveness, those elements of the local JOGG-approach and adaptive implementation process that contributed to the accomplished outcomes in the community. Taken together, the answers to the research questions in phase 1 provide insights in the success of the adaptive planning and implementation process, the external factors that played a role and the outcomes that were accomplished through this process. In this last step we critically review the JOGG-approach’s contribution to these outcomes. The principles of effectiveness that are identified in this phase of the evaluation fill the ‘practice-based knowledge bucket’ from which the national JOGG-organisation informs their programme theory. Validating these principles across contexts is part of phase 2 in the evaluation.

Design and methods

In order to answer the research questions in phase 1, we use a multiple case study design. Such a design is suitable for studying dynamic programme implementation and development within the real-life context, from multiple perspectives [56]. The large number of communities using the JOGG-approach makes it possible to do multiple case studies, considering one community as a case, and to identify principles for effectiveness that work across a range of contexts. In the multiple case studies we will use qualitative methods to answer the research questions. Qualitative methods are well suited to collect new information on what happens and why [57]. Iteratively, qualitative data will help us understand how the programme unfolded in a community and how it contributed to outcomes [58].

Research questions regarding the local programme theory will be answered applying document analysis of local programme documents and interviews with the local JOGG-team. Then, two recently developed methods will be applied to shape data collection and analysis to answer the research questions regarding implementation, adaption, external influences and outcomes, i.e. Ripple Effects Mapping [59] and the Critical Event Card [60, 61]. Both methods involve the participation of relevant stakeholders by sharing their knowledge and experiences, thereby providing insight in the complex interplay between implementation, context and outcomes. The Critical Event Card tool specifically provides insight into how and why the local JOGG-approach developed over time [60, 61], through the analysis of the critical events that characterize the evolution of the complex and adaptive programme. Ripple Effects Mapping [59] focusses on the outcomes of the JOGG-approach. This method was specifically designed to evaluate both intended and emergent outcomes of community-based programmes, across different levels in the community. It involves a group session in which different community-stakeholders participate to provide their perspective on the outcomes and collaboratively explore the contribution of the JOGG-approach.

Analysis: explanation building and principles for effectiveness

From each case community, practice-based principles for effectiveness will be formulated: the underlying principles that contributed to effectiveness in the local context and may provide guidance for effective action in other contexts [23, 29]. These can be formulated through the process of explanation building, which constitutes building a causal explanation about how and why certain outcomes have occurred in a case [62]. The answers to the research questions in the framework regarding local programme theory, implementation, adaption, outcomes and context, resources, external influences and feedback loops at play in each case study help build this explanation. Explanations can be visualized using a systems-based logic model [63,64,65], which provides insight into the complex causal relations between programme components, outcomes and their interaction (with context). By iteratively comparing explanatory propositions and the logic model from a case with findings from new cases, we expect to be able to gradually refine the model, for example by adding or unpacking components or processes [63].

The principles for effectiveness form a set of hypothesis that can be validated in the second phase of the study. Principles should eventually provide simple and pragmatic guidance for action [66] for effective community-based obesity prevention across contexts [29]. This means identifying those (combinations of) programme components that underly significant and/or structural impact in communities, for example because they set into motion a reinforcing feedback loop. To be generalizable across contexts, principles should describe the function of a programme component or action, rather than the exact form [18, 67,68,69]. For example, rather than formulating the action ‘organising stakeholder meetings’ a principle should provide insight in the function of this action, for example ‘building a trusting relationship with and among community stakeholders’. The exact formulation of principles for effectiveness remains to be debated and we believe it is relevant to further discuss, share ideas and experiment with different approaches. Principles could for example be formulated as simple guidance for action (e.g. ‘ensure visibility of stakeholders’ successes to other stakeholders’) or include more complex hypothesis on the causal relations between programme components, context factors and outcomes (e.g. as a causal loop diagram or parts of the systems-based logic model).

Phase 2 validating principles for effectiveness

The second phase of the study focusses on the validation of the formulated principles for effectiveness. In this phase, additional qualitative or quantitative data may be collected. Principles may be validated using more traditional summative evaluation questions and methods. The decision on the specific methods for the second phase of the evaluation will be taken depending on the results of phase 1. The evaluation approach itself is in this way responsive to the emerging understanding of the programme through evaluation [23, 29, 68]. It remains to be debated how to adequately consider complexity in this phase of the study, especially when applying more traditional summative methods for evaluation.

Discussion

We have proposed an evaluation approach for the JOGG-approach, a large scale community-based programme for childhood obesity prevention in the Netherlands. The evaluation of comparable programmes has been challenging, given their complexity. Which is characterized by objectives that vary locally, adaptive programme developments over time, emergent outcomes and non-linear causality. We argued that with the aim to learn about the programme components that successfully contribute to obesity prevention, the focus on only predetermined programme components and outcomes in evaluation is insufficient. The current evaluation proposal therefore focusses on the whole adaptive implementation process. The presented evaluation framework includes research questions that focus on the national and local programme theory, implementation, the interaction with local context, both intended and unintended outcomes and adaption. By focussing on these components, using complexity sensitive qualitative research methods (e.g. Ripple Effects Mapping), we aim to extract practice-based knowledge about effective principles of the programme. We expect that these principles can be used to further develop the programme and further research can focus on validating these principles for effectiveness.

The framework will be applied retrospectively, which comes with some limitations. Indeed, it is subject to several threats to the validity, including recall bias in stakeholders and selection bias due to poor documentation or staff turnover [62]. Clearly, the risk of bias increases when the period covered in the framework increases. As some communities have implemented the JOGG-approach for over ten years, this may result in spurious causal inferences about the contribution of the JOGG-approach to outcomes in the community. In order to mitigate potential biases, several methodological elements are incorporated in the framework. First, combining data sources and methods (document analysis and interviews with different and purposively selected stakeholders) aims to increase the accuracy, as different perspectives on the same event they can be checked against each other [62]. Second, the use of timelining within the interviews helps situating participants in the past [60] and the use of the Critical Event Card tool within analyses allows to separate material evidence of events from stakeholders’ experiences and compare these [61]. Further, when identifying principles for effectiveness, cross-case analysis and explanation building are important analytical techniques to make evidence based and correct causal inferences in case studies [62]. The second study phase is explicitly aimed at validating formulated principles for effectiveness.

Despite the limitations of retrospective research, the application of the proposed framework provides valuable information. Given the fact that much is still unknown about the factors underlying success and failure of such programmes [5, 7, 21] and the impact and sustainability of these programmes in the long run [70], the expected results of the proposed evaluation, i.e. the principles for effectiveness, address some highly relevant gaps in the body of evidence of CBOP programmes. The evaluation of the JOGG-approach, which has been implemented in many different community contexts over the past 10 years, provides a unique opportunity to unravel these factors and their complex interplay (with context).

The application of a complexity perspective on the evaluation of public health programmes and interventions is not new. Over the past years different approaches and methods have been developed and applied [26, 30], for example system mapping and modelling, network analysis and the use of a complexity perspective as a lens in qualitative data analysis [58]. The proposed evaluation of the JOGG-approach however, does differ from what has been developed so far in several ways. First, existing complex evaluations mostly build on the premise that the programme was or will be designed with a complexity perspective in mind (e.g. [71, 72]). Like many other CBOP programmes however, the JOGG-approach was originally designed as a ‘simple’ programme, in which over time a greater acknowledgement of complexity was integrated. The proposed evaluation provides an opportunity to evaluate the JOGG-approach, a programme that was not specifically designed from a complexity perspective. The proposed evaluation approach may therefore be relevant for many other CBOP programmes.

Second, most existing evaluation methods and approaches that consider the complexity perspective, require to follow changes in a community system from before the start of the programme to draw conclusions about system changes over time. However, as programmes and policies are also developed in practice, without involvement of researchers in its development and implementation, other methods are also needed [25]. The proposed evaluation of the JOGG-approach is suitable to evaluate a programme that, like the JOGG-approach, was developed in practice and learn from the adaptive implementation of the programme.

Third, existing complex evaluation methods and approaches that consider the complexity perspective rarely facilitate the translation of the findings to programme planners and decision makers [26]. The proposed evaluation of the JOGG-approach is specifically aimed at generating results that inform programme planners, policy makers and health promotion practitioners to further improve the programme as well as scientist to inform the development and implementation of other CBOP programmes.

Conclusion

The evaluation proposal described in this paper takes into account the complexity of a CBOP programme and is designed to provide insight in how the programme works. Such approaches to public health evaluation have not been given much priority in the past, and only limited guidance exists in how to apply such perspective [27]. By sharing our evaluation approach we aimed to contribute to the necessary debate and progress in this area.

Our approach is yet to be tested, and requires further specifications and developments, for example on how principles for effectiveness can best be unravelled and validated. We therefore call on other researchers to share their ideas, trials, reflections and lessons. The evaluation of complex CBOP programmes, or complex health promotion in general is not easy and relatively new, sharing ideas, approaches and lessons will support the development of appropriate and feasible evaluation approaches and methods.

Availability of data and materials

Not applicable.

Abbreviations

CBOP:

Community-based obesity prevention

JOGG (Jongeren Op Gezond Gewicht, Dutch):

Youth At a Healthy Weight

References

  1. OECD. The heavy burden of obesity. Paris: OECD Health Policy Studies, OECD Publishing; 2019.

    Book  Google Scholar 

  2. World Health Organization. Monitoring health for the SDGs, sustainable development goals. Geneva: World Health Organization; 2020.

  3. United Nations Children’s Fund, World Health Organization, World Bank Group. UNICEF/WHO/The World Bank Group joint child malnutrition estimates: levels and trends in child malnutrition: key findings of the 2021 edition. 2021. Available from: https://www.who.int/publications/i/item/9789240025257

  4. Bemelmans WJE, Wijnhoven TMA, Verschuuren M, Breda J. Overview of 71 European community-based initiatives against childhood obesity starting between 2005 and 2011: general characteristics and reported effects. BMC Public Health. 2014;14(1):758.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Bagnall A-M, Radley D, Jones R, Gately P, Nobles J, Van Dijk M, et al. Whole systems approaches to obesity and other complex public health challenges: a systematic review. BMC Public Health. 2019;19(1):8.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Bleich SN, Segal J, Wu Y, Wilson R, Wang Y. Systematic review of community-based childhood obesity prevention studies. Pediatrics. 2013;132(1):e201-10. https://0-doi-org.brum.beds.ac.uk/10.1542/peds.2013-0886.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Ewart-Pierce E, Mejía Ruiz MJ, Gittelsohn J. “Whole-of-Community” obesity prevention: a review of challenges and opportunities in multilevel multicomponent interventions. Curr Obes Rep. 2016;5(3):361–74.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Wolfenden L, Wyse R, Nichols M, Allender S, Millar L, McElduff P. A systematic review and meta-analysis of whole of community interventions to prevent excessive population weight gain. Prev Med. 2014;62:193–200.

    Article  PubMed  Google Scholar 

  9. Campbell K, Waters E, O’Meara S, Kelly S, Summerbell C. Interventions for preventing obesity in children. Cochrane Database Syst Rev. 2002;2:CD001871.

    Google Scholar 

  10. Summerbell CD, Waters E, Edmunds L, Kelly SA, Brown T, Campbell KJ. Interventions for preventing obesity in children. Summerbell CD, editor. Cochrane Database Syst Rev. 2005 Jul 20;2005(3 Art. No.: CD001871). https://0-doi-org.brum.beds.ac.uk/10.1002/14651858.CD001871.pub2

  11. WHO. Obesity: preventing and managing the global epidemic: report of a WHO consultation. Geneva: World Health Organization; 2000.

  12. Swinburn B, Egger G, Raza F. Dissecting obesogenic environments: the development and application of a framework for identifying and prioritizing environmental interventions for obesity. Prev Med. 1999;29:563–70.

    Article  CAS  PubMed  Google Scholar 

  13. Swinburn BA, Sacks G, Hall KD, McPherson K, Finegood DT, Moodie ML, et al. The global obesity pandemic: shaped by global drivers and local environments. Lancet. 2011;378(9793):804–14.

    Article  Google Scholar 

  14. World Health Organization. Population-based approaches to childhood obesity prevention. Geneva: World Health Organization; 2012. p. 233–40.

    Google Scholar 

  15. Finegood DT, Merth TDN, Rutter H. Implications of the Foresight Obesity System Map for Solutions to Childhood Obesity. Obesity. 2010;18(S1):S13-6. https://0-doi-org.brum.beds.ac.uk/10.1038/oby.2009.426.

    Article  PubMed  Google Scholar 

  16. Swinburn BA, Kraak VI, Allender S, Atkins VJ, Baker PI, Bogard JR, et al. The global syndemic of obesity, undernutrition, and climate change: the lancet commission report. Lancet. 2019;393(10173):791–846.

    Article  PubMed  Google Scholar 

  17. Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281–3.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3–4):267–76.

    Article  PubMed  Google Scholar 

  19. Petticrew M, Knai C, Thomas J, Rehfuess EA, Noyes J, Gerhardus A, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Heal. 2019;25(4):e000899.

    Article  Google Scholar 

  20. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4.

    Article  PubMed  Google Scholar 

  21. Karacabeyli D, Allender S, Pinkney S, Amed S. Evaluation of complex community-based childhood obesity prevention interventions. Obes Rev. 2018;19(8):1080–92.

    Article  CAS  PubMed  Google Scholar 

  22. Dooris M. Healthy settings: challenges to generating evidence of effectiveness. Health Promot Int. 2005;21(1):55–65.

    Article  PubMed  Google Scholar 

  23. Jolley G. Evaluating complex community-based health promotion: addressing the challenges. Eval Program Plann. 2014;45:71–81.

    Article  PubMed  Google Scholar 

  24. White M, Adams J. Different scientific approaches are needed to generate stronger evidence for population health improvement. PLoS Med. 2018;15(8):e1002639.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Ogilvie D, Adams J, Bauman A, Gregg EW, Panter J, Siegel KR, et al. Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head. J Epidemiol Community Health. 2020;74(2):203–8.

    Article  PubMed  Google Scholar 

  26. McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc Sci Med. 2021;272:113697.

    Article  PubMed  Google Scholar 

  27. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of medical research council guidance. BMJ. 2018;2021(374):1–11.

    Google Scholar 

  28. Pawson R, Tilley N. Evaluation for the 21st Century: A Handbook. Thousand Oaks, California: SAGE Publications, Inc.; 1997.

    Google Scholar 

  29. Patton M. Developmental evaluation: applying compexity concepts to enhance innovation and use. New York: The Guildford Press; 2011.

    Google Scholar 

  30. Carey G, Malbon E, Carey N, Joyce A, Crammond B, Carey A. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open. 2015;5(12):e009002.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Gates EF. Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Eval Program Plann. 2016;59:62–73.

    Article  PubMed  Google Scholar 

  32. Borys JM, Le Bodo Y, Jebb SA, Seidell JC, Summerbell C, Richard D, et al. EPODE approach for childhood obesity prevention: methods, progress and international development. Obes Rev. 2011;13(4):299–315.

    Article  PubMed  Google Scholar 

  33. Romon M, Lommez A, Tafflet M, Basdevant A, Oppert JM, Bresson JL, et al. Downward trends in the prevalence of childhood overweight in the setting of 12-year school- and community-based programmes. Public Health Nutr. 2009;12(10):1735–42.

    Article  PubMed  Google Scholar 

  34. Van Koperen TM, Jebb SA, Summerbell CD, Visscher TLS, Romon M, Borys JM, et al. Characterizing the EPODE logic model: unravelling the past and informing the future. Obes Rev. 2013;14(2):162–70.

    Article  PubMed  Google Scholar 

  35. Brown T, Moore TH, Hooper L, Gao Y, Zayegh A, Ijaz S, et al. Interventions for preventing obesity in children. Cochrane Database Syst Rev. 2019;2019(7 Art. No.: CD001871).

  36. Bemelmans WJE, Wijnhoven TMA, Verschuuren M, Breda J. Overview of 71 European community-based initiatives against childhood obesity starting between 2005 and 2011: General characteristics and reported effects. BMC Public Health. 2014;14(1):758.

  37. Boelsen-Robinson T, Peeters A, Beauchamp A, Chung A, Gearon E, Backholer K. A systematic review of the effectiveness of whole-of-community interventions by socioeconomic position. Obes Rev. 2015;16(9):806–16.

    Article  CAS  PubMed  Google Scholar 

  38. Borys JM, Valdeyron L, Levy E, Vinck J, Edell D, Walter L, et al. Epode - a model for reducing the incidence of obesity and weight-related comorbidities. Eur Endocrinol. 2013;9(2):116–20.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Mantziki K, Renders CM, Westerman MJ, Mayer J, Borys JM, Seidell JC. Tools for a systematic appraisal of integrated community-based approaches to prevent childhood obesity. BMC Public Health. 2018;18(1):1–12.

    Article  Google Scholar 

  40. McLeroy KR, Bibeau D, Steckler A, Glanz K. An ecological perspective on health promotion programs. Health Educ Q. 1988;15(4):351–77.

    Article  CAS  Google Scholar 

  41. van der Kleij MR, Crone M, Reis R, Paulussen T. Unravelling the factors decisive to the implementation of EPODE-derived community approaches targeting childhood obesity: A longitudinal, multiple case study. Int J Behav Nutr Phys Act. 2016;13(1):98.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Traverso-Yepez M, Maddalena V, Bavington W, Donovan C. Community capacity building for health: a critical look at the practical implications of this approach. SAGE Open. 2012;2(2):215824401244699.

    Article  Google Scholar 

  43. King L, Gill T, Allender S, Swinburn B. Best practice principles for community-based obesity prevention: development, content and application. Obes Rev. 2011;12(5):329–38.

    Article  CAS  PubMed  Google Scholar 

  44. Skinner S. Strengthening communities: a guide to capacity building for communities and the public sector. London: Community Development Foundation; 2006.

    Google Scholar 

  45. Kremers SPJ, Visscher TLS, Schuit AJ. Effect in zijn context (Dutch). Tijdschr voor gezondheidswetenschappen. 2018;96(3–4):128–31.

    Article  Google Scholar 

  46. van der Kleij RM, Crone MR, Paulussen TG, van de Gaar VM, Reis R. A stitch in time saves nine? A repeated cross-sectional case study on the implementation of the intersectoral community approach youth at a healthy weight. BMC Public Health. 2015;15(1):1032.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Collard D, Slot-Heijs J, Dellas V, van der Poel H. Monitor Jongeren Op Gezond Gewicht 2017 (Dutch). Utrecht: Mulier Instituut; 2018.

  48. Blokstra A, Schipper M, de Hollander EL, Schurink-van ’t Klooster TM. Werkt de JOGG-aanpak? Veranderingen in overgewicht en beweeggedrag bij kinderen en jongeren (Dutch). Bilthoven: Rijksinstituut voor Volksgezondheid en Milieu; 2020. Available from: https://www.rivm.nl/documenten/factsheet-werkt-jogg-aanpak.

  49. Reijgersberg N, Pulles I, Breedveld K, de Hollander E. Monitor Jongeren Op Gezond Gewicht 2015 (Dutch). Utrecht: Mulier Instituut; 2016.

  50. Mantziki K, Renders CM, Vassilopoulos A, Radulian G, Borys J-M, du Plessis H, et al. Inequalities in energy-balance related behaviours and family environmental determinants in European children: changes and sustainability within the EPHE evaluation study. Int J Equity Health. 2016;15(1):160.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Weiss C. Theory-based evaluation: past, present and future. New Dir Progr Eval. 1997;76:41–55.

    Article  Google Scholar 

  52. Nutbeam D. Evaluating health promotion - progress, problems and solutions. Health Promot Int. 1998;13(1):27–44.

    Article  Google Scholar 

  53. Ling T. Evaluating complex and unfolding interventions in real time. Evaluation. 2012;18(1):79–91.

    Article  Google Scholar 

  54. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):1–11.

    Article  Google Scholar 

  55. Rogers PJ. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008;14(1):29–48.

    Article  Google Scholar 

  56. Simons H. Case study research in practice. London: SAGE Publications, Inc.; 2009.

    Book  Google Scholar 

  57. Green J, Thorogood N. Qualitatieve methods for health research. London: Sage; 2013.

    Google Scholar 

  58. McGill E, Marks D, Er V, Penney T, Petticrew M, Egan M. Qualitative process evaluation from a complex systems perspective: a systematic review and framework for public health evaluators. PLoS Med. 2020;17(11):e1003368. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pmed.1003368.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Chazdon S, Emery M, Hansen D, Higgins L, Sero R. A Field guide to ripple effects mapping. 2017. 1–216 p. Available from: https://conservancy.umn.edu/handle/11299/190639

  60. Borvil AD, Kishchuk N, Potvin L. The use of critical incident technique in population health intervention research: lessons learned. Int J Public Health. 2018;63(3):429–30.

    Article  PubMed  Google Scholar 

  61. Figueiro AC, de Araújo Oliveira SR, Hartz Z, Couturier Y, Bernier J, do Socorro Machado Freire M, et al. A tool for exploring the dynamics of innovative interventions for public health: the critical event card. Int J Public Health. 2017;62(2):177–86.

    Article  PubMed  Google Scholar 

  62. Yin RK. Case study research and applications: design and methods. Sixth. Thousand Oaks: SAGE Publications, Inc; 2018.

    Google Scholar 

  63. Rehfuess EA, Booth A, Brereton L, Burns J, Gerhardus A, Mozygemba K, et al. Towards a taxonomy of logic models in systematic reviews and health technology assessments: a priori, staged, and iterative approaches. Res Synth Methods. 2018;9(1):13–24.

    Article  PubMed  Google Scholar 

  64. Rohwer A, Pfadenhauer L, Burns J, Brereton L, Gerhardus A, Booth A, et al. Series: clinical epidemiology in South Africa. Paper 3: logic models help make sense of complexity in systematic reviews and health technology assessments. J Clin Epidemiol. 2017;83:37–47. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2016.06.012.

    Article  PubMed  Google Scholar 

  65. Mills T, Lawton R, Sheard L. Advancing complexity science in healthcare research: the logic of logic models. BMC Med Res Methodol. 2019;19(1):1–11.

    Article  Google Scholar 

  66. Cambon L, Alla F. Understanding the complexity of population health interventions: assessing intervention system theory (ISyT). Heal Res Policy Syst. 2021;19(1):1–11. https://0-doi-org.brum.beds.ac.uk/10.1186/s12961-021-00743-9.

    Article  Google Scholar 

  67. Moore GF, Evans RE, Hawkins J, Littlecott H, Melendez-Torres GJ, Bonell C, et al. From complex social interventions to interventions in complex social systems: future directions and unresolved questions for intervention development and evaluation. Evaluation. 2019;25(1):23–45.

    Article  PubMed  Google Scholar 

  68. Preskill H, Gopal S, Mack K, Cook J. Evaluating complexity: propositions for improving practice. 2015. Available from: https://www.fsg.org/publications/evaluating-complexity

  69. Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomised controlled trial be? BMJ. 2004;328(7455):1561–3.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Whelan J, Love P, Millar L, Allender S, Bell C. Sustaining obesity prevention in communities: a systematic narrative synthesis review. Obes Rev. 2018;19(6):839–51.

    Article  CAS  PubMed  Google Scholar 

  71. Waterlander WE, Pinzon AL, Verhoeff A, Den Hertog K, Altenburg T, Dijkstra C, et al. A system dynamics and participatory action research approach to promote healthy living and a healthy weight among 10–14-year-old adolescents in Amsterdam: The LIKE programme. Int J Environ Res Public Health. 2020;17(14):1–18.

    Article  Google Scholar 

  72. Allender S, Brown AD, Bolton KA, Fraser P, Lowe J, Hovmand P. Translating systems thinking into practice for community action on childhood obesity. Obes Rev. 2019;20(S2):179–84.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Not applicable

Funding

This research is funded by the national JOGG-organisation, supported by the Dutch ministry of Public Health, Welfare and Sport. The national JOGG-organisation had no role in the design of the study or writing of the manuscript.

Author information

Authors and Affiliations

Authors

Contributions

DC and AS secured funding for this study and were responsible for the overall development of the JOGG evaluation project. DC, AS and IH contributed to the conceptualisation of the evaluation approach presented in the paper. IH drafted the original manuscripts. DS, AS, FvL and MC all contributed considerably to revising the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Irma Huiberts.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huiberts, I., Singh, A., van Lenthe, F.J. et al. Evaluation proposal of a national community-based obesity prevention programme: a novel approach considering the complexity perspective. Int J Behav Nutr Phys Act 19, 31 (2022). https://0-doi-org.brum.beds.ac.uk/10.1186/s12966-022-01271-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12966-022-01271-7

Keywords