We developed a survey (see Online Appendix 1) to assess researchers’ self-perceived confidence in evidence synthesis skills before and after GELA. The survey was based on a tool by Bidonde^16,17 but adapted to fit the context of this study.

The survey comprised four sections: respondent demographics, core skills (for this study, core skills refer to technical skills), soft skills (project management and interpersonal skills) and capacity development. Core skills consisted of four separate sections: systematic reviews of effectiveness, qualitative evidence synthesis (QES), health economic evidence and methods related to evidence-to-decision processes. Researchers answered only the sections relevant to the research they conducted with the GELA project. Each core skill section consisted of different domains containing specific skills that align with conducting various types of evidence syntheses. Box 1 shows an example of the domain ‘Formulating a review question’ as part of the ‘Systematic review of effectiveness’ section and the specific skills in the domain. Details of all the domains for each section are presented in Online Appendix 1.

Respondents rated their self-perceived baseline confidence in executing different evidence syntheses skills. If they indicated that their skills for a specific domain were built as part of GELA, they also rated their confidence after participating in GELA activities. The confidence ratings for core skills were: ‘No knowledge of this’ (Level 1), ‘I have heard of this topic, but I don’t feel confident to do it (Level 2)’, ‘Slightly confident to do it (Level 3)’ and ‘I have mastered this topic (Level 4)’.

The section on soft (Project management and interpersonal) skills was not separated into domains but listed the specific interpersonal skills, e.g. ‘project coordination meetings’ and ‘chairing project meetings’ (see Online Appendix 1). The confidence ratings for project management and interpersonal skills were: ‘I don’t feel confident’ (Level 1), ‘I feel slightly confident’ (Level 2), ‘I feel moderately confident’ (Level 3) and ‘I feel highly confident’ (Level 4). Finally, we also asked respondents if they developed their skills through participating in the GELA activities, to measure their perceived capacity development.

The survey was designed on REDCap, an electronic data capture tool hosted by the South African Medical Research Council and invitations containing the link to the survey were sent via email. Participation was voluntary, and responses were anonymous; however, there was an option to provide an email address to enable linking survey responses with interview data.

All interviews were conducted virtually on Microsoft Teams except for the five SA member interviews, which were conducted face to face. The WP leads, interviews lasted an average of 52 min, and the WP members’ interviews an average of 41 min. The recordings were transcribed using Autokest.¹⁸ During the interviews, the interviewers (Retsedisitsoe P. Mazibuko and Willem Odendaal) took observational notes.

To ensure we understood how WPs were set up and what their activities were, we interviewed the WP leads first. The interviews for both WP leads and members focused on their perceptions and experiences of the GELA project more broadly, and whether and how capacity development happened for them. Some questions differed between leads and members, based on their roles within the WP (see Online Appendix 2).

Quantitative data obtained from the online survey were analysed using descriptive statistics with Stata 18.¹⁹ We reported the proportions of GELA researchers who indicated that their skills were developed through GELA activities. For self-perceived confidence levels, we reported the number of responses for each skill per level of confidence (from level 1 confidence, ‘no knowledge’, to level 4 confidence, ‘mastery’) before and after participating in GELA. We assumed that the level of confidence for researchers who had mastered a skill before GELA and thus indicated that they did not build their capacity, remained the same after GELA. We therefore reported on the total number of researchers with level 4 confidence at baseline and after GELA (i.e. the researchers with baseline level 4 confidence plus those with level 4 confidence after participating in GELA activities).

The interview transcripts were analysed with ATLAS.ti,²⁰ using framework analysis as proposed by Spencer and Ritchie.²¹ Framework analysis follows a systematic process of data analysis that produces structured outputs of the data.^21,22 It is flexible and not associated with a particular epistemological, philosophical or theoretical approach.²² In addition, framework analysis can be deductive or inductive depending on the research question.²² To ensure reliability during analysis, we consulted with the author team to review the analysis process.

In the first step, Familiarisation, the lead author started by listening to each recording, followed by her verifying the correctness of the transcriptions and anonymising the data. During this process, she made notes of salient aspects, keeping the objectives and aim of the research in mind. This was followed by the Thematical Framework development. Using the study objectives and interview schedules, the lead author and co-interviewer developed the initial code list and organised it into groups and themes, which were uploaded onto ATLAS.ti. They coded 10% of the transcripts independently and thereafter compared their coding. After reaching a satisfactory agreement, the lead author continued with Indexing, that is, coding the remaining transcriptions and periodically consulted with the co-interviewer to refine the code list. During the next step, Charting, the lead author grouped the codes according to the themes identified in the Thematical framework development stage. In the final step, Mapping, the lead author reviewed the quotations to confirm additional patterns and interpretations of the data.

Participants reported that their respective WPs required specific and different technical skills to achieve the WP objectives (see Table 1).

WP1 focused on scoping priority setting. In this WP, members gained skills in stakeholder mapping and identifying priority topics. For others, GELA strengthened their stakeholder engagement skills.

WP2 focused on finding, appraising and synthesising evidence required to develop clinical guidelines. It required an understanding of three evidence synthesis methods: systematic review of effectiveness, QES and health economics evidence synthesis.

Skills that were strengthened under WP2 included problem-solving in the context of guideline development and evidence synthesis. Some participants mentioned that their critical appraisal skills were strengthened. This included using tools such as the Cochrane Risk of Bias 2 (RoB2) tool, and the AGREE-II tool. Furthermore, they gained skills in assessing the certainty of qualitative evidence using the GRADE CERQual approach. Members were able to strengthen their evidence synthesis skills through conducting QESs and health economic evidence syntheses. Some of the members were also able to learn how systematic reviews are used in guideline development:

WP3 required an understanding of CPG development methods. Experience as a methodologist and that of a systematic reviewer in the context of guideline development was also highlighted as important in this WP. Some of the existing skills that were strengthened included learning about guideline development in a national context, rather than the international context, where various researchers had gained experience prior to joining GELA. Researchers were also able to learn the ADOLOPMENT process of guideline development.

WP4 focused on the dissemination of different formats of CPGs and communication within the project and with external stakeholders. It required knowledge translation skills, such as using evidence to inform decision-making. A participant reported that some of the required technical skills for this WP were the skills to understand study designs and study conduct and the use of evidence to inform decision-making and communicating with decision-makers:

In WP5, some of the required skills included an understanding of capacity development methods, developing workshop content, knowledge of CPG development and evidence synthesis for teaching and learning.

Some of the skills that were required in WP6 were protocol writing skills, operationalising of research questions skills and user testing skills. Members were able to strengthen their observation skills through observing guideline development meetings and steering group meetings, specifically, through the iteration of observation tools. An additional skill that was strengthened was developing infographics at a national level.

Both technical and soft skills were important to achieve WP objectives. WP leads reported various soft skills that were important to lead their respective WP. Project management, communication and organisation were the skills reported by all leads across all WPs. Work package members were able to build collaboration, planning, project management and time management skills. Both WP members and WP leads emphasised that the GELA project required great collaboration and people skills to be able to carry out their respective roles:

The GELA project, in nature, was a collaborative effort involving partnerships between countries. This collaboration allowed capacity development in the form of learning from others. There were cases of partners from different countries learning from each other, an example of this being partners filling capacity gaps by drawing on the experience of partners in another country with more capacity. Partners would give input on the work performed by other partners to provide guidance, exemplifying learning through collaboration:

In addition to learning between countries, there was learning within the countries where individuals within the respective country teams were able to learn from one another. There were instances where researchers from the same country within a WP would meet to discuss areas that others did not understand, and they would take time to share experiences and help their fellow country colleagues understand. It was mentioned that good relationships were important to facilitate learning between individuals:

The collaborative nature of GELA facilitated both informal and formal learning. Examples of formal learning mechanisms within the project were short courses, webinars and workshops. The short courses were provided by Stellenbosch University, one of the GELA partners. This partnership made it easier for the researchers to partake in courses offered. Furthermore, the GELA project provided bursaries for researchers to register for these courses. Another advantage of this collaboration was that the university had courses that were specific to the various required technical skills for CPG development. Some of the webinars were hosted by the Norwegian researchers, while the workshops were hosted by other researchers within the project.

Participants shared their preferences for learning through formal or informal mechanisms. Although the participants had different preferences, some highlighted that formal mechanisms can be more applicable to novice researchers, while those with experience may learn more through informal mechanisms. Many participants suggested that both mechanisms can be complementary:

Capacity development deliverables in WP5 of GELA mostly focused on using evidence in the guideline development process. Capacity development addressing gaps in producing evidence (mostly related to WP2) happened within work packages. Because of this, there were reports of challenges in balancing meeting WP deliverables and dedicating time to capacity development. A WP lead thought that there was not enough time to meet both the project deliverables and capacity development. As a result of this, leads had to prioritise meeting the WP deliverables, and capacity development became less of a priority:

In addition, one WP lead pointed out that there were challenges in implementing the scheduled capacity development activities as detailed in the GELA proposal, and that, some of the planned activities were more challenging to implement in reality than anticipated:

There were reports of the iterative nature of capacity development, and researchers highlighted that the process of learning involved identifying gaps as you progress through the work of the project. WP leads filled these gaps by providing learning resources to their members. One lead took the initiative to consolidate materials that were deemed important to strengthening the skills of the members before the start of the WP activities. In another WP, the leader of a small sub-team started journal clubs to assist with strengthening the skills of the researchers and finding ways to meet the team’s deliverables. Within WP2, there were smaller teams that focused on one of the evidence synthesis methods: