Adaptation and cross-cultural validation of the United States Primary Care Assessment Tool (expanded version) for use in South Africa

Background Measuring primary care is important for health sector reform. The Primary Care Assessment Tool (PCAT) measures performance of elements essential for cost-effective care. Following minor adaptations prior to use in Cape Town in 2011, a few findings indicated a need to improve the content and cross-cultural validity for wider use in South Africa (SA). Aim This study aimed to validate the United States of America-developed PCAT before being used in a baseline measure of primary care performance prior to major reform. Setting Public sector primary care clinics, users, practitioners and managers in urban and rural districts in the Western Cape Province. Methods Face value evaluation of item phrasing and a combination of Delphi and Nominal Group Technique (NGT) methods with an expert panel and user focus group were used to obtain consensus on content relevant to SA. Original and new domains and items with > = 70% agreement were included in the South African version – ZA PCAT. Results All original PCAT domains achieved consensus on inclusion. One new domain, the primary healthcare (PHC) team, was added. Three of 95 original items achieved < 70% agreement, that is consensus to exclude as not relevant to SA; 19 new items were added. A few items needed minor rephrasing with local healthcare jargon. The demographic section was adapted to local socio-economic conditions. The adult PCAT was translated into isiXhosa and Afrikaans. Conclusion The PCAT is a valid measure of primary care performance in SA. The PHC team domain is an important addition, given its emphasis in PHC re-engineering. A combination of Delphi and NGT methods succeeded in obtaining consensus on a multi-domain, multi-item instrument in a resource- constrained environment.


Introduction
Measuring primary care performance Primary health care (PHC), considered the backbone of a country's health system, 1 is a complex, multifaceted range of activities, a unique integration of knowledge, values and skills drawn from clinical, public health, behavioural and anthropological sciences. In addition to diagnosis, treatment and rehabilitation, the range of skills and activities include person-centred communication, prevention and health promotion applied in a comprehensive family-and community-orientated approach to care. A primary care measurement strategy therefore needs a range of dimensions for it to be a valid measure. Primary care dimensions (e.g. comprehensive care) are themselves multifaceted. Each dimension requires a variety of indicators (items) to describe and measure it (content validity); and item phrasing has to be congruent with the indicator and dimension being measured (face validity).
The Primary Care Assessment Tool (PCAT, http://www. jhsph.edu/pcpc/pca_tools.html) is a multi-dimension, multi-item instrument developed and tested for reliability and validity by Starfield and Shi, Primary Care Policy Center, Johns Hopkins School of Public Health. 2 The PCAT measures primary care organisation and performance on four core dimensions (access, continuity, coordination, comprehensiveness)and three derivative dimensions (community orientation, family-centredness, cultural competence)known as domains defined in the PCAT manual (http://www.jhsph.edu/pcpc/pca_tools.html). They have been shown to be essential for cost-effective primary care in developing and developed contexts. 3,4,5 When these essential features are available to primary care users and implemented in their care, the outcomes include improved health and satisfaction, reduced cost, and reduced inequity. 5,6 The domains are also in line with the Alma Ata Declaration, 7 universally accepted definitions of primary care, 4,8,9 and the principles of family medicine, 10,11 including principles relevant to sub-Saharan Africa. 12 By surveying the three main primary care stakeholders (users, providers/ practitioners and clinic managers) the PCAT measures the extent to which users' experience of primary care approximates what is essential for cost-effective care.
The extent of primary care clinic adherence in each domain is determined by complex summing of participant responses to a range of items (questions) pertinent to that domain. The PCAT is also able to determine the size of differences between the three stakeholders' domain scores when the respective user, provider and manager instruments are used. Whilst the dimensions are considered universal, their generalisability may be limited in different cultural and socio-economic settings. Given the focus on health systems strengthening worldwide, it is not surprising that the PCAT is increasingly being subjected to cross-cultural validation to extend its generalisability to countries wanting to align with cost-effective care. 13,14,15,16,17 There is also evidence supporting the PCAT's ability to measure the impact of changes to PHC systems. 17,18,19,20 This article describes the process used to adapt and validate the adult expanded version (PCAT AE) for use in South Africa (SA). Whilst there are other measures of PHC that have been found to be valid in contexts other than that in which they originate, 21 the literature suggests that the PCAT is the most widely used and adapted. Other PHC measures include the Primary Care Assessment Survey (PCAS), the Components of Primary Care Index, the EUROPEP Interpersonal Processes of Care, the Primary Care Evaluation tool (PCET) used mainly in Europe, and the General Practice Assessment Survey (GPAS), all comprising multi-item dimensions.

Study background
In 2011, following an earlier visit by PCAT author, the late Professor Barbara Starfield, a study team in the Division of Family Medicine at the University of Cape Town conducted a pilot audit of primary care in one Cape Town health substructure (two sub-districts) in collaboration with the service provider, Cape Town Metro District Health Services (MDHS). The adult expanded (AE) version of the original United States of America (USA) PCAT was scrutinised by the study team under the supervision of the author to determine whether changes were necessary prior to local use. Adhering to the author's guidelines for changes to the PCAT domains (B-K) b only minor adaptations were made. a These involved rephrasing some items to improve local comprehension using colloquial phrasing and local healthcare jargon.
Given the high prevalence of tuberculosis (TB) in SA and in the Western Cape in particular, the item screening for lead exposure was replaced with TB screening to improve the content validity of G domain (comprehensiveness − services available). Changes were made to the introduction (A), health insurance (L) and demographic (N) sections to fit the South African context. The adapted version was piloted on 10 patients before the English version was finalised and translated into isiXhosa and Afrikaans to cover the three major languages spoken in Cape Town. Translation included bilingual translators, back translation and piloting with Xhosa-and Afrikaans-speaking users. The translations were further scrutinised by bilingual fieldworkers during their two-day training to administer the PCAT AE. Minor changes were made to the wording of a few items and they were piloted on patients by the fieldworkers as part of their training before the translations were finalised.
a Edits to item phrasing were made in B2, B3, C3, C6, C8, C9, D7, D12, E1, E2, F1, F3,  G2, G3, G10, G12, G13, G14, G15, G18, G20, G21, G22, G23, G25, H2, H8, H12, J1, J2 The distribution of the dimension (domain) scores confirmed what might be expected locally, providing evidence for the PCAT's construct validity in the South African context. This was further supported by service managers and providers generally accepting the results as a reflection of primary care performance in the substructure. The 2013 study team nevertheless wished to strengthen the content and crosscultural validity of the PCAT before wider use in SA, and to improve its alignment with provincial and national health plans. The 2011 findings suggested that a few domain items may not have been well understood by primary care users. Although none of the items approximated > 50% 'Do not know' responses (requiring adjusted coding and analysis), some were much higher than others. In addition, the users' PCAT score for ongoing care (domain D) was considerably higher than the provider and manager scores and higher than expected by study team members who had years of experience working in the clinics studied.

Research design and method
Approval for the study was obtained from the Human Ethics Research Committee in the University of Cape Town's Faculty of Health Sciences and the Western Cape Provincial Department of Health Research Committee. There were no risks to participants in the study. All data provided were analysed and reported anonymously.
The study was conducted in two parts: (1) an expert panel and a primary care user focus group reviewed the wording of PCAT domain items for their face value; and (2) a combination of modified Delphi and nominal group technique (NGT) methods was used to determine consensus on domain and item relevance for SA. The ZA PCAT was piloted at a primary care clinic before being finalised and translated.

Part 1: Review of item wording for local use Expert panel (1a)
This panel, consisting of 2011 PCAT study investigators (two family physician researchers, a family physician in charge of clinical governance at a clinic and a fieldworker) had face value evaluation meetings to review the wording of all PCAT items in domains B-K and to rephrase items where necessary to improve comprehension by primary care users in SA. Problematic wording identified in 2011 was given particular attention. Changes were agreed upon by simple consensus. Selected items were referred to the user focus group to assist with improving user comprehension.

User focus group (1b)
Two members of the PCAT study team purposively selected and invited six primary care users from amongst patients attending a clinic in the baseline measure study sampling frame to join a focus group. Six consenting adult patients who had attended the clinic at least four times and who were conversant in two of the three main languages spoken in the Western Cape (namely English and Afrikaans) were selected. A modified form of the NGT method was used to obtain group consensus on the phrasing of items selected in part 1a. The process involves each item in turn being presented to the group on a flipchart in one of the

Number of items Mean s.d. Range
First contact-access 12 2.5 0.6 1.0-3.8 two languages. Each participant records her and/or his back translation on a blank sheet without discussion. All responses are recorded anonymously on a flip chart, after which each participant, without discussion, chooses and records the one response that she or he feels most accurately reflects the item presented. Responses (phrasing) achieving > = 70% agreement 22 are accepted. The steps are repeated for each item. The investigators determine whether the phrasing chosen maintains the intention of the original item; if not, investigators rephrase the item to maintain the original intention, taking into account the information obtained from participants by that stage. The above steps are repeated to determine consensus on the item thus phrased.

Part 2: Obtaining consensus on domain and item relevance for South Africa Delphi process (2a) (steps 1-4 in Appendix 1)
A second expert panel comprising 2 family physicians, 2 clinical nurse practitioners, 2 clinic managers, and 2 family physician educator/researchers were purposively selected by the study team to examine and determine consensus on the relevance of all PCAT domains and domain items (B-K) for the primary care context in SA. The task of each panellist was to determine from her/his own experience, (i) which items were relevant, and (ii) which items, if any, should be added. Background information given to the panellists included a summary of the PCAT, its purpose, and relevance to PHC and family medicine. The panellists were informed of the 2011 PCAT study; that lessons from 2011 were being applied; that rephrasing of some questions might be necessary; and that the process outlined was to validate the ZA PCAT for use in SA by using a combination of the Delphi and NGT methods (Appendix 1). Inviting, consenting and providing information to the panellists were done via email. Panellists were asked to scrutinise all the PCAT domains and their respective items and to score them for relevance on a scale of 0 to 3 (not at all relevant -very relevant). At the end of this stage the emailed data were collated and the percentage agreement calculated for the 9 domains (B-K) and their items.
The results together with new items were tabulated.

Nominal Group Technique process (2b) (steps 5-7 in Appendix 1)
The expert panel in 2a was convened by two study investigators and part 2a results presented (items which achieved < 70% agreement; new items generated; and comments) by displaying them verbatim in poster form for panellists to view and to ensure that their items and comments had been recorded. Three questions were provided to guide the selection and scoring of new items generated in part 2a: (i) Is it important and relevant to primary care?; (ii) In which domain does it belong?; and (iii) Can clinic staff do anything about it?
The method requires that the meaning of each new item be clarified and similar items merged (step 5) in a facilitated group discussion. Given that the above discussion may stimulate further thinking on domains and items to include, panellists are given a second opportunity on their own (i.e. NGT silent phase repeated) to generate items they wished to include.
All new items thus generated are in turn clarified to ensure they are understood by all participants and that they differ from existing items (step 5 repeated). These were added to a table along with new items generated in the Delphi process (2a) and original PCAT items that achieved < 70% agreement and scored for relevance as in 2a (i.e. items that achieved < 70% agreement in 2a were re-scored), and were recorded independently. Each panellist then verbally submitted her/ his scores in round-robin fashion (step 6). The scores were recorded and summed by the investigators, and the summed scores presented in a projected table (step 7). The percentage agreement for each item was calculated to determine which items achieved consensus (step 7), that is > = 70% agreement. Repetition of steps 3-7 above substituted for the usual successive rounds in the Delphi method, which was limited to one round in part 2a. The percentage agreement for each item is calculated, where 100% = the number of panellists x 3, that is the maximum score per item. Only domains and items on which consensus is achieved (i.e. > =70% agreement) are retained (step 8).

Item review and rephrasing (1a and 1b)
Domain and item relevance was not considered at this stage. The phrasing of domain items (questions) considered problematic by the 2011 study teamrequired minimal rephrasing. The team nevertheless wanted user assistance on five questions (D4, F1, G2, J2, K2) to improve user comprehension. These were presented to the six focus group participants for scrutiny as described in 1b of the methods section. Most participants had difficulty understanding the task, especially back translation and its purpose, and preferred to add and discuss questions on problems they encountered with the service. After unsuccessful attempts to explain and guide participants through the planned NGT process, this was abandoned. Instead an open discussion on the phrasing of the questions was conducted to gain as much information from patients as possible for item rephrasing.

The Delphi process (2a)
Results of the Delphi process (2a)

The Nominal Group Technique process (2b)
In the NGT process (2b) a total of 19 new items were generated and scored, as described in methods sections 2a and 2b. The results are presented in Table 2. These items related to the first contact -access (C), ongoing care (D), coordination This was followed by a final round of scoring (individually and without discussion), which included all the original PCAT items which received < 100% agreement in 2a and all new items generated as described above. Whilst rescoring the few original items that achieved < 100% but > 70% in part 2a was not essential, the panel felt that a final round of scoring that included these items should be the final step. Table 2 shows the final percentage agreement for all original items which achieved < 100% agreement, all new items (coded as 'NGT') as well as the PHC team domain.

E9bNGT
How long did it take for you to be given your appointment by your CHC? 95

E9cNGT
From the time that you were given your appointment date, how long before you actually saw the specialist? 90

E14NGT
Would the CHC assist you to get medical-legal or insurance reports if required? 71

G26NGT
Checking for weight problems? 95 G27NGT Access to termination of pregnancy services at or via your CHC if required? 100

H15NGT
Advice and treatment on sexually transmitted infections 100 H9* Ask if you have a gun, its storage or its security 10

K4bNGT
Do you think your CHC understands/respects your culture? 100

K4cNGT
Do you feel comfortable discussing religious or cultural issues that affect your health with the staff at the CHC? 100

P1.
Can you see a social worker if you need to? E.g. for help with counselling for a family problem or advice about social services? -

P2.
Can you see a physiotherapist (and occupational therapist) at your CHC if you need to? E.g. to help with muscle sprains or movement following a stroke.

P3.
Can you be visited in your home by a community health worker linked to your CHC if you need it? E.g. for home-based care for TB, HIV or basic care such as wound dressings. -

P4.
Can you be seen by a health promoter/dietician for advice on these topics? -

P5.
Can you be seen by a mental health worker at your CHC for help with any mental health problems? -

P6.
Can you be seen by a dental/oral health worker at/or linked to your CHC if you need it? E.g. any problems with your teeth. -

P7.
Can a child (under 12 yrs) be seen at your CHC? -ZA PCAT validation. DELPHI-NGT comments and additional questions. 7 Participants; score 0-3; max score per item = 21, that is 100% agreement. NGT, items added via Delphi-NGT process; CHC, community health centre. *, Retained in the printed version; explained in discussion section. explained in the discussion below. The provider (PE) and manager (FE) instruments were accordingly aligned with the ZA PCAT AE by the study team. The additions to the English AE version were translated and back translated for the isiXhosa and Afrikaans versions and existing items in the 2011 translations were reviewed by bilingual study team members before being finalised.

Discussion
Following lessons from the 2011 pilot study, we sought to validate the PCAT content and cross-cultural applicability for use in SA, that is improve its relevance to primary care in SA and its comprehension in three languages before extending the audit to other health districts. Minimal item rephrasing was necessary following that done in 2011. The fact that no domains were removed, only one new domain was added, only three items achieved < 70% agreement, and a high median percentage agreement on domain and item relevance was achieved, indicates the high content validity of the original PCAT for SA.
The silent phases in parts 2a and 2b ensured that participants were able to generate items and to agree/disagree independently of each other. The discussion during the clarification stage and the range of scores in part 2b suggest that participants were not unduly led by each other's views or scores. The addition of the PHC team domain aligns the PCAT with the emphasis on the PHC team in local and international policy documents and research. 23,24,25,26,27 In a study identifying key principles of family medicine in sub-Saharan Africa, practising as members of a PHC team emerged as important and included nurse practitioners. 12 Including the PHC team as a domain in the ZA PCAT is therefore likely to be supported by primary care physicians in Africa.
Nurse practitioner (CNP) was not generated as a PHC team item. This can be explained by the fact that PHC in SA is a nurse-led service. 23 In most primary care clinics CNPs are the only clinical practitioners. Where there are doctors, CNPs practice as members of the clinical team alongside them.
The PHC team domain uses the same Likert scale for rating responses to items and is analysed in the same way as existing domains. PHC team scores are included in the total primary care score -a summing of all the domain scores. New items which do not use the PCAT Likert scale (patient waiting times, travel costs and patient satisfaction) are included under the relevant domains but are analysed separately. The study team elected to retain original items C6, C7, H9 in the printed version b used for the data collection in order to allow for international comparison of results where the original PCAT is used. Analysis of the data will include or exclude these items as required. Cronbach alpha and factor analyses on the ZA PCAT 2013 will be included in the second article reporting the results of the PCAT survey of users, providers and managers.
The method used to improve the wording of domain items for better comprehension by users did not follow the standardised seven-step method described by Sousa and Rojjanasrirat 28 in their review of methodological approaches to translation, adaptation and cross-cultural validation of research instruments; also used in the Korean adaptation of the PCAT. 15 A key component of our cross-cultural method was the user focus group (part 1b). However, most of the participants had difficulty understanding the task, especially back translation and its purpose, which led to the planned NGT process being abandoned. Users' preference for raising and discussing problems they encountered with the service is highlighted in patient satisfaction surveys.
The research team nevertheless constituted an expert panel with years of experience practising and teaching in local primary care services. The same team conducted the 2011 study and included a research assistant with experience in developing and administering questionnaires in other research projects in local primary care services. The combined experience was applied in updating and improving the translations of the English PCAT into the two other main languages spoken in the Western Cape. Edits to the demographic section (N) to align it with Western Cape demographic and socio-economic features were also made. Even though the translation method did not follow all the steps suggested by Sousa and Rojjanasrirat, 28 the bilingualism advised for cross-cultural validation was well represented on the panel and the research assistantwas trilingual.
The pilot conducted on 10 patients before finalising the ZA PCAT AE 2013 will have reduced the impact of our failure to achieve the primary objective of the focus group, by providing another opportunity to identify difficulties with phrasing and administering the PCAT to users. In addition, the 2011 translation conducted by the study team was also scrutinised by the bilingual fieldworkers. Their practical training on patients served as a second pilot of the isiXhosa and Afrikaans translations in 2011.
The focus group experience nevertheless provided important insights, including the depth of feeling amongst users about their frustrations with primary care; the challenges faced by users who are functionally illiterate (which complicated the task of the researchers in this context); the importance of a participatory method when involving user stakeholders and the need for researchers to be flexible. The communication challenge between informant and researcher reflects a common feature of the primary care consultation, namely the potential conflict between patient and practitioner agendas, where 'give and take' is required to achieve a therapeutic partnership.
Combining the Delphi and NGT methods in a two-stage process had the benefit of participants independently generating responses in at least one round before meeting for the NGT, thus reducing the impact of attrition of participants and delayed responses that can bedevil a Delphi process. The combination of the Delphi and NGT methods may be better considered a modification of the NGT. 22 In verbal feedback at the end of part 2b the expert panel reported that, as busy clinicians and managers, they found the combination of Delphi and NGT methods useful in seeking consensus on the content and wording and preferred to meet once-off rather than have a number of iterations as required by the Delphi method. They also preferred the opportunity to interact with each other, especially in the prioritisation and clarification stages of the NGT.
In addition, the modification provided the opportunity for doctors, nurse practitioners and managers to contribute as equals in a key stakeholder group. Modelling such a method that can be used in clinics by managers and practitioners to obtain multi-stakeholder consensus on complex activities and interventions in a time-constrained and limited-resource context may also be useful. Involving these key stakeholders should also increase the likelihood of the ZA PCAT's use in ongoing monitoring, evaluation and quality improvement, also guiding the revitalisation of PHC and assisting health sector reform in SA. Whilst the study did not specifically address aligning the PCAT with provincial and national health policies and plans, it can be argued that including managers, practitioners and educators with years of experience in health services in SA guided the work with these in mind -evidenced by the addition of the PHC team.
As noted above, the findings concur with key principles of family medicine in sub-Saharan Africa and therefore point to the potential for the ZA PCAT to be similarly used in other parts of Africa, as well as to measure the impact of interventions aimed at strengthening PHC systems. This includes measuring the impact of postgraduate training in family medicine (primary care physicians) on person-centred comprehensive, community-based primary care as promoted in the Victoria Falls Statement. 24 Given that primary care in SA is nurse-driven, the finding that the PCAT content is highly relevant to SA suggests that CNPs should also be trained to apply the essential elements of primary care. If the benefits of a PHC team are to be realised, training in the family medicine approach to primary care -currently the preserve of primary care physicians -should be extended to CNPs. Primary care physicians and CNPs should be trained together on the content, application and measurement of the essential dimensions of primary care. Primary care facility and district managers should be aware of the importance of these elements and trained in methods that improve access to them when allocating and managing resources. They should also be trained in the use of the PCAT to monitor the organisation and performance of the primary care services they manage.
Limitations of the cross-cultural validation method used, when compared with those suggested by a scholarly review, 28 are discussed above. This will need to be considered if there are significant differences in scores between the three language groups after the baseline results are analysed.
Other study limitations include the expert panel and patient informants being local only. The full Delphi method would have permitted wider representation on the expert panel and could have included participants in other sub-Saharan countries. The items in the PHC team domain do not describe team functioning or the quality of team-based care and therefore limit the potential value of this measure. Items that describe team function should be developed and added in future studies.
Is the ZA PCAT 2013 suitable for the Western Cape only? Given the diversity across the nine provinces in SA, translation into local languages and some rephrasing will be necessary, along with changes to the demographic section, depending on region or province. However, we think it unlikely that major domain and item changes will be necessary.
These limitations notwithstanding, the findings are in keeping with those of PCAT validation studies in other countries 15,16,17,18,19,20 where the essential features of primary care measured by the PCAT were also found to apply. The Brazilian study 18 kept closely to the original items, whereas other country studies removed a number of domain items, such as in the Chinese PCAT. 17 Haggerty et al. 21 examined the validity of a number of instruments that measure PHC from the user perspective in the Canadian context, including the PCAT, the PCAS, the Components of Primary Care Index, and the EUROPEP Interpersonal Processes of Care -all composed of multi-item dimensions. The study found that these instruments performed similarly in Canada as in their original contexts. The Brazilian PCAT study is of note for SA. It showed that the PCAT also applies in a developing context. 18 The PCET, not included in the Haggerty study, is also of interest. 29 It includes the four core primary care dimensions measured by the PCAT (access, continuity, comprehensiveness and coordination) as well as four health system functions (financing, creating resources, stewardship and delivery services) that are worthy of assessment in a primary care audit. These functions can be included alongside a PCAT audit without having to adapt the PCAT.
Other audit instruments used in SA currently include the National Core Standards (NCS) instrument 26  Part 2 (steps 5-7) will be conducted with the panellists meeting as a group. We envisage that each part can be done in 90mins or less. By the end of step 7 an enhanced ZA PCAT will have been agreed on via the consensus process outlined.
The attached PCAT questionnaire has 9 primary care domains (essential elements) each with a number of items posed as questions to describe the presence and practice of each domain. You are being asked to: • rate the relevance of domain items on a scale of 0-3.
• add any domains and / or items that you feel are important in the SA context that are missing in the current version.

(comment on italicised text if you wish and have the time)
The ratings of the expert panel will be combined to obtain an overall score for each item; depending on the scores, domains items will be retained, removed or added. Items that receive consistently low scores will be removed at the end of part 1. The remaining and/or additional items will go into round 2 and follow the same process above. Further rounds will follow if necessary until consensus is achieved by the panel on which items should remain, which should be removed and which if any items or domains should be added. Consensus is defined as 70% agreement and will constitute the revised ZA version of the PCAT to be used in the study.

Introduction
Participants and investigators are introduced. A brief overview of the purpose, objectives and process of the consensus method is given and any questions for clarification dealt with.

New item presentation, clarification and rationalisation
Any new items and domains generated in step 3 are presented to the expert panel and their meaning clarified in the group to ensure they are understood by all and how they differ from existing items. New domains and items with similar meanings and intentions will be merged. The relevance of items is not discussed at this