Evaluation of problem-based learning curriculum implementation in a clerkship rotation of a newly established African medical training institution: lessons from the University of Botswana

Evaluation of problem-based learning curriculum implementation in a clerkship rotation of a newly established African medical training institution: lessons from the University of Botswana

Stephane Tshitenge Tshitenge^1,&, Chiratidzo Ellen Ndhlovu², Radiance Ogundipe¹

 

¹Family Physician, Faculty of Medicine, University of Botswana, Gaborone, Botswana, Department of Family Medicine and Primary Health Care, Botswana, ²University of Zimbabwe College of Health Sciences, Harare, Zimbabwe

 

 

^&Corresponding author
Stephane Tshitenge, Faculty of Medicine, University of Botswana, Department of Family Medicine and Primary Health Care, Gaborone, Botswana

 

 

 

 

Problem-based Learning (PBL) is an approach to learning that is used extensively by medical training institutions for the past four decades. A 'typical' PBL consists of a 'seven-jump' procedure that includes in the first session, identifying facts/problems, generating hypotheses, listing needs to know, organizing and prioritizing learning issues/objectives, self-directed learning, and the second session is essentially a feedback session [1]. In a 'hybrid' PBL curricula, prior five lectures per week are reasonable to support PBL learning at the preclinical phase; while in the clerkship, there should be fewer lectures along with seminars, bedside teaching and skills laboratories [2]. The task of the PBL facilitator is to ease the intellectual and relational process for the group [3]. PBL was primarily designed as a learning platform for preclinical phase training and has been extensively studied and adopted in this phase [4]. However, some medical training institutions extend its use to clerkship training [5]. PBL curricula require systematic evaluation as there is a risk of implementing a dysfunctional PBL curriculum [6]. The implementation of PBL is not easy more especially in new institutions; as challenges encountered may include resources, organizational issues and a shift in culture for the academic staff [7]. Literature on the implementation and use of PBL in clinical phase in Sub Saharan Africa is very limited; also literature that assesses the discrepancy in implementation of the PBL curriculum within new institutions is scarce. The University of Botswana-Faculty of Medicine (UB-FOM), a new medical institution that just graduated three cohorts of medical students, adopted a 'hybrid' PBL as a mode of teaching and learning. The UB-FOM's PBL curriculum implementation and facilitation in clerkship was yet to be evaluated, though an early study from UB-FOM conducted on one group of 26 students who were commencing their rotation in emergency medicine attested that more than 90% of students supported the PBL approach [8]. This study intended to evaluate the PBL curriculum delivery from the perspective of the clerkship students at UB-FOM.

 

 

A cross-sectional study was conducted among year 3, year 4 and year 5 clerkship students rotating in family medicine, paediatrics, surgery and internal medicine departments of the UB-FOM from the 1st October 2015 to the 20th November 2015. Students rotate in small groups of on average 10 students for an 8-week rotation block in these disciplines. The survey instrument, a three-part self-administered and anonymous questionnaire, was constructed as follows; (i) Part I: demographic data, (ii) Part II: CP students' evaluation of the PBL process, and (iii) Part III: CP students' evaluation of the PBL facilitation. Part I of the questionnaire had items that included the clerkship student's class, the rotation block, number of students the PBL group, number of discipline-specific teaching sessions (lectures) in a week, and whether the PBL session 1 and 2 took place. Part II of the questionnaire consisted of seven items adapted from the 'seven-jump' procedure checking whether (1) case scenario provided open-ended problems, (2) unfamiliar terms were identified and clarified, (3) the problem was defined, (4) brainstorming was done, (5) there was a review the definition of the problem, (6) there was formulation of learning objectives, (7) there was contribution to the discussion by all students [1, 5]. Part III of the questionnaire consisted of 11 items adapted from the Short-Questionnaire-to-Evaluate-the-Effectiveness-of-Tutors (SQEET-PBL) tool to evaluate the PBL facilitation [9]. It is one of the instruments to evaluate the facilitation of PBL that has high validity and reliability; it was founded on contemporary constructivist approach theories with five different facilitator competencies which are (i) constructive, (ii) self-directed, (iii) contextual, (iv) collaborative learning and (v) intra-personal behavior [9].

 

Each of these items was evaluated using a five point Likert scale indicating from (1) strongly disagree to (5) strongly agree with each statement. At the end of the questionnaire, students were asked to provide an overall evaluation of PBL process and facilitation using a scale from 1 (very poor) to 10 (excellent); and an open-ended section that permitted the students to write in short sentence tips to improve the PBL process and the PBL facilitation. We conducted a pilot study using a group of 10 students to test the user friendliness of the questionnaire; and we corrected the tool where it was necessary. In UB-FOM, the first PBL session generally takes place at the beginning a week (Mondays) and the second session is held towards the end of a week (Fridays). PBL sessions are, in a majority of the case, facilitated by academic staff, and in a few cases by residents. During a 4-week period, each respondent completed weekly a questionnaire based survey tool from day one of the PBL week and then submitted it at the end of the week by placing the forms in a specified collection box. All clerkship students rotating in these four disciplines during the study period were asked to participate. We excluded those who did not consent or those who were in a rotation block where PBL sessions were not organised such as MBBS 5 internal medicine. We summarized data in frequency, median ± interquartile range (IQR), tables and figures where it was appropriate. The qualitative data were identified in themes and summarized in frequency. We used a Spearman test to assess the correlation between the quality of PBL process and PBL facilitators. R software version 3.3.1 was used to capture and analyze the data. The level of statistical significance will be taken as below 0.05. For qualitative data, we read through the responses repeatedly to familiarize with the data. Thematic indexes were developed using the Atlas-ti qualitative analysis software. We systematically applied the codes in the thematic index to all the data. Charting was done to bring all data with the same codes together; these were then interpreted by the researchers. We obtained ethical clearance for the study from the University of Botswana (UBR/RES/IRB/1569, X-REF: UBR/RES/ETHI/07). No student or PBL facilitator identifiers, such as names and addresses were captured for purposes of this study. Respondents signed the consent form before embarking on the study, they were also told that they could withdraw at any time they wished. The risk of the study was negligible, since the study did not involve any interventions.

 

 

Out of the 81 eligible CP students in Internal Medicine, Pediatrics, Surgery and Family Medicine, 89% (n=72) responded. Out of the expected 288 forms to return (72 respondents X 4 weeks), 141 (49%) forms were collected back. The majority (n=84; 60%) of the forms were from MBBS year 3 respondents Table 1.

 

Evaluation of the problem based learning process: PBL first sessions took place all the time only in Family Medicine and in about three quarters (75%) of time in Pediatrics but none were conducted in the other disciplines. PBL case scenarios derived from patients clerked by students were used in Family Medicine all the time and in Pediatrics in 70% of the cases. In disciplines such as Internal Medicine and Surgery, paper based cases were used as PBL case scenarios. The second session PBL took place in the four disciplines, all the time. Table 2 illustrates scores assigned by students when evaluating the PBL process. Students attributed a median score of 4 (agree) to all the items. Table 3 summarized qualitative data on tips to improve the PBL process. Students expressed that they would like to see PBL first session to take place. 'First session of PBL is very important; I need also to have my learning objectives' (IMYns) Students appeared to have differing opinions on the preferred approach to the nature of patient problems that the PBL sessions should be structured around. Some of the students expressed preference for the case scenarios provided by the facilitators as it was in their pre-clinical phase, while others opted for actual patients clerked by the students. 'The PBL in Internal MED does not analyze the real patients from the wards, but has learning objectives already formulated. This arrangement makes it easy for us to know what to learn and it's a good arrangement' (IMY4) 'Use of case scenarios like in phase 1 stimulate more learning practically than topics' (PY5) 'We should bring into discussion the clerked patient and try to manage the patient' (FMY5) 'Case should be clerked by the student and reviewed by the facilitator before the first session' (PY5).

 

Evaluation of the problem based learning facilitation: Table 4 shows how students assigned scores per item when evaluating PBL facilitation. The students allotted a median score of 4 (agree) in all the items; one quarter of the students attributed a score of 2 (disagree) in collaborative learning items. For qualitative data on PBL facilitation Table 3, the most recurring themes in the students' responses were the lack of uniformity in the quality of the PBL facilitation. They appreciated the facilitation differently depending on the tutor and rotations. 'The tutors are always knowledgeable, but sometimes lack the skills of teaching or guiding us learn the necessary.'(PY3-1) 'Train staff in this form of curriculum. It is sometimes poorly taught and sometimes okay. Some rotations teach better than others' (PY3-2) Many of the students were against reading from books and notes during the second PBL session as this practice does not augur well for deep learning. 'Allocate questions randomly to students and don't allow reading out feedback as it does not help students learn' (IMY3-1) A guiding, clarifying style and clear expectations without unnecessary intrusions was preferred 'Was involved in the discussion which was good and asked for clarification here and there. Provided a good environment for learning' (FMY3-1) 'From beginning of PBL session states how he prefers his PBL to be' (FMY3-2) 'Allow students to express themselves with minimal intervening' (FMYns) Another concern expressed by the students was the way the PBL sessions were graded. They felt the grading was subjective and suggested that the grading be discussed with the students as a formative feedback that is effective and constructive. 'The facilitator should discuss the grading with us. This will sort of help us know where to improve and how to study'(PY3-3). 'Tutors should at least have a short session after PBL to give a short lecture about the week's topic and how to approach and handle the patient with such condition '(PYns-).

 

Overall clinical phase students' score of PBL process and PBL facilitators: the students overall responses showed that the students gave a median 'good' score (8, IQR: 7-9) to the PBL process and a median 'very good score' (9, IQR: 7-9) to the PBL facilitation. There was a statistically significant positive correlation between the favorable grade allocated for PBL process and PBL facilitation (rs =0. 67, p=0. 01); and between the students allocated score to the first item assessing the PBL process (PBL case scenario provides open-ended problems that stimulate inquiry, not a single problem with a well-defined solution) and PBL facilitation (rs =0. 24, p=0. 01).

 

 

The present study evaluated the PBL curriculum delivery as it was viewed by clerkship students at the UB-FOM. This study found that PBL process varied from one rotation block to another in clerkship the UB-FOM. The first sessions took place all the time only in one rotation block (Family Medicine); while in other rotation blocks it was partly implemented (about three quarters of time in Pediatrics) or not at all in the other disciplines. Although most academic staff undergoes induction training on PBL, the non-experience of some of the academic staff in clerkship may have played a role on the unequal way of implementing the PBL. With the non-implementation of the PBL first session where five of the 'seven-jump' take place, one would wonder if such training platform has created a dysfunctional curriculum with a PBL façade [1, 5]. This study found that there was no uniformity in the way PBL case scenarios were generated; they were derived from clerked patients by students in Family Medicine, while paper cases were used in other rotation blocks. If one group of students commented that having a case paper was easing their learning, another group of students had an opposite point of view as using a real patient stimulated their learning. Although there may be no significant difference between the two approaches, a study found that the use of a real patient as a PBL case scenario was considered significantly more interesting than the paper case by students and it may significantly improve their understanding of the learning objectives and make them feel confident in upcoming patient encounters [10]. Although it may be easy to find a good real patient PBL case scenario during an 8-week rotation block in primary care settings such as Family Medicine, it may not be as easy to find a suitable patient with the week theme scenario in secondary care during an 8-week rotation block. In our study, students felt that overall, the PBL process was 'good' (median score 8, IQR: 7-9). A study from the same institution conducted among Year 3 students rotating in Emergency Department reported that over 90% of respondents were satisfied with PBL and found PBL and Emergency Medicine as an effective combination [8].

 

In this study, although students felt that overall the PBL facilitation was 'very good' (median score 9, IQR: 7-9), one quarter of the students´ responses allotted a score of 2 (disagree) in collaborative learning items. PBL in UB-FOM is facilitated by faculty, staff and residents in few cases. Collaborative learning as well as stimulating active learning and self-directed learning is considered to be the most important tutor competencies that trigger a higher group functioning, and better achievement in PBL [11]. Academic staff, as a subject expert, may be more likely than non-academic staff (residents) facilitators to display the collaborative competency [12]. Many of the students were against reading from books and notes during the second PBL session as this practice does not augur well for deep learning. By reading from books during feedback session, students give the impression that they are engaged actively in the debate, but that discussion is superficial and do not create embellishment and activation of prior knowledge. As a result PBL becomes a ritual behavior; when such situation arises, the facilitator is to ask students to reflect on their own process. In the study, students suggested that PBL facilitators should adopt a guiding, clarifying style and provide clear expectations without unnecessary intrusions. The students felt that the grading were subjective; they suggested that the grading be discussed with the students as a formative feedback that is effective and constructive. The tool used by UB-FOM to rate the students´ performances during the PBL session has not been validated therefore inappropriate grading of students may be possible and may have contributed to students´ dissatisfaction about the grade. Method of rating students' performances during the PBL session may include tutor´s assessment, peer assessment, and self-assessment; though in summative assessment, peer- and self-assessment marks may be reliable but they lack validity [13]. In our study, there was a statistically significant positive correlation between the favorable grade allocated for PBL process and PBL facilitation (rs =0. 67, p=0. 01) and between the students allocated score to the first item assessing the PBL process (PBL case scenario provides open-ended problems that stimulate inquiry, not a single problem with a well-defined solution) and PBL facilitation (rs =0. 24, p=0. 01). Similar findings were reported in a Japanese study where a rating of 'excellent' regarding case scenarios was associated with excellent tutor evaluations (OR of 12.43 [95% CI: 10.28-15.03) [14].

 

Limitations of the study: the present study did not attempt to assess the proportion of PBL facilitation by non-academic staff tutors and how their PBL facilitation was rated by student. On the other end, we did assess whether there was a difference in rating of the PBL facilitation between lower class students and high class students. As a result of PBL experience, high class students have tended to give a favorable score to facilitators, they become less reliant on their facilitators to make the group functioning and they are more focused on gaining difficult information [15]. Although the response rate to the questionnaire was 89%, which is good in itself, we collected back only 49% of the total forms. This may be a source of non-response bias and may have affected our results as those non respondents may have not had a good experience with the PBL; as a result, our findings should be considered with caution.

 

Recommendations: we recommend a minimum variation of PBL implementation in clerkship within the same institution (UB-FOM). To avoid a possible dysfunctional PBL curriculum, first PBL sessions should take place in all clinical disciplines; but clinical disciplines may choose the approach of PBL case scenario selection as it is not easy to find a good real case during an 8 week rotation block in all disciplines. The institution should organize induction course training of PBL facilitation for non- academic staff (residents), and it should also organize regular PBL facilitation refresher training for both academic staff and non- academic staff that emphasizes on how to provide an effective feedback to students during a PBL session. We also recommend a regular review of the implementation of PBL curriculum in new institutions to detect dysfunctional curriculum on time, and further study that looks at issues such as tools to evaluate students´ performance during PBL sessions.

 

 

Despite students rating PBL process as 'good' and facilitation as 'very good', PBL first sessions were not consistently undertaken. Students seemed not to be satisfied with collaborative learning skills of facilitators. We also recommend a regular review of the implementation of PBL curriculum in new institutions to detect dysfunctional curriculum on time, and methods of evaluation of students' performance during PBL sessions.

 

 

The authors declare no conflicts of interest.

 

 

ST was responsible for the conceptualization of the study, data analysis, write up and compilation of the manuscript. CEN contributed to the conceptualization of the study, for reviewing and editing the manuscript; and RO contributed to qualitative data analysis, for reviewing and editing the manuscript.

 

 

We highly appreciate the support this study got from the Sub-Saharan Africa-FAIMER Regional Institute (SAFRI).

 

 

Table 1: clerkship students’ respondents’ distribution per Bachelor of Medicine class, Faculty of Medicine, University of Botswana, October to November 2015

Table 2: clerkship students evaluating the problem based learning process, University of Botswana, October to November 2015

Table 3: qualitative analysis of clerkship students’ responses on tips to improve problem-based learning (PBL) process and facilitation at the University of Botswana, Faculty of Medicine, October to November 2015

Table 4: clerkship students evaluating the problem based learning facilitation, University of Botswana, October to November 2015

 

 

1. Gwee MC. Globalization of Problem-based learning (PBL): cross-cultural implications. Kaohsiung J Med Sci. 2008; 24(3): S14-22. PubMed | Google Scholar


2. Azer SA. What makes a great lecture; Use of lectures in a hybrid PBL curriculum. Kaohsiung J Med Sci. 2009; 25(3): 109-115. PubMed | Google Scholar


3. Gresham CL, Philp JR. Problem-based learning in clinical medicine. Teach Learn Med. 1996; 8:111-115. PubMed | Google Scholar


4. Macallan DC, Kent A, Holmes SC, Farmaer EA, McCRorie P. A model of clinical problem-based learning for clinical attachments in medicine. Med Educ. 2009 Aug; 43(8):799-807. PubMed | Google Scholar


5. Wood DF. Problem based learning; ABC of learning and teaching in medicine. BMJ. 2003 Feb 8; 326(7384):328-330. PubMed | Google Scholar


6. Dolmans DH, Schmidt HG. What Drives the Student in Problem-based Learning. Med Educ. 1994 Sep;28(5):372-380. PubMed | Google Scholar


7. Amoako-Sakyi D, Amonoo-Kuofi H. Problem-based learning in resource-poor settings: lessons from a medical school in Ghana. BMC Medical Education. 2015;15:221. PubMed | Google Scholar


8. Cox M, Chandra A. Undergraduate emergency medicine in an African medical school-Experiences from Botswana. AJEM. 2013;3:157-163. PubMed | Google Scholar


9. Dolmans D, Ginns P. A short questionnaire to evaluate the effectiveness of tutors in PBL: validity and reliability. Med Teach. 2005 Sep; 27(6):534-538. PubMed | Google Scholar


10. Takkunen M, Turpeinen H, Viisanen H, Wigren HK, Aarnio M, Pitkäniemi J. Introduction of real patients into problem-based learning in preclinical first-year anatomy curriculum. Med Teach. 2011;33(10):854-856. PubMed | Google Scholar


11. Van Berkel HJM, Dolmans DHJM. The influence of tutoring competencies on problems, group functioning and student achievement in problem-based learning. Med Educ. 2006; 40(8):730-736. PubMed | Google Scholar


12. Groves M, Régo P, O'Rourke P. Tutoring in problem-based learning medical curricula: the influence of tutor background and style on effectiveness. BMC Medical Education. 2005;5(20):1-7. PubMed | Google Scholar


13. Machado JLM, Machado VMP, Grec W, Bollela VR, Vieira JE. Self- and peer assessment may not be an accurate measure of PBL tutorial process. BMC Medical Education. 2008 Nov 27;8:55. PubMed | Google Scholar


14. Oda Y, Onishi H, Sakemi T. Effectiveness of Student Tutors in Problem-Based Learning of Undergraduate Medical Education. Tohoku J Exp Med. 2014;232(3):223-237. PubMed | Google Scholar


15. Bochner D, Badovinac RL, Howell TH, Karimbux NY. Tutoring in a problem-based curriculum: expert versus nonexpert. J Dent Educ. 2002 Nov;66(11):1246-1251. PubMed | Google Scholar