Navigating edtech evaluation: Lessons from online testing in low-fee control schools in South Africa
·
Abstract
In this blog we outline the key learnings from our recent baseline assessments of learners in control schools using online assessments administered individually on tablets. We found that:
1. Using standardized digital assessment tools is possible, efficient, scalable, and removes variability associated with face-to-face assessments
2. Staffing doesn’t need to be complicated to deliver edtech evaluations
3. Think ahead and plan for the worst - systems integration and power interruptions
4. Potential digital skills gaps can be addressed using the language children know best
5. Support - from the provincial to the school level - is an important enabler
Research Data Availability Statement
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
Keywords: Educational technology, South Africa, Evaluation
This science communication piece was posted on LinkedIn on 1 December 2023.
Recently Click Learning launched an initiative to evaluate various edtech products that support English literacy and numeracy. In the Eastern Cape, our focus is on supporting English literacy skills at schools which have isiXhosa as the language of learning and teaching (LoLT). In Gauteng, our focus is on supporting numeracy skills for learners in Grades 1 - 4 at English LoLT schools, and learners in Grades 4 - 7 which had an African Language as the LoLT in Foundation Phase.
Our programme enables children to attend computer lab sessions where they work on one of the literacy or numeracy products assigned to their school for four sessions of 30 minutes per week, during school time. We track learner progress through learning analytics on the applications, through time spent on the products measured through our portal, and through two online administered quizzes that measure foundational English literacy and numeracy skills.
In order to understand the impact of our programme, we wanted to compare learners’ progress in our programs to learners in ‘business-as-usual’ (control) schools, who do not have edtech programs nor computer labs in their schools. For comparative purposes we wanted to ensure that assessment was as similar as possible in our program schools and the control schools, and thus we used a single, standard online quiz.
In this blog we outline the key learnings from our recent baseline assessments of learners in control schools1 using online assessments administered individually on tablets. We found that:
1. Using standardized digital assessment tools is possible, efficient, scalable, and removes variability associated with face-to-face assessments
2. Staffing doesn’t need to be complicated to deliver edtech evaluations
3. Think ahead and plan for the worst - systems integration and power interruptions
4. Potential digital skills gaps can be addressed using the language children know best
5. Support - from the provincial to the school level - is an important enabler
1. Measuring progress in literacy and numeracy using online eQuizzes Our online eQuizzes were carefully designed by research leaders in the literacy and numeracy fields in South Africa. These quizzes are self-contained individually completed assessments that are scored automatically, do not need to be facilitated (other than a general orientation), and do not require specialised software to be installed on the laptop or tablets. The quizzes include video instructions, oral prompts, and multiple choice and input response options which learners complete in real time and submit. Learners complete the quizzes in one group, each working on their own tablet with headphones. Thus, the test administration is standardised across learners, and only three assessors are needed to set up the devices, log learners in to the portal, and provide the instructions. These quizzes take approximately 25 minutes each to complete for a group.
2. Staffing doesn’t have to be complicated
Our quizzes are easy to use, so we were able to hire assessors who don't need as much training as those who work on one-on-one tests. Other assessments, such as the Early Grade Reading Assessment, require assessors to be more highly trained (sometimes for six days) since they have to score the test in real time. With our quizzes, we were able to train our youth facilitators who usually work in our computer labs to set up devices, log learners into our portal, collect learners from class, and close out at a school. All this, with eight hours of training and a visit to a pilot school in the Eastern Cape. In Gauteng, we ran training online for three hours to accommodate our distributed teams, and facilitators were still sufficiently trained.
3. Think ahead and plan for the worst - keeping track of learners and overcoming load shedding using systems integration and technology
The eQuiz is delivered automatically to each learner based on their user details registered on our portal. Through agreements with provinces we are able to request school administrative data. This integration enables us to more accurately track learners over time. We also used this data to randomly select learners, and replacement learners, in advance of attending the school, saving valuable time that is often spent on selecting learners on the test day.
We brought all the necessary technology to the schools to conduct the quizzes. This included one tablet and one set of headphones per learner, for a total of 26 tablets per school in the Eastern Cape, and 52 tablets per school in Gauteng. We also included one additional tablet and set of headphones as a backup option. Our eQuizzes rely on internet connectivity, so we also provided a router, with a choice of two internet service providers. Additionally, we mitigated load shedding (electricity blackouts) by including a UPS, allowing us to work through the worst of extended power outages.
4. Addressing potential digital skills gaps through the language children know best One way that children may differ between our program schools and the control schools is exposure to digital technology. We were not able to roll out our online testing prior to launching schools on our programs, so some learners in our programs had up to 8 hours of time on the apps. However, we had no idea how much exposure to technology children in control schools had had. To mitigate this potential difference, we included a technology orientation session at the start of the quiz in the control schools which included explaining what a tablet and headset are and how we were going to use them. We also included an orientation to the online test format which showed children how to listen to the instructions, how to choose an answer, and how to move on. Learners in the program schools also received this same orientation to the technology so that testing was standardized across both groups. For example, in the Eastern Cape, these orientation sessions were conducted in isiXhosa, the language most spoken by the learners in these schools. We found that learners were able to independently navigate the assessment, especially from Grade 3 upwards. Learners in Grades 1 and 2 required some additional prompting from facilitators that they were allowed to move on independently through the quiz and do not need to wait to be told to move on. Nevertheless, teachers that observed the quizzes were surprised at how easily learners were able to navigate the quiz.
5. Co-operation at the province, district and school level goes a long way to ensure the success of the project
Besides the use of technology, and carefully developed and piloted quizzes, the project was a success because of cooperation at the province, district and school levels. The Eastern Cape Department of Education, and the Ekurhuleni North, Ekurhuleni South and Gauteng East districts support the implementation and evaluation of our programme in control schools. This support extended to school principals, teachers and learners who were very welcoming and excited to be part of this project.
Where online testing may not be a solution
We’ve presented the successes of testing in control schools using internet connected devices, even in somewhat rural areas. However, offline testing, for example via Tangerine, SurveyCTO or Kobo Toolbox, could be better if testing in remote areas with no internet access. Additionally, if there isn’t existing access to tablets the costs could be prohibitive.
Looking ahead
In our next steps we will be comparing the data from these control school learners to data from the learners in our Double Click program in the Eastern Cape and Gauteng, setting our baseline. We’ll learn from these lessons to improve our assessment processes for the midline assessment in 2024.
From 11 - 15 September 2023, a team of three assessors collected data from 515 Grade 1 - 4 learners in five low-fee schools (two rural, three semi-urban) in the Eastern Cape of South Africa. All learners completed a literacy quiz, and 102 Grade 4 learners completed a numeracy quiz in English. From 24 October - 3 November, two teams of four assessors collected data from approximately 1800 Grade 1 - 7 learners from 13 schools in Ekurhuleni South, Ekurhuleni North and Gauteng East districts of Gauteng. Eighty percent of these learners completed the literacy quiz, 68% completed the numeracy quiz. Depending on their cohort, some learners completed both quizzes, and some learners completed only one quiz.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
