SpacerLogo Peer Learning Toolbox
SpacerSearch Button

A toolkit for the evaluation of ICT in schools
Schools are complex organisations. That means that to do them justice through evaluation (be it external-, self- or peer-evaluation) there are many things to take into account. Many inspectorates have adopted a formalised procedure and methodologies to ensure reliable and fair evaluations of schools. Such procedures by the way cannot only be of help when carrying out external evaluations, but could mutatis mutandis also be used when schools carry out self- and peer-evalution (material for a future post maybe). When carrying out external evaluations with inspectors from different countries with a diversity of responsibilities, experience and mandates, having such formalised methologies in place is even more important.
P2V WP6 is a cooperation of 6 inspectorates, each of them evaluating 4 schools in their own country. They use the same evaluation framework during those visits, but 'simply' having agreement on the indicators and criteria is not enough to ensure consistency and reliabilty of observations - when agreeing with 6 European inspectorates on evaluation indicators there was a feeling of shared achievement but also of anticipation: will our understanding of things be as uniform once we enter a school? Earlier experience showed that it is necesarry to develop a number of support materials, not only to help experienced inspectors in getting consistent evaluations but also with the aim of possibly extending the use of the framework to people whose everyday activity is not observing and evaluation. To an untrained eye the original P2P framework in itself leaves too many things open, although some schools and inspectors who used the framework 'stand-alone' reported back with enthusiasm. As part of P2V we decided to enhance the framework by adding a number of supports:

  1. an evaluator guidance
  2. a self-evaluation questionnaire 
  3. interview guidance
  4. a lesson observation form
  5. a template report

Besides these materials, we also improved the framework by adding the possibility to add notes and a column in which scores can be assigned to each indicator. Also the framework now contains clues about the sources that can be used to find evidence about a specific indicator. In another post I will elaborate more in the framework itself.

The evaluator guidance contains all kinds of practical information and helps the local evaluator in setting up the visit and in making appropriate chices before, during and after the visit. This document gives tips about making observations, about documenting evidence and about writing and referencing the school report.

Prior to each visit a self-evaluation questionnaire should be filled in by the school that is going to be visited. This questionnaire should be made available to the evaluators before the visit. It contains questions about hardware, vision and policies. The schools are also requested to send policy briefs etc to the evaluators. This information helps making the visit more efficient (as no time is lost during the visit itself to gather this info) and effective (as the visit is more focussed on themes and hypotheses that were identified based on the materials).

The interview guidances help the evaluators during the school visit. There are three guidances: one for teachers, one for management and one for pupils. As part of the methodology, interviews should be scheduled with each of these groups. The questions are directly derived from the quality indicators and evidence pointers in the framework, thus helping the evalutor in making informed decisions.

An essential part of each school visist are the lesson observations. Each evaluator makes notes on a lesson observation form, on which also appropriate boxes can be checked about the subject, type of ICT used, type of classroom etc.

After the visit the local inspector writes a report. In this report he/she elaborates on the scores, providing strengths and weaknesses for each Quality Area. A template report is provided, with a clear format that is linked to the framework.

To an inspector or someone familiar with inspections, most of the above is not radically new. What is new however that we try to develop this toolkit specifically for ICT, aimed at transnational use and building on the body of knowledge and expertise of inspectors in several countries. During our first trials it proved that sometimes a small adjustment in wording, in the order of criteria or in the guidance was needed to prevent confusion on part of the school or the evaluators. The iterative process of the school visits will mean that by the summer of 2008 we will have a toolkit that has passed its practical test.

by Bert Jaap van Oel (originally posted on the P2V Blog-Inspection)

Web Editor: Paul Gerhard
Last changed: Thursday, 26 February 2009
Curved Line