Contents:
Summary
1. Introduction
2. Part 1: the views of users of assessment
2.1 The views of employers
2.2 The views of parents
2.3 The views of higher education
2.4 The views of students
2.5 Overview of users' views
3. Part 2: Project plans and outcomes
3.1 Review of Working Paper 3
3.2 The role of case studies
3.3 Recommendations / guidelines / propositions
3.4 Future plans
Appendix A: List of participants
Appendix B: Seminar programme
Appendix C: Internal summative assessment in National Vocational Qualifications (NVQs): the concerns of employers (Kathryn Ecclestone)
Appendix D: Views of Scottish parents about what matters in learning and assessment (Carolyn Hutchinson)
Appendix E: Overview of some parents' views of summative assessment (Jean de Rijke)
Appendix F: Views of students on summative assessment: what does research tell us? (Judy Sebba)
Appendix G: Report of Open Space Events for Pupils (Carolyn Hutchinson)
Summary
The fifth in the series of seminars held by the ASF project took place at Lucy Cavendish College, Cambridge on July 11 and 12, 2005. It was in two parts, the first of which was a full day devoted to considering the views of 'users' of assessment on the evidence, arguments and principles as set out in project's Working Papers 1 and 2. Inputs included background papers prepared by members of the Core Group, evidence from invited experts and responses from users: employers, parents, higher education and students. In the second part the Core Group met, first to review the evidence from users and then, on the second day, to discuss the content and further development of an early draft of Working Paper 3.
1. Introduction
The seminar was in two parts. In the first, taking place on July 11 th , the Core Group was joined by visitors to discuss the views of various users of student summative assessment: employers, parents, higher education and students themselves. The second part of the seminar, on July 12 th , was an extended meeting of the Core Group to discuss the draft of Working Paper 3, produced since the last meeting, and the further work in the final six months of the project. Appendix A lists the participants and Appendix B the programme for each day.
On the first day each session, with the exception of the short one concerning higher education, began with some evidence from research, followed by the views of those with direct experience as users of particular forms of summative assessment. In the original plan of the project a seminar with this focus was to have been held in January 2005, following the consultation conferences on Working Paper 1, so the participants were responding to the ideas presented in that paper. The day began, therefore, with a brief overview of the definitions, principles, descriptive framework and recommendations in Working Paper 1.
2.
Part 1: the views of users of assessment
2.1 Employers
This session considered the nature of assessment that is used by employers either in recruitment or in training their workforce. The focus was on National Vocational Qualifications (NVQs) as an example from a range of vocational qualifications. In presenting her background paper (Appendix C) Kathryn Ecclestone noted that there is not a great deal of research information about the views of NVQs of employers or evaluations of their effectiveness. She explained the purpose of NVQs and described the role of colleges and of employers in the process. The stated aim is to ensure that training meets the competence requirements of employment. Detailed specification of competence criteria guide the assessment and meeting these criteria comprises both the training and the evidence of competence. This on-going assessment can be formative, in identifying where skills still need to be improved, or summative. In work place assessment the employer is the trainer and the assessor, both the source of assessment information and the user of that information. Thus emphasis is on validity and authenticity, awarding bodies focusing on the extent to which procedures have been correctly implemented rather than on moderating decisions.
In response, Murray Butcher, Director of Compliance, City & Guilds spoke about the process of assessment in NVQ qualifications. A key feature of NVQ courses is that assessment is, or is part of, the learning programme; opportunities to learn are equally opportunities for assessment, and vice versa. Work-place assessors need occupational competence and competence in assessment. Training is provided for the latter, amounting to about 30 to 60 hours of coursework and the creation of a portfolio on assessment. Murray spoke of levels in the process of moderation that are needed to ensure dependability of the assessment. The assessment conducted by the assessor working with the learner (level 1) is moderated by an internal verifier (level 2) who in turn is moderated by an external verifier, on behalf of the awarding body (level 3), the whole process being under the surveillance of the QCA. Thus the number of people involved is considerable, with 1,500 employed by City and Guilds alone, from about 150 awarding bodies recognised by QCA. A good deal of the work within NVQs is the recognition of skills that people already have. Learners who were not successful initially could try again and be assessed as many times as required until they met the criteria. However, unlike an examination such as GCSE, which can be passed without success in all parts, it was necessary to meet all the criteria in order to be awarded an NVQ.
Key points from the discussion were the following: The assessment process was complex, but it was judged to be working well. ‘Teacher assessment' meaning ‘employer/trainer assessment' was appropriate and provided high levels of validity. When a written test was also required as part of a vocational qualification, the greatest reason for drop out was failure to take or to pass the test. The training of assessors and verifiers had to be a constant process, as employee assessors moved from job to job. Training in assessment was the key to the successful operation and to the gradual improvement in the system that was reported. Learners and teachers were both aware of the criteria and therefore self-assessment was a feature at all levels in the moderation processes.
2.2 Parents
Carolyn Hutchinson introduced her report of the views of parents expressed in Open Space meetings held in Scotland by challenging seminar participants to suggest questions that parents would want to ask teachers about their children. These were later compared with the questions found to be of most concern to parents. Parents were least concerned about how their child compared with others or with overall standards and most concerned about how to collaborate in helping their child's progress. The findings were presented in terms of five key themes emerging from a series of Open Space Events. The themes and a brief account of the events are described in Appendix D.
In response, Jean de Rijke, convener of the Devon Parent Teachers' Association, noted that her research findings and impression from working with parents accorded with the Scottish findings. She had collected view of parents in Tower Hamlets, Plymouth and Devon and presented the findings from the 20 responses she had received. An overview of her results is in Appendix E. About two thirds of the parents thought that testing was overall a good thing; they were aware of the effects it can have on the curriculum and on children's motivation, particularly of the lower achievers. As in Scotland , parents' views on the curriculum were linked to their views on assessment. They wanted children to have a wider range of experiences and for out of school learning to be recognised. They thought that children would benefit more from teacher assessment as opposed to external testing and that it would support children's motivation and self-esteem.
In the discussion it was suggested that there were differences between primary and secondary schools in relation to how close the communication between teachers and parents can be. It was also suggested that some teachers like examinations and tests because they provide protection from parents' challenges to their judgments. However it was pointed out that teachers' judgements should be supported by evidence and the criteria of judgement. Better communication at this more detailed level would enable parents to understand and to trust teachers' judgments. The survey had not addressed questions of high stakes external testing such as GCSE, but the evidence in relation to national tests indicated that parents wanted a greater range of information to be included in judging children's achievements.
2.3 Higher Education
Angela Milln, director of admissions at the University of Bristol , described her role in developing the university's admissions policy and ensuring that departments adhered to it. The latter part of the role was to some extent comparable to the ‘internal verifier' role in the NVQ, but in regard to the process of admission rather than the performance of individual students. She described the problems facing Bristol as being the same as those of other universities in trying to identify students who not only had performed well to date but who could benefit from university study. However the high status of Bristol brought additional problems of selection, when a large majority of the 30,000 applicants each year had straight ‘A's.
Responding to questions about how the candidates' teachers' opinions in the school reference were viewed, Angela reported that these were not in general trusted, since schools were anxious to promote their students' chances of admission. Where the school was known to the admissions tutors and a relationship had been established, then the school reference was more likely to be used. What was needed was evidence and honest opinions about whether a student would ‘stay the course' and had learned how to learn. Schools needed to be more honest with their students about their abilities in these respects, so that unrealistic expectations were not built up and there was more self-assessment and self-selection. Evidence from an extended project carried out by a candidate, for instance, was likely to give more relevant information than high marks on tests and examinations that could be affected by cramming. Universities were well aware that many students had spent many years ‘chasing grades'.
Despite the unreliability of grades, universities such as Bristol were looking for finer discrimination between A* and A** etc., which technically makes little sense. More were resorting to setting their own examinations, only adding to students' pressures. In the later discussion it was suggested that there was an opportunity for a revised system, based to a greater extent on moderated and dependable teachers' assessment to provide the information that would be really helpful to universities.
2.4 Students
The research background was briefly summarised by Judy Sebba. She reported that the research specifically relating to students views of summative assessment was ‘thin', most studies having focused on students' views of conditions for learning or their views of formative assessment. Her report is in Appendix F.
John Harland, Director the NFER Northern Office, then reported students' views gathered in the Northern Ireland Curriculum Cohort Study, which followed students from Key Stage 2 through to Key Stage 4, beginning in 1996. The longitudinal study tracked 2,500 pupils through an annual questionnaire, with a sub-sample of 60 pupils interviewed and observed biannually.
Students were reported as finding all methods of assessment helpful, with the exception of records of achievement. The general response of students to tests and examinations was found to be positive, more so at KS4 than at KS3. Lower attaining students saw their test results as a sign of their ability while higher attainers saw tests as showing them how they could improve. At Key Stage 4 students liked tests because they gave a clear cut measure of progress based on ‘right or wrong', enabled them to see their strong and weak points, made them revise for examinations and were good practice for the ‘real' examinations. Thus in a system dominated by examinations, what students wanted was help in passing them.
Students in the Northern Ireland study were concerned about the fairness of project work and impressionistic marking of course work. At Key Stage 4 they were, however, aware that the curriculum was dominated by what was needed to gain the necessary results to enter higher education, but took an instrumental view that this was necessary despite admitting that what they were doing was not particularly useful in its own right for their future lives. What they wanted from assessment was evidence of progress towards the goals viewed in this way; tests gave them this evidence, so they were satisfied.
Carolyn Hutchinson then reported briefly on the Open Space Events for Pupils in Scotland (see Appendix G). The questions raised and discussed by pupils mainly concerned the content of the curriculum, how they learned best and actions relating to dress codes. Pupils wanted more lively teaching (and teachers), more variety and alternatives for study, less ‘teacher talk', more encouragement to do their best at whatever level they have reached. Perhaps testing and examinations did not appear high in their priorities because these do not dominate their daily experience in Scotland as much as in England .
Two students from Hastingsbury School, Hannah Mayston and Lindsay O'Brien, accompanied by one of their teachers, responded to the matters raised in the project. Both students were active in the Student Voice initiative, and spoke confidently about their views and experience. Their views on assessment by teachers were initially negative: they thought that there would be bias in their teachers' judgments, that it would impose an extra strain on teachers and that ‘all staff mark differently'. However, they recognised that the advantage would be that teachers could take the context of work into account, and that with strong, clear criteria and teacher collaboration in using them, bias could be minimised. They described an experience of having to mark their own work and being given the criteria, as being ‘fantastically helpful'. Similarly they could identify pros and cons of tests, recognising that tests impose too much pressure, cause cramming, and detract from the enjoyment of education. They were also aware that current summative assessment did not give them credit for life skills, such as being able to talk to a roomful of academics, education administrators and senior teachers! The question of gender differences in preference for examination or teacher assessment was raised.
These students were probably exceptional in their experience of reflecting upon and speaking about their education, yet they were unaware of basic ideas about the nature and purposes of assessment. A thought expressed later was that we should not romanticise the pupil voice.
2.5 Overview of users' views
Certain themes were perceived as emerging from the discussions of the various user perspectives. They included:
- The key role of fairness in assessment needs to be evident and overt.
- The transparency of the procedures was essential.
- Good communication between all players and users was a high priority.
- The cultural setting was highly relevant at the country level (cf different reactions in Scotland and Northern Ireland ) and in relation to the learning focus (vocational vs academic).
- The notion that teachers' assessment was for the less able and examinations for the more able needed to be explicitly refuted.
- The variability of quality of teachers' assessment has to be faced; teachers need both subject competence and assessment competence
- Maintaining quality in teachers' assessment requires permanent opportunities for training and the establishment of robust moderation procedures.
- There was a need to shift the basis of high stakes judgements both for individual students and institutions.
- The extent to which assessment created outside the classroom (or other place of learning) can truly reflect the range of learning inside the place of learning ought to be questioned.
3.
Part 2: project plans and outcomes
3.1
Review of Working Paper 3
The general view of WP3 was that was helpful in clarifying concepts and pros and cons of assessment for different purposes. The content was considered useful for influencing policy-makers when making their decisions relating to their own educational contexts. It lacked, however, a statement about costs. Also, some terminology needed to be revised, both to avoid jargon and to avoid presenting a biased view of some of the arguments. More could be said about the purposes of the assessment in each of the case studies. The differences between purposes of assessment affecting individual students and those relating to impact of student summative assessment need to be drawn out.
There was some difference of view as to extent to which the paper should take a position. Some urged a bold approach in comparing summative assessment by tests and by teachers' assessment, addressing the ‘hard choices' full on. This means taking a stand about the nature of education that we want to promote. Others preferred a more circumspect approach, arguing pros and cons, etc. In either case it was necessary to make explicit the basis of the arguments in terms of what is considered to be of most value to students' education as well as of value to those who want information about students' achievement. Giving students greater responsibility for their learning has implications for assessment processes. Current policy statements are sympathetic to such arguments, providing an opportunity to be seized.
3.2 The role of case studies
The case studies were considered to be a useful part of the paper but one that could take the arguments further given some revision. Richard agreed to co-ordinate the development the case studies as a separate section of the paper – or even a further Working Paper – with the design sections and properties more clearly linked to the ‘components' of WP1. These would also be a further section for each case study going beyond description and analysis to suggest, where appropriate, what could be done to make more appropriate use of assessment by teachers and how this would affect the properties of the assessment in relation to its purpose and to maximise the quality of summative assessment .
3.3 Recommendations / guidelines / propositions
The great complexity of different summative assessment sub-systems revealed in the project's work to date raised questions about the feasibility of identifying overall recommendations. Indeed, challenged individually to write a statement of recommendation, members tended to identify one particular curriculum area and content area in order to be sufficiently specific. Statements then attempted to encompass the procedures needed to ensure that the impact of the assessment was a positive as possible. (For example: a recommendation ‘that teachers use their observations and interaction with learners to make judgements about learning achieved, based of the latest and best evidence, judged against explicit progressive criteria, shared among teachers and with learners and with judgements moderated within and across schools').
In the light of this discussion it was agreed that recommendations could not be made to suit each context and that it was preferable to identify some propositions that could be applied in any situation by those setting up or modifying assessment systems. The final sections of the case studies, as noted above, would indicate the kinds of actions that would meet the criteria reflected in the propositions. The aim was to support thinking that starts from the concept of what is good summative assessment in a particular context and for a particular purpose.
The recommendations already in Working Paper 1 were revisited during this discussion. They had been described by some at the consultation conferences as too ‘bland', possibly because they attempted to include a rationale for each recommendation. Also, in being addressed at the system level, they were necessarily broad and somewhat vague. Rewording into tighter statements was suggested and Judy undertook to collate suggestions and produce a revised list, that was more in line with the suggestions for propositions.
3.4 Future plans
Actions agreed were to reorganise Working Paper 3 to provide a justified theoretical framework and implications for the kind of information that is needed and the evidence from practice and research of gathering and using it in various ways. Reference to the case studies and to an appendix on illustrative costs would provide some guidance as to how the principles can be put into practice. The aim was to inform without recommending or dictating specific actions. Wynne would draft this for the next meeting and collect information on costs from Jon, Janet and Carolyn, while David Bartlett offered to ask David Budge about the figures he used at the consultation conference and Kathryn would follow up costs of City & Guilds NVQ s through Murray Butcher. Wynne will also make an initial draft of a short booklet.