1. Four Criteria of Effective Assessment and Evaluation
    2. The Need For Some Criteria Of Effective Evaluation
    3. ##Maybe box these
    4. Criteria of Effective Assessment and Evaluation

 
SESSION 6
ASSESSMENT AND PLANNING FOR A BALANCED LITERACY PROGRAM
 
 
GROUP REFLECTIONS ON SESSION 5
30 minutes
Purpose  The purpose of this time is:
•  to share participants' experiences of the Between Session Instructional Strategy and the insights they gained from the Between Session Reading.
 
groups 3-4
Ask participants to share their experiences using the Instructional Strategy in their classrooms. Encourage participants to show any samples of work from their students and if possible display these for all to view.
 
After 15 minutes remind participants that they need to discuss the Between Session Reading. Use OH2:2 as a discussion starter if necessary.
 
INTRODUCTION TO SESSION 6
5 minutes
whole group  In this session we are going to focus on a framework for thinking about assessment and planning for a balanced literacy program. We will highlight the important links between beliefs about learning, teaching practices and assessment and evaluation practices that have been highlighted in each of the Implications for Evaluation in the past sessions. We will explore four criteria for effective assessment and evaluation, and we will argue that if these are in place we are in a better position to not only understand what our students can do with respect to literacy learning, but also to plan teaching/learning experiences that lead to increased student learning. We will consider how this is reflected in practice by viewing a classroom in action.
 
Finally we will consider the question: Where to now? as we each develop a personal plan of action.
 
Remind participants of key points identified in previous sessions by showing OH6:1 and talking to each point briefly
 
Show OH6:1
Evaluation is an ongoing process, occurring before, during and after instruction.
Evaluation includes the process not just the product.
Evaluation is both formative and summative.
Evaluation drives instruction.
Evaluation views approximation as a demonstration of a learner's valid attempt to achieve the conventional form.
Evaluation uses learning logs, interviews, observations and careful examination of products as the evidence of student learning.
Evaluation is carried out over the range of genres of language.
Evaluation reflects the teacher's beliefs of what makes an effective reader/writer.
Evaluation should result in optimal learning for all those involved.
 
Close by pointing out that they will have an opportunity to discuss these these key points in the first workshop.
 
INPUT/WORKSHOP 1: FOUR CRITERIA FOR EFFECTIVE EVALUATION OF OUR STUDENTS' LITERACY DEVELOPMENT
50 minutes
whole group
Introduction.
5 of 50mins
In the past five sessions we have focused our thinking on the assessment and evaluation of our students’ literacy learning during the section Implications for Evaluation. If we were to categorise the outcomes of our discussions and thinking we believe four criteria would emerge. These criteria can be used to frame our thinking about the assessment and evaluation of our students. In this section we are going to consider each of these criterIn the past five sessions we have focused our thinking on the assessment and evaluation of our students’ literacy learning during the section Implications for Evaluation. If we were to categorise the outcomes of our discussions and thinking we believe four criteria would emerge. These criteria can be used to frame our thinking about the assessment and evaluation of our students. In this section we are going to consider each of these criteria and what each means for our assessment and evaluation practices.
 
NOTE: It needs to be pointed out that a focus on assessment and evaluation can only be a cursory consideration in this session. An Assessment and Evaluation Module has been developed for a far more indepth study of assessment and evaluation theory and practices, use of standards and reporting to stakeholders. Point out to participants that they may want to consider participating in this Module at a later date.
 
Purpose: the purposes of this workshop are to:
Procedure:
Refer participants to ‘Four Criteria of Effective Assessment and Evaluation’ on pages** in their Notebook.
 
Show OH6:2
OH6:2
Criteria of Effective Assessment and Evaluation
  1. Assessment and evaluation must result in optimal learning for all involved.
  2. Assessment and evaluation must inform, support and justify teacher decision making.
  3. Assessment and evaluation practices must reflect the theories of language, learning and literacy which guide our teaching.
  4. The findings which result from our assessment and evaluation practices must be accurate, valid, reliable, and perceived to be rigorous by all who use them.

NOTE: You will note that the terms ‘assessment’ and ‘evaluation’ have mostly been used together. We define ‘assessment’ to be the gathering of ‘data’; whereas ‘evaluation’ is the making of value judgments on such data. It is these judgments - the outcomes of the evaluation process - that we use to report to others about the students’ progress and to make informed decisions about future teaching/learning experiences. We find it difficult to have one process happening without the other so tend to use then together. These definitions are in the next input section - however you may want to share them with your participants at this point and possibly have them already written up on a chart so there is a shared meaning of the terms.
 
Refer participants to page?? in their Notebooks to the short article written by Brian Cambourne and Jan Turbill, Four Criteria of Effective Assessment and Evaluation.
 
Explain that they are to read the article and at given points you will ask them to stop reading and work collaboratively in their group on the reflective tasks listed.
 
NOTE: You may want to ask groups to list their responses of charts so that all can share the responses at the end of the workshop. The timing for each section is as follows:
 
Section 1
15 of 45 minutes
 
Section 2
10 of 30 minutes
 
Section 3
10 of 20 minutes
 
Section 4
10 of 10 minutes
 
Ask participants to begin the first section of the reading. Tell them that when they have read this first section to continue with the reflective task. Point out that your role will be to move around the groups and remind them of the timing for each section. Remind them that the timing for each section is in their Notebook.
 
##LINDA THIS NEXT PART IS TO GO IN THE NOTEBOOK. I HAVE LEFT IT HERE SO THAT PEOPLE WHO READ THIS AND COMMENT CAN SEE WHAT IT IS THAT THE FACILITATOR IS WORKING WITH. BUT EVENTUALLY IT SHOULD BE IN THE NOTEBOOK ONLY. THE BOXED REFLECTIVE TASKS NEED TO BE SHADED OR SOMETHING SO THEY LOOK DIFFERENT TO THE WORK GRIDS. IT WOULD ALSO BE GOOD IF WE COULD PUT THE TEXT AND REFLECTIVE TASKS ON THE LEFT PAGE AND THE WORK GRIDS ON THE RIGHT. JUST A SUGGESTION.
 


Four Criteria of Effective Assessment and Evaluation



Four Criteria of Effective Assessment and Evaluation
Brian Cambourne & Jan Turbill 1

Section 1 15 minutes


The Need For Some Criteria Of Effective Evaluation



The Need For Some Criteria Of Effective Evaluation
In our work with teachers in the Assessment and Evaluation Project 2 we realised that many of us had different notions of what constituted ‘effective’ or ‘good’ evaluation. This became obvious when we began the process of trying to develop sharing meanings around these concepts. We realised as we tried to make explicit what we thought effective evaluation looked like in practice, that this was something we had taken for granted (‘everyone knows what effective evaluation is’), but had never really thought too deeply about. Our initial discussions on this subject were shallow and we found much of what we came up with relatively trivial.
 
We all agreed that unless we could come to some agreement on this issue, that unless we could at least tentatively agree on some working definitions of effective assessment and evaluation we would not get very far.
 
Our first step was to develop working definitions for the terms ‘assessment’ and ‘evaluation’ that we could all accept. We agreed on the following:
 
Assessment and evaluation are interdependent parts of the same process.
Assessment refers to the collection of information or ‘data’.
Evaluation refers to the interpretations and placing of values on the information that results from the process of assessing.
 
We soon found however, that as soon as one begins to collect information or ‘data’ as we began to call it, one begins to make value judgments on that data, thus turning the bits of data into informed judgments. And one cant make such judgements without sufficient data collection. Thus we soon found ourselves referring to the terms always linked - ‘assessment and evaluation’ and then to the acronym A&E. If nothing else this demonstrated to us how inextricably linked the two processes are. We also recognised that often we made judgments ie. Made evaluative judgements about students learning without collecting sufficient data to justify our judgments. The process of developing a shared meaning around these two definitions led to a great deal more understanding than we began with.
 
As well as developing shared meanings about what the terms 'assessment' and 'evaluation' meant, we decided to try to develop some shared meanings about the nature of ‘effective assessment and evaluation’. We soon discovered that the issue of what constituted effective assessment and evaluation was not easy to determine either. After much reading, discussion and debate (often heated) we agreed to these criteria as a tentativeAs well as developing shared meanings about what the terms 'assessment' and 'evaluation' meant, we decided to try to develop some shared meanings about the nature of ‘effective assessment and evaluation’. We soon discovered that the issue of what constituted effective assessment and evaluation was not easy to determine either. After much reading, discussion and debate (often heated) we agreed to these criteria as a tentative beginning.


##Maybe box these



##Maybe box these


Criteria of Effective Assessment and Evaluation



Criteria of Effective Assessment and Evaluation
  1. Assessment and evaluation must result in optimal learning for all involved.
  2. Assessment and evaluation must inform, support and justify teacher decision making.
  3. Assessment and evaluation practices must reflect the theories of language, learning and literacy which guide our teaching.
  4. The findings which result from our assessment and evaluation practices must be accurate, valid, reliable, and perceived to be rigorous by all who use them.

In what follows we shall elaborate on each of these criteria.
 
1. Assessment and evaluation must result in optimal learning for all involved.
In one sense this is a kind of 'parenthood' criterion. Like parenthood, everyone would agree that it's a 'good' thing to have an assessment and evaluation procedure that resulted in optimal learning for all. It's almost impossible not to agree with such a laudable criterion. It's what lies behind it, the things that it presupposes that we found difficult . It meant that we had to begin to address such issues as:
What is optimal learning?
Who should be included in the ‘all’?
Who should benefit from any evaluation that is carried out?
 
It didn’t take us long to realise that we couldn’t address the question ‘What is optimal learning?’ until we had sorted out what effective reading, writing, language development, learning, etc., were.
 
This forced us to examine our values and beliefs about learning, literacy, language, and teaching. We discovered that not all of us had the same ideas about what 'optimal learning' actually was. Neither did we agree with who should comprise the 'all'. However, the fact that we didn't share the same beliefs about these things did not prove to be a hindrance. We realised that we could tolerate differences of opinion in these areas, as long as we had gone through the process of trying to make them explicit for ourselves and each other.
 
Reflective Task 1
Read the key points that we have learned in this course listed in the grid. Beside each note one or two classroom practices that reflect such a belief. The first two have some examples. You may think of more.
 
 
Beliefs About Language And Learning This Means In My Classroom I ...
language is a social process learned through social interaction for social purposes
  • have lots of group work
  • encourage students to support each other
we use language to achieve our needs and to make sense of the world around us
  • ask questions that help students make connections
  • make explicit the purpose of activities
we learn language, we learn through language and we learn about language simultaneously as we use language  
learning language is a life-long process of learning to use the appropriate genre in various situations in which we find ourselves  
the greater the repertoire of genres we have learned, the more empowered we are as language users  
language is more than simply communication; it is a powerful tool for thinking, for coming to know, for learning.  
the components of language - talking, listening, reading and writing - are highly interrelated.  
reading and writing are connected; when we read we are also learning about writing and that when we are writing we are also learning about reading. We learn to spell through reading for writing.    

Section 2. 10 minutes
2. Assessment and evaluation must inform, support and justify teacher decision making.
The previous criterion ‘optimal learning for all’ influenced the generation of this one. We strongly believed that among those who should ‘optimally learn’ from any evaluation procedure should at least be the teacher who is doing the evaluation, and the learner being taught, and the learner’s parents. We could see little point in an evaluation procedure which doesn’t help teachers, learners, and parents make wise decisions about what should be done next. We would argue for example that standardised and/or diagnostic tests of reading required by the District or State given for the sole purpose of accountability is not good evaluation practice. These tests may serve an important purpose for the administrators and politicians but rarely do they inform the teacher, student or parent in ways that they could say they have learned from them. There are however, many of the ‘alternate assessment’ procedures of portfolio assessment, observation surveys, running records, listening and questioning students that will and do provide teachers with rich data that they can use to not only report on students’ progress but also can use to inform, support and justify their future teaching/learning experiences.
 
Reflective Task 3. List the many ways that you use to collect information or ‘data’ on your students’ progress.
 
## MAKE THIS LARGER IN NOTEBOOK
‘Ways’ to collect ‘data’ or information
  • observe students

 
 
 
 
 
 

 
Section 3 10minutes
3. Assessment and evaluation practices must reflect the theories of language, learning and literacy which guide our teaching.
We agreed that there was little benefit from teaching in a way which for example encouraged learners to go through the process of drafting, revising, editing and proofreading to produce a piece of writing, and then evaluate what has been learned by making the learner compose a one draft, perfect piece without the opportunity to draft, revise and edit. We could see no point in encouraging young learners to use the ‘have-a-go’ strategy in spelling and them evaluate them on a right/wrong basis in a weekly test. In fact any evaluation practice which was not congruent with the way we taught language could only serve to send mixed messages to learners, which could only confuse them. How could optimal learning for all result from such incongruity?
 
Reflective Task 4 Consider some of the assessment evaluation practices that you are using now that reflect your theory of literacy as well as any that don’t and that you would like to change (even if you can’t)?
 
##NEED TO ADD MORE ROWS TO THIS
Assessment and evaluation practice Supports my beliefs Need to change because it does not support my beliefs,
• analysing students’ spelling ‘miscues’ children need to approximate in their learning; they use a variety of spelling strategies  
• spelling test   children not given opportunity to examine own words and ‘fix’ them.

Section 4 10minutes
4. The findings which result from our assessment and evaluation practices must be accurate, valid, reliable, and perceived to be rigorous by all who use them.
The teachers who participated in the project were all aware that the prevailing model of evaluation was one which insisted that evaluation could only be valid if it was measurement-based, objective and had something called ‘high reliability’. Many of them had done courses such as ‘Statistics 101’ while at University, and were uncomfortable about using assessment and evaluation procedures which could be labelled ‘subjective’, ‘biased’, ‘unreliable’, ‘soft’, or ‘invalid’. They also realised that many of those to whom they were accountable (parents, principals. system-wide administrators, policy makers, resource-givers, politicians, etc) were locked within the measurement -based paradigm and could be highly critical of any approach to evaluation which had even a tinge of subjectivity or ‘softness’ about them. Therefore we decided that this criterion was absolutely fundamental to any assessment and evaluation procedure. Unless one could demonstrate that the procedures which were being used were at least as ‘scientific’ as measurement-based evaluation, then it would not result in optimal learning for all.
 
How to ensure the results are ‘rigorous’ and ‘scientifically respectable’
Reputable scientists who work in fields like anthropology, ethnography, sociology, investigative journalism, and program evaluation have identified a range of procedures which they argue can be applied to data in order to ensure that research findings are 'trustworthy' (i.e. 'reliable') and 'credible' (i.e. 'valid'). These procedures have technical names which might be off-putting at first glance. However the concepts that these labels represent are really quite logical Many of them are things that teachers are already doing, but are not aware of. When teachers are consciously aware of what they are, how they work together, and why they need to be consciously used, that they become empowered evaluators.
 
Table *** summarises what these procedures are, and how teachers in the project put them into place.
 
##THIS COULD GO ON THE RIGHT SIDE.
Scientific procedures social scientists use
How teachers can implement these in the classroom
• prolonged engagement on the site
 
Being there
  		classroom teachers, by the very nature of their job, can't help applying this procedure. It's a by-product of the job
• persistent observation,
 
Observing until you know what's salient
  		observe, in a focused way, what goes on in classrooms
• triangulation
 
Crosschecking
  		crosschecking by using multiple and different sources of data about the same literacy behavior.
• peer debriefing
 
Having a peer keep you honest   		 checking findings with a peer. For example at weekly grade meetings teachers discuss interpretations of different students' learning growth, justifying how these conclusions were arrived at.
• negative case analysis
 
Consciously looking for examples that could disprove your emerging interpretations 		 
  teachers shouldn’t let themselves get too carried away by first impressions, or by any unconscious biases they might have
• referential adequacy
 
Keeping lots of products ie ‘references 		 
' that confirm your interpretations
  teachers already do a lot of this by using 'folders' or 'portfolios' of student products.
member checking.
 
Taking your interpretations back to the members of the 'tribe' and asking them to tell you whether they agree   		 taking judgements of student learning back to the students themselves or parents or perhaps other teachers and asking them to comment on the findings and conclusions.

Reflective task 4. Share with the people in your group how you put these procedures in place now, or could if you feel you don’t now.
 
Recap and Closure
10 minutes
Close workshop by sharing key points from groups or charts displayed. Recap by reiterating that
INPUT/WORKSHOP TWO TOWARDS A BALANCED LITERACY PROGRAM
50 minutes
NOTE: There are two videos. One is a Grade 1 American teacher. The other a Grade 6 Australian teacher. You will only have time to show one video so you will need to choose which one best suits the needs of your group. If you have a K-6 group you may want to use two videos and monitors so that the videos can be show simultaneously to the appropriate group.
 
5 of 50 minutes
Introduction  We are going to observe video clips of teachers who believe in integrated learning philosophy teaching their students. We will examine the balanced literacy programs presented in these classrooms and consider not only what they do, that is, their teaching practice and organization, but why they do what they do, that is, their beliefs about language and learning. Ultimately all teachers should be able to articulate the beliefs underlying their teaching practices as these teachers can do.
 
 
Purpose  The purposes of this Workshop are:
 •  to observe integrated language teachers in action
 •  to focus specifically on:
   -  what the teachers do
   -  how they talk to the students
   -  how they organize the classroom
 •  to consider the beliefs that these teachers appear to hold and how their beliefs drive their practices.
 
Procedure  Show OH 6:3 Discussion Questions for Videos. Ask participants to choose one question and as they watch the video make notes in response to the chosen question. Make sure that all questions are covered. You may decide to allocate questions if you have a small group. Suggest that participants respond to their chosen question(s) in the grid as they watch the video.
 
Tell participants that they will continue to fill out the grid on page ?? of the Notebook when they have finished viewing the videos. They will consider not only what the teacher is doing but why we think she does what she does.
 
Finally participants will consider what they are doing now, what they want to change and how they might begin to do this.
 
whole group
OH 6:3
Discussion Questions for Video
 
1  How do the teachers organize their students? Why do you think they do it that way?
2  How do the teachers organize the physical space? Why do you think they do it that way?
3  How do the teachers organize time? Why do you think they do it that way?
4  How do they interact with/question students? Why do you think they do it this way?
5  What Instructional Strategies do they use? Why do you think they choose to use these?
6  What resources do they use/do they have in the room? Why do you think they use/have these resources?
 
Remind participants that these questions are also in their Notebook on page **
 
30 of 45 minutes  Show videos.
 
Workshop
groups 3-4
15 of 15 minutes  Ask the groups to respond to their questions, jotting down their responses in the grid entitled 'The Teacher's Actions and Beliefs' on page ?? in their Notebook. Remind them that they will be expected to share their responses with the other groups at the end of the Workshop and combine their knowledge to complete the Group's Grid.
 
Model one question and response using OH 6:4. The Teacher's Actions and Beliefs.
 
NOTE: Many of the responses will overlap. You may need to point this out and remind participants that the teachers' beliefs drive their actions and thus their beliefs will come through consistently. Examples have been included in this Manual for your benefit. These are not in the Course Notebook.
 
Show OH 6.4
Teacher’s Actions/Practices Teacher’s Beliefs
1. students sometimes as a whole class; sometimes in groups language is learned through social interaction; they need opportunities to ‘have a go’ in non-threatening contexts
2. teacher needs space for students to be together so she can read to students need to be immersed in the language to be learned
need to feel a sense of ‘community’
3.
  		 
4.  
5.  
6.    

Ask participants to share responses and complete the group's grid.
 
Point out to participants the importance of being able to articulate the beliefs which drive the teachers' practices.
 
Where to Now?
Individual and whole groups 10 minutes
 
Point out to participants that they will finish course by considering their future plans.
 
Ask them to turn the Action Plan Grid in their Notebook on page ?? and list two to three things that they would like to change in their teaching of literacy. In the second column ask them to indicate what they will need to do to begin such a change.
 
Show OH6:5 and demonstrate with the example listed.
 
What I would like to change How I will go about this change
More group work • reread Instructional Strategies to help me see where groups can be used
• identify ‘meaningful’ learning tasks for groups
• reorganiz
e my classroom so it easier for students to work in groups  
   

 
 
Remind participants that although the Course has finished, their learning has really only just begun.
 
It is important that they take time to revisit the readings, try each of the Instructional Strategies and continue to reflect on their classroom practices in their Learning Journal.
 
Point out that there are other Framework courses that they can participate in at a later time.
 
These are:
 
A six session Assessment and Evaluation Module
A four session Spelling Module
A four session Phonics Module.
 
Now it is party time! Let’s celebrate!
 

Back to top