Welcome to the Evaluation section within the first part of the programme you will have (1) identified a problem, (2) carried out some reconnaissance of the problem and (3) focused your inquiry.

Within this phase of the course, we have looked already at (4) how to plan your evaluation. We have focused on tools for data collection including interviews and focus groups, surveys, documentary evidence, pupils' work, observations and field notes and diaries. We have also considered (5) your implementation of your enquiry. We will now go on to look at the evaluation of your enquiry.

Moving from describing to evaluation

In writing an account of implementation you are very likely to have touched on your initial evaluation of your innovation. For example, it is very difficult, if not impossible, to explain why you might have changed track in your implementation without mentioning the critical feedback from your peers, pupils or learners. Further, you might have found find it unnatural to talk about carrying out data collection without giving an overview of the results. There is no easy distinction between reporting and offering an evaluation of your project. You will need to draw the line where it seems most sensible for your project. However, no matter how you do it your evaluation will need to address:

• Describing what took place
• Collating the different evidence on what took place
• Analysing what took place
• Comparing what took place to other types of innovation reported elsewhere

Describing what took place

In describing results you are being asked to organise the data you have collected. One way of doing this is to follow a three step process:

• Review the data
• Represent the data
• Select and amplify the data

Reviewing the data

In reviewing your data you are going trying to get an overview of what you have found out. A key decision here is to decide whether to approach this in a top down or bottom up fashion. In a top down approach you are looking for what was said or reported about the particular themes or issues which you have already identified as important. In a bottom up approach you are trying to see the data afresh: if you did not know anything about the topic what would strike you as most important? Of course, in practice you are likely to alternate between a bottom up and top down approach. For example, even if you are looking for what pupils are saying about a particular implementation in a lesson but discover, through listening to interviews, that you have uncovered significant, but unexpected, worries about school in general you are unlikely to ignore this finding. Likewise, you can never look at the data with completely fresh eyes. One of the readings we gave you (Strauss and Corbin 1990) describe the bottom up approach very well in a scenario concerning the social scientist at the restaurant. However, even in the restaurant scenario think about how much prior understanding the deliberately 'naive' researcher brings to the task. It is worth thinking about these two approaches as an orientation to reviewing the data - and we would encourage you to approach the data with fresh eyes before homing in on the issues that most concern you.

 

For you to do Review your data, this could be tapes of interviews, diaries, pupils' work, survey returns (even the physical surveys before you collate the results) and ask yourself what general impression are you gaining of the innovation.

Representing the data

Your earlier task of reviewing the data was almost certainly made difficult by its inconvenient format. For example, conversations are time consuming to listen to, survey returns unwieldy, pupils' work can be overwhelming. You will have quickly realised that you need to represent the data in a more condensed and accessible form. How to this appears more obvious in the case of survey data. There are numerous options for displaying tabulated data other than simple tables. Graphical options include bar charts, line graphs, pie charts and so on. In addition such data can be summarised as an average figure. An introductory overview of using data and statistics is available online here. Presenting numerical data need not be a daunting task. Simple charts and tables can help you get a handle on patterns in your data as well as provide a clear and concise way to report your findings. Even if you feel a sense of dread at the thought of 'doing some maths' you no doubt make sense of and use tabulated information in your everyday life.

 

A practical way to create charts and graphs is to use Microsoft Excel. An overview of how to generate a graph or chart from a table of numerical data is available online here. Note if you do not have a copy of Excel on your machine you are able to buy a discounted copy though the University (link).

 

For you to do Have a go at using Excel to present some tabulated data from your project. Consider what you want people to take from the data and experiment with different representations (table, pie chart and so on). Write a paragraph or two of explanation.

What did you learn from or discover in the data that you did not know before? Which representation is the most appropriate for your purpose? Why? What does your text description bring to the data?

More work is required to make sense of qualitative data for example that interview data. The first task is usually to transcribe the data. This is a time-consuming process and 5 minutes of audio recording can take up to one hour to transcribe. Something is invariably 'lost in transcription' but it does force the researcher to pay close attention to detail and it can missed insights. On a small project undertaken by one person there are obvious time limits on how much you can transcribe and tit may be perfectly adequate to make detailed notes and verbatim quotations rather than a full transcription. Look for key illustrative quotes which you might want to include in your study.

 

A detailed overview on how to prepare and transcribe data for analysis is available from the Center for Applied Ethics and Professional Practice. Note that various new technologies can assist with transcribing (1) and processing data (2).

Transcribing Machine

For you to do If you have collected audio-visual data consider:

How much of it are you likely to need to transcribe? Can you think of any of the data that definitely deserves transcription? Why? Can you think of any of the data that definitely does not deserve transcription? Why not? How will you identify which other parts are worth transcribing?

 

 

Once your data is transcribed or written up what do you do with it? One approach is to code your data so you can easily summarise what one person has said about a topic or more likely to compare and contrast what a set of people have said about something. How do you know what to code? As discussed earlier you might approach this in a top down way marking up each section of text which relates to your key issue or a bottom up way making up codes that best fit the data in front of you. More likely you will go for a combination of both. Researchers often speak about reaching a saturation point so that they have enough codes to deal with the data they are working with.

For you to do This is a brief except from an interview between a teacher and a 12 year old girl in a small scale action research project involving the use of multi media in school, it is discussed in Collins, Hammond and Wellington (2001). The teacher has been using a CD ROM of Shakespeare's play Macbeth in order to investigate whether multimedia discs might increase learner engagement in the text. Data collection methods have included an observation schedule for children at a machine and a questionnaire for children who have worked on the disk. The first scene of this disc shows an image of the three witches accompanied by a sound track. On clicking a certain button the text appears and by clicking another button the text is read out by actors as if performing the play. T: Can you tell me about what you have been looking at? P: It is the scene with the witches telling Macbeth about what is going to happen. T: What do you learn from the video clip? P: You can see they are bad from the way they look at each other. They are in a storm, the voice tells you she's quite old, she's not posh..She looks like she's an outcast left to live in a cave or something. Her voice is gravely and rough - she might be ill or something... T: What about the backdrop? P: It is depressing scenery, bleak like a wasteland, it reminds me of the moors, quite scary, like you're on the moors in the middle of a storm. T: Would you rather read it or look at the pictures? P: Without the pictures you would not really know what they looked like, you might think the witches are young and beautiful, they could be like in a fairy tale with Medusa or a talking frog but here you can tell they are real and you can tell who's speaking and things like who has got a limp and what they look like. T: What do you think about hearing the text. P: The music tells you it is about horror...and tell you the periods it was from - it was sort of medievalish....when you are reading you might say it really fast...you might say it the wrong way, put the full stops in the wrong places.

How might you code this data?

REFLECTION The project was asking in general terms if pupils liked using multi media. Question 4 asks this question directly and it might be possible to code the pupil's response (which is thoughtful and ambiguous) with those from other pupils. The more likely approach to coding this data would be to mark up what the child says about different media, for example the second pupil response is about learning from the video clip, the third response is about static images, the third is about images in general, the forth is about learning from sound. However the text may be treated in radically different ways. For example you might want to mark up the metaphors she uses ('bleak like a wasteland', 'like a fairy tale' and compare these with pupils talk inspired by print media.

An overview of How and what to code can be found at the Online QDA website.

Coding allows you to make sense of your data and the same principles may apply in many other contexts beyond interviewing, eg open ended interview questions, pupils' work and so on

 

Top of page

Selecting and amplifying the data

Now you have got your data into a more manageable form the temptation is to show off the whole set of charts and tables you have produced to a reader. However this would be overwhelming for the reader and even if you will need to make continual reference to your charts and table be discriminating in what you include in your report. Help the reader by providing descriptive, explanatory text. This text can of course be heavily influential in how the reader views the data. Consider this data which arose out of the reconnaissance phase of a project aimed at developing the use of homework in school.   strongly agree agree disagree strongly disagree I find homework useful 28 20 38 14 An accompanying text could quite easily state that most pupils did not find homework useful many pupils did not find homework useful but some rated it as very useful many pupils found homework useful, and more than a quarter strongly agreed that it was useful, but some disagreed more pupils strongly agreed that homework was useful than strongly disagreed Each of these statements can lead the reader to see the data in different ways. This might be seen as an open invitation to manipulate the reader but we would rather see it as showing the importance of taking care in your presentation of data and trying to follow a consistent pattern, for example always showing the majority view before the minority view and being consistent in, and explaining, your use of words such as many (for example this could refer to more than two thirds of respondents), some (these could be a third or less of respondents), most and few.

 

Examples of the reporting of tabulated data can be seen in the articles The Implementation of Peer Assessment: an action research approach (3) and Improving University Teaching: Responding to feedback from students (4) in your Readings for LCBE (2) folder. Note that tabulated displays are not confined to multiple choice responses. You may present or summarise the results of responses to an open survey in a table or offer a tally of the number of instances of a certain code in transcribed data appears as in the example of an evaluation of an action research programme (Hammond 2006) which we gave you in your set of readings.

An example of a discussion of data analysis can be seen on page 7 and Appendix D of this write up.

An example of analysis of transcribed video recordings of teachers can be found in the following conference paper.

Many online journal articles can be accessed (using your ATHENS login) that contain verbatim excerpts of transcribed data. For example, the following paper describes observations of a teacher's mathematics lessons.

For you to do Have a go at presenting this data as you might in your final report. Consider...

What raw data, if any, to include... transcribed excerpts? ...coding categories? ...tabulated findings? How best to describe the data including... context? ...descriptive prose? ...interpretations? ...explanations?

 

Top of page

 

Collating the information

You will have reviewed, represented, selected and amplified the data you have collected. Your next step is to bring this all together to present a credible description of your innovation. Your data may of course be a single data source at a single point in time (for example a particular innovation seen through the view point of a pupil) but is much more likely to consist of different data sources (or different groups such as students, teachers, observers) at different times (e.g. interviews before and after the innovation took place. Each project different and it is very difficult to lay down rules as to how you might want to treat these different data sources, but one approach is to triangulate data against each other.

Triangulation is a term taken from surveying and consists of reaching a mid point by taking three or more readings. This is not always a healthy metaphor for practitioner research. For example it is precisely the differences within groups of participants or between groups of participants which we might be most interested in or the same group at different times. There may be no obvious mid point between your perspective and say some of your colleagues or your pupils. Your interest may be in knowing why views diverge, access to which experiences or sets of knowledge have made your participants see things in different ways? In contrast knowing that there are shared perspectives between groups or within groups at different times can lead to greater confidence in interpreting your data. For example if questionnaires and interviews both indicate that your innovation was a great success then you may feel more confident in saying this; likewise if all your colleagues felt that your project ignored certain learning styles again you may feel more confident that stating that learning styles were an important issue. It is very difficult to predict which sets of data you might want to bring together but a useful approach in many studies is to construct a table showing the main findings and how these findings were shared or diverged between different groups. Again provide a commentary to go alongside the table, below are some the kinds of statements made by teachers in reporting on different projects: The innovation was well received. Most students agreed and most teachers agreed and most observers agreed. The innovation met the needs of more able students. Interviews and questionnaire returns showed this. The innovation was constrained by time. Most students raised concerns over the amount of time spent on the project in an exam year and all teacher colleagues expressed concern over how well the project goals matched the exam syllabus. The innovation met with a mixed response both from students and from colleagues. Students who were setted in lower ability groups and teachers of these groups were more supportive of the project aims than teachers and pupils in higher ability classes.

 

Top of page

Analysing what took place

We have looked at the process of making your data manageable and reporting your key findings to readers and comparing data from different sources or different methods. We shall now look at offering an explicit analysis of what took place, we go about this in the following sections:

• Critical Reflection
• Explaining your findings
• Model making
• Comparing with what others have said
• Informing your professional practice

Top of page

Critical Reflection

The first challenge to reflecting critically on your findings is: you are so intimate with your data that you will have lost sight of the big picture. It is essential to take a step back so that you can see the wood from the trees. Notwithstanding the pressure you may feel to get finished it is important to make sure you give yourself a break now and again. When feeling utterly lost, not looking at your project for 2 or 3 days can be more insightful than relentlessly slogging away into the small hours.

 

A key aspect in critical thinking is to identify and question what you have taken for granted, this may be through received 'wisdom' in the literature, your professional intuition developed through your experience as a teacher, the staff room culture of your workplace or some other source. What seems like common-sense may turn out to be ungrounded pre-supposition. Starting your analysis is a good moment to pause and consider that which you may have taken for granted.

Playing devil's advocate with yourself is one way to approach reflection. Sharing your ideas and asking others to look for holes in your thinking can be an excellent exercise. Reading literature which is takes a different perspective to your own can be helpful too. This does not mean you have to agree with or incorporate the ideas of authors who have a different philosophical or methodological stance to your own - but exposing yourself to different and contrary viewpoints can help you see where the weak points in your own understanding might lie.

 

For you to do Try to focus on the basic assumptions you have brought to your action research as well as those you have acquired whilst carrying it out. See if you can make a list of 5 or 6 pre-suppositions that you might not have critically reflected on. You might consider assumptions...

...about the purpose of doing action research. ...about the nature of what makes good practice. ...from the standard working practices in your school. ...from commonly accepted pedagogical paradigms such as constructivism. ...from your personal political, religious and philosophical views.

 

Top of page

Explaining your findings

So far you have mostly been concerned with describing your findings - though as we have stressed there is no great leap between describing and explaining; they are points along a continuum. To move along this continuum you need to be clear about the phenomenon associated with your project and how to account for it.

 

For you to do Have a go at identifying the main findings of your research as you perceive them at the moment.

Write a single paragraph of about 200 words summarising your findings. Write another paragraph summarising your findings, this time about 100 words. Write a single sentence summarising your findings.

REFLECTIONS This exercise is surprisingly difficult for most of us to do but unless you can summarise your findings succinctly you will not be able to identify the central 'phenomenon' associated with your project and you will not be able to analyse what led to this phenomenon. Reported 'phenomena' in previous studies have included: "a peer review approach to assessment which engaged many learners in the assessment process" "an innovatory approach to group work which was not understood by most pupils" "an inservice training programme which was received well but led to little impact in the classroom" "a creative learning initiative which impacted on engagement of pupils" You may find it useful to repeat this exercise during the analysis phase every time you feel your thinking has changed significantly until you feel you have really settled on the core meaning of the project.

There is no one recipe for how to account for the phenomenon you have identified. However on many projects researchers have found it useful to look back at the main findings, with all the necessary provisos arising through triangulation and critical reflection, which appear as credibly associated with the phenomenon. Take for example a study into an inservice pro gramme designed to impact on teachers use of group work. The central phenomenon identified was that this an inservice training programme which was received well but led to little impact in the classroom. Some of the main findings associated with the study were that:

• Teachers believed groupwork could have a positive impact on pupil learning
• Teachers commented favourably on the planning and delivery of the inservice sessions
• Teachers said they did not have enough time to plan to use groupwork in their classes
• Some Teachers had difficulty in organising groupwork due to the layout of their classes
• Teachers saw the training as generic but lacked detail on applying groupwork in their context
• Teachers felt many pupils would find it challenging to take part in less structured groupwork activities
• Teacher felt there were too many constraints on using groupwork

At first sight the associated between these findings and phenomenon seem fairly clear: the inservice sessions led to little or no change in teacher behaviour because there were time constraints, physical constraints in the classroom and fears of pupils' response. However, even if this was the case which of these is more important and if you took away one would that be enough to change the outcome? And is one best seen as a cause of the other. For example there are always time constraints, if lack of time was the overriding problem then nothing would ever change. So is it more a question that lack of confidence in managing pupils led to a perception that it would not be worth investing time in this project, ie time constraints are better seen as an outcome of lack of confidence not a cause of lack of implementation? Or perhaps this is being cavalier about the constraints we are working under, the key factor may lie in the overriding pressure on us as teachers - there is simply too much asked of us to make any meaningful changes. Then again perhaps this is wrong, the key factor was simply that the sessions were too general, if they had been geared to the kinds of lessons the teachers wanted to carry out then perhaps the teacher would have been more confident. Relationships between factors are rarely clear cut and this search for analysis will take time as you review your data and continue the search for a credible story to link all or most of the findings together. The key here is you are searching for credibility not arriving at a single inconvertible truth.

 

For you to do Now you will need to ask: what factors led up to the central phenomena associated with your project?

REFLECTIONS As you carry out this activity try to present a hierarchy of factors and suggest how they might link together.

Top of page

Model Making

 

One possible way to get a grip on your data is to produce a concept map. This can take the form of a top-down approach in which you express your intuitions about how the data might be modelled in graph form. A whole host of examples of concept maps can be seen online here. Note that free concept mapping software is available for download here.

 

For you to do Think about your intuitions about your data and potential findings. Make a concept map to describe your findings. You may wish to do so using a large piece of paper and a pencil or using freely available software.

REFLECTIONS Remember that generating a model is an iterative process Start at the top-most level of your thinking and add increasing detail - indeed concept mapping lends itself well to this. Don't be afraid of screwing up your map and starting again, or trying to make an improved version once you have finished it. Note that although using software takes longer to get going, once you are used to it redesigning it and experimenting with new connections and relationships becomes more efficient. This allows for a greater number of iterations leading to a more refined final version.

 

Top of page

Comparing with what others have said

There are aspects of your innovation that are likely to be particular to your own circumstances. However, you might well be surprised by the amount of literature covering aspects of cases not dissimilar to yours. In your reconnaissance you will have looked at aspects of this literature and now is the time to go back again and use search strategies to find out more. For example use key word searches in the library database or in tools such as Google scholar but also use snowballing - using the references at the end of one article to locate new articles.

 

For you to do Try to find, say, 4 or 5 studies and take detailed notes on them (there will be a time limit to how much you can realistically and productively look at, of course). You can treat this process as another aspect of data collection: rather than conducting an interview, someone else is telling you about their study. As with all textual data you can code the account. For example, you might look for...

...when it was. ...where it was. ...which similar key factors were enablers of innovation. ...which similar key factors were inhibitors. ...main conclusions from the study.

REFLECTIONS You should now be in a position to draw out similarities and differences in your own study. Again you might like to present this in a tabular display and compare the literature against your own findings. Your study might bring out differences - highlight these differences as a contribution to the field and then try to account for these differences. Conversely you might feel a sense of disappointment if someone else has carried out a similar study with similar results - why did you bother? But don't be down-hearted - the aim of the innovation was to do something practical to develop your teaching and yourself. There are no fixed truths about education and your study is adding to what we know - it is as useful to confirm as to counter existing material.

 

Top of page

Informing your professional practice

In this final section we will cover the impact doing this action research project has had on you, your teaching, your learners and your institution. We will also ask: where to from here?

There are two essential ways in which the completion of an action research project is likely to have an impact. The first is the implications of the literal findings and conclusions of your research. For example, perhaps you set out to establish the pros and cons of a particular approach to coursework. Your findings are likely to inform how coursework is carried out in your practice in a direct manner. The second is the experience of the process of having carried out an action research project. Although this is a less "official" outcome of a research project it may well be the richer, more profound and ultimately most significant impact on your practice. For example, you may have discovered a renewed enthusiasm for certain aspects of your work, or perhaps re-ignited a reflective approach where complacency had begun to creep into your practice.

 

For you to do Consider the impact of (i) the findings and conclusions and (ii) the process and experience your action research project has had on...

...you as a person. ...you as a professional practitioner. ...your teaching in the classroom. ...your work outside the classroom (planning, marking, meetings and so on). ...your students. ...the department you work in. ...the wider institution you work in.

REFLECTIONS It may be that not all the above apply to you or your research project, and some are likely to be more weighted than others. For example, consider the different impacts likely for research into children's perceptions of group work compared to research into teachers' attitudes to the use of plenaries.

 

The findings and conclusions of your research are likely to have led you to ask certain questions. What would you do differently were you to repeat the project? How could you build upon the research were you to undertake another project? Similarly, the experience of carrying out a research project is likely to cause you to question your future approach to your professional practice. What aspects of researching will you incorporate into your day to day work? For example, you may decide to focus and reflect on a single aspect of your classroom practice every term in a semi-formalised manner.

 

Top of page

 

 

 

 

 

 

 

 

 

Page Content

Moving from describing to evaluation
Describing what took place
Reviewing the data
Representing the data
Selecting and amplifying the data
Collating the information

Evaluating what took place
Critical Reflection
Explaining your findings
Model making
Comparing with what others have said
Informing your professional practice