4Matrix Frequently Asked Questions

The following are Frequently Asked Questions about 4Matrix.
If your question is not answered here please contact us.

Q. What is 4Matrix?
A. 4Matrix is a set of interactive tools built into an application which contains school examination and other data extracted from a school MIS. 4Matrix extracts the most recent GCSE results, analyses them, generates a series of reports and creates an interactive data analysis and research toolkit. 4Matrix is a research tool for use by subject leaders and teachers to research the factors influencing pupil performance in their subject.
A version of 4Matrix, called 4MForecaster, will process year 10 examination data to forecast a complete comparative achievement picture for these pupils one year into the future.

Q. How does it work?
A. 4Matrix is an online service that works by remotely extracting examination results and a range of contextual data from a school's MIS before processing it into a set of interactive tools. This remote service is operated by a SIMS Premier Partner. Once created, the tools can be used on any computer and no longer needs to connect to the school MIS. All data remains in the school.

Q. We do lots of data analysis - what does your system offer that we can't do already? A. There are many good reasons for using 4Matrix.
     As well as it being very affordable, 4Matrix will:
1. Save an assistant head many hours of crunching numbers in Excel - 4Matrix takes minutes
2. Not make any mistakes - school grade lists are frequently wrong at the first attempt
3. Go much further than grade lists by providing automated commentaries about performance
4. Provide a real independent alternative to RAISEonline data
5. Provide comprehensive measures of Within School Variation (No other tools do this)
6. Allow teachers themselves to research the reasons for performance variations
7. ...that's a start - there are many other good reasons - see this web site for more information.

Q. At what time of the year is 4Matrix available to schools
A. 4Matrix can be used at any time of the year to process the most recent examination results, but the most valuable time to use 4Matrix is the day the examination results arrive in late August. This way, a complete analysis of subject performance is available in advance of the start of the new year and some time before Fisher Family Trust and RAISEonline data becomes available.
However, our Forecasting tools can be used at any time during the year to analyse target grades and forecast the future GCSE performance of year 10 or year 11 pupils.

Q. Our School Improvement Partner (SIP) visit is based on RAISEonline data . . .
A. RAISEonline is the primary source of evidence used to judge the outputs of schools, so it will likely be the main focus for the SIP visit. However, if RAISEonline data doesn't put the school in the best light, what additional evidence will the school use to show how well it is doing? 4Matrix provides a very different approach to looking at the outputs of schools and hence provides a different source of measures about subject and school performance. Read the comments made by headteachers under 'what users say' to see how schools are using 4Matrix. Secondary SIPs will also want to know how the school is proving that 'Every Child Matters' because future funding re:14-19 developments will depend on this. 4Matrix is just the tool to supply this essential information. NEWS  We have provided training for all of the secondary School Improvement Advises for a large County LA in the use of 4Matrix. We will report on the impact of this on the effectiveness of the SIP role in the months to come.

Q. Will our School Improvement Partner (SIP) know about 4Matrix?
A. We have worked closely with one large Local Authority to train every secondary SIP in the use of 4Matrix.  We will be able to judge whether this has improved the effectiveness of the SIP role later in the year.

Q. What is 'Within School Variation' and why is it important?
A. 4Matrix calculates a set of Within School Variation (WSV) measures for every subject, ethnic category and teaching group - or for selected groups of pupils. WSV is important because it reveals where pupils who perform well in some subjects underperform in others. If schools concentrate on reducing this effect it can raise standards by 10% - according to the DCSF. This is the most obvious target for school improvement because it is achievable - we know that the target groups of pupils are capable of doing better - because they demonstrate it across their other subjects.

Q. We publish grade lists and use target grades to focus on improvement.
    What more could we do?
A. 4Matrix goes much further in its analysis of variation than just headline grades. It is a subject leader's research tool for investigating hypotheses about under and overachievement across a large range of factors (including gender, academic range, ethnic group etc.) as well as selected groups (like pupils from a particular primary school, frequent absentees, differences between teaching groups, main lessons on a Friday, pupils with poor language skills, pupils in the revision club, those students who went on the Geography field trip etc.) It can help answer questions like "Do pupils with good maths skills do better in ICT?" or "Do pupils with high non-verbal reasoning CAT scores do better if they choose Diplomas rather than academic subjects"?

Q. Is Contextual Value Added (CVA) a good way to compare schools?
A. CVA adjustments discount one or more variables (ethnicity, gender, etc) so that comparisons can be made on a fairer basis (The technique is called multi-level regression analysis).
CVA is a useful technique to help filter out chosen factors to help us see what remains. 
However, it is a statistical technique that was designed to work with large sample sizes. It has no meaning at class level because a good teacher will not expect less from individual pupils because of their ethnicity or gender, etc.
The use of contextual value added (CVA) data is a helpful way to make comparisons between schools fairer, but the high correlation between CVA scores and inspection judgments of leadership quality (as reported in the TES) suggests that there may be too much reliance on this one source of evidence. With 4Matrix, unadjusted data will be the starting point for school-level research into the local contextual circumstances of learners.
CVA will usually be a major component of an Ofsted inspector's evidence base used to compare one school with another, and indirectly to judge the quality of leadership in schools. In the absence of any other measures about a school's performance an inspection team is likely to use the ones they have, i.e. RAISEonline scores. The point here is that schools really need to gather additional measures to place alongside RAISEonline. 4Matrix provides a powerful complementary set of measures which help paint a broader picture.

Q. Our Contextual Value Added (CVA) score for KS2-4 is 1028.
This means that we are good doesn't it?
A. CVA compensates for factors like deprivation. An inner city school will have its value-added totals 'adjusted' so as to try to remove the deprivation factor from a consideration of how well the school is doing. We have undertaken work for a number of local authorities that shows the relationship between CVA, 'standards' (i.e. %5A-C grades) and 'Value Added Variation'. The evidence of this work shows an interesting non-linear relationship between these three measures.
We have seen seen strong evidence to suggest that CVA often gives an 'inflated' view of the performance of categories of school.  The answer to the question posed here is that a judgment of school effectiveness based only on CVA, or only on 'standards', or both, will not necessarily paint the most accurate picture of how good a school is at the time of inspection. 

Q. Contextual Value Added (CVA) scores that the Ofsted team will use don't reflect our current intake.  What can we do?
A. CVA data may not reflect the mix of any current year group in a school. The best way to influence judgments arising from inspectors' use of CVA data will be to have secure independent, quantitative, up-to-date measures of the performance of pupils now in the school. By using school norm referencing, variation from expected grade, as well as comparisons with national standards, a selective school can be compared equally to a school containing a large number of disadvantaged pupils. Where CVA scores are not up-to-date a school can use Matrix to provide measures of in-school-variation and unadjusted Value Added to place alongside official CVA measures.

Q. We are a National Challenge school.   How can 4Matrix help us?
A. Within School Variation is a theme that will be used in the £400M Natonal Challenge programme to help schools with less than 30% A-Cs to improve. 4Matrix is a well-tested tool designed specifically to promote action-research approaches to school improvement.
We have worked with schools that have used 4Matrix to help with their journey out of special measures.

Q. We have 65% A*-C GCSE grades. We are a good school aren't we?
A. Maybe. It depends what you mean by 'good'.
There is not a concise definition of 'good'. Are all selective schools automatically 'good' and all schools working with challenging pupils 'not good'?  'Standards' are not necessarily the route to answering this question. (See our presentation on this subject)
CVA measures were a sensible way to try to compare schools working in different circumstances.
But we believe that additional measures of 'Within School Variation' get closest to measuring how good provision is for every group of learner. We know that the work of schools is often skewed by attempts to raise their points scores. This will show up as wider variation, i.e. a greater level of inequality of provision. We think it is important to look at this where schools ar claiming improvements in 'standards'. Although 'Every Child Matters' is the key theme of education, at the present time the system that judges schools doesn't appear to give this a particularly high priority.  Schools using 4Matrix can demonstrate that have high achievement, coupled with a high consistency of provision. This is a 'good' school in our judgement.

Q. But does 4Matrix provide any Value-Added measures?
A. Our online system for extracting and processing data held in an MIS will also create a 'Value Added Variation' version of 4Matrix. Together with the standard WSV version of 4Matrix, these two tools will provide the essential measures of both Value Added and WSV needed to show that a good school is, arguably, one with High Achievement and Low Variation.

Q. We already get RAISEonline and Fischer Family Trust data.  Isn't this enough?
A. 4Matrix provides a school-level, independent approach to analysing a schools' outputs. If Contextual Value Added data from RAISEonline says that the school should be doing better, then that is what the Ofsted report will say - unless the school can prove there is a different story to tell. (See 'What user's say' for a good example of this.) 4Matrix provides the independent performance analysis needed, especially when a school is due for inspection.

Q. Why should we need to do any more data analysis than we already do?
A. Schools that don't know themselves well put themselves at risk, and lead others to question their leadership. Ofsted inspection findings would never be a surprise to a school that knows itself well. Poor inspection findings should not occur in school that knows itself and has the means to improve. 4Matrix has been designed for school leaders and teachers to use for themselves as a process of research and development to find out where particular groups of pupils have the capacity for doing better. In the best practice seen in trials the 4Matrix application was used by subject leaders and teachers to explore and report on variation factors - with subject leaders then reporting to school leaders their plans to reduce variation (a process of non-judgmental internal review and development).

Q. Could 'Within School Variation' metrics also be an indicator of leadership quality?
A. If one believes that the most important item of evidence of good leadership is a school's outputs, then our techniques will be useful because they provide evidence of the consistency of the school's outputs. After all, even a sausage factory is judged on how good the sausages are and how many are perfect - not how good the factory's systems are, or how charismatic the director is, or how well trained the staff are, etc.
The link is feasible because, boiled down to the essentials, a school that has good leadership at all levels will be one that has high - and consistently high - achievement.  And to 'customers' - parents and pupils - that will really be all that matters.

Q. How accurate are the commentaries generated by 4Matrix?
A. The commentaries are generated directly from the examination data supplied, and by reference to published national performance statistics. We always use the most recent national comparative data available at the time. Some of this data may relate to previous year national averages - but in practice there is no great statistically significant difference in making a comparison with the previous year, except in the case of a school which has had a dramatic change in circumstances. This year we have been given access to NCER comparative data - the best available source.
It is worth remembering that 4Matrix does not know anything about the context in which the subject was taught. The subject leader's contextual comments (which can be entered into 4Matrix) will be important in providing a balanced interpretation.

Q. How can our school get 4Matrix?
A. 4Matrix is available within extended trial arrangements, as part of our consultancy package, and via LA-wide arrangements. 4Matrix 2007 is also now available by direct sale. Contact us for details.

Q. Which Management Information System (MIS) does 4Matrix work with?
A. We specialise in providing services to schools which make use of SIMS.
We make use of a remote extraction service provided by a SIMS Premier Partner which reliably extracts the school's examination data, prior to 4Matrix processing this data into a 4Matrix school application.

Q. Will 4Matrix work with other MIS'?
A. At the present time we do not have a similar remote service to reliably extract a school's examination data from systems other than SIMS. Although we can make use of data extracted by schools themselves from other MIS' this data often needs cleaning up, e.g. it may not use official QCA subject names and might use non-standard grading notation. We can only guarantee accuracy with our current extraction method using SIMS.

Q. What do I do if I have a technical problem with running the 4Matrix application
A. Please see our help page for details of all problems experienced to date and their solutions.