EVALUATING ACADEMIC QUALITY

A lecture by Professor Margit Osterloh Frey, Warwick Business School

How do academics regard methods of ranking universities? Is peer review a really reliable method of measuring academic success and how far can the public rely on league tables? In her inaugural lecture at Warwick Business School, Professor Margit Osterloh Frey questioned the way forward in evaluating academia.

Israeli scientist Daniel Shechtman’s work was heavily criticised by his colleagues and rejected by peer review journals. Subsequently thrown out of his research group, his academic career seemed to be over. In 2011 he won the Nobel Prize for Chemistry.

What Shechtman’s case illustrates, argues Professor Osterloh Frey, is that peers and the peer review process used in academia to evaluate and assess the quality and originality of research can make huge mistakes. “There is a lot of empirical literature about the quality of the peer review system. What we know today is that the reliability is very, very, low... it is important to notice that the reliability is better with articles that are rejected and much lower with articles accepted.” Peer review is what she calls the Republic of Science, where scientists are responsible for their own quality control. Is external evaluation, which she deems New Public Management, any more reliable?

Not so. “The Times Higher Education World University Rankings 2011-12 were released in a way that resembles the publishing of the hit parade,” says Professor Osterloh Frey. “Harvard was no longer number one but had to share numbers two and three with Stanford. Oxford moved from six to four.” The public, she believes, can’t rely on these rankings. She goes on to explain why. “It is important to know that the editor of the Times HE rankings last year admitted big mistakes during the last six years producing this ranking. He wrote to an internal audience that ‘The rankings of the world’s top universities that my magazine has been publishing for the past six years which have attracted enormous global attention has serious weaknesses’. He described the method as ‘embarrassing’ and promised to improve the methods.” What this shows is that attempts to give the public an insight into what is good research and the quality of universities is problematic.

So who should we trust when it comes to evaluating academic quality? Based on her research with Professor Bruno Frey, Professor Osterloh Frey offers improvements to mitigate the problems and tensions between the Republic of Science and New Public Management.

An informed peer review system would include many rankings. “You can use rankings but only in an informed way – you know about the criteria and the weight given to them and can take these into account.” The disadvantage of this system is that the accountability to the public is low.

Another alternative is input control. Instead of emphasising too much on output control we should emphasise more on input control, says Professor Osterloh Frey. This is an idea she got from management control theory. Input control would involve the “rigorous selection and socialisation of scholars instead of permanent output monitoring”. Examples of institutions where this method works well include Harvard and Queensland universities.

The advantages of input control are that it allows great autonomy and unorthodox research including informed peer review. However, a key disadvantage is that the method’s success is only measurable after a long time. The public is not really able to comprehend the whole picture.

Rewards, explains Professor Osterloh Frey, are great motivators for academics. The system could rely more on them. They are hard to manipulate and give an easy-to-understand picture to the public. On the other hand, she goes on to say, this only mitigates and doesn't eliminate the problems of qualitative peer review. Again, success is only measurable after a long time.

Finally, she makes her most provocative suggestion: allotment. This is “partially random selection after a control for minimum standards”. Allotment specifically downplays referees’ bias and avoids the cronyism present in the current peer review system, the kind of which rejected Daniel Shechtman’s work. From the public's point of view, the success of an academic’s work is easy to measure by the number of citations. On the negative side, allotment does not discriminate between good and bad papers, but then again, research has shown that selection by peers is sometimes close to randomness. Indeed, the degree of randomness in allotment can be varied in a clear and transparent manner.

To conclude, Professor Osterloh Frey states that New Public Management and the Republic of Science can complement each other when the role of rankings and peer reviews are played down by input control, awards and allotment.

Margit Osterloh Frey joined WBS in 2010 as Professor of Management Science, after holding a chair in Human Resources at the University in Lüneburg, Germany, and then a chair in Business Administration and Management of Technology and Innovation at the University of Zürich, Switzerland. She is based in the Marketing & Strategic Management Group at WBS.

She has published nine books as author, co-author, or editor, and published over 160 scholarly articles. She was awarded an honorary doctorate from Leuphana University Lüneburg and is research director of CREMA (Centre for Research in Economics, Management, and the Arts), Zürich. She is a former board member of three large companies in Switzerland and Germany, and a current member of Deutscher Wissenschaftsrat (German Council for Science and Humanities).

Her research interests cover organisation theory, knowledge management, philosophy of science, corporate governance and research governance.

By Penelope Jenkins