Episode Transcript
00:00 Peter McColl: This is Peter McColl, Senior Associate of the Consultation Institute and an alumnus of the Young Academy of Scotland. It's time for a new podcast on the future of tertiary education. And in the episode today we focus on the Research Excellence Framework and the other key performance indicators as well as alternative ways to evaluate success in tertiary education. To explore this topic, I'm joined by Dr. Iain Donald and by Ian Ritchie. Iain Donald is a lecturer in design and UX at Edinburgh Napier University. He gained his PhD in the field of history. He has an MSc in information systems and worked in the games industry prior to joining academia in 2010. His recent work examines the intersection of games, digital media and history, with a focus on commemoration and memorialization, using game design and technology to exploit collective and communal memory in communities and virtual worlds. Ian Ritchie is the non-executive chairman of technology investment company Tern PLC, Computer Application Services Limited and Krotos Limited. He serves on the boards of the Royal Lyceum Theatre and the Scottish Council for Development and Industry. He founded OWL in 1984, which pioneered hypertext technology, a forerunner to the World Wide Web, and which was sold to Panasonic in 1989. Since he has been involved in over 50 startup high tech businesses, and Research Excellence Framework panels in 2014 and 2021 for Computer Science Research. We're going to start, I suppose, with a little bit of looking back at how the Research Excellence Framework has operated. So since it was introduced in 2014, the Research Excellence Framework, which we'll be referring to through the rest of the podcasts as the REF, but don't think of Pierluigi Collina or whichever football referee it was you had in your youth. So the REF has been used to assess research success in academia. We've got somebody from academia, we've got somebody from industry. So I just wanted to give you the opportunity to talk a little bit about how you've seen the Research Excellence Framework change during your career. I'm going to start with Ian Ritchie, if that's okay?
02:11 Ian Ritchie: Yeah, sure. Well, I've never been an academic, I've been an honorary, visiting professor, an honorary professor or something. I did do a stint of helping at Heriot-Watt at one time, but I've never been a proper academic. But I've always been involved in startup technology companies, and many of those have arisen from technology that's been created in universities. So I have a connection really with, particularly in my area which is computer science. And I was on the RAE panel in 2001, and in 2008. And then it was on the REF panel, REF panel in 2014 and 2021. I'm hopefully I'll be allowed off now, for good behaviour. And I won't be invited back again, because I'm now in my 70s [unclear]. But it was interesting to see these developments. There's not an awful lot that changed, to be honest, it's a very expensive, laborious process, and it involves the universities putting a hell of a lot of effort into preparing material, that then you have, you know, 25 of the top researchers in the field have to meet several times and read as many of these papers as they can and come up with their scoring and so forth. And it's… and of course, there's conflict of interest. So it's very complicated. There's data protection issues, because there's individuals involved. And it's a very, very complicated system. And you wonder whether it is appropriate that it should be quite so complicated.
03:43 Peter McColl: And I suppose, before I come to the other Iain, there's a tendency for these processes to become more complex as time goes on, not simpler. And I wonder if that's if that's what you've observed?
03:56 Ian Ritchie: Well, actually, but it was an accident, the pandemic made it simpler. So the pandemic meant we didn't all have to go to be tripping off to London or Manchester. We've had a couple of meetings in Manchester. But basically, most of the stuff was done on Zoom, which actually was fine. And that made it simpler I have to say, but otherwise, it's still it's still pretty complex…
04:18 Peter McColl: And has got more complex or has changed its complexity, or…
04:23 Ian Ritchie: Oh, I don't think so. I think it's just, I mean, there've been debating for a long time. And most almost all the research work is done on, based on the published papers. And these are published in reputable places and so forth. And the journals have got a sort of scoring system of the quality of a journal and so a lot of argument about whether you could do it automatically. You could just say, you know, a paper in Communications of the ACM is a number one paper, a paper in proceeding of such and such is a number three paper, whatever, then do a quick calculation. If you've managed to get a paper in one of the prestigious journals, Nature or something, then obviously that's sitting at the top. I wonder if they did that calculation whether it would be much different frankly from the process of people having to go through them all and score them individually.
05:09 Peter McColl: Iain Donald, you are a practitioner academic, you are as it were at the coalface, do want to talk a little bit about what it's like for an academic to go through this process, because I think a lot of people outside of academia will see these things turned into league tables, which I think most of us who understand the process are sceptical about, or we'll see a headline once every five or six years about university performance. But that's not the reality in academia is it? It's much more engaged than that for you.
05:39 Dr Iain Donald: Some of the challenges that are in the REF, you know, as soon as you have anything that is being measured, right? The universities then look towards them, you know, to improving, it should be... the REF should really be about nudging behaviours, and encouraging universities and institutions to support their staff to do the best research possible. And I agree that there's challenges within that. I think one of the challenges about, you know, using automated metrics would be, you know, how that could be gamed, if you like, because, what are you going to use? Are you going to use impact factors? Are you going to use citations? To a certain extent, you lose the value of the peer review. That said, as a researcher, I think the focus on the REF can somewhat distract from trying to do the best thing for the research. You know, research is naturally about learning and discovering, and often that involves failing. One of the challenges within that is the pressures that are put to deliver, you know, three-star, four-star research are difficult to actually support, how do you know if something is going to be original, significant, you have challenges in any research project that may impact its rigor. And those are the criteria that the REF measures, but in my own experience of starting at a vocational university, where it wasn't research focused, so we had a lot of learning in the process for the REF, coupled with the challenges of securing funding, conducting the research, publishing it, with workloads that were increasingly focused on teaching responsibilities and not research. I've only been an academic since 2010, you know, I think there's a shift towards industry-focused outputs and research, but depending on the discipline and field, universities aren't always going to be best placed to undertake that research. If a research grant based academic for a day or two a week very simply, they can't move at the speed at which industry can, there's a disconnect somewhere between industry and academia. I think both see the other as better funded for research, you know, whereas the reality is, it's somewhere in the middle.
07:52 Peter McColl: There are a couple of interesting things I want to pick out there. So I think the first thing is Iain Ritchie referred to the Research Assessment Exercise, which was the previous mechanism, the thing that came before REF. Certainly one of the criticisms of that was that it took a scientific model of understanding where you would publish things in papers, and that was the mode of progress and applied it to disciplines in which that wasn't the case. And I remember sadly, much-missed Larry Hurtado telling me at one point that he had spent the previous four years writing a major book on I think it was the early church fathers, that would be discipline changing. And the Research Assessment Exercise just wouldn't capture that because it was set up to capture scientific papers or papers published in the scientific model. I think one of one of the reasons why we moved to the Research Excellence Framework, rather than the RAE was that it was meant to capture some of those things. I don't know how successful that's been, I'd be interested in observations, I suppose from listeners. But I think also there's a kind of broader point. And I'm going to, I'm going to try not to butcher Goodhart’s law. But some of you may be familiar with Goodhart’s law, which is that when a measure becomes a target, it ceases to be a good measure. So once you set out a target and begin to measure it, what people do is change their activity, focus on the things that are measured, rather than focusing on the things that might actually make the biggest difference, you know, the classic example is a target not to have patients on trolleys in hospitals, so they employed porters to lift the trolleys, take the wheels off them, and thus turn the same thing, sitting in the same place into not a trolley, but a bed. So you've achieved your target, but you haven't actually achieved the substantive change that you might want to, you might want to achieve. So I wonder about that question of teaching to the target, teaching to the test, you know, that sort of approach where measures become in themselves a problem. Ian Ritchie does that, does that reflect…?
09:53 Ian Ritchie: It does. In 2014 I was on supervisory panel, though I was assessing the submissions from math, applied math, statistics and computer science, and we found that the computer scientists had a very high opinion of themselves. They were… about two thirds of them thought their research was of international standard. But as the math, applied math and statistics, it was about 1/3. And the reason for that, I concluded, was the computer science is very topical, there are fashions. At the moment is AI, big data and AI and so forth. If you're working in this field, you'll see your research, might think to be internationally significant. 10 years later, you look back and you'll find that a lot of it was, wasn't really, but obviously at the time, they treat it that way. So actually measuring is a tough one. And you're absolutely right, if your target’s set obviously, I mean, every university is gunning for these targets. And there's a great, you know, transfer window, where the top academic is, you know, hired by another university to pull them in and get their score higher, and so forth. So there's an awful lot of game playing in that, in that field. Yeah.
11:07 Peter McColl: I mean, certainly, when I was on the governing body of Edinburgh University, we had a real problem with the gender pay gap. And one of the conclusions was that this was being driven by that transfer market, the male academics were much more likely to seek to be bought out by another institution, and to take their research team with them. Whereas women are much less likely to do that, regardless of the quality of the research, that was for social factors. And so in some ways, the REF was seen to be driving a gender pay gap, which was one of the things another thing that the universities were being asked to address. So they're being, they're being pushed and pulled in both directions. Iain Donald, any observations on that?
11:51 Dr Iain Donald: I mean, one of the things I think's really interesting there, as you mentioned, like ignoring the gender issue, but the idea that a research academic can be hired, and they can take their team with them, you know, for the vast majority of universities, that just doesn't exist, right? You know, I don't have a team. I've never had a team, I work with colleagues, but we are, we might be working on a project together, usually, that's not even funded. You know, it's things that we are building together to try and get funding, you know, and I think one of the things that's really interesting at the moment is, the REF is still very traditional. It is very much based on fields and disciplines, whereas the vast majority of it… really research breaks disciplines, breaks the norms, and where you have social scientists working with historians, where you have them working with computer scientists, you know. We don't, you know, I've been teaching and researching games since 2010. Where does game, where do games fit within the REF, right? We submitted to, my previous institution, we submitted to the unit of assessment 32, which was art and design, could easily be in computing science, could also be in unit of assessment 33, you have this, this discrepancy of where that research fits, right. And you're relying on, and one of the things I think is really interesting is you're, you're often relying on institutions to support you in making those decisions. And the institutions are often driven by people that don't necessarily have the experience of the panels, don't necessarily have the experience of research, some of them. You know, some of them are making decisions on the assumption that researchers X, you know, whether it's science, you know, and when we're already seeing the impact of government shifts in, you know, we're still very much in a process where STEM is being pushed to the detriment of our arts and humanities, yet, I can understand that, I can accept that. But the prospect of not having arts and humanities is going to have an impact upon STEM. We need to be better. You know, as a general thing I always talk about, it's more important for us, especially in Scotland to be more collaborative than competitive. We're a small country, I don't think the institutions, we have 19 higher education or institutions that are, and that's phenomenal. For a small country to have 19 is phenomenal. But the REF, and the way that we fund our universities, puts them in direct competition, and that creates challenges. So, you know, for example, you know, we have, we have students going into a vocational university, because they have a desire to learn to go into a specific career. Right? That is great. If you are 17-18 and you know what career you want to have. That's fantastic. If you graduate when you're 21-22 and that's still the same decision. Wow. You know, I think the way that we, Scotland had a lot of benefits with the old system where you could go in and study a whole bunch of, you know, a smorgasbord of disciplines and fields and subjects. And yes, at some point, you had to make a decision to move on to honours and then from that, you could build on what you wanted to do and specialize. And I think that was a strength in the Scottish system. I think vocational degrees, we lose some of that. Going off tangent, but we now have metrics about, we still have metrics where we're measuring where students are six months after they graduate, not where they are two years, five years, 10 years. It's one of those things, like your degree does not necessarily define you. And it doesn't define research. And it doesn't, you know, and I think one of the things we need to get better at is looking at how we bring all of this together in a more collaborative environment.
15:33 Peter McColl: And I think I mean, interesting, I did a bit of work on future jobs, when I was at Nesta, and the, the Scottish degree as was, looked like something that would prepare you very much for those jobs of the future. And what the REF has done is to push that degree in the opposite direction, and into specialization, and into focus, which is probably not where we need people to be, because if they focus on something at university, I mean, I think, you know, programming language is a good example, you learn a programming language at university is very unlikely you'll use that for the rest of your working career. What you needed the skills to learn the relevant languages, and that's where the focus has to be. Ian Ritchie, you've come from that sort of industry background. Does that, does that reflect your understanding of what's happening here? And if so, how did we end up where we ended up?
16:26 Ian Ritchie: Yeah, no, you're absolutely right. And, obviously, the concentration, I was also on the Funding Council, and the concentration on funding blue sky research, you know, cutting edge research, all that sort of thing, is good for the universities, but it's not particularly good for the economy, or the rest of us really. And I, the last couple of times, I've been involved in the impact, because I'm no longer an active researcher, I've been involved in the impact panels. So you try and assess the impact. The day I was at a meeting in Manchester, I got an email saying that one of the companies I'd invested in, which started life at university, had just raised $275 million dollars’ worth of investment. And that company wasn't an impact, because it wasn't directly related to some blue-sky research. But it's now actually a $22 billion company. And so is a massive impact, but it didn't score. And I've noticed over the years, I've seen this in practice that academic research, the finest bits of academic research are actually about 20 years ahead of market. So you can't just take them out and turn them into companies. And my God, they do, they do try, the fun things coming out of research labs. And we try and take them, turn them into companies, but they're usually concept-led rather than the market-led. And I think the whole impact model is broken, it should not be related to academic, scientific research of three and four quality, it should be related to the way it has an economic impact on the world. I mean, for example, when I was on that panel, there was a post 92 university, I think it was Teesside or something. And they'd built themselves an innovation hub. And they were building spinout companies like crazy, and they were really doing a great job. Didn't get a penny because obviously it wasn't related to particular bits of academic research. And I think there should be a completely separate funding system for impact. And universities like Napier and so forth would benefit from that. Because, you know, they tend to be closer to industry. In fact, if we wanted an economic benefit from our universities, you know, we currently claim we've got about three universities in the top 20. But if we wanted to have an economic benefit of universities, we should be trying to encourage them to work with industry. American universities tend to, academics in America tend to go into industry and back again, you know, that never happens here. For an academic goes into industries pretty he's much written off, and then getting back in... No, it's true. I mean, it's, I know, people who've gone back into university after running a company and they don't have the same… because they don't have the same research record, you know. And you're talking about women, there's obviously a lot of care taken not to damage a woman's career because of maternity leave and things, but it does, her research record isn't there and therefore she doesn't rise and it's just inevitable. So there's an awful lot wrong with the current system which tends to focus on you know, the blue sky scientific breakthroughs and so forth. Is that the right thing? Is that the right thing for our economy in general or social society? No, I'm quite... Iain was saying about the games industry as a case in point. The games industry is huge economic benefit for Scotland, precisely educational games. Minecraft for example, is, you know, a Scottish… developed into consoles and a massive educational tool around the world, but wouldn't be recognized under these current systems.
20:00 Peter McColl: I mean, Iain, your previous role at Abertay was, at the university that that focused itself very much on video games as a selling point. I mean, I suppose for me there's a sort of... Iain Donald, I'm falling foul of the two Ian rule... For me, there's an interesting question here, you really, I think clearly set out, the benefits and the costs to the Research Excellence Framework. For me, we need to do a little bit more thinking ahead of the 2028 REF round, about how it is that we can build in some of those points that Ian Ritchie was making about industry, but certainly, from my perspective, we need to look at things that aren't just industry, but also our societal based approaches. So, you know, we've decided that we're going to adopt very, very challenging targets for Net Zero, how is it that we're going to achieve those? Are universities going to play a role in that? I would imagine so, but how are they being directed to play that role? And I wonder if that points to a different model of designing impact or measuring impact. I'm really interested in mission-based approaches. And, you know, we've seen a bit of uptake of that in government, but there hasn't been a great deal of it in higher education. And I mean, I did a bit of work with one higher education institution on setting challenges, and it was already good, until they decided that the challenges had to be restricted to one academic discipline, at which point I thought, have you understood any of this at all, because the idea that you could, you know, throw open a challenge, and see who could address it, but only within the confines of one academic unit seemed to me to miss the point, almost entirely. I mean, you know, maybe a better way of working even within that economic unit. But it seemed to really miss the point of can you deal with antibiotic resistance by taking a biotech approach? Or can you take, can you deal with it by taking an approach from behavioural economics, you know. Can you encourage people to wash their hands more, or it doesn't have to be a biotech-led approach. And, you know, both of those things, I think, are valid, both of them have a role to play. But if you narrow it down to one discipline, then you're never going to get those two approaches working together and the dynamic interaction between which I think, is one of the really exciting prospects. I don't know if that reflects your thinking on this, Iain Donald, but I'd be really interested to hear whether that's how it works within an institution.
22:29 Dr Iain Donald: I have to, I mean, I have to be careful here about, it's very easy to veer into the other issues wrong with academia, you know, in terms of like, funding, we have research councils that specialize in areas, and yet within those areas, you have crossover research, right, so who's allowed to fund what. You know, when something is funded, are the best people getting funded, or the best people that were appropriate, that applied to that research council. You know, when we look at collaboration, collaboration tends to happen more granularly, you know, it's more kind of, from the ground up, academics working on things together, because they are interested in that kind of field. When we talk about the challenges of industry, you know, coming back to that point where I said, the expectation between academia and industry is, each seems to think the other is better funded, you know, and one of the things that I find incredibly interesting is... I finished my PhD in 1999, I didn't become an academic until 2010. So I spent a good time in industry, but this expectation that industry will give their time for free, you know, in terms of like, we aren't funded, we will fund an academic for one or two days a week, right, through the Research Council. But the research that's being done from the industry side might not be funded at all, and they're expected to give in kind support and all this and that, even if it's more than one or two days a week, even if it's like five days a week, I do think there is a big disconnect with ... and maybe this is my experience of having worked in the games industry, where, you know, the speed at which things are developed, right? You know, without naming games, companies, but if games companies want to investigate something specific, that they think will be a good feature for their new game, if they're big enough, large enough, they will dedicate 20-30 resources to that for six months, 12 months, right? And that will be a calculated decision, that's going to be for the benefit of the game and the product, and hopefully, it will sell considerably, and companies do. At the same time, if you went to that same, the same idea in academia and asked for 20 people for six months, you know, you'd be laughed at, right? It's just not going to happen. And even if you did get some of those resources, they're still going to be expected to teach and that's going to slowly progress rather than what industry can do. So, you know, I think there has to be a better way. And I think what Ian Ritchie was saying, so this idea of what we do about having that transferability, between industry and academia, you know, becomes critical. One of the things I loved about working in the games industry was I saw people develop, you know, in disciplines. And, and then when you read it on the forums, people saying that the game developers don't do research. I know of artists and designers who could easily have got a PhD in a very specific area, because they got to know something so well, you know, they're probably experts in that area, because they spent four years studying, developing, researching, implementing. And then they stop and they move on to next game. I am sure other aspects of computer science are like that. I know less about science but I know that there's, I agree with Ian's point, there are lots of amazing academics who are lost to industry because, to be honest, industry pays better, you know. And at the same side, I think one of the things that we struggle with is bringing people from industry into academia. Because it is a weird type of environment. I've often described it as, it's almost like you're being freelanced, or you're employed, but you're also a freelancer, independent, you're expected to do your own research. Sometimes that's with people, sometimes it's not. So it's a very fluid system. And I think, you know, I agree with Ian Ritchie, that there's lots of things that could be improved. I'm just not sure, you know, what the REF can improve, compared to what has to be considered a wider issue. And, you know, one of the challenges at the moment is you have universities, very much acting as if they are businesses and being run as if they are businesses without necessarily having people that have any experience of running businesses in charge of that.
26:56 Peter McColl: I mean, I think, I think there are a couple of points that I'd like to tease out there. So the first one is that your point about research councils was, I think, part of the thinking behind the creation of UK Research and Innovation. So bring the research councils, bring Innovate UK, all under the one roof so that so that you get some integration. Now, of course, it's going to take some time. And I think this is the first Research Excellence Framework for which UKRI has been up and running with a good lead in time, I think it was created just about the time of the definition of the last REF. So I think there's an interesting point there and it may be something we would want to take up about how research council funding feeds into the REF. And how do you how do you close that circle? Is there a circle to be closed there? That I think is an interesting point, because the funding of things to be assessed is an obvious starting point to generate change. And I think there's a second point there about universities as institutions. And I think one of the things that I'd observed as somebody who's worked with universities, with private companies, is that the university institutional form is very different to what you find elsewhere. And I mean, Iain Donald will be very familiar with things like sprints, you know, to create change. That's just simply something you wouldn't see in any university environment. And I do wonder if there might be some learning there. Because while there's a lot of talk about universities as businesses, they don't look like businesses that I see elsewhere in the economy, they look like businesses that you might have seen at some period in the past. And for reasons of politeness, I won't, I won't say which decade I think that was. I mean, Ian Ritchie does that align with your experience?
28:47 Ian Ritchie: Yes, you could change the REF, you could absolutely change the REF. The REF is doing the job of making sure that top quality scientific and other research is being is being done. But it's not doing the job of... if you compare us to the States or to Germany, you know, the higher education system is much, much more integrated with the wider economy, professors in the States or in Germany, for that matter, would have three or four startup companies in their career, you know, this is the way it's done. And there's an awful lot of transfer. I mean, obviously, Germany doesn't have an Oxford or Cambridge or an Edinburgh or whatever. Does that hurt their economy? Doesn't seem to, you know, they seem to manage quite well. And in fact, their sort of middle-sized companies tend to be very, very connected with local higher education and so forth. Those are coming and going back and forward. That isn't in the UK. And I think the REF in particular causes that division, because it emphasizes so much. And then it's got this pretend thing about having impact. And actually, it's not the real impact. So I believe strongly that there should be a separate funding body external to the REF, which assesses real impact, real academic connection with the wider industry. And it's not just me that says, I mean, you know, the former head of engineering at Cambridge University said, you know, said this, that it was really important. If an academic in Britain, does, you know, a couple of years at Princeton, or Harvard or Stanford or something, that's regarded as fantastic. If they got a couple of years working at Rolls Royce, that's regarded as, you know, that damages their career. What kind of nonsense is this, that kind of interaction should be normal for academics.
30:43 Dr Iain Donald: I think one of the things that is really interested about the changes that have been made to the REF, the move away from individuals, though, towards a focus on institutions. I think that's, you know, that's positive. You know, I keep coming back to this issue of like, if we don't have the REF, what do we do? And in some of my moments, I'm like: 'That's great. We'll just distribute all the money between all the universities, and that will be fine, right?'. There has to be some method to how we do this. And how do you compete with, you know, how do you have the smaller vocational universities that probably do have those better connections with industry, that may not be doing the cutting-edge blue-sky research, but are doing stuff that actually impacts the value of that company? And how do you compete with universities that have been established for 500 years, but don't necessarily engage with industry to the same level? The REF is trying to create the right behaviours for institutions. I think many of the problems with the REF and how it's perceived and how it's interpreted, are to do with institutions and people within institutions rather than the REF. I mean, you know, one of the one of the strengths of the REF is that everything is out there, you know, it's a transparent process, you know, everything is published in advance, you know what you're being measured on, you know what the criteria is? Whether we agree with all of those is a different issue. But it's all out there. You know, I come, like I said, I keep coming back to this, what would we replace it with? You know, it's somewhere in the region of 2 billion pounds worth of funding. Above that, maybe more, two and a half? I don't know. How do you distribute that? You know, and how do we look for impact? You know, I agree, it's kind of when you, you look at impact… Minecraft is a great example. Minecraft, you know, the console versions are made in Scotland, but the game was made in Sweden, and bought by Microsoft. And, you know, it's this, Minecraft was bought for $2 billion, $1.4 went to the creator, I think, or $1.2. We look at, you know, again, if you look at some of the biggest companies, tech companies in Scotland, they're no longer owned in Scotland, they're owned by American companies, or Japanese companies or Chinese companies. You have this, you know, and I, again, I'd really be interested in Ian Ritchie's take on this, but I mean, how do we foster an environment where we're building on Scottish success?
33:29 Ian Ritchie: Yeah, unfortunately, we have the wrong kind of economy for this, we don't really have risk capital in Scotland in the way that they tend to have in other countries. I mean, one of the companies I'm involved in, a share and investment company, we created a company that does satellite technology and, and we've floated it on NASDAQ North in Sweden, because that's a market actually does respect high tech. And a lot of companies obviously go to the States for the same reason. The UK, the markets are really, really dozy, and they don't, they tend to go for mining and their traditional biking companies and things like that. They don't tend to be interested in a Talan technology. But there is a movement around at the moment that the government is seeking to pension companies: 'Look, you guys are here long term, buy, why don't you invest in long term tech'. The top five, six companies in the world today are all tech companies, the Amazon, Apple and Google and so forth. That's where the money is. But the UK has completely bypassed all of that. And, of course, the food chain for technology companies is you build yourself a reputation, you build yourself an interested market, and one of the big guys buys you because they can access the market. It's just, I mean, that's not a Scottish thing, that's a universal, that happens everywhere. But there should be at least a few real companies that do manage to stay as contributors to the Scottish economy.
35:04 Peter McColl: I mean, it's interesting that Sweden has had such enormous success in this area. There's a lot of reasons why we can't replicate Silicon Valley. There are, I think, many fewer reasons why we couldn't replicate the Swedish model where you've got Klarna and I think Spotify, is Spotify Swedish?
35:25 Ian Ritchie: I think 15 unicorns come out of Sweden, or something like 10 unicorns come from Estonia, for God's sake. And it's only 1.2 million people. There's only two unicorns ever come out of Scotland or two tech unicorns to come out of Scotland, and there's no sign of any new ones coming through. So we're failing. I know, what is, what is it about Scotland? You know, we're bright enough. I just don't, I just don't understand it. You know, you want a scary statistic, the average Irish person is over 50% richer than the average Scottish person now. For goodness sake. You know, I mean, this is ridiculous. We're supposed to have all the great universities.
36:05 Peter McColl: As an Irish person, I take a little bit of affect of the idea that Scots should always be richer than they are. Sorry, I'm joking. I mean, I think the key is that Ireland doesn't have the natural resources that Scotland has had, and that it hasn't had the history of higher education that Scotland has had would point in the opposite direction. I think there are some really interesting questions to be to be asked there. Iain Donald asked a little bit about what we'd replace REF with. And I think it's really interesting, it's quite difficult to have this conversation. I don't think I'm telling tales out of class here to say it's been difficult to get people to contribute to this conversation. And I think that in itself is really interesting. I think we need to open up a space where we could think about alternatives. I have some quite strong ideas about this. And mission based and I do often wonder, I mean, I think Ian Ritchie was talking about the amount of resource that goes towards REF. It shouldn't be hard to repurpose some of that resource to other ways of assessing our impact the way in which the money that goes into universities changes the economy and society and the environment. And I think for me, there's, the REF, very much operates in a world before mass communication. You know, it's not this, this is something that doesn't lend itself to being communicated to non-experts. And that is in itself problematic. And I think if we could have REF outputs that were digestible for a range of audiences, that that might be a really interesting thing that we could do at very little additional, additional cost. And I think also there's, there's a big question about how it is we set the parameters for this. So how, how is it that we're deciding what our good impact should look like? Can we do that in conjunction with other institutions, other elements of society? How can we, how can we look at people's lives and say, what is it that needs to change over the next 10,20,30 years that will be informed by research and feed that into the process? Because I don't see that at the moment. What I see at the moment is, things happened and we said whether they were good or not at the end of that process. So there's very little strategy going into this up front. And I think those are some of the areas where you could, without throwing everything up in the air and hoping that it all lands in a sensible order, change the process. So I don't know what you think about some of those some of those ideas and the ideas that came out of the Tertiary Education Futures project that that have helped to inform some of the RSE thinking on this? Ian Ritchie?
38:50 Ian Ritchie: I think you could improve the system, you could certainly automate much more of it. Now we're living in a world of big data, you can basically extract information in a way that we could never do before. So the complexity and the machine, the REF machine, could be stripped back and made much more efficient. So I think that would be one thing to do. I think, on the other hand, I think you need more investment in the impact side of things. I really do. And I think you should be trying to encourage and reward the universities and actually, I know you were talking about the post-92, and they’re obviously closer to industry, but actually the big, you know, the big research intensive universities, I see still do create, you know, companies that that can go on to be very effective as well. So I think I think there are things that can be done with this. I think UK is a funny place. We're very traditional, very conservative, policy conservative, we don't like change, but universities are probably the most resilient organizations. I mean, you know what, St Andrews is 600 years old or something. If you manage for 600 years, that's pretty good. And my job is to know how to duck and dive, and work out… and so they need to change. And they need to become more effective. And I think they've got themselves a niche, well, crikey, where universities these days are basically spending all the time educating foreigners for money.
40:21 Peter McColl: That's a major export industry for Scotland.
40:27 Ian Ritchie: Is that the right approach to the Scottish economy?
40:27: Peter McColl: Iain Donald, any observations on that? Maybe what you think the future of tertiary education will bring?
40:32 Dr Iain Donald: Yeah, I think the pandemic has given us a wee bit of an insight into… the pandemic created an environment where we all had to change and adapt. And I look at how universities tried to do that. There were definitely challenges within that where, you know, accessibility, inclusiveness, allowing the opportunity for people to, you know, once the restrictions were relaxed to be on campus, but also having to support people who didn't want to come. You know, one of the challenges within all of that is your idea of a university that does that incredibly well. The Open University have been doing remote learning, work from home for over 50 years, they do it phenomenally well, which is why they're successful. The idea that smaller universities could pivot to doing, or even larger institutions, could pivot to doing that and doing it as well, was a bit ridiculous, and it wasn't a strength. And we're now in a situation where we look at what, again, we're moving probably away from the REF, but we're at a stage where what is happening within higher education has different challenges, right? We are moving towards, you know, are we moving towards a model where we will see less students going to university, not just because of demographics, but because that traditional, in Scotland, four-year degree but elsewhere three-year degree? I mean, we already probably see less Scots going to English universities. Because why would you, because it costs a lot more money, right? But are we going to see that change because of graduate apprenticeships? And within all of that, if we want to bring it back to research, I think one of the most interesting areas will be that we tend to think of institutions as teaching-led, vocational-led, research-led. And the idea of research-led being you focus on the research that you're doing and relate it to your teaching, right? But maybe there's a call for students to be more involved in the big research questions at an earlier stage of their degree. They are the ones that are going to be bringing it forward and five years, 10 years, 20 years, involving them in the research process where we celebrate failure, rather than ... I'm not saying we need to move to a system where there aren't grades, but I mean, you know, when everything is driven by the grade and the degree that you get, there is an abject fear of failing. And then there's an abject fear of that having an impact on your, your life. And, you know, I watched the... how social media and the media in general, you know, we have that criticism of the currently the government saying that they want to get rid of pointless degrees, or 'Mickey Mouse' degrees. What are those? Because 20 years ago, that would have been games development. You know, one of the ones that is routinely named is Golf Management, you know. Scotland has a massive industry based around golf, we need people that can manage everything around that, you know, I think there's huge challenges ahead. But I think one of the things that, you know, is exciting is that we have, the pandemic has given us an opportunity to reflect, we need to think about what it is best for. When you look at how other countries do things a year from home or two years from home where it's more online based learning, bringing people together for, you know, semesters where they work together in teams to do more industry-led challenges and projects and getting them involved in the big research questions that academics haven't been able to answer because working together as teams collaboratively. I think it's interesting, I look at my kids and I think about whether they're going to go, you know, I already don't expect them to go through a four-year degree. I'm looking at them and thinking, well, what are they probably going to do? Would it be a graduate apprenticeship? Will it be something else? You know, there's a lot to be said for the two plus two model, where two years of college, two years at university, but when you start looking at the costs involved now, ignoring fees, you know. How are you going to support your child to be paying, you know, 1000 pounds a month in rent? And that's, that seems to be a relatively normal figure. I mean, some of the figures for halls, I was like: 'Oh, that seems reasonable’. And then I realized it was per week, not per month. I think those are the challenges.
45:10 Peter McColl: I think that points to some of the broader economic pressures. So you know, we've had Ian Ritchie talking a little bit about the investment culture, Iain Donald talking a little bit about factors like housing and social change. I do wonder about some of those questions around involving people more in the process. And I think students are in some ways, the best academics say that they learn from their students. And I think there, there may be a question around how we can build that into the system more rigorously so that it's not just conversations through which that learning happens. But it's through research design as well because certainly, when you look at things like online harms, it is very difficult for somebody of my age to understand what the online world is like for somebody much younger than me. So looking at those sorts of things, I think there's, there's an important element of co-production in some of this. I think, there may also be questions around whether we do address things like high rent as a question of research and as an economic drag, and where that might happen, an economic drag, also a huge social problem. So looking at those things, I think, I think offer some really interesting things. It's probably not to do with the REF, though. So if you have any sort of final observations around the REF, and what you'd like to see from the next REF, and if you have one, maybe two key proposals, I think they'll be really nice for us to go away with. Ian Ritchie?
46:48 Ian Ritchie: Well, I've probably said it all already. But I think the REF could be streamlined, I think could be a much simpler system. I think a lot of the measurement could be automated. So I think the panels could be provided with basically a structure of what the answers are, and then... And maybe it's a matter of, you know, looking at individual institutions holistically, and saying, you know, is this working, we've got the score from the research papers, and so forth. But obviously, actually working, or we should be doing things differently. So I think that's one thing, I think the other thing is that there should be a completely separate system for impact. So this should be a measure of the academic, or the economic benefit of academic activity, whatever academic, so, you know, right through from Oxford, right down to the post-92, it should all be measured in that way. And I think they would win. In many cases, the vocational universities as Iain says, they would win a lot of that battle, because they are much more industrially connected, and they would be encouraged to be more industrially connected, because our funding system would push them towards that. So I think that's the two things I would say.
47:57 Peter McColl: Iain, any final thoughts?
47:59 Dr Iain Donald: The REF isn't perfect, I think is a little unfairly maligned, you know, in terms of ... the aim of it is to foster a research environment for institutions. How that is interpreted, and how that is developed is very much institutional based. And it's very much based on people within those institutions. And that's where the challenges arise, you know, it's a bit like, while I agree that some of the REF matrix can probably be automated, I would still be concerned that that automation could be, and would be by larger institutions with better resources, could be not fabricated, but could be shifted to give a more favourable metric. And that's the challenge. That's why I think peer review... One of the things about the REF that I think is really good, is that we start from a system where almost all that research that goes forward, has already been through peer review, right? And then it's going through further peer review, through multiple experts. We talk about the different panels, but if there is cross disciplinary research, it's often referred to other panels. Is the system perfect? No. Could it be improved? Yes. Like I said, I mean, it depends on what universities want to do. You know, I think we have this challenge where, you know, I agree with Ian Ritchie there in that impact is incredibly difficult to look at. And referring back to that comment about when we're looking at impact and students and research like, what is the impact of something there six months to two years, five years, 10 years, 20 years from now, you know? I think a lot of the research on... I mean, we're going back a very long time, but, you know, I did my master's research in artificial intelligence, and I mean, where we are now, there's a lot of things, everything is AI, AI, AI. And yet, what is the research that is underneath that. To take away from the technology, if you look at the ethics and the intellectual property issues that we have for Chat GPT, or mid journey, or all of these sorts of things, yet they are going to be tools, because they... you can't put the genie back in the bowl, those are going to be tools that the students that we have now are going to be incorporating into their working life. How that's done? Those are interesting challenges. But we've very much focused on the research of 'can AI generate a picture of x?'. And it turns out, yes, it can. How is it generating it? We don't really know. And I think the REF is really good for dri[ving], encouraging the correct behaviours, but it relies on institutions to interpret them equally. And that's maybe the challenge.
50:51 Peter McColl: Okay. I think that's those are really interesting insights. It's been a really interesting conversation. This is part of an ongoing conversation around tertiary education futures that the Royal Society of Edinburgh has been engaged in. There's a microsite that you can go to that's linked in the show notes. So please do go and have a look at the microsite, have a look at the report that we produced. And please do keep in touch with us. And if you have any suggestions for what the REF should look like, then we'd be really interested to hear those because, there's a review ongoing and we'll be feeding into that. Thank you to Ian Ritchie. Thank you to Iain Donald. Thank you to Cristina Clopot, who helped to pull this together. And I hope you've enjoyed listening.
51:34 Ian Ritchie: Thank you.
51:34 Dr Iain Donald: Thank you very much.
