Smart KPIs for Biotechs

Welcome to Cut the Chat: Life Science Insider. Episode 1: Smart KPIs for Biotechs.
Cut the Chat is an informative podcast series designed to quite literally cut the unnecessary talk, and get straight to the point and heart of many life science and biotech conversations that need to happen for our industry to evolve, innovate, and get smarter, faster. We'll facilitate the conversations you've been dying to hear, and tackle the real crux of our industry's challenges, while addressing the subjects of what we can do as life science leaders to be more successful in a shorter period of time — conversations that might actually help each other when we dare to open up.
Today's episode is hosted by Seuss+ co-founder and CEO Sabine Hutchison. She'll be joined by a very special guest. So let's get to it — let's cut the chat.

Well, Alan, welcome. You and I go way back to the days of MDS — that dates us, doesn't it? Remember that?

It certainly does. It certainly does, yeah.

Yeah, I still have lots of wonderful relationships from that company, so I'm happy that we've continued to see each other and stay in touch since then. Alan, you've been in the industry for a very long time — 25 years, so lots of experience you bring with you to this conversation today. And what I find interesting is that you actually started on the financial side of the business — your first positions were with AstraZeneca and GSK, but more on the financial side, and then you moved into general management roles, worked for clinical research organizations, large CROs — you were VP of Europe for Covance, which is now LabCorp, president of ICON Clinical Research, and the list goes on. What I find really interesting as well is that you've not only worked with the larger organizations, but you've also worked for a lot of midsize companies and really helped scale their businesses — RPS, MDS, as we mentioned earlier, Quanticate, and lastly working with Excelia, a really nice midsize CRO that you helped grow and expand. But what's super interesting is that you've not only worked for the CROs, but you were also the founder and CEO of a life science company, Concept Life Sciences, a discovery, chemistry, and toxicology business. So lots of experience you bring, and I'm really curious to hear how you're going to bring all these insights into our conversation.

Yeah, I would say lots of gray hair, but there's no hair left — it would have been gray, but stubble, stubble now.

So Alan, what's interesting is you're actually moving into a new phase of your career — you're working with a lot of organizations now, consulting, working in the private equity space, continuing to support Excelia as a non-executive advisor, also working as a non-executive chair at Tantronics Limited, a genetic toxicology, early-stage CRO backed by private equity, and you're stepping into the tech space as an advisor for Nuva.di, a US-based AI tech company.

Yeah, a portfolio of different things — a really exciting next stage of my career for me.

And Alan, on a personal note — I've always held you in very high regard, we've known each other through lots of different jobs, experiences, clients, crazy travel, flights to companies. One of the things I really value about you is your strategic insight — you always have the ability to keep the big-picture focus and guide people through processes and the organizations you've worked with, and I've been fortunate enough to work with you firsthand on some of that, something I really, really value about you. And what I'm excited about with this conversation is that I know you're always genuinely honest and very authentic, and that continues to inspire me in all the conversations we have. So I wanted to say thank you, first of all, that you've agreed and graciously accepted our invitation to be a guest on this upcoming launch of our podcast — thanks for being part of this journey.

Yeah, thank you very much for inviting me. As I say, we've worked together over a long, long, long time — you were very young, you were very young at the start! And, you know, Seuss+ has done some fabulous work recently to support me in different organizations over the years, so I'm delighted to have the opportunity to have this conversation.

And actually, it's really interesting to see how biotechs run their projects a little differently than mid-sized and larger pharma. When you get up close and interact with biotechs, do you see that their mindset is a bit different?

Yeah, absolutely — and that's why we're here today, of course, to look at that. Our industry is heavily regulated, and we tend, at times, to fall into ruts in the way we review and look at metrics.

And that's what we want to talk about today, specifically for biotechs. We want to talk about the importance of measuring, but how we can be smarter about measuring what truly matters — specifically in clinical trials, so that's going to be our focus today. And the significance of finding smart KPIs for biotechs — in some of our past conversations, biotechs can sometimes minimize or overlook KPIs and metrics, especially smaller companies, whereas larger ones can potentially fall into the habit of what large pharma does — overly implementing KPIs that maybe aren't so relevant to what they're doing. I'd like to get some of your insights around that.

Absolutely. I'd say some of this is really recent — I was CEO of Excelia, a roughly 900-person CRO, up to the end of April, and as you say I've now moved into a non-executive role. But over the last three years at Excelia, which has a lot of biotech clients for its end-to-end services, I've had a lot of direct experience sitting with biotech management teams and looking at how to support them in their studies — plus a broader perspective over a number of years from mid-size and larger CROs as well.
I'd just echo what you said — there's almost two very distinct, polarised approaches from biotech companies. I can think of a recent interaction with a really large biotech that's got, I guess you'd say, all of the characteristics of a large pharma company, in terms of size of revenue base, number of employees, and global spread. That really does feel like a very large pharma company, where you collect and report on a book of metrics that might run over six or seven pages, really detailed — what I'd characterize as a lot of overlapping metrics, when you're thinking about predictive metrics versus outcome metrics — lots of outcome metrics, "this is what it was." And honestly, a lot of CROs collect these metrics for large pharma companies and these large biotechs because they're required to do so, and you sort of wonder what that company actually does with that set of metrics, because there are so many data points that it must be really difficult to understand what the key elements actually are. Typically, when you go through a governance discussion with the steering committee, or the strategic committee, or whatever's been formed to review the relationship, you end up focusing on one or two key metrics — so you've got a book, and most of it's really thrown away. It's a reporting mechanism, and to some extent I see that as part of the organization's responsibility to show oversight of the CRO they're using, for GCP purposes — and that certainly ticks that box. But how effective is that, in terms of actually being able to run a better study?
The flip side of that, which you also alluded to, is really small, not-quite-virtual businesses — but definitely businesses that are resource-constrained — where, when they're looking at setting up metrics, they basically have no agenda for metrics at all. So it's the opposite end of the spectrum — sometimes they're not even looking for any of those metrics, they're focused on what the study budget is, and that's often a very hard number for them, similarly for academic institutions — they're just very focused on the money element, they've got a fixed amount to spend, and maybe first-patient-in, if that's linked to their funding requirements — but that's really it. In those circumstances, you really do need to sit down with the customer and start to plan with them: what are the critical points in their study, probably for funding purposes? And that's a big element of the challenge a biotech client has that a large pharma company doesn't have in the same way — they need to be able to meet certain critical points for their investors, and actually teasing that out is a really important part of the initial phase of building a relationship with a small biotech client.

Yeah, absolutely. And it's interesting, you mentioned first-patient-in — I think that one is always at the top of the list, isn't it, often when you think about KPIs for a project. And I think it's interesting as well, because a lot of biotechs aren't necessarily looking at and thinking about the commercial impacts, which come later on — thinking about endpoints, and some of these pieces that could also be very beneficial when starting your trial, and not necessarily always just jumping into first-patient-in.

No, no, absolutely. There's a vast array of possible metrics between signing the contract and getting to that first-patient-in, around those timepoints. Honestly, the big challenge, whether it's biotech companies or mid-size or large pharma companies, is around amendments to protocols. Protocol amendments are really crippling — a crippling challenge for the drug development industry, in terms of cost and delay. So yes, you can get first-patient-in pretty quickly, but if the trade-off is that you haven't done sufficient work to really look at things like potential changes to inclusion and exclusion criteria, then you get your first patient in, you suddenly work out patient accrual isn't working to the extent you assumed it would, and maybe one of the reasons is the exclusion criteria. Then the client decides, maybe with CRO advice, that they need to change something like the inclusion/exclusion criteria — and you've got to go through all the challenges of ethics approval to move that forward. And that's a delay.
So the rush to first-patient-in is often an investor-driven focus, as opposed to an operational-driven one — there is an element of operational focus around it, but if you could wave a magic wand and dramatically improve the outcome, both in terms of cost and time, I'd say investing up front in a really robust process to minimize protocol amendments has got to pay off. Because you see, time after time, the challenges of protocol amendments — it's almost like Groundhog Day, the industry isn't learning that this is a big problem. It's a big problem for time, a big problem for money, and a big problem for patient adherence in trials, as trials get extended and the risk of patient dropout increases.
We've talked in other circles about the role of the investigator meeting, for example, in terms of training investigators to be able to deliver the study protocol the way it's described. Often, for smaller biotech companies, there's maybe one key opinion leader who's really been the pivotal person contributing to that protocol. What I'd suggest, and do suggest, to customers is that they really take a bit of time and challenge that — look for critical challenge around the protocol. In a nice way, obviously — you've got to keep your principal investigator on board, they're a really important person — but really pressure-test all of the, maybe, logistical elements of running the trial, to be able to drive patient adherence to the protocol.
Honestly, you'd be better off spending an additional few weeks at the very start making sure you've really robustly tested the protocol — looked, in a scientific way, at similar protocols run by other organizations, and tried to ascertain what the challenges might have been for them, and invested up front around that. You see quite a lot of pushback when you talk to customers about that, because they're focused on first-patient-in — country selection, site selection. But that rush to first-patient-in potentially has consequences in other elements of the trial, which have a fundamental impact on the success of that trial.

Yeah, absolutely. We see it, and I'm sure you've seen it too, during a vendor selection process, or when a request comes in — often we just have a synopsis, and that synopsis isn't even complete at that point. We're basing a request to a CRO on a synopsis, and there's so much still potentially missing, and you're absolutely right, then we have to revisit again once the protocol gets a little further, but then the budget gets impacted again too. So if you look at the timing, I think that's just such a valid point — look at the protocol, spend the time, and you're right about the key opinion leaders, but also patient advocacy — there's so much available for organizations to tap into. And the idea of pressure-testing is so valuable, because in the long run, patient accrual could actually be even faster if the protocol is structured properly.

Yeah, and the point about patient advocacy groups is really important, because more recent changes to ICH GCP require you to more specifically consider the voice of the patient in trial design — a more explicit requirement at this point in time. That's healthy from a number of perspectives, but specifically when you think about the challenges of patient adherence to a protocol, it's really important that you get the voice of the patient reflected in that trial design. Honestly, there's a vast industry of specialist service providers that's grown up in the last seven or eight years to keep patients engaged in trials — and there's a reason for that, because there's so much patient dropout in trials, and it's so expensive, such a waste. So again, that comes down to a really thoughtful approach to study design, which isn't simply about the mechanics of the action of the drug, but the whole impact on the patient and the whole ecosystem — how you're going to deliver the study, the logistics of delivering it — you really have to consider the complete package needed to deliver the study efficiently. And again, that's an element that both smaller biotechs and maybe some smaller CROs don't necessarily fully embrace as they're thinking about vendor selection, the pitch process, trying to present their credentials to a customer.

Yeah, absolutely. And it's an interesting point you mentioned earlier — when you look at biotechs and their drivers, they obviously have drivers from investors, and one of those key pieces is first-patient-in. So how can biotechs, how can the leadership team, actually push back to their investors and change the mindset around this?

Yeah, I think it is difficult — I certainly don't diminish the challenges, because first-patient-in has become such a focus. I think it's around, though, when you look at the metrics around the number of protocol amendments in a trial — the data speaks here. You can actually pull up data showing the average number of protocol amendments in certain therapeutic areas — that data exists, and you can present it. So actually bringing home the risk of protocol amendments is a really important part of that challenge.
But there are other pieces here — when you look at, for example, the data around FDA submissions, at this point in time you're really talking about maybe 50% of submissions getting approved with no questions coming back from regulators. If you do get regulator questions, the average delay from an FDA question is close to a year. So that's a serious delay in the data. There are some very specific points that speak to the importance of, let's call it, quality metrics around those delay pieces. I'd point people to Applied Clinical Trials, a really interesting article published by Avoca and Pfizer on quality metrics — if you're listening to the podcast, I'd suggest going and looking at that, it's focused on quality metrics around clinical trials.
As you think about the risk of delay from regulators because of the quality of the data in your submission, that's another reason you've got to be really focused on not rushing to get one metric right — the first-patient-in metric — because ultimately the only metric that's really going to be valuable to you is the approval metric. You don't want questions coming back from a regulator — you need to effectively be approved first time, or you're potentially waiting another year for your submission.
So as you try to discuss this, and I'd say the primary stakeholder group here is financial investors — to some extent it might be your own management team, but I think they're probably easier to bring into the fold, to help them understand the importance of a holistic approach to your set of metrics. But for hardened bankers in the financial markets, you need to go in with data to make sure they understand it. And I understand that's probably a bit of an uphill push for CEOs and chief medical officers to make that pitch. But when you think about some of the, let's call them shortcuts, you might be tempted to take to get that first-patient-in — you might select a particular country, for example, to get a really fast approval, fast ethics — all your CEOs will quote you which countries you can get through fast. Ultimately, when you're opening up a country to run a clinical study, if you only get one patient out of that country, that's a massively inefficient approach. So you've really got to think about the sustainability of country and site selection. Just because you might be able to get, I don't know, Australia up faster than France — if ultimately you're in a particular disease area and there aren't so many patients in Australia, that's a very costly first-patient-in, and you'd be better off waiting and picking a country with a sufficient patient population to deliver your trial. Because honestly, for any company, whether a biotech or a large pharma company, the days of having lots of backup countries in your trial is a really expensive way of running your trials. The smart players are looking at the minimum country selection necessary to deliver the right number of patients within the timelines. So again, that's another reason not to be too blinded by the first-patient-in focus — think about the whole patient-accrual process.

Which actually extends the time again, right — looking at bringing in one patient in one country, to your point, the trial is going to be delayed, and there are cost implications too, having to open up new countries. It's a very valid point.

Yeah, I should give a shout-out to somebody we both worked with a long time ago at Icon — Malcolm Burgess, who's now retired from the clinical marketplace. I remember having multiple discussions with him about first-patient-in, ironically — he was head of data management at one point for ICON, and his point was really around database points locked, that you should be focused on how you get those data points locked in your database — that's a really focused metric for CROs to look at, because you go through that whole data-cleaning process, and you really need to focus on what data points you've got locked into your database.
On the country-selection piece, I'd like to make another point, because I think when biotechs in particular are talking to CRO partners, you tend to see an alignment of size — biotechs tend to want to work with smaller or mid-sized CROs, and there are lots of really positive reasons why: cultural fit, management attention, and experience — the smaller and mid-sized CROs have got a lot of experience working with smaller biotechs that maybe haven't got a complete complement of staff, so they have to fill in the gaps. But many CROs in that category have got incomplete service lines, potentially around country distribution, potentially some services, and the whole raft of services to support the broader requirements for bigger endpoints that exist today that didn't exist before — we're talking central laboratory, logistics, the whole supply chain to get investigational product into markets, as well as specialist device support you might need for particular procedures in a trial.
I think the challenge for some of the mid-sized CRO sector is that a lot of that is outsourced to multiple other vendors, so the CRO themselves doesn't necessarily have the in-house experience to understand the challenges of that broader supply chain. I recently picked up a rescue client whose study went on hold because they ran out of investigational product — the previous CRO hadn't taken account of that, and all of a sudden everything's on hold because that element of the process hadn't been looked at. So even if your CRO is outsourcing elements of the study — and they're obviously going to have to, for specialist things like investigational product — they still need to take a project-management perspective, looking at a full end-to-end solution for the client. Because unlike a big pharma company, many biotech clients don't have that in-house capability. So there's another element of the difference for a biotech: when your CRO presents a set of metrics to you, the tendency in many CROs is to present metrics that are in their own direct control, and not necessarily look at the bigger picture of what the client actually needs.

Yeah, going back to the point you mentioned around data — which is just so important, having clean data — one of the things we know is that when biotechs are being assessed, or potentially looked at for acquisition in the future, that's what's going to be looked at and have an impact in the due diligence process — not necessarily first-patient-in, but the quality of the data at the end.

Yeah, there are always trade-offs here. Here's an interesting perspective — large pharma companies today, the top three, are going through a very explicit exercise of trying to limit the number of endpoints in their trials, because they recognize — you've got committee on committee on committee, and this is large pharma, we'll come back to biotech in a second — really trying to cover every conceivable use of that data, with a view to getting as broad a label as possible. But what we see very clearly, not only from my Excelia experience but previous stats-focused experience at Quanticate and elsewhere, is that there's a real challenge with that, because it comes back to the whole challenge of clinical development. If your protocol's too complicated, the chances are it may not get through ethics approval, because you're trying to collect data points that are irrelevant or unnecessary for the study design — so you have a risk at ethics approval. But even if you get through that, and it's data you don't really need to get a defendable experiment and hypothesis approved to get your drug approved, you're risking extending the trial, or getting other data points that could cause issues, other ramifications and delays.
Big pharma is learning that lesson — there are a couple of companies I know of that have actually set an absolute cap on the number of data points for their trials, and they're pushing those down; I probably shouldn't name them. But when you look at biotechs, a lot of them fall into that same trap — yes, they want their drug approved, and yes, they want potentially to be able to trade that drug on after phase two, so they're thinking, "let's collect this piece of data, let's collect that piece of data." The reality is, the more data you collect, yes, you increase the possibility of exciting potentially new investors, or a potential acquirer of the drug — but you also run the risk of complicating your study, delaying it, and increasing the cost. So there's a balance between getting a really tight dataset necessary to secure approval — which, on the face of it, is where my starting point would be — versus what other data you can legitimately collect within the confines of the ethical approval, that doesn't add a lot of ramifications in terms of complexity around endpoints and study design.
If you throw in, say, an additional piece of spirometry onto your study, or a 24-hour urine collection — there's all sorts of things people will throw in that run the risk of patient non-adherence and dropout, which is a really big challenge. So if you throw in unnecessary procedures — particularly things like 24-hour urine collection — you're putting yourself at risk in terms of the complexity of the study. One of the core metrics I know for a top-10 pharma company is endpoints, and there's a physical limit on endpoints for studies. I'd say that's hard to set for smaller companies, but you really need to think about a really tight design, and then move away from that carefully. Clearly, biotechs don't necessarily have all the committees-on-committees that create the same risk for large pharma companies, but they do have that commercial awareness of "we might want to do other things with this data." My counsel would be: stick to what you really need to get your study approved, because that's the thing that's going to be of the most value to you — and then a few additional data points on top, providing they don't have negative ramifications for patient adherence to the study, maybe you can consider those.

Thanks, Alan. Well, as I knew, we would not struggle to fill 30, 40 minutes' worth of conversation! So thanks so much. And as we wrap up this discussion around smart KPIs for biotechs — if you were to give three tips to the leadership team of a biotech, as they're about to set off and set their KPIs for their next clinical trial, what would those three tips be?

I think the first thing to say is that the thing that's going to damage you really, really significantly is protocol amendments — you've really got to focus on the design of your trial to minimize them, even if that takes longer, because that's going to be a strong backbone to the delivery of your study. So really think about that, and pressure-test — even if you've got the best investigator in the world, pressure-test the protocol, not just from a medical perspective, but from a whole logistical and patient-adherence perspective.
The link to that is around the voice of the patient in the protocol, because you're now required under ICH GCP to take a more explicit input from the patient — it's really important to think about patient adherence, and patient advocacy groups in particular are really where you need to focus at this point in time. The best CROs have been spending the last three years or so really building relationships with those patient advocacy groups, and you'll be able to tell that from your vendor selection, because they'll know those areas.
We've talked a lot about first-patient-in, and the myopic approach to it among a lot of biotechs — and I do appreciate it's a challenge for financial investors to be so focused on first-patient-in. But again, don't get trapped into treating first-patient-in as your primary metric without thinking about the other elements, because it's the second, the third, and the hundredth patient you've got to be thinking about — until you've got the requisite number of patients in your study, you're not going to have a significant enough population to make your submission.
And the last point — I've probably gone over your three — is really think about quality metrics. I mentioned that Applied Clinical Trials article between Pfizer and Avoca, because I think it's a really interesting example, because if you submit data that's of poor quality and get referrals from the regulatory agencies — yes, you may be able to get those cleared up, probably about 60 or 70% of referrals are cleared up, but they're cleared up after a period of time that's really, really expensive. You've got to bake the quality metrics in, and don't lose focus on those — it's really, really important.

Yeah, absolutely — obviously we're a regulated industry, so very key. Well, Alan, thanks so much, as always a pleasure — really enjoyed the conversation, and I wish you a good weekend, since this is a Friday.

No, thank you — I really, really enjoyed talking to you today. I think there are some really important things there for the industry to think about, and there's a challenge to really think about doing a bit more work upfront, being really thoughtful around it. And, as you say, big pharma has benefits and challenges that are different from biotechs, but some of the same things are still there — that issue around endpoints is the same challenge for both, it's come about from maybe a different source, but it's the same point.

Absolutely — tighter endpoints, faster trials, that's the word of the day. Thanks, Alan.

Okay, thank you very much. Take care. Bye-bye.

We hope you enjoyed this week's episode of Cut the Chat: Life Science Insider — Smart KPIs for Biotechs. Make sure you're following Seuss+ on LinkedIn, and be the first to know about upcoming podcasts. See you again soon.