Beyond the Checklist: The Hidden Risks in Clinical Systems Compliance

Thank you. Wonderful to be here. I'm back.

Kieran @ Seuss+ (00:51)
Yeah, back by popular demand. Yes, we are very excited about that. And what's been so lovely is that it's not as if we haven't been speaking and interacting — a lot has continued to happen in this landscape for our clients, alongside you having a beautiful baby boy. And it's exciting to hear, because I think it's actually really interesting when you're not in the details day to day, but a step back — what do we notice when we have a slightly different perspective? So I think it's exciting to speak about this today, at this moment, with you. And one of the things I really—

Sorry to interrupt — I was just going to say, it's actually really lovely because what it's shown me is the pace of change. Just being out for a few months to have my baby, and then coming back, the pace of change is suddenly brought right to the fore, because I can see, in that very short period of time, there's been a lot of change — in guidelines, in current practices around data integrity, in what data we're using, what innovations we're using. It's amazing. It's really helped put into perspective for me how important it is to continuously evolve and check what's going on in this space, because it moves so fast.

Kieran @ Seuss+ (02:28)
Yeah. And I think it's also fun, because the topics and challenges we're actually executing on now, we started talking about four years ago — that's when you and I signaled a risk together. We started saying, this is something so important, it's coming, we must all be compliant. It's a space where many decision-makers in smaller biotech are less comfortable, and therefore make a lot more assumptions. And it took a while, didn't it, for the people making the decisions to actually understand what that meant. Initially we had a lot of conversations where people said, but this is irrelevant, it's not so important, it's under control, there's no problem here.

So it's wonderful to think that we started talking about this three and a half years ago, then two and a half years ago you joined Seuss+, and now, looking at where we are today — that evolution hasn't felt quick day to day, but when you look back you can see how much has changed. So that's really exciting. The first question I'd like to ask is about the hidden risks — the things others don't see when they're in it, or that they make assumptions about. What are actually the most common blind spots in digital trials, especially as more and more systems and tools enter the clinical landscape?

To your point, you're completely right — until biotechs and investors actually look at this space, it's a big blind spot in itself. Data integrity, and all the evolutions in regulations and guidelines we've seen over the last few years, is a giant blind spot in itself. And it's not until the activities start taking place that I think we're going to turn around and go, "I can't believe we never did that." It's unbelievable that we missed this giant elephant in the room and just continued without addressing the main point of the clinical trial — the thing that lives with us for the 25 years to come: the data. Sometimes we don't even know which system it's in, or who's checking it, or who's responsible for it. By putting these vital steps into place, we can start to build a picture of where the data is and how much we can trust it.

We're in a period of fake news — have we got fake data? How real is this? It's also about documentation. Do we have the right documentation in place, so that once we've submitted our data, regulators — and anyone who comes to the table to look at it, including investors — can ask, how much do we trust this? Do we believe in where it's come from? Let's look at the audit trail — who entered that, who made a change here, anything that might affect the metadata, for example.

Practically, we've had a lot of clients who've looked into this and improved the robustness of their data. But in some cases we've seen clients submit a data flow diagram without having done all the other pieces — so they've told part of the story, but they don't have the other documentation in place that actually establishes trust in the data. It might be very good data, but if you can't show you've done the right oversight — systems inventory, system risk assessment, monitoring your audit trails, whatever it might be — we can't build the full picture that the data has the integrity and quality we're now asked to deliver.

In some cases it even comes back to validation. The data's fine, the audit trail is fine, they have everything they need — but if you go back to step one, they're not collecting it with a device that's validated for the patient population. So after all that effort to gather the data, it hasn't actually been captured in a way we can use. That's a huge hurdle.

Kieran @ Seuss+ (07:40)
Yeah. And I think it's interesting, because in every industry, guidelines are generally behind — not running ahead of general practice, in a lot of ways. But looking at the guidelines over the past two or three years, there's been an evolution where, in certain ways, they're actually now ahead of standard industry practice. It's interesting to see this sense of "we need to catch up" — but isn't it under control? I think the beauty of it is that the regulations are driving things like knowing your systems inventory — knowing what systems are being used, not just as one signal, but so the whole core team knows what's in place. Even that first step has been interesting: how much is that information shared within the teams actually in control of how things happen? You might have a project lead who doesn't quite know which systems are in place or how they're set up, and oversight doesn't know either, because we assume it's in place. So the systems inventory — knowing what's there and what it does — is a beautiful first step. It's not about control, or choice, or change, it's about knowing. And then, what are the actual risks tied to understanding how that data flows?

The regulations push us toward more control and a better understanding, and I think that's the beauty of this evolution — it's maturing the industry. I don't think it's about curbing all the options out there — the hundreds of specific systems, tech providers, and device providers that each solve one problem. It's about how that integrates into the whole. What's most important? And at that level of detail, I see many clients where somehow the primary or secondary endpoint data isn't prioritized enough. That's what's exciting about your risk analysis — it makes sure that's at the top of the list, and gives a better understanding of the knock-on impact when a decision shifts which endpoint or which data takes priority, because that changes what kind of oversight or review is needed for validation, and so on.

So I think it's a very good thing for our industry, because it leads into my next question too — it's part of a much wider conversation. Whenever I bring up the words "data integrity," there's this visceral reaction, because there's no client or investor we've worked with for whom that isn't just a given — it must be, similar to "patients must be safe." That's ingrained very deep in all of us, and it's a hugely important part of what we do. But when you start digging into what that actually means, and how you show it, and what the risks are — that's where it gets interesting. "Of course it's fine, of course our data has integrity" — but when you look at how you prove that, that's where things are sometimes less in control than we'd expect, and less than the guidelines require.

But it links further than that. Data integrity isn't just about control for your study and increasing your chance of approval — it also rolls into investor confidence, and potentially, as you've mentioned to me before, into a partnership, or an exit to a big pharma. So how does strong systems oversight link to inspection readiness, investor confidence, and ultimately the milestone success that matters to big pharma? That's a big question — start anywhere.

Essentially, yes — you need the right documentation in place to demonstrate you've put the right oversight into practice. You've mentioned some key documents there that get scrutinized because of the changing guidelines — these aren't optional, they're required, which is already step one, because it's about people knowing they're now required. We find that's often missed.

A huge part of this is the whole data lifecycle — it's not a check-and-done exercise, it's about what oversight you've maintained throughout the trial to manage the full data lifecycle across its duration. What's your frequency for checking whether the data has integrity, resolving issues as you find them? Because quite often, when you do look, you do find issues. We had a client where we found that a percentage of their primary endpoint data had been lost — a huge oversight gap.

And because we looked, we were actually able to recover most of it. If you don't look, and you wait until submission or database lock, you might miss that opportunity, because some of that data was actually captured and sitting in temporary folders. If you don't realize it's gone and go rescue it, it's gone. A lot of our clients are in the earlier phases — phase one, phase two, phase 2B — where you're not relying on that many patients, which means every single patient counts. You can't afford to lose one patient's data. So although looking into this might, on the surface, seem like a load of extra work, I promise you it's needed — one, because it's a regulatory requirement, and two, because it has huge benefit. It raises your chance of success.

Another very important document is the data flow diagram, which again isn't a static requirement — it's something you need to keep working on. One important aspect is how data moves from one system to another. And a really important point: if you don't know which system your data is in, how can you say it has integrity? You can't. So the data flow diagram helps create a visualization of where everything actually is — it's like my roadmap, the starting point I use to ask, where is the risk in this diagram? A huge part of that is how data flows between systems — the data transfer. Is it an integration? If so, who's managing that, if it's two different systems? In an ideal world you'd be using the same data model in the background, but that's not always the case.

You'll sometimes have the same database — which is rare, though there are a few vendors now using the same database, which is exciting to see. But in most cases, even in the large CROs — and I'd say especially the large CROs — there are more systems, so this challenge is often greater than in some of the newer CROs that have built their own homegrown technology, designed to connect everything from the start. Instead, it's often an acquisition strategy — trying to fit one Lego piece to another and seeing how it goes. And often we rely on manual transfer — a manual download, a manual upload, and reconciliation between the two.

We've got a client relying on manual download and upload for their eligibility criteria, and you look at that and think, how quickly can we manage this process, how fast is that reconciliation? Because we need speed to make sure they can meet their recruitment timelines. So there are all these moving pieces, and it all needs to form part of your oversight strategy.

Kieran @ Seuss+ (17:29)
Yeah. What I love — Excel, reports, and data analysis genuinely excite me, contracts too, so I know I sit in this, not everyone's the same — but what I love about the digital data flow that's part of the clinical systems analysis we do is that it gives everyone a centralized, aligned, visual agreement of how things work. It lets so many different functions and layers understand the same thing. For me, it serves a similar purpose to KPIs and study metrics — I love KPIs because they let us have a conversation grounded in the same base data. We can both see, visually, that there's a signal, that something's not going quite the way it should. The digital data flow does the same for me — you get people from different functional departments and layers in the organization going, "wait a minute, I didn't realize that — so what does that mean?" Because otherwise we just assume it's a system, it's data — we assume it's perfect, or we assume it's fraught with error, but we don't actually know why, or how to fix it, and we're often layers away from where the fix actually happens, dealing with non-client-facing staff. When we all understand the same thing, at least we're having the conversation in the same language, and we can agree on what the risks actually are and what we're going to do to mitigate or understand them.

It makes me sad when I see companies say, "yeah, we have a digital data flow, we're fine" — and then I ask where their biggest risk areas are, and it's crickets, or "yeah, but leave it alone, it's fine." Well, when it's not fine, you don't find out until you have data you can't recoup, costs you have to catch up on, more patients you need to recruit, or just huge friction in relationships, because we're not speaking the same language — or we don't even know which language we should be speaking together. So I think it brings together the fixed systems and the human understanding, and therefore lets us align to decide what needs to be done.

Something you mentioned to me before that I wanted to make sure we talk about — you've talked about investor confidence, and that's not just linked to our chances of approval, it's also about having the data in order to attract the next step, whether that's the next round of fundraising for further development, a potential partnership with a pharma, or an exit to a pharma. How do you see that linking together? This is why I wanted you to talk about it here.

Yeah, I'd say it forms part of the due diligence — are we going to acquire this asset or not, do you have the right data and documentation in place. If you can't show your workings, and show you've followed the recent guidance on how we should be checking that our data has integrity — if you haven't followed those steps, then in due diligence, the investor or big pharma would rightly question what they can actually do with the asset. Would they still want to acquire it? Would they want to acquire it for the value you believe it's worth, based on your studies, if they then have to redo the study to get the assurance they need? So yes, I'd say investors and big pharma are putting more steps into place — and if they're not, they should be, to properly do due diligence on the assets they're acquiring.

Kieran @ Seuss+ (21:49)
And I think that due diligence is happening. Someone with decades of experience at a very big pharma once said to me, when I told him what we were doing, that what's wonderful about smaller biotechs and pharma adopting this mindset is that, as the end user of that transaction, he knows what he can use, what he doesn't need to redo, because he can't prove the value of that data, and knows how to integrate it or not. That's huge end value. So when that rolls back to impacting a transaction, it may take a long time to fully play out, but as the landscape evolves more quickly, I think those checks will happen earlier and earlier, rather than "I've got this asset, this team, I need to incorporate it and figure out how to use it." So — thank you for that.

At Seuss+ we're all about that interaction between vendors and their clients — our clients, the biotechs and pharma. Where there are humans, there are errors, but where there are humans, there are also assumptions. This is a wonderful way to drive an asset through clinical development — we need to use vendors, and most vendors are actually much better suited to this than trying to do it in-house. Not everything, but there's specialty expertise and knowledge needed, so that collaboration and relationship matters. But part of the job of a sponsor, and of our clients, is oversight. We sometimes talk about the myths of vendor compliance — this assumption of who's doing what, and what you can assume: "it's your system, therefore you're responsible, and I assume you've got it under control." But — you've said this to me before — we run into sponsors assuming their vendors handle things like system validation. What's the danger of that mindset? On the one hand, you're paying millions, so you should be able to assume certain things are in place. What's the danger of that complete assumption?

So the danger of that assumption is that, well, it's incorrect — because if you don't check, and you assume, and then you're caught out, you're still at fault. You can delegate as many tasks as you like as a sponsor, but you're still ultimately responsible — including for your data integrity, and for ensuring your systems are fit for purpose. So if you haven't got a process in place that—

Kieran @ Seuss+ (24:43)
—and you're still at fault.

—can show the systems are fit for purpose — and that includes the labs you're working with, any of your vendors. We've seen this from capturing your primary endpoint through ePRO or eCOA, particularly since one of the benefits of working with biotechs is that they're so innovative — we've worked with plenty who've used continuous monitoring endpoints and wearables, which are genuinely at the forefront of what we can capture now, which is exciting. But if you haven't ensured those systems are fit for purpose, and shown you've done the steps, you can't show your workings, and it leaves you in a very vulnerable place. Unless you've taken those steps, you're in a vulnerable position, and your data may not be accepted at the end of the day.

Kieran @ Seuss+ (26:06)
Is it a correct assumption that the vendor is responsible for validating the system? Is that true?

Ultimately, the sponsor is responsible for that. You've got to make sure you've checked that the process the vendor used to validate it is correct, and that the validation documents themselves are up to scratch. We've seen quite a few clients say, "yeah, I'm sure it's validated" — that's not good enough anymore.

Kieran @ Seuss+ (26:42)
Well, that's the assumption. The way I translate that in my head is: it's not that a vendor would ever say, "we haven't validated that, that's your problem, sponsor" — well, it could be, if they did say that — but it's up to the sponsor to check it's in place. In essence, whoever is selling you the system should be the one investing in that validation. Or not?

Well, yes — as a vendor, you'd hope they've done everything they need to.

Kieran @ Seuss+ (27:12)
No, no — but then you're into "hope" and "assumption," and those are far too close together.

Well, if as a sponsor you don't check — yes, if I was talking to a vendor, I'd say you should be validating, making sure you've selected the right system that's fit for purpose. But as a sponsor, you've got to make sure you've checked, and that you've put the right process in place — whether that's a computer systems validation audit, or however you want to approach it. That's probably the more robust way we'd recommend. Some people try to squeeze it all into a qualification audit, but the danger there is: do you go into enough detail to actually demonstrate you've looked at the specific validation documentation, as well as the process? So it's a question of how you've achieved it — not just whether you've achieved it.

Kieran @ Seuss+ (28:10)
Yeah, I think that's true of a lot of things — we often see clients thinking, and saying, "we've got all of these things to do," and they want to get it right the first time, always, which I understand. But with a lot of these things, it's about taking the steps: "this is what we thought at the time, this is where the risks were that we identified, this is what we put in place to address it, this is how we checked it." I don't think we often see pushback for getting it wrong if you missed something — you may have missed it because no one knew it was a risk at that point until it was, and it's about how you dealt with it. It's about making sure you've done it, and that you're complying with it for yourself. That's what I love about this application of risk — that's my happy place: the documentation that says, "these are the risks we're going to mitigate, these are the ones we know are there but won't mitigate, we'll put signals in place for those, and that's just how it is — we can do nothing about them, it's part of the business, we accept it." That process alignment lets you reduce internal capacity needs, and manage the stakeholders involved with opinions.

But it applies here too — you make decisions, and you've checked that validation is in place and fit for purpose for your needs. Not every system you use needs the same level of documentation or validation as another — it's not black and white. And I think, when we're insecure as humans and as leaders, we think, "it has to be perfect, we need to eliminate this risk" — but that's not really the point. It's: what's our biggest risk, what are the main priorities, and what do we make sure is in place?

No, it should be very much a risk-based approach. And as soon as you're aware there's a gap, you just deal with it. We're doing the best we can — it's a big job, with a lot of steps, but as soon as you're aware of a gap, it's about how you go about resolving it.

Kieran @ Seuss+ (30:43)
Yeah. So the last topic I wanted to touch on — an exciting one, for us, for our clients, for our industry, for the world — is the incorporation of AI tools. It's an area that feels both exciting and scary and risk-fraught, but it may also let us do things faster and cheaper, which is a huge draw, especially for smaller biotech. We're living to a budget that's been fundraised, investors have a timeline to exit and make sure their investors get a return on that fund — so it's very much tied to time and budget. But it's also exciting, because it can reduce some of the tedious, difficult work. Would you mind giving us a few thoughts on incorporating AI into clinical development — the attractiveness, the risks it brings, and, as you've mentioned, the regulations already in place?

Yeah — this is partly why I said so much has changed since I came back from maternity leave, even though it wasn't that long. AI is suddenly being used so much more within the clinical trial space, which is really exciting to see. If you think back to 2016, there was one application to the FDA that included AI. Last year, there were 300 — and by the time I came back from maternity leave, we suddenly had our first biotech clients all using AI. It's wild. But it does mean other regulations now need to be considered — we can't treat the use of AI the same way as any other system. There are additional steps and considerations, and we just need to make sure that, as with anything we include, we're following the right steps.

Suddenly, with the use of AI, we have the EU AI Act in place, and the new FDA regulatory guidance with its seven-step credibility assessment framework to consider — all these additional things. But should that make us fearful of including it? No — from a regulatory standpoint and an investment standpoint, I think investors are genuinely excited about it, just as people are excited about it generally. It's just about making sure we're doing it the right way. I think biotechs, as generally more nimble organizations, are perfectly placed to start exploring and including it — but it's about making sure we do it in a way that checks off the guidelines in place, because that de-risks it for us, and helps ensure it will be accepted at the end. Without following the right steps, we just put ourselves at risk of it not being accepted.

But from an excitement standpoint, I think it's right for the science of what we do — that we're considering it, looking at it, and its potential in our organizations. So I'm really excited about it. The number of applications being used is far greater than ever before, and practically speaking, on an individual level, we're all probably using AI more in our own lives too, which helps make it more accessible in our minds. But again, it's about knowing when to use it, and how to use it safely.

Kieran @ Seuss+ (34:45)
Yeah, very good, thank you. So Lauren, I think we're at time, because we could talk for hours, and I look forward to talking more about these topics in the years to come. Thank you for giving us a masterclass in what compliance means, in the risks we need to identify, and for touching on some of these topics. And for our listeners — thank you for listening. Don't let vendor assurances, or any of the myriad assumptions you hold, either lull you into comfort or drive you into the panic zone. Ask the hard questions, ask for help — it becomes clearer and clearer what needs to be done. But we do think it's a good evolution. Own your data. Compliance is not a checkbox — it's a process of evolution. As Lauren said, it's sometimes evolving, and that's okay. We just need to make sure — and I hate that word, "just" — but we need to make sure we have oversight, and that we know how this is flowing through our clinical development design. So thank you, Lauren.

And if you'd like to read more, I've written a white paper on this called Hidden Risks — it covers more of the areas you might need to look at, to help make sure you're covering some of the areas we've seen biotechs fall down on. That's on my LinkedIn page — please have a look if you'd like more information. And in the coming months, I'll be writing more on AI too.

Kieran @ Seuss+ (36:21)
Exciting, exciting. So reach out if you have questions — we're all, as an industry, in the same evolution, and it's an exciting time. Thank you for your time, and have a wonderful day.