Resolving the Evidence Generation paradox

Psyrin is an early-stage start-up aiming to transform psychosis care pathways by leveraging speech-based machine learning technology for triage and assessment. Here, co-founder Edwin Wong pens some reflections on their experiences, following completion of the DigitalHealth.London Evidence Generation Bootcamp programme.

We almost turned down our offer to join the first Evidence Generation Bootcamp cohort – not because we had doubts or reservations, but because the initial offer email was sent to our spam folder! Thankfully we checked our junk mail 12 hours before the acceptance deadline, and thus began a 10-week journey with DigitalHealth.London.

The programme has been a  steep learning curve, which has expanded our perspectives and helped us to reconsider old assumptions. While we were previously cognisant of evidence generation, the programme supplied clarity and structure to this ambiguous process. We cannot claim yet to have successfully navigated the evidence generation landscape, but we now have some idea of how to start this journey – here we offer some thoughts that we hope you find useful!

What is evidence generation?

Prior to the Bootcamp, we were very confident in our rigid definition of “evidence generation” – as an early-stage startup leveraging machine learning, we were focused solely on large, cross-sectional trials demonstrating scientific effectiveness in a research context. After all, having spent months working to develop our cool technology, surely the only priority would be validating it’s scientific efficacy?

Nope.

Evidence includes a focus on understanding user experience, enabled by descriptive and qualitative studies with our potential users.

Evidence includes health economic analyses, a fascinating yet complex domain of research on its own, worthy of an entirely separate essay.

Evidence includes safety evaluation, which is often overlooked in digital health – safety is assessed with various techniques, such as clinical audits and prospective hazard analysis.

Even within our initial definition of generating effectiveness data, there was much more to consider, including the various possible study designs and their use cases. Thanks to the Evidence Generation Bootcamp, we were signposted to several key, publically-available resources; for instance, see this link for some great guidance on the evidence generation journey.

Armed with this new knowledge, our next steps are to act on these lessons and collect evidence. This , however, is far easier said than done…

The evidence generation paradox

Digital health enthusiasts will surely be familiar with the central conundrum underlying the evidence generation process: to generate evidence, you must deploy your product, but to deploy your product, you need solid evidence. Now the important question – has the Evidence Generation Bootcamp offered a solution resolving this paradox?

Well, we have come to realise that there is no singular solution; instead, navigating this paradox requires clarity and a deep understanding of relevant stakeholders and your product’s value propositions. To this end, the programme has certainly thrust us forward:

• We were introduced to the NICE evidence standards framework (ESF) for digital health technologies, and guided in conducting a gap analysis structured around this framework. While fairly content-heavy, the ESF is a fantastic way to frame your objectives and work towards resolving the evidence generation paradox
• Every week involved a new collection of leading experts in various domains of the evidence generation process – providing unique insight and access to brilliant minds who offered custom advice and guidance
• Over the various seminars, we were introduced to an impressive array of guest speakers, many of whom were founders in adjacent digital health areas. Drawing parallels between our journeys has re-affirmed our belief in the possibility of “solving” the potential issues that may arise during evidence generation.

In summary, one resolves the aforementioned paradox by shifting the goalposts – take one step at a time and adopt a holistic view of the evidence generation journey.

Strength in numbers (of enthusiasts)

While the content delivery during this Bootcamp was great, we feel that the biggest value in this course were (and yes this is cliché) the people we’ve met along the way. Our cohort was filled with a diversity of startups, brilliant individuals all keen on expanding their knowledge of evidence generation in the digital health sector. Each and every startup worked on cutting-edge technology in a health area of dire need – and some even worked adjacent to us in the mental health domain (shoutout to MyMind and heyr). There was a range of experience and maturity across the startups, making for fascinating conversation and real insights.

Moreover, the DigitalHealth.London team has been an absolute joy to work with – we want to make special mention of our NHS Guide Sam Kyffin. He was an absolute machine, always available for a call, and very quick to respond over email. Sam alone was able to answer a plethora of my queries, but for the more nuanced issues, he was quick to broker an expert connection to add further clarity. While the bootcamp has now drawn to a close, we feel truly connected with the DigitalHealth.London team, and will certainly continue to pester them for assistance and advice.

Final thoughts

As the digital health landscape continues to evolve at breakthtaking pace, it is becoming increasingly difficult to navigate all the twists and turns of building a company in this domain. There is clear recognition of this issue – for instance the MHRA has announced a Software and AI as a Medical Device Change Programme, while the NHS has set up a multi-agency advisory service to provide regulatory advice.

But even with clearer written guidance available online, it is organisations like DigitalHealth.London, and programmes like the Evidence Generation Bootcamp, that prove pivotal in helping companies comprehend and embrance the (necessary) journey of deploying a product in the NHS. The interaction and adaptibility inherent to these programmes are invaluable – they allow for deep discussions, key clarifications, and chance connections that no written guide could fully encompass.

This programme has been truly transformational to the strategy of Psyrin moving forward, and we are happy to share more about our experience – if you have any questions or, more generally, want to chat about anything digital health, feel free to reach out to us via LinkedIn or at hello@psryin.co.uk. And remember, always check your junk folder!

DigitalHealth.London is delighted to publish blogs by the NHS staff and digital health companies we support through our programmes, as well as sector thought-leaders, experts and academics. Any opinions expressed within blogs published on our website are those of the author and not necessarily held by DigitalHealth.London. For more information, or if you would like to write a blog for our website, please email info@digitalHealth.london.