Digital health needs better evaluations, but there are significant barriers to doing that. Public Health England worked with digital developers across the private and public sectors to determine what help they needed. This work led to the Evaluating Digital Health Products resource on GOV.UK, an easy-to-use guide to evaluation. This resource was the starting point for the second webinar in the #EvaluateDigiHealth series and led to a lively discussion about what help developers and commissioners of digital health need, and what help might be coming in the future.
Find out more about the series and sign up for upcoming webinars here.
Chair: Bobak Saadat, Public Health England
Charlotte Fountaine, Co-Founder and CPO, Kalda
Dr Sonia Ponzo, Head of Clinical Research, Thrive
Prof Henry Potts, Professor of Health Informatics, University College London
We’ve summarised the key themes and questions from the discussion:
The creation of gov.uk’s Evaluating Digital Health Products resources
Henry kicked-off the discussion by providing the background to Public Health England’s (PHE’s) creation of the gov.uk Evaluating Digital Health Products resources. PHE recognised that the digital health community at large, including developers in the public and private sectors, needed more support with evaluation. The resources aim to make it easier for developers to evaluate their products but also for commissioners to understand how they interpret an evaluation.
The resources include content which can be used by organisations to run workshops on developing a logic model for your product and prioritising what you should be measuring.
Charlotte, who was also involved in the creation of the resources, explained that a great deal of research went into the creation of the resources and it therefore meets the needs of lots of different people developing digital health products whether they are someone in the NHS or working in industry for example. She felt it was important for ensuring companies build the right metrics from the start and can have better conversations with academics or analysts they work with.
Sonia, as head of Clinical Research at digital health company, Thrive, felt the gov.uk resource was extremely useful. She also pointed out the NICE guidelines as another resource available to innovators but recognised that it can be easy for companies to gravitate to certain categories which are simpler – RCTs can be expensive for example so they might want to avoid these.
The need for a digital health evaluation mandate
Charlotte is Co-Founder and CPO of another digital health company, Kalda, which is earlier in their journey. She shared that Digitalhealth.London’s Evidence Generator workshops had been very useful for them at an early stage to identify the evaluation they need to undertake but flagged that the lack of a digital health evaluation mandate can make this difficult.
Henry shared that there is currently a lot of work coming from government bodies on this. The NICE framework doesn’t currently cover Artificial Intelligence (AI) products, but an AI section is coming in the future. He explained that a collaboration between NHSX, NICE, MHRA and CQC is bringing together advice around regulation for AI and that this may also lead to update regulations.
Kalda is currently facing challenges in understanding the best way to measure their impact and Charlotte pointed out that there was a tension between the measures that investors want to see (e.g. number of downloads) compared to what NHS stakeholders want to see.
Sonia agreed that clarity is needed on what is a requirement vs what is nice to have. Thrive have done all they feel they can, including registering a medical device, but they need to know what they should do now. There is a needed for guidance on what is required for companies at every stage in their life cycle.
Paying for evaluation – public vs private
Henry flagged the need for a conversation in the community about what companies should pay for or what the public bodies should pay for. He pointed out that there is a lot of cross-fertilisation between public and private sectors so we can’t assume that companies should pay for themselves – they are developing new products which provide health benefit to us all. One role Henry identified for the public sector here is defining standards and production of resources for example the PHE resource. Going forward he hopes to see NHSX providing infrastructure to support evaluation and a publicly funded community where innovators can learn from each other and swap case studies.
Henry felt that public money should pay for general research on how we measure engagement with technology. He felt that the public sector should be spending money on looking at how well certain methods of intervention work, for example virtual cognitive behavioural therapies, which covers a broad range of innovations.
Sonia felt public sector money would be well spent in creating a third-party evaluator to ensure safety of new products, acknowledging the complexity of this but also the real need.
Joining up guidance from different countries
One audience member asked about international guidance for evaluation. Henry steered audience members to view last year’s webinar Can we move beyond regulation stifling innovation? – International perspectives on regulation and evaluating digital health products.
Thrive is talking to clients in Italy, Europe and Australia and Sonia shared that generally in other countries not enough is known about evaluation and that usually the discussions are more about security and privacy.
Henry raised that some national bodies want evaluation of efficacy in a local population which adds to costs and can be a challenge. Sonia felt that for certain types of solutions, especially in mental health, there is a definite value in evaluating cultural differences.
Patient privacy and confidentially
The final question of the day was around patient privacy and confidentiality. Sonia shared that an app like Thrive which develops medically relevant information needs to be GDPR compliant. When going through NHS assessments companies need to ensure that data is safe and stored with the end user in mind.
Charlotte highlighted that lots of apps need end-to-end date encrypted systems, including for financial data too. At Kalda they only collect the minimum amount of data and she advised taking as little as possible and then storing correctly. For example, with advertising, she advised not to use google analytics and instead use a more reliable version.