PHILIPS Image Guided Therapy Systems User Guide

Mind the Gap: taking AI from Lab to Field
Dennis Dams (TNO-ESI)
Lena Filatova (Philips Healthcare, IGT-S)
16 April 2024

Philips Image Guided Therapy Systems (IGT-S)

“We develop seamlessly integrated health systems, including interventional X-ray systems and software solutions, that enable physicians to treat patients with personalized and minimally invasive procedures.” 

Quality expectations with growing complexity

• Hundreds of configurations
• Millions lines of code
• High safety requirements
• Obligation to maintain and update for 10+ years

Bug resolution process optimization

Can these steps be accelerated by tapping into the available process data?The steps we focused on, and the technologies used
Can these steps be accelerated by tapping into the available process data?Bug Triaging – Which team has to fix this bug?

Bug Triaging – Machine Learning approach

Training an ML classifier this is done on known cases: Using a trained ML classifier (aka model) in deployment – on new cases:

Bug Triaging –We tried everything under the sun

Our findings:

• “Classical” classifiers (like Logistic Regression) outperform Deep Learning (Neural Networks)
• The best classifier, in deployment, predicts the right team in 46% of the cases

Bug Triaging – 46% is not as bad as it seems

• Random guessing would result in about 14% correct predictions (as there are 7 teams)
• Triaging by humans (Change Control Board – CCB) is first-time-right in about 64% of the cases

Detailed error analysis on 195 cases (after a few months of deployment):

• Random guessing would result in about 14% correct predictions (as there are 7 teams)
• Triaging by humans (Change Control Board – CCB) is first-time-right in about 64% of the cases

Detailed error analysis on 195 cases (after a few months of deployment):

• Random guessing would result in about 14% correct predictions (as there are 7 teams)
• Triaging by humans (Change Control Board – CCB) is first-time-right in about 64% of the cases

Detailed error analysis on 195 cases (after a few months of deployment):

Lessons learned

1. Bug triaging is intrinsically difficult
   – Different authors of bug reports use different language
   – Bug tossing is part of the process
   – Expertise areas of teams may be overlapping

2. Introducing Machine Learning in the triaging process is not trivial
   – Introducing ML requires adaptation of workflow and sufficient confidence
   – The (perceived) value of ML depends on more than the accuracy score
   – Keeping the ML model in shape (re-training to deal with data and concept drift)

Bug Report Creation – Is it already in the database?

Bug Report Creation – Encouraging better bug reports

We introduced a wizard (GUI) offering the following features:

• List potential duplicate reports as the user starts typing a new report

• Suggest terms to use, by domain specific, interactive query expansion
  – query expansion: when query contains “automobile”, also search for “car”
  – domain specific: when query contains “System Version 18.2.ext”, also search for “DeLuxeEdition”
  – interactive: additional terms are offered as suggestion for the user to include

• Suggest/generate suitable report headlines

• Apply guidelines for wording and structuring the report
  – previously, these were communicated by email, which made it difficult to remember them

Bug Report Creation – Encouraging better bug reports

We introduced a wizard (GUI) offering the following features:

• List potential duplicate reports as the user starts typing a new report

• Suggesting terms to use, by domain specific, interactive query expansion
  – query expansion: when query contains “automobile”, also search for “car”
  – domain specific: when query contains “System Version 18.2.ext”, also search for “DeLuxeEdition”
  – interactive: additional terms are offered as suggestion for the user to include

• Suggesting/generating suitable report headlines

• Applying guidelines for wording and structuring the report
  – previously, these were communicated by email, which made it difficult to remember them

Bug Localization – Which piece of code is to blame?

Bug localization – Via direct correlation

Use correlation between terms in the bug report to terms in the source codeUse correlation between terms in the bug report to terms in the source codeThe testers who write bug reports “speak a different language” than the programmers
Use knowledge about how similar bugs were fixed in the pastUse knowledge about how similar bugs were fixed in the pastReport similarity (x) versus change-set similarity (y) for each pair of issues:

Bug localization – Interactive, Visual Approach:

To analyze bugs, most engineers inspect logs (usually not attached to the bug report)Tracy (and related utility) available in open source:
https://github.com/TNO/vscode-tracy
https://github.com/TNO/vscode-tracy-csv-converter
Other ESI partners are trying out and contributing to Tracy

Taking AI from the lab to the field – Conclusions

• Theory and practice do not always match
  – Ill-defined problems
  – Data quality and balancing

• Embedding AI or other advanced tools into the organization can be challenging
  – Tool for the organization or organization for the tool
  – Usability, maintainability, explainability

• People perception and trust

PS. We believe that these conclusions generalize beyond our specific use case (bug resolution) to other applications of AI
Thanks to:

• Alwyn van den Berg
• Bart Boeren
• Patrick Bronneberg
• Philip Brouwers
• Gergely Mincsovics
• Daan van der Munnik
• Rogier Vermeulen
• Jelte Peter Vink| • Ronald Begeer
• Erik Boertjes
• Teun Buijs
• Koen Kanters
• Christos Kitsanelis
• Saskia Lensink
• Luna Li
• Anca Lichiardopol
• Filip Zamfirov
---|---

© Koninklijke Philips N.V.
… and many more!

Documents / Resources

| PHILIPS Image Guided Therapy Systems [pdf] User Guide
Image Guided Therapy Systems, Image Guided Therapy Systems, Guided Therapy Systems, Therapy Systems, Systems
---|---

References

• GitHub - TNO/vscode-tracy: Tracy is a Visual Studio Code plugin made to simplify log analysis
• GitHub - TNO/vscode-tracy-csv-converter: A CSV converter example for Tracy

Read User Manual Online (PDF format)

Download This Manual (PDF format)

Download this manual  >>