Black Box Verification of GLP-Validated Image Analysis
Digital tissue image analysis is a computational method for analyzing whole-slide images and extracting large, complex, and quantitative data sets. Our scientists have implemented a robust “black box” verification testing protocol that ensures the accuracy of the algorithm’s output.
The qualification process requires a quorum of three trained experts, each assigned to one of the following roles:
- design of the algorithm
- quality control of the algorithm
- verification of the algorithm’s output
These experts are board-certified pathologists or a qualified designee with documented training in the identification of the object of interest (OOI). An OOI is defined as a specific region, lesion, cell type, organelle, or particle characterized by the animal model, the local tissue, and the special stain or assay used to identify it.
Every trained expert working on an algorithm can independently identify the region(s) of interest (ROI) where the OOI is being measured, distinguish the OOI from all other objects within the ROI, and can perform all of the measurements necessary to quantify the dependent variable or variables (endpoints). Trained users are required to fulfill the first two roles of the qualification process (design and quality control of the algorithm). These qualified experts also know how to use the platform that generates the algorithm.
Image Analysis & Stereology Services
Charles River’s image analysis and stereology services can aid your early drug discovery, animal model development, efficacy testing, and safety testing.
After reviewing the creation of the study material – from harvest to production of digital images – our trained experts critically evaluate numerous aspects of the project:
- image features used by the algorithm
- classification method used for segmentation of the OOI
- threshold or training data used to develop the algorithm’s heuristics
- application of the trained algorithm to novel test material
- any post-processing of the metadata generated by the algorithm
- all measurements and calculations used by the algorithm to generate study endpoints
What happens when you apply the speed and precision of deep machine learning algorithms to the experience of trained pathologists? Together, these assets create a powerful tool for quantifying pathology data rapidly, making numerical endpoints more accessible than ever.
Charles River now offers GLP-compliant, AI based image analysis, delivering data you can trust.
Through a robust qualification process, an experienced pathologist manually annotates the object of interest within the assigned field of view. These annotations are imported into the image analysis platform and compared to the algorithm’s performance in the same field of view.
Agreement between the trained expert and the algorithm is assessed using a DSC coefficient, Kappa index, or another metric of inter-rater reliability and the algorithm is refined when necessary.
Algorithm revisions are included in a new version which is then requalified and verified. Any discrepancies between the algorithm and trained expert are analyzed visually and discussed in a final report.
To learn more about digital pathology, talk to our team, or visit us at www.criver.com/pathology.
After approving the algorithm for use on all or a portion of the data, the reviewing member versions and locks the algorithm in a secure location. The name and version of the qualified algorithm is electronically associated with any data generated by it.
After application of the algorithm to the study material, a portion of the regions of interest are randomly selected to be verified by the third trained expert, usually a board-certified veterinary pathologist who is naïve to the algorithm.
Thus, by using the depth and breadth of expertise in our human resources and verifying the results of an automated analysis phenomenologically against that of a trained expert’s, we can deliver GLP-compliant automated quantification of study data.