Fig. 2

Consensus Labeling of Annotated ROIs. a WSI as viewed in the HistomicsUI application, containing annotated ROIs. ROIs are annotated by experts and/or nonvices for Pre-NFTs (blue circles/boxes) and iNFTs (red circles/boxes) using the point annotation tool and these points are converted to bounding boxes using watershed and manual corrections. Green circle in ROI marks a completely annotated ROI. b Process used to create labels for unlabeled ROIs using pre-trained models. The best models for each annotator are used as an initial guess of the Pre-NFT/iNFT labels. These sets of predictions are then matched between each other: for each prediction in an ROI, find if any predictions from other ROIs match, using the IoU metric (threshold of 0.5). When creating the final set of labels of the ROI, set a minimum number of models that must agree on a label to be given as the “ground truth”: n. The bottom row of images shows that as n is set higher, the number of labels decreases as more models must agree. Setting n to 1 includes all predictions from all models, with the label (Pre-NFT/iNFT) being set by the label most models agree with. In cases of ties, iNFTs takes precedence over Pre-NFT as the label. Close ups of iNFT and Pre-NFT predictions are also shown with bounding boxes for n = 1, 4, and 8. NFT annotations and workflow for consensus labeling