FDPNP4: Artificial Intelligence under the Microscope (Part 3)
Preface: I want to initiate a comprehensive series, “Fundamentals of Digital Pathology for Novice Pathologists (FDPNP)”. This endeavor aims to disseminate knowledge about the utilization of computer vision in the realm of modern pathology to pathologists who want to know more about digital pathology.
Introduction
In Part 1, we delved into the definitions of artificial intelligence (AI) and associated concepts. In Part 2, we explored the components of AI that play a role in digital pathology. Continuing with this mini-review, we further investigate their involvement in digital pathology. To achieve this, we will discuss the tasks that AI performs in the realm of digital pathology. These tasks are represented under the term “output” in Part 2.
What is a task?
In the realm of AI, the term “task” is frequently used to denote a specific problem or operation that the system is engineered to solve or execute. Despite its sophisticated connotation, this term remains somewhat abstract and ambiguous due to its encompassment of a wide array of problems.
To render a more precise definition, let’s narrow our focus to the tasks within digital pathology. I propose a division of AI tasks in digital pathology into four primary categories:
- Prediction
- Segmentation
- Detection
- Generation
Overview of AI-related tasks in digital pathology — Image by Author
Prediction
Prediction tasks involve forecasting an outcome or a feature of interest. When the outcome is a numerical value, such as age, gene expression, or protein expression, the task is referred to as a regression task. If the outcome is a categorical variable, the task is known as classification. When the task involves predicting a patient’s prognosis, it is termed a survival prediction task. Consequently, regression, classification, and survival estimation are categorized under Prediction tasks.
Examples include:
- Predicting gene expression from histology (regression)
- Grading tumors based on histology (classification)
- Cancer detection (classification)
- Risk stratification using histology (survival prediction)
These tasks often involve the application of machine learning or deep learning models. These models are trained on extensive datasets with labeled examples.
Examples of prediction tasks — Image by Author
Segmentation
Segmentation tasks entail dividing an image into multiple segments, with each segment corresponding to a distinct object or region of interest. In the realm of cell-level segmentation, basic classes often encompass normal tissue, tumor cells, and stroma. More advanced classes may include lymphocytes, fibroblasts, eosinophils, and collagenous stroma.
Examples include:
- Quantifying tumor-infiltrating lymphocytes (TIL)
- Quantifying tumor-stroma ratio (TSR)
- Characterizing tumor microenvironment
An example of a cell-level segmentation task — Image by Author
In addition to cell-level segmentation, there are also architecture-level and nuclei-level segmentation tasks in histopathology. In architecture-level segmentation, the classes include various architectural structures such as glandular, solid, or papillary structures. On the other hand, nuclei-level segmentation involves classes such as nuclei and non-nuclei.
An example of a nuclei-level segmentation task — Image by Author
Detection
Object detection tasks involve training models to identify and quantify specific objects or features in images. By detecting and counting important features, AI models can also provide a more objective and precise evaluation of diseases in digital pathology.
Examples include:
- Mitosis counting
- Vessel counting
- Nuclear counting
- Detecting acid-fast bacilli (AFB) in Ziehl-Neelsen stained slides
Detecting and counting mitosis — Image by Author
Generation
Generation tasks entail creating new images or texts. There are various types of generation models. Unconditional models generate images without any prompts, producing synthetic images akin to the training images. Conversely, conditional models generate images based on their training and input, which can be either text or an image.
Examples without input:
- Image generation
Examples with an image as input:
- Stain transformation
- Frozen Section — FFPE translation.
- Image-to-text generation
Examples with text as input:
- Text-to-image generation
Frozen section translation to formalin-fixed, paraffin-embedded (FFPE) section, this image is modified from the original image in [1]
Conclusion
AI tasks can be broadly classified into four main categories: prediction, segmentation, detection, and generation. Each of these tasks has its unique applications in addressing various challenges in digital pathology.
In our forthcoming blog posts, we will delve into the workings of deep-learning models, specifically how they process histological images to produce insightful outputs.
For those who may be interested, I hope you enjoy this mini-review.
Reference
- Ozyoruk KB, Can S, Darbaz B, Başak K, Demir D, Gokceler GI, Serin G, Hacisalihoglu UP, Kurtuluş E, Lu MY, Chen TY, Williamson DFK, Yılmaz F, Mahmood F, Turan M. A deep-learning model for transforming the style of tissue images from cryosectioned to formalin-fixed and paraffin-embedded. Nat Biomed Eng. 2022 Dec;6(12):1407–1419. doi: 10.1038/s41551–022–00952–9. Epub 2022 Dec 23. PMID: 36564629.