Yushan Zheng

Gigapixel image analysis, particularly for whole slide images (WSIs), often relies on multiple instance learning (MIL). Under the paradigm of MIL, patch image representations are extracted and then fixed during the training of the MIL classifiers for efficiency consideration. However, the invariance of representations makes it difficult to perform data augmentation for WSI-level model training, which significantly limits the performance of the downstream WSI analysis. The current data augmentation methods for gigapixel images either introduce additional computational costs or result in a loss of semantic information, which is hard to meet the requirements for efficiency and stability needed for WSI model training. In this paper, we propose a Promptable Representation Distribution Learning framework (PRDL) for both patch-level representation learning and WSI-level data augmentation. Meanwhile, we explore the use of prompts to guide data augmentation in feature space, which achieves promptable data augmentation for training robust WSI-level models. The experimental results have demonstrated that the proposed method stably outperforms state-of-the-art methods.

Large-scale pre-training models have promoted the development of histopathology image analysis. However, existing self-supervised methods for histopathology images primarily focus on learning patch features, while there is a notable gap in the availability of pre-training models specifically designed for WSI-level feature learning. In this paper, we propose a novel self-supervised learning framework for pan-cancer WSI-level representation pre-training with the designed position-aware masked autoencoder (PAMA). Meanwhile, we propose the position-aware cross-attention (PACA) module with a kernel reorientation (KRO) strategy and an anchor dropout (AD) mechanism. The KRO strategy can capture the complete semantic structure and eliminate ambiguity in WSIs, and the AD contributes to enhancing the robustness and generalization of the model. We evaluated our method on 7 large-scale datasets from multiple organs for pan-cancer classification tasks. The results have demonstrated the effectiveness and generalization of PAMA in discriminative WSI representation learning and pan-cancer WSI pre-training. The proposed method was also compared with 8 WSI analysis methods. The experimental results have indicated that our proposed PAMA is superior to the state-of-the-art methods.

Multiple instance learning (MIL) based whole slide image (WSI) classification is often carried out on the representations of patches extracted from WSI with a pre-trained patch encoder. The performance of classification relies on both patch-level representation learning and MIL classifier training. Most MIL methods utilize a frozen model pre-trained on ImageNet or a model trained with self-supervised learning on histopathology image dataset to extract patch image representations and then fix these representations in the training of the MIL classifiers for efficiency consideration. However, the invariance of representations cannot meet the diversity requirement for training a robust MIL classifier, which has significantly limited the performance of the WSI classification. In this paper, we propose a Self-Supervised Representation Distribution Learning framework (SSRDL) for patch-level representation learning with an online representation sampling strategy (ORS) for both patch feature extraction and WSI-level data augmentation. The proposed method was evaluated on three datasets under three MIL frameworks. The experimental results have demonstrated that the proposed method achieves the best performance in histopathology image representation learning and data augmentation and outperforms state-of-the-art methods under different WSI classification frameworks.

Fine-grained cross-modal retrieval (FGCR) for digital pathology plays a crucial role in diagnostic support systems, enabling targeted searches using detailed descriptive queries. However, current research on FGCR for histopathology images faces three main challenges: the domain gap between medical language and histopathology images, the lack of robust fine-grained feature representation, and the absence of a comprehensive dataset to support such analysis. In this paper, we present a novel medical language and image representation learning framework specifically designed for FGCR in digital pathology. Our approach addresses the domain gap through a novel Knowledge-guided Contrastive Learning (KCL) strategy for representation alignment between texts and images in the histopathology domain. We also introduce a Dual-transformer network that enhances multimodal feature learning by integrating both global context and local details from pathology images. To facilitate research in this area, we have created the BreaKHis Cross-Modal Retrieval (BKCMR) dataset, containing over 5,000 image-text pairs with detailed morphological descriptions created by pathologists. Extensive experiments demonstrate that our method significantly outperforms current state-of-the-art approaches in cross-modal retrieval tasks, achieving substantial improvements across all evaluation metrics. This research represents an important step toward enabling more precise and effective cross-modal searches in digital pathology, with potential applications in diagnostic assistance and educational platforms.

Attention-based transformer networks have shown immense promise for enhancing feature representation by capturing token correlations. However, the standard transformer architecture may not fully exploit the complex contextual dependencies or spatial relationships present in histopathology whole slide image (WSI) patches. This paper introduces a kernel attention transformer (KAT) architecture designed specifically for histopathology WSI analysis. Unlike traditional transformers, our KAT network learns kernel-based attention patterns that offer more varied attention to effectively model the complex relationships in histopathology images. Specifically, we introduced a designed kernel structure to represent attention patterns, which significantly expands the representational capacity beyond what the traditional dot-product attention allows, while maintaining training efficiency. Further, to improve its computational efficiency on high-resolution histopathology WSIs, we developed a memory-token-based KAT classifier that efficiently models global dependencies across the entire slide while being computationally tractable. The proposed method was evaluated on four public histopathology WSI datasets for classification. Experimental results demonstrated that our method outperforms state-of-the-art approaches, achieving significantly better accuracy, particularly on the most challenging histopathology tasks.

Histopathological whole slide images (WSIs) are gigapixel images widely used in cancer diagnosis. Each WSI may include multiple diagnostically relevant regions, making them challenging to represent. Meanwhile, the spatial relationships between tissue regions are often diagnostically significant. This paper proposes a novel location-aware graph encoding network (LaGeNet) for representing histopathology WSIs that captures both region-wise feature representations and spatial relationships. Unlike existing multiple-instance learning (MIL) methods that focus solely on region features, our approach builds a location graph from region nodes with positional encoding and creates a diagnostic node for global feature learning. The model is trained end-to-end using a contrastive learning strategy to capture diagnostic similarities between WSIs. We evaluate our approach on five datasets from different organs, demonstrating superior performance in diagnostically-relevant region retrieval compared to state-of-the-art methods. Our LaGeNet framework provides an effective approach for encoding gigapixel histopathology images while preserving spatial information.