Skip to content

Bioimaging techniques yield deeper insights into the molecular and cellular processes of the human body than ever before. The procedure of identifying and detecting the boundaries of specific areas of interest such as single cells within a microscopic imaging is called segmentation. In recent years, image segmentation based on deep learning has achieved great success in biomedical research. The algorithm usually learns via a large amount of “ground truth” – training data annotated by humans. This supervised learning method requires researchers to label a large amount of data. Even though unsupervised methods that can train deep learning models without human annotation exist, their performance is often less accurate than that of supervised methods – and thus not sufficient for biomedical research. That is why scientist working on the project »Smart Human-in-the-loop Segmentation« aim to develop a powerful deep learning model that is trained with a minimum amount of human effort.

Open source deep learning workflow enables more efficient image segmentation

The overall method generalises and further develops the iterative deep learning workflow of the Allen Cell and Structure Segmenter. The mentioned Segmenter is a toolkit for the 3D segmentation of intracellular structures in fluorescence microscope images, co-developed by Dr Jianxu Chen and the Allen Institute for Cell Science. The first step of the »Smart Human-in-the-loop Segmentation« is the development of effective strategies to automatically select representative samples for the initial curation or annotation based on the project team’s previous work on suggestive annotation, a deep active learning framework that combines a fully convolutional network with active learning. Next, a few samples will be selected for a rough manual annotation by the researchers. These are used for the initial training of the algorithm for the segmentation of images, which will then be applied to all images in the data set. After that, a number of images will be revised manually and entered into the workflow to train the algorithm.

Overview of the human-in-the- loop segmentation workflow for the automated analysis of the volume of B cell follicles and T cell zone in a Peyer’s patch, a grouping of lymphoid follicles in the mucus membrane of the small intestine. The images on the left are the raw light sheet fluorescence microscopy images and the ones on the right are the images after segmentation.

© ISAS / Jianxu Chen

For each iteration, the researchers are going to develop methods to enable the deep learning model to automatically detect potential errors that will then initiate human curation. Finally, effective active learning and continual learning methods will be developed to avoid so called catastrophic forgetting. Meaning the drastic loss of previously learned information after being trained on a new task, in the iterative deep learning workflow. This enables accurate image segmentation with a low amount of human intervention. Eventually, the developed algorithms will be open source and made available as a graphical user interface tool for the broader biomedical image analysis community.

Share

Select publications

JMIR Serious Games, Vol. 12, 2024, P. e52785

Egger J, Gsaxner C, Luijten G, Chen J, Chen X, Bian J, Kleesiek J, Puladi B.

Is the Apple Vision Pro the Ultimate Display? A First Perspective and Survey on Entering the Wonderland of Precision Medicine

https://doi.org/10.2196/52785

Nature Methods, Vol. 21, No. 8, 2024, P. 1390-1393

Zhang S, Dai G, Huang T, Chen J.

Multimodal large language models for bioimage analysis

https://doi.org/10.1038/s41592-024-02334-2

Journal of Imaging Informatics in Medicine, 2024

Krieger K, Egger J, Kleesiek J, Gunzer M, Chen J.

Multisensory Extended Reality Applications Offer Benefits for Volumetric Biomedical Image Analysis in Research and Medicine

https://doi.org/10.1007/s10278-024-01094-x

2024 IEEE International Symposium on Biomedical Imaging (ISBI), 2024

Zhou Y, Zhao S, Sonneck J, Chen J.

2D Label-Free Prediction of Multiple Organelles Across Different Transmitted-Light Microscopy Images with Bag-of-Experts

https://doi.org/10.1109/ISBI56570.2024.10635298

2024 IEEE International Symposium on Biomedical Imaging (ISBI), 2024

Yao J, Hagemann N, Xiong Q, Chen J, Hermann DM, Chen C.

Topological Analysis of Mouse Brain Vasculature via 3d Light-Sheet Microscopy Images

https://doi.org/10.1109/ISBI56570.2024.10635226

Nature Methods, Vol. 21, No. 2, 2024, P. 182-194

Reinke A, Tizabi MD, Baumgartner M, Eisenmann M, Heckmann-Nötzel D, Kavur AE, Rädsch T, Sudre CH, Acion L, Antonelli M, Arbel T, Bakas S, Benis A,…

Understanding metric-related pitfalls in image analysis validation

https://doi.org/10.1038/s41592-023-02150-0

GigaScience, 2024

Sonneck J, Zhou Y, Chen J.

MMV_Im2Im: an open-source microscopy machine vision toolbox for image-to-image transformation

https://doi.org/10.1093/gigascience/giad120

Nature Methods, Vol. 21, No. 3, 2024, P. 368-369

Zhou Y, Cao J, Sonneck J, Banerjee S, Dörr S, Grüneboom A, Lorenz K, Zhang S, Chen J.

EfficientBioAI: making bioimaging AI models efficient in energy and latency

https://doi.org/10.1038/s41592-024-02167-z

Nature Communications, Vol. 14, 2023, P. 1-13

Cibir Z, Hassel J, Sonneck J, Kowitz L, Beer A, Kraus A, Hallekamp G, Rosenkranz M, Raffelberg P, Olfen S, Smilowski K, Burkard R, Helfrich I, Tuz…

ComplexEye:a multi-lens array microscope for high-throughput embedded immune cell migration analysis

https://doi.org/10.1038/s41467-023-43765-3

Nature Machine Intelligence, Vol. 5, 2023, P. 687-698

Jia Z, Chen J, Xu X, Kheir J, Hu J, Xiao H, Peng S, Hu X, Chen D, Shi Y.

The importance of resource awareness in artificial intelligence for healthcare

https://doi.org/10.1038/s42256-023-00670-0

Nature Methods, Vol. 20, 2023, P. 968-970

Chen J, Viana M, Rafelski S.

When seeing is not believing: application-appropriate validation matters for quantitative bioimage analysis

https://doi.org/10.1038/s41592-023-01881-4

Cell Reports Methods, Vol. 3, No. 3, 2023, P. 100436

Spangenberg P, Hagemann N, Squire A, Förster N, Krauß SD, Qi Y, Mohamud Yusuf A, Wang J, Grüneboom A, Kowitz L, Korste S, Totzeck M, Cibir Z, Tuz AA,…

Rapid and fully automated blood vasculature analysis in 3D light-sheet image volumes of different organs

https://doi.org/10.1016/j.crmeth.2023.100436

IEEE Transactions on Medical Imaging, Vol. 41, No. 10, 2022, P. 2582-2597

Liang P, Zhang Y, Ding Y, Chen J, Madukoma CS, Weninger T, Shrout JD, Chen DZ.

H-EMD: A Hierarchical Earth Mover's Distance Method for Instance Segmentation

https://doi.org/10.1109/TMI.2022.3169449

Immunological Reviews, Vol. 306, No. 1, 2021, P. 293-303

Grüneboom A, Aust O, Cibir Z, Weber F, Hermann DM, Gunzer M.

Imaging innate immunity

https://doi.org/10.1111/imr.13048