Recent Trends of Fluorescence Lifetime Imaging Microscopy Analysis Using Machine Learning

Ilya D. Shchechkin (Login required)
Privolzhsky Research Medical University, Institute of Experimental Oncology and Biomedical Technologies, Nizhny Novgorod, Russian Federation
N.I. Lobachevsky Nizhny Novgorod National Research State University, Russian Federation


Paper #9185 received 24 Oct 2024; revised manuscript received 21 Jan 2025; accepted for publication 19 Feb 2025; published online 23 Mar 2025.

DOI: 10.18287/JBPE25.11.010201

Abstract

Fluorescence Lifetime Imaging Microscopy (FLIM) is an advanced imaging technique that provides quantitative information about molecular interactions and changes in the microenvironment by measuring the emission decay lifetimes of molecules undergoing fluorescence. The integration of FLIM with machine learning holds great promise for advancing biomedical research and clinical diagnostics by enabling precise quantification of cellular processes and microenvironmental changes. As FLIM instrumentation becomes more accessible and analytical methods continue to improve, the technique is poised to have a significant impact across various fields of biology and medicine. This review considers a range of methods of applying Machine Learning to FLIM data processing.

Keywords

deep learning; segmentation; denoising; upscaling

Full Text:

PDF

References


1. R. Datta, T. M. Heaster, J. T. Sharick, A. A. Gillette, and M. C. Skala, “Fluorescence lifetime imaging microscopy: fundamentals and advances in instrumentation, analysis, and applications,” Journal of Biomedical Optics 25(07), 1 (2020).

2. P. A. Bonilla, R. Shrestha, “FLIM-FRET Protein-Protein Interaction Assay,” in KRAS, A. G. Stephen and D. Esposito (Eds.), 2797, Springer US, 261–269 (2024).

3. P. P. Provenzano, K. W. Eliceiri, and P. J. Keely, “Multiphoton microscopy and fluorescence lifetime imaging microscopy (FLIM) to monitor metastasis and the tumor microenvironment,” Clinical & Experimental Metastasis 26(4), 357–370 (2009).

4. S. Karpf, C. T. Riche, D. Di Carlo, A. Goel, W. A. Zeiger, A. Suresh, C. Portera-Cailliau, and B. Jalali, “Spectro-temporal encoded multiphoton microscopy and fluorescence lifetime imaging at kilohertz frame-rates,” Nature Communications 11(1), 2062 (2020).

5. M. Evers, N. Salma, S. Osseiran, M. Casper, R. Birngruber, C. L. Evans, and D. Manstein, “Enhanced quantification of metabolic activity for individual adipocytes by label-free FLIM,” Scientific Reports 8(1), 8757 (2018).

6. T. Sanchez, T. Wang, M. V. Pedro, M. Zhang, E. Esencan, D. Sakkas, D. Needleman, and E. Seli, “Metabolic imaging with the use of fluorescence lifetime imaging microscopy (FLIM) accurately detects mitochondrial dysfunction in mouse oocytes,” Fertility and Sterility 110(7), 1387–1397 (2018).

7. W. Becker, A. Bergmann, L. Braun, and Becker, “Metabolic Imaging with the DCS-120 Confocal FLIM System : Simultaneous FLIM of NAD ( P ) H and FAD,” Application note (2018).

8. A. D. Komarova, S. D. Sinyushkina, I. D. Shchechkin, I. N. Druzhkova, S. A. Smirnova, V. M. Terekhov, A. M. Mozherov, N. I. Ignatova, E. E. Nikonova, E. A. Shirshin, L. E. Shimolina, S. V. Gamayunov, V. I. Shcheslavskiy, and M. V. Shirmanova, “Insights into metabolic heterogeneity of colorectal cancer gained from fluorescence lifetime imaging,” eLife 13, RP94438 (2024).

9. Y. Ouyang, Y. Liu, Z. M. Wang, Z. Liu, and M. Wu, “FLIM as a Promising Tool for Cancer Diagnosis and Treatment Monitoring,” Nano-Micro Letters 13(1), 133 (2021).

10. B. W. Weyers, M. Marsden, T. Sun, J. Bec, A. F. Bewley, R. F. Gandour-Edwards, M. G. Moore, D. G. Farwell, and L. Marcu, “Fluorescence lifetime imaging for intraoperative cancer delineation in transoral robotic surgery,” Translational Biophotonics 1(1–2), e201900017 (2019).

11. J. T. Smith, R. Yao, N. Sinsuebphon, A. Rudkouskaya, N. Un, J. Mazurkiewicz, M. Barroso, P. Yan, and X. Intes, “Fast fit-free analysis of fluorescence lifetime imaging via deep learning,” Proceedings of the National Academy of Sciences 116(48), 24019–24030 (2019).

12. A. Buades, B. Coll, and J.-M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Modeling and Simulation 4 (2), 490–530 (2005).

13. A. Shah, J. I. Bangash, A. W. Khan, I. Ahmed, A. Khan, A. Khan, and A. Khan, “Comparative analysis of median filter and its variants for removal of impulse noise from gray scale images,” Journal of King Saud University-Computer and Information Sciences 34(3), 505–519 (2022).

14. W. Zhou, Y. Miura, “Denoising method using a moving-average filter for the moving-object recognition of low-illuminance video-image,” in 2016 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), IEEE (2016).

15. D. Bhonsle, V. Chandra, and G. R. Sinha, “Medical Image Denoising Using Bilateral Filter,” International Journal of Image, Graphics and Signal Processing 4(6), 36–43 (2012).

16. S. Shrestha, “Image Denoising using New Adaptive Based Median Filters,” Signal & Image Processing: An International Journal (SIPIJ) 5(4), 1−13 (2014).

17. Z. Shan, J. Yang, M. A. F. Sanjuán, C. Wu, and H. Liu, “A novel adaptive moving average method for signal denoising in strong noise background,”The European Physical Journal Plus 137(1), 50 (2022).

18. R. Tian, G. Sun, X. Liu, and B. Zheng, “Sobel Edge Detection Based on Weighted Nuclear Norm Minimization Image Denoising,” Electronics 10(6), 655 (2021).

19. B. Jakisc, R. Ivokovic, B. Gara, M. Petrovic, and P. Spalevic, “Analysis of different influence of compression algorithm on the image filtered Laplacian, Prewitt and Sobel operator,” International Journal of Darshan Institute on Engineering Research & Emerging Technologies 2(1), 68–76 (2013).

20. W. Piao, Y. Yuan, and H. Lin, “A Digital Image Denoising Algorithm Based on Gaussian Filtering and Bilateral Filtering,” ITM Web of Conferences 17, 01006 (2018).

21. S. Paris, S. W. Hasinoff, and J. Kautz, “Local Laplacian filters: edge-aware image processing with a Laplacian pyramid,” Communications of the ACM 58(3), 81–91 (2015).

22. Y. Raphiphan, S. Wattanakaroon, and S. Khetkeeree, “Adaptive High Boost Filtering for Increasing Grayscale and Color Image Details,” in 2020 International Conference on Power, Energy and Innovations (ICPEI), IEEE, 69–72 (2020).

23. Z. Shi, Y. Chen, E. Gavves, P. Mettes, and C. G. M. Snoek, “Unsharp Mask Guided Filtering,” IEEE Transactions on Image Processing 30, 7472–7485 (2021).

24. M. A. Digman, V. R. Caiolfa, M. Zamai, and E. Gratton, “The phasor approach to fluorescence lifetime imaging analysis,” Biophysical Journal 94(2), L14–L16 (2008).

25. V. Mannam, Y. Zhang, X. Yuan, T. Hato, P. C. Dagher, E. L. Nichols, C. J. Smith, K. W. Dunn, and S. Howard, “Convolutional neural network denoising in fluorescence lifetime imaging microscopy (FLIM),” in Multiphoton Microscopy in the Biomedical Sciences XXI, A. Periasamy, P. T. So, and K. König (Eds.), SPIE Proceedings 11648, 116481C (2021).

26. B. Fu, X. Zhao, Y. Li, X. Wang, and Y. Ren, “A convolutional neural networks denoising approach for salt and pepper noise,” Multimedia Tools and Applications 78, 30707–30721 (2019).

27. P. C. Tripathi, S. Bag, “CNN-DMRI: A Convolutional Neural Network for Denoising of Magnetic Resonance Images,” Pattern Recognition Letters 135, 57–63 (2020).

28. Y. Kim, J. W. Soh, G. Y. Park, and N. I. Cho, “Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3482–3492 (2020).

29. V. Kapitany, A. Fatima, V. Zickus, J. Whitelaw, E. McGhee, R. Insall, L. Machesky, and D. Faccio, “Single-sample image-fusion upsampling of fluorescence lifetime images,” Science Advances 10(21), eadn0139 (2024).

30. C. Callenberg, A. Lyons, D. D. Brok, A. Fatima, A. Turpin, V. Zickus, L. Machesky, J. Whitelaw, D. Faccio, and M. B. Hullin, “Super-resolution time-resolved imaging using computational sensor fusion,” Scientific Reports 11, 1689 (2021).

31. B. Huang, J. Li, B. Yao, Z. Yang, E. Y. Lam, J. Zhang, W. Yan, and J. Qu, “Enhancing image resolution of confocal fluorescence microscopy with deep learning,” PhotoniX 4, 2 (2023).

32. D. Xiao, Z. Zang, W. Xie, N. Sapermsap, Y. Chen, and D. D. Uei Li, “Spatial resolution improved fluorescence lifetime imaging via deep learning,” Optics Express 30(7), 11479 (2022).

33. M. Adhikari, R. Houhou, J. Hniopek, and T. Bocklitz, “Review of Fluorescence Lifetime Imaging Microscopy (FLIM) Data Analysis Using Machine Learning,” Journal of Experimental and Theoretical Analyses 1(1), 44–63 (2023).

34. W. Shi, D. E. S. Koo, M. Kitano, H. J. Chiang, L. A. Trinh, G. Turcatel, B. Steventon, C. Arnesano, D. Warburton, S. E. Fraser, and F. Cutrale, “Pre-processing visualization of hyperspectral fluorescent data with Spectrally Encoded Enhanced Representations,” Nature Communications 11(1), 726 (2020).

35. Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: A Nested U-Net Architecture for Medical Image Segmentation,” in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, D. Stoyanov, Z. Taylor, G. Carneiro, T. Syeda-Mahmood, A. Martel, L. Maier-Hein, J. M. R. S. Tavares, A. Bradley, J. P. Papa, V. Belagiannis, J. C. Nascimento, Z. Lu, S. Conjeti, M. Moradi, H. Greenspan, and A. Madabhushi (Eds.), Springer International Publishing, 11045, 3–11 (2018).

36. W. Tingxi, T. Binbin, L. Yu, P. Ting, D. Yu, C. Yuping, and Z. Shanshan, “Review of research on the instance segmentation of cell images,” Computer Methods and Programs in Biomedicine 227, 107211 (2022).

37. S. M. A. Burney, H. Tariq, “K-Means Cluster Analysis for Image Segmentation,” International Journal of Computer Applications 96(4), 1–8 (2014).

38. W. Gao, “Random Forest in Image Segmentation,” Machine Learning Theory and Practice 1, 11–19 (2020).

39. Z. Kato, “Markov Random Fields in Image Segmentation,” Foundations and Trends® in Signal Processing 5(1–2), 1–155 (2011).

40. B. Karthicsonia, M. Vanitha, “Edge Based Segmentation in Medical Images,” International Journal of Engineering and Advanced Technology 9(1), 449–451 (2019).

41. O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi (Eds.), Springer International Publishing, 9351, 234–241 (2015).

42. D. Bannon, E. Moen, M. Schwartz, E. Borba, T. Kudo, N. Greenwald, V. Vijayakumar, B. Chang, E. Pao, E. Osterman, W. Graf, and D. Van Valen, “DeepCell Kiosk: scaling deep learning–enabled cellular image analysis with Kubernetes,” Nature Methods18(1), 43–45 (2021).

43. A. E. Carpenter, T. R. Jones, M. R. Lamprecht, C. Clarke, I. H. Kang, O. Friman, D. A. Guertin, J. H. Chang, R. A. Lindquist, J. Moffat, P. Golland, and D. M. Sabatini, “CellProfiler: image analysis software for identifying and quantifying cell phenotypes,” Genome Biology 7(10), R100 (2006).

44. K. He, X. Zhang, S. Ren, and J. Sun, “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence 37(9), 1904–1916 (2015).

45. Y. Wang, W. Wang, D. Liu, W. Hou, T. Zhou, and Z. Ji, “GeneSegNet: a deep learning framework for cell segmentation by integrating gene expression and imaging,” Genome Biology 24(1), 235 (2023).

46. J. Wu, R. Fu, H. Fang, Y. Zhang, Y. Yang, H. Xiong, H. Liu, and Y. Xu, “MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model,” in Medical Imaging with Deep Learning, Proceedings of Machine Learning Research 227, 1623–1639 (2024).

47. J. Ma, F. Li, and B. Wang, “U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation,” arXiv preprint arXiv:2401.04722 (2024).

48. J. Ruan, S. Xiang, “VM-UNet: Vision Mamba UNet for Medical Image Segmentation,” arXiv preprint arXiv:2402.02491 (2024).

49. Z. Xing, T. Ye, Y. Yang, G. Liu, and L. Zhu, “SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Seg-mentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer Nature Switzerland, Cham, 578–588 (2024).

50. Z. Lu, C. She, W. Wang, and Q. Huang, “LM-Net: A light-weight and multi-scale network for medical image segmentation,” Computers in Biology and Medicine 168, 107717 (2024).

51. C. You, W. Dai, Y. Min, F. Liu, D. Clifton, S. K. Zhou, L. Staib, and J. Duncan, “Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective,” Advances in neural information processing systems, 36, 9984–10021 (2023).

52. D. Müller, I. Soto-Rey, and F. Kramer, “Towards a guideline for evaluation metrics in medical image segmentation,” BMC Research Notes 15(1), 210 (2022).

53. H. Khastavaneh, H. Ebrahimpour-komleh, “Automated Segmentation of Abnormal Tissues in Medical Images,” Journal of Biomedical Physics & Engineering 11(4), 415–424 (2019).

54. P. Qin, J. Chen, J. Zeng, R. Chai, and L. Wang, “Large-scale tissue histopathology image segmentation based on feature pyramid,” EURASIP Journal on Image and Video Processing 2018, 75 (2018).

55. A. K. Kuchibhotla, L. D. Brown, A. Buja, and J. Cai, “All of Linear Regression,” arXiv preprint arXiv:1910.06386 (2019).

56. H.-H. Huang, Q. He, “Nonlinear regression analysis,” in International Encyclopedia of Education, 4th Ed., Elsevier, 558–567 (2023).

57. R. S. Turley, “Polynomial Fitting”, Faculty Publication, 2237, 1 Jan 2018 (accessed 22 October 2024). [https://scholarsarchive.byu.edu/facpub/2237].

58. D. C. Vargas, E. J. Rodríguez, M. Flickner, and J. L. C. Sanz, “Splines and Spline Fitting Revisited,” in Image Technology: Advances in Image Processing, Multimedia and Machine Vision, J. L. C. Sanz (Ed.), Springer Berlin Heidelberg, 405–437 (1996).

59. M. Li, L. D. Li, “A Novel Method of Curve Fitting Based on Optimized Extreme Learning Machine,” Applied Artificial Intelligence 34(12), 849–865 (2020).

60. V. P, D. R. Rao, “Machine Learning Techniques on Multidimensional Curve Fitting Data Based on R- Square and Chi-Square Methods,” International Journal of Electrical and Computer Engineering 6(3), 974–979 (2016).

61. S. S. Sidharth, “Curve Fitting Simplified: Exploring the Intuitive Features of CurvPy,” arXiv preprint arXiv:2307.06377 (2023).

62. N. Tumbarkova, N. Deliiski, “Applying the software package TableCurve 2D for computation of processing air medium temperature during freezing in a freezer and defrosting of logs,” Innovations in Woodworking Industry and Engeneering Design 11, 69–76 (2017).

63. A. Dakkak, T. Wickham-Jones, and W.-m. Hwu, “The design and implementation of the wolfram language compiler,” Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization, San Diego, CA, USA, 212–228 (2020).

64. T. Cepowski, ndCurveMaster (8.7), SigmaLab [Computer Software], (2025) (accessed 22 October 2024). [https://www.ndcurvemaster.com].

65. D. Gao, P. R. Barber, J. V. Chacko, Md. A. Kader Sagar, C. T. Rueden, A. R. Grislis, M. C. Hiner, and K. W. Eliceiri, “FLIMJ: An open-source ImageJ toolkit for fluorescence lifetime image data analysis,” PLoS ONE 15(12), e0238327 (2020).

66. D. Nečas, P. Klapetek, “Gwyddion: an open-source software for SPM data analysis,” Central European Journal of Physics 10, 181–188 (2011).

67. L. Hu, N. Wang, J. D. Bryant, L. Liu, L. Xie, A. P. West, and A. J. Walsh, “Machine learning prediction of cancer cell metabolism from autofluorescence lifetime images,” bioRxiv [preprint], 2022.2012.16.520759 (2022).

68. N. I. Nizam, V. Pandey, I. Erbas, J. T. Smith, and X. Intes, “A Novel Technique for Fluorescence Lifetime Tomography,” bioRxiv [preprint], 2024.09.19.613888 (2024).

69. D. Xiao, N. Sapermsap, Y. Chen, and D. D. U. Li, “Deep learning enhanced fast fluorescence lifetime imaging with a few photons,” Optica 10(7), 944 (2023).

70. Z. Zang, D. Xiao, Q. Wang, Z. Jiao, Y. Chen, and D. D. U. Li, “Compact and robust deep learning architecture for fluorescence lifetime imaging and FPGA implementation,” Methods and Applications in Fluorescence 11(2), 025002 (2023).

71. D. Xiao, Y. Chen, and D. D.-U. Li, “One-Dimensional Deep Learning Architecture for Fast Fluorescence Lifetime Imaging,” IEEE Journal of Selected Topics in Quantum Electronics 27(4), 1–10 (2021).

72. Q. Wang, Y. Li, D. Xiao, Z. Zang, Z. Jiao, Y. Chen, and D. D. U. Li, “Simple and Robust Deep Learning Approach for Fast Fluorescence Lifetime Imaging,” Sensors 22(19), 7293 (2022).

73. Y.-I. Chen, Y.-J. Chang, S.-C. Liao, T. D. Nguyen, J. Yang, Y.-A. Kuo, S. Hong, Y.-L. Liu, H. G. Rylander, S. R. Santacruz, T. E. Yankeelov, and H.-C. Yeh, “Deep learning enables rapid and robust analysis of fluorescence lifetime imaging in photon-starved conditions,” bioRxiv [preprint], 2020.2012.02.408195 (2020).

74. F. Wang, S. H. Kim, Y. Zhao, A. Raghuram, A. Veeraraghavan, J. Robinson, and A. H. Hielscher, “High-Speed Time-Domain Diffuse Optical Tomography With a Sensitivity Equation-Based Neural Network,” IEEE Transactions on Computational Imaging 9, 459–474 (2023).

75. M. Fazel, S. Jazani, L. Scipioni, A. Vallmitjana, S. Zhu, E. Gratton, M. A. Digman, and S. Pressé, “Building Fluorescence Lifetime Maps Photon-by-Photon by Leveraging Spatial Correlations,” ACS Photonics 10(10), 3558–3569 (2023).

76. D.-M. Koh, N. Papanikolaou, U. Bick, R. Illing, C. E. Kahn, J. Kalpathi-Cramer, C. Matos, L. Martí-Bonmatí, A. Miles, S. K. Mun, S. Napel, A. Rockall, E. Sala, N. Strickland, and F. Prior, “Artificial intelligence and machine learning in cancer imaging,” Communications Medicine 2(1), 133 (2022).

77. V. Mannam, Y. Zhang, X. Yuan, C. Ravasio, and S. S. Howard, “Machine learning for faster and smarter fluorescence lifetime imaging microscopy,” Journal of Physics: Photonics 2(4), 042005 (2020).

78. N. Thuerey, P. Holl, M. Mueller, P. Schnell, F. Trost, and K. Um, “Physics-based Deep Learning,” arXiv preprint arXiv:2109.05237 (2021).

79. A. Arzani, L. Yuan, P. Newell, and B. Wang, “Interpreting and generalizing deep learning in physics-based problems with functional linear models,” Engineering with Computers 41(1), 135–157 (2025).

80. Y. Sun, Q. Sun, and K. Qin, “Physics-Based Deep Learning for Flow Problems,” Energies 14(22), 7760 (2021).

81. D. Zhang, Y. Chen, and S. Chen, “Filtered partial differential equations: a robust surrogate constraint in physics-informed deep learning framework,” Journal of Fluid Mechanics 999, A40 (2024).

82. S. Kaltenbach, P.-S. Koutsourelakis, “Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems,” Journal of Computational Physics 419, 109673 (2020).




Bookmark and Share


Сontact

34 Moskovskoe shosse, Samara, 443086, Russian Federation
Email: j-bpe@ssau.ru
Phone: +7-846-267-4550
© 2014-2025 J-BPE