Individual Behavior Recognition in Laboratory Rats: a Comparative Analysis of Computer Vision Methods

Dmitrii Krasnov; Aleksey Shmonin; Maxim Volynsky; Nikita Margaryants; Alexandr Gusev; Maria Maltseva; Elena Korotkova

doi:10.18287/JBPE26.12.010306

Authors

Dmitrii Krasnov ITMO University, Saint-Petersburg, Russian Federation http://orcid.org/0009-0003-9469-6618 (unauthenticated)
Aleksey Shmonin Federal Scientific and Clinical Center for Resuscitation and Rehabilitation Pavlov First Saint Petersburg State Medical University, Moscow region, Solnechnogorsk, Russian Federation http://orcid.org/0000-0003-2232-4332 (unauthenticated)
Maxim Volynsky ITMO University, Saint-Petersburg, Russian Federation http://orcid.org/0000-0003-1679-4450 (unauthenticated)
Nikita Margaryants ITMO University, Saint-Petersburg, Russian Federation Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation http://orcid.org/0000-0002-3767-3618 (unauthenticated)
Alexandr Gusev ITMO University, Saint-Petersburg, Russian Federation http://orcid.org/0000-0003-2437-8537 (unauthenticated)
Maria Maltseva Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation http://orcid.org/0000-0002-1698-4038 (unauthenticated)
Elena Korotkova Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation http://orcid.org/0009-0001-1213-3287 (unauthenticated)

DOI:

https://doi.org/10.18287/JBPE26.12.010306

Keywords:

сomputer vision, deep learning, machine learning, rat behavior, rodent behavior, action recognition, keypoints, convolutional neural network, Wistar rat

Abstract

Computer-vision methods have been applied to automated behavior recognition in laboratory rodents. Namely, we explored the possibility of classifying certain behavior classes from still images; compared keypoint-based methods with approaches based on visual embeddings; studied the feasibility of transferring models between rats and mice; and evaluated the relevance of the number and the accuracy of the detected keypoints. We collected a dataset of a freely moving Wistar rat to train six pose-based classifiers with Long Short-Term Memory (LSTM) using six sets of keypoints produced by two detectors and four Convolutional Neural Network (CNN) classifiers using images and optical flow frames. The results demonstrated the highest mean average precision (mAP) of 65.7% for the CNN-based methods and 34.3% for the LSTM classifiers, the feasibility of recognizing visually distinct classes (rearing and body grooming) from still images, and the applicability of a keypoint detector trained on mice. The results of this study can be applied to the design of a computer vision system for automating long-term monitoring of laboratory rodent behavior

Author Biographies

Dmitrii Krasnov, ITMO University, Saint-Petersburg, Russian Federation

PhD student, Technical Vision Lab, ITMO University
Aleksey Shmonin, Federal Scientific and Clinical Center for Resuscitation and Rehabilitation Pavlov First Saint Petersburg State Medical University, Moscow region, Solnechnogorsk, Russian Federation

MD, PhD, Professor, Department of Psychology, Pedagogy and Rehabilitation Technologies, Federal Scientific and Clinical Center for Resuscitation and Rehabilitation, Associate Professor, Department of Pathophysiology with a course in Clinical Pathophysiology, Head of the laboratory, Laboratory of Pathophysiology, Pavlov First Saint Petersburg State Medical University
Maxim Volynsky, ITMO University, Saint-Petersburg, Russian Federation

PhD, Director, Associate Professor, Technical Vision Lab, ITMO University
Nikita Margaryants, ITMO University, Saint-Petersburg, Russian Federation Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation

PhD, Associate Professor, Technical Vision Lab, ITMO University, Associate Professor, Department of Physics, Math, and Informatics, Pavlov First Saint Petersburg State Medical University
Alexandr Gusev, ITMO University, Saint-Petersburg, Russian Federation

PhD, Lead Engineer, Technical Vision Lab, ITMO University
Maria Maltseva, Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation

PhD, Associate Professor, Department of Pedagogy and Psychology, Pavlov First Saint Petersburg State Medical University
Elena Korotkova, Pavlov First Saint Petersburg State Medical University, Saint-Petersburg, Russian Federation

Student, Pavlov First Saint Petersburg State Medical University

References

F. Rotaru, S. I. Bejinariu, H. N. Costin, R. Luca, and C. D. Nita, “Captive animal behavior study by video analysis,” Sensors 23(18), 7928 (2023).

Himanshu, Dharmila, D. Sarkar, and Nutan, “A review of behavioral tests to evaluate different types of anxiety and anti-anxiety effects,” Clinical Psychopharmacology and Neuroscience: The Official Scientific Journal of the Korean College of Neuropsychopharmacology 18(3), 341–351 (2020).

F. Grieco, B. J. Bernstein, B. Biemans, et al., “Measuring behavior in the home cage: Study design, applications, challenges, and perspectives,” Frontiers in Behavioral Neuroscience 15, 735387 (2021).

K. Luxem, J. J. Sun, S. P. Bradley, K. Krishnan, E. Yttri, J. Zimmermann, T. D. Pereira, and M. Laubach, “Open-source tools for behavioral video analysis: Setup, methods, and best practices,” eLife 12, e79305 (2023).

M. Perez and C. Toler-Franklin, “CNN-based action recognition and pose estimation for classifying animal behavior from videos: A survey,” arXiv:2301.06187 (2023).

S. İşık, G. Ünal, “Open-source software for automated rodent behavioral analysis,” Frontiers in Neuroscience 17, 1149027 (2023).

C. J. Gabriel, Z. Zeidler, B. Jin, C. Guo, C. M. Goodpaster, A. Q. Kashay, A. Wu, M. Delaney, J. Cheung, L. E. DiFazio, M. J. Sharpe, D. Aharoni, S. A. Wilke, and L. A. DeNardo, “BehaviorDEPOT is a simple, flexible tool for automated behavioral detection based on markerless pose tracking,” eLife 11, e74314 (2022).

M. Kabra, A. A. Robie, M. Rivera-Alba, S. Branson, and K. Branson, “JAABA: interactive machine learning for automatic annotation of animal behavior,” Nature Methods 10(1), 64–67 (2013).

M. Lorbach, R. Poppe, E. A. Van Dam, L. P. J. J. Noldus, and R. C. Veltkamp, “Automated Recognition of Social Behavior in Rats: The Role of Feature Quality,” in Image Analysis and Processing – ICIAP 2015, V. Murino and E. Puppo (Eds.), Springer International Publishing, 9280, 565–574 (2015).

C. Segalin, J. Williams, T. Karigo, M. Hui, M. Zelikowsky, J. J. Sun, P. Perona, D. J. Anderson, and A. Kennedy, “The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice,” eLife 10, e63720 (2021).

A. I. Hsu, E. A. Yttri, “B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors,” Nature Communications 12, 5188 (2021).

E. A. Van Dam, L. P. J. J. Noldus, and M. A. J. Van Gerven, “Deep learning improves automated rodent behavior recognition within a specific experimental setup,” Journal of Neuroscience Methods 332, 108536 (2020).

J. Nie, L. Yan, X. Wang, and J. Chen, “A Novel 3D Convolutional Neural Network for Action Recognition in Infrared Videos,” in 2021 4th International Conference on Information Communication and Signal Processing (ICICSP), IEEE, 420–424 (2021).

C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional two-stream network fusion for video action recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1933–1941 (2016).

Q. Wang, J. Du, K. Yan, and S. Ding, “Seeing in flowing: Adapting CLIP for action recognition with motion prompts learning,” Proceedings of the 31st ACM International Conference on Multimedia (ACM), 5339–5347 (2023).

Z. Bai, Y. Li, Z. Ru, S. Li, W. Li, and F. Duan, “Rats Behavior Recognition Based on Improved YOLOv8,” in 2024 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, 1082–1086 (2024).

K. M. Hatton-Jones, C. Christie, T. A. Griffith, A. G. Smith, S. Naghipour, K. Robertson, J. S. Russell, J. N. Peart, J. P. Headrick, A. J. Cox, and E. F. Du Toit, “A YOLO based software for automated detection and analysis of rodent behaviour in the open field arena,” Computers in Biology and Medicine 134, 104474 (2021).

N. G. Nguyen, D. Phan, F. R. Lumbanraja, M. R. Faisal, B. Abapihi, B. Purnama, M. K. Delimayanti, K. R. Mahmudah, M. Kubo, and K. Satou, “Applying Deep Learning Models to Mouse Behavior Recognition,” Journal of Biomedical Science and Engineering 12(02), 183–196 (2019).

J. Carreira, A. Zisserman, “Quo Vadis, action recognition? A new model and the Kinetics dataset,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724–4733 (2017).

D. I. Krasnov, M. A. Volynsky, and A. A. Gusev, “Preprocessing of skeletal keypoints trajectories in the task of laboratory animal behavior recording automation,” Scientific and Technical Journal of Information Technologies, Mechanics and Optics 25(2), 295–302 (2025). [in Russian]

D. I. Krasnov, M. A. Volynsky, and A. A. Gusev, “State estimation of an agent with stochastic dynamics using recurrent filters,” Informatics and Automation 24(5), 1355–1378 (2025). [in Russian]

N. L. Goodwin, J. J. Choong, S. Hwang, K. Pitts, L. Bloom, A. Islam, Y. Y. Zhang, E. R. Szelenyi, X. Tong, E. L. Newman, K. Miczek, H. R. Wright, R. J. McLaughlin, Z. C. Norville, N. Eshel, M. Heshmati, S. R. O. Nilsson, and S. A. Golden, “Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience,” Nature Neuroscience 27(7), 1411–1424 (2024).

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778 (2016).

K. Hara, H. Kataoka, and Y. Satoh, “Learning spatio-temporal features with 3D residual networks for action recognition,” Proceedings of the IEEE International Conference on Computer Vision Workshop (ICCVW), 3154–3160 (2017).

G. Farnebäck, “Two-Frame Motion Estimation Based on Polynomial Expansion,” in Image Analysis, J. Bigun and T. Gustavsson (Eds.), Springer Berlin Heidelberg, 2749, 363–370 (2003).

C. Lyu, W. Zhang, H. Huang, Y. Zhou, Y. Wang, Y. Liu, S. Zhang, and K. Chen, “RTMDet: An empirical study of designing real-time object detectors,” arXiv:2212.07784 (2022).

J. Huang, Z. Zhu, F. Guo, and G. Huang, “The devil is in the details: Delving into unbiased data processing for human pose estimation,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 5699–5708 (2020).