Hyperspectral Camera – Attachment for Microscopy

. The paper presents a microscope equipped with a hyperspectral ocular nozzle with a phase diffraction grating, which is used as a dispersing element. The spectral range of the device is 400 – 1000 nm, with the spectral resolution of 5 nm. The paper describes a microscope software that is used to solve problems of spectral analysis. © 2021 Journal of Biomedical Photonics & Engineering.


Introduction
Hyperspectral equipment that can distinguish a large number of spectral channels, in contrast to conventional three-color imaging, when using hyperspectral image processing, allows you to see subtle color differences in the visible range of optical radiation, as well as wavelengths invisible to the human eye in the nearinfrared range [1]. This ability makes it possible to distinguish a number of chemical compounds of the underlying surface in remote sensing of the Earth, and is also increasingly used in biology and medicine [2]. In addition to the classical search signatures, in biology and medicine, a number of compounds with the use of illumination of certain wavelengths can cause the effect of photoluminescence, fluorescence, Raman scattering, etc., which allows you to use the hyperspectral equipment in criminology [3], materials science [4], mineralogy [5]. Such studies can be carried out only in the laboratory, and the use of a combination of microscopy and hyperspectral equipment allows you to see the smallest particles that cannot be seen by the eye [6].
A microscope with hyperspectral imaging can use narrow-band radiation of an object, or radiation transmitted through an optical monochromator [7]. Hyperspectral imaging systems based on acousto-optic modulators are known [8]. Most often, hyperspectrometers use an optical scheme of the scanning type, similar to that described in Ref. [9]. Using a microscope-spectrometer, it is possible to obtain the spectrum of individual pixels in the wavelength range from 400 nm to 1000 nm with a spectral resolution of 5 nm [10][11], depending on the dispersing element used and the resolution of the recording matrix.
We analyze well-known microscopes equipped with spectrum recording systems, and some of their shortcomings become obvious. One of them is the complexity and high cost of the optical or electronic part of the devices. For example, the LOMO MSFU-K microscope-spectrophotometer uses a photoelectron multiplier PMF R928 from Hamamatsu (Japan). Many devices do not allow you to get spectral data in each individual pixel of the image. Most instruments do not have an option of selecting a microscope. The complex can only be installed on certain microscopes.
In this regard, in this paper, the task is to develop and study a hyperspectrometer adapted to work as part of a microscope in the form of an ocular nozzle. Scanning can be carried out by moving the movable table of the microscope. This concept makes it possible to equip almost any microscope for biological research with a hyperspectral imaging system.

Optical scheme of a hyperspectral camera to a microscope
The optical unit of a hyperspectral camera for a microscope is made according to a fairly well-known scheme of a slit hyperspectrometer based on a grating dispersion element. Most schemes have a prism or lattice dispersion element. Since the grid has a more compact size, it was decided to use it. A standard microscope lens with a magnification of 10×, 20×, 40×, 100× is used as an input lens. This set of magnifications is most often used in biomedical researches [12][13]. The image from these lenses is built on a slit aperture having a slit width of about 10 µm. Next, a telescopic system based on two lenses builds an image of the gap on the matrix. A binary lattice with a period of 6 µm is installed inside the telescope. Such grating provides about 250 channels in the focal plane in the spectral range of 400-1000 nm.
The design elements of the hyperspectral nozzle were developed according to the optical scheme (see Fig. 1) using three-dimensional modeling of 3D printing parts made of plastic on a 3D lapper. An optical hyperspectrometer consists of the following elements: 1) light source; 2) motorized table; 3) focusing lens; 4) entrance slit; 5) collimator; 6) dispersion grid; 7) focusing lens; 8) optical sensor, L -wavelength; a, b -2D detector arrays.
For illumination, both the built-in halogen lamp of the microscope and the external one with an upper position can be used. In this case, the generated spectrum on the sensor is recorded as the transmission spectrum and the reflection spectrum, respectively.

Hyperspectral survey control system as part of a microscope
The fact that the camera is based on slit scanning will be used as the basis of the shooting control system for obtaining hyperspectral images (hypercubes). In our case, the hyperspectral camera remains stationary, while relative to it, the movement is carried out by scanning a sample located on a movable microscope table. The microscope has precise mechanical movement elements. Usually, the operator manually rotates these handles and visually observes the area on the slide table. For automated control of these movements, it is possible to install a stepper motor. Thus, with the help of a special control system, it is possible to install the step and speed of movement from the control unit. The control unit of the complex is based on a singleboard microcomputer with software for shooting and an optical sensor of the camera synchronized with the shooting and a motion control system for the scanning table. The control is carried out by sending control signals to the stepper motor driver. The camera registers the signal during the time allotted for the accumulation of light by the optical sensor, at the end of which the next signal for movement is formed. At the end of the movement, the next frame begins to be shot synchronously. The time of accumulation of the signal from the matrix, the number of shooting frames, etc. shooting parameters are set using the developed user interface. The structure of the complex is shown in Fig. 2. In order to control the shooting, a compact, sufficiently powerful computer is needed.
The complex uses a Raspberry Pi 4 microcomputer due to the large number of GPIO pins, a large set of USB and HDMI interfaces, the presence of Wi-Fi and Bluetooth. The manufacturer is "Raspberry Pi Foundation", England.
Also, the A3967SLB stepper motor driver (Allegro MicroSystems, USA) is used in the complex. The complex uses a monochrome camera "Basler acA 1920-40 um" (BASLER Fashion GmbH, Germany). The monochrome camera was chosen because of its greater light sensitivity and it can capture a larger range, unlike a color camera.
The information collected in this way is processed by special software for recognizing objects with similar spectral characteristics.
A stepper motor is attached to the movable microscope substrate. The stepper motor is connected to the movable table of the microscope through a gearbox consisting of two gears. Gear ratio equals to 3. The image of the engine design is shown in Fig. 3. Fig. 3 (a) The image shows a microscope, (b) motor design for moving the microscope table.
The stepper motor driver is controlled by a Raspberry Pi 4 single-board computer via GPIO ports [14].
The Python program controls the signals "1" and "0" on the GPIO ports using the python library RPi.GPIO.
The presented complex uses the stepper motor driver "A3967LBT".
The signals are fed to the inputs of the A3967LBT driver and, thus, the stepper motor is controlled [15]. The presented stepper motor driver allows you to work with step crushing modes in the range from full-step to 1/8 step, which on the microscope under study provides an accuracy of movement from 32.4 µm per step in fullstep mode, to 4.05 µm in 1/8 step mode.
The Basler camera is connected to the computer via a USB 3.0 port. The camera parameters were changed using the Python SDK using the pypylon library. Shooting settings: shutter speed, if using a color camera, the white balance value [16].
To set the settings for shooting, you need an interface (Fig. 4). To create it, we used the PyQt5 program. This may be necessary to adjust the focus. The "Start over" button clears the fields. The "START" button starts shooting.
Shooting can be controlled remotely via Wi-Fi, via a separate computer or smartphone using the VNC viewer remote access program.

Results and Discussion
We tested the resulting scanning and hyperspectral shooting control system, carried out the necessary tests, calibrations, scanning speed settings, exposure, etc. We also determined the optical characteristics of the hyperspectral attachment to the microscope. A hyperspectral image (hypercube) is obtained by assembling frames from a sequence taken at each step of the stepper motor using specially developed software.
When testing the complex (see Fig. 5), a multicolored paper printed on an inkjet printer was photographed (see Fig. 5a).
The image of a single frame fixed on the CCD matrix of the hyperspectral camera, illuminated by a halogen lamp (Fig. 5b). In order to carry out the correct color synthesis for this hyperspectral camera, laser calibration was carried out with a wavelength of 532 nm. The view of the image of the spectral decomposition of the image in the slot from the object that is additionally illuminated by the laser is shown in Fig. 5c. From this image, it is also possible to estimate the spectral resolution of the hyperspectrometer, which, as can be seen, allows distinguishing about 120 spectral channels in the range of 400-1000 nm. In this case, it is easy to get that the spectral resolution in this case becomes about 5 nm.
The spectrum data was obtained and converted into a hypercube using special software. The three layers of the hypercube, which are responsible for the blue, green, and red components, were used to create the RGB image (see Fig. 5d).
Some "stripes" were skipped due to the 1/2 shooting mode, and therefore the proportions of the length and width of the image in Fig. 5d are equal to each other. You can set a smaller step (shooting mode 1/8), then all the proportions will be observed.
We can observe the spectrum at each point of this image. An example of the spectrum of pixels from the green area of colored paper (Fig. 5b) "rainbow" is shown in Fig. 6.   7 The result of the classification of the object in the image. The dimensions of the shooting area are 4.5 mm wide and 14 mm long. Some "stripes" were skipped due to the 1/2 shooting mode, and therefore the proportions of the length and width of the image in Fig. 5d are equal to each other. You can set a smaller step (shooting mode 1/8), then all the proportions will be observed. Fig. 6 shows the spectral analysis of the "rainbow" image -the green pixel is selected.
In this case, the ENVI program was chosen for spectral analysis. ENVI is a software for image processing and analysis. It is used by image analysts, GIS specialists and scientists to extract timely, reliable and accurate information from geospatial images.
The developer of the ENVI program "L3 Harris Technologies" is an American technology company, a defense contractor and an information technology service provider.
For example, we also carried out the classification of color bands with threshold processing by the well-known Euclidean distance method [17]. The result of selecting one strip from the image of colored paper (Fig. 5) is shown in Fig. 7.
For this purpose, a special software has been developed, with the help of which it is possible to quickly find objects of interest to research with a spectrum similar to the spectrum specified as the search standard. The spectrum can be loaded as a file or specified as an object in the image. We also obtained medical hyperspectral images of histological specimens for studying of rat's spleen from a set of micropreparations according to the catalog [18], produced by the company "Retinoids", produced in compliance with ethical requirements and international recommendations for conducting biomedical research using animals. An example of an image obtained in a microscope with an 40× magnification and its colorsynthesized image is shown in Fig. 8. Fig. 8a shows an image of a histological sample using an 10× lens and a ToupCam U3CMOS05100KPA microscope camera. The field of view is 0.9 × 0.9 mm. Fig. 8b shows the image obtained by scanning in the 1/8 step mode, the size of the scanning area is 0.4 mm high and 1.2 mm wide. The proportions of the image are not observed due to the limited scanning step. Fig. 8c also shows an example of the spectrum in the cross-hair marker (Fig. 8b) area in the image. The hyperspectral additive for the microscope gives an additional magnification of about 2×.
In the future, we plan to continue our research to obtain methods for classifying objects in the image of medical samples. In this work, the task of determining the accuracy of the classification of anomalies associated with diseases was not set. To conduct this kind of research, we plan to conduct a survey of a large set of various biological preparations. In this work, the task of determining the accuracy of the classification of anomalies associated with diseases was not set. To conduct this kind of research, we plan to conduct a survey of a large set of various biological preparations. Previously, studies of hyperspectral imaging were conducted in Refs. [19,20] to determine the types of human skin neoplasms. These studies have shown a fairly high accuracy in determining pathologies. Also, a hyperspectral camera based on an acousto-optic monochromator was described in Ref. [21]. This design J of Biomedical Photonics & Eng 7 (3) 30 Sep 2021 © J-BPE 030405 -7 eliminates the need for additional scanning systems. At the same time, the production of a control unit and shooting for a hyperspectral camera in this case turns out to be more expensive than in the device presented in this paper. Also, the used wavelength range was limited to the range of 400-750 nm, while in our study, the spectral wavelengths range is limited only by the sensitivity range of the sensor. The presented control system and an optical camera can be equipped with almost any microscope, which is almost 10 times cheaper in comparison with expensive analogs of microscopes.

Conclusion
The article presents a hardware and software complex for controlling hyperspectral photography, as well as software for spectral analysis. During the research, the optical characteristics of the complex were determined, software was developed for controlling the shooting and obtaining primary spatial-spectral data that are then converted into hyperspectral images by processing, which allows, as we hope, to obtain detailed and highly reliable data for establishing an accurate diagnosis from histological samples. During the study, the parameters of the shooting speed were obtained depending on the magnification of the lens.
Advantages of this complex: 1) simplicity of the scheme; 2) flexibility in the selection of specific components; 3) remote and automatic shooting control; 4) obtaining individual spectra in individual pixels of the image.
A software for spectral analysis was also developed.

Disclosures
All authors declare that there is no conflict of interests in this paper.