近日，美国杜克大学的Roarke Horstmeyer课题组与美国Ramona Optics公司的研究人员合作，提出了一种高速的并行计算3D视频显微镜技术，可用于捕获自由移动生物体的高速3D地形视频，每秒可达数十亿像素。相关成果已于2023年3月20日在国际权威学术期刊《自然—光子学》上发表。
该研究团队提出了一种名为3D-RAPID的计算显微镜，该显微镜基于54个相机的同步阵列，可以在一个135平方厘米的区域内捕获高速的3D地形视频，达到每秒230帧的速度，时空吞吐量超过5亿像素每秒。3D-RAPID采用一种3D重建算法，对于每个同步快照，将所有54个图像融合成一个组合体，其中包括一个协同注册的3D高度图。该算法采用自监督三维重建算法训练神经网络，利用立体重叠冗余和光线传播物理作为唯一的监督机制，将原始光度图像映射到3D地形图。因此，重建过程对泛化误差具有鲁棒性，并可从任意大小相机阵列扩展到任意长视频。该研究利用数个自由行动生物的收集展示了3D-RAPID的广泛适用性，包括蚂蚁、果蝇和斑马鱼幼虫。
据悉，具有高速和空间分辨率的广角显微镜尤其适用于研究自由移动生物的行为，但同时优化所有这些属性的光学仪器的设计具有挑战性。现有技术通常需要获取序列图像快照来观察大面积或测量3D信息，从而牺牲了速度和吞吐量。
附：英文原文
Title: Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second
Author: Zhou, Kevin C., Harfouche, Mark, Cooke, Colin L., Park, Jaehee, Konda, Pavan C., Kreiss, Lucas, Kim, Kanghyun, Jnsson, Joakim, Doman, Thomas, Reamey, Paul, Saliu, Veton, Cook, Clare B., Zheng, Maxwell, Bechtel, John P., Bgue, Aurlien, McCarroll, Matthew, Bagwell, Jennifer, Horstmeyer, Gregor, Bagnat, Michel, Horstmeyer, Roarke
Issue&Volume: 2023-03-20
Abstract: Wide-field-of-view microscopy that can resolve three-dimensional (3D) information at high speed and spatial resolution is particularly desirable for studying the behaviour of freely moving organisms. However, it is challenging to design an optical instrument that optimizes all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising speed and throughput. Here we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135 cm2 area, achieving up to 230 frames per second at a spatiotemporal throughput exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with several collections of freely behaving organisms: ants, fruit flies and zebrafish larvae.
DOI: 10.1038/s41566-023-01171-7
Source: https://www.nature.com/articles/s41566-023-01171-7