Amphibious and panoramic artificial vision inspired by eyes of fiddler crab

Inspired by the compound eye of the fiddler crab eye in nature, we have developed a novel panoramic and amphibious imaging device by integrating a flat and graded-refractive-index micro-lens array with a comb-shape silicon photodiode array.
Amphibious and panoramic artificial vision inspired by eyes of fiddler crab

 With the advent of novel industrial applications of cameras, such as autonomous vehicles, drones, and virtual/augmented reality (VR/AR), the high-performance, miniaturized, and wide field-of-view (FoV) imaging device has become a key technological asset. In particular, an advanced camera with a compact form factor, a panoramic visual field (almost 360° FoV), and all-weather compatibility enables to envision a variety of next-generation imaging applications. However, the commercial cameras for 360° vision cannot suppress optical aberrations including image distortion due to the complicated configuration of multiple fisheye lenses. Moreover, the size of the entire imaging system is too bulky and heavy to apply it to various circumstances. Thus, there has been a need for the development of a novel imaging device as a replacement of the conventional 360° cameras.

Figure 1. Geometrical and functional features of the panoramic and amphibious visions in nature: (a) Schematic illustrations for geometrical comparison of the ommatidium of three kinds of crabs (e.g., vampire crab, fiddler crab, and spanner crab) according to their habitat; (b) Photographs (left) and SEM image (right) of the fiddler crab eye; (c) SEM image in top view (left) and TEM image in cross-sectional view (right) for the ommatidia of fiddler crab; (d) Distribution map of ommatidia of the fiddler crab eye.

 Aiming for the development of a miniatured imaging system with panoramic visual field and amphibious imaging capability, we focused on the eyes of intertidal crabs (e.g., fiddler crab) (Fig. 1a). Unlike other types of crabs that live only on land or under water, the intertidal crabs have evolved ellipsoidal-shape eyes with flat corneal lenses, on which ommatidia are distributed to cover nearly 360°. Figure 1b shows photographs of the fiddler crab eye from various angles, and its magnified view (right inset: scanning electron microscope (SEM) image). The ommatidia are distributed on the almost entire surface of the ellipsoidal eye. Figure 1c shows flat cornea facet lenses (left frame: top view, SEM image) and a multi-layer structure of the corneal lens (right frame: cross-sectional view, transmission electron microscope (TEM) image). The flat lens with the multi-layer structure achieves a consistent focal length regardless of external refractive index change between air and water. A visual map that indicates the distribution range of the ommatidia confirms a panoramic vision of the fiddler crab (Figure 1d). From these structural and functional features, we could find out that the fiddler crab has an amphibious and panoramic vision, which inspired the design and fabrication of a novel amphibious and panoramic artificial vision.

Figure 2. Artificial version of a panoramic and amphibious vision inspired by the fiddler crab eye: (a) Structural comparison between a biological ommatidium and an artificial ommatidium; (b) Cross-sectional optical image of an artificial corneal lens (i.e., graded refractive index micro-lens (g-ML)); (c) Exploded view of a comb-shape image sensor array, which includes g-MLs and photodetectors; (d) Schematics that show the pixel distribution on the 3D structure from top and side views; (e) Photograph of the artificial vision. The image sensor array and g-ML array are combined and integrated on the 3D structure.

 In our recent publication, we have reported a novel panoramic and amphibious imaging device inspired by the fiddler crab vision. The artificial vision features panoramic FoV (horizontal FoV (FoVH): 300°, vertical FoV (FoVV): 160°), amphibious imaging capability, minimal optical aberrations, miniaturized form factor, and large depth-of-field (DoF) (1 cm ~ infinity). Specifically, we developed an artificial ommatidium by stacking polymeric layers with graded refractive indices (i.e., g-MLs) by mimicking the flat multi-layer corneal lens (Fig. 2a–b). This artificial corneal lens exhibits optical properties comparable to those of the natural fiddler crab eye lens. We also integrated a comb-shaped silicon photodetector array with the g-ML array (Fig. 2c), and mounted the integrated arrays on a spherical structure (Fig. 2d–e).

Figure 3. Imaging demonstration of the panoramic and amphibious artificial vision: (a) Experimental setup for the panoramic imaging demonstration; (b) Exploded view of the panoramic imaging results; (c) Experimental setup for the amphibious imaging demonstration; (d) Images obtained by the optical simulation (left) and experiment (right).

 We demonstrated the imaging capability of the artificial vision by using a lab-made setup (Fig. 3). Imaging demonstrations were performed with five objects projected from all directions (Fig. 3a). We could successfully obtain the panoramic imaging result (Fig. 3b). Also, our artificial vision successfully achieved amphibious imaging, whose result was comparable to the optical simulation result (Fig. 3c–d). Even when a half of the device was immersed in water, the imaging performance was comparable to the case when the device was outside the water. According to these imaging demonstrations, we could verify that the artificial vision has a panoramic and amphibious imaging capability.

 This work can bring a significant advance with regards to the development of a novel panoramic vision system with an amphibious imaging capability. Such an artificial vision can provide many new opportunities for the next-generation artificial vision applications including mobile electronics and robotics. For more detailed information, please see the latest article published in Nature Electronics: An amphibious artificial vision system with a panoramic visual field.