CVPR 2022

Technology 01 On-the-fly 6DoF object pose estimation

For applications involving interaction with the real world, such as AR, 3 dimensional(6DoF) pose estimation is a key technology. However, modern ML methods require a large amount of labeling effort, which prevents users to improvise and register their favorites.
Our technology solves this issue while achieving a high level of accuracy.

In the right part of the clip above, a 3D model of the object is rendered based on the pose estimation results.

Pick up any object you like and turn it around in front of the camera. This is all it takes to register it.

• Attribute(PDF)

Technology 02 Lensless Imaging Technology

What is a lensless camera?

A lensless camera is a camera consisting of an image sensor and a patterned mask. In contrast to a standard camera with a lens, a sensor image of a lensless camera does not resemble the scene image and should be decoded. For example, in Fig. 1, we show a basic imaging pipeline for a lensless camera. A lensless camera can be an attractive alternative to a lens-based camera despite the necessary image reconstruction step. As one of the largest image sensors and sensing electronics manufacturers, Sony is developing lensless imaging technology with a perspective of applying it in our future products.

Fig. 1. Lensless imaging pipeline

Lensless imaging in long-wave infrared

One advantage of a lensless camera is that it can be attractive when light focusing with a lens is impossible or too expensive. One such example is long-wave infrared imaging (LWIR), where one of the biggest challenges is the high price of optics. In our recent project, we successfully constructed an LWIR lensless camera and demonstrated how it could be used in combination with a conventional RGB camera to improve the accuracy of pedestrian detection in challenging illumination scenarios. In Fig. 2, we show a schematic of the image processing pipeline. See [1] for more details.

Fig. 2. LWIR lensless + RGB imaging pipeline

Lensless mismatched aspect ratio imaging

Lensless imaging technology can also be used to achieve results that are impossible with a regular camera. Recently, we demonstrated a lensless camera for imaging with a resolution significantly larger than the sensor's resolution in one of its dimensions by virtually rearranging the available pixels. For example, in Fig. 3, we demonstrate the reconstruction result from our lensless camera, which allows us to capture scene images with a resolution of 3171 x 103 pixels from a 453 x 721 sensor image. I.e., the horizontal resolution of the result is more than four times higher than the vertical resolution of the sensor. For more details, see [2].

Fig. 3. Sensor image (left), reconstruction result (right)

Efficient optical implementation for lensless technology

In a lensless camera, a patterned mask is often the only optical element between the scene and the sensor. Therefore, it is important to ensure that the mask works with maximum efficiency. Typically, many lensless cameras employ binary or phase masks. However, these masks have several issues, including low light efficiency and diffraction-related artifacts. Recently, we have proposed a framework for migration from diffractive to refractive masks, which allows us to significantly improve imaging quality without adding extra computational or manufacturing costs. Fig. 4 shows the reconstruction quality improvement by a simple switching from a binary to a refractive mask. For more details, see [3].

Fig. 4. Reconstruction results using a binary mask (top) and a refractive mask (bottom)

[1] I. Reshetouski, H. Oyaizu, K. Nakamura, R. Satoh, S. Ushiki, R. Tadano, A. Ito, and J. Murayama, "Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and its Application for Human Detection,“ 2020 European Conference on Computer Vision (ECCV)
[2] I. Reshetouski, R. Tadano, H. Oyaizu, K. Nakamura and J. Murayama, "Lensless Mismatched Aspect Ratio Imaging," 2021 IEEE International Conference on Computational Photography (ICCP), 2021, pp. 1-12
[3] To appear in 2022 Imaging and Applied Optics Congress

Technology 03 Polarization Image Sensor Technology Polarsens

What is Polarization? What is Polarsens? Why does an on chip Polarizer have different filter angles?
This video provides a general overview of Sony Semiconductor Solutions' Polarization Image Sensor.

This video introduces application examples of polarization image sensor: reflection removal, visual inspection, improve visibility, 3D/Stress detection and human/vehicle detection.

Technology 04 Life with DepthSense™

DepthSense™ solutions with ToF sensor brings a new experience to your life.

Technology 05 AR Gaming with RGB / Depthsense enabled SLAM

Integration of RGB-D Fusion and SLAM into a real-time Mobile AR Demo Application. Capitalizing on Time of Flight for instant SLAM/3D, low-light/texture SLAM/3D, night vision and scene mesh. Dense depth (full-field) and sparse depth (spots) based pipelines.

Workshop 01 A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-off-Flight Depth Maps

Sparse active illumination enables precise time-of-flight depth sensing as it maximizes signal-to-noise ratio for low power budgets. However, depth completion is required to produce dense depth maps for 3D perception. We address this task with realistic illumination and sensor resolution constraints by simulating ToF datasets for indoor 3D perception with challenging sparsity levels. We propose a quantized convolutional encoder-decoder network for this task. Our model achieves optimal depth map quality by means of input pre-processing and carefully tuned training with a geometry-preserving loss function. We also achieve low memory footprint for weights and activations by means of mixed precision quantization-at-training techniques. The resulting quantized models are comparable to the state of the art in terms of quality, but they require very low GPU
times and achieve up to 14-fold memory size reduction for the weights w.r.t. their floating point counterpart with minimal impact on quality metrics.
This is a joint work between Sony Depthsensing Solutions, Sony R&D Center Stuttgart Lab 1, and the University of Padua, Italy.

Workshop 02 M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation

Emotion Recognition in Conversations (ERC) is crucial in developing sympathetic human-machine interaction. In conversational videos, emotion can be present in audio, visual, and transcript.
However, due to the inherent characteristics of these modalities, multi-modal ERC has always been considered a challenging undertaking. This paper proposes a Multi-modal Fusion Network (M2FNet) that extracts emotion-relevant features from visual, audio, and text modality. It further employs a multi-head attention-based fusion mechanism to combine emotion-rich latent representations of the input data.
A new feature extractor is also introduced to extract features from audio and visual modalities. The proposed M2FNet architecture outperforms the other methods and sets a new state-of-the-art performance.

Workshop 03 SqueezeNeRF: Further factorized FastNeRF for memory-efficient inference

Neural Radiance Fields (NeRF) has emerged as the state-of-the-art method for novel view generation of complex scenes, but is very slow during inference.
Recently, there have been multiple works on speeding up NeRF inference, but the state of the art methods for real-time NeRF inference rely on caching the neural network output, which occupies several giga-bytes of disk space that limits their applicability in embedded systems. In this work, we propose SqueezeNeRF, which is more than 60 times memory-efficient than the state of the art methods and is still able to render at more than 190 frames per second on a high spec GPU during inference.