When we think of cameras, we usually think of them being used by humans to take pictures.
Although humans are generally considered to be the primary consumers of images captured
by cameras, machines have also started consuming images on a very large scale.
Today, virtually all computer vision applications use cameras optimized for human viewers.
The image signal processors in those cameras are usually tuned based on photography-driven
image quality characteristics that are important to the human visual system.
However, tuning an imaging pipeline for the optimal image quality does not guarantee optimal
results for computer vision tasks.
For instance, heavier noise reduction might result in better perceptual image quality,
whereas it might be more beneficial for a machine vision system to tolerate a higher
level of noise in exchange for more information.
An image processed for optimal computer vision performance can look vastly different from
an image processed for optimal perceptual image quality.
We propose a set of processing blocks, which we collectively call the VisionISP, to build
an image processing pipeline optimized for machine consumption.
The blocks in VisionISP are simple, content-aware, and trainable.
The first block is a computer-vision-driven denoiser that tunes an existing ISP without
modifying the underlying algorithms and hardware design.
Our ISP tuning algorithm tunes the denoising parameters to minimize a high-level content
loss in the denoised images.
We calculate this loss on feature maps extracted from a target neural network that is pre-trained
to perform a particular computer vision task.
In this way, the denoiser learns to preserve what's important in the image for the target
machine vision task.
The second block in our pipeline is a trainable local tone mapping operator which reduces
the bit-depth of its input.
Reducing the bit depth translates into simpler hardware, reduced bandwidth, and significant
savings in power.
Unlike uniform bit-depth reduction, our method makes sure that the features that are essential
for computer vision applications are preserved after bit-depth reduction.
We do this by using a global non-linear transformation followed by a local detail boosting operator
before bit-depth reduction.
The non-linear transform acts as a trainable global tone mapping operator while the detail
boosting operator acts locally to preserve the details in the low-bit-per-pixel output.
The last block in our pipeline is a very simple convolutional neural network that acts as
a preprocessor for a subsequent computer vision task.
The first layer in this block is a 1x1 convolution layer that can be thought of as a trainable
color space converter that finds an optimal color space automatically.
The following layer is a 7x7 convolution layer that extracts low-level features, such as
edges and textures while reducing the input resolution.
This layer has a flexible stride parameter that allows for adjusting the downscaling
rate, without retraining.
Many computer vision systems use downscaled images to be able to run in real-time.
However, conventional downscaling methods, such as bilinear interpolation, are content-agnostic.
Therefore, small details in a scene, such as pedestrians, can easily be discarded during
such downscaling.
This feature extraction layer processes full resolution data and helps downscale images
without dropping the features that would be needed in the next stages.
The final layer in the block projects the output feature maps into three channels since
computer vision systems typically expect 3-channel images as inputs.
Although those 3-channel inputs do not look natural to human viewers when visualized as
pseudo-color images, they `look good' to machines in the sense that they provide a very efficient
representation of what a camera should feed into a computer vision system to perform well.
For example, this demo video shows how object detection results look on frames processed
by the VisionISP as compared to a basic image signal processor that is not optimized for
computer vision.
The VisionISP significantly reduces data transmission needs between an image signal processor and
a computer vision engine while preserving the information relevant for a computer vision
system.
Take a look at our paper to see our experimental results and learn more about how VisionISP
works.