welcome to this four-part series on targeting vision applications to FPGA or ASIC hardware
a Lane detection design will be used throughout but the topics are applicable to designing any
vision processing application for Hardware the first part will discuss some of the concepts
you will need to pay attention to part 2 will give an overview of the workflow part 3 will
illustrate specific considerations in the Lane detection example and part 4 will show how to
target the design of Hardware automated driving applications rely heavily on vision processing
of video from onboard cameras these applications must process large amounts of data at high speed
after all you don't want any delay in corrective action at highway speeds and even a 720p camera
has nearly 1 million active pixels in a frame and at 30 frames per second that requires high-speed
processing FPGA hardware delivers this kind of performance with efficient power and size
footprints but programming complex algorithms and hardware description languages can be challenging
system-on-chip platforms combine FPGA hardware with a processor to run software so the high
speed and well-defined tasks can run on the hardware to deliver a smaller set of data
to the software which handles the intelligence and control but processing in Hardware requires
adapting algorithms to process streams of pixel data and logic to manage those data
streams processing can be parallelized which accelerates computation but also requires
coordinating the timing of parallel paths and one of the biggest constraints with hardware is
that it is fixed with a finite set of resources the data streams in and out via a fixed set of
pins there is a certain amount of on chip RAM in an FPGA that's called block RAM but it's not
enough to store entire image frames so if you need to store a full frame typically it needs
to be sent to off chip Ram via memory controller intellectual property blocks IP blocks such as
this are pre verified building blocks that perform common tasks and are available in ASIC or FPGA
hardware and of course all Hardware has random logic and clocking structures for programming
the customized functionality FPGA is also have pre-configured DES P slices for performing heavy
arithmetic these are efficient implementations of multipliers with some other built-in math and
registers to synchronize timing mapping multiply operations to these is more efficient than using
random logic but they are also a fixed resource typically taking inputs up to 18 by 18 or 25 by
18 bits if your word sizes grow beyond that a multiplication operation gets mapped to multiple
DSP slices and you can use them up pretty quickly so data word sizes are something to constantly pay
attention to when designing for hardware it might be obvious that storing say 32-bit data is going
to be used more memory than 12 bit data but it will also require more fixed resources to move
around manage perform operations on and so forth hardware designers spend a lot of time balancing
the size of data types versus the amount of accuracy needed with a rough understanding
of the constraints of hardware let's focus specifically on vision processing at the
algorithm level you can process a frame at a time and access any pixel randomly as an XY coordinate
to do this in hardware would require streaming in a frame storing it to RAM reading out what's
needed then storing the resulting frame in another Ram location for every stage of the algorithm this
is slow and resource intensive it's much more efficient to operate on a continuous stream of
pixels but that can require algorithm changes and that's pretty straightforward if you're doing say
color space conversion you just convert each pixel as it passes through but if you need to
do something like filtering or edge detection you need a window to operate on or a region of
interest which requires storing multiple lines of pixels in what's called line buffer memory
that small amount will typically fit into the block Ram on the FPGA but obviously you will
need to adapt your algorithms to work on this sliding window so image processing in a frame
based world is very different from implementing these algorithms as pixel streaming Hardware our
experience has been that best results are achieved when the image processing engineers collaborate
with the hardware design engineers vision HDL toolbox helps facilitate that collaboration it
has many standard and processing algorithms that operate in pixel streaming mode and the
algorithms that require a region of interest have easy settings for line buffer sizes using
HDL coder these blocks generate efficient Verilog or VHDL to target an FPGA or ASIC device it also
includes utilities such as blocks that convert frames to pixels and vice versa for a variety
of frame sizes generating the control signals needed by the hardware and there are a number of
reference application examples that illustrate real implementations using these blocks around
what's surrounding logic in the next part of this series will outline a recommended workflow
to take frame based image processing algorithms in MATLAB to pixel streaming FPGA implementations