DOSA: Organic Compilation for Neural Network Inference on Distributed FPGAs

The computational requirements of artificial intel- ligence workloads are growing exponentially. At the same time, Dennard scaling has ended and Moore’s law is winding down. In addition, more and more compute is moved towards the edge due to latency or localization constraints. These trends cre- ated an opportunity for specialized accelerators including field- programmable gate arrays (FPGAs), but the poor support and usability of today’s tools prevents FPGAs from being deployed at scale for deep neural network (DNN) inference applications. In this work, we propose an organic compiler — DOSA — that drastically lowers the barrier for deploying FPGAs. DOSA builds on the operation set architecture concept and mixes the DNN accelerator components generated by existing DNN-to-FPGA frameworks to produce an overall efficient solution. DOSA starts from DNNs represented in the community standard ONNX and automatically implements model- and data-parallelism, based on the performance targets and resource footprints provided by the user. Deploying a DNN using DOSA on 9 FPGAs exhibits a speedup of up to 52 times compared to a CPU and 18 times compared to a GPU.