University of Zurich (UZH) researchers have combined a novel bio-inspired camera with AI to develop a system that can detect obstacles around a car much more quickly than current systems.

Quicker detection using less computational power

Cameras currently used for driver assistance on cars typically capture 30 to 50 frames per second. An artificial neural network can be trained to recognize objects in their images. But if something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late, according to the researchers.

The solution would be increasing the frame rate, but that translates into more data that needs to be processed in real-time and more computational power, the scientists note in an open-access Nature paper published today.

Combining the best of two camera types with AI to emulate human eyes

Event (“neuromorphic”) cameras are a recent innovation based on a different principle. Instead of a constant frame rate, they record information every time they detect fast movements, similar to how human eyes perceive images. But they can miss things that move slowly and their images are not easily converted into the kind of data that is used to train the AI algorithm.

The researches created a hybrid system that combines the best of both worlds: It includes a standard camera that collects 20 images per second (a relatively low frame rate). Its images are processed by a convolutional neural network, that is trained to recognize cars or pedestrians. The data from the event camera is coupled to a different type of AI system, called an asynchronous graph neural network, which is particularly apt for analyzing 3-D data that change over time.

Detections from the event camera are used to anticipate detections by the standard camera and also boost its performance. The result is a visual detector that can detect objects just as quickly as a standard camera taking 5,000 images per second would do but only requires the same bandwidth as a standard 50-frame-per-second camera.

One hundred times faster detections using less data

The team tested their system against the best cameras and visual algorithms currently on the automotive market, finding that it leads to one hundred times faster detections while reducing the amount of data that must be transmitted between the camera and the onboard computer as well as the computational power needed to process the images without affecting accuracy.

Crucially, the system can effectively detect cars and pedestrians that enter the field of view between two subsequent frames of the standard camera, providing additional safety for both the driver and traffic participants, which can make a huge difference, especially at high speeds.

According to the scientists, the method could be made even more powerful in the future by integrating cameras with LiDAR sensors, like the ones currently used on self-driving cars.

Citation: Gehrig, D., Scaramuzza, D. Low-latency automotive vision with event cameras. Nature 629, 1034–1040 (2024). https://doi.org/10.1038/s41586-024-07409-w (open access)

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.