Unlocking Better AI Performance by Rethinking Decision-Making in Deep Learning Layers

Deep Learning (DL) performs classification tasks using a series of layers. To effectively execute these tasks, local decisions are performed progressively along the layers. But can we perform an all-encompassing decision by choosing the most influential path to the output rather than performing these decisions locally?

Study: Enhancing the accuracies by performing pooling decisions adjacent to the output layer. Image Credit: cono0430 / Shutterstock

In an article published today in the journal Scientific Reports, researchers from Bar-Ilan University in Israel answer this question with a resounding "yes." By updating the most influential paths to the output, preexisting deep architectures have been improved.

"One can think of it as two children who wish to climb a mountain with many twists and turns. One of them chooses the fastest local route at every intersection while the other uses binoculars to see the entire path ahead and picks the shortest and most significant route, just like Google Maps or Waze. The first child might get a head start, but the second will end up winning," said Prof. Ido Kanter of Bar-Ilan's Department of Physics and Gonda (Goldschmied) Multidisciplinary Brain Research Center, who led the research.

"This discovery can pave the way for better enhanced AI learning, by choosing the most significant route to the top," added Yarden Tzach, a PhD student and one of the key contributors to this work.

This exploration of a deeper comprehension of AI systems by Prof. Kanter and his experimental research team, led by Dr. Roni Vardi, aims to bridge between the biological world and machine learning, thereby creating an improved, advanced AI system. To date, they have discovered evidence for efficient dendritic adaptation using neuronal cultures, as well as how to implement those findings in machine learning, showing how shallow networks can compete with deep ones, and finding the mechanism underlying successful deep learning.

Enhancing existing architectures using global decisions can pave the way for improved AI, which can improve its classification tasks without the need for additional layers. 

VGG16 and A-VGG16 architectures. (A) VGG16 architecture (13 (3×3)(3×3) CLs and in between 5 (2×2)(2×2) MP operators, followed by 3 FC layers) for the CIFAR10 database consisting of 32×3232×32 RGB inputs. A CL is defined by its square filters with dimension K and depth D, (K,D)(K, D). (B) A-VGG16 architecture for CIFAR10 inputs consisting of 7 (3×3)(3×3) CLs, (4×4)(4×4) average pooling (AP), 6 (3×3)(3×3) CLs, (2×2)(2×2) MP and 3 FC layers.