Efficient Multitask Learning with Compact Neural Networks

By Aryaman PattnayakReviewed by Susha Cheriyedath, M.Sc.Aug 30 2023

A recent paper submitted to the arxiv* server introduces an effective technique to train compact neural networks for multitask learning scenarios. The method achieves improved optimization and generalization in resource-constrained applications by overparameterizing the network during training and sharing parameters more efficiently.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.

Compact Multitask Learning

Multitask learning allows a single neural network to tackle multiple related tasks by sharing representations between them. This provides benefits like reduced inference time and computational costs. However, designing compact yet accurate multitask models poses challenges. The network must balance efficiency with effectively sharing knowledge across tasks.

Smaller models typically achieve lower performance on complex tasks. Careful network design is needed to maximize accuracy under tight parameter budgets. Multitask learning adds further difficulties as tasks compete for limited resources. Achieving strong generalization with compact networks remains an open problem, especially for multitasking scenarios.

Overparameterization for Training

The proposed method expands the network during training by factorizing layers into multiple matrices. The core idea is that overparameterization can improve optimization and generalization even if representational capacity is unchanged. Previous work has shown the benefits of overparameterization for plain neural networks. The authors extend this approach to multitask learning, where it provides additional advantages. Factoring into shared and task-specific components enables more efficient pooling of knowledge.

The factorization uses spatial singular value decomposition, yielding matrices U, V, and a diagonal matrix M. U and V contain shared knowledge and are tuned on all tasks. M captures task-specific variations and is split into versions M(1)....M(t) for each task. This decomposition allows pooling knowledge in U and V while tuning M(i) to align parameters to each task's data singular vectors. An iterative training strategy separates updates to shared and task-specific components.

Validating Effectiveness

The method was evaluated on multitask datasets NYUv2 and COCO for semantic segmentation, depth estimation, and other vision tasks. It improved performance over single-task learning baselines using compact versions of networks like SegNet.

Experiments showed benefits on both smaller and larger models. The technique generalizes across diverse architectures. It was applied to convolutional and Residual Network architectures and consistently improved results. Ablations quantified the impact of factorizing versus sharing different components. Results demonstrated the importance of task-specific tuning of M. The approach also outperformed prior overparameterization techniques like RepVGG.

The capability to deploy highly efficient multitask models with strong performance opens up exciting possibilities across domains like robotics, autonomous systems, and mobile devices. Compact networks that excel at diverse perceptual tasks will enable embedding intelligent capabilities into more platforms. Systems from drones to wearables to household appliances can benefit from on-device deep learning if models are sufficiently lightweight.

The proposed technique targets multitask scenarios common in embedded vision systems for automation, navigation, inspection, and more. Sharing knowledge through overparameterization unlocks substantial accuracy gains under tight computational budgets. Advanced driver assistance systems are prime examples of applications where compact networks must perform semantic segmentation, object detection, depth estimation, and other tasks concurrently.

Effective multitasking learning will allow more sensing and analysis to be performed locally, reducing reliance on connectivity and improving speed, security, and robustness. The ability to simultaneously capture multiple facets of an environment provides contextual advantages. Deploying multitask networks on low-powered devices can progress embedded intelligence and edge computing. As research enhances the optimization of compact models, the scope of on-device deep learning will continue.

Future Outlook

This paper presents a proficient methodology for training exceptionally concise yet precise models for multitasking learning. By strategically introducing an excess of parameters and facilitating parameter sharing, the optimization and generalization of models are enhanced. The efficacy of this approach is exemplified across various applications, including the realm of autonomous driving, where the availability of computational resources is constrained. This advancement holds the potential to facilitate the embedded deployment of multitask deep learning across a diverse spectrum of platforms.

The current contribution holds significant implications, as it introduces an adept technique to cultivate compact networks that concurrently exhibit remarkable multitasking performance. This, in turn, extends the practical utility of artificial intelligence at the periphery of computational infrastructure. Furthermore, ongoing research endeavors may refine optimization strategies and bolster knowledge-sharing mechanisms for streamlined multitask models of reduced complexity.

*Important notice: arXiv publishes preliminary scientific reports that are not peer-reviewed and, therefore, should not be regarded as definitive, used to guide development decisions, or treated as established information in the field of artificial intelligence research.