On The Biological Plausibility Of Orthogonal Initialisation For Solving Gradient Instability In Deep Neural Networks

# On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks

**arXiv ID:** 2211.08408
**Authors:** Nikolay Manchev, Michael Spratling
**Published:** 2022-10-27T06:08:06Z
**Abstract:**
Initialising the synaptic weights of artificial neural networks (ANNs) with orthogonal matrices is known to alleviate vanishing and exploding gradient problems. A major objection against such initialisation schemes is that they are deemed biologically implausible as they mandate factorization techniques that are difficult to attribute to a neurobiological process. This paper presents two initialisation schemes that allow a network to naturally evolve its weights to form orthogonal matrices, provides theoretical analysis that pre-training orthogonalisation always converges, and empirically confirms that the proposed schemes outperform randomly initialised recurrent and feedforward networks.

## Skill Description

This skill is generated from the arXiv paper: On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networks (2211.08408).

## How to Use

[To be filled in by the user or by future automation]

## References

- [arXiv:2211.08408](http://arxiv.org/abs/2211.08408v1)