The Loss Landscape Of Overparameterized Neural Networks

# The loss landscape of overparameterized neural networks

**arXiv ID:** 1804.10200
**Authors:** Y Cooper
**Published:** 2018-04-26T17:58:45Z
**Abstract:**
We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from $\mathbb{R}^n$ to $\mathbb{R}$ - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has $n$ parameters and is trained on $d$ data points, with $n>d$, we show that the locus $M$ of global minima of $L$ is usually not discrete, but rather an $n-d$ dimensional submanifold of $\mathbb{R}^n$. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that $M$ is typically a very high-dimensional subset of $\mathbb{R}^n$.

## Skill Description

This skill is generated from the arXiv paper: The loss landscape of overparameterized neural networks (1804.10200).

## How to Use

[To be filled in by the user or by future automation]

## References

- [arXiv:1804.10200](http://arxiv.org/abs/1804.10200v1)