- #1
madness
- 815
- 70
- TL;DR Summary
- Are there methods for exhaustively finding local optima? Could one do so iteratively by adding a cost function which avoids previously explored optima?
Hi all,
Let me give some background to my question. In computational neuroscience it's now fashionable to train recurrent neural networks (RNNs) to solve some task (working memory for example). We do so by defining some cost function and training the model weights to minimize it (using gradient descent). However one usually finds that there are multiple unique solutions which solve the task well but in conceptually distinct ways. I would like to find a way to iteratively search through and catalogue all the solutions that an RNN can come up with for a task. (Notice that this problem is not unique to RNNs - it could arise in many non-convex optimization problems).
The first idea that comes to mind is to add a term to the cost function which penalizes solutions that are close to a previous solution. One could then repeat this iteratively to find further solutions. Does anyone know if methods based on this principle exist?
Immediate problems that spring to mind are: 1) RNNs have so much symmetry that the network could find the same solution, for example with permuted neuron indices. This could be address by choosing an appropriate pentalty. 2) Adding these penalties to the cost function could destroy or change the other local optima we are looking for.
Thanks for your help.
Let me give some background to my question. In computational neuroscience it's now fashionable to train recurrent neural networks (RNNs) to solve some task (working memory for example). We do so by defining some cost function and training the model weights to minimize it (using gradient descent). However one usually finds that there are multiple unique solutions which solve the task well but in conceptually distinct ways. I would like to find a way to iteratively search through and catalogue all the solutions that an RNN can come up with for a task. (Notice that this problem is not unique to RNNs - it could arise in many non-convex optimization problems).
The first idea that comes to mind is to add a term to the cost function which penalizes solutions that are close to a previous solution. One could then repeat this iteratively to find further solutions. Does anyone know if methods based on this principle exist?
Immediate problems that spring to mind are: 1) RNNs have so much symmetry that the network could find the same solution, for example with permuted neuron indices. This could be address by choosing an appropriate pentalty. 2) Adding these penalties to the cost function could destroy or change the other local optima we are looking for.
Thanks for your help.