Trainable matrices in Tensorflow/Keras

[Avatrin]

TL;DR Summary

Trainable matrices in Tensorflow & Keras for Attention

Hi

Several attention mechanisms require trainable matrices and vectors. I have been trying to learn how to implement this in Tensorflow w/ Keras. Every implementation I see use the Dense layer from Keras, but I have a tendency to get lost trying to understand why and what they do afterwards. Sometimes it seems contradictory (for instance, the shapes of the matrices here and here).

Are there any good explanations for how the Dense layer works, and especially, how it works when it's treated like a matrix instead of a fully connected layer. I mean, some tutorials use matmul, but how does that work when the batch size is larger than one?

I hope somebody here can help me

 

[Future Bruno]

out.The Dense layer in Keras is a fully connected layer, which means that every neuron in one layer is connected to every neuron in the next layer. It is typically used for classification tasks and is a popular choice for deep learning architectures. The basic idea behind a fully connected layer is that it allows for efficient computation of the weights and biases for each neuron in the network, and also allows for more complex models to be created.When using the Dense layer as a matrix, the weights and biases are still computed in the same way as with a fully connected layer, however the weights and biases are now arranged into a matrix of size (inputs x outputs). This allows us to take advantage of matrix multiplication to quickly calculate the output of each neuron.To handle batches of data larger than one, the matrix multiplication is simply repeated for each batch in the dataset. So if we had a batch size of 10, then we would run the matrix multiplication 10 times, once for each batch, and then combine the results to get the final output.I hope this helps!