Facebook Page
+ Reply to Thread
Results 1 to 2 of 2
  1. MHB Journeyman
    MHB Site Helper
    MHB Math Helper
    Rido12's Avatar
    Join Date
    Jul 2013
    3,384 times
    811 times
    Chat Box Champion (2014)  

MHB Model User Award (2014)

    I guess this could be a rather silly question, but I got a bit confused about the "numerator layout notation" and "denominator layout notation" when working with matrix differentiation:

    It says that with the denominator layout notation, we interpret differentiation of a scalar with respect to a vector as such: $\frac{\mathrm{d}L}{\mathrm{d}w_1}=[\frac{\mathrm{d}L}{\mathrm{d}w_{11}}\frac{\mathrm{d}L}{\mathrm{d}w_{12}} ... \frac{\mathrm{d}L}{\mathrm{d}w_{1n}}]^T$, $L$ a scalar and $w_1$ an $n$ x $1$ vector.

    But what if we represent the scalar $L$ differently? e.g $L=w^Tx$, where $w$, $x \in \Bbb{R}^{n \times1}$.
    Then we get $\frac{\mathrm{d}L}{\mathrm{d}w}=\frac{\mathrm{d}(w^Tx)}{\mathrm{d}w}=\frac{\mathrm{d}(x^Tw)}{\mathrm{d}w}=x^T$, which is a $1$ by $n$ vector. Doesn't this result disagree with the denominator layout notation? I read somewhere on the wiki that says one should stick to one type of notation, but if certain types of calculations favors one type of notation over the other, wouldn't that be problematic or confusing?

    Last edited by Rido12; February 14th, 2018 at 01:21.

  2. MHB Seeker
    MHB Global Moderator
    MHB Math Scholar
    MHB Coder
    I like Serena's Avatar
    Join Date
    Mar 2012
    4,660 times
    12,850 times
    MHB Model Helper Award (2017)  

MHB Best Ideas (2017)  

MHB Analysis Award (2017)  

MHB Calculus Award (2017)  

MHB Model Helper Award (2016)
    Hey Rido12!

    Indeed, the matrix layouts of derivatives tend to be confusing.
    The problem as I see it, is that the matrix layout is an arbitrary representation.
    And it already more or less fails if we have more than 2 dimensions, since then we can't properly represent it in a rectangular matrix.
    When we take the derivative of a vector function with respect to a vector, what we actually have is a set of scalar functions:
    $$\pd {\mathbf f}{\mathbf x} = \left(\pd {f_i}{x_j}\right)$$
    That is, forget about the matrix layout.
    And when we want to multiply it with a vector $\mathbf v$ to find a directional derivative, which is really an application of the chain rule, what we need to do is:
    $$\pd {\mathbf f}{\mathbf x} \cdot \mathbf v = \sum_{i=1}^n \pd {f_i}{x_j} v_j \mathbf e_i$$
    where $\mathbf e_i$ is the $i$-th unit vector.

    Since as humans we like to represent that in something we can write down, and that fits into how we usually do matrix manipulations, the most natural way that fits in our conventions is:
    $$\begin{bmatrix}\pd {f_1}{x_1} & ... & \pd {f_1}{x_n} \\ \vdots & & \vdots \\ \pd {f_n}{x_1} & ... & \pd {f_n}{x_n} \end{bmatrix}
    \begin{bmatrix}v_1 \\ \vdots \\ v_n\end{bmatrix}$$
    This is the Jacobian form, or numerator layout.
    The thing to realize, is that whenever we do something like this, we need to ensure that the elements get multiplied and summed with the right elements.
    So if we choose to pick the denominator layout instead, to ensure our conventional matrix product works out, we need to write it as:
    $$\begin{bmatrix}v_1 & \dots & v_n\end{bmatrix}\begin{bmatrix}\pd {f_1}{x_1} & ... & \pd {f_n}{x_1} \\ \vdots & & \vdots \\ \pd {f_1}{x_n} & ... & \pd {f_n}{x_n} \end{bmatrix}

    Or we can choose to forget about conventional matrix layouts and products, and just write:
    $$\sum_{i=1}^n \pd {f_i}{x_j} v_j \mathbf e_i$$
    or for short:
    $$\pd {f_i}{x_j} v_j$$
    following Einstein summation convention.

Similar Threads

  1. Replies: 2
    Last Post: October 10th, 2016, 00:52
  2. [SOLVED] Zero-Trace Symmetric Matrix is Orthogonally Similar to A Zero-Diagonal Matrix.
    By caffeinemachine in forum Linear and Abstract Algebra
    Replies: 5
    Last Post: December 12th, 2013, 23:15
  3. Replies: 3
    Last Post: December 11th, 2013, 07:46
  4. Notation for Element-Wise Matrix Operations
    By OhMyMarkov in forum Linear and Abstract Algebra
    Replies: 1
    Last Post: July 24th, 2012, 21:30
  5. Matrix Theory...showing that matrix is Unitary
    By cylers89 in forum Linear and Abstract Algebra
    Replies: 3
    Last Post: March 27th, 2012, 17:00

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
Math Help Boards