Ordinary Differentiation for Machine Learning Instead of Matrix Calculus

The Challenge of Matrix Calculus

Matrix calculus is frequently encountered in machine learning contexts, particularly when dealing with differentiation of functions involving vector or matrix variables. It involves a set of complex rules that vary based on vector and matrix orientations, making it difficult to apply consistently.

An Alternative: Ordinary Differentiation with Indices

An alternative approach, historically used by physicists for tensors, involves writing out formulas with indices and applying ordinary differentiation. This method is presented as a more straightforward and universally applicable solution compared to matrix calculus. It eliminates the need to learn specific matrix calculus rules.

Advantages of the Indexed Approach

The primary advantage of using ordinary differentiation with indices is its universal applicability; it works for all formulas. It also requires knowledge only of ordinary differentiation, potentially saving time compared to memorizing and applying the specific rules of matrix calculus. This method is demonstrated with an example of a neuron activation function.

Practical Application Example

The article illustrates the method by differentiating a neuron activation function, \(\mathsf{activation}(x) = \max(0, Ax + b)\). By expanding the matrix multiplication into indexed form, \(\mathsf{activation}(x)_i = \sum_j A_{ij}x_j + b_i\), ordinary differentiation can be applied with respect to \(x_k\), \(A_{kl}\), or \(b_k\) by setting the corresponding primed variable to 1 and others to 0.