45. Adv: tensor products, intro to multilinear algebra
§23.1's bilinear form took two vectors in and produced one scalar. This lesson asks what happens when two vectors go in and a vector comes out instead — the tensor product — and shows it isn't a new invention so much as the object every bilinear form (and every matrix outer product already used since §16.2) was quietly built from.
The tensor product of two vectors
For \vec u\in\mathbb{R}^m, \vec v\in\mathbb{R}^n, the tensor product \vec u\otimes\vec v is the m\times n matrix
\vec u\otimes\vec v=\vec u\vec v^T
— exactly the outer product already used in §21.4's SVD sum A=\sum_i\sigma_i\vec u_i\vec v_i^T, now given its proper name. Entrywise, (\vec u\otimes\vec v)_{ij}=u_iv_j — every possible product of one entry from \vec u and one from \vec v, arranged in a grid.
Bilinearity: (\vec u+\vec u')\otimes\vec v=\vec u\otimes\vec v+\vec u'\otimes\vec v and (c\vec u)\otimes\vec v=c(\vec u\otimes\vec v), both immediate from \vec u\vec v^T's distributive properties (§16.2). This is the defining property of a tensor product in general — it's the universal way to turn a bilinear pairing into a genuinely new object, rather than collapsing it down to a scalar the way §23.1's B(\vec u,\vec v) did.
Rank exactly 1 (when both \vec u,\vec v\neq\vec0): every row of \vec u\otimes\vec v is a scalar multiple of \vec v^T (row i is u_i\vec v^T), so the row space is 1-dimensional. This is precisely why §21.4's SVD sum, built from a sum of r rank-1 tensor products, reconstructs a rank-r matrix — and why §21.4's truncated SVD (keeping only the largest terms) gives the best low-rank approximation: it's discarding the smallest-magnitude rank-1 pieces from the sum.
Every bilinear form, revisited through tensors
§23.1's B(\vec x,\vec y)=\vec x^TA\vec y can be rewritten using the Frobenius inner product (§20.1's matrix inner product, \langle M,N\rangle=\operatorname{tr}(M^TN)):
B(\vec x,\vec y)=\vec x^TA\vec y=\operatorname{tr}(A^T(\vec x\otimes\vec y))=\langle A,\ \vec x\otimes\vec y\rangle
(using \vec x^TA\vec y=\operatorname{tr}(\vec x^TA\vec y), a 1\times1 "matrix" trivially equal to its own trace, then cycling the trace, §19.3). This says every bilinear form is literally an inner product against a tensor — the matrix A pairs with the tensor \vec x\otimes \vec y the same way two vectors pair via a dot product, just one dimension up.
Multilinear maps, briefly
A multilinear map takes k vector arguments and is linear in each one separately — bilinear (§23.1) is the case k=2; a linear functional (§23.0) is k=1. The determinant (§16.4), viewed as a function of A's n columns (fix all but one column, and \det is linear in the remaining one — this is exactly the row-operation scaling rule from §16.4, restated as multilinearity in the columns), is the single most important multilinear map this course has already used without naming it as one. General tensors of order k (needing k indices, generalizing vectors — order 1 — and matrices — order 2) are the natural home for such maps, and are the objects deep learning frameworks manipulate directly (a "tensor" in that context is exactly this: a multi-index array, with the same linearity structure).
Doing it in Python
import numpy as np
u = np.array([1., 2., 3.])
v = np.array([4., 5.])
T = np.outer(u, v) # u tensor v
print("u (x) v =")
for row in T.tolist():
print(row)
print("rank:", np.linalg.matrix_rank(T))
# Bilinearity check
u2 = np.array([0., 1., -1.])
lhs = np.outer(u + u2, v)
rhs = np.outer(u, v) + np.outer(u2, v)
print("\n(u+u2) (x) v == u(x)v + u2(x)v:", bool(np.allclose(lhs, rhs)))
u (x) v =
[4.0, 5.0]
[8.0, 10.0]
[12.0, 15.0]
rank: 1
(u+u2) (x) v == u(x)v + u2(x)v: True
Confirming the Frobenius-inner-product identity for a bilinear form:
import numpy as np
A = np.array([[2., 1.], [3., 4.]])
x, y = np.array([1., 2.]), np.array([-1., 3.])
direct = x @ A @ y
via_tensor = np.trace(A.T @ np.outer(x, y))
print("x^T A y =", direct)
print("<A, x (x) y> =", via_tensor)
print("match:", np.isclose(direct, via_tensor))
x^T A y = 19.0
<A, x (x) y> = 19.0
match: True
Worked example
Compute \vec u\otimes\vec v for \vec u=(2,-1), \vec v=(1,3,0), and verify it has rank 1.
\vec u\otimes\vec v=\begin{pmatrix}2\\-1\end{pmatrix}\begin{pmatrix}1&3&0\end{pmatrix}=\begin{pmatrix}2&6&0\\-1&-3&0\end{pmatrix}
\boxed{\vec u\otimes\vec v=\begin{pmatrix}2&6&0\\-1&-3&0\end{pmatrix}}
Sanity check. Row 2 is exactly -\tfrac12\timesrow 1 (-\tfrac12(2,6,0)=(-1,-3,0) ✓) — the rows are dependent, confirming rank 1 directly by inspection, matching the general claim that every nonzero tensor product of two vectors has rank exactly 1.
Your turn
1. Compute \vec e_1\otimes\vec e_2 for the standard basis vectors of \mathbb{R}^2. What familiar matrix is this?
2. Is \vec u\otimes\vec v=\vec v\otimes\vec u in general (for \vec u,\vec v\in\mathbb{R}^n, so both sides have matching shape)?
3. True or false: the sum of two rank-1 tensor products always has rank 1.
Solutions
1. \vec e_1\otimes\vec e_2=\begin{pmatrix}1\\0\end{pmatrix}\begin{pmatrix}0&1\end{pmatrix}=\begin{pmatrix}0&1\\0&0\end{pmatrix} — an elementary matrix with a single 1 (§17.2's "basis of M_{m\times n} made of single-entry matrices," now recognized as exactly the tensor products \vec e_i\otimes\vec e_j).
2. No, not in general. \vec u\otimes\vec v=\vec u\vec v^T while \vec v\otimes\vec u=\vec v\vec u^T=(\vec u\otimes\vec v)^T — they're transposes of each other, equal only when \vec u\otimes\vec v is itself symmetric (e.g. \vec u=\vec v, or \vec u,\vec v parallel).
3. False. E.g. \vec e_1\otimes\vec e_1+\vec e_2\otimes\vec e_2=\begin{pmatrix}1&0\\0&0\end{pmatrix}+\begin{pmatrix}0&0\\0&1\end{pmatrix}=I, which has rank 2, not 1. This is exactly why §21.4's SVD sum needs multiple rank-1 terms to reconstruct a higher-rank matrix — a single tensor product could never do it alone, and summing them generally increases rank rather than preserving it.
Check yourself in code
Compute \vec u\otimes\vec v for \vec u=(3,1), \vec v=(2,-1,4), and verify B(\vec x,\vec y)=\vec x^TA\vec y matches \langle A,\vec x\otimes\vec y\rangle for A=\vec u\otimes\vec v paired with itself conceptually — instead, verify the outer product directly against u_iv_j.
Print exactly this:
u (x) v =
[6, -3, 12]
[2, -1, 4]
rank: 1
import numpy as np
u = np.array([3, 1])
v = np.array([2, -1, 4])
T = np.outer(u, v)
print("u (x) v =")
for row in T.tolist():
print(row)
# print the rank of T
import numpy as np
u = np.array([3, 1])
v = np.array([2, -1, 4])
T = np.outer(u, v)
print("u (x) v =")
for row in T.tolist():
print(row)
print("rank:", np.linalg.matrix_rank(T))
The tensor product \vec u\otimes\vec v=\vec u\vec v^T is the outer product already used since §16.2 and §21.4, bilinear and always rank-1 (for nonzero factors). Every bilinear form is secretly a Frobenius inner product against a tensor, B(\vec x,\vec y)=\langle A,\vec x\otimes\vec y\rangle, and the determinant is this course's running example of a multilinear map beyond the bilinear case.
Next, closing this module: what changes — and what surprisingly doesn't — when the real numbers are replaced by a different number system entirely.