The Matrix Cookbook Kaare Brandt Petersen Michael Syskind Pedersen VERSION:JANUARY 5,2005 What is this?These pages are a collection of facts (identities,approxima- tions,inequalities,relations,...)about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference Disclaimer:The identities,approximations and relations presented here were obviously not invented but collected,borrowed and copied from a large amount of sources.These sources include similar but shorter notes found on the internet and appendices in books-see the references for a full list. Errors:Very likely there are errors,typos,and mistakes for which we apolo- gize and would be grateful to receive corrections at kbpaimm.dtu.dk. Its ongoing:The project of keeping a large repository of relations involving matrices is naturally ongoing and the version will be apparent from the date in the header. Suggestions:Your suggestion for additional content or elaboration of some topics is most welcome at kbpaimm.dtu.dk. Acknowledgements:We would like to thank the following for discussions, proofreading,extensive corrections and suggestions:Esben Hoegh-Rasmussen and Vasile Sima. Keywords:Matrix algebra,matrix relations,matrix identities,derivative of determinant,derivative of inverse matrix,differentiate a matrix. 1
The Matrix Cookbook Kaare Brandt Petersen Michael Syskind Pedersen Version: January 5, 2005 What is this? These pages are a collection of facts (identities, approximations, inequalities, relations, ...) about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference . Disclaimer: The identities, approximations and relations presented here were obviously not invented but collected, borrowed and copied from a large amount of sources. These sources include similar but shorter notes found on the internet and appendices in books - see the references for a full list. Errors: Very likely there are errors, typos, and mistakes for which we apologize and would be grateful to receive corrections at kbp@imm.dtu.dk. Its ongoing: The project of keeping a large repository of relations involving matrices is naturally ongoing and the version will be apparent from the date in the header. Suggestions: Your suggestion for additional content or elaboration of some topics is most welcome at kbp@imm.dtu.dk. Acknowledgements: We would like to thank the following for discussions, proofreading, extensive corrections and suggestions: Esben Hoegh-Rasmussen and Vasile Sima. Keywords: Matrix algebra, matrix relations, matrix identities, derivative of determinant, derivative of inverse matrix, differentiate a matrix. 1
CONTENTS CONTENTS Contents 1 Basics 1.1 Trace and Determinants l.2 The Special Case2x2..··.··············· 6 2 Derivatives 7 2.1 Derivatives of a Determinant.··················· 7 2.2 Derivatives of an Inverse······················ 8 2.3 Derivatives of Matrices,Vectors and Scalar Forms 2.4 Derivatives of Traces................... 11 2.5 Derivatives of Structured Matrices·.....···. 12 3 Inverses 14 3.1 Exact Relations... 14 3.2 Implication on Inverses.··············· 14 3.3 Approximations....·· 15 3.4 Generalized Inverse.... 15 3.5 Pseudo Inverse 15 4 Complex Matrices 17 4.1 Complex Derivatives.··.. 17 5 Decompositions 20 5.1 Eigenvalues and Eigenvectors····. 20 5.2 Singular Value Decomposition.....·.·.... 20 5.3 Triangular Decomposition.············: ”· 6 General Statistics and Probability 22 6.1 Moments of any distribution 22 62 Expectations·························· 2 7 Gaussians 24 7.1 One Dimensional....·········· 24 7.3 Moments 27 7.4 Miscellaneous.·················· 2 7.5 One Dimensional Mixture of Gaussians........ 29 7.6 Mixture of Gaussians 。。。。。。。。。。。。。。。 30 8 Miscellaneous 31 8.1 Functions and Series......... 4 1 8.2 Indices,Entries and Vectors.·.··.·. 8.3 Solutions to Systems of Equations.... 35 8.4 Block matrices..·.·...··.···· 8.5 Matrix Norms.······················· 37 8.6 Positive Definite and Semi-definite Matrices............ 38 PETERSEN PEDERSEN,THE MATRIX COOKBOOK (VERSION:JANUARY 5,2005),PAGE 2
CONTENTS CONTENTS Contents 1 Basics 5 1.1 Trace and Determinants . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 The Special Case 2x2 . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Derivatives 7 2.1 Derivatives of a Determinant . . . . . . . . . . . . . . . . . . . . 7 2.2 Derivatives of an Inverse . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Derivatives of Matrices, Vectors and Scalar Forms . . . . . . . . 9 2.4 Derivatives of Traces . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Derivatives of Structured Matrices . . . . . . . . . . . . . . . . . 12 3 Inverses 14 3.1 Exact Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2 Implication on Inverses . . . . . . . . . . . . . . . . . . . . . . . . 14 3.3 Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.4 Generalized Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5 Pseudo Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4 Complex Matrices 17 4.1 Complex Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 17 5 Decompositions 20 5.1 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . 20 5.2 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 20 5.3 Triangular Decomposition . . . . . . . . . . . . . . . . . . . . . . 21 6 General Statistics and Probability 22 6.1 Moments of any distribution . . . . . . . . . . . . . . . . . . . . . 22 6.2 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 7 Gaussians 24 7.1 One Dimensional . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 7.2 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 7.3 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 7.4 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 7.5 One Dimensional Mixture of Gaussians . . . . . . . . . . . . . . . 29 7.6 Mixture of Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . 30 8 Miscellaneous 31 8.1 Functions and Series . . . . . . . . . . . . . . . . . . . . . . . . . 31 8.2 Indices, Entries and Vectors . . . . . . . . . . . . . . . . . . . . . 32 8.3 Solutions to Systems of Equations . . . . . . . . . . . . . . . . . 35 8.4 Block matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 8.5 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 8.6 Positive Definite and Semi-definite Matrices . . . . . . . . . . . . 38 Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 2
CONTENTS CONTENTS 8.7 Integral Involving Dirac Delta Functions:··...:······. 39 8.8 Miscellaneous..···.···.:················· 40 A Proofs and Details 41 PETERSEN PEDERSEN,THE MATRIX COOKBOOK (VERSION:JANUARY 5,2005),PAGE 3
CONTENTS CONTENTS 8.7 Integral Involving Dirac Delta Functions . . . . . . . . . . . . . . 39 8.8 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 A Proofs and Details 41 Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 3
CONTENTS CONTENTS Notation and Nomenclature A Matrix Matrix indexed for some purpose Ai Matrix indexed for some purpose A可 Matrix indexed for some purpose A Matrix indexed for some purpose or The n.th power of a square matrix A-1 The inverse matrix of the matrix A A+ The pseudo inverse matrix of the matrix A A1/2 The square root of a matrix (if unique),not elementwise (A) The (i,j).th entry of the matrix A A The (i,j).th entry of the matrix A a Vector 男 Vector indexed for some purpose ai The i.th element of the vector a a Scalar 咒x Real part of a scalar z Real part of a vector Z Real part of a matrix Bz Imaginary part of a scalar Bz Imaginary part of a vector SZ Imaginary part of a matrix det(A） Determinant of A A Matrix norm (subscript if any denotes what norm) Transposed matrix A* Complex conjugated matrix AH Transposed and complex conjugated matrix AoB Hadamard (elementwise)product A⑧B Kronecker product 0 The null matrix.Zero in all entries. The identity matrix J的 The single-entry matrix,1 at (i,j)and zero elsewhere A positive definite matrix A diagonal matrix PETERSEN PEDERSEN,THE MATRIX COOKBOOK (VERSION:JANUARY 5,2005),PAGE 4
CONTENTS CONTENTS Notation and Nomenclature A Matrix Aij Matrix indexed for some purpose Ai Matrix indexed for some purpose Aij Matrix indexed for some purpose An Matrix indexed for some purpose or The n.th power of a square matrix A−1 The inverse matrix of the matrix A A+ The pseudo inverse matrix of the matrix A A1/2 The square root of a matrix (if unique), not elementwise (A)ij The (i, j).th entry of the matrix A Aij The (i, j).th entry of the matrix A a Vector ai Vector indexed for some purpose ai The i.th element of the vector a a Scalar <z Real part of a scalar <z Real part of a vector <Z Real part of a matrix =z Imaginary part of a scalar =z Imaginary part of a vector =Z Imaginary part of a matrix det(A) Determinant of A ||A|| Matrix norm (subscript if any denotes what norm) AT Transposed matrix A∗ Complex conjugated matrix AH Transposed and complex conjugated matrix A ◦ B Hadamard (elementwise) product A ⊗ B Kronecker product 0 The null matrix. Zero in all entries. I The identity matrix J ij The single-entry matrix, 1 at (i, j) and zero elsewhere Σ A positive definite matrix Λ A diagonal matrix Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 4
1 BASICS 1 Basics (AB)-1=B-1A-1 (ABC)-1=C-1B-1A-1 (AT)-1=(A-1)T (A+B)T=AT+BT (AB)T=BTAT (ABC..T=...CTBTAT (A)1=(A-1)H (A+B)H=AH+B (AB)H=BHAH (ABC...)=.CHBA 1.1 Trace and Determinants Tr(A)=∑AH=∑A, A:=eig(A) Tr(A)=Tr(AT) Tr(AB)=Tr(BA) Tr(A+B)=Tr(A)+Tr(B) Tr(ABC)=Tr(BCA)=Tr(CAB) det(a)=Πx: Ai=eig(A) det(AB)=det(A)det(B), if A and B are invertible det(A-1)=det(A) 1 PETERSEN PEDERSEN,THE MATRIX COOKBOOK (VERSION:JANUARY 5,2005),PAGE 5
1 BASICS 1 Basics (AB) −1 = B −1A−1 (ABC...) −1 = ...C−1B −1A−1 (AT ) −1 = (A−1 ) T (A + B) T = AT + B T (AB) T = B T AT (ABC...) T = ...CT B T AT (AH) −1 = (A−1 ) H (A + B) H = AH + B H (AB) H = B HAH (ABC...) H = ...CHB HAH 1.1 Trace and Determinants Tr(A) = X i Aii = X i λi , λi = eig(A) Tr(A) = Tr(AT ) Tr(AB) = Tr(BA) Tr(A + B) = Tr(A) + Tr(B) Tr(ABC) = Tr(BCA) = Tr(CAB) det(A) = Y i λi λi = eig(A) det(AB) = det(A) det(B), if A and B are invertible det(A−1 ) = 1 det(A) Petersen & Pedersen, The Matrix Cookbook (Version: January 5, 2005), Page 5