This book was distributed courtesy of:

For your own Unlimited Reading and FREE eBooks today, visit:

http://www.Free-eBooks.net

Share this eBook with anyone and everyone automatically by selecting any of the options below:

To show your appreciation to the author and help others have wonderful reading experiences and find helpful information too, we'd be very grateful if you'd kindly

post your comments for this book here.

COPYRIGHT INFORMATION

Free-eBooks.net respects the intellectual property of others. When a book's copyright owner submits their work to Free-eBooks.net, they are granting us permission to distribute such material. Unless otherwise stated in this book, this permission is not passed onto others. As such, redistributing this book without the copyright owner's permission can constitute copyright infringement. If you believe that your work has been used in a manner that constitutes copyright infringement, please follow our Notice and Procedure for Making Claims of Copyright Infringement as seen in our Terms of Service here:

http://www.free-ebooks.net/tos.html

Basics of Algebra, Topology, and Differential Calculus

Jean Gallier Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104, USA e-mail: jean@cis.upenn.edu

c Jean Gallier

December 3, 2013

2

Contents

1 Introduction 9

2 Vector Spaces, Bases, Linear Maps 11

2.1 Groups, Rings, and Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.3 Linear Independence, Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4 Bases of a Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5 Linear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.6 Quotient Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Matrices and Linear Maps 45

3.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.2 Haar Basis Vectors and a Glimpse at Wavelets . . . . . . . . . . . . . . . . . 61

3.3 The Effect of a Change of Bases on Matrices . . . . . . . . . . . . . . . . . . 77

3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4 Direct Sums, The Dual Space, Duality 81

4.1 Sums, Direct Sums, Direct Products . . . . . . . . . . . . . . . . . . . . . . 81

4.2 The Dual SpaceE∗ and Linear Forms . . . . . . . . . . . . . . . . . . . . . . 94

4.3 Hyperplanes and Linear Forms . . . . . . . . . . . . . . . . . . . . . . . . . . 107

4.4 Transpose of a Linear Map and of a Matrix . . . . . . . . . . . . . . . . . . . 108

4.5 The Four Fundamental Subspaces . . . . . . . . . . . . . . . . . . . . . . . . 117

4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5 Determinants 123

5.1 Permutations, Signature of a Permutation . . . . . . . . . . . . . . . . . . . 123

5.2 Alternating Multilinear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . 126

5.3 Definition of a Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

5.4 Inverse Matrices and Determinants . . . . . . . . . . . . . . . . . . . . . . . 136

5.5 Systems of Linear Equations and Determinants . . . . . . . . . . . . . . . . 140

5.6 Determinant of a Linear Map . . . . . . . . . . . . . . . . . . . . . . . . . . 141

5.7 The Cayley–Hamilton Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 141

3 5.8 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

6 Gaussian Elimination, LU, Cholesky, Echelon Form 149

6.1 Motivating Example: Curve Interpolation . . . . . . . . . . . . . . . . . . . 149

6.2 Gaussian Elimination andLU-Factorization . . . . . . . . . . . . . . . . . . 153

6.3 Gaussian Elimination of Tridiagonal Matrices . . . . . . . . . . . . . . . . . 175

6.4 SPD Matrices and the Cholesky Decomposition . . . . . . . . . . . . . . . . 178

6.5 Reduced Row Echelon Form . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

6.6 Transvections and Dilatations . . . . . . . . . . . . . . . . . . . . . . . . . . 197

6.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

7 Vector Norms and Matrix Norms 205

7.1 Normed Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

7.2 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

7.3 Condition Numbers of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 223

7.4 An Application of Norms: Inconsistent Linear Systems . . . . . . . . . . . . 231

7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

8 Iterative Methods for Solving Linear Systems 235

8.1 Convergence of Sequences of Vectors and Matrices . . . . . . . . . . . . . . . 235

8.2 Convergence of Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . 238

8.3 Methods of Jacobi, Gauss-Seidel, and Relaxation . . . . . . . . . . . . . . . 240

8.4 Convergence of the Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 245

8.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

9 Euclidean Spaces 253

9.1 Inner Products, Euclidean Spaces . . . . . . . . . . . . . . . . . . . . . . . . 253

9.2 Orthogonality, Duality, Adjoint of a Linear Map . . . . . . . . . . . . . . . . 261

9.3 Linear Isometries (Orthogonal Transformations) . . . . . . . . . . . . . . . . 271

9.4 The Orthogonal Group, Orthogonal Matrices . . . . . . . . . . . . . . . . . . 274

9.5 QR-Decomposition for Invertible Matrices . . . . . . . . . . . . . . . . . . . 276

9.6 Some Applications of Euclidean Geometry . . . . . . . . . . . . . . . . . . . 278

9.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

10 QR-Decomposition for Arbitrary Matrices 281

10.1 Orthogonal Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

10.2 QR-Decomposition Using Householder Matrices . . . . . . . . . . . . . . . . 284

10.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

11 Hermitian Spaces 291

11.1 Hermitian Spaces, Pre-Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . 291

11.2 Orthogonality, Duality, Adjoint of a Linear Map . . . . . . . . . . . . . . . . 300

11.3 Linear Isometries (Also Called Unitary Transformations) . . . . . . . . . . . 304

11.4 The Unitary Group, Unitary Matrices . . . . . . . . . . . . . . . . . . . . . . 306

11.5 Orthogonal Projections and Involutions . . . . . . . . . . . . . . . . . . . . . 308

11.6 Dual Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

11.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

12 Eigenvectors and Eigenvalues 317

12.1 Eigenvectors and Eigenvalues of a Linear Map . . . . . . . . . . . . . . . . . 317

12.2 Reduction to Upper Triangular Form . . . . . . . . . . . . . . . . . . . . . . 324

12.3 Location of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326

12.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

13 Spectral Theorems 331

13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

13.2 Normal Linear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

13.3 Self-Adjoint and Other Special Linear Maps . . . . . . . . . . . . . . . . . . 340

13.4 Normal and Other Special Matrices . . . . . . . . . . . . . . . . . . . . . . . 346

13.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350

14 Bilinear Forms and Their Geometries 351

14.1 Bilinear Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351

14.2 Sesquilinear Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358

14.3 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362

14.4 Adjoint of a Linear Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366

14.5 Isometries Associated with Sesquilinear Forms . . . . . . . . . . . . . . . . . 369

14.6 Totally Isotropic Subspaces. Witt Decomposition . . . . . . . . . . . . . . . 372

14.7 Witt’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385

14.8 Symplectic Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389

14.9 Orthogonal Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393

15 Introduction to The Finite Elements Method 401

15.1 A One-Dimensional Problem: Bending of a Beam . . . . . . . . . . . . . . . 401

15.2 A Two-Dimensional Problem: An Elastic Membrane . . . . . . . . . . . . . . 412

15.3 Time-Dependent Boundary Problems . . . . . . . . . . . . . . . . . . . . . . 415

16 Singular Value Decomposition and Polar Form 423

16.1 Singular Value Decomposition for Square Matrices . . . . . . . . . . . . . . . 423

16.2 Singular Value Decomposition for Rectangular Matrices . . . . . . . . . . . . 431

16.3 Ky Fan Norms and Schatten Norms . . . . . . . . . . . . . . . . . . . . . . . 434

16.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435

17 Applications of SVD and Pseudo-inverses 437

17.1 Least Squares Problems and the Pseudo-inverse . . . . . . . . . . . . . . . . 437

17.2 Data Compression and SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . 445

17.3 Principal Components Analysis (PCA) . . . . . . . . . . . . . . . . . . . . . 446

17.4 Best Affine Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454

18 Quadratic Optimization Problems 459

18.1 Quadratic Optimization: The Positive Definite Case . . . . . . . . . . . . . . 459

18.2 Quadratic Optimization: The General Case . . . . . . . . . . . . . . . . . . 467

18.3 Maximizing a Quadratic Function on the Unit Sphere . . . . . . . . . . . . . 471

18.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476

19 Basics of Affine Geometry 477

19.1 Affine Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477

19.2 Examples of Affine Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484

19.3 Chasles’s Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486

19.4 Affine Combinations, Barycenters . . . . . . . . . . . . . . . . . . . . . . . . 487

19.5 Affine Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490

19.6 Affine Independence and Affine Frames . . . . . . . . . . . . . . . . . . . . . 495

19.7 Affine Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500

19.8 Affine Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507

19.9 Affine Geometry: A Glimpse . . . . . . . . . . . . . . . . . . . . . . . . . . . 509

19.10Affine Hyperplanes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512

19.11Intersection of Affine Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 514

19.12Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517

20 Polynomials, Ideals and PID’s 531

20.1 Multisets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531

20.2 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532

20.3 Euclidean Division of Polynomials . . . . . . . . . . . . . . . . . . . . . . . . 538

20.4 Ideals, PID’s, and Greatest Common Divisors . . . . . . . . . . . . . . . . . 540

20.5 Factorization and Irreducible Factors inK[X] . . . . . . . . . . . . . . . . . 548

20.6 Roots of Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552

20.7 Polynomial Interpolation (Lagrange, Newton,

Hermite) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559

21 UFD’s, Noetherian Rings, Hilbert’s Basis Theorem 565

21.1 Unique Factorization Domains (Factorial Rings) . . . . . . . . . . . . . . . . 565

21.2 The Chinese Remainder Theorem . . . . . . . . . . . . . . . . . . . . . . . . 579

21.3 Noetherian Rings and Hilbert’s Basis Theorem . . . . . . . . . . . . . . . . . 584

21.4 Futher Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588

22 Annihilating Polynomials; Primary Decomposition 589

22.1 Annihilating Polynomials and the Minimal Polynomial . . . . . . . . . . . . 589

22.2 Minimal Polynomials of Diagonalizable Linear Maps . . . . . . . . . . . . . . 591

22.3 The Primary Decomposition Theorem . . . . . . . . . . . . . . . . . . . . . . 595

22.4 Nilpotent Linear Maps and Jordan Form . . . . . . . . . . . . . . . . . . . . 600

23 Tensor Algebras 605

23.1 Tensors Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605

23.2 Bases of Tensor Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613

23.3 Some Useful Isomorphisms for Tensor Products . . . . . . . . . . . . . . . . 615

23.4 Duality for Tensor Products . . . . . . . . . . . . . . . . . . . . . . . . . . . 616

23.5 Tensor Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618

23.6 Symmetric Tensor Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624

23.7 Bases of Symmetric Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . 627

23.8 Some Useful Isomorphisms for Symmetric Powers . . . . . . . . . . . . . . . 630

23.9 Duality for Symmetric Powers . . . . . . . . . . . . . . . . . . . . . . . . . . 630

23.10Symmetric Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632

23.11Exterior Tensor Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634

23.12Bases of Exterior Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638

23.13Some Useful Isomorphisms for Exterior Powers . . . . . . . . . . . . . . . . . 640

23.14Duality for Exterior Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . 641

23.15Exterior Algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643

23.16The Hodge∗-Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646

23.17Testing Decomposability; Left and Right Hooks . . . . . . . . . . . . . . . . 648

23.18Vector-Valued Alternating Forms . . . . . . . . . . . . . . . . . . . . . . . . 655

23.19The Pfaffian Polynomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658

24 Introduction to Modules; Modules over a PID 663

24.1 Modules over a Commutative Ring . . . . . . . . . . . . . . . . . . . . . . . 663

24.2 Finite Presentations of Modules . . . . . . . . . . . . . . . . . . . . . . . . . 671

24.3 Tensor Products of Modules over a Commutative Ring . . . . . . . . . . . . 676

24.4 Extension of the Ring of Scalars . . . . . . . . . . . . . . . . . . . . . . . . . 679

24.5 The Torsion Module Associated With An Endomorphism . . . . . . . . . . . 682

24.6 Torsion Modules over a PID; Primary Decomposition . . . . . . . . . . . . . 690

24.7 Finitely Generated Modules over a PID . . . . . . . . . . . . . . . . . . . . . 695

25 Normal Forms; The Rational Canonical Form 711

25.1 The Rational Canonical Form . . . . . . . . . . . . . . . . . . . . . . . . . . 711

25.2 The Rational Canonical Form, Second Version . . . . . . . . . . . . . . . . . 716

25.3 The Jordan Form Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . 717

25.4 The Smith Normal Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 719

26 Topology 733

26.1 Metric Spaces and Normed Vector Spaces . . . . . . . . . . . . . . . . . . . . 733

26.2 Topological Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737

26.3 Continuous Functions, Limits . . . . . . . . . . . . . . . . . . . . . . . . . . 742

26.4 Connected Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747

26.5 Compact Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753

26.6 Continuous Linear and Multilinear Maps . . . . . . . . . . . . . . . . . . . . 767

26.7 Normed Affine Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772

26.8 Futher Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 772

27 A Detour On Fractals 773 27.1 Iterated Function Systems and Fractals . . . . . . . . . . . . . . . . . . . . . 773

28 Differential Calculus 781

28.1 Directional Derivatives, Total Derivatives . . . . . . . . . . . . . . . . . . . . 781

28.2 Jacobian Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 789

28.3 The Implicit and The Inverse Function Theorems . . . . . . . . . . . . . . . 794

28.4 Tangent Spaces and Differentials . . . . . . . . . . . . . . . . . . . . . . . . . 798

28.5 Second-Order and Higher-Order Derivatives . . . . . . . . . . . . . . . . . . 799

28.6 Taylor’s formula, Fa`a di Bruno’s formula . . . . . . . . . . . . . . . . . . . . 805

28.7 Vector Fields, Covariant Derivatives, Lie Brackets . . . . . . . . . . . . . . . 809

28.8 Futher Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 811

29 Extrema of Real-Valued Functions 813

29.1 Local Extrema and Lagrange Multipliers . . . . . . . . . . . . . . . . . . . . 813

29.2 Using Second Derivatives to Find Extrema . . . . . . . . . . . . . . . . . . . 822

29.3 Using Convexity to Find Extrema . . . . . . . . . . . . . . . . . . . . . . . . 825

29.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 833

30 Newton’s Method and its Generalizations 835

30.1 Newton’s Method for Real Functions of a Real Argument . . . . . . . . . . . 835

30.2 Generalizations of Newton’s Method . . . . . . . . . . . . . . . . . . . . . . 836

30.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 842

31 Appendix: Zorn’s Lemma; Some Applications 843

31.1 Statement of Zorn’s Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . 843

31.2 Proof of the Existence of a Basis in a Vector Space . . . . . . . . . . . . . . 844

31.3 Existence of Maximal Proper Ideals . . . . . . . . . . . . . . . . . . . . . . . 845

Bibliography 845

Chapter 1 Introduction

10 CHAPTER 1. INTRODUCTION

Chapter 2 Vector Spaces, Bases, Linear Maps

2.1 Groups, Rings, and Fields

In the following three chapters, the basic algebraic structures (groups, rings, fields, vector spaces) are reviewed, with a major emphasis on vector spaces. Basic notions of linear algebra such as vector spaces, subspaces, linear combinations, linear independence, bases, quotient spaces, linear maps, matrices, change of bases, direct sums, linear forms, dual spaces, hyperplanes, transpose of a linear maps, are reviewed.

The set R of real numbers has two operations +: R×R→ R (addition) and∗: R×R→ R (multiplication) satisfying properties that make R into an abelian group under +, and R− {0} = R∗ into an abelian group under∗. Recall the definition of a group.

Definition 2.1. A group is a setG equipped with a binary operation·:G×G→G that associates an elementa·b∈G to every pair of elementsa, b∈G, and having the following properties:· is associative, has an identity elemente∈G, and every element inG is invertible (w.r.t.·). More explicitly, this means that the following equations hold for alla, b, c∈G:

(G1) a· (b·c) = (a·b)·c.

(G2) a·e =e·a =a.

(associativity); (identity); (G3) For everya∈G, there is somea−1 G such thata·a−1 =a−1 a =e∈ · A groupG is abelian (or commutative) if

a·b =b·a

for alla, b∈G.

(inverse).

A set M together with an operation·:M×M→M and an elemente satisfying only conditions (G1) and (G2) is called a monoid. For example, the set N ={0,1, . . . , n, . . .} of natural numbers is a (commutative) monoid under addition. However, it is not a group.

Some examples of groups are given below.

Example 2.1.

1. The set Z ={. . . ,−n, . . . ,−1,0,1, . . . , n, . . .} of integers is a group under addition, with identity element 0. However, Z∗ = Z− {0} is not a group under multiplication.

2. The set Q of rational numbers (fractionsp/q withp, q∈ Z andq = 0) is a group under addition, with identity element 0. The set Q∗ = Q− {0} is also a group under multiplication, with identity element 1.

3. Similarly, the sets R of real numbers and C of complex numbers are groups under addition (with identity element 0), and R∗ = R− {0} and C∗ = C− {0} are groups under multiplication (with identity element 1).

4. The sets Rn and Cn ofn-tuples of real or complex numbers are groups under componentwise addition:

(x_1, . . . , xn) + (y_1, . . . , yn) = (x_1 +yn, . . . , xn +yn),

with identity element (0, . . . ,0). All these groups are abelian.

5. Given any nonempty set S, the set of bijectionsf :S→S, also called permutations ofS, is a group under function composition (i.e., the multiplication off andg is the compositiongæf), with identity element the identity function idS. This group is not abelian as soon asS has more than two elements.

6. The set ofn×n matrices with real (or complex) coefficients is a group under addition of matrices, with identity element the null matrix. It is denoted by Mn(R) (or Mn(C)).

7. The set R[X] of polynomials in one variable with real coefficients is a group under addition of polynomials.

8. The set of n×n invertible matrices with real (or complex) coefficients is a group under matrix multiplication, with identity element the identity matrixIn. This group is called the general linear group and is usually denoted by GL(n,R) (or GL(n,C)).

9. The set of n×n invertible matrices with real (or complex) coefficients and determinant +1 is a group under matrix multiplication, with identity element the identity matrix In. This group is called the special linear group and is usually denoted by SL(n,R) (or SL(n,C)).

10. The set of n×n invertible matrices with real coefficients such thatRR =In and of determinant +1 is a group called the special orthogonal group and is usually denoted by SO(n) (whereR is the transpose of the matrixR, i.e., the rows ofR are the columns ofR). It corresponds to the rotations in Rn.

11. Given an open interval ]a, b[, the setC(]a, b[) of continuous functionsf : ]a, b[→ R is a group under the operationf +g defined such that

(f +g)(x) =f(x) +g(x)

for allx∈]a, b[.

It is customary to denote the operation of an abelian groupG by +, in which case the inversea−1 of an elementa∈G is denoted by−a.

The identity element of a group is unique. In fact, we can prove a more general Fact: Fact 1. If a binary operation·:M×M→M is associative and ife∈M is a left identity ande∈M is a right identity, which means that

e·a =a for all a∈M (G2l)

and a·e =a for all a∈M, (G2r) thene =e .

Proof. If we leta =e in equation (G2l), we get

e·e =e ,

and if we leta =e in equation (G2r), we get

e·e =e ,

and thus e =e·e =e , as claimed.

Fact 1 implies that the identity element of a monoid is unique, and since every group is a monoid, the identity element of a group is unique. Furthermore, every element in a group has a unique inverse. This is a consequence of a slightly more general fact:

Fact 2. In a monoidM with identity elemente, if some elementa∈M has some left inverse a∈M and some right inversea∈M, which means that

a·a =e (G3l)

and a·a =e, (G3r) thena =a .

Proof. Using (G3l) and the fact thate is an identity element, we have

(a·a)·a =e·a =a .

Similarly, Using (G3r) and the fact that e is an identity element, we have a· (a·a ) =a·e =a .

However, sinceM is monoid, the operation· is associative, so

a =a· (a·a ) = (a·a)·a =a , as claimed.

Remark: Axioms (G2) and (G3) can be weakened a bit by requiring only (G2r) (the existence of a right identity) and (G3r) (the existence of a right inverse for every element) (or (G2l) and (G3l)). It is a good exercise to prove that the group axioms (G2) and (G3) follow from (G2r) and (G3r).

If a group G has a finite numbern of elements, we say thatG is a group of ordern. If G is infinite, we say thatG has infinite order. The order of a group is usually denoted by |G| (ifG is finite).

Given a group,G, for any two subsetsR, S⊆G, we let

RS ={r·s|r∈R, s∈S}.

In particular, for anyg∈G, ifR ={g}, we write

gS ={g·s|s∈S}

and similarly, ifS ={g}, we write

Rg ={r·g|r∈R}.

From now on, we will drop the multiplication sign and writeg_1g_2 forg1·g_2.

For any g∈G, defineLg, the left translation byg, byLg(a) =ga, for alla∈G, and Rg, the right translation byg, byRg(a) =ag, for alla∈G. Observe thatLg andRg are bijections. We show this forLg, the proof forRg being similar.

If Lg(a) =Lg(b), thenga =gb, and multiplying on the left byg−1, we geta =b, soLg injective. For anyb∈G, we haveLg(g−1b) =gg−1b =b, soLg is surjective. Therefore,Lg is bijective.

Definition 2.2. Given a groupG, a subsetH ofG is a subgroup ofG iff (1) The identity element,e, ofG also belongs toH (e∈H);

(2) For allh_1, h2∈H, we haveh_1h2∈H;

(3) For allh^1 H.∈H, we haveh− ∈

The proof of the following proposition is left as an exercise.

Proposition 2.1. Given a groupG, a subsetH⊆G is a subgroup ofG iffH is nonempty and wheneverh_1, h2∈H, thenh_1h−1 H.

2 ∈

If the group G is finite, then the following criterion can be used.

Proposition 2.2. Given a finite groupG, a subset,H⊆G is a subgroup ofG iff (1) e∈H;

(2) H is closed under multiplication.

Proof. We just have to prove that condition (3) of Definition 2.2 holds. For anya∈H, since the left translationLa is bijective, its restriction toH is injective, and sinceH is finite, it is also bijective. Sincee∈H, there is a uniqueb∈ H such thatLa(b) =ab =e. However, if a−1 is the inverse ofa inG, we also haveLa(a− ) =aa−1 =e, and by injectivity ofLa, we havea−1 =b∈H.

Definition 2.3. IfH is a subgroup ofG andg∈G is any element, the sets of the formgH are called left cosets ofH inG and the sets of the formHg are called right cosets ofH in G.

The left cosets (resp. follows: For allg_1, g2∈G, right cosets) ofH induce an equivalence relation,∼, defined as g1∼g_2 iff g_1H =g_2H (resp.g1∼g_2 iffHg_1 =Hg_2). Obviously,∼ is an equivalence relation. Now, we claim that

g

1

H

=

g

2

H

iff

g

1

2

g

1

H

=

H

iff

g

− 1

g1∈H.

2

If we apply the bijection

L

g

−1 to bothg_1H andg_2H we getLg− (g_1H) =g− 1

2^2

L

1H and2 g 1 1H =H. Ifg− 1 1H =H, since 1∈H, we getg− (g_2H) =H, sog_1H =g_2H iffg− g g2 2 2

g− 1 1∈H. Conversely, ifg− 1 1∈H, sinceH is a group, the left translationLg−1 is a2 g g

2 2 g_1

bijection of

H

, so

g

− 1 1H =H. Thus,g− 1 1H =H iffg^1

2

g1∈H.2 g g

2

It follows that the equivalence class of an element g∈G is the cosetgH (resp. Hg). SinceLg is a bijection betweenH andgH, the cosetsgH all have the same cardinality. The mapLg−1æRg is a bijection between the left cosetgH and the right cosetHg, so they also

have the same cardinality. Since the distinct cosetsgH form a partition ofG, we obtain the following fact:

Proposition 2.3. (Lagrange) For any finite groupG and any subgroupH ofG, the order h ofH divides the ordern ofG.

The ration/h is denoted by (G :H) and is called the index ofH inG. The index (G :H) is the number of left (and right) cosets ofH inG. Proposition 2.3 can be stated as |G| = (G :H)|H|.

The set of left cosets of H inG (which, in general, is not a group) is denotedG/H. The “points” ofG/H are obtained by “collapsing” all the elements in a coset into a single element.

It is tempting to define a multiplication operation on left cosets (or right cosets) by setting

(g_1H)(g_2H) = (g_1g_2)H,

but this operation is not well defined in general, unless the subgroupH possesses a special property. This property is typical of the kernels of group homomorphisms, so we are led to Definition 2.4. Given any two groups,G, G , a functionÕ:G→G is a homomorphism iff

Õ(g_1g_2) =Õ(g_1)Õ(g_2), for allg_1, g2∈G.

Takingg_1 =g_2 =e (inG), we see that Õ(e) =e , and takingg_1 =g andg_2 =g−1, we see that

Õ(g−1) =Õ(g)−1.

If Õ:G→G andψ:G→G are group homomorphisms, thenψæÕ:G→G is also a homomorphism. IfÕ:G→G is a homomorphism of groups andH⊆G andH⊆G are two subgroups, then it is easily checked that

ImH =Õ(H) ={Õ(g)|g∈H} is a subgroup ofG (ImH is called the image ofH byÕ) and

Õ−1(H ) ={g∈G|Õ(g)∈H} is a subgroup ofG. In particular, whenH ={e}, we obtain the kernel, KerÕ, ofÕ. Thus, KerÕ ={g∈G|Õ(g) =e}.

It is immediately verified that Õ:G→G is injective iff KerÕ ={e}. (We also write KerÕ = (0).) We say thatÕ is an isomorphism if there is a homomorphism,ψ:G→G, so that

ψæÕ = idG and Õæψ = idG. In this case,ψ is unique and it is denotedÕ−1. WhenÕ is an isomorphism we say the the groupsG andG are isomorphic. It is easy to see that a bijective hmomorphism is an isomorphism. WhenG =G, a group isomorphism is called an automorphism.

The left translationsLg and the right translationsRg are group isomorphisms. We claim thatH = KerÕ satisfies the following property:

gH =Hg, for allg∈G. (∗)

First, note that (∗) is equivalent to

gHg−1 =H, for allg∈G,

and the above is equivalent to

gHg−1 H, for allg∈G. (∗∗)⊆

This is becausegHg−1 H impliesH⊆g−1Hg, and this for allg∈G. But,⊆

Õ(ghg−1) =Õ(g)Õ(h)Õ(g−1) =Õ(g)e Õ(g)−1 =Õ(g)Õ(g)−1 =e ,

for allh∈H = KerÕ and allg∈G. Thus, by definition ofH = KerÕ, we havegHg−1 H.⊆ Definition 2.5. For any group,G, a subgroup,N⊆G, is a normal subgroup ofG iff

gNg−1 =N, for allg∈G.

This is denoted byN G. Observe that ifG is abelian, then every subgroup ofG is normal.

If N is a normal subgroup ofG, the equivalence relation induced by left cosets is the same as the equivalence induced by right cosets. Furthermore, this equivalence relation,∼, is a congruence, which means that: For allg_1, g_2, g_1, g2∈G,

(1) Ifg_1N =g_1N andg_2N =g_2N, theng_1g_2N =g_1g_2N, and (2) Ifg_1N =g_2N, theng−1 N =g−1 N.1 2

As a consequence, we can define a group structure on the setG/∼ of equivalence classes modulo∼, by setting

(g_1N)(g_2N) = (g_1g_2)N.

This group is denotedG/N and called the quotient ofG byN. The equivalence class,gN, of an elementg∈G is also denotedg (or [g]). The mapπ:G→G/N given by

π(g) =g =gN, is clearly a group homomorphism called the canonical projection.

Given a homomorphism of groups,Õ:G→G , we easily check that the groupsG/KerÕ and ImÕ =Õ(G) are isomorphic. This is often called the first isomorphism theorem.

A useful way to construct groups is the direct product construction. Given two groupsG anH, we letG×H be the Cartestian product of the setsG andH with the multiplication operation· given by

(g_1, h_1)· (g_2, h_2) = (g_1g_2, h_1h_2).

It is immediately verified thatG×H is a group. Similarly, given anyn groupsG_1, . . . , Gn, we can define the direct productG1× · · · ×Gn is a similar way.

IfG is an abelian group andH_1, . . . , Hn are subgroups ofG, the situation is simpler. Consider the map

a:H1× · · · ×Hn→G given by

a(h_1, . . . , hn) =h_1 +· · · +hn,

using + for the operation of the group G. It is easy to verify thata is a group homomorphism, so its image is a subgroup ofG denoted byH_1 +· · ·+Hn, and called the sum of the groups Hi. The following proposition will be needed.

Proposition 2.4. Given an abelian groupG, ifH_1 andH_2 are any subgroups ofG such thatH1∩H_2 ={0}, then the mapa is an isomorphism

a:H1×H2→H_1 +H_2.

Proof. The map is surjective by definition, so we just have to check that it is injective. For this, we show that Kera ={(0,0)}. We havea(a_1, a_2) = 0 iffa_1+a_2 = 0 iffa_1 =−a_2. Since a1∈H_1 anda2∈H_2, we see thata_1, a2∈H1∩H_2 ={0}, soa_1 =a_2 = 0, which proves that Kera ={(0,0)}.

Under the conditions of Proposition 2.4, namely H1∩H_2 ={0}, the groupH_1 +H_2 is called the direct sum ofH_1 andH_2; it is denoted byH1⊕H_2, and we have an isomorphism H

1

×

H

=

2∼H1⊕H_2.

The groups Z,Q,R, C, and Mn(R) are more than an abelian groups, they are also commutative rings. Furthermore, Q, R, and C are fields. We now introduce rings and fields. Definition 2.6. A ring is a setA equipped with two operations +:A×A→A (called addition) and∗:A×A→A (called multiplication) having the following properties:

(R1) A is an abelian group w.r.t. +;

(R2)∗ is associative and has an identity element 1∈A; (R3)∗ is distributive w.r.t. +.

The identity element for addition is denoted 0, and the additive inverse of a∈ A is denoted by−a. More explicitly, the axioms of a ring are the following equations which hold for alla, b, c∈A:

a + (b +c) = (a +b) +c (associativity of +) (2.1) a +b =b +a (commutativity of +) (2.2) a + 0 = 0 +a =a (zero) (2.3) a + (−a) = (−a) +a = 0 (additive inverse) (2.4) a∗ (b∗c) = (a∗b)∗c (associativity of∗) (2.5) a∗ 1 = 1∗a =a (identity for∗) (2.6) (a +b)∗c = (a∗c) + (b∗c) (distributivity) (2.7) a∗ (b +c) = (a∗b) + (a∗c) (distributivity) (2.8)

The ringA is commutative if a∗b =b∗a for alla, b∈A.

From (2.7) and (2.8), we easily obtain

a∗ 0 = 0∗a = 0 (2.9) a∗ (−b) = (−a)∗b =−(a∗b). (2.10)

Note that (2.9) implies that if 1 = 0, then a = 0 for alla∈A, and thus,A ={0}. The ringA ={0} is called the trivial ring. A ring for which 1 = 0 is called nontrivial. The multiplicationa∗b of two elementsa, b∈A is often denoted byab.

Example 2.2.

1. The additive groups Z,Q,R,C, are commutative rings.

2. The group R[X] of polynomials in one variable with real coefficients is a ring under multiplication of polynomials. It is a commutative ring.

3. The group ofn×n matrices Mn(R) is a ring under matrix multiplication. However, it is not a commutative ring.

4. The groupC(]a, b[) of continuous functionsf : ]a, b[→ R is a ring under the operation f·g defined such that

(f·g)(x) =f(x)g(x) for allx∈]a, b[.

When ab = 0 withb = 0, we say thata is a zero divisor. A ringA is an integral domain (or an entire ring) if 0 = 1,A is commutative, andab = 0 implies thata = 0 orb = 0, for alla, b∈A. In other words, an integral domain is a nontrivial commutative ring with no zero divisors besides 0.

Example 2.3.

1. The rings Z,Q,R,C, are integral domains.

2. The ringR[X] of polynomials in one variable with real coefficients is an integral domain.

3.

4. For any positive integer,p∈ N, define a relation on Z, denotedm≡n (modp), as follows:

m≡n (modp) iff m−n =kp for somek∈ Z.

The reader will easily check that this is an equivalence relation, and, moreover, it is compatible with respect to addition and multiplication, which means that ifm1≡n_1

(modp) andm2≡n_2 (modp), thenm_1 +m2≡n_1 +n_2 (modp) andm_1m2≡n1n_2

(modp). Consequently, we can define an addition operation and a multiplication operation of the set of equivalence classes (modp):

[m] + [n] = [m +n]

and [m]· [n] = [mn].

Again, the reader will easily check that the ring axioms are satisfied, with [0] as zero and [1] as multiplicative unit. The resulting ring is denoted by Z/pZ.1 Observe that ifp is composite, then this ring has zero-divisors. For example, ifp = 4, then we have

2· 2≡ 0 (mod 4).

However, the reader should prove that Z/pZ is an integral domain ifp is prime (in fact, it is a field).

5. The ring ofn×n matrices Mn(R) is not an integral domain. It has zero divisors.

A homomorphism between rings is a mapping preserving addition and multiplication (and 0 and 1).

1The notationZp is sometimes used instead ofZ/pZ but it clashes with the notation for the p-adic integers so we prefer not to use it.

Definition 2.7. Given two ringsA andB, a homomorphism betweenA andB is a function h:A→B satisfying the following conditions for allx, y∈A:

h (x +y) =h(x) +h(y) h(xy) =h(x)h(y) h(0) = 0

h(1) = 1.

Actually, becauseB is a group under addition,h(0) = 0 follows from

h(x +y) =h(x) +h(y). Example 2.4. 1. IfA is a ring, for any integern∈ Z, for anya∈A, we definen·a by n·a =a +· · · +a

n

ifn≥ 0 (with 0·a = 0) and n·a =−(−n)·a ifn < 0. Then, the maph: Z→A given by

h(n) =n· 1A

is a ring homomorphism (where 1A is the multiplicative identity ofA). 2. Given any realλ∈ R, the evaluation mapηλ: R[X]→ R defined by

ηλ(f(X)) =f(λ)

for every polynomialf(X)∈ R[X] is a ring homomorphism.

A ring homomorphism h:A→B is an isomorphism iff there is a homomorphismg:B→ A such thatgæf = idA andfæg = idB. Then,g is unique and denoted byh−1. It is easy to show that a bijective ring homomorphismh:A→B is an isomorphism. An isomorphism from a ring to itself is called an automorphism.

Given a ring A, a subsetA ofA is a subring ofA ifA is a subgroup ofA (under addition), is closed under multiplication, and contains 1. Ifh:A→ B is a homomorphism of rings, then for any subringA , the imageh(A ) is a subring ofB, and for any subringB ofB, the inverse imageh−1(B ) is a subring ofA.

A field is a commutative ring K for whichA− {0} is a group under multiplication. Definition 2.8. A setK is a field if it is a ring and the following properties hold: (F1) 0 = 1; (F2) K∗ =K− {0} is a group w.r.t.∗ (i.e., everya = 0 has an inverse w.r.t.∗); (F3)∗ is commutative.

If∗ is not commutative but (F1) and (F2) hold, we say that we have a skew field (or noncommutative field).

Note that we are assuming that the operation∗ of a field is commutative. This convention is not universally adopted, but since∗ will be commutative for most fields we will encounter, we may as well include this condition in the definition.

Example 2.5.

1. The rings Q, R, and C are fields.

2. The set of (formal) fractionsf(X)/g(X) of polynomialsf(X), g(X)∈ R[X], where g(X) is not the null polynomial, is a field.

3. The ringC(]a, b[) of continuous functionsf : ]a, b[→ R such thatf(x) = 0 for all x∈]a, b[ is a field.

4. The ring Z/pZ is a field wheneverp is prime.

A homomorphismh:K1→K_2 between two fieldsK_1 andK_2 is just a homomorphism between the ringsK_1 andK_2. However, becauseK∗ andK∗ are groups under multiplication,

1 2

a homomorphism of fields must be injective.

First, observe that for anyx = 0,

1 =h(1) =h(xx−1) =h(x)h(x−1)

and 1 =h(1) =h(x−1x) =h(x−1)h(x), soh(x) = 0 and

h(x−1) =h(x)−1.

But then, ifh(x) = 0, we must havex = 0. Consequently,h is injective.

A field homomorphism h:K1→ K_2 is an isomorphism iff there is a homomorphism g:K2→K_1 such thatgæf = idK1 andfæg = idK2. Then,g is unique and denoted byh−1. It is easy to show that a bijective field homomorphismh:K1→K_2 is an isomorphism. An isomorphism from a field to itself is called an automorphism.

Since every homomorphismh:K1 →K_2 between two fields is injective, the imagef(K_1) is a subfield ofK_2. We also say thatK_2 is an extension ofK_1. A fieldK is said to be 2.2. VECTOR SPACES 23

algebraically closed if every polynomialp(X) with coefficients inK has some root inK; that is, there is somea∈K such thatp(a) = 0. It can be shown that every fieldK has some minimal extension & which is algebraically closed, called an algebraic closure ofK. For example, C is the algebraic closure of both Q and C.

Given a fieldK and an automorphismh:K→K ofK, it is easy to check that the set

Fix(h) ={a∈K|h(a) =a}

of elements ofK fixed byh is a subfield ofK called the field fixed byh.

If K is a field, we have the ring homomorphismh: Z→K given byh(n) =n· 1. Ifh is injective, thenK contains a copy of Z, and since it is a field, it contains a copy of Q. In this case, we say thatK has characteristic 0. Ifh is not injective, thenh(Z) is a subring of K, and thus an integral domain, which is isomorphic to Z/pZ for somep≥ 1. But then,p must be prime since Z/pZ is an integral domain iff it is a field iffp is prime. The primep is called the characteristic ofK, and we also says thatK is of finite characteristic.

2.2 Vector Spaces

For everyn≥ 1, let Rn be the set ofn-tuplesx = (x_1, . . . , xn). Addition can be extended to Rn as follows:

( x_1, . . . , xn) + (y_1, . . . , yn) = (x_1 +y_1, . . . , xn +yn).

We can also define an operation·: R×Rn Rn as follows:→

λ· (x_1, . . . , xn) = (λx_1, . . . , λxn).

The resulting algebraic structure has some interesting properties, those of a vector space.

Definition 2.9. Given a fieldK, a vector space overK (orK-vector space) is a setE (of vectors) together with two operations +:E×E→E (called vector addition),2 and ·:K×E→E (called scalar multiplication) satisfying the following conditions for allα, β∈K and allu, v∈E;

(V0) E is an abelian group w.r.t. +, with identity element 0;3 (V1) α· (u +v) = (α·u) + (α·v); (V2) (α +β)·u = (α·u) + (β·u); (V3) (α∗β)·u =α· (β·u);

2 The symbol + is overloaded, since it denotes both addition in the field K and addition of vectors in E. It is usually clear from the context which + is intended.

3The symbol 0 is also overloaded, since it represents both the zero in K (a scalar) and the identity element of E (the zero vector). Confusion rarely arises, but one may prefer using 0 for the zero vector.

(V4) 1·u =u. In (V3),∗ denotes multiplication in the fieldK. Givenα∈K andv∈E, the elementα·v is also denoted byαv. The fieldK is often called the field of scalars.

Unless specified otherwise or unless we are dealing with several different fields, in the rest of this chapter, we assume that allK-vector spaces are defined with respect to a fixed field K. Thus, we will refer to aK-vector space simply as a vector space. In most cases, the field K will be the field R of reals.

From (V0), a vector space always contains the null vector 0, and thus is nonempty. From (V1), we getα· 0 = 0, andα· (−v) =−(α·v). From (V2), we get 0·v = 0, and (−α)·v =−(α·v).

Another important consequence of the axioms is the following fact: For anyu∈E and anyλ∈K, ifλ = 0 andλ·u = 0, thenu = 0.

Indeed, sinceλ = 0, it has a multiplicative inverseλ−1, so fromλ·u = 0, we get

λ−1 (λ·u) =λ−1 0.· ·

However, we just observed thatλ−1 0 = 0, and from (V3) and (V4), we have·

λ−1 (λ·u) = (λ−1λ)·u = 1·u =u,·

and we deduce thatu = 0.

Remark: One may wonder whether axiom (V4) is really needed. Could it be derived from the other axioms? The answer is no. For example, one can take E = Rn and define

·: R×Rn Rn by→ λ· (x_1, . . . , xn) = (0, . . . ,0)

for all ( x_1, . . . , xn)∈ Rn and allλ∈ R. Axioms (V0)–(V3) are all satisfied, but (V4) fails. Less trivial examples can be given using the notion of a basis, which has not been defined yet.

The fieldK itself can be viewed as a vector space over itself, addition of vectors being addition in the field, and multiplication by a scalar being multiplication in the field.

Example 2.6.

1. The fields R and C are vector spaces over R.

2. The groupsn and Cn are vector spaces over R, and Cn is a vector space over C. R

3. The ring R[X] of polynomials is a vector space over R, and C[X] is a vector space over R and C. The ring ofn×n matrices Mn(R) is a vector space over R.

4. The ringC(]a, b[) of continuous functionsf : ]a, b[→ R is a vector space over R.

Let E be a vector space. We would like to define the important notions of linear combination and linear independence. These notions can be defined for sets of vectors inE, but it will turn out to be more convenient to define them for families (vi)i∈I, whereI is any arbitrary index set.

2.3 Linear Independence, Subspaces

One of the most useful properties of vector spaces is that there possess bases. What this means is that in every vector space,E, there is some set of vectors,{e_1, . . . , en}, such that every, vector,v∈E, can be written as a linear combination,

v =λ_1e_1 +· · · +λnen,

of theei, for some scalars,λ_1, . . . , λn∈K. Furthermore, then-tuple, (λ_1, . . . , λn), as above is unique.

This description is fine when E has a finite basis,{e_1, . . . , en}, but this is not always the case! For example, the vector space of real polynomials, R[X], does not have a finite basis but instead it has an infinite basis, namely

1, X, X^2, . . . , Xn, . . .

One might wonder if it is possible for a vector space to have bases of different sizes, or even to have a finite basis as well as an infinite basis. We will see later on that this is not possible; all bases of a vector space have the same number of elements (cardinality), which is called the dimension of the space. However, we have the following problem: If a vector space has an infinite basis,{e_1, e_2, . . . ,}, how do we define linear combinations? Do we allow linear combinations

λ_1e_1 +λ_2e_2 +· · ·

with infinitely many nonzero coefficients?

If we allow linear combinations with infinitely many nonzero coefficients, then we have to make sense of these sums and this can only be done reasonably if we define such a sum as the limit of the sequence of vectors,s_1, s_2, . . . , sn, . . ., withs_1 =λ_1e_1 and

sn+1 =sn +λn+1en+1.

But then, how do we define such limits? Well, we have to define some topology on our space, by means of a norm, a metric or some other mechanism. This can indeed be done and this is what Banach spaces and Hilbert spaces are all about but this seems to require a lot of machinery.

A way to avoid limits is to restrict our attention to linear combinations involving only finitely many vectors. We may have an infinite supply of vectors but we only form linear combinations involving finitely many nonzero coefficients. Technically, this can be done by introducing families of finite support. This gives us the ability to manipulate families of scalars indexed by some fixed infinite set and yet to be treat these families as if they were finite. With these motivations in mind, let us review the notion of an indexed family.

Given a setA, a family (ai)i∈I of elements ofA is simply a functiona:I→A.

Remark: When considering a family (ai)i∈I, there is no reason to assume thatI is ordered. The crucial point is that every element of the family is uniquely indexed by an element of I. Thus, unless specified otherwise, we do not assume that the elements of an index set are ordered.

If A is an abelian group (usually, whenA is a ring or a vector space) with identity 0, we say that a family (ai)i∈I has finite support ifai = 0 for alli∈I−J, whereJ is a finite subset ofI (the support of the family).

We can deal with an arbitrary set X by viewing it as the family (Xx)x∈X corresponding to the identity function id:X→X. We agree that whenI =∅, (ai)i∈I =∅. A family (ai)i∈I is finite ifI is finite.

Given two disjoint sets I andJ, the union of two families (ui)i∈I and (vj)j∈J, denoted as (ui)i∈I∪ (vj)j∈J, is the family (wk)k∈(I∪J) defined such thatwk =uk ifk∈I, andwk =vk ifk∈J. Given a family (ui)i∈I and any elementv, we denote by (ui)i∈I∪k (v) the family (wi)i∈I∪{k} defined such that,wi =ui ifi∈I, andwk =v, wherek is any index such that k /∈I. Given a family (ui)i∈I, a subfamily of (ui)i∈I is a family (uj)j∈J whereJ is any subset ofI.

In this chapter, unless specified otherwise, it is assumed that all families of scalars have finite support.

Definition 2.10. LetE be a vector space. A vectorv∈E is a linear combination of a family (ui)i∈I of elements ofE if there is a family (λi)i∈I of scalars inK such that v = λiui.

i∈I

WhenI =∅, we stipulate thatv = 0. We say that a family (ui)i∈I is linearly independent if for every family (λi)i∈I of scalars inK,

λiui = 0 implies that λi = 0 for alli∈I.

i∈I

Equivalently, a family (ui)i∈I is linearly dependent if there is some family (λi)i∈I of scalars inK such that

λiui = 0 and λj = 0 for somej∈I.

i∈I

We agree that when I =∅, the family∅ is linearly independent. A family (ui)i∈I is linearly dependent iff someuj in the family can be expressed as a linear combination of the other vectors in the family. Indeed, there is some family (λi)i∈I of scalars inK such that

λiui = 0 and λj = 0 for somej∈I,

i∈I

which implies that u

j

=

−

λ

1

j λiui.

i∈(I−{j})

The above shows that a family ( ui)i∈I is linearly independent iff eitherI =∅, orI consists of a single elementi andui = 0, or|I| ≥ 2 and no vectoruj in the family can be expressed as a linear combination of the other vectors in the family.

When I is nonempty, if the family (ui)i∈I is linearly independent, note thatui = 0 for alli∈I. Otherwise, ifui = 0 for somei∈I, then we get a nontrivial linear dependence i∈Iλiui = 0 by picking any nonzeroλi and lettingλk = 0 for allk∈I withk =i, since λi 0 = 0. If|I| ≥ 2, we must also haveui =uj for alli, j∈I withi =j, since otherwise we get a nontrivial linear dependence by pickingλi =λ andλj =−λ for any nonzeroλ, and lettingλk = 0 for allk∈I withk =i, j.

Example 2.7.

1. Any two distinct scalarsλ, µ = 0 inK are linearly dependent.

2. In R^3, the vectors (1,0,0), (0,1,0), and (0,0,1) are linearly independent.

3. In R^4, the vectors (1,1,1,1), (0,1,1,1), (0,0,1,1), and (0,0,0,1) are linearly independent.

4. In R^2, the vectorsu = (1,1),v = (0,1) andw = (2,3) are linearly dependent, since

w = 2u +v.

Note that a family ( ui)i∈I is linearly independent iff (uj)j∈J is linearly independent for every finite subsetJ ofI (even whenI =∅). Indeed, wheni∈Iλiui = 0, the family (λi)i∈I of scalars inK has finite support, and thusi∈Iλiui = 0 really means that j∈Jλjuj = 0 for a finite subsetJ ofI. WhenI is finite, we often assume that it is the setI ={1,2, . . . , n}. In this case, we denote the family (ui)i∈I as (u_1, . . . , un).

The notion of a subspace of a vector space is defined as follows.

Definition 2.11. Given a vector spaceE, a subsetF ofE is a linear subspace (or subspace) ofE ifF is nonempty andλu +µv∈F for allu, v∈F, and allλ, µ∈K. It is easy to see that a subspaceF ofE is indeed a vector space, since the restriction of +:E×E→E toF×F is indeed a function +:F×F→F, and the restriction of ·

:K×E→E toK×F is indeed a function·:K×F→F.

It is also easy to see that any intersection of subspaces is a subspace. Since F is nonempty, if we pick any vectoru∈F and if we letλ =µ = 0, thenλu +µu = 0u + 0u = 0, so every subspace contains the vector 0. For any nonempty finite index setI, one can show by induction on the cardinality ofI that if (ui)i∈I is any family of vectorsui∈F and (λi)i∈I is any family of scalars, theni∈Iλiui∈F.

The subspace{0} will be denoted by (0), or even 0 (with a mild abuse of notation). Example 2.8.

1. In R^2, the set of vectorsu = (x, y) such that

x +y = 0

is a subspace.

2. In R^3, the set of vectorsu = (x, y, z) such that

x +y +z = 0

is a subspace.

3. For anyn≥ 0, the set of polynomialsf(X)∈ R[X] of degree at mostn is a subspace of R[X].

4. The set of upper triangularn×n matrices is a subspace of the space ofn×n matrices.

Proposition 2.5. Given any vector spaceE, ifS is any nonempty subset ofE, then the smallest subspace S (or Span(S)) ofE containingS is the set of all (finite) linear combinations of elements fromS.

Proof. We prove that the set Span(S) of all linear combinations of elements ofS is a subspace ofE, leaving as an exercise the verification that every subspace containingS also contains Span(S).

First, Span(S) is nonempty since it containsS (which is nonempty). Ifu =i∈Iλiui andv =j∈Jµjvj are any two linear combinations in Span(S), for any two scalarsλ, µ∈ R, λu +µv =λ λiui +µ µjvj

i∈I j∈J

= λλiui + µµjvj

i∈I j∈J

= λλiui + (λλi +µµi)ui + µµjvj,

i∈I−J i∈I∩J j∈J−I

which is a linear combination with index setI∪J, and thusλu +µv∈ Span(S), which proves that Span(S) is a subspace.

One might wonder what happens if we add extra conditions to the coefficients involved in forming linear combinations. Here are three natural restrictions which turn out to be important (as usual, we assume that our index sets are finite):

(1) Consider combinationsi∈Iλiui for which

λi = 1.

i∈I

These are called affine combinations. One should realize that every linear combination i ∈Iλiui can be viewed as an affine combination. For example, ifk is an index not

inI , if we letJ =I∪ {k},uk = 0, andλk = 1− i∈Iλi, thenj∈Jλjuj is an affine

combination and

λiui = λjuj.

i∈I j∈J

However, we get new spaces. For example, in R^3, the set of all affine combinations of the three vectorse_1 = (1,0,0), e_2 = (0,1,0), ande_3 = (0,0,1), is the plane passing through these three points. Since it does not contain 0 = (0,0,0), it is not a linear subspace.

(2) Consider combinationsi∈Iλiui for which

λi≥ 0, for alli∈I.

These are called positive (or conic) combinations It turns out that positive combinations of families of vectors are cones. They show naturally in convex optimization.

(3) Consider combinationsi∈Iλiui for which we require (1) and (2), that is

λi = 1, and λi≥ 0 for alli∈I.

i∈I

These are called convex combinations. Given any finite family of vectors, the set of all convex combinations of these vectors is a convex polyhedron. Convex polyhedra play a very important role in convex optimization.

2.4 Bases of a Vector Space

Given a vector space E, given a family (vi)i∈I, the subsetV ofE consisting of the null vector 0 and of all linear combinations of (vi)i∈I is easily seen to be a subspace ofE. Subspaces having such a “generating family” play an important role, and motivate the following definition. Definition 2.12. Given a vector spaceE and a subspaceV ofE, a family (vi)i∈I of vectors vi∈V spansV or generatesV if for everyv∈V , there is some family (λi)i∈I of scalars in K such that

v = λivi.

i∈I

We also say that the elements of ( vi)i∈I are generators ofV and thatV is spanned by (vi)i∈I, or generated by (vi)i∈I. If a subspaceV ofE is generated by a finite family (vi)i∈I, we say thatV is finitely generated . A family (ui)i∈I that spansV and is linearly independent is called a basis ofV .

Example 2.9.

1. In R^3, the vectors (1,0,0), (0,1,0), and (0,0,1) form a basis.

2. The vectors (1 ,1,1,1),(1,1,−1,−1),(1,−1,0,0),(0,0,1,−1) form a basis of R^4 known as the Haar basis. This basis and its generalization to dimension 2n are crucial in wavelet theory.

3. In the subspace of polynomials in R[X] of degree at mostn, the polynomials 1, X, X^2, . . . , Xn form a basis.

4. The Bernstein polynomials n (1−X)kXn−k fork = 0, . . . , n, also form a basis ofk

that space. These polynomials play a major role in the theory of spline curves.

It is a standard result of linear algebra that every vector space E has a basis, and that for any two bases (ui)i∈I and (vj)j∈J,I andJ have the same cardinality. In particular, ifE has a finite basis ofn elements, every basis ofE hasn elements, and the integern is called the dimension of the vector spaceE. We begin with a crucial lemma.

Lemma 2.6. Given a linearly independent family (ui)i∈I of elements of a vector spaceE, if v∈E is not a linear combination of (ui)i∈I, then the family (ui)i∈I∪k(v) obtained by adding v to the family (ui)i∈I is linearly independent (wherek /∈I).

Proof. Assume thatµv+i∈Iλiui = 0, for any family (λi)i∈I of scalars inK. Ifµ = 0, then µ has an inverse (becauseK is a field), and thus we havev =− i∈I(µ−1λi)ui, showing thatv is a linear combination of (ui)i∈I and contradicting the hypothesis. Thus, µ = 0. But then, we havei∈Iλiui = 0, and since the family (ui)i∈I is linearly independent, we have λi = 0 for alli∈I.

The next theorem holds in general, but the proof is more sophisticated for vector spaces that do not have a finite set of generators. Thus, in this chapter, we only prove the theorem for finitely generated vector spaces.

Theorem 2.7. Given any finite familyS = (ui)i∈I generating a vector spaceE and any linearly independent subfamilyL = (uj)j∈J ofS (whereJ⊆I), there is a basisB ofE such thatL⊆B⊆S.

Proof. Consider the set of linearly independent familiesB such thatL⊆B⊆S. Since this set is nonempty and finite, it has some maximal element, sayB = (uh)h∈H. We claim that B generatesE. Indeed, ifB does not generateE, then there is someup∈S that is not a linear combination of vectors inB (sinceS generatesE), withp /∈H. Then, by Lemma 2.6, the familyB = (uh)h∈H∪{p} is linearly independent, and sinceL⊆ B⊂ B⊆S, this contradicts the maximality ofB . Thus,B is a basis ofE such thatL⊆B⊆S.

Remark: Theorem 2.7 also holds for vector spaces that are not finitely generated. In this case, the problem is to guarantee the existence of a maximal linearly independent familyB such thatL⊆B⊆S. The existence of such a maximal family can be shown using Zorn’s lemma, see Appendix 31 and the references given there.

The following proposition giving useful properties characterizing a basis is an immediate consequence of Theorem 2.7.

Proposition 2.8. Given a vector spaceE, for any familyB = (vi)i∈I of vectors ofE, the following properties are equivalent:

(1) B is a basis ofE. (2) B is a maximal linearly independent family ofE. (3) B is a minimal generating family ofE.

The following replacement lemma due to Steinitz shows the relationship between finite linearly independent families and finite families of generators of a vector space.

Proposition 2.9. (Replacement lemma) Given a vector spaceE, let (ui)i∈I be any finite linearly independent family inE, where|I| =m, and let (vj)j∈J be any finite family such that everyui is a linear combination of (vj)j∈J, where|J| =n. Then, there exists a setL and an injectionρ:L→J such thatL∩I =∅ ,|L| =n−m, and the families (ui)i∈I∪ (vρ(l))l∈L and (vj)j∈J generate the same subspace ofE. In particular,m≤n.

Proof. We proceed by induction on|I| =m. Whenm = 0, the family (ui)i∈I is empty, and the proposition holds trivially withL =J (ρ is the identity). Assume|I| =m+ 1. Consider the linearly independent family (ui)i∈(I−{p}), wherep is any member ofI. By the induction hypothesis, there exists a setL and an injectionρ:L→ J such thatL∩ (I− {p}) =∅, |L| =n−m, and the families (ui)i∈(I −{p})∪(vρ(l))l∈L and (vj)j∈J generate the same subspace ofE. Ifp∈L, we can replaceL by (L− {p})∪ {p} wherep does not belong toI∪L, and replaceρ by the injectionρ which agrees withρ onL− {p} and such thatρ (p ) =ρ(p). Thus, we can always assume thatL∩I =∅. Sinceup is a linear combination of (vj)j∈J and the families (ui)i∈(I−{p})∪ (vρ(l))l∈L and (vj)j∈J generate the same subspace ofE,up is a linear combination of (ui)i∈(I−{p})∪ (vρ(l))l∈L. Let

up = λiui + λlvρ(l). (1)

i∈(I−{p}) l∈L

Ifλl = 0 for alll∈L, we have

λiui−up = 0,

i∈(I−{p})

contradicting the fact that (ui)i∈I is linearly independent. Thus,λl = 0 for somel∈L, say l =q. Sinceλq = 0, we have

vρ(q) = (−λ− 1 λi)ui +λ− 1 up + (−λ− 1 λl)vρ(l). (2)

q q q

i∈(I−{p}) l∈(L−{q})

We claim that the families ( ui)i∈(I−{p})∪ (vρ(l))l∈L and (ui)i∈I∪ (vρ(l))l∈(L−{q}) generate the same subset ofE. Indeed, the second family is obtained from the first by replacingvρ(q) byup, and vice-versa, andup is a linear combination of (ui)i∈(I−{p})∪(vρ(l))l∈L, by (1), andvρ(q) is a linear combination of (ui)i∈I∪(vρ(l))l∈(L−{q}), by (2). Thus, the families (ui)i∈I∪(vρ(l))l∈(L−{q}) and (vj)j∈J generate the same subspace ofE, and the proposition holds forL− { q} and the restriction of the injectionρ:L→J toL− {q}, sinceL∩I =∅ and|L| =n−m imply that (L− {q})∩I =∅ and|L− {q}| =n− (m + 1).

The idea is that m of the vectorsvj can be replaced by the linearly independentui’s in such a way that the same subspace is still generated. The purpose of the functionρ:L→J is to pickn−m elementsj_1, . . . , jn−m ofJ and to relabel theml_1, . . . , ln−m in such a way that these new indices do not clash with the indices inI; this way, the vectorsvj1, . . . , vjn−m who “survive” (i.e. are not replaced) are relabeledvl1, . . . , vln−m, and the otherm vectorsvj withj∈J− {j_1, . . . , jn−m} are replaced by theui. The index set of this new family isI∪L.

Actually, one can prove that Proposition 2.9 implies Theorem 2.7 when the vector space is finitely generated. Putting Theorem 2.7 and Proposition 2.9 together, we obtain the following fundamental theorem.

Theorem 2.10. LetE be a finitely generated vector space. Any family (ui)i∈I generatingE contains a subfamily (uj)j∈J which is a basis ofE. Furthermore, for every two bases (ui)i∈I and (vj)j∈J ofE, we have|I| =|J| =n for some fixed integern≥ 0.

Proof. The first part follows immediately by applying Theorem 2.7 withL =∅ andS = (ui)i∈I. Assume that (ui)i∈I and (vj)j∈J are bases ofE. Since (ui)i∈I is linearly independent and (vj)j∈J spansE, proposition 2.9 implies that|I| ≤ |J|. A symmetric argument yields

|J| ≤ |I|.

Remark: Theorem 2.10 also holds for vector spaces that are not finitely generated. This can be shown as follows. Let (ui)i∈I be a basis ofE, let (vj)j∈J be a generating family ofE, and assume thatI is infinite. For everyj∈J, letLj⊆I be the finite set

Lj ={i∈I|vj = λiui, λi = 0}.

i∈I

Let L = j∈JLj. By definitionL⊆I, and since (ui)i∈I is a basis ofE, we must haveI =L, since otherwise ( ui)i∈L would be another basis ofE, and this would contradict the fact that (ui)i∈I is linearly independent. Furthermore,J must be infinite, since otherwise, because theLj are finite,I would be finite. But then, sinceI =j∈ JLj withJ infinite and theLj finite, by a standard result of set theory,|I| ≤ | J|. If (vj)j∈J is also a basis, by a symmetric argument, we obtain|J| ≤ |I|, and thus,|I| =|J| for any two bases (ui)i∈I and (vj)j∈J ofE.

When E is not finitely generated, we say thatE is of infinite dimension. The dimension of a vector spaceE is the common cardinality of all of its bases and is denoted by dim(E). Clearly, if the fieldK itself is viewed as a vector space, then every family (a) wherea∈K anda = 0 is a basis. Thus dim(K) = 1. Note that dim({0}) = 0.

IfE is a vector space, for any subspaceU ofE, if dim(U) = 1, thenU is called a line; if dim(U) = 2, thenU is called a plane. If dim(U) =k, thenU is sometimes called ak-plane. Let (ui)i∈I be a basis of a vector spaceE. For any vectorv∈E, since the family (ui)i∈I generatesE, there is a family (λi)i∈I of scalars inK, such that

v = λiui.

i∈I

A very important fact is that the family (λi)i∈I is unique.

Proposition 2.11. Given a vector spaceE, let (ui)i∈I be a family of vectors inE. Letv∈E, and assume thatv =i∈Iλiui. Then, the family (λi)i∈I of scalars such thatv =i∈Iλiui is unique iff (ui)i∈I is linearly independent.

Proof. First, assume that (ui)i∈I is linearly independent. If (µi)i∈I is another family of scalars inK such thatv =i∈Iµiui, then we have

(λi−µi)ui = 0,

i∈I

and since ( ui)i∈I is linearly independent, we must haveλi −µi = 0 for alli∈I, that is,λi =µi for alli∈I. The converse is shown by contradiction. If (ui)i∈I was linearly dependent, there would be a family (µi)i∈I of scalars not all null such that

µiui = 0

i∈I

andµj = 0 for somej∈I. But then,

v = λiui + 0 = λiui + µiui = (λi +µi)ui,

i∈I i∈I i∈I i∈I

withλj =λj+µj sinceµj = 0, contradicting the assumption that (λi)i∈I is the unique family such thatv =i∈Iλiui.

If (ui)i∈I is a basis of a vector spaceE, for any vectorv∈E, if (xi)i∈I is the unique family of scalars inK such that

v = xiui,

i∈I

eachxi is called the component (or coordinate) of indexi ofv with respect to the basis (ui)i∈I. Given a fieldK and any (nonempty) setI, we can form a vector spaceK(I) which, in some sense, is the standard vector space of dimension|I|.

Definition 2.13. Given a fieldK and any (nonempty) setI, letK(I) be the subset of the cartesian productKI consisting of all families (λi)i∈I with finite support of scalars inK.4 We define addition and multiplication by a scalar as follows:

(λi)i∈I + (µi)i∈I = (λi +µi)i∈I, and λ· (µi)i∈I = (λµi)i∈I.

It is immediately verified that addition and multiplication by a scalar are well defined. Thus,K(I) is a vector space. Furthermore, because families with finite support are considered, the family (ei)i∈I of vectorsei, defined such that (ei)j = 0 ifj =i and (ei)i = 1, is clearly a basis of the vector spaceK(I). WhenI ={1, . . . , n}, we denoteK(I) byKn. The functionι:I→K(I), such thatι(i) =ei for everyi∈I, is clearly an injection.

WhenI is a finite set,K(I) =KI, but this is false whenI is infinite. In fact, dim(K(I)) = |I|, but dim(KI) is strictly greater whenI is infinite.

Many interesting mathematical structures are vector spaces. A very important example is the set of linear maps between two vector spaces to be defined in the next section. Here is an example that will prepare us for the vector space of linear maps.

Example 2.10. LetX be any nonempty set and letE be a vector space. The set of all functionsf :X→E can be made into a vector space as follows: Given any two functions f :X→E andg:X→E, let (f +g):X→E be defined such that

for allx∈X, and for everyλ∈K, letλf :X→E be defined such that

(f +g)(x) =f(x) +g(x) 4Where KI denotes the set of all functions from I to K.

(λf)(x) =λf(x)

for all x∈X. The axioms of a vector space are easily verified. Now, letE =K, and letI be the set of all nonempty subsets ofX. For everyS∈I, letfS:X→E be the function such thatfS(x) = 1 iffx∈S, andfS(x) = 0 iffx /∈S. We leave as an exercise to show that (fS)S∈I is linearly independent.

2.5 Linear Maps

A function between two vector spaces that preserves the vector space structure is called a homomorphism of vector spaces, or linear map. Linear maps formalize the concept of linearity of a function. In the rest of this section, we assume that all vector spaces are over a given fieldK (say R).

Definition 2.14. Given two vector spacesE andF, a linear map betweenE andF is a functionf :E→F satisfying the following two conditions:

f(x +y) =f(x) +f(y) for allx, y∈E;

f(λx) =λf(x) for allλ∈K, x∈E.

Setting x =y = 0 in the first identity, we getf(0) = 0. The basic property of linear maps is that they transform linear combinations into linear combinations. Given a family (ui)i∈I of vectors inE, given any family (λi)i∈I of scalars inK, we have

f( λiui) = λif(ui).

i∈I i∈I

The above identity is shown by induction on the size of the support of the family (λiui)i∈I, using the properties of Definition 2.14.

Example 2.11.

1. The mapf : R^2 R^2 defined such that→

x = x−y y = x +y is a linear map. The reader should check that it is the composition of a rotation by π/4 with a magnification of ratio√2.

2. For any vector spaceE, the identity map id:E→E given by

id(u) =u for allu∈E

is a linear map. When we want to be more precise, we write idE instead of id. 3. The mapD: R[X]→ R[X] defined such that

D(f(X)) =f (X),

wheref (X) is the derivative of the polynomialf(X), is a linear map. 4. The map Φ:C([a, b])→ R given by

b

Φ(f) = f(t)dt,

a

whereC([a, b]) is the set of continuous functions defined on the interval [a, b], is a linear map.

5. The function−,−:C([a, b])× C([a, b])→ R given by

b

f, g = f(t)g(t)dt,

a

is linear in each of the variablef,g. It also satisfies the properties f, g = g, f and f, f = 0 ifff = 0. It is an example of an inner product.

Definition 2.15. Given a linear mapf :E→F, we define its image (or range) Imf =f(E), as the set

Imf ={y∈F| (∃x∈E)(y =f(x))},

and its Kernel (or nullspace) Kerf =f−1(0), as the set

Kerf ={x∈E|f(x) = 0}.

Proposition 2.12. Given a linear mapf :E→F, the set Imf is a subspace ofF and the set Kerf is a subspace ofE. The linear mapf :E→F is injective iff Kerf = 0 (where 0 is the trivial subspace{0}).

Proof. Given anyx, y∈ Imf, there are someu, v∈E such thatx =f(u) andy =f(v), and for allλ, µ∈K, we have

f(λu +µv) =λf(u) +µf(v) =λx +µy,

and thus,λx +µy∈ Imf, showing that Imf is a subspace ofF.

Given anyx, y∈ Kerf, we havef(x) = 0 andf(y) = 0, and thus,

f(λx +µy) =λf(x) +µf(y) = 0,

that is,λx +µy∈ Kerf, showing that Kerf is a subspace ofE.

First, assume that Ker f = 0. We need to prove thatf(x) =f(y) implies thatx =y. However, iff(x) =f(y), thenf(x)−f(y) = 0, and by linearity off we getf(x−y) = 0. Because Kerf = 0, we must havex−y = 0, that isx =y, sof is injective. Conversely, assume thatf is injective. Ifx∈ Kerf, that isf(x) = 0, sincef(0) = 0 we havef(x) = f(0), and by injectivity,x = 0, which proves that Kerf = 0. Therefore,f is injective iff Kerf = 0.

Since by Proposition 2.12, the image Imf of a linear mapf is a subspace ofF, we can define the rank rk(f) off as the dimension of Imf.

A fundamental property of bases in a vector space is that they allow the definition of linear maps as unique homomorphic extensions, as shown in the following proposition.

Proposition 2.13. Given any two vector spacesE andF, given any basis (ui)i∈I ofE, given any other family of vectors (vi)i∈I inF, there is a unique linear mapf :E→F such thatf(ui) =vi for alli∈I. Furthermore,f is injective iff (vi)i∈I is linearly independent, andf is surjective iff (vi)i∈I generatesF.

Proof. If such a linear mapf :E→F exists, since (ui)i∈I is a basis ofE, every vectorx∈E can written uniquely as a linear combination

x = xiui,

i∈I

and by linearity, we must have f(x) = xif(ui) = xivi.

i∈I i∈I

Define the functionf :E→F, by letting f(x) = xivi

i∈I

for everyx =i∈Ixiui. It is easy to verify thatf is indeed linear, it is unique by the previous reasoning, and obviously, f(ui) =vi.

Now, assume thatf is injective. Let (λi)i∈I be any family of scalars, and assume that λivi = 0.

i∈I

Sincevi =f(ui) for everyi∈I, we have f( λiui) = λif(ui) = λivi = 0.

i∈I i∈I i∈I

Sincef is injective iff Kerf = 0, we have λiui = 0,

i∈I

and since ( ui)i∈I is a basis, we haveλi = 0 for alli∈I, which shows that (vi)i∈I is linearly independent. Conversely, assume that (vi)i∈I is linearly independent. Since (ui)i∈I is a basis ofE, every vectorx∈E is a linear combinationx =i∈Iλiui of (ui)i∈I. If

f(x) =f( λiui) = 0,

i∈I

then λivi = λif(ui) =f( λiui) = 0,

i∈I i∈I i∈I

and λi = 0 for alli∈I because (vi)i∈I is linearly independent, which means thatx = 0. Therefore, Kerf = 0, which implies thatf is injective. The part wheref is surjective is left as a simple exercise.

By the second part of Proposition 2.13, an injective linear map f :E→F sends a basis (ui)i∈I to a linearly independent family (f(ui))i∈I ofF, which is also a basis whenf is bijective. Also, whenE andF have the same finite dimensionn, (ui)i∈I is a basis ofE, and f :E→F is injective, then (f(ui))i∈I is a basis ofF (by Proposition 2.8).

We can now show that the vector space K(I) of Definition 2.13 has a universal property that amounts to saying thatK(I) is the vector space freely generated byI. Recall that ι:I→K(I), such thatι(i) =ei for everyi∈I, is an injection fromI toK(I).

Proposition 2.14. Given any setI, for any vector spaceF, and for any functionf :I→F, there is a unique linear mapf :K(I) F, such that→

f =fæι, as in the following diagram: I

CCC

ι K(I)

CCCCCC ff

F

Proof. If such a linear mapf :K(I) F exists, sincef =fæι, we must have→

f(i) =f(ι(i)) =f(ei),

for every i∈ I. However, the family (ei)i∈I is a basis ofK(I), and (f(i))i ∈I is a family of vectors inF, and by Proposition 2.13, there is a unique linear mapf :K(I) F such that→ f(ei) =f(i) for everyi∈I, which proves the existence and uniqueness of a linear mapf such thatf =fæι.

The following simple proposition is also useful.

Proposition 2.15. Given any two vector spacesE andF, withF nontrivial, given any family (ui)i∈I of vectors inE, the following properties hold:

(1) The family (ui)i∈I generatesE iff for every family of vectors (vi)i∈I inF, there is at most one linear map f :E→F such thatf(ui) =vi for alli∈I.

(2) The family (ui)i∈I is linearly independent iff for every family of vectors (vi)i∈I inF, there is some linear mapf :E→F such thatf(ui) =vi for alli∈I. Proof. (1) If there is any linear mapf :E→F such thatf(ui) =vi for alli∈I, since (ui)i∈I generatesE, every vectorx∈E can be written as some linear combination

x = xiui,

i∈I

and by linearity, we must have

f(x) = xif(ui) = xivi.

i∈I i∈I

This shows that f is unique if it exists. Conversely, assume that (ui)i∈I does not generateE. SinceF is nontrivial, there is some some vectory∈F such thaty = 0. Since (ui)i∈I does not generateE, there is some vectorw∈E that is not in the subspace generated by (ui)i∈I. By Theorem 2.7, there is a linearly independent subfamily (ui)i∈I0 of (ui)i∈I generating the same subspace. Since by hypothesis,w∈E is not in the subspace generated by (ui)i∈I0, by Lemma 2.6 and by Theorem 2.7 again, there is a basis (ej)j∈I0∪J ofE, such thatei =ui, for alli∈I_0, andw =ej0, for somej0∈J. Letting (vi)i∈I be the family inF such thatvi = 0 for alli∈I, definingf :E→F to be the constant linear map with value 0, we have a linear map such thatf(ui) = 0 for alli∈I. By Proposition 2.13, there is a unique linear map g:E→F such thatg(w) =y, andg(ej) = 0, for allj∈ (I0∪J)− { j0}. By definition of the basis (ej)j∈I0∪J ofE, we have,g(ui) = 0 for alli∈I, and sincef =g, this contradicts the fact that there is at most one such map.

(2) If the family ( ui)i∈I is linearly independent, then by Theorem 2.7, (ui)i∈I can be extended to a basis ofE, and the conclusion follows by Proposition 2.13. Conversely, assume that (ui)i∈I is linearly dependent. Then, there is some family (λi)i∈I of scalars (not all zero) such that

λiui = 0.

i∈I

By the assumption, for any nonzero vector,y∈F, for everyi∈I, there is some linear map fi:E→F, such thatfi(ui) =y, andfi(uj) = 0, forj∈I− {i}. Then, we would get

0 =fi( λiui) = λifi(ui) =λiy,

i∈I i∈I

and sincey = 0, this impliesλi = 0, for everyi∈I. Thus, (ui)i∈I is linearly independent.

Given vector spacesE,F, andG, and linear mapsf :E→F andg:F→G, it is easily verified that the compositiongæf :E→G off andg is a linear map.

A linear mapf :E→F is an isomorphism iff there is a linear mapg:F→E, such that

gæf = idE and fæg = idF. (∗) Such a mapg is unique. This is because ifg andh both satisfygæf = idE,fæg = idF, hæf = idE, andfæh = idF, then

g =gæ idF =gæ (fæh) = (gæf)æh = idEæh =h.

The mapg satisfying (∗) above is called the inverse off and it is also denoted byf−1.

Proposition 2.13 implies that if E andF are two vector spaces, (ui)i∈I is a basis ofE, andf :E→F is a linear map which is an isomorphism, then the family (f(ui))i∈I is a basis ofF.

One can verify that iff :E→F is a bijective linear map, then its inversef−1:F→E is also a linear map, and thusf is an isomorphism.

Another useful corollary of Proposition 2.13 is this:

Proposition 2.16. LetE be a vector space of finite dimensionn≥ 1 and letf :E→E be any linear map. The following properties hold:

(1) Iff has a left inverseg, that is, ifg is a linear map such thatgæf = id, thenf is an isomorphism andf−1 =g.

(2) Iff has a right inverseh, that is, ifh is a linear map such thatfæh = id, thenf is an isomorphism andf−1 =h.

Proof. (1) The equationgæf = id implies thatf is injective; this is a standard result about functions (iff(x) =f(y), theng(f(x)) =g(f(y)), which implies thatx =y since gæf = id). Let (u_1, . . . , un) be any basis ofE. By Proposition 2.13, sincef is injective, (f(u_1), . . . , f(un)) is linearly independent, and sinceE has dimensionn, it is a basis of E (if (f(u_1), . . . , f(un)) doesn’t spanE, then it can be extended to a basis of dimension strictly greater thann, contradicting Theorem 2.10). Then,f is bijective, and by a previous observation its inverse is a linear map. We also have

g =gæ id =gæ (fæf−1) = (gæf)æf−1 = idæf−1 =f−1.

(2) The equation fæh = id implies thatf is surjective; this is a standard result about functions (for anyy∈E, we havef(g(y)) =y). Let (u_1, . . . , un) be any basis ofE. By Proposition 2.13, sincef is surjective, (f(u_1), . . . , f(un)) spansE, and sinceE has dimension n, it is a basis ofE (if (f(u_1), . . . , f(un)) is not linearly independent, then because it spans E, it contains a basis of dimension strictly smaller thann, contradicting Theorem 2.10). Then,f is bijective, and by a previous observation its inverse is a linear map. We also have

h = idæh = (f−1 f)æh =f−1 (fæh) =f−1 id =f−1.æ æ æ

This completes the proof.

2.6. QUOTIENT SPACES 41

The set of all linear maps between two vector spaces E andF is denoted by Hom(E, F) or byL(E;F) (the notationL(E;F) is usually reserved to the set of continuous linear maps, whereE andF are normed vector spaces). When we wish to be more precise and specify the fieldK over which the vector spacesE andF are defined we write HomK(E, F).

The set Hom(E, F) is a vector space under the operations defined at the end of Section 2.1, namely

(f +g)(x) =f(x) +g(x) for allx∈E, and

(λf)(x) =λf(x)

for allx∈E. The point worth checking carefully is thatλf is indeed a linear map, which uses the commutativity of∗ in the fieldK. Indeed, we have

(λf)(µx) =λf(µx) =λµf(x) =µλf(x) =µ(λf)(x).

When E andF have finite dimensions, the vector space Hom(E, F) also has finite dimension, as we shall see shortly. WhenE =F, a linear mapf :E→E is also called an endomorphism. It is also important to note that composition confers to Hom(E, E) a ring structure. Indeed, composition is an operationæ: Hom(E, E)× Hom(E, E)→ Hom(E, E), which is associative and has an identity idE, and the distributivity properties hold:

(g_1 +g_2)æf =g1æf +g2æf; gæ (f_1 +f_2) =gæf_1 +gæf_2.

The ring Hom( E, E) is an example of a noncommutative ring. It is easily seen that the set of bijective linear mapsf :E→E is a group under composition. Bijective linear maps are also called automorphisms. The group of automorphisms ofE is called the general linear group (ofE), and it is denoted by GL(E), or by Aut(E), or whenE =Kn, by GL(n, K), or even by GL(n).

Although in this book, we will not have many occasions to use quotient spaces, they are fundamental in algebra. The next section may be omitted until needed.

2.6 Quotient Spaces

LetE be a vector space, and letM be any subspace ofE. The subspaceM induces a relation ≡M onE, defined as follows: For allu, v∈E,

u≡Mv iffu−v∈M.

We have the following simple proposition.

Proposition 2.17. Given any vector spaceE and any subspaceM ofE, the relation≡M is an equivalence relation with the following two congruential properties:

1. Ifu1≡Mv_1 andu2≡Mv_2, thenu_1 +u2≡Mv_1 +v_2, and 2. ifu≡Mv, thenλu≡Mλv.

Proof. It is obvious that≡M is an equivalence relation. Note thatu1≡Mv_1 andu2≡Mv_2 are equivalent tou1−v_1 =w_1 andu2−v_2 =w_2, withw_1, w2∈M, and thus,

(u_1 +u_2)− (v_1 +v_2) =w_1 +w_2,

andw_1 +w2∈M, sinceM is a subspace ofE. Thus, we haveu_1 +u2≡M v_1 +v_2. If u−v =w, withw∈M, then

λu−λv =λw, andλw∈M, sinceM is a subspace ofE, and thusλu≡Mλv.

Proposition 2.17 shows that we can define addition and multiplication by a scalar on the setE/M of equivalence classes of the equivalence relation≡M.

Definition 2.16. Given any vector spaceE and any subspaceM ofE, we define the following operations of addition and multiplication by a scalar on the setE/M of equivalence classes of the equivalence relation≡M as follows: for any two equivalence classes [u],[v]∈E/M, we have

[u] + [v] = [u +v], λ[u] = [λu].

By Proposition 2.17, the above operations do not depend on the specific choice of representatives in the equivalence classes [u],[v]∈E/M. It is also immediate to verify thatE/M is a vector space. The functionπ:E→E/F, defined such thatπ(u) = [u] for everyu∈E, is a surjective linear map called the natural projection ofE ontoE/F. The vector spaceE/M is called the quotient space ofE by the subspaceM.

Given any linear mapf :E→F, we know that Kerf is a subspace ofE, and it is immediately verified that Imf is isomorphic to the quotient spaceE/Kerf.

2.7 Summary

The main concepts and results of this chapter are listed below: Groups, rings and fields.•

• The notion of a vector space.

Families of vectors.•

2.7. SUMMARY 43

•

Linear combinations of vectors; linear dependence and linear independence of a family of vectors.

• Linear subspaces.

•

Spanning (or generating) family; generators, finitely generated subspace; basis of a subspace.

• Every linearly independent family can be extended to a basis (Theorem 2.7).

•

A familyB of vectors is a basis iff it is a maximal linearly independent family iff it is a minimal generating family (Proposition 2.8).

• The replacement lemma (Proposition 2.9).

•

Any two bases in a finitely generated vector spaceE have the same number of elements; this is the dimension ofE (Theorem 2.10).

Hyperlanes .•

• Every vector has a unique representation over a basis (in terms of its coordinates).

• The notion of a linear map.

• The image Imf (or range) of a linear mapf.

• The kernel Kerf (or nullspace) of a linear mapf. rank rk(f) of a linear mapf.• The

•

The image and the kernel of a linear map are subspaces. A linear map is injective iff its kernel is the trivial space (0) (Proposition 2.12).

•

The unique homomorphic extension property of linear maps with respect to bases (Proposition 2.13 ).

• Quotient spaces.

Chapter 3

Matrices and Linear Maps

3.1 Matrices

Proposition 2.13 shows that given two vector spaces E andF and a basis (uj)j∈J ofE, every linear mapf :E→F is uniquely determined by the family (f(uj))j∈J of the images underf of the vectors in the basis (uj)j∈J. Thus, in particular, takingF =K(J), we get an isomorphism between any vector spaceE of dimension|J| andK(J). IfJ ={1, . . . , n}, a vector spaceE of dimensionn is isomorphic to the vector spaceKn. If we also have a basis (vi)i∈I ofF, then every vectorf(uj) can be written in a unique way as

f(uj) = ai jvi,

i∈I

where j∈J, for a family of scalars (ai j)i∈I. Thus, with respect to the two bases (uj)j∈J ofE and (vi)i∈I ofF, the linear mapf is completely determined by a possibly infinite “I×J-matrix”M(f) = (ai j)i∈I, j∈J.

Remark: Note that we intentionally assigned the index setJ to the basis (uj)j∈J ofE, and the indexI to the basis (vi)i∈I ofF, so that the rows of the matrixM(f) associated withf :E→F are indexed byI, and the columns of the matrixM(f) are indexed byJ. Obviously, this causes a mildly unpleasant reversal. If we had considered the bases (ui)i∈I of E and (vj)j∈J ofF, we would obtain aJ×I-matrixM(f) = (aj i)j∈J, i∈I. No matter what we do, there will be a reversal! We decided to stick to the bases (uj)j∈J ofE and (vi)i∈I of F, so that we get anI×J-matrixM(f), knowing that we may occasionally suffer from this decision!

WhenI andJ are finite, and say, when|I| = m and|J| = n, the linear mapf is determined by the matrixM(f) whose entries in thej-th column are the components of the

45 vectorf(uj) over the basis (v_1, . . . , vm), that is, the matrix ëa1 1 a1 2 . . . a1 nö

ì

÷

ì. . ø

a

2 1

a

2 2

. . . a

2 n

÷

M(f) =ì .... ÷ íam 1 am 2 . . . am n whose entry on rowi and columnj isai j (1≤i≤m, 1≤j≤n).

We will now show that when E andF have finite dimension, linear maps can be very conveniently represented by matrices, and that composition of linear maps corresponds to matrix multiplication. We will follow rather closely an elegant presentation method due to Emil Artin.

Let E andF be two vector spaces, and assume thatE has a finite basis (u_1, . . . , un) and thatF has a finite basis (v_1, . . . , vm). Recall that we have shown that every vectorx∈E can be written in a unique way as

x =x_1u_1 +· · · +xnun,

and similarly every vectory∈F can be written in a unique way as

y =y_1v_1 +· · · +ymvm.

Letf :E→F be a linear map betweenE andF. Then, for everyx =x_1u_1 +· · · +xnun in E, by linearity, we have

f(x) =x_1f(u_1) +· · · +xnf(un). Let

f(uj) =a1 jv_1 +· · · +am jvm,

or more concisely,m

f(uj) = ai jvi,

i=1

for everyj, 1≤j≤n. This can be expressed by writing the coefficientsa1j, a2j, . . . , amj of f(uj) over the basis (v_1, . . . , vm), as thejth column of a matrix, as shown below: f(u_1) f(u_2) . . . f(un)

v_1 ë a_11 a_12 . . . a1n ö

v

2

ì

÷

í

a

21

a

22

. . . a

2

n

÷

ì .... ÷. ì. . ø

vm am1 am2 . . . amn

Then, substituting the right-hand side of eachf(uj) into the expression forf(x), we get

m m

f(x) =x_1( ai 1vi) +· · · +xn( ai nvi),

i=1 i=1

which, by regrouping terms to obtain a linear combination of thevi, yields

n n

f(x) = ( a1 jxj)v_1 +· · · + ( am jxj)vm.

j=1 j=1

Thus, lettingf(x) =y =y_1v_1 +· · · +ymvm, we have

n

yi = ai jxj (1)

j=1

for alli, 1≤i≤m.

To make things more concrete, let us treat the case wheren = 3 andm = 2. In this case,

f (u_1) =a_11v_1 +a_21v_2

f(u_2) =a_12v_1 +a_22v_2

f(u_3) =a_13v_1 +a_23v_2,

which in matrix form is expressed by

f(u_1) f(u_2) f(u_3)

v1 a11 a12 a13 ,v2 a21 a22 a23

and for anyx =x_1u_1 +x_2u_2 +x_3u_3, we have

f (x) =f(x_1u_1 +x_2u_2 +x_3u_3)

=x_1f(u_1) +x_2f(u_2) +x_3f(u_3)

=x_1(a_11v_1 +a_21v_2) +x_2(a_12v_1 +a_22v_2) +x_3(a_13v_1 +a_23v_2) = (a_11x_1 +a_12x_2 +a_13x_3)v_1 + (a_21x_1 +a_22x_2 +a_23x_3)v_2.

Consequently, since y =y_1v_1 +y_2v_2, we have

y_1 =a_11x_1 +a_12x_2 +a_13x_3 y_2 =a_21x_1 +a_22x_2 +a_23x_3.

This agrees with the matrix equation ëx1ö y1 = a_11 a_12 a13 íx2ø.y2 a21 a22 a23 x3 Let us now consider how the composition of linear maps is expressed in terms of bases.

Let E, F, and G, be three vectors spaces with respective bases (u_1, . . . , up) for E, (v_1, . . . , vn) forF, and (w_1, . . . , wm) forG. Letg:E→F andf :F→G be linear maps. As explained earlier,g:E→F is determined by the images of the basis vectorsuj, and f :F→G is determined by the images of the basis vectorsvk. We would like to understand howfæg:E→G is determined by the images of the basis vectorsuj.

Remark: Note that we are considering linear mapsg:E→F andf :F→G, instead off :E→ F andg:F→ G, which yields the compositionfæg:E→ G instead of gæf :E→G. Our perhaps unusual choice is motivated by the fact that iff is represented by a matrixM(f) = (ai k) andg is represented by a matrixM(g) = (bk j), thenfæg:E→G is represented by the productAB of the matricesA andB. If we had adopted the other choice wheref :E→F andg:F→G, thengæf :E→G would be represented by the productBA. Personally, we find it easier to remember the formula for the entry in rowi and column ofj of the product of two matrices when this product is written byAB, rather than BA. Obviously, this is a matter of taste! We will have to live with our perhaps unorthodox choice.

Thus, let

m

f(vk) = ai kwi,

i=1

for everyk, 1≤k≤n, and let

n

g(uj) = bk jvk,

k=1

for everyj, 1≤j≤p; in matrix form, we have

f(v_1) f(v_2) . . . f(vn) w_1 ë a_11 a_12 . . . a1n ö w

2

ì a_21 a_22 . . . a2n ÷

ì... ÷

.

÷. ì

í . . ø wm am1 am2 . . . amn

and

g(u_1) g(u_2) . . . g(up) v_1 ë b_11 b_12 . . . b1p ö v

2

ì b_21 b_22 . . . b2p ÷

ì... ÷ . ÷. ì. . øv í

n bn1 bn2 . . . bnp By previous considerations, for every x =x_1u_1 +· · · +xpup,

lettingg(x) =y =y_1v_1 +· · · +ynvn, we have

p

yk = bk jxj (2)

j=1

for all k, 1≤k≤n, and for every y =y_1v_1 +· · · +ynvn,

lettingf(y) =z =z_1w_1 +· · · +zmwm, we have

n

zi = ai kyk (3)

k=1

for alli, 1≤i≤m. Then, ify =g(x) andz =f(y), we havez =f(g(x)), and in view of (2) and (3), we have

n p

zi = ai k( bk jxj)

k=1 j=1

n p

= ai kbk jxj

k=1 j=1

p n

= ai kbk jxj

j=1 k=1

p n

= ( ai kbk j)xj.

j=1 k=1

Thus, definingci j such that

n

ci j = ai kbk j,

k=1

for 1≤i≤m, and 1≤j≤p, we have

p

zi = ci jxj (4)

j=1

Identity (4) suggests defining a multiplication operation on matrices, and we proceed to do so. We have the following definitions.

Definition 3.1. Given a fieldK, anm×n-matrix is a family (ai j)1≤i≤m, 1≤j≤n of scalars in K, represented as an arrayëa1 1 a1 2 . . . a1 nö

ì

÷

ì. . ø

a

2 1

a

2 2

. . . a

2

n

÷

ì .... ÷

íam 1 am 2 . . . am n

In the special case wherem = 1, we have a row vector, represented as

(a1 1· · ·a1 n) and in the special case wheren = 1, we have a column vector, represented as ëa1 1ö

ì. ÷

ía ø

m 1

In these last two cases, we usually omit the constant index 1 (first index in case of a row, second index in case of a column). The set of allm×n-matrices is denoted by Mm,n(K) or Mm,n. Ann×n-matrix is called a square matrix of dimension n. The set of all square matrices of dimensionn is denoted by Mn(K), or Mn.

Remark: As defined, a matrixA = (ai j)1≤i≤m, 1≤j≤n is a family, that is, a function from {

1 ,2, . . . , m} × {1,2, . . . , n} toK. As such, there is no reason to assume an ordering on the indices. Thus, the matrix A can be represented in many different ways as an array, by adopting different orders for the rows or the columns. However, it is customary (and usually convenient) to assume the natural ordering on the sets{1,2, . . . , m} and{1,2, . . . , n}, and to representA as an array according to this ordering of the rows and columns.

We also define some operations on matrices as follows.

Definition 3.2. Given twom×n matricesA = (ai j) andB = (bi j), we define their sum A +B as the matrixC = (ci j) such thatci j =ai j +bi j; that is,

ëa1 1 a1 2 . . . a1 nö ëb1 1 b1 2 . . . b1 nö

ìa2 1 a2 2 . . . a2 n÷ ì

÷ ÷

ì ì

b

2 1

b

2 2

. . . b

2

n

÷

ì .... ÷ + ì .... ÷

í . . ø í . . ø

am 1 am 2 . . . am n bm 1 bm 2 . . . bm n

ë a1 1 +b1 1 a1 2 +b1 2 . . . a1 n +b1 n ö ì a2 1 +b2 1 a2 2 +b2 2 . . . a2 n +b2 n ÷ =

ì... ÷

.

÷ .ì

í . . øam 1 +bm 1 am 2 +bm 2 . . . am n +bm n For any matrixA = (ai j), we let−A be the matrix (−ai j). Given a scalarλ∈K, we define the matrixλA as the matrixC = (ci j) such thatci j =λai j; that is

ëa1 1 a1 2 . . . a1 nö ë λa1 1 λa1 2 . . . λa1 nö ìa2 1 a2 2 . . . a2 n÷ ì

÷ ÷ ì

í

λa

2 1

λa

2 2

. . . λa

2

n

÷

λì ... . ÷ = ì ... . ÷ .ì . . ø í . . øam 1 am 2 . . . am n λam 1 λam 2 . . . λam n

Given anm×n matricesA = (ai k) and ann×p matricesB = (bk j), we define their product AB as them×p matrixC = (ci j) such that

n

ci j = ai kbk j,

k=1

for 1≤i≤m, and 1≤j≤p. In the productAB =C shown below

ëa1 1 a1 2 . . . a1 nö ëb1 1 b1 2 . . . b1 pö ëc1 1 c1 2 . . . c1 p ö ìa2 1 a2 2 . . . a2 n÷ ìb2 1 b2 2 . . . b2 p÷ ì

÷ ì ÷ ÷ ì ì

c

2 1

c

2 2

. . . c

2 p

÷ ì .... ÷ ì. . .... ÷ =ì .... ÷

í . . ø í ø í . . ø am 1 am 2 . . . am n bn 1 bn 2 . . . bn p cm 1 cm 2 . . . cm p

note that the entry of index i andj of the matrixAB obtained by multiplying the matrices A andB can be identified with the product of the row matrix corresponding to thei-th row ofA with the column matrix corresponding to thej-column ofB:

ëb1 jön

(ai 1· · ·ai n)ì .÷ = ai kbk j.í ø

bn j k=1

The square matrixIn of dimensionn containing 1 on the diagonal and 0 everywhere else is called the identity matrix. It is denoted as

ë1 0 . . . 0ö

ì

÷

ì

0

1

. . .

0

÷

ì ....÷

í . . ø

0 0 . . . 1

Given anm×n matrixA = (ai j), its transposeA = (aj i), is then×m-matrix such thataj i =ai j, for alli, 1≤i≤m, and allj, 1≤j≤n.

The transpose of a matrix A is sometimes denoted byAt, or even bytA. Note that the transposeA of a matrixA has the property that thej-th row ofA is thej-th column of A. In other words, transposition exchanges the rows and the columns of a matrix.

The following observation will be useful later on when we discuss the SVD. Given any m×n matrixA and anyn×p matrixB, if we denote the columns ofA byA^1, . . . , An and the rows ofB byB_1, . . . , Bn, then we have

AB =A^1B_1 +· · · +AnBn.

For every square matrix A of dimensionn, it is immediately verified thatAIn =InA =A. If a matrixB such thatAB =BA =In exists, then it is unique, and it is called the inverse ofA. The matrixB is also denoted byA−1. An invertible matrix is also called a nonsingular matrix, and a matrix that is not invertible is called a singular matrix.

Proposition 2.16 shows that if a square matrix A has a left inverse, that is a matrixB such thatBA =I, or a right inverse, that is a matrixC such thatAC =I, thenA is actually invertible; soB =A−1 andC =A−1. These facts also follow from Proposition 4.14.

It is immediately verified that the set Mm,n(K) ofm×n matrices is a vector space under addition of matrices and multiplication of a matrix by a scalar. Consider them×n-matrices Ei,j = (eh k), defined such thatei j = 1, andeh k = 0, ifh =i ork =j. It is clear that every matrixA = (ai j)∈ Mm,n(K) can be written in a unique way as

m n

A = ai jEi,j.

i=1 j=1

Thus, the family (Ei,j)1≤i≤m,1≤j≤n is a basis of the vector space Mm,n(K), which has dimensionmn.

Remark: Definition 3.1 and Definition 3.2 also make perfect sense whenK is a (commutative) ring rather than a field. In this more general setting, the framework of vector spaces is too narrow, but we can consider structures over a commutative ringA satisfying all the axioms of Definition 2.9. Such structures are called modules. The theory of modules is (much) more complicated than that of vector spaces. For example, modules do not always have a basis, and other properties holding for vector spaces usually fail for modules. When a module has a basis, it is called a free module. For example, whenA is a commutative ring, the structureAn is a module such that the vectorsei, with (ei)i = 1 and (ei)j = 0 for j =i, form a basis ofAn. Many properties of vector spaces still hold forAn. Thus,An is a free module. As another example, whenA is a commutative ring, Mm,n(A) is a free module with basis (Ei,j)1≤i≤m,1≤j≤n. Polynomials over a commutative ring also form a free module of infinite dimension.

Square matrices provide a natural example of a noncommutative ring with zero divisors. Example 3.1. For example, lettingA, B be the 2× 2-matrices

A =1 0 , B =0 0 ,0 0 1 0 then AB =1 0 0 0 =0 0 ,0 0 1 0 0 0

and

BA =0 0 1 0 =0 0 .1 0 0 0 1 0

We now formalize the representation of linear maps by matrices.

Definition 3.3. LetE andF be two vector spaces, and let (u_1, . . . , un) be a basis forE, and (v_1, . . . , vm) be a basis forF. Each vectorx∈E expressed in the basis (u_1, . . . , un) as x =x_1u_1 +· · · +xnun is represented by the column matrix

ëx1ö

M

(

x

) =

ì. ÷

í ø

xn

and similarly for each vectory∈F expressed in the basis (v_1, . . . , vm).

Every linear mapf :E→F is represented by the matrixM(f) = (ai j), whereai j is the i-th component of the vectorf(uj) over the basis (v_1, . . . , vm), i.e., where

m

f(uj) = ai jvi, for everyj, 1≤j≤n.

i=1

The coefficientsa1j, a2j, . . . , amj off(uj) over the basis (v_1, . . . , vm) form thejth column of the matrixM(f) shown below:

f(u_1) f(u_2) . . . f(un) v

ë ö 1 a_11 a_12 . . . a1n v

2

ì a_21 a_22 . . . a2n ÷

ì... ÷

.

÷ .. ì

í . . ø vm am1 am2 . . . amn

The matrix M(f) associated with the linear mapf :E→ F is called the matrix off with respect to the bases (u_1, . . . , un) and (v_1, . . . , vm). WhenE =F and the basis (v_1, . . . , vm) is identical to the basis (u_1, . . . , un) ofE, the matrixM(f) associated withf :E→E (as above) is called the matrix off with respect to the base (u_1, . . . , un).

Remark: As in the remark after Definition 3.1, there is no reason to assume that the vectors in the bases (u_1, . . . , un) and (v_1, . . . , vm) are ordered in any particular way. However, it is often convenient to assume the natural ordering. When this is so, authors sometimes refer to the matrixM(f) as the matrix off with respect to the ordered bases (u_1, . . . , un) and (v_1, . . . , vm).

Then, given a linear map f :E→F represented by the matrixM(f) = (ai j) w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm), by equations (1) and the definition of matrix multiplication, the equationy =f(x) correspond to the matrix equationM(y) =M(f)M(x), that

is, ëy_1ö ë a1 1 . . . a1 nöëx_1ö

ì. ÷ = ì. .... ÷ì. ÷.íy ø í øí ø

m am 1 . . . am n xn

Recall that

ëa1 1 a1 2 . . . a1 nö ëx1ö ëa1 1ö ëa1 2 ö ëa1 nö

ì a2 1 a2 2 . . . a2 n÷ ìx_2÷ ìa2 1÷ ì a2 2 ÷ ì ÷ ì ÷ ì ÷ ì ÷ ì ÷

ì

a

2 n

÷ ì .... ÷ ì. ÷ =x1 ì. ÷ +x2ì. ÷ +· · · +xnì. ÷ .

í . . ø í ø í ø í ø í ø am 1 am 2 . . . am n xn am 1 am 2 am n

Sometimes, it is necessary to incoporate the bases ( u_1, . . . , un) and (v_1, . . . , vm) in the notation for the matrixM(f) expressingf with respect to these bases. This turns out to be a messy enterprise!

We propose the following course of action: writeU = (u_1, . . . , un) andV = (v_1, . . . , vm)

for the bases of E andF, and denote byMU,V(f) the matrix off with respect to the basesUandV. Furthermore, writexU for the coordinatesM(x) = (x_1, . . . , xn) ofx∈E w.r.t. the basisU and writeyV for the coordinatesM(y) = (y_1, . . . , ym) ofy∈F w.r.t. the basisV . Then,

y =f(x) is expressed in matrix form by

yV =MU,V(f)xU. WhenU =V, we abbreviateMU,V(f) asMU(f).

The above notation seems reasonable, but it has the slight disadvantage that in the expressionMU,V(f)xU, the input argumentxU which is fed to the matrixMU,V(f) does not appear next to the subscriptU inMU,V(f). We could have used the notationMV,U(f), and some people do that. But then, we find a bit confusing thatV comes beforeU whenf maps from the spaceE with the basisU to the spaceF with the basisV. So, we prefer to use the notationMU,V(f).

Be aware that other authors such as Meyer [77] use the notation [ f]U,V, and others such as Dummit and Foote [30] use the notationMVU(f), instead ofMU,V(f). This gets worse! You may find the notationMUV (f) (as in Lang [65]), orU[f]V, or other strange notations.

Let us illustrate the representation of a linear map by a matrix in a concrete situation. LetE be the vector space R[X]4 of polynomials of degree at most 4, letF be the vector space R[X]3 of polynomials of degree at most 3, and let the linear map be the derivative mapd: that is,

d(P +Q) =dP +dQ

d(λP) =λdP,

with λ∈ R. We choose (1, x, x^2, x^3, x^4) as a basis ofE and (1, x, x^2, x^3) as a basis ofF. Then, the 4× 5 matrixD associated withd is obtained by expressing the derivativedxi of each basis vector fori = 0,1,2,3,4 over the basis (1, x, x^2, x^3). We find

ë 0 1 0 0 0ö

ì0 0 2 0 0÷.í0 0 0 3 0÷D = ì

ø 0 0 0 0 4

Then, if P denotes the polynomial P = 3x^4 5x^3 +x^2 7x + 5,− −

we have

dP = 12x^3 15x^2 + 2x− 7,−

the polynomialP is represented by the vector (5,−7,1,−5,3) anddP is represented by the vector (−7,2,−15,12), and we have

ë0 1 0 0 0öë 5ö ë

ì

ì ì

7 ö

ì

0

0

2

0

0

÷

ì

−

7

÷

÷ − ÷,í0 0 0 3 0÷ ì_1 ÷ =ì 2 ÷

øì 5÷ í−15ø 0 0 0 0 4í−ø 12

3

as expected! The kernel (nullspace) ofd consists of the polynomials of degree 0, that is, the constant polynomials. Therefore dim(Kerd) = 1, and from

dim(E) = dim(Kerd) + dim(Imd) (see Theorem 4.11), we get dim(Imd) = 4 (since dim(E) = 5).

For fun, let us figure out the linear map from the vector space R[X]3 to the vector space R[X]4 given by integration (finding the primitive, or anti-derivative) ofxi, fori = 0,1,2,3). The 5× 4 matrixS representing with respect to the same bases as before is

ë0 0 0 0ö

ì1 0 0 0÷

S

=

ì0 1/2 0 0÷

ì ÷ .ì0 0 1/3 0÷

í ø

0 0 0 1/4

We verify thatDS =I_4,

ë0 1 0 0 0ö ë0 0 0 0 ö ë1 0 0 0ö

ì

0

0

2

0

0

ì

÷ ì1 0 0 0 ÷ 0 1 0 0÷,í0 0 0 3 0÷ ì0 1/2 0 0 ÷= ì

ì ÷ ì0 0 1 0÷

øì0 0 1/3 0 ÷ í ø 0 0 0 0 4í ø 0 0 0 10 0 0 1/4

as it should! The equationDS =I_4 show thatS is injective and hasD as a left inverse. However,SD =I_5, and instead

ë0 0 0 0öë0 1 0 0 0öë0 0 0 0 0ö

ì1 0 0 0 ÷0 0 2 0 0 ì

÷

=

ì

0

1

0

0

0

÷

ì0 1/2 0 0÷ì 0 0 1 0 0÷

ì ÷ì ,ì0 0 1/3 0 ÷í0 0 0 3 0÷ ì ÷

ø

ì0 0 0 1 0÷ í ø0 0 0 0 4 í ø 0 0 0 1/4 0 0 0 0 1 because constant polynomials (polynomials of degree 0) belong to the kernel ofD.

The function that associates to a linear map f :E→F the matrixM(f) w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm) has the property that matrix multiplication corresponds to composition of linear maps. This allows us to transfer properties of linear maps to matrices. Here is an illustration of this technique:

Proposition 3.1. (1) Given any matricesA∈ Mm,n(K),B∈ Mn,p(K), andC∈ Mp,q(K), we have

(AB)C =A(BC);

that is, matrix multiplication is associative.

(2) Given any matricesA, B∈ Mm,n(K), andC, D∈ Mn,p(K), for allλ∈K, we have

( A +B)C =AC +BC

A(C +D) =AC +AD

(λA)C =λ(AC)

A(λC) =λ(AC),

so that matrix multiplication·: Mm,n(K)× Mn,p(K)→ Mm,p(K) is bilinear.

Proof. (1) Everym×n matrixA = (ai j) defines the functionfA:Kn Km given by→

fA(x) =Ax,

for all x∈ Kn. It is immediately verified thatfA is linear and that the matrixM(fA) representingfA over the canonical bases inKn andKm is equal toA. Then, formula (4) proves that

M(fAæfB) =M(fA)M(fB) =AB, so we get M((fAæfB)æfC) =M(fAæfB)M(fC) = (AB)C and

M(fAæ (fBæfC)) =M(fA)M(fBæfC) =A(BC),

and since composition of functions is associative, we have (fAæfB)æfC =fAæ (fBæfC), which implies that

(AB)C =A(BC).

(2) It is immediately verified that iff_1, f2∈ HomK(E, F),A, B∈ Mm,n(K), (u_1, . . . , un) is any basis ofE, and (v_1, . . . , vm) is any basis ofF, then

M(f_1 +f_2) =M(f_1) +M(f_2) fA+B =fA +fB.

Then we have

( A +B)C =M(fA+B)M(fC)

=M(fA+BæfC)

=M((fA +fB)æfC))

=M((fAæfC) + (fBæfC)) =M(fAæfC) +M(fBæfC) =M(fA)M(fC) +M(fB)M(fC) =AC +BC.

The equation A(C +D) = AC +AD is proved in a similar fashion, and the last two equations are easily verified. We could also have verified all the identities by making matrix computations.

Note that Proposition 3.1 implies that the vector space Mn(K) of square matrices is a (noncommutative) ring with unitIn. (It even shows that Mn(K) is an associative algebra.) The following proposition states the main properties of the mappingf→M(f) between Hom(E, F) and Mm,n. In short, it is an isomorphism of vector spaces.

Proposition 3.2. Given three vector spacesE, F, G, with respective bases (u_1, . . . , up), (v_1, . . . , vn), and (w_1, . . . , wm), the mappingM : Hom(E, F)→ Mn,p that associates the matrixM(g) to a linear mapg:E→F satisfies the following properties for all x∈E, all g, h:E→F, and allf :F→G:

M(g(x)) =M(g)M(x) M(g +h) =M(g) +M(h) M(λg) =λM(g)

M(fæg) =M(f)M(g).

Thus, M : Hom(E, F)→ Mn,p is an isomorphism of vector spaces, and whenp = n and the basis (v_1, . . . , vn) is identical to the basis (u_1, . . . , up),M : Hom(E, E)→ Mn is an isomorphism of rings.

Proof. ThatM(g(x)) =M(g)M(x) was shown just before stating the proposition, using identity (1). The identitiesM(g +h) =M(g) +M(h) andM(λg) =λM(g) are straightforward, andM(fæg) =M(f)M(g) follows from (4) and the definition of matrix multiplication. The mappingM : Hom(E, F)→ Mn,p is clearly injective, and since every matrix defines a linear map, it is also surjective, and thus bijective. In view of the above identities, it is an isomorphism (and similarly forM : Hom(E, E)→ Mn).

In view of Proposition 3.2, it seems preferable to represent vectors from a vector space of finite dimension as column vectors rather than row vectors. Thus, from now on, we will denote vectors of Rn (or more generally, ofKn) as columm vectors.

It is important to observe that the isomorphism M : Hom(E, F)→ Mn,p given by Proposition 3.2 depends on the choice of the bases (u_1, . . . , up) and (v_1, . . . , vn), and similarly for the isomorphismM : Hom(E, E)→ Mn, which depends on the choice of the basis (u_1, . . . , un). Thus, it would be useful to know how a change of basis affects the representation of a linear mapf :E→F as a matrix. The following simple proposition is needed.

Proposition 3.3. LetE be a vector space, and let (u_1, . . . , un) be a basis ofE. For every family (v_1, . . . , vn), letP = (ai j) be the matrix defined such thatvj =n ai jui. The matrix P is invertible iff (v_1, . . . , vn) is a basis ofE.i=1

Proof. Note that we haveP =M(f), the matrix associated with the unique linear map f :E→E such thatf(ui) =vi. By Proposition 2.13,f is bijective iff (v_1, . . . , vn) is a basis ofE. Furthermore, it is obvious that the identity matrixIn is the matrix associated with the identity id:E→E w.r.t. any basis. Iff is an isomorphism, thenfæf−1 =f−1 f = id, and

by Proposition 3.2, we get

M

(

f

)

M

(

f

−

1

) =

M

(

f

−

1

æ

)M(f) =In, showing thatP is invertible and thatM(f−1) =P−1.

Proposition 3.3 suggests the following definition.

Definition 3.4. Given a vector spaceE of dimensionn, for any two bases (u_1, . . . , un) and (v_1, . . . , vn) ofE, letP = (ai j) be the invertible matrix defined such that

n

vj = ai jui,

i=1

which is also the matrix of the identity id: E→E with respect to the bases (v_1, . . . , vn) and (u_1, . . . , un), in that order. Indeed, we express each id(vj) =vj over the basis (u_1, . . . , un). The coefficientsa1j, a2j, . . . , anj ofvj over the basis (u_1, . . . , un) form thejth column of the matrixP shown below:

v_1 v_2 . . . vn u_1 ëa_11 a_12 . . . a1nö u

2

ìa_21 a_22 . . . a2n÷

ì... ÷

.

÷ .. ì

í . . ø un an1 an2 . . . ann

The matrixP is called the change of basis matrix from (u_1, . . . , un) to (v_1, . . . , vn).

Clearly, the change of basis matrix from ( v_1, . . . , vn) to (u_1, . . . , un) isP−1. SinceP = (ai,j) is the matrix of the identity id:E→E with respect to the bases (v_1, . . . , vn) and (u_1, . . . , un), given any vectorx∈E, ifx =x1u_1+· · ·+xnun over the basis (u_1, . . . , un) and x =x_1v_1 +· · · +xnvn over the basis (v_1, . . . , vn), from Proposition 3.2, we have

ëx_1ö ëa1 1 . . . a1 nöëxö

1

ì. ÷ = ì. .... ÷ì. ÷,íx ø í øí ø

n an 1 . . . an n xn

showing that the old coordinates (xi) ofx (over (u_1, . . . , un)) are expressed in terms of the new coordinates (xi) ofx (over (v_1, . . . , vn)).

Now we face the painful task of assigning a “good” notation incorporating the bases U = (u_1, . . . , un) andV = (v_1, . . . , vn) into the notation for the change of basis matrix from U toV. Because the change of basis matrix fromU toV is the matrix of the identity map idE with respect to the basesV andU in that order, we could denote it byMV,U(id) (Meyer [77] uses the notation [I]V,U), which we abbreviate as

PV,U.

Note that P

U

,

V

=

P

−

1

,U.V

Then, if we writexU = (x_1, . . . , xn) for the old coordinates ofx with respect to the basisUandxV = (x1, . . . , xn) for the new coordinates ofx with respect to the basisV, we have

x

U =PV,UxV, xV =P−, 1x

U U

.

V

The above may look backward, but remember that the matrix MU,V(f) takes input expressed over the basisU to output expressed over the basisV. Consequently,PV,U takes input expressed over the basisV to output expressed over the basisU, andxU =PV,UxVmatches this point of view!

Beware that some authors (such as Artin [3]) define the change of basis matrix fromU toV asPU,V =P−,U U is expressed in terms of

the new basis

1. Under this point of view, the old basis

V

V . We find this a bit unnatural. Also, in practice, it seems that the new basis is often expressed in terms of the old basis, rather than the other way around.

Since the matrix P =PV,U expresses the new basis (v_1, . . . , vn) in terms of the old basis (u_1, . . .,un), we observe that the coordinates (xi) of a vectorx vary in the opposite direction of the change of basis. For this reason, vectors are sometimes said to be contravariant. However, this expression does not make sense! Indeed, a vector in an intrinsic quantity that does not depend on a specific basis. What makes sense is that the coordinates of a vector vary in a contravariant fashion.

Let us consider some concrete examples of change of bases.

Example 3.2. LetE =F = R^2, withu_1 = (1,0),u_2 = (0,1),v_1 = (1,1) andv_2 = (−1,1). The change of basis matrixP from the basisU = (u_1, u_2) to the basisV = (v_1, v_2) is

1 −1P =1 1

and its inverse is

P−1 =1/2 1/2 .

−1/2 1/2

The old coordinates (x_1, x_2) with respect to (u_1, u_2) are expressed in terms of the new coordinates (x_1, x_2) with respect to (v_1, v_2) by

x1 = 1 −1 x1 ,x2 1 1 x2

and the new coordinates (x_1, x_2) with respect to (v_1, v_2) are expressed in terms of the old coordinates (x_1, x_2) with respect to (u_1, u_2) by

x1 = 1/2 1/2 x1 .x2 −1/2 1/2 x2

Example 3.3. Let E = F = R[X]3 be the set of polynomials of degree at most 3, and consider the basesU = (1, x, x^2, x^3) andV = (B^3(x), B^3(x), B^3(x), B^3(x)), where B

3

(

x

)

, B

3

(

x

)

, B

3

(

x

)

, B

3

0 1 2 3

( x) are the Bernstein polynomials of degree 3, given by0 1 2 3

B^3(x) = (1−x)3 B^3(x) = 3(1−x)2x B^3(x) = 3(1−x)x^2 B^3(x) =x^3.0 1 2 3

By expanding the Bernstein polynomials, we find that the change of basis matrixPV,U is given byë 1 0 0 0ö

ì 3 3 0 0÷.V,U =ì−6 3 0÷Pí 3 − ø

3 1−1 3 −

We also find that the inverse ofPV,U is ë1 0 0 0ö ì1 1/3 0 0÷.

V

,

U

í

1

2

/

3

1

/

3

0 ÷P− 1 = ì

ø

1 1 1 1

Therefore, the coordinates of the polynomial 2x^3 x + 1 over the basisV are ë_1 ö ë1 0 − 0öë_1 ö_0

ì2/3÷ = ì1 1/3 0 0÷ì 1÷

ì ,í1/3÷ ì1 2/3 1/3 0÷ì− ÷

ø í øí_0 ø

2 1 1 1 1 2

and so

2

x

3 ) + 2B^3(x) + 1B^3(x) + 2B^3(x).−x + 1 =B^3

0(x 1_3 2 3_3

Our next example is the Haar wavelets, a fundamental tool in signal processing.

3.2 Haar Basis Vectors and a Glimpse at Wavelets

We begin by considering Haar wavelets in R^4. Wavelets play an important role in audio and video signal processing, especially for compressing long signals into much smaller ones than still retain enough information so that when they are played, we can’t see or hear any difference.

Consider the four vectorsw_1, w_2, w_3, w_4 given by

ë_1ö ë_1 ö ë_1 ö ë_0 ö

ì1÷ w_2 =ì 1 ÷ w_3 =ì 1÷ w_4 =ì 0 ÷.1 = ì_1÷ ì_1÷ ì− ÷ ì_1 ÷wí ø í 0 ø í ø 1

ø í−1 0 −1−

Note that these vectors are pairwise orthogonal, so they are indeed linearly independent (we will see this in a later chapter). LetW ={w_1, w_2, w_3, w4} be the Haar basis, and let U ={e_1, e_2, e_3, e4} be the canonical basis of R^4. The change of basis matrixW =PW,U from U toW is given byë1 1 1 0 ö

ì1 1 −1 0 ÷,í1 −1 0 1 ÷W = ì

ø 1 −1 0 −1

and we easily find that the inverse ofW is given by

ë1/4 0 0 0ö ë1 1 1 1ö W

−

1 = ì ÷

ø í

ì 0 1/4 0 0 ÷ ì1 1 −1 −1÷.í 0 0 1/2 0 ÷ ì1 −1 0 0 ø

0 0 0 1/2 0_0 1 −1 So, the vectorv = (6,4,5,1) over the basisU becomesc = (c_1, c_2, c_3, c_4) over the Haar basis W, with

ëc1ö ë1/4 0 0 0 öë1 1 1 1 ö ë_6ö ë_4ö

ìc_2÷ = ì 0 1/4 0 0 ÷ì1 1 −1 −1÷ ì4÷ = ì1÷ ì ÷ ì ÷ ì .íc3ø í 0 0 1/2 0 ÷ì1 −1 0 0 ø í5÷ ì_1÷

øí ø í ø c_4 0 0 0 1/2 0_0 1 −1 1 2

Given a signalv = (v_1, v_2, v_3, v_4), we first transformv into its coefficientsc = (c_1, c_2, c_3, c_4) over the Haar basis by computingc =W−1v. Observe that

c

1

=

v_1 +v_2 +v_3 +v_4

4

is the overall average value of the signalv. The coefficientc_1 corresponds to the background of the image (or of the sound). Then,c_2 gives the coarse details ofv, whereas,c_3 gives the details in the first part ofv, andc_4 gives the details in the second half ofv.

Reconstruction of the signal consists in computingv =W c. The trick for good compression is to throw away some of the coefficients ofc (set them to zero), obtaining a compressed signalc, and still retain enough crucial information so that the reconstructed signalv =W c looks almost as good as the original signalv. Thus, the steps are:

inputv −→ coefficientsc =W−1v −→ compressedc−→ compressedv =W c.

This kind of compression scheme makes modern video conferencing possible. It turns out that there is a faster way to findc =W−1v, without actually usingW−1. This has to do with the multiscale nature of Haar wavelets.

Given the original signal v = (6,4,5,1) shown in Figure 3.1, we compute averages and half differences obtaining Figure 3.2. We get the coefficientsc_3 = 1 andc_4 = 2. Then, again we compute averages and half differences obtaining Figure 3.3. We get the coefficientsc_1 = 4 andc_2 = 1.

6451

Figure 3.1: The original signalv

55 331 2

−1 2− Figure 3.2: First averages and first half differences

11

4444

−1 −1

Figure 3.3: Second averages and second half differences

Note that the original signal v can be reconstruced from the two signals in Figure 3.2, and the signal on the left of Figure 3.2 can be reconstructed from the two signals in Figure 3.3.

This method can be generalized to signals of any length 2n. The previous case corresponds ton = 2. Let us consider the casen = 3. The Haar basis (w_1, w_2, w_3, w_4, w_5, w_6, w_7, w_8) is given by the matrix

ë 1 1 1 0 1 0 0 0ö

ì1 1 1 0 −1 0 0 0 ÷

ì1 1 −1 0_0 1 0 0÷

ì ÷

ì1 1 −1 0 0 −1 0 0÷

W

=

ì ÷

ì1 −1 0 1 0_0 1 0÷

ì ÷

ì1 −1 0 1 0 0 −1 0÷

ì ÷

ì1 −1 0 −1 0 0_0 1÷

í ø

1 −1 0 −1 0 0 0 −1

The columns of this matrix are orthogonal and it is easy to see that

W−1 = diag(1/8,1/8,1/4,1/4,1/2,1/2,1/2,1/2)W .

A pattern is begining to emerge. It looks like the second Haar basis vector w_2 is the “mother” of all the other basis vectors, except the first, whose purpose is to perform averaging. Indeed, in general, given

w_2 = (1, . . . ,1,−1, . . . ,−1),

2n

the other Haar basis vectors are obtained by a “scaling and shifting process.” Starting from w_2, the scaling process generates the vectors

w_3, w_5, w_9, . . . , w2j+1, . . . , w2n−1+1,

such thatw2j+1+1 is obtained fromw2j+1 by forming two consecutive blocks of 1 and−1 of half the size of the blocks inw2j+1, and setting all other entries to zero. Observe that w2j+1 has 2j blocks of 2n−j elements. The shifting process, consists in shifting the blocks of 1 and−1 inw2j+1 to the right by inserting a block of (k− 1)2n−j zeros from the left, with 0≤j≤n− 1 and 1≤k≤ 2j. Thus, we obtain the following formula forw2j+k:

ñ0 1≤i≤ (k− 1)2n−j

ô

ò

1

(

k

−

1)2

n−j + 1≤i≤ (k− 1)2n−j + 2n−j−1 w2j+k(i) = 1 (k− 1)2n−j + 2n−j−1 + 1≤i≤k2n−j

ô

ó− k2n−j + 1≤i≤ 2n,0

with 0≤j≤n− 1 and 1≤k≤ 2j. Of course w_1 = (1, . . . ,1).

2n

The above formulae look a little better if we change our indexing slightly by letting k vary from 0 to 2j 1 and using the indexj instead of 2j. In this case, the Haar basis is denoted by−

w_1, h^0, h^1, h^1, h^2, h^2, h^2, h^2, . . . , hj, . . . , hn−1 ,0 0 1 0 1 2 3 k 2n −1 1−

and ñ0 1≤i≤k2n−j

ô

ò

1

k

2

n−j + 1≤i≤k2n−j + 2n−j−1

h

j

k(i) = 1 k2n−j + 2n−j−1 + 1≤i≤ (k + 1)2n−j

ô

ó

0

− (k + 1)2n−j + 1≤i≤ 2n,

with 0j 1.≤j≤n− 1 and 0≤k≤ 2−

It turns out that there is a way to understand these formulae better if we interpret a vectoru = (u_1, . . . , um) as a piecewise linear function over the interval [0,1). We define the function plf(u) such that

plf(u)(x) =ui,i− 1 x < im, 1≤i≤m.

m ≤

In words, the function plf( u) has the valueu_1 on the interval [0,1/m), the valueu_2 on [1/m,2/m), etc., and the valueum on the interval [(m−1)/m,1). For example, the piecewise linear function associated with the vector

u = (2.4,2.2,2.15,2.05,6.8,2.8,−1.1,−1.3) is shown in Figure 3.4.

Then, each basis vectorhj corresponds to the functionk

ψj = plf(hj).k k

7

6

5

4

3

2

1

0

−1

−2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 3.4: The piecewise linear function plf(u)

In particular, for alln, the Haar basis vectors

h^0 =w_2 = (1, . . . ,1,−1, . . . ,−1)0

2n

yield the same piecewise linear functionψ given by ñ

ô1 if 0≤x < 1/2 ψ(x) =ò 1 if 1/2≤x < 1

ô

ó

0

− otherwise,

whose graph is shown in Figure 3.5. Then, it is easy to see thatψj is given by the simplek

1

0^1

−1

Figure 3.5: The Haar waveletψ

expression ψj(x) =ψ(2jx−k), 0≤j≤n− 1, 0≤k≤ 2j 1.k −

The above formula makes it clear thatψj is obtained fromψ by scaling and shifting. The function

φ

0

k

= plf(w_1) is the piecewise linear function with the constant value 1 on [0,1), and_0

the functionsψj together withÕ^0 are known as the Haar wavelets.k 0

Rather than using W−1 to convert a vectoru to a vectorc of coefficients over the Haar basis, and the matrixW to reconstruct the vectoru from its Haar coefficientsc, we can use faster algorithms that use averaging and differencing.

Ifc is a vector of Haar coefficients of dimension 2n, we compute the sequence of vectors u_0, u_1, . . .,un as follows:

u_0 =c

uj+1 =uj

uj+1(2i− 1) =uj(i) +uj(2j +i) uj+1(2i) =uj(i)−uj(2j +i),

forj = 0, . . . , n− 1 andi = 1, . . . ,2j. The reconstructed vector (signal) isu =un. Ifu is a vector of dimension 2n, we compute the sequence of vectorscn, cn−1, . . . , c_0 as follows:

cn =u

cj =cj+1

cj(i) = (cj+1(2i− 1) +cj+1(2i))/2 cj(2j +i) = (cj+1(2i− 1)−cj+1(2i))/2, forj =n− 1, . . . ,0 andi = 1, . . . ,2j. The vector over the Haar basis isc =c_0.

We leave it as an exercise to implement the above programs in Matlab using two variables u andc, and by building iteratively 2j. Here is an example of the conversion of a vector to its Haar coefficients forn = 3.

Given the sequenceu = (31,29,23,17,−6,−8,−2,−4), we get the sequence

c_3 = (31,29,23,17,−6,−8,−2,−4) c_2 = (30,20,−7,−3,1,3,1,1) c_1 = (25,−5,5,−2,1,3,1,1)

c_0 = (10,15,5,−2,1,3,1,1),

soc = (10,15,5,−2,1,3,1,1). Conversely, givenc = (10,15,5,−2,1,3,1,1), we get the sequence

u_0 = (10,15,5,−2,1,3,1,1)

u_1 = (25,−5,5,−2,1,3,1,1)

u_2 = (30,20,−7,−3,1,3,1,1) u_3 = (31,29,23,17,−6,−8,−2,−4),

which gives backu = (31,29,23,17,−6,−8,−2,−4).

There is another recursive method for constucting the Haar matrix Wn of dimension 2n that makes it clearer why the above algorithms are indeed correct (which nobody seems to prove!). If we splitWn into two 2n 2n−1 matrices, then the second matrix containing the last 2

n

−

1

×

columns ofWn has a very simple structure: it consists of the vector

(1,−1,0, . . . ,0)

2n

and 2n−1 1 shifted copies of it, as illustrated below forn = 3:−

ë 1 0 0 0ö ì 1 0 0 0 ÷

ì−1 0 0 ÷ ì_0 ÷

ì 0 −1 0 0÷

ì ÷

ì .ì 0_0 1 0÷

÷

ì 0 0 −1 0÷

ì ÷

ì 0 0_0 1÷ í ø 0 0 0 −1

Then, we form the 2n 2n−2 matrix obtained by “doubling” each column of odd index, which×

means replacing each such column by a column in which the block of 1 is doubled and the block of− 1 is doubled. In general, given a current matrix of dimension 2n 2j, we form a

2

n 2j− matrix by doubling each column of odd index, which means that we replace each×

×

such column by a column in which the block of 1 is doubled and the block of−1 is doubled. We repeat this processn− 1 times until we get the vector

(1, . . . ,1,−1, . . . ,−1) .

2n

The first vector is the averaging vector (1, . . . ,1). This process is illustrated below forn = 3:

2n

ë_1 ö ë1 0 ö ë1 0 0 0 ö

ì 1 ÷ ì 1 0 ÷ ì 1 0 0 0 ÷ ì_1 ÷ ì1 0 ÷ ì−1 0 0 ÷ ì ÷ ì ÷ ì_0 ÷ ì_1 ÷ ì−1 0 ÷ ì0 −1 0 0 ÷ ì ÷= ì ÷= ì ÷ ì_1÷ ì−1 ÷ ì0 0 1 0 ÷ ì ÷ ⇐ ì_0 ÷ ⇐ ì ÷ ì−1÷ ì0 1 ÷ ì0 0 −1 0 ÷ ì ÷ ì ÷ ì ÷ ì−1÷ ì0 −1÷ ì0 0 0 1 ÷

í− ø í ø í ø 1 0 −1 0 0 0 −1−

Adding (1, . . . ,1,1, . . . ,1) as the first column, we obtain

2n

ë 1 1 1 0 1 0 0 0ö ì1 1 1 0 −1 0 0 0 ÷ ì1 1 −1 0_0 1 0 0÷

ì ÷

ì1 1 −1 0 0 −1 0 0÷ W

3

=

ì ÷

ì .ì1 −1 0 1 0_0 1 0÷

÷

ì1 −1 0 1 0 0 −1 0÷

ì ÷

ì1 −1 0 −1 0 0_0 1÷ í ø 1 −1 0 −1 0 0 0 −1

Observe that the right block (of size 2n 2n−1) shows clearly how the detail coefficients×

in the second half of the vectorc are added and subtracted to the entries in the first half of the partially reconstructed vector aftern− 1 steps.

An important and attractive feature of the Haar basis is that it provides a multiresolution analysis of a signal. Indeed, given a signalu, ifc = (c_1, . . . , c2n) is the vector of its Haar coefficients, the coefficients with low index give coarse information aboutu, and the coefficients with high index represent fine information. For example, ifu is an audio signal corresponding to a Mozart concerto played by an orchestra,c_1 corresponds to the “background noise,”c_2 to the bass,c_3 to the first cello,c_4 to the second cello,c_5, c_6, c_7, c_7 to the violas, then the violins, etc. This multiresolution feature of wavelets can be exploited to compress a signal, that is, to use fewer coefficients to represent it. Here is an example.

Consider the signal

u = (2.4,2.2,2.15,2.05,6.8,2.8,−1.1,−1.3),

whose Haar transform is c = (2,0.2,0.1,3,0.1,0.05,2,0.1).

The piecewise-linear curves corresponding to u andc are shown in Figure 3.6. Since some of the coefficients inc are small (smaller than or equal to 0.2) we can compressc by replacing them by 0. We get

c_2 = (2,0,0,3,0,0,2,0),

and the reconstructed signal is

u_2 = (2,2,2,2,7,3,−1,−1).

The piecewise-linear curves corresponding tou_2 andc_2 are shown in Figure 3.7.

7_3

6

2.5

5

4 2

3

1.5

2

1^1

0

0.5 −1

−

2

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 3.6: A signal and its Haar transform

7^3

6 2.5

5

2 4

3 1.5

2_1

1

0.5

0

−

1

0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Figure 3.7: A compressed signal and its compressed Haar transform

An interesting (and amusing) application of the Haar wavelets is to the compression of audio signals. It turns out that if your type load handel in Matlab an audio file will be loaded in a vector denoted byy, and if you type sound(y), the computer will play this piece of music. You can converty to its vector of Haar coefficients,c. The length ofy is 73113, so first tuncate the tail ofy to get a vector of length 65536 = 2^16. A plot of the signals corresponding toy andc is shown in Figure 3.8. Then, run a program that sets all coefficients ofc whose absolute value is less that 0.05 to zero. This sets 37272 coefficients to 0. The resulting vectorc_2 is converted to a signaly_2. A plot of the signals corresponding toy_2 andc_2 is shown in Figure 3.9. When you type sound(y2), you find that the music doesn’t differ much from the original, although it sounds less crisp. You should play with other numbers greater than or less than 0.05. You should hear what happens when you type sound(c). It plays the music corresponding to the Haar transformc ofy, and it is quite funny.

Another neat property of the Haar transform is that it can be instantly generalized to

0.8 0.6

0.60.4

0.4 0.2

0.2

0 0

−0.2 −0.2

−0.4 −0.4

−0.6 −0.6

−0.8 0 1 2 3 4 5 6 7−0.80 1 2 3 4 5 6 7 x 104 4x 10

Figure 3.8: The signal “handel” and its Haar transform

1 0.6

0.8

0.4

0.6

0.40.2

0.2 0

0

−0.2 −0.2

−0.4 −0.4

−0.6 −0.6 −0.8

−10 1 2 3 4 5 6 7−0.80 1 2 3 4 5 6 7 x 10^4 x 10^4

Figure 3.9: The compressed signal “handel” and its Haar transform

matrices (even rectangular) without any extra effort! This allows for the compression of digital images. But first, we address the issue of normalization of the Haar coefficients. As we observed earlier, the 2n 2n matrixWn of Haar basis vectors has orthogonal columns,×

but its columns do not have unit length. As a consequence,Wn is not the inverse ofWn, but rather the matrix

W−1 =DnW

n n

withDn = diag 2−n, 2−n,2−(n−1),2−(n−1),2−(n−2), . . . ,2−(n−2), . . . ,2−1, . . . ,2−1 .

2^0 21 22 2n−1

Therefore, we define the orthogonal matrix

1

Hn =WnD^2 whose columns are the normalized Haar basis vectors, with

1

D

2

= diag

2

−2

n, 2 n n−1 n−1 n−2 n−2 1 1

−2 ,2− 2 ,2− 2 ,2− 2 , . . . ,2− 2 , . . . ,2−2, . . . ,2−2 .

2^0 21 22 2n−1

We callHn the normalized Haar transform matrix. BecauseHn is orthogonal,H−1 =Hn .

n

Given a vector (signal) u, we callc =Hnu the normalized Haar coefficients ofu. Then, a moment of reflexion shows that we have to slightly modify the algorithms to computeHnu andHnc as follows: When computing the sequence ofujs, use

uj+1(2i− 1) = (uj(i) +uj(2j +i))/√2 uj+1(2i) = (uj(i)−uj(2j +i))/√2, and when computing the sequence ofcjs, use

cj(i) = (cj+1(2i− 1) +cj+1(2i))/√2 cj(2j +i) = (cj+1(2i− 1)−cj+1(2i))/√2.

Note that things are now more symmetric, at the expense of a division by√2. However, for long vectors, it turns out that these algorithms are numerically more stable.

Remark: Some authors (for example, Stollnitz, Derose and Salesin [99]) rescalec by 1/√2n andu by√2n. This is because the norm of the basis functionsψj is not equal to 1 (under

1k

the inner product f, g =0f(t)g(t)dt). The normalized basis functions are the functions √2jψj.k

Let us now explain the 2D version of the Haar transform. We describe the version using the matrixWn, the method usingHn being identical (except thatH−1 =Hn , but this does

not hold for

W

−

1

). Given a 2

m

2

n

n

matrixA, we can first convert the rows ofA to their

n ×

Haar coefficients using the Haar transformW−

n

1, obtaining a matrixB, and then convert the columns ofB to their Haar coefficients, using the matrixW−1 . Because columns and rows

m

are exchanged in the first step,

B =A(W−1) ,

n

and in the second stepC =W−1B, thus, we have

m

C =W−1A(W−1) =DmWmAWnDn.m n

In the other direction, given a matrix C of Haar coefficients, we reconstruct the matrixA (the image) by first applyingWm to the columns ofC, obtainingB, and thenWn to the rows ofB. Therefore

A =WmCWn.

Of course, we dont actually have to invertWm andWn and perform matrix multiplications. We just have to use our algorithms using averaging and differencing. Here is an example. If the data matrix (the image) is the 8× 8 matrix

ë 64 2 3 61 60 6 7 57ö ì 9 55 54 12 13 51 50 16÷ ì17 47 46 20 21 43 42 24÷

ì ÷

ì40 26 27 37 36 30 31 33÷ A

=

ì ÷

ì ,ì32 34 35 29 28 38 39 25÷

÷

ì41 23 22 44 45 19 18 48÷

ì ÷

ì49 15 14 52 53 11 10 56÷ í ø 8 58 59 5 4 62 63 1

then applying our algorithms, we find that

ë32.5 0 0 0 0 0 0 0ö ì 0 0 0 0 0 0 0 0÷ ì 0 0 0 0 4 −4 4 −4÷

ì ÷

ì 0 0 0 0 4 − 4 4 − 4÷ C

=

ì ÷

ì .ì 0 0 0.5 0.5 27 −25 23 −21÷

÷

ì 0 0 −0.5 −0.5 −11 9 −7 5 ÷

ì ÷

ì 0 0_0.5_0.5 −5 7 −9 11÷ í ø

0 0 −0.5 −0.5 21 −23 25 −27 As we can see,C has a more zero entries thanA; it is a compressed version ofA. We can further compressC by setting to 0 all entries of absolute value at most 0.5. Then, we get

ë32.5 0 0 0 0 0 0 0ö

ì 0 0 0 0 0 0 0 0 ÷

ì 0 0 0 0 4 −4 4 −4 ÷

ì ÷

ì 0 0 0 0 4 − 4 4 − 4÷

C

2

=

ì ÷

ì .ì 0 0 0 0 27 −25 23 −21÷

÷

ì 0 0 0 0 −11 9 −7 5÷

ì ÷

ì 0 0 0 0 −5 7 −9 11÷

í ø

0 0 0 021 −23 25 −27

We find that the reconstructed image is

ë 63.5 1.5 3.5 61.5 59.5 5.5 7.5 57.5ö ì 9.5 55.5 53.5 11.5 13.5 51.5 49.5 15.5÷ ì17.5 47.5 45.5 19.5 21.5 43.5 41.5 23.5÷

ì ÷

ì39.5 25.5 27.5 37.5 35.5 29.5 31.5 33.5÷ A

2

=

ì ÷

ì ,ì31.5 33.5 35.5 29.5 27.5 37.5 39.5 25.5÷

÷

ì41.5 23.5 21.5 43.5 45.5 19.5 17.5 47.5÷

ì ÷

ì49.5 15.5 13.5 51.5 53.5 11.5 9.5 55.5÷ í ø 7.5 57.5 59.5 5.5 3.5 61.5 63.5 1.5 which is pretty close to the original image matrixA.

It turns out that Matlab has a wonderful command, image(X), which displays the matrix X has an image in which each entry is shown as a little square whose gray level is proportional to the numerical value of that entry (lighter if the value is higher, darker if the value is closer to zero; negative values are treated as zero). The images corresponding toA andC are shown in Figure 3.10. The compressed images corresponding toA_2 andC_2 are shown in

Figure 3.10: An image and its Haar transform

Figure 3.11. The compressed versions appear to be indistinguishable from the originals!

Figure 3.11: Compressed image and its Haar transform

If we use the normalized matricesHm andHn, then the equations relating the image matrixA and its normalized Haar transformC are

C =HmAHn A =HmCHn.

The Haar transform can also be used to send large images progressively over the internet. Indeed, we can start sending the Haar coefficients of the matrixC starting from the coarsest coefficients (the first column from top down, then the second column, etc.) and at the receiving end we can start reconstructing the image as soon as we have received enough data.

Observe that instead of performing all rounds of averaging and differencing on each row and each column, we can perform partial encoding (and decoding). For example, we can perform a single round of averaging and differencing for each row and each column. The result is an image consisting of four subimages, where the top left quarter is a coarser version of the original, and the rest (consisting of three pieces) contain the finest detail coefficients. We can also perform two rounds of averaging and differencing, or three rounds, etc. This process is illustrated on the image shown in Figure 3.12. The result of performing one round, two rounds, three rounds, and nine rounds of averaging is shown in Figure 3.13. Since our images have size 512×512, nine rounds of averaging yields the Haar transform, displayed as the image on the bottom right. The original image has completely disappeared! We leave it as a fun exercise to modify the algorithms involving averaging and differencing to perform k rounds of averaging/differencing. The reconstruction algorithm is a little tricky.

A nice and easily accessible account of wavelets and their uses in image processing and computer graphics can be found in Stollnitz, Derose and Salesin [99]. A very detailed account is given in Strang and and Nguyen [102], but this book assumes a fair amount of background in signal processing.

We can find easily a basis of 2n 2n = 22n vectorswij for the linear map that reconstructs×

an image from its Haar coefficients, in the sense that for any matrixC of Haar coefficients, the image matrixA is given by

2n_2n

A = cijwij.

i=1 j=1

Indeed, the matrixwj is given by the so-called outer product

wij =wi(wj) .

Similarly, there is a basis of 2n 2n = 22n vectorshij for the 2D Haar transform, in the sense×

that for any matrixA, its matrixC of Haar coefficients is given by

2n_2n

C = aijhij.

i=1 j=1

Figure 3.12: Original drawing by Durer

IfW−1 = (w−1 ), thenij

hij =wi−1(wj−1) .

We leave it as exercise to compute the bases (wij) and (hij) forn = 2, and to display the corresponding images using the command imagesc.

Figure 3.13: Haar tranforms after one, two, three, and nine rounds of averaging

3.3 The Effect of a Change of Bases on Matrices

The effect of a change of bases on the representation of a linear map is described in the following proposition.

Proposition 3.4. LetE andF be vector spaces, letU = (u_1, . . . , un) andU = (u_1, . . . , un) be two bases ofE, and letV = (v_1, . . . , vm) andV = (v_1, . . . , v

be the change of basis matrix from be the change of basis matrix fromf :E

m) be two bases ofF. Let

P =PU,U toV . For any linear mapU toU , and letQ =PV,V (f) be the matrix associated tofV w.r.t. the basesU andV, and letM→F, letM(f) =MU,V be the matrix associated (f) =MU,V (f)

tof w.r.t. the basesU andV . We have

M (f) =Q−1M(f)P,

or more explicitly

MU,V (f) =P−1 MU,V(f)PU,U =PV,VMU,V(f)PU,U.

V,V

Proof. Sincef :E→F can be written asf = idF æfæ idE, sinceP is the matrix of idE w.r.t. the bases (u_1, . . . , un) and (u_1, . . . , un), andQ−1 is the matrix of idF w.r.t. the bases (v_1, . . . , vm) and (v_1, . . . , vm), by Proposition 3.2, we haveM (f) =Q−1M(f)P.

As a corollary, we get the following result.

Corollary 3.5. LetE be a vector space, and letU = (u_1, . . . , un) andU = (u_1, . . . , un) be two bases ofE. LetP =PU,U be the change of basis matrix fromU toU . For any linear mapf :E→E, letM(f) =MU(f) be the matrix associated tof w.r.t. the basisU, and let

M (f) =MU (f) be the matrix associated tof w.r.t. the basisU . We have

M (f) =P−1M(f)P,

or more explicitly,

MU (f) =P− 1 MU(f)PU,U =PU,UMU(f)PU,U.

U,U

Example 3.4. LetE = R^2,U = (e_1, e_2) wheree_1 = (1,0) ande_2 = (0,1) are the canonical basis vectors, letV = (v_1, v_2) = (e_1, e1−e_2), and let

A =2 1 .0 1

The change of basis matrixP =PV,U fromU toV is

P =1 1 ,0 −1 and we check that P−1 =P. Therefore, in the basisV, the matrix representing the linear mapf defined byA is

A =P−1AP =P AP =1 1 2 1 1 1 =2 0 =D,0 −1 0 1 0 −1 0 1

a diagonal matrix. Therefore, in the basisV, it is clear what the action off is: it is a stretch by a factor of 2 in thev_1 direction and it is the identity in thev_2 direction. Observe thatv_1 andv_2 are not orthogonal.

What happened is that we diagonalized the matrixA. The diagonal entries 2 and 1 are the eigenvalues ofA (andf) andv_1 andv_2 are corresponding eigenvectors. We will come back to eigenvalues and eigenvectors later on.

The above example showed that the same linear map can be represented by different matrices. This suggests making the following definition:

Definition 3.5. Twon×n matricesA andB are said to be similar iff there is some invertible matrixP such that

B =P−1AP.

It is easily checked that similarity is an equivalence relation. From our previous considerations, twon×n matricesA andB are similar iff they represent the same linear map with respect to two different bases. The following surprising fact can be shown: Every square matrixA is similar to its transposeA . The proof requires advanced concepts than we will not discuss in these notes (the Jordan form, or similarity invariants).

IfU = (u_1, . . . , un) andV = (v_1, . . . , vn) are two bases ofE, the change of basis matrix

ëa_11 a12 · · · a1nö ìa_21 a22 · · · a2n÷ P

=

P

V

,

U

=

ì ÷ ì. . ... . ÷

í ø an1 an2 · · · ann

from (u_1, . . . , un) to (v_1, . . . , vn) is the matrix whosejth column consists of the coordinates ofvj over the basis (u_1, . . . , un), which means that

n

vj = aijui.

i=1

product of a matrix times the vector ëv1ö

It is natural to extend the matrix notation and to express the vector

ø

ì .÷ inEn as theívnëu1ö

ì .÷ inEn, namely así ø

un

ëv1ö ëa11 a21 · · · an1ö ëu1ö

ìv_2÷ ìa_12 a22 · · · an2÷ ìu_2÷

ì ÷= ì ÷ ì ÷,ì . ÷ ì . . ... . ÷ ì . ÷

ív ø í ø í ø

n a1n a2n · · · ann un

but notice that the matrix involved is notP, but its transposeP .

This observation has the following consequence: ifU = (u_1, . . . , un) andV = (v_1, . . . , vn) are two bases ofE and ifëv1ö ëu1ö

ì. ÷ =Aì. ÷,ív ø í ø

n un

that is,n

vi = aijuj,

j=1

for any vectorw∈E, ifn n

w = xiui = ykvk,

i=1 k=1

then ëx1ö ëy1ö

ì. ÷ =A ì. ÷,íx ø í ø

n yn

and so ëy1ö ëx1ö

ì .÷ = (A )−1 ì .÷.íy ø í ø

n xn

It is easy to see that ( A )−1 = (A−1) . Also, ifU = (u_1, . . . , un),V = (v_1, . . . , vn), and W = (w_1, . . . , wn) are three bases ofE, and if the change of basis matrix fromU toV is P =PV,U and the change of basis matrix fromV toW isQ =PW,V, then

ëv1ö ëu1öëw1ö ëv1ö

ì. ÷ =P ì. ÷, ì. ÷ =Q ì. ÷,ív ø í ø í ø í ø

n un wn vn

so ëw1ö ëu1ö ëu1ö

ì. ÷ =Q P ì. ÷ = (P Q) ì. ÷,íw ø í ø í ø

n un un

which means that the change of basis matrixPW,U fromU toW isP Q. This proves that PW,U =PV,UPW,V.

3.4 Summary

The main concepts and results of this chapter are listed below:

The representation of linear maps by matrices.•

• The vector space of linear maps HomK(E, F).

The vector space Mm,n(K) ofm×n matrices over the fieldK; The ring Mn(K) of• n×n matrices over the fieldK.

Column vectors, row vectors.•

• Matrix operations: addition, scalar multiplication, multiplication.

•

The matrix representation mapping M : Hom(E, F)→ Mn,p and the representation isomorphism (Proposition 3.2).

• Haar basis vectors and a glimpse at Haar wavelets. Change of basis matrix and Proposition 3.4.•

Chapter 4 Direct Sums, The Dual Space, Duality

4.1 Sums, Direct Sums, Direct Products

Before considering linear forms and hyperplanes, we define the notion of direct sum and prove some simple propositions. There is a subtle point, which is that if we attempt to define the direct sumE F of two vector spaces using the cartesian productE×F, we don’t quite get the right notion because elements ofE×F are ordered pairs, but we want E F =F E. Thus, we want to think of the elements ofE F as unordrered pairs of elements. It is possible to do so by considering the direct sum of a family (Ei)i∈{1,2}, and more generally of a family (Ei)i∈I. For simplicity, we begin by considering the case where I ={1,2}.

Definition 4.1. Given a family (Ei)i∈{1,2} of two vector spaces, we define the (external) direct sumE_1 E_2 (or coproduct) of the family (Ei)i∈{1,2} as the set

E_1 E_2 ={{1, u , 2, v} |u∈E_1, v∈E2},

with addition

{1, u_1, 2, v1} +{1, u_2, 2, v2} ={1, u_1 +u_2, 2, v_1 +v2},

and scalar multiplication

λ{1, u , 2, v} ={1, λu , 2, λv}.

We define the injections in_1:E1→E_1 E_2 andin_2:E2→E_1 E_2 as the linear maps defined such that,

in_1(u) ={1, u , 2,0}, and

in_2(v) ={1,0, 2, v}.

81 Note that E_2 E_1 ={{2, v , 1, u} |v∈E_2, u∈E1} =E_1 E_2. Thus, every member{1, u , 2, v} ofE_1 E_2 can be viewed as an unordered pair consisting of the two vectorsu andv, tagged with the index 1 and 2, respectively.

Remark: In fact,E_1 E_2 is just the producti∈{1,2 E

} i of the family (Ei)i∈{1,2}. This is not to be confused with the cartesian productE1×E_2. The vector spaceE1×E_2 is the set of all ordered pairs u, v , whereu∈ E_1, andv∈ E_2, with addition and multiplication by a scalar defined such that

u_1, v_1 + u_2, v_2 = u_1 +u_2, v_1 +v_2,

λ u, v = λu, λv .

There is a bijection betweeni∈{1,2} i and E1×E_2, but as we just saw, elements of

i∈{1,2

E

Ei are certain sets. The productE1

}can also be defined. We will do this shortly. × · · · ×En of any number of vector spaces

The following property holds.

Proposition 4.1. Given any two vector spaces, E_1 andE_2, the setE_1 E_2 is a vector space. For every pair of linear maps,f :E1→G andg:E2 →G, there is a unique linear map,f +g:E_1 E2→G, such that (f +g)æin_1 =f and (f +g)æin_2 =g, as in the following diagram:

E1 PPPPPPPPf

in_1 PPPP

Ef+g PPPPG_1 E2 nnnn

in2 nnnnnnnnn n

g

nnE_2

Proof. Define

(f +g)({1, u , 2, v}) =f(u) +g(v),

for everyu∈E_1 andv∈E_2. It is immediately verified thatf +g is the unique linear map with the required properties.

We already noted thatE_1 E_2 is in bijection withE1×E_2. If we define the projections π_1:E_1 E2→E_1 andπ_2:E_1 E2→E_2, such that

π_1({1, u , 2, v}) =u,

and

π_2({1, u , 2, v}) =v,

we have the following proposition.

Proposition 4.2. Given any two vector spaces,E_1 andE_2, for every pair of linear maps, f :D→E_1 andg:D→E_2, there is a unique linear map,f×g:D→E_1 E_2, such that π1æ (f×g) =f andπ2æ (f×g) =g, as in the following diagram:

fnnnnnnnnE_1

D

nnnnnnnn π1

Pf×g E_1 E2PPPPPP

g π_2PPPPPPPPP E2

Proof. Define (f×g)(w) ={1, f(w), 2, g(w)},

for everyw∈D. It is immediately verified thatf×g is the unique linear map with the required properties.

Remark: It is a peculiarity of linear algebra that direct sums and products of finite families are isomorphic. However, this is no longer true for products and sums of infinite families.

When U, V are subspaces of a vector spaceE, lettingi_1:U→ E andi_2:V→E be the inclusion maps, ifU V is isomomorphic toE under the mapi1 +i_2 given by Proposition 4.1, we say thatE is a direct sum ofU andV , and we writeE =U V (with a slight abuse of notation, sinceE andU V are only isomorphic). It is also convenient to define the sum U_1 +· · · +Up and the internal direct sumU1⊕ · · · ⊕Up of any number of subspaces ofE.

Definition 4.2. Givenp≥ 2 vector spacesE_1, . . . , Ep, the productF =E1× · · · ×Ep can be made into a vector space by defining addition and scalar multiplication as follows:

(u_1, . . . , up) + (v_1, . . . , vp) = (u_1 +v_1, . . . , up +vp) λ(u_1, . . . , up) = (λu_1, . . . , λup),

for allui, vi ∈Ei and allλ∈K. With the above addition and multiplication, the vector spaceF =E1× · · · ×Ep is called the direct product of the vector spacesE_1, . . . , Ep.

As a special case, whenE_1 =· · · =Ep =K, we find again the vector spaceF =Kp. The projection mapspri:E1× · · · ×Ep→Ei given by

pri(u_1, . . . , up) =ui

are clearly linear. Similarly, the maps ini:Ei→E1× · · · ×Ep given by

ini(ui) = (0, . . . ,0, ui,0, . . . ,0) are injective and linear. If dim(Ei) =ni and if (ei, . . . , eini) is a basis ofEi fori = 1, . . . , p,1

then it is easy to see that then_1 +· · · +np vectors

(e^1,0, . . . ,0), . . . , (e^1n1,0, . . . ,0),1

. . .

(0, . . . ,0, ei,0, . . . ,0), . . . , (0, . . . ,0, eini,0, . . . ,0),1

. .

(0, . . . ,0, e

. p), . . . , (0, . . . ,0, epnp)1

form a basis ofE1× · · · ×Ep, and so

dim(E1× · · · ×Ep) = dim(E_1) +· · · + dim(Ep).

Let us now consider a vector spaceE andp subspacesU_1, . . . , Up ofE. We have a map

a:U1× · · · ×Up→E given by a(u_1, . . . , up) =u_1 +· · · +up,

withui∈Ui fori = 1, . . . , p. It is clear that this map is linear, and so its image is a subspace ofE denoted by

U_1 +· · · +Up

and called the sum of the subspacesU_1, . . . , Up. It is immediately verified thatU_1 +· · ·+Up is the smallest subspace ofE containingU_1, . . . , Up.

If the mapa is injective, then Kera = 0, which means that ifui∈Ui fori = 1, . . . , p and if

u_1 +· · · +up = 0

thenu_1 =· · · =up = 0. In this case, everyu∈U_1 +· · · +Up has a unique expression as a sum

u =u_1 +· · · +up,

withui∈Ui, fori = 1, . . . , p. It is also clear that for anyp nonzero vectorsui∈Ui,u_1, . . . , up are linearly independent.

Definition 4.3. For any vector spaceE and anyp≥ 2 subspacesU_1, . . . , Up ofE, if the mapa defined above is injective, then the sumU_1 +· · · +Up is called a direct sum and it is denoted by

U1⊕ · · · ⊕Up. The spaceE is the direct sum of the subspacesUi if

E =U1⊕ · · · ⊕Up. Observe that when the mapa is injective, then it is a linear isomorphism between U1× · · · × Up andU1⊕ · · · ⊕Up. The difference is thatU1× · · · ×Up is defined even if the spacesUi are not assumed to be subspaces of some common space.

Now, ifp = 2, it is easy to determine the kernel of the mapa:U1×U2→E. We have a(u_1, u_2) =u_1 +u_2 = 0 iff u_1 =−u_2, u1∈U_1, u2∈U_2, which implies that Kera ={(u,−u)|u∈U1∩U2}.

Now,U1∩U_2 is a subspace ofE and the linear mapu→ (u,−u) is clearly an isomorphism, so Kera is isomorphic toU1∩U_2. As a result, we get the following result: Proposition 4.3. Given any vector spaceE and any two subspacesU_1 andU_2, the sum U_1 +U_2 is a direct sum iffU1∩U_2 = (0).

An interesting illustration of the notion of direct sum is the decomposition of a square matrix into its symmetric part and its skew-symmetric part. Recall that ann×n matrix A∈ Mn is symmetric ifA =A, skew -symmetric ifA =−A. It is clear that

S(n) ={A∈ Mn|A =A} and Skew(n) ={A∈ Mn|A =−A}

are subspaces of Mn, and that S(n)∩Skew(n) = (0). Observe that for any matrixA∈ Mn, the matrixH(A) = (A +A )/2 is symmetric and the matrixS(A) = (A−A )/2 is skewsymmetric. Since

A

=

H

(

A

) +

S

(

A

) =

A

+

A

+

A

A

− ,2 2

we see that Mn = S(n) +Skew(n), and since S(n)∩Skew(n) = (0), we have the direct sum Mn = S(n)⊕Skew(n).

Remark: The vector space Skew(n) of skew-symmetric matrices is also denoted by so(n). It is the Lie algebra of the group SO(n).

Proposition 4.3 can be generalized to anyp≥ 2 subspaces at the expense of notation. The proof of the following proposition is left as an exercise.

Proposition 4.4. Given any vector spaceE and anyp≥ 2 subspacesU_1, . . . , Up, the following properties are equivalent:

(1) The sumU_1 +· · · +Up is a direct sum.

(2) We have

p

U

i

∩

j=1,j=i

Uj = (0), i = 1, . . . , p. (3) We have

i−1

U

i

∩

j=1

Uj = (0), i = 2, . . . , p.

Because of the isomorphism

U1× · · · ×Up≈U1⊕ · · · ⊕Up,

we have

Proposition 4.5. IfE is any vector space, for any (finite-dimensional) subspacesU_1, . . ., Up ofE, we have

dim(U1⊕ · · · ⊕Up) = dim(U_1) +· · · + dim(Up).

IfE is a direct sum E =U1⊕ · · · ⊕Up,

since every u∈E can be written in a unique way as u =u_1 +· · · +up

for someui∈Ui fori = 1. . . , p, we can define the mapsπi:E→Ui, called projections, by πi(u) =πi(u_1 +· · · +up) =ui.

It is easy to check that these maps are linear and satisfy the following properties:

π

j

æ

π

i

=

πi ifi =j 0 ifi =j, π_1 +· · · +πp = idE. For example, in the case of the direct sum

Mn = S(n)⊕Skew(n), the projection onto S(n) is given by

π

1

(

A

) =

H

(

A

) =

A +A 2 ,

and the projection onto Skew(n) is given by

π

2

(

A

) =

S

(

A

) =

A

A

− .

2

Clearly,H(A)+S(A) =A,H(H(A)) =H(A),S(S(A)) =S(A), andH(S(A)) =S(H(A)) = 0.

A functionf such thatfæf =f is said to be idempotent. Thus, the projectionsπi are idempotent. Conversely, the following proposition can be shown:

Proposition 4.6. LetE be a vector space. For anyp≥ 2 linear mapsfi:E→E, if

f

j

æ

f

i

=

fi ifi =j 0 ifi =j, f_1 +· · · +fp = idE,

then if we letUi =fi(E), we have a direct sum

E =U1⊕ · · · ⊕Up.

We also have the following proposition characterizing idempotent linear maps whose proof is also left as an exercise.

Proposition 4.7. For every vector spaceE, iff :E→E is an idempotent linear map, i.e., fæf =f, then we have a direct sum

E = Kerf⊕ Imf,

so thatf is the projection onto its image Imf.

We now give the definition of a direct sum for any arbitrary nonempty index set I. First, let us recall the notion of the product of a family (Ei)i∈I. Given a family of sets (Ei)i∈I, its producti∈IEi, is the set of all functionsf :I→ i∈IEi, such that,f(i)∈Ei, for every i∈I . It is one of the many versions of the axiom of choice, that, if Ei =∅ for everyi∈I, theni ∈IEi =∅. A memberf∈ i∈IEi, is often denoted as (fi)i∈I. For everyi∈I, we have the projection πi:i∈IEi→Ei , defined such that,πi((fi)i∈I) =fi. We now define direct sums.

Definition 4.4. LetI be any nonempty set, and let (Ei)i∈I be a family of vector spaces. The (external) direct sumi∈IEi (or coproduct) of the family (Ei)i∈I is defined as follows: Ei consists of allf∈ i∈IEi, which have finite support, and addition and multii∈I

plication by a scalar are defined as follows:

(fi)i∈I + (gi)i∈I = (fi +gi)i∈I, λ(fi)i∈I = (λfi)i∈I.

We also have injection mapsini:Ei→ i∈IEi, defined such that,ini(x) = (fi)i∈I, where fi =x, andfj = 0, for allj∈ (I− {i}).

The following proposition is an obvious generalization of Proposition 4.1. Proposition 4.8. LetI be any nonempty set, let (Ei)i∈I be a family of vector spaces, and letG be any vector space. The direct sum

(hi)i∈I of linear mapshi:Ei→G

i∈IEi is a vector space, and for every family

, there is a unique linear map

hi :

i∈I

Ei→G,

i∈I

such that, ( i∈Ihi)æini =hi, for everyi∈I.

Remark: WhenEi =E, for alli∈I, we denotei∈IEi byE(I). In particular, when Ei =K, for alli∈I, we find the vector spaceK(I) of Definition 2.13.

We also have the following basic proposition about injective or surjective linear maps.

Proposition 4.9. LetE andF be vector spaces, and letf :E→F be a linear map. If f :E→F is injective, then there is a surjective linear mapr:F→E called a retraction, such that ræf = idE. Iff :E→F is surjective, then there is an injective linear map

s:F→E called a section, such that fæs = idF.

Proof. Let (ui)i∈I be a basis ofE. Sincef :E→F is an injective linear map, by Proposition 2.13, (f(ui))i∈I is linearly independent inF. By Theorem 2.7, there is a basis (vj)j∈J ofF, whereI⊆J, and wherevi =f(ui), for alli∈I. By Proposition 2.13, a linear mapr:F→E can be defined such thatr(vi) =ui, for alli∈I, andr(vj) =w for allj∈ (J−I), wherew is any given vector inE, sayw = 0. Sincer(f(ui)) =ui for alli∈I, by Proposition 2.13, we haveræf = idE.

Now, assume that f :E→ F is surjective. Let (vj)j∈J be a basis ofF. Sincef :E→F is surjective, for everyvj∈F, there is someuj∈E such thatf(uj) =vj. Since (vj)j∈J is a basis ofF, by Proposition 2.13, there is a unique linear maps:F→E such thats(vj) =uj. Also, sincef(s(vj)) =vj, by Proposition 2.13 (again), we must havefæs = idF.

The converse of Proposition 4.9 is obvious. We now have the following fundamental Proposition.

Proposition 4.10. LetE,F andG, be three vector spaces,f :E→F an injective linear map,g:F→G a surjective linear map, and assume that Imf = Kerg. Then, the following properties hold. (a) For any sections:G→F ofg, we haveF = Kerg⊕ Ims, and the linear mapf +s:E⊕G→F is an isomorphism.1

(b) For any retractionr:F→E off, we haveF = Imf⊕ Kerr.2

f g

Er Fs G 1The existence of a section s: G→F of g follows from Proposition 4.9. 2The existence of a retraction r : F→E of f follows from Proposition 4.9.

Proof. (a) Sinces:G→F is a section ofg, we havegæs = idG, and for everyu∈F,

g(u−s(g(u))) =g(u)−g(s(g(u))) =g(u)−g(u) = 0.

Thus,u−s(g(u))∈ Kerg, and we haveF = Kerg + Ims. On the other hand, ifu∈Kerg∩ Ims, thenu =s(v) for somev∈G becauseu∈ Ims,g(u) = 0 becauseu∈ Kerg,

and so,

g(u) =g(s(v)) =v = 0,

because gæs = idG, which shows thatu =s(v) = 0. Thus,F = Kerg⊕ Ims, and since by assumption, Imf = Kerg, we haveF = Imf⊕ Ims. But then, sincef ands are injective, f +s:E⊕G→F is an isomorphism. The proof of (b) is very similar.

Note that we can choose a retractionr:F→E so that Kerr = Ims, since F = Kerg⊕ Ims = Imf⊕ Ims andf is injective so we can setr≡ 0 on Ims. Given a sequence of linear mapsEf Fg G, when Imf = Kerg, we say that the sequence

E

f

F

g

G

is

exact at

−→ −→

F. If in addition to being exact atF,f is injective−→ −→

andg is surjective, we say that we have a short exact sequence, and this is denoted as

0 −→E f F g G−→ 0.−→ −→

The property of a short exact sequence given by Proposition 4.10 is often described by saying that 0−→E f Fg G−→ 0 is a (short) split exact sequence.−→ −→

As a corollary of Proposition 4.10, we have the following result.

Theorem 4.11. LetE andF be vector spaces, and letf :E→F be a linear map. Then, E is isomorphic to Kerf⊕ Imf, and thus,

dim(E) = dim(Kerf) + dim(Imf) = dim(Kerf) + rk(f).

Proof. Consider Kerfi E f Imf,−→ −→

where Kerfi E is the inclusion map, andEf Imf is the surjection associated

with Ef −→ −→ s E off to−→ F. Then, we apply Proposition 4.10 to any section Imf−→ get an isomorphism betweenE and Kerf⊕ Imf, and Proposition 4.5, to get dim(E) = dim(Kerf) + dim(Imf).

Remark: The dimension dim(Kerf) of the kernel of a linear mapf is often called the nullity off.

We now derive some important results using Theorem 4.11.

Proposition 4.12. Given a vector spaceE, ifU andV are any two subspaces ofE, then

dim(U) + dim(V ) = dim(U +V ) + dim(U∩V ),

an equation known as Grassmann’s relation.

Proof. Recall thatU +V is the image of the linear map

a:U×V→E

given by a(u, v) =u +v,

and that we proved earlier that the kernel Kera ofa is isomorphic toU∩V . By Theorem 4.11,

dim(U×V ) = dim(Kera) + dim(Ima),

but dim(U×V ) = dim(U) + dim(V ), dim(Kera) = dim(U∩V ), and Ima =U +V , so the Grassmann relation holds.

The Grassmann relation can be very useful to figure out whether two subspace have a nontrivial intersection in spaces of dimension> 3. For example, it is easy to see that in R^5, there are subspacesU andV with dim(U) = 3 and dim(V ) = 2 such thatU∩V = 0; for example, letU be generated by the vectors (1,0,0,0,0),(0,1,0,0,0), (0,0,1,0,0), andV be generated by the vectors (0,0,0,1,0) and (0,0,0,0,1). However, we claim that if dim(U) = 3 and dim(V ) = 3, then dim(U∩V )≥ 1. Indeed, by the Grassmann relation, we have

dim(U) + dim(V ) = dim(U +V ) + dim(U∩V ),

namely 3 + 3 = 6 = dim(U +V ) + dim(U∩V ), and sinceU +V is a subspace of R^5, dim(U +V )≤ 5, which implies

6≤ 5 + dim(U∩V ),

that is 1≤ dim(U∩V ).

As another consequence of Proposition 4.12, if U andV are two hyperplanes in a vector space of dimensionn, so that dim(U) =n− 1 and dim(V ) =n− 1, the reader should show that

dim(U∩V )≥n− 2, and so, ifU =V , then

dim(U∩V ) =n− 2.

Here is a characterization of direct sums that follows directly from Theorem 4.11. Proposition 4.13. IfU_1, . . . , Up are any subspaces of a finite dimensional vector spaceE, then

dim(U_1 +· · · +Up)≤ dim(U_1) +· · · + dim(Up), and

dim(U_1 +· · · +Up) = dim(U_1) +· · · + dim(Up) iff theUis form a direct sumU1⊕ · · · ⊕Up.

Proof. If we apply Theorem 4.11 to the linear map

a:U1× · · · ×Up→U_1 +· · · +Up

given bya(u_1, . . . , up) =u_1 +· · · +up, we get

dim(U_1 +· · · +Up) = dim(U1× · · · × Up)− dim(Kera) = dim(U_1) +· · · + dim(Up)− dim(Kera), so the inequality follows. Sincea is injective iff Kera = (0), theUis form a direct sum iff the second equation holds.

Another important corollary of Theorem 4.11 is the following result:

Proposition 4.14. Let E and F be two vector spaces with the same finite dimension dim(E) = dim(F) = n. For every linear mapf :E→ F, the following properties are equivalent:

(a) f is bijective.

(b) f is surjective. (c) f is injective. (d) Kerf = 0.

Proof. Obviously, (a) implies (b).

Iff is surjective, then Imf =F, and so dim(Imf) =n. By Theorem 4.11, dim(E) = dim(Kerf) + dim(Imf), and since dim(E) = n and dim(Imf) = n, we get dim(Kerf) = 0, which means that Kerf = 0, and sof is injective (see Proposition 2.12). This proves that (b) implies (c). Iff is injective, then by Proposition 2.12, Kerf = 0, so (c) implies (d). Finally, assume that Kerf = 0, so that dim(Kerf) = 0 andf is injective (by Proposition 2.12). By Theorem 4.11,

dim(E) = dim(Kerf) + dim(Imf), and since dim(Kerf) = 0, we get

dim(Imf) = dim(E) = dim(F),

which proves thatf is also surjective, and thus bijective. This proves that (d) implies (a) and concludes the proof.

One should be warned that Proposition 4.14 fails in infinite dimension. The following Proposition will also be useful.

Proposition 4.15. LetE be a vector space. IfE =U⊕V andE =U⊕W, then there is an isomorphismf :V→W betweenV andW.

Proof. LetR be the relation betweenV andW, defined such that

v, w∈R iff w−v∈U.

We claim that R is a functional relation that defines a linear isomorphismf :V→ W betweenV andW, wheref(v) =w iff v, w∈R (R is the graph off). Ifw−v∈U and w−v∈U, thenw−w∈U, and sinceU⊕W is a direct sum,U∩ W = 0, and thus w−w = 0, that isw =w. Thus,R is functional. Similarly, ifw−v∈U andw−v∈U, thenv− v∈U, and sinceU⊕V is a direct sum,U∩V = 0, andv =v. Thus,f is injective. SinceE =U⊕V , for everyw∈W, there exists a unique pair u, v ∈U× V , such that w =u +v. Then,w−v∈U, andf is surjective. We also need to verify thatf is linear. If

w−v =u

and w−v =u , whereu, u∈U, then, we have

(w +w )− (v +v ) = (u +u ),

whereu +u∈U. Similarly, if w−v =u whereu∈U, then we have

λw−λv =λu, whereλu∈U. Thus,f is linear.

Given a vector space E and any subspaceU ofE, Proposition 4.15 shows that the dimension of any subspaceV such thatE =U⊕V depends only onU. We call dim(V ) the codimension ofU, and we denote it by codim(U). A subspaceU of codimension 1 is called a hyperplane.

The notion of rank of a linear map or of a matrix is an important one, both theoretically and practically, since it is the key to the solvability of linear equations. Recall from Definition 2.15 that the rank rk(f) of a linear mapf :E→F is the dimension dim(Imf) of the image subspace Imf ofF.

We have the following simple proposition.

Proposition 4.16. Given a linear mapf :E→F, the following properties hold:

(i) rk(f) = codim(Kerf).

(ii) rk(f) + dim(Kerf) = dim(E).

(iii) rk(f)≤ min(dim(E),dim(F)).

Proof. Since by Proposition 4.11, dim(E) = dim(Kerf) + dim(Imf), and by definition, rk(f) = dim(Imf), we have rk(f) = codim(Kerf). Since rk(f) = dim(Imf), (ii) follows from dim(E) = dim(Kerf) + dim(Imf). As for (iii), since Imf is a subspace ofF, we have rk(f)≤ dim(F), and since rk(f) + dim(Kerf) = dim(E), we have rk(f)≤ dim(E).

The rank of a matrix is defined as follows.

Definition 4.5. Given am×n-matrixA = (ai j) over the fieldK, the rank rk(A) of the matrixA is the maximum number of linearly independent columns ofA (viewed as vectors inKm).

In view of Proposition 2.8, the rank of a matrix A is the dimension of the subspace of Km generated by the columns ofA. LetE andF be two vector spaces, and let (u_1, . . . , un) be a basis ofE, and (v_1, . . . , vm) a basis ofF. Letf :E→ F be a linear map, and letM(f) be its matrix w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm). Since the rank rk(f) off is the dimension of Imf, which is generated by (f(u_1), . . . , f(un)), the rank off is the maximum number of linearly independent vectors in (f(u_1), . . . , f(un)), which is equal to the number of linearly independent columns ofM(f), sinceF andKm are isomorphic. Thus, we have rk(f) = rk(M(f)), for every matrix representingf.

We will see later, using duality, that the rank of a matrixA is also equal to the maximal number of linearly independent rows ofA.

If U is a hyperplane, thenE =U⊕V for some subspaceV of dimension 1. However, a subspaceV of dimension 1 is generated by any nonzero vectorv∈V , and thus we denote V byKv, and we writeE =U⊕Kv. Clearly,v /∈U. Conversely, letx∈E be a vector such thatx /∈U (and thus,x = 0). We claim thatE =U⊕Kx. Indeed, sinceU is a hyperplane, we haveE =U⊕Kv for somev /∈U (withv = 0). Then,x∈E can be written in a unique way asx =u +λv, whereu∈U, and sincex /∈U, we must haveλ = 0, and thus,v =−λ−1u +λ−1x. SinceE =U⊕Kv, this shows thatE =U +Kx. Sincex /∈U, we haveU∩Kx = 0, and thusE =U⊕Kx. This argument shows that a hyperplane is a maximal proper subspaceH ofE.

In the next section, we shall see that hyperplanes are precisely the Kernels of nonnull linear mapsf :E→K, called linear forms.

4.2 The Dual SpaceE∗ and Linear Forms

We already observed that the field K itself is a vector space (over itself). The vector space Hom(E, K) of linear maps fromE to the fieldK, the linear forms, plays a particular role. We take a quick look at the connection betweenE and Hom(E, K), its dual space. As we will see shortly, every linear mapf :E→F gives rise to a linear mapf :F∗→E∗, and it turns out that in a suitable basis, the matrix off is the transpose of the matrix off. Thus, the notion of dual space provides a conceptual explanation of the phenomena associated with transposition. But it does more, because it allows us to view subspaces as solutions of sets of linear equations and vice-versa.

Consider the following set of two “linear equations” in R^3,

x−y +z = 0

x−y−z = 0,

and let us find out what is their set V of common solutions (x, y, z)∈ R^3. By subtracting the second equation from the first, we get 2z = 0, and by adding the two equations, we find that 2(x−y) = 0, so the setV of solutions is given by

y =x

z = 0.

This is a one dimensional subspace of R^3. Geometrically, this is the line of equationy =x in the planez = 0.

Now, why did we say that the above equations are linear? This is because, as functions of (x, y, z), both mapsf_1: (x, y, z)→x−y +z andf_2: (x, y, z)→x−y−z are linear. The set of all such linear functions from R^3 to R is a vector space; we used this fact to form linear combinations of the “equations”f_1 andf_2. Observe that the dimension of the subspaceV is 1. The ambient space has dimensionn = 3 and there are two “independent” equations f_1, f_2, so it appears that the dimension dim(V ) of the subspaceV defined bym independent equations is

dim(V ) =n−m,

which is indeed a general fact.

More generally, in Rn, a linear equation is determined by ann-tuple (a_1, . . . , an)∈ Rn, and the solutions of this linear equation are given by then-tuples (x_1, . . . , xn)∈ R such that

a_1x_1 +· · · +anxn = 0; these solutions constitute the kernel of the linear map (x_1, . . . , xn)→a_1x_1 +· · · +anxn. The above considerations assume that we are working in the canonical basis (e_1, . . . , en) of Rn, but we can define “linear equations” independently of bases and in any dimension, by viewing them as elements of the vector space Hom(E, K) of linear maps fromE to the field K.

Definition 4.6. Given a vector spaceE, the vector space Hom(E, K) of linear maps fromE to the fieldK is called the dual space (or dual) ofE. The space Hom(E, K) is also denoted byE∗, and the linear maps inE∗ are called the linear forms, or covectors. The dual space E∗∗ of the spaceE∗ is called the bidual ofE.

As a matter of notation, linear formsf :E→K will also be denoted by starred symbol, such asu∗,x∗, etc.

IfE is a vector space of finite dimensionn and (u_1, . . . , un) is a basis ofE, for any linear formf∗∈E∗, for everyx =x_1u_1 +· · · +xnun∈E, we have

f∗(x) =λ_1x_1 +· · · +λnxn,

where λi =f∗(ui)∈K, for everyi, 1≤i≤n. Thus, with respect to the basis (u_1, . . . , un), f∗(x) is a linear combination of the coordinates ofx, and we can view a linear form as a linear equation, as discussed earlier.

Given a linear form u∗∈E∗ and a vectorv∈E, the resultu∗(v) of applyingu∗ tov is also denoted by u∗, v . This defines a binary operation−,−:E∗×E→K satisfying the following properties:

u∗ +u∗, v = u∗, v + u∗, v1 2 1 2 u∗, v_1 +v_2 = u∗, v_1 + u∗, v_2 λu∗, v =λ u∗, v

u∗, λv =λ u∗, v .

The above identities mean that−,− is a bilinear map, since it is linear in each argument. It is often called the canonical pairing betweenE∗ andE. In view of the above identities, given any fixed vectorv∈E, the map evalv:E∗→K (evaluation atv) defined such that

evalv(u∗) = u∗, v =u∗(v) for everyu∗∈E∗ is a linear map fromE∗ toK, that is, evalv is a linear form inE∗∗. Again, from the above identities, the map evalE:E→E∗∗, defined such that

evalE(v) = evalv for everyv∈E, is a linear map. Observe that evalE(v)(u∗) = u∗, v =u∗(v), for allv∈E and allu∗∈E∗. We shall see that the map evalE is injective, and that it is an isomorphism whenE has finite dimension.

We now formalize the notion of the set V 0 of linear equations vanishing on all vectors in a given subspaceV⊆E, and the notion of the setU^0 of common solutions of a given set U⊆ E∗ of linear equations. The duality theorem (Theorem 4.17) shows that the dimensions ofV andV 0, and the dimensions ofU andU^0, are related in a crucial way. It also shows that, in finite dimension, the mapsV→V 0 andU→U^0 are inverse bijections from subspaces of E to subspaces ofE∗.

Definition 4.7. Given a vector spaceE and its dualE∗, we say that a vectorv∈E and a linear formu∗∈E∗ are orthogonal if u∗, v = 0. Given a subspaceV ofE and a subspaceU ofE∗, we say thatV andU are orthogonal if u∗, v = 0 for everyu∗ ∈U and everyv∈V . Given a subsetV ofE (resp. a subsetU ofE∗), the orthogonalV 0 ofV is the subspaceV 0 ofE∗ defined such that

V 0 ={u∗∈E∗| u∗, v = 0, for everyv∈V} (resp. the orthogonalU^0 ofU is the subspaceU^0 ofE defined such that

U^0 ={v∈E| u∗, v = 0, for everyu∗∈U}).

The subspace V 0 E∗ is also called the annihilator ofV . The subspaceU^0 E⊆ ⊆ annihilated byU⊆E∗ does not have a special name. It seems reasonable to call it the linear subspace (or linear variety) defined byU.

Informally,V 0 is the set of linear equations that vanish onV , andU^0 is the set of common zeros of all linear equations inU.

We can also defineV 0 by

V 0 ={u∗∈E∗|V⊆ Keru∗} andU^0 by

U^0 = Keru∗.

u∗ U

Observe that

E

0 = 0, and{00 ∈

} =E∗. Furthermore, ifV1⊆V2⊆E, thenV 0 V 0 E∗, and if

U

1

⊆

U

2

⊆

E

∗

, then

U

0

U

0

2

E

.

⊆1⊆ 2⊆1⊆

Indeed, ifV

f1 0 0 we havef∗(v) = 0 for allv∈V2, and thus1 ⊆V2⊆E, then for anyf∗∈V2 0, we∗(v) = 0 for allv∈V1, sof∗∈V 0 . Similarly, ifU1⊆ U2⊆E∗, then for anyv∈U_2 havef∗(v) = 0 for allf∗∈U_2, sof∗(v) = 0 for allf∗∈U_1, which means thatv∈U1. Here are some examples. LetE = M_2(R), the space of real 2× 2 matrices, and letV be the subspace of M_2(R) spanned by the matrices

0 1 ,1 0 ,0 0 .1 0 0 0 0 1

We check immediately that the subspaceV consists of all matrices of the form

b a ,a c

that is, all symmetric matrices. The matrices

a_11 a_12 a_21 a_22

inV satisfy the equation a12−a_21 = 0,

and all scalar multiples of these equations, soV 0 is the subspace ofE∗ spanned by the linear form given byu∗(a_11, a_12, a_21, a_22) =a12−a_21. We have

dim(V 0) = dim(E)− dim(V ) = 4− 3 = 1.

The above example generalizes to E = Mn(R) for anyn≥ 1, but this time, consider the spaceU of linear forms asserting that a matrixA is symmetric; these are the linear forms spanned by then(n− 1)/2 equations

aij−aji = 0, 1≤i < j≤n;

Note there are no constraints on diagonal entries, and half of the equations

aij−aji = 0, 1≤i =j≤n

are redudant. It is easy to check that the equations (linear forms) for which i < j are linearly independent. To be more precise, letU be the space of linear forms inE∗ spanned by the linear forms

u∗ij(a_11, . . . , a1n, a_21, . . . , a2n, . . . , an1, . . . , ann) =aij−aji, 1≤i < j≤n.

Then, the setU^0 of common solutions of these equations is the space S(n) of symmetric matrices. This space has dimension

n(n + 1)=n2 n(n− 1).2 − 2

We leave it as an exercise to find a basis of S(n).

IfE = Mn(R), consider the subspaceU of linear forms inE∗ spanned by the linear forms

u ∗ij(a_11, . . . , a1n, a_21, . . . , a2n, . . . , an1, . . . , ann) =aij +aji, 1≤i < j≤n u∗ii(a_11, . . . , a1n, a_21, . . . , a2n, . . . , an1, . . . , ann) =aii, 1≤i≤n. It is easy to see that these linear forms are linearly independent, so dim(U) =n(n + 1)/2.

The spaceU^0 of matricesA∈ Mn(R) satifying all of the above equations is clearly the space Skew(n) of skew-symmetric matrices. The dimension ofU^0 is

n(n− 1)=n2 n(n + 1).

2 − 2

We leave it as an exercise to find a basis of Skew(n).

For yet another example, withE = Mn(R), for anyA∈ Mn(R), consider the linear form inE∗ given by

tr(A) =a_11 +a_22 +· · · +ann,

called the trace ofA. The subspaceU^0 ofE consisting of all matricesA such that tr(A) = 0 is a space of dimensionn^2 1. We leave it as an exercise to find a basis of this space.−

The dimension equations

dim(V ) + dim(V 0) = dim(E) dim(U) + dim(U^0) = dim(E)

are always true (ifE is finite-dimensional). This is part of the duality theorem (Theorem 4.17).

In constrast with the previous examples, given a matrixA∈ Mn(R), the equations asserting thatA A =I are not linear constraints. For example, forn = 2, we have

a^2 +a^2 = 111 21

a^2 +a^2 = 121 22

a_11a_12 +a_21a_22 = 0.

Remarks:

(1) The notation V 0 (resp.U^0) for the orthogonal of a subspaceV ofE (resp. a subspace U ofE∗) is not universal. Other authors use the notationV⊥ (resp. U⊥). However, the notationV⊥ is also used to denote the orthogonal complement of a subspaceV with respect to an inner product on a spaceE, in which caseV⊥ is a subspace ofE and not a subspace ofE∗ (see Chapter 9). To avoid confusion, we prefer using the notationV 0.

(2) Since linear forms can be viewed as linear equations (at least in finite dimension), given a subspace (or even a subset)U ofE∗, we can define the setZ(U) of common zeros of the equations inU by

Z (U) ={v∈E|u∗(v) = 0, for allu∗∈U}. Of courseZ(U) =U^0, but the notionZ(U) can be generalized to more general kinds of equations, namely polynomial equations. In this more general setting,U is a set of polynomials inn variables with coefficients inK (wheren = dim(E)). Sets of the form Z(U) are called algebraic varieties. Linear forms correspond to the special case where homogeneous polynomials of degree 1 are considered.

If V is a subset ofE, it is natural to associate with V the set of polynomials in K[X_1, . . . , Xn] that vanish onV . This set, usually denotedI(V ), has some special properties that make it an ideal. IfV is a linear subspace ofE, it is natural to restrict our attention to the spaceV 0 of linear forms that vanish onV , and in this case we identifyI(V ) andV 0 (although technically,I(V ) is no longer an ideal).

For any arbitrary set of polynomials U⊆K[X_1, . . . , Xn] (respV⊆E) the relationship betweenI(Z(U) andU (resp.Z(I(V )) andV ) is generally not simple, even though we always have

U⊆ I(Z(U) (resp. V⊆ Z(I(V ))).

However, when the field K is algebraically closed, thenI(Z(U) is equal to the radical of the idealU, a famous result due to Hilbert known as the Nullstellensatz (see Lang [65] or Dummit and Foote [30]). The study of algebraic varieties is the main subject of algebraic geometry, a beautiful but formidable subject. For a taste of algebraic geometry, see Lang [65] or Dummit and Foote [30].

The duality theorem (Theorem 4.17) shows that the situation is much simpler if we restrict our attention to linear subspaces; in this case

U =I(Z(U) and V =Z(I(V )).

We claim thatV⊆V 00 for every subspaceV ofE, and thatU⊆U^00 for every subspace U ofE∗.

Indeed, for any v∈ V , to show thatv∈V 00 we need to prove thatu∗(v) = 0 for all u∗∈V 0. However,V 0 consists of all linear formsu∗ such thatu∗(y) = 0 for ally∈V ; in particular, sincev∈V ,u∗(v) = 0 for allu∗∈V 0, as required.

Similarly, for any u∗ ∈ U, to show thatu∗∈ U^00 we need to prove thatu∗(v) = 0 for allv∈U^0. However,U consists of all vectorsv such thatf∗(v) = 0 for all f∗∈U; in particular, sinceu∗∈U,u∗(v) = 0 for allv∈U^0, as required.

We will see shortly that in finite dimension, we haveV =V 00 andU =U^00.

However, even thoughV =V 00 is always true, whenE is of infinite dimension, it is not always true thatU =U^00.

Given a vector spaceE and any basis (ui)i∈I forE, we can associate to eachui a linear formu∗∈E∗, and theu∗ have some remarkable properties.

Definition 4.8. Given a vector spaceE and any basis (ui)i∈I forE, by Proposition 2.13, for everyi∈I, there is a unique linear formu∗ such that

1 ifi =ju∗(uj) = 0 ifi =j,

for everyj∈I. The linear formu∗ is called the coordinate form of indexi w.r.t. the basis (ui)i∈I.

Given an index setI, authors often define the so called “Kronecker symbol”δi j, such

that 1 ifi =jδi j = 0 ifi =j,

for alli, j∈I. Then,u∗(uj) =δi j.

The reason for the terminology coordinate form is as follows: IfE has finite dimension and if (u_1, . . . , un) is a basis ofE, for any vector

v =λ_1u_1 +· · · +λnun,

we have

u∗(v) =u∗(λ_1u_1 +· · · +λnun)

=λ_1u∗(u_1) +· · · +λiu∗(ui) +· · · +λnu∗(un)

=λi,

sinceu∗(uj) =δi j. Therefore,u∗ is the linear function that returns theith coordinate of a vector expressed over the basis (u_1, . . . , un).

Given a vector space E and a subspaceU ofE, by Theorem 2.7, every basis (ui)i∈I ofU can be extended to a basis (uj)j∈I∪J ofE, whereI∩J =∅. We have the following important theorem adapted from E. Artin [2] (Chapter 1).

Theorem 4.17. (Duality theorem) LetE be a vector space. The following properties hold: (a) For every basis (ui)i∈I ofE, the family (u∗)i∈I of coordinate forms is linearly independent.

(b) For every subspaceV ofE, we haveV 00 =V .

(c) For every subspaceV of finite codimensionm ofE, for every subspaceW ofE such thatE =V⊕W (whereW is of finite dimensionm), for every basis (ui)i∈I ofE such

that (u1, . . . , um) is a basis ofW, the family (u∗, . . . , u∗m) is a basis of the orthogonal

V 0 ofV inE∗, so that^1

dim(V 0) = codim(V ).

Furthermore, we haveV 00 =V .

(d) For every subspaceU of finite dimensionm ofE∗, the orthogonalU^0 ofU inE is of finite codimensionm, so that

codim(U^0) = dim(U). Furthermore,U^00 =U.

Proof. (a) Assume that

λiu∗ = 0,

i∈I

for a family (λi)i∈I (of scalars inK). Since (λi)i∈I has finite support, there is a finite subset J ofI such thatλi = 0 for alli∈I−J, and we have

λju∗ = 0.

j∈J

Applying the linear formj∈Jλju∗ to eachuj (j∈J), by Definition 4.8, sinceu∗(uj) = 1 if i =j and 0 otherwise, we get λj = 0 for allj∈J, that isλi = 0 for alli∈I (by definition ofJ as the support). Thus, (u∗)i∈I is linearly independent.

(b) Clearly, we have V⊆V 00. IfV =V 00, then let (ui)i∈I∪J be a basis ofV 00 such that (ui)i∈I is a basis ofV (whereI∩J =∅). SinceV =V 00,uj0∈V 00 for somej0∈J (and thus,j_0 /∈I). Sinceuj0∈V 00,uj0 is orthogonal to every linear form inV 0. Now, we have u∗0(ui) = 0 for alli∈I, and thusu∗0∈V 0. However,u∗0(uj0) = 1, contradicting the fact thatuj0 is orthogonal to every linear form inV 0. Thus,V =V 00.

(c) LetJ =I− {1, . . . , m}. Every linear formf∗∈V 0 is orthogonal to everyuj, for j∈J, and thus,f∗(uj) = 0, for allj∈J. For such a linear formf∗∈V 0, let g∗ =f∗(u_1)u∗ +· · · +f∗(um)u∗m.1

We have g∗(ui) =f∗(ui), for everyi, 1≤ i≤m. Furthermore, by definition,g∗ vanishes on alluj, wherej∈J. Thus,f∗ andg∗ agree on the basis (ui)i∈I ofE, and so,g∗ =f∗. This shows that (u_1 , . . . , u∗m) generatesV 0, and since it is also a linearly independent family, (u∗, . . . , u∗m) is a basis ofV 0. It is then obvious that dim(V 0) = codim(V ), and by part (b),1

we haveV 00 =V .

(d) Let ( u∗, . . . , u∗m) be a basis ofU. Note that the maph:E→Km defined such that_1

h(v) = (u∗(v), . . . , u∗m(v))1

for every v∈ E, is a linear map, and that its kernel Kerh is preciselyU^0. Then, by Proposition 4.11,

E≈ Ker (h)⊕ Imh =U^0 Imh,

and since dim(Im

h

)

≤

m

, we deduce that

U

0

⊕

is a subspace ofE of finite codimension at

most m, and by (c), we have dim(U^00) = codim(U^0)≤m = dim(U). However, it is clear thatU⊆U^00, which implies dim(U)≤ dim(U^00), and so dim(U^00) = dim(U) =m, and we must haveU =U^00.

Part (a) of Theorem 4.17 shows that

dim(E)≤ dim(E∗).

WhenE is of finite dimensionn and (u_1, . . . , un) is a basis ofE, by part (c), the family (u∗, . . . , u∗n) is a basis of the dual spaceE∗, called the dual basis of (u_1, . . . , un).1

By part (c) and (d) of theorem 4.17, the maps V→ V 0 andU→ U^0, whereV is a subspace of finite codimension ofE andU is a subspace of finite dimension ofE∗, are inverse bijections. These maps set up a duality between subspaces of finite codimension of E, and subspaces of finite dimension ofE∗.

One should be careful that this bijection does not extend to subspaces ofE∗ of infinite dimension.

When E is of infinite dimension, for every basis (ui)i∈I ofE, the family (u∗)i∈I of coordinate forms is never a basis ofE∗. It is linearly independent, but it is “too small” to generateE∗. For example, ifE = R(N), where N ={0,1,2, . . .}, the mapf :E→ R that sums the nonzero coordinates of a vector inE is a linear form, but it is easy to see that it cannot be expressed as a linear combination of coordinate forms. As a consequence, when E is of infinite dimension,E andE∗ are not isomorphic.

Here is another example illustrating the power of Theorem 4.17. Let E = Mn(R), and consider the equations asserting that the sum of the entries in every row of a matrix∈ Mn(R) is equal to the same number. We haven− 1 equations

n

(aij−ai+1j) = 0, 1≤i≤n− 1,

j=1

and it is easy to see that they are linearly independent. Therefore, the spaceU of linear forms inE∗ spanned by the above linear forms (equations) has dimensionn− 1, and the spaceU^0 of matrices sastisfying all these equations has dimensionn^2 n + 1. It is not so obvious to find a basis for this space.−

When E is of finite dimensionn and (u_1, . . . , un) is a basis ofE, we noted that the family (u∗, . . . , u∗n) is a basis of the dual spaceE∗ (called the dual basis of (u_1, . . . , un)). Let us see_1

how the coordinates of a linear formÕ∗ over the dual basis (u∗, . . . , u∗n) vary under a change_1

of basis.

Let (u_1, . . . , un) and (v_1, . . . , vn) be two bases ofE, and letP = (ai j) be the change of basis matrix from (u_1, . . . , un) to (v_1, . . . , vn), so that

n

vj = ai jui,

i=1

and letP−1 = (bi j) be the inverse ofP, so that

n

ui = bj ivj.

j=1

Sinceu∗(uj) =δi j andv∗(vj) =δi j, we get

n

v∗(ui) =v∗( bk ivk) =bj i,

k=1

and thusn

v∗ = bj iu∗,

i=1

andn

u∗ = ai jv∗.

j=1

This means that the change of basis from the dual basis (u∗, . . . , u∗n) to the dual basis_1

(v∗, . . . , v∗) is (P−1) . Since

1 n n

Õ∗ = Õiu∗ = Õiv∗,

i=1 i=1

we getn

Õj = ai jÕi,

i=1

so the new coordinatesÕj are expressed in terms of the old coordinatesÕi using the matrix P . If we use the row vectors (Õ_1, . . . , Õn) and (Õ_1, . . . , Õn), we have

(Õ_1, . . . , Õn) = (Õ_1, . . . , Õn)P. Comparing with the change of basis

n

vj = ai jui,

i=1

we note that this time, the coordinates ( Õi) of the linear formÕ∗ change in the same direction as the change of basis. For this reason, we say that the coordinates of linear forms are covariant. By abuse of language, it is often said that linear forms are covariant, which explains why the term covector is also used for a linear form.

Observe that if (e_1, . . . , en) is a basis of the vector spaceE, then, as a linear map from E toK, every linear formf∈E∗ is represented by a 1×n matrix, that is, by a row vector (λ_1, . . . , λn), with respect to the basis (e_1, . . . , en) ofE, and 1 ofK, wheref(ei) =λi. A vectoru = n uiei∈E is represented by an× 1 matrix, that is, by a column vectori=1

ëu1ö

ì. ÷,í ø

un

and the action off onu, namelyf(u), is represented by the matrix product ëu1ö

λ

1

· · ·

λ

ì

n í .÷ =λ_1u_1 +· · · +λnun.ø

un

On the other hand, with respect to the dual basis (e∗, . . . , e∗n) ofE∗, the linear formf is

1

represented by the column vectorëλ1ö

ì. ÷.íλ ø

n

Remark: In many texts using tensors, vectors are often indexed with lower indices. If so, it is more convenient to write the coordinates of a vectorx over the basis (u_1, . . . , un) as (xi), using an upper index, so thatn

x = xiui,

i=1

and in a change of basis, we haven

vj = aijui

i=1

andn

xi = aijxj.

j=1

Dually, linear forms are indexed with upper indices. Then, it is more convenient to write the coordinates of a covectorÕ∗ over the dual basis (u∗1, . . . , u∗n) as (Õi), using a lower index, so thatn

Õ∗ = Õiu∗i

i=1

and in a change of basis, we have

n

u∗i = aijv∗j j=1

andn

Õj = aijÕi.

i=1

With these conventions, the index of summation appears once in upper position and once in lower position, and the summation sign can be safely omitted, a trick due to Einstein. For example, we can write

Õj =aijÕi as an abbreviation forn

Õj = aijÕi.

i=1

For another example of the use of Einstein’s notation, if the vectors (v_1, . . . , vn) are linear combinations of the vectors (u_1, . . . , un), with

n

vi = aijuj, 1≤i≤n,

j=1

then the above equations are witten as

vi =ajuj, 1≤i≤n.i

Thus, in Einstein’s notation, then×n matrix (aij) is denoted by (aj), a (1,1)-tensor.i

Beware that some authors view a matrix as a mapping between coordinates, in which case the matrix (aij) is denoted by (aij).

We will now pin down the relationship between a vector spaceE and its bidualE∗∗. Proposition 4.18. LetE be a vector space. The following properties hold: (a) The linear map evalE:E→E∗∗ defined such that

evalE(v) = evalv for allv∈E,

that is, evalE(v)(u∗) = u∗, v =u∗(v) for everyu∗∈E∗, is injective. (b) WhenE is of finite dimensionn, the linear map evalE:E→E∗∗ is an isomorphism (called the canonical isomorphism).

Proof. (a) Let (ui)i∈I be a basis ofE, and letv =i∈Iviui. If evalE(v) = 0, then in particular, evalE(v)(u∗) = 0 for allu∗, and since

evalE(v)(u∗) = u∗, v =vi, we havevi = 0 for alli∈I, that is,v = 0, showing that evalE:E→E∗∗ is injective.

If E is of finite dimensionn, by Theorem 4.17, for every basis (u_1, . . . , un), the family (u∗, . . . , u∗n) is a basis of the dual spaceE∗, and thus the family (u_1, . . . , u∗∗ ) is a basis of1 n the bidualE∗∗. This shows that dim(E) = dim(E∗∗) =n, and since by part (a), we know that evalE:E→E∗∗ is injective, in fact, evalE:E→E∗∗ is bijective (because an injective map carries a linearly independent family to a linearly independent family, and in a vector space of dimensionn, a linearly independent family ofn vectors is a basis, see Proposition 2.8).

When a vector spaceE has infinite dimension,E and its bidualE∗∗ are never isomorphic.

WhenE is of finite dimension and (u_1, . . . , un) is a basis ofE, in view of the canonical isomorphism evalE:E→E∗∗, the basis (u_1, . . . , u∗∗ ) of the bidual is identified with

n

( u_1, . . . , un).

Proposition 4.18 can be reformulated very fruitfully in terms of pairings.

Definition 4.9. Given two vector spacesE andF overK, a pairing betweenE andF is a bilinear mapÕ:E×F→K. Such a pairing is nondegenerate iff

(1) for everyu∈E, ifÕ(u, v) = 0 for allv∈F, thenu = 0, and (2) for everyv∈F, ifÕ(u, v) = 0 for allu∈E, thenv = 0.

A pairing Õ:E×F→ K is often denoted by−,−:E×F→K. For example, the map−,−:E∗×E→K defined earlier is a nondegenerate pairing (use the proof of (a) in Proposition 4.18).

Given a pairingÕ:E×F→K, we can define two mapslÕ:E→F∗ andrÕ:F→E∗ as follows: For everyu∈E, we define the linear formlÕ(u) inF∗ such that lÕ(u)(y) =Õ(u, y) for everyy∈F ,

and for everyv∈F, we define the linear formrÕ(v) inE∗ such that

rÕ(v)(x) =Õ(x, v) for everyx∈E.

We have the following useful proposition.

Proposition 4.19. Given two vector spaces E and F over K, for every nondegenerate pairingÕ:E×F→K betweenE andF, the mapslÕ:E→F∗ andrÕ:F→E∗ are linear and injective. Furthermore, ifE andF have finite dimension, then this dimension is the same andlÕ:E→F∗ andrÕ:F→E∗ are bijections.

Proof. The mapslÕ:E→F∗ andrÕ:F→E∗ are linear because a pairing is bilinear. If lÕ(u) = 0 (the null form), then

lÕ(u)(v) =Õ(u, v) = 0 for everyv∈F , 4.3. HYPERPLANES AND LINEAR FORMS 107

and since Õ is nondegenerate,u = 0. Thus,lÕ:E→F∗ is injective. Similarly,rÕ:F→E∗ is injective. WhenF has finite dimensionn, we have seen thatF andF∗ have the same dimension. SincelÕ:E→F∗ is injective, we havem = dim(E)≤ dim(F) =n. The same argument applies toE, and thusn = dim(F)≤ dim(E) =m. But then, dim(E) = dim(F), andlÕ:E→F∗ andrÕ:F→E∗ are bijections.

When E has finite dimension, the nondegenerate pairing−,−:E∗ ×E→ K yields another proof of the existence of a natural isomorphism betweenE andE∗∗. Interesting nondegenerate pairings arise in exterior algebra. We now show the relationship between hyperplanes and linear forms.

4.3 Hyperplanes and Linear Forms

Actually, Proposition 4.20 below follows from parts (c) and (d) of Theorem 4.17, but we feel that it is also interesting to give a more direct proof.

Proposition 4.20. LetE be a vector space. The following properties hold: (a) Given any nonnull linear formf∗∈E∗, its kernelH = Kerf∗ is a hyperplane.

(b) For any hyperplaneH inE, there is a (nonnull) linear formf∗∈E∗ such thatH = Kerf∗.

(c) Given any hyperplaneH inE and any (nonnull) linear formf∗∈E∗ such thatH = Kerf∗, for every linear formg∗∈E∗,H = Kerg∗ iffg∗ =λf∗ for someλ = 0 inK. Proof. (a) Iff∗∈E∗ is nonnull, there is some vectorv0∈E such thatf∗(v_0) = 0. Let H = Kerf∗. For everyv∈E, we have

f∗(v) f∗(v) f∗ v− f∗(v0)v_0 =f∗(v)− f∗(v0)f∗(v_0) =f∗(v)−f∗(v) = 0.

Thus, f∗(v) v− f∗(v0)v_0 =h∈H,

and

v =h + f∗(v)v_0,f∗(v0)

that is,E =H +Kv_0. Also, sincef∗(v_0) = 0, we havev_0/H, that is,H∩Kv_0 = 0. Thus, E =H⊕Kv_0, andH is a hyperplane.

(b) If H is a hyperplane,E =H⊕Kv_0 for somev_0 /H. Then, everyv∈E can be written in a unique way asv =h +λv_0. Thus, there is a well-defined functionf∗:E→K, such that,f∗(v) =λ, for everyv =h +λv_0. We leave as a simple exercise the verification thatf∗ is a linear form. Sincef∗(v_0) = 1, the linear formf∗ is nonnull. Also, by definition, it is clear thatλ = 0 iffv∈H, that is, Kerf∗ =H.

(c) Let H be a hyperplane inE, and letf∗∈E∗ be any (nonnull) linear form such that H = Kerf∗. Clearly, ifg∗ =λf∗ for someλ = 0, thenH = Kerg∗. Conversely, assume that H = Kerg∗ for some nonnull linear formg∗. From (a), we haveE =H⊕Kv_0, for somev_0 such thatf∗(v_0) = 0 andg∗(v_0) = 0. Then, observe that

g∗(v_0)g∗− f∗(v_0)f∗

is a linear form that vanishes onH, since bothf∗ andg∗ vanish onH, but also vanishes on Kv_0. Thus,g∗ =λf∗, with

λ =g∗(v_0).f∗(v0)

We leave as an exercise the fact that every subspace V =E of a vector spaceE, is the intersection of all hyperplanes that containV . We now consider the notion of transpose of a linear map and of a matrix.

4.4 Transpose of a Linear Map and of a Matrix

Given a linear mapf :E→F, it is possible to define a mapf :F∗→E∗ which has some interesting properties.

Definition 4.10. Given a linear mapf :E→F, the transposef :F∗→E∗ off is the linear map defined such that

f (v∗) =v∗æf, for everyv∗∈F∗,

as shown in the diagram below: E

BBB

f F

BBBBB v∗ f (v∗)

K. Equivalently, the linear mapf :F∗→E∗ is defined such that

v∗, f(u) = f (v∗), u ,

for allu∈E and allv∗∈F∗. It is easy to verify that the following properties hold:

(f +g) =f +g

(gæf) =f æg

idE = idE∗.

Note the reversal of composition on the right-hand side of (gæf) =f æg . The equation (gæf) =f æg implies the following useful proposition. Proposition 4.21. Iff :E→F is any linear map, then the following properties hold: (1) Iff is injective, thenf is surjective.

(2) Iff is surjective, thenf is injective.

Proof. Iff :E→ F is injective, then it has a retractionr:F→E such thatræf = idE, and iff :E→F is surjective, then it has a sections:F→E such thatfæs = idF. Now, iff :E→F is injective, then we have

( ræf) =f ær = idE∗,

which implies thatf is surjective, and iff is surjective, then we have

(fæs) =s æf = idF∗,

which implies thatf is injective.

We also have the following property showing the naturality of the eval map. Proposition 4.22. For any linear mapf :E→F, we have

f æ evalE = evalFæf,

or equivalently, the following diagram commutes:

E

∗∗

f F∗∗

evalE evalF

Ef F.

Proof. For everyu∈E and everyÕ∈F∗∗, we have

(f æ evalE)(u)(Õ) = f (evalE(u)), Õ = evalE(u), f (Õ) = f (Õ), u = Õ, f(u) = evalF(f(u)), Õ = (evalFæf)(u), Õ = (evalFæf)(u)(Õ),

which proves thatf æ evalE = evalFæf, as claimed. IfE andF are finite-dimensional, then evalE and then evalF are isomorphisms, so Proposition 4.22 shows that if we identifyE with its bidualE∗∗ andF with its bidualF∗∗ then

(f ) =f.

As a corollary of Proposition 4.22, if dim(E) is finite, then we have

Ker (f ) = evalE(Ker (f)).

Indeed, if E is finite-dimensional, the map evalE:E→E∗∗ is an isomorphism, so every Õ∈E∗∗ is of the formÕ = evalE(u) for someu∈E, the map evalF :F→F∗∗ is injective, and we have

f (Õ) = 0 iff f (evalE(u)) = 0 iff evalF(f(u)) = 0 iff f(u) = 0

iff u∈ Ker (f)

iff Õ∈ evalE(Ker (f)),

which proves that Ker (f ) = evalE(Ker (f)).

The following proposition shows the relationship between orthogonality and transposition.

Proposition 4.23. Given a linear mapf :E→F, for any subspaceV ofE, we have

f(V )0 = (f )−1(V 0) ={w∗∈F∗|f (w∗)∈V 0 .}

As a consequence,

Kerf = (Imf)0 and Kerf = (Imf )0. Proof. We have w∗, f(v) = f (w∗), v ,

for all v∈E and allw∗∈F∗, and thus, we have w∗, f(v) = 0 for everyv∈V , i.e. w∗∈f(V )0, iff f (w∗), v = 0 for everyv∈V , ifff (w∗)∈V 0, i.e. w∗∈ (f)−1(V 0), proving that

f(V )0 = (f )−1(V 0).

Since we already observed thatE^0 = 0, lettingV =E in the above identity, we obtain that

Kerf = (Imf)0.

From the equation

w∗, f(v) = f (w∗), v , we deduce thatv∈ (Imf )0 iff f (w∗), v = 0 for allw∗∈ F∗ iff w∗, f(v) = 0 for all w∗∈F∗. Assume thatv∈ (Imf )0. If we pick a basis (wi)i∈I ofF, then we have the linear formsw∗ :F→K such thatw∗(wj) =δij, and since we must have w∗, f(v) = 0 for all i∈I and (wi)i∈I is a basis ofF, we conclude thatf(v) = 0, and thusv∈ Kerf (this is because w∗, f(v) is the coefficient off(v) associated with the basis vectorwi). Conversely, ifv∈ Kerf, then w∗, f(v) = 0 for allw∗∈ F∗, so we conclude thatv∈ (Imf )0. Therefore,v∈ (Imf )0 iffv∈ Kerf; that is,

Kerf = (Imf )0, as claimed.

The following proposition gives a natural interpretation of the dual (E/U)∗ of a quotient spaceE/U.

Proposition 4.24. For any subspaceU of a vector spaceE, ifp:E→E/U is the canonical surjection ontoE/U, thenp is injective and

Im( p ) =U^0 = (Ker (p))0. Therefore,p is a linear isomorphism between (E/U)∗ andU^0.

Proof. Sincep is surjective, by Proposition 4.21, the mapp is injective. Obviously,U = Ker (p). Observe that Im(p ) consists of all linear formsψ∈E∗ such thatψ =Õæp for someÕ∈ (E/U)∗, and since Ker (p) =U, we haveU⊆ Ker (ψ). Conversely for any linear formψ∈E∗, ifU⊆ Ker (ψ), thenψ factors throughE/U asψ =ψæp as shown in the following commutative diagram

E

CCC

p E/U

CCCCCCψψ

K, whereψ:E/U→K is given by

ψ(v) =ψ(v), v∈E,

where v∈E/U denotes the equivalence class ofv∈E. The mapψ does not depend on the representative chosen in the equivalence classv, since ifv =v, that isv−v =u∈U, then ψ(v ) =ψ(v +u) =ψ(v) +ψ(u) =ψ(v) + 0 =ψ(v). Therefore, we have

Im(p ) ={Õæp|Õ∈ (E/U)∗} ={ψ:E→K|U⊆ Ker (ψ)}=U0,

which proves our result. Proposition 4.24 yields another proof of part (b) of the duality theorem (theorem 4.17) that does not involve the existence of bases (in infinite dimension).

Proposition 4.25. For any vector spaceE and any subspaceV ofE, we haveV 00 =V .

Proof. We begin by observing thatV 0 =V 000. This is because, for any subspaceU ofE∗, we haveU⊆U^00, soV 0 V 000. Furthermore,V⊆ V 00 holds, and for any two subspaces⊆

M, N ofE, ifM^0 N^0, so we getV 000 V 0. WriteV_1 =V 00, so that V

0 =V 000 =V 0 ⊆N thenN⊆ ⊆

1 . We wish to prove thatV_1 =V .

SinceV⊆V_1 =V 00, the canonical projectionp_1:E→E/V_1 factors asp_1 =fæp as in the diagram below,

E

CC

p E/V CCCCCCC fp_1

E/V_1

where p:E→ E/V is the canonical projection ontoE/V andf :E/V→ E/V_1 is the quotient map induced byp_1, withf(uE/V) =p_1(u) =uE/V1, for allu∈E (sinceV⊆V_1, if u−u =v∈V , thenu−u =v∈V_1, sop_1(u) =p_1(u )). Sincep_1 is surjective, so isf. We wish to prove thatf is actually an isomorphism, and for this, it is enough to show thatf is injective. By transposing all the maps, we get the commutative diagram

E

p

∗ (E/V )∗HHHHHHHHHHf

(E/V_1)∗,

but by Proposition 4.24, the maps p : (E/V )∗→ V 0 and p_1 : (E/V_1)∗ → V 0 are isomorphism, and since

V

0

=

V

0

1

, we have the following diagram where bothp andp_1 are_1

isomorphisms:

V 0 p (E/V )∗HHHHHHHHHH f

(E/V_1)∗.

Therefore,f = (p )1 p_1 is an isomorphism. We claim that this implies thatf is injective.− æ

If f is not injective, then there is somex∈E/V such thatx = 0 andf(x) = 0, so for everyÕ∈ (E/V_1)∗, we havef (Õ)(x) =Õ(f(x)) = 0. However, there is linear form ψ∈ (E/V )∗ such thatψ(x) = 1, soψ =f (Õ) for allÕ∈ (E/V_1)∗, contradicting the fact thatf is surjective. To find such a linear formψ, pick any supplementW ofKx inE/V , so thatE/V =Kx⊕W (W is a hyperplane inE/V not containingx), and defineψ to be zero onW and 1 onx.3 Therefore,f is injective, and since we already know that it is surjective, it is bijective. This means that the canonical mapf :E/V→E/V_1 withV⊆V_1 is an isomorphism, which implies thatV =V_1 =V 00 (otherwise, ifv∈V1−V , thenp1(v) = 0, so f(p(v)) =p_1(v) = 0, butp(v) = 0 sincev /∈V , andf is not injective).

The following theorem shows the relationship between the rank off and the rank off . Theorem 4.26. Given a linear mapf :E→F, the following properties hold.

(a) The dual (Imf)∗ of Imf is isomorphic to Imf =f (F∗); that is,

(Imf)∗≈ Imf .

(b) rk(f)≤ rk(f ). If rk(f) is finite, we have rk(f) = rk(f ). Proof. (a) Consider the linear maps

Ep Imfj F,−→ −→

where

E

p Imf is the surjective map induced byEf F, and Imfj

−→

injective inclusion map of Imf intoF. By definition,f =jæ p

−→j

F is the−→ −→p. To simplify the notation, letI = Imf. By Proposition 4.21, sinceEp I is surjective,I∗ −→E∗ is injective, and since Imfj F is injective,F∗ −→I∗ is surjective. Sincef =jæp, we also have−→

f = (jæp) =p æj ,

and since

F

j p

∗ −→ I∗ is surjective, andI∗ −→ E∗ is injective, we have an isomorphism between (Imf)∗ andf (F∗).

(b) We already noted that part (a) of Theorem 4.17 shows that dim( E)≤ dim(E∗), for every vector space E. Thus, dim(Imf)≤ dim((Imf)∗), which, by (a), shows that rk(f)≤ rk(f ). When dim(Imf) is finite, we already observed that as a corollary of Theorem 4.17, dim(Imf) = dim((Imf)∗), and thus, by part (a) we have rk(f) = rk(f ).

If dim(F) is finite, then there is also a simple proof of (b) that doesn’t use the result of part (a). By Theorem 4.17(c)

dim(Imf) + dim((Imf)0) = dim(F),

and by Theorem 4.11 dim(Kerf ) + dim(Imf ) = dim(F∗).

3Using Zorn’s lemma, we pick W maximal among all subspaces of E/V such that Kx∩W = (0); then, E/V = Kx⊕W .

Furthermore, by Proposition 4.23, we have

Kerf = (Imf)0,

and sinceF is finite-dimensional dim(F) = dim(F∗), so we deduce

dim(Imf) + dim((Imf)0) = dim((Imf)0) + dim(Imf ),

which yields dim(Imf) = dim(Imf ); that is, rk(f) = rk(f ).

Remarks:

1. If dim(E) is finite, following an argument of Dan Guralnik, we can also prove that rk(f) = rk(f ) as follows.

We know from Proposition 4.23 applied tof :F∗→E∗ that

Ker (f ) = (Imf )0,

and we showed as a consequence of Proposition 4.22 that

Ker (f ) = evalE(Ker (f)).

It follows (since evalE is an isomorphism) that

dim((Imf )0) = dim(Ker (f )) = dim(Ker (f)) = dim(E)− dim(Imf),

and since dim(Imf ) + dim((Imf )0) = dim(E), we get

dim(Imf ) = dim(Imf).

2. As indicated by Dan Guralnik, if dim(E) is finite, the above result can be used to prove that

Imf = (Ker (f))0. From

f (Õ), u = Õ, f(u)

for all Õ∈F∗ and allu∈E, we see that ifu∈ Ker (f), then f (Õ), u = Õ,0 = 0, which means thatf (Õ)∈ (Ker (f))0, and thus, Imf ⊆ (Ker (f))0. For the converse, since dim(E) is finite, we have

dim((Ker (f))0) = dim(E)− dim(Ker (f)) = dim(Imf), but we just proved that dim(Imf ) = dim(Imf), so we get

dim((Ker (f))0) = dim(Imf ),

and since Imf ⊆ (Ker (f))0, we obtain

Imf = (Ker (f))0,

as claimed. Now, since (Ker (f))00 = Ker (f), the above equation yields another proof of the fact that

Ker (f) = (Imf )0, whenE is finite-dimensional.

3. The equation

Imf = (Ker (f))0 is actually valid even if whenE if infinite-dimensional, as we now prove. Proposition 4.27. Iff :E→F is any linear map, then the following identities hold: Imf = (Ker (f))0

Ker (f ) = (Imf)0

Imf = (Ker (f )0

Ker (f) = (Imf )0.

Proof. The equation Ker (f ) = (Imf)0 has already been proved in Proposition 4.23.

By the duality theorem (Ker ( f))00 = Ker (f), so from Imf = (Ker (f))0 we get Ker (f) = (Imf )0. Similarly, (Imf)00 = Imf, so from Ker (f ) = (Imf)0 we get Imf = (Ker (f )0. Therefore, what is left to be proved is that Imf = (Ker (f))0.

Letp:E→E/Ker (f) be the canonical surjection,f :E/Ker (f)→ Imf be the isomorphism induced byf, andj: Imf→F be the inclusion map. Then, we have f =jæfæp,

which implies that

Sincep is surjective, p f =p æf æj .

is injective, sincej is injective, j is surjective, and sincef is

bijective, f is also bijective. It follows that (E/Ker (f))∗ = Im(f æj ), and we have Imf = Imp .

Sincep:E→E/Ker (f) is the canonical surjection, by Proposition 4.24 applied toU = Ker (f), we get

Imf = Imp = (Ker (f))0, as claimed.

In summary, the equation Imf = (Ker (f))0 applies in any dimension, and it implies that

Ker (f) = (Imf )0.

The following proposition shows the relationship between the matrix representing a linear mapf :E→F and the matrix representing its transposef :F∗→E∗.

Proposition 4.28. LetE andF be two vector spaces, and let (u_1, . . . , un) be a basis for E, and (v_1, . . . , vm) be a basis forF. Given any linear mapf :E→F, ifM(f) is the m×n-matrix representingf w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm), then×m-matrix M(f ) representingf :F∗→E∗ w.r.t. the dual bases (v∗, . . . , v∗m) and (u∗, . . . , u∗n) is the1 1 transposeM(f) ofM(f).

Proof. Recall that the entryai j in rowi and columnj ofM(f) is thei-th coordinate of f(uj) over the basis (v_1, . . . , vm). By definition ofv∗, we have v∗, f(uj) =ai j. The entry aj i in rowj and columni ofM(f ) is thej-th coordinate of

f

(

v

∗

) =

a

1

i

u

∗

1 +· · · +aj iu∗ +· · · +an iu∗n

over the basis (u∗, . . . , u∗n), which is justaj i =f (v∗)(uj) = f (v∗), uj . Since_1

v∗, f(uj) = f (v∗), uj,

we haveai j =aj i, proving thatM(f ) =M(f) .

We now can give a very short proof of the fact that the rank of a matrix is equal to the rank of its transpose.

Proposition 4.29. Given am×n matrixA over a fieldK, we have rk(A) = rk(A ).

Proof. The matrixA corresponds to a linear mapf :Kn Km, and by Theorem 4.26,→

rk(f) = rk(f ). By Proposition 4.28, the linear mapf corresponds toA . Since rk(A) = rk(f), and rk(A ) = rk(f ), we conclude that rk(A) = rk(A ).

Thus, given an m×n-matrixA, the maximum number of linearly independent columns is equal to the maximum number of linearly independent rows. There are other ways of proving this fact that do not involve the dual space, but instead some elementary transformations on rows and columns.

Proposition 4.29 immediately yields the following criterion for determining the rank of a matrix:

4.5. THE FOUR FUNDAMENTAL SUBSPACES 117

Proposition 4.30. Given anym×n matrixA over a fieldK (typicallyK = R orK = C), the rank ofA is the maximum natural number r such that there is an invertibler×r submatrix ofA obtained by selectingr rows andr columns ofA.

For example, the 3× 2 matrix ëa_11 a_12ö A = ía_21 a_22ø a_31 a_32 has rank 2 iff one of the three 2× 2 matrices

a_11 a_12 a_11 a_12 a_21 a_22 a_21 a_22 a_31 a_32 a_31 a_32

is invertible. We will see in Chapter 5 that this is equivalent to the fact the determinant of one of the above matrices is nonzero. This is not a very efficient way of finding the rank of a matrix. We will see that there are better ways using various decompositions such as LU, QR, or SVD.

4.5 The Four Fundamental Subspaces

Given a linear mapf :E→F (whereE andF are finite-dimensional), Proposition 4.23 revealed that the four spaces

Imf, Imf , Kerf, Kerf

play a special role. They are often called the fundamental subspaces associated withf. These spaces are related in an intimate manner, since Proposition 4.23 shows that 0Kerf = (Imf )

Kerf = (Imf)0,

and Theorem 4.26 shows that rk(f) = rk(f ).

It is instructive to translate these relations in terms of matrices (actually, certain linear algebra books make a big deal about this!). If dim(E) =n and dim(F) =m, given any basis (u_1, . . . , un) ofE and a basis (v_1, . . . , vm) ofF, we know thatf is represented by anm×n matrixA = (ai j), where thejth column ofA is equal tof(uj) over the basis (v_1, . . . , vm). Furthermore, the transpose mapf is represented by then×m matrixA (with respect to the dual bases). Consequently, the four fundamental spaces

Imf, Imf , Kerf, Kerf correspond to (1) The column space ofA, denoted by ImA orR(A); this is the subspace of Rm spanned by the columns ofA, which corresponds to the image Imf off.

(2) The kernel or nullspace ofA, denoted by KerA orN(A); this is the subspace of Rn consisting of all vectorsx∈ Rn such thatAx = 0.

(3) The row space ofA, denoted by ImA orR(A ); this is the subspace of Rn spanned by the rows ofA, or equivalently, spanned by the columns ofA , which corresponds to the image Imf off .

(4) The left kernel or left nullspace ofA denoted by KerA orN(A ); this is the kernel (nullspace) ofA , the subspace of Rm consisting of all vectorsy∈ Rm such that A y = 0, or equivalently,y A = 0.

Recall that the dimension r of Imf, which is also equal to the dimension of the column space ImA =R(A), is the rank ofA (andf). Then, some our previous results can be reformulated as follows:

1. The column spaceR(A) ofA has dimensionr.

2. The nullspaceN(A) ofA has dimensionn−r.

3. The row spaceR(A ) has dimensionr.

4. The left nullspaceN(A ) ofA has dimensionm−r.

The above statements constitute what Strang calls the Fundamental Theorem of Linear Algebra, Part I (see Strang [101]).

The two statements

Kerf = (Imf )0 0Kerf = (Imf)

translate to (1) The nullspace ofA is the orthogonal of the row space ofA. (2) The left nullspace ofA is the orthogonal of the column space ofA.

The above statements constitute what Strang calls the Fundamental Theorem of Linear Algebra, Part II (see Strang [101]).

Since vectors are represented by column vectors and linear forms by row vectors (over a basis inE orF), a vectorx∈ Rn is orthogonal to a linear formy if

yx = 0. Then, a vectorx∈ Rn is orthogonal to the row space ofA iffx is orthogonal to every row ofA, namelyAx = 0, which is equivalent to the fact thatx belong to the nullspace ofA. Similarly, the column vectory∈ Rm (representing a linear form over the dual basis ofF∗) belongs to the nullspace ofAiffA y = 0, iffy A = 0, which means that the linear form given byy (over the basis inF) is orthogonal to the column space ofA.

Since (2) is equivalent to the fact that the column space of A is equal to the orthogonal of the left nullspace ofA, we get the following criterion for the solvability of an equation of the formAx =b:

The equationAx =b has a solution iff for ally∈ Rm, ifA y = 0, theny b = 0. Indeed, the condition on the right-hand side says thatb is orthogonal to the left nullspace ofA, that is, thatb belongs to the column space ofA.

This criterion can be cheaper to check that checking directly thatb is spanned by the columns ofA. For example, if we consider the system

x1−x_2 =b_1

x2−x_3 =b_2

x3−x_1 =b_3

which, in matrix form, is written Ax =b as below: ë1 −1 0 öëx1ö ëb1ö

í 01 −1øíx_2ø = íb_2ø, −1 0_1 x_3 b_3

we see that the rows of the matrix A add up to 0. In fact, it is easy to convince ourselves that the left nullspace ofA is spanned byy = (1,1,1), and so the system is solvable iffy b = 0, namely

b_1 +b_2 +b_3 = 0.

Note that the above criterion can also be stated negatively as follows:

The equationAx =b has no solution iff there is somey∈ Rm such thatA y = 0 and y b = 0.

4.6 Summary

The main concepts and results of this chapter are listed below: Direct products, sums, direct sums.•

• Projections.

• The fundamental equation dim(E) = dim(Kerf) + dim(Imf) = dim(Kerf) + rk(f)

(Proposition 4.11). Grassmann’s relation•

dim(U) + dim(V ) = dim(U +V ) + dim(U∩V ).

• Characterizations of a bijective linear mapf :E→F. Rank of a matrix.•

• The dual spaceE∗ and linear forms (covector). The bidualE∗∗. bilinear pairing−,−:E∗×E→K (the canonical pairing).• The

Evaluation atv: evalv:E∗→K.•

• The map evalE:E→E∗∗.

Othogonality

between a subspace

V

of

E

and a subspace

U

of

E

∗

; the orthogonal V

0

• and the orthogonalU^0.

Coordinate forms.•

• The Duality theorem (Theorem 4.17).

dual basis of a basis.• The

• The isomorphism evalE:E→E∗∗ when dim(E) is finite.

Pairing between two vector spaces; nondegenerate pairing; Proposition 4.19.•

• Hyperplanes and linear forms.

transposef :F∗→E∗ of a linear mapf :E→F.• The

• The fundamental identities:

Kerf = (Imf)0 and Kerf = (Imf )0 (Proposition 4.23).

• IfF is finite-dimensional, then rk(f) = rk(f ). (Theorem 4.26).

•

The matrix of the transpose mapf is equal to the transpose of the matrix of the map f (Proposition 4.28).

•

For anym×n matrixA, rk(A) = rk(A ).

•

Characterization of the rank of a matrix in terms of a maximal invertible submatrix (Proposition 4.30).

• The four fundamental subspaces:

Imf, Imf , Kerf, Kerf .

• The column space, the nullspace, the row space, and the left nullspace (of a matrix).

•

Criterion for the solvability of an equation of the formAx =b in terms of the left nullspace.

Chapter 5 Determinants

5.1 Permutations, Signature of a Permutation

This chapter contains a review of determinants and their use in linear algebra. We begin with permutations and the signature of a permutation. Next, we define multilinear maps and alternating multilinear maps. Determinants are introduced as alternating multilinear maps taking the value 1 on the unit matrix (following Emil Artin). It is then shown how to compute a determinant using the Laplace expansion formula, and the connection with the usual definition is made. It is shown how determinants can be used to invert matrices and to solve (at least in theory!) systems of linear equations (the Cramer formulae). The determinant of a linear map is defined. We conclude by defining the characteristic polynomial of a matrix (and of a linear map) and by proving the celebrated Cayley-Hamilton theorem which states that every matrix is a “zero” of its characteristic polynomial (we give two proofs; one computational, the other one more conceptual).

Determinants can be defined in several ways. For example, determinants can be defined in a fancy way in terms of the exterior algebra (or alternating algebra) of a vector space. We will follow a more algorithmic approach due to Emil Artin. No matter which approach is followed, we need a few preliminaries about permutations on a finite set. We need to show that every permutation onn elements is a product of transpositions, and that the parity of the number of transpositions involved is an invariant of the permutation. Let [n] ={1,2. . . , n}, wheren∈ N, andn > 0.

Definition 5.1. A permutation onn elements is a bijectionπ: [n]→ [n]. Whenn = 1, the only function from [1] to [1] is the constant map: 1→ 1. Thus, we will assume thatn≥ 2. A transposition is a permutationτ : [n]→ [n] such that, for somei < j (with 1≤i < j≤n), τ(i) =j,τ(j) =i, andτ(k) =k, for allk∈ [n]− {i, j}. In other words, a transposition exchanges two distinct elementsi, j∈ [n]. A cyclic permutation of orderk (ork-cycle) is a permutationσ: [n]→ [n] such that, for somei_1, i_2, . . . , ik, with 1≤i_1< i_2< . . . < ik≤n, andk≥ 2,

σ(i_1) =i_2, . . . , σ(ik−1) =ik, σ(ik) =i_1,

123 andσ(j) =j, forj∈ [n]− {i_1, . . . , ik}. The set{i_1, . . . , ik} is called the domain of the cyclic permutation, and the cyclic permutation is sometimes denoted by (i_1, i_2, . . . , ik).

If τ is a transposition, clearly,τæτ = id. Also, a cyclic permutation of order 2 is a transposition, and for a cyclic permutationσ of orderk, we haveσk = id. Clearly, the composition of two permutations is a permutation and every permutation has an inverse which is also a permutation. Therefore, the set of permutations on [n] is a group often denoted Sn. It is easy to show by induction that the group Sn hasn! elements. We will also use the terminology product of permutations (or transpositions), as a synonym for composition of permutations.

The following proposition shows the importance of cyclic permutations and transpositions.

Proposition 5.1. For everyn≥ 2, for every permutationπ: [n]→ [n], there is a partition of [n] intor subsets, with 1≤r≤n, where each setJ in this partition is either a singleton

i}, or it is of the form{ J ={i, π(i), π^2(i), . . . , πri−1(i)},

where ri is the smallest integer, such that, πri(i) = i and 2≤ ri≤ n. Ifπ is not the identity, then it can be written in a unique way as a composition π =σ1æ. . .æσs of cyclic permutations (where 1≤ s≤ r). Every permutation π: [n]→ [n] can be written as a nonempty composition of transpositions.

Proof. Consider the relationRπ defined on [n] as follows:iRπj iff there is somek≥ 1 such thatj =πk(i). We claim thatRπ is an equivalence relation. Transitivity is obvious. We claim that for everyi∈ [n], there is some leastr (1≤r≤n) such thatπr(i) =i. Indeed, consider the following sequence ofn + 1 elements:

i, π(i), π^2(i), . . . , πn(i).

Since [n] only hasn distinct elements, there are someh, k with 0≤h < k≤n such that πh(i) =πk(i),

and since π is a bijection, this impliesπk−h(i) =i, where 0≤k−h≤n. Thus,Rπ is reflexive. It is symmetric, since ifj =πk(i), lettingr be the leastr≥ 1 such thatπr(i) =i, then

i =πkr(i) =πk(r−1)(πk(i)) =πk(r−1)(j).

Now, for everyi∈ [n], the equivalence class ofi is a subset of [n], either the singleton{i} or a set of the form

J ={i, π(i), π^2(i), . . . , πri−1(i)},

where ri is the smallest integer such thatπri(i) =i and 2≤ri≤n, and in the second case, the restriction ofπ toJ induces a cyclic permutationσi, andπ =σ1æ. . .æσs, wheres is the number of equivalence classes having at least two elements.

5.1. PERMUTATIONS, SIGNATURE OF A PERMUTATION 125

For the second part of the proposition, we proceed by induction on n. Ifn = 2, there are exactly two permutations on [2], the transpositionτ exchanging 1 and 2, and the identity. However, id_2 =τ^2. Now, letn≥ 3. Ifπ(n) =n, since by the induction hypothesis, the restriction ofπ to [n− 1] can be written as a product of transpositions,π itself can be written as a product of transpositions. Ifπ(n) =k =n, lettingτ be the transposition such thatτ(n) =k andτ(k) =n, it is clear thatτæπ leavesn invariant, and by the induction hypothesis, we haveτæπ =τmæ. . .æτ_1 for some transpositions, and thus

π =τæτmæ. . .æτ_1,

a product of transpositions (sinceτæτ = idn).

Remark: Whenπ = idn is the identity permutation, we can agree that the composition of 0 transpositions is the identity. The second part of Proposition 5.1 shows that the transpositions generate the group of permutations Sn.

In writing a permutation π as a compositionπ =σ1æ. . .æσs of cyclic permutations, it is clear that the order of theσi does not matter, since their domains are disjoint. Given a permutation written as a product of transpositions, we now show that the parity of the number of transpositions is an invariant.

Definition 5.2. For everyn≥ 2, since every permutationπ: [n]→ [n] defines a partition ofr subsets over whichπ acts either as the identity or as a cyclic permutation, let (π), called the signature ofπ, be defined by (π) = (−1)n−r, wherer is the number of sets in the partition.

If τ is a transposition exchangingi andj, it is clear that the partition associated with τ consists ofn− 1 equivalence classes, the set{i, j}, and then− 2 singleton sets{k}, for k∈ [n]− {i, j}, and thus, (τ) = (−1)n−(n−1) = (−1)1 =−1.

Proposition 5.2. For everyn≥ 2, for every permutationπ: [n]→ [n], for every transpositionτ, we have

(τæπ) =− (π).

Consequently, for every product of transpositions such thatπ =τmæ. . .æτ_1, we have

( π) = (−1)m, which shows that the parity of the number of transpositions is an invariant.

Proof. Assume thatτ(i) =j andτ(j) =i, wherei < j. There are two cases, depending whetheri andj are in the same equivalence classJl ofRπ, or if they are in distinct equivalence classes. Ifi andj are in the same classJl, then if

Jl ={i_1, . . . , ip, . . . iq, . . . ik}, whereip =i andiq =j, since

τ(π(π−1(ip))) =τ(ip) =τ(i) =j =iq and τ(π(iq−1)) =τ(iq) =τ(j) =i =ip,

it is clear that Jl splits into two subsets, one of which is{ip, . . . , iq−1}, and thus, the number of classes associated withτæπ isr + 1, and (τæπ) = (−1)n−r−1 =−(−1)n−r =− (π). Ifi andj are in distinct equivalence classesJl andJm, say

{i_1, . . . , ip, . . . ih}

and

where ip =i andjq =j, since{j1, . . . , jq, . . . jk}, τ(π(π−1(ip))) =τ(ip) =τ(i) =j =jq

and

τ(π(π−1(jq))) =τ(jq) =τ(j) =i =ip,

we see that the classesJl andJm merge into a single class, and thus, the number of classes associated withτæπ isr− 1, and (τæπ) = (−1)n−r+1 =−(−1)n−r =− (π). Now, letπ =τmæ. . .æτ_1 be any product of transpositions. By the first part of the proposition, we have

(π) = (−1)m−1 (τ_1) = (−1)m−1(−1) = (−1)m, since (τ_1) =−1 for a transposition.

Remark: Whenπ = idn is the identity permutation, since we agreed that the composition of 0 transpositions is the identity, it it still correct that (−1)0 = (id) = +1. From the proposition, it is immediate that (πæπ) = (π ) (π). In particular, sinceπ−1 π = idn, we get (π−1) = (π).æ

We can now proceed with the definition of determinants.

5.2 Alternating Multilinear Maps

First, we define multilinear maps, symmetric multilinear maps, and alternating multilinear maps.

Remark: Most of the definitions and results presented in this section also hold whenK is a commutative ring, and when we consider modules overK (free modules, when bases are needed).

Let E_1, . . . , En, andF, be vector spaces over a fieldK, wheren≥ 1.

Definition 5.3. A functionf :E1×. . .×En→F is a multilinear map (or ann-linear map) if it is linear in each argument, holding the others fixed. More explicitly, for everyi, 1≤i≤n, for allx1∈E_1. . .,xi−1∈Ei−1,xi+1∈Ei+1, . . .,xn∈En, for allx, y∈Ei, for all λ∈K,

f(x_1, . . . , xi−1, x +y, xi+1, . . . , xn) =f(x_1, . . . , xi−1, x, xi+1, . . . , xn) +f(x_1, . . . , xi−1, y, xi+1, . . . , xn), f(x_1, . . . , xi−1, λx, xi+1, . . . , xn) =λf(x_1, . . . , xi−1, x, xi+1, . . . , xn).

When F =K, we callf ann-linear form (or multilinear form). Ifn≥ 2 andE_1 = E_2 =. . . =En, ann-linear mapf :E×. . .×E→F is called symmetric, iff(x_1, . . . , xn) = f(xπ(1), . . . , xπ(n)), for every permutationπ on{1, . . . , n}. Ann-linear mapf :E×. . .×E→ F is called alternating, iff(x_1, . . . , xn) = 0 wheneverxi =xi+1, for somei, 1≤i≤n−1 (in other words, when two adjacent arguments are equal). It does not harm to agree that when n = 1, a linear map is considered to be both symmetric and alternating, and we will do so.

When n = 2, a 2-linear mapf :E1×E2→F is called a bilinear map. We have already seen several examples of bilinear maps. Multiplication·:K×K→K is a bilinear map, treatingK as a vector space over itself. More generally, multiplication·:A×A→A in a ringA is a bilinear map, viewingA as a module over itself.

The operation−,−:E∗×E→K applying a linear form to a vector is a bilinear map. Symmetric bilinear maps (and multilinear maps) play an important role in geometry (inner products, quadratic forms), and in differential calculus (partial derivatives). A bilinear map is symmetric iff(u, v) =f(v, u), for allu, v∈E.

Alternating multilinear maps satisfy the following simple but crucial properties. Proposition 5.3. Letf :E×. . .×E→F be ann-linear alternating map, withn≥ 2. The following properties hold:

(1) f(. . . , xi, xi+1, . . .) =−f(. . . , xi+1, xi, . . .)

(2) f(. . . , xi, . . . , xj, . . .) = 0, wherexi =xj, and 1≤i < j≤n.

(3) f(. . . , xi, . . . , xj, . . .) =−f(. . . , xj, . . . , xi, . . .), where 1≤i < j≤n.

(4) f(. . . , xi, . . .) =f(. . . , xi +λxj, . . .), for anyλ∈K, and wherei =j.

Proof. (1) By multilinearity applied twice, we have

f(. . . , xi +xi+1, xi +xi+1, . . .) =f(. . . , xi, xi, . . .) +f(. . . , xi, xi+1, . . .) +f(. . . , xi+1, xi, . . .) +f(. . . , xi+1, xi+1, . . .), and sincef is alternating, this yields 0 =f(. . . , xi, xi+1, . . .) +f(. . . , xi+1, xi, . . .),

that is,f(. . . , xi, xi+1, . . .) =−f(. . . , xi+1, xi, . . .).

(2) Ifxi =xj andi andj are not adjacent, we can interchangexi andxi+1, and thenxi andxi+2, etc, untilxi andxj become adjacent. By (1),

f(. . . , xi, . . . , xj, . . .) =f(. . . , xi, xj, . . .), where = +1 or−1, butf(. . . , xi, xj, . . .) = 0, sincexi =xj, and (2) holds. (3) follows from (2) as in (1). (4) is an immediate consequence of (2).

Proposition 5.3 will now be used to show a fundamental property of alternating multilinear maps. First, we need to extend the matrix notation a little bit. LetE be a vector space overK. Given ann×n matrixA = (ai j) overK, we can define a mapL(A):En En as follows:→

L(A)1(u) =a1 1u_1 +· · · +a1 nun,

. . .

L(A)n(u) =an 1u_1 +· · · +an nun,

for all u_1, . . . , un∈E, withu = (u_1, . . . , un). It is immediately verified thatL(A) is linear. Then, given twon×n matriceA = (ai j) andB = (bi j), by repeating the calculations establishing the product of matrices (just before Definition 3.1), we can show that

L(AB) =L(A)æL(B). It is then convenient to use the matrix notation to describe the effect of the linear mapL(A),

as ëL(A)1(u)ö ëa1 1 a1 2 . . . a1 nö ëu_1ö

ìL(A)2(u)÷ ìa2 1 a2 2 . . . a2 n÷ ìu_2÷

ì ÷= ì... ÷ ì ÷

.

÷ ì. ÷ .ì. ÷ ì

í

L(A)n(u)ø ían 1 an 2 . . . an n

. . ø íu ø

n

Lemma 5.4. Letf :E×. . .×E→F be ann-linear alternating map. Let (u_1, . . . , un) and (v_1, . . . , vn) be two families ofn vectors, such that,

v_1 =a1 1u_1 +· · · +an 1un,

. . .

vn =a1 nu_1 +· · · +an nun. Equivalently, lettingëa1 1 a1 2 . . . a1 nö

ì

÷

ì

a

2 1

a

2 2

. . . a

2 n

÷

A =ì .... ÷ í . . ø an 1 an 2 . . . an n

assume that we haveëv1ö ëu1ö

ìv_2÷ ìu_2÷ ì ÷=A ì ÷.ì . ÷ ì . ÷

ív ø í ø

n un

Then,

f(v_1, . . . , vn) = (π)aπ(1) 1· · ·aπ(n) n f(u_1, . . . , un),

π∈Sn

where the sum ranges over all permutationsπ on{1, . . . , n}. Proof. Expandingf(v_1, . . . , vn) by multilinearity, we get a sum of terms of the form

aπ(1) 1· · ·aπ(n) nf(uπ(1), . . . , uπ(n)),

for all possible functions π:{1, . . . , n} → {1, . . . , n}. However, becausef is alternating, only the terms for whichπ is a permutation are nonzero. By Proposition 5.1, every permutation π is a product of transpositions, and by Proposition 5.2, the parity (π) of the number of transpositions only depends onπ. Then, applying Proposition 5.3 (3) to each transposition inπ, we get

aπ(1) 1· · ·aπ(n) nf(uπ(1), . . . , uπ(n)) = (π)aπ(1) 1· · ·aπ(n) nf(u_1, . . . , un). Thus, we get the expression of the lemma.

The quantity det(A) = (π)aπ(1) 1· · ·aπ(n) n

π∈Sn

is in fact the value of the determinant of A (which, as we shall see shortly, is also equal to the determinant ofA ). However, working directly with the above definition is quite ackward, and we will proceed via a slightly indirect route

5.3 Definition of a Determinant

Recall that the set of all squaren×n-matrices with coefficients in a fieldK is denoted by Mn(K).

Definition 5.4. A determinant is defined as any map

D: Mn(K)→K,

which, when viewed as a map on ( Kn)n, i.e., a map of then columns of a matrix, isn-linear alternating and such thatD(In) = 1 for the identity matrixIn. Equivalently, we can consider a vector spaceE of dimensionn, some fixed basis (e_1, . . . , en), and define

D:En K→

as ann-linear alternating map such thatD(e_1, . . . , en) = 1.

First, we will show that such maps D exist, using an inductive definition that also gives a recursive method for computing determinants. Actually, we will define a family (Dn)n≥1 of (finite) sets of mapsD: Mn(K)→K. Second, we will show that determinants are in fact uniquely defined, that is, we will show that eachDn consists of a single map. This will show the equivalence of the direct definition det(A) of Lemma 5.4 with the inductive definition D(A). Finally, we will prove some basic properties of determinants, using the uniqueness theorem.

Given a matrixA∈ Mn(K), we denote itsn columns byA^1, . . . , An.

Definition 5.5. For everyn≥ 1, we define a finite setDn of mapsD: Mn(K)→ K inductively as follows:

Whenn = 1,D1 consists of the single mapD such that,D(A) =a, whereA = (a), with a∈K.

Assume thatD n −1 has been defined, wheren≥ 2. We define the setDn as follows. For every matrixA∈ Mn(K), letAi j be the (n− 1)× (n− 1)-matrix obtained fromA = (ai j) by deleting rowi and columnj. Then,Dn consists of all the mapsD such that, for somei, 1≤i≤n,

D(A) = (−1)i+1ai 1D(Ai 1) +· · · + (−1)i+nai nD(Ai n),

where for everyj, 1≤j≤n,D(Ai j) is the result of applying anyD inDn−1 toAi j.

We confess that the use of the same letter D for the member ofDn being defined, and for members ofDn−1, may be slightly confusing. We considered using subscripts to distinguish, but this seems to complicate things unnecessarily. One should not worry too much anyway, since it will turn out that eachDn contains just one map.

Each (−1)i+jD(Ai j) is called the cofactor ofai j, and the inductive expression forD(A) is called a Laplace expansion ofD according to thei-th row. Given a matrixA∈ Mn(K), eachD(A) is called a determinant ofA.

We can think of each member ofDn as an algorithm to evaluate “the” determinant ofA. The main point is that these algorithms, which recursively evaluate a determinant using all possible Laplace row expansions, all yield the same result, det(A).

We will prove shortly thatD(A) is uniquely defined (at the moment, it is not clear that Dn consists of a single map). Assuming this fact, given an×n-matrixA = (ai j),

ëa1 1 a1 2 . . . a1 nö

ì

÷

ì. . ø

a

2 1

a

2 2

. . . a

2

n

÷

A =ì .... ÷

ían 1 an 2 . . . an n

its determinant is denoted byD(A) or det(A), or more explicitly by

a1 1 a1 2 . . . a1 n a2 1 a2 2 . . . a2 ndet(A) = . . ... .

an 1 an 2 . . . an n

First, let us first consider some examples. Example 5.1.

1. Whenn = 2, if A = a b c d expanding according to any row, we have

D(A) =ad−bc.

2. Whenn = 3, ifëa1 1 a1 2 a1 3ö

A = ía2 1 a2 2 a2 3ø a3 1 a3 2 a3 3 expanding according to the first row, we have

a2 2 a2 3 a2 1 a2 3 +a a2 1 a2 2D(A) =a1 1 a3 2 a3 3−a1 2 a3 1 a3 3 1 3 a3 1 a3 2 that is, D(A) =a1 1(a2 2a3 3−a3 2a2 3)−a1 2(a2 1a3 3−a3 1a2 3) +a1 3(a2 1a3 2−a3 1a2 2), which gives the explicit formula D(A) =a1 1a2 2a3 3 +a2 1a3 2a1 3 +a3 1a1 2a2 3−a1 1a3 2a2 3−a2 1a1 2a3 3−a3 1a2 2a1 3.

We now show that eachD∈ Dn is a determinant (map).

Lemma 5.5. For everyn≥ 1, for everyD∈ Dn as defined in Definition 5.5,D is an alternating multilinear map such that D(In) = 1.

Proof. By induction on n, it is obvious that D(In) = 1. Let us now prove that D is multilinear. Let us show thatD is linear in each column. Consider any columnk. Since D(A) = (i+1ai 1D(Ai 1) +· · · + (−1)i+jai jD(Ai j) +· · · + (−1)i+nai nD(Ai n),−1)

if j =k, then by induction,D(Ai j) is linear in columnk, andai j does not belong to column k, so (−1)i+jai jD(Ai j) is linear in columnk. Ifj =k, thenD(Ai j) does not depend on columnk =j, sinceAi j is obtained fromA by deleting rowi and columnj =k, andai j belongs to columnj =k. Thus, (−1)i+jai jD(Ai j) is linear in columnk. Consequently, in all cases, (−1)i+jai jD(Ai j) is linear in columnk, and thus,D(A) is linear in columnk.

Let us now prove that D is alternating. Assume that two adjacent rows ofA are equal, sayAk =Ak+1. First, letj =k andj =k + 1. Then, the matrixAi j has two identical adjacent columns, and by the induction hypothesis,D(Ai j) = 0. The remaining terms of D(A) are

(−1)i+kai kD(Ai k) + (−1)i+k+1ai k+1D(Ai k+1).

However, the two matrices Ai k andAi k+1 are equal, since we are assuming that columnsk andk + 1 ofA are identical, and sinceAi k is obtained fromA by deleting rowi and column k, andAi k+1 is obtained fromA by deleting rowi and columnk + 1. Similarly,ai k =ai k+1, since columnsk andk + 1 ofA are equal. But then,

(i+kai kD(Ai k) + (−1)i+k+1ai k+1D(Ai k+1) = (−1)i+kai kD(Ai k)−(−1)i+kai kD(Ai k) = 0.−1)

This shows thatD is alternating, and completes the proof.

Lemma 5.5 shows the existence of determinants. We now prove their uniqueness. Theorem 5.6. For everyn≥ 1, for everyD∈ Dn, for every matrixA∈ Mn(K), we have

D(A) = (π)aπ(1) 1· · ·aπ(n) n,

π∈Sn

where the sum ranges over all permutations π on{1, . . . , n}. As a consequence,Dn consists of a single map for everyn≥ 1, and this map is given by the above explicit formula. Proof. Consider the standard basis (e_1, . . . , en) ofKn, where (ei)i = 1 and (ei)j = 0, for j =i. Then, each columnAj ofA corresponds to a vectorvj whose coordinates over the basis (e_1, . . . , en) are the components ofAj, that is, we can write

v_1 =a1 1e_1 +· · · +an 1en,

. . .

vn =a1 ne_1 +· · · +an nen.

Since by Lemma 5.5, eachD is a multilinear alternating map, by applying Lemma 5.4, we get

D(A) =D(v_1, . . . , vn) = (π)aπ(1) 1· · ·aπ(n) n D(e_1, . . . , en),

π∈Sn

where the sum ranges over all permutationsπ on{1, . . . , n}. ButD(e_1, . . . , en) =D(In), and by Lemma 5.5, we haveD(In) = 1. Thus,

D(A) = (π)aπ(1) 1· · ·aπ(n) n,

π∈Sn

where the sum ranges over all permutationsπ on{1, . . . , n}.

From now on, we will favor the notation det(A) overD(A) for the determinant of a square matrix.

Remark: There is a geometric interpretation of determinants which we find quite illuminating. Givenn linearly independent vectors (u_1, . . . , un) in Rn, the set

Pn ={λ_1u_1 +· · · +λnun| 0≤λi≤ 1, 1≤i≤n}

is called a parallelotope. Ifn = 2, thenP_2 is a parallelogram and ifn = 3, thenP_3 is a parallelepiped, a skew box havingu_1, u_2, u_3 as three of its corner sides. Then, it turns out that det(u_1, . . . , un) is the signed volume of the parallelotopePn (where volume means n-dimensional volume). The sign of this volume accounts for the orientation ofPn in Rn.

We can now prove some properties of determinants.

Corollary 5.7. For every matrixA∈ Mn(K), we have det(A) = det(A ). Proof. By Theorem 5.6, we have

det(A) = (π)aπ(1) 1· · ·aπ(n) n,

π∈Sn

where the sum ranges over all permutationsπ on{1, . . . , n}. Since a permutation is invertible, every product

aπ(1) 1· · ·aπ(n) n

can be rewritten as a1 π−1(1)· · ·an π−1(n),

and since (π−1) = (π) and the sum is taken over all permutations on{1, . . . , n}, we have

(π)aπ(1) 1· · ·aπ(n) n = (σ)a1 σ(1)· · ·an σ(n), π∈Sn σ∈Sn

whereπ andσ range over all permutations. But it is immediately verified that

det(A ) = (σ)a1 σ(1)· · ·an σ(n).

σ∈Sn

A useful consequence of Corollary 5.7 is that the determinant of a matrix is also a multilinear alternating map of its rows. This fact, combined with the fact that the determinant of a matrix is a multilinear alternating map of its columns is often useful for finding short-cuts in computing determinants. We illustrate this point on the following example which shows up in polynomial interpolation.

Example 5.2. Consider the so-called Vandermonde determinant

1 1 . . . 1 x_1 x_2 . . . xn V

(

x

1

, . . . , x

n

) =

x

2x2

1 2 . . . x^2n .

. . ....

xn−1 xn−1 . . . xn−11 2 n

We claim that V (x_1, . . . , xn) = (xj−xi),

1≤i<j≤n

with V (x_1, . . . , xn) = 1, whenn = 1. We prove it by induction onn≥ 1. The casen = 1 is obvious. Assumen≥ 2. We proceed as follows: multiply rown− 1 byx_1 and substract it from rown (the last row), then multiply rown− 2 byx_1 and subtract it from rown− 1, etc, multiply rowi− 1 byx_1 and subtract it from rowi, until we reach row 1. We obtain the following determinant:

1 1 . . . 1

0 x2−x_1 . . . xn−x_1 V (x_1, . . . , xn) = 0 x_2(x2−x_1) . . . xn(xn−x_1)

. . ....

0 xn−2(x2−x_1) . . . xn−2(xn−x_1)2 n

Now, expanding this determinant according to the first column and using multilinearity, we can factor (xi−x_1) from the column of indexi− 1 of the matrix obtained by deleting the first row and the first column, and thus

V (x_1, . . . , xn) = (x2−x_1)(x3−x_1)· · ·(xn−x_1)V (x_2, . . . , xn), which establishes the induction step.

Lemma 5.4 can be reformulated nicely as follows.

Proposition 5.8. Letf :E×. . .×E→F be ann-linear alternating map. Let (u_1, . . . , un) and (v_1, . . . , vn) be two families ofn vectors, such that

v_1 =a1 1u_1 +· · · +a1 nun,

. . .

vn =an 1u_1 +· · · +an nun. Equivalently, lettingëa1 1 a1 2 . . . a1 nö

ì

÷

ì. . ø

a

2 1

a

2 2

. . . a

2 n

÷

A =ì .... ÷ ían 1 an 2 . . . an n assume that we haveëv1ö ëu1ö

ì v_2÷ ìu_2÷ ì ÷=Aì ÷ ì .

í

.

÷ ì. ÷

ø í ø vn un Then,

f(v_1, . . . , vn) = det(A)f(u_1, . . . , un).

Proof. The only difference with Lemma 5.4 is that here, we are usingA instead ofA. Thus, by Lemma 5.4 and Corollary 5.7, we get the desired result.

As a consequence, we get the very useful property that the determinant of a product of matrices is the product of the determinants of these matrices.

Proposition 5.9. For any twon×n-matricesA andB, we have det(AB) = det(A) det(B). Proof. We use Proposition 5.8 as follows: let (e_1, . . . , en) be the standard basis ofKn, and let ëw1ö ëe1ö

ìw_2÷ ìe_2÷

ì ÷=ABì ÷.ì . ÷ ì . ÷

íw ø í ø

n en

Then, we get det(w_1, . . . , wn) = det(AB) det(e_1, . . . , en) = det(AB), since det(e_1, . . . , en) = 1. Now, letting

ëv1ö ëe1ö

ìv_2÷ ìe_2÷

ì ÷=Bì ÷,ì . ÷ ì . ÷

ív ø í ø

n en

we get det(v_1, . . . , vn) = det(B), and sinceëw1ö ëv1ö

ì w_2÷ ìv_2÷ ì ÷=Aì ÷ ì ,

í

.

÷ ì. ÷

ø í ø wn vn we get

det(w_1, . . . , wn) = det(A) det(v_1, . . . , vn) = det(A) det(B).

It should be noted that all the results of this section, up to now, also holds when K is a commutative ring, and not necessarily a field. We can now characterize when ann×n-matrix A is invertible in terms of its determinant det(A).

5.4 Inverse Matrices and Determinants

In the next two sections,K is a commutative ring and when needed, a field. Definition 5.6. LetK be a commutative ring. Given a matrixA∈ Mn(K), letA = (bi j) be the matrix defined such that

bi j = (−1)i+j det(Aj i),

the cofactor ofaj i. The matrixA is called the adjugate ofA, and each matrixAj i is called a minor of the matrixA.

Note the reversal of the indices in

bi j = (−1)i+j det(Aj i).

Thus,A is the transpose of the matrix of cofactors of elements ofA. We have the following proposition. Proposition 5.10. LetK be a commutative ring. For every matrixA∈ Mn(K), we have

AA =AA = det(A)In.

As a consequence,A is invertible iffdet(A) is invertible, and if so,A−1 = (det(A))−1A.

Proof. IfA = (bi j) andAA = (ci j), we know that the entryci j in rowi and columnj ofAA is

ci j =ai 1b1 j +· · · +ai kbk j +· · · +ai nbn j, which is equal to

ai 1(−1)j+1 det(Aj 1) +· · · +ai n(−1)j+n det(Aj n).

Ifj =i, then we recognize the expression of the expansion of det(A) according to thei-th row:

ci i = det(A) =ai 1(−1)i+1 det(Ai 1) +· · · +ai n(−1)i+n det(Ai n).

If j =i, we can form the matrixA by replacing thej-th row ofA by thei-th row ofA. Now, the matrixAj k obtained by deleting rowj and columnk fromA is equal to the matrix Aj k obtained by deleting rowj and columnk fromA , sinceA andA only differ by thej-th row. Thus,

det(Aj k) = det(Aj k), and we have

ci j =ai 1(−1)j+1 det(Aj 1) +· · · +ai n(−1)j+n det(Aj n).

However, this is the expansion of det( A ) according to thej-th row, since thej-th row ofA is equal to thei-th row ofA, and sinceA has two identical rowsi andj, because det is an alternating map of the rows (see an earlier remark), we have det(A ) = 0. Thus, we have shown thatci i = det(A), andci j = 0, whenj =i, and so

AA = det(A)In.

It is also obvious from the definition ofA, that

A =A .

Then, applying the first part of the argument toA , we have

A A = det(A )In,

and since, det(A ) = det(A),A =A , and (AA) =A A , we get

det(A)In =A A =A A = (AA) , that is, (AA) = det(A)In, which yields

AA = det(A)In, sinceIn =In. This proves that

AA =AA = det(A)In.

As a consequence, if det( A) is invertible, we haveA−1 = (det(A))−1A. Conversely, ifA is invertible, fromAA−1 =In, by Proposition 5.9, we have det(A) det(A−1) = 1, and det(A) is invertible.

When K is a field, an elementa∈K is invertible iffa = 0. In this case, the second part of the proposition can be stated asA is invertible iff det(A) = 0. Note in passing that this method of computing the inverse of a matrix is usually not practical.

We now consider some applications of determinants to linear independence and to solving systems of linear equations. Although these results hold for matrices over an integral domain, their proofs require more sophisticated methods (it is necessary to use the fraction field of the integral domain,K). Therefore, we assume again thatK is a field.

LetA be ann×n-matrix,x a column vectors of variables, andb another column vector, and letA^1, . . . , An denote the columns ofA. Observe that the system of equationAx =b,

ëa1 1 a1 2 . . . a1 nö ëx_1ö ëb_1ö

ìa2 1 a2 2 . . . a2 n÷ ìx_2÷ ìb_2÷

ì... ÷ ì ÷

.

÷

ì

.

÷

=

ì ÷

ì. . ì

ø

í

ø

í

.

÷

í ø

an 1 an 2 . . . an n xn bn

is equivalent to x_1A^1 +· · · +xjAj +· · · +xnAn =b, since the equation corresponding to thei-th row is in both cases

ai 1x_1 +· · · +ai jxj +· · · +ai nxn =bi.

First, we characterize linear independence of the column vectors of a matrixA in terms of its determinant.

Proposition 5.11. Given ann× n-matrixA over a fieldK, the columnsA^1, . . . , An of A are linearly dependent iff det(A) = det(A^1, . . . , An) = 0. Equivalently,A has rankn iff det(A) = 0.

Proof. First, assume that the columnsA^1, . . . , An ofA are linearly dependent. Then, there arex_1, . . . , xn∈K, such that

x_1A^1 +· · · +xjAj +· · · +xnAn = 0,

wherexj = 0 for somej. If we compute

det(A^1, . . . , x_1A^1 +· · · +xjAj +· · · +xnAn, . . . , An) = det(A^1, . . . ,0, . . . , An) = 0,

where 0 occurs in thej-th position, by multilinearity, all terms containing two identical columnsAk fork =j vanish, and we get

xj det(A^1, . . . , An) = 0.

Sincexj = 0 andK is a field, we must have det(A^1, . . . , An) = 0.

Conversely, we show that if the columns A^1, . . . , An ofA are linearly independent, then det(A^1, . . . , An) = 0. If the columnsA^1, . . . , An ofA are linearly independent, then they form a basis ofKn, and we can express the standard basis (e_1, . . . , en) ofKn in terms of A^1, . . . , An. Thus, we have

ëe_1ö ëb1 1 b1 2 . . . b1 nö ëA1ö

ìe_2÷ ìb2 1 b2 2 . . . b2 n÷ ì

÷ ì ÷

ì

A

2÷

ì ÷ =ì .... ÷ ì. ÷,

í

.

÷ ì

ø í . . ø ínø

en bn 1 bn 2 . . . bn n A

for some matrixB = (bi j), and by Proposition 5.8, we get

det(e_1, . . . , en) = det(B) det(A^1, . . . , An),

and since det( e_1, . . . , en) = 1, this implies that det(A^1, . . . , An) = 0 (and det(B) = 0). For the second assertion, recall that the rank of a matrix is equal to the maximum number of linearly independent columns, and the conclusion is clear.

If we combine Proposition 5.11 with Proposition 4.30, we obtain the following criterion for finding the rank of a matrix.

Proposition 5.12. Given anym×n matrixA over a fieldK (typicallyK = R orK = C), the rank ofA is the maximum natural number r such that there is anr× r submatrixB of A obtained by selectingr rows andr columns ofA, and such that det(B) = 0.

6

5.5 Systems of Linear Equations and Determinants

We now characterize when a system of linear equations of the formAx =b has a unique solution.

Proposition 5.13. Given ann×n-matrixA over a fieldK, the following properties hold: (1) For every column vectorb, there is a unique column vectorx such thatAx =b iff the only solution toAx = 0 is the trivial vectorx = 0, iff det(A) = 0.

(2) If det(A) = 0, the unique solution ofAx =b is given by the expressions det(

A

1, . . . , Aj−1, b, Aj+1, . . . , An

xj = ) ,det(A^1, . . . , Aj−1, Aj, Aj+1, . . . , An)

known as Cramer’s rules.

(3) The system of linear equationsAx = 0 has a nonzero solution iff det(A) = 0. Proof. Assume thatAx =b has a single solutionx_0, and assume thatAy = 0 withy = 0. Then,

A(x_0 +y) =Ax_0 +Ay =Ax_0 + 0 =b,

and x_0 +y =x_0 is another solution ofAx =b, contadicting the hypothesis thatAx =b has a single solutionx_0. Thus,Ax = 0 only has the trivial solution. Now, assume thatAx = 0 only has the trivial solution. This means that the columnsA^1, . . . , An ofA are linearly independent, and by Proposition 5.11, we have det(A) = 0. Finally, if det(A) = 0, by Proposition 5.10, this means thatA is invertible, and then, for everyb,Ax =b is equivalent tox =A−1b, which shows thatAx =b has a single solution.

(2) Assume that Ax =b. If we compute

det(A^1, . . . , x_1A^1 +· · · +xjAj +· · · +xnAn, . . . , An) = det(A^1, . . . , b, . . . , An),

whereb occurs in thej-th position, by multilinearity, all terms containing two identical columnsAk fork =j vanish, and we get

x^1, . . . , An) = det(A^1, . . . , Aj−1, b, Aj+1, . . . , An),j det(A

for everyj, 1≤j≤n. Since we assumed that det(A) = det(A^1, . . . , An) = 0, we get the desired expression.

(3) Note that Ax = 0 has a nonzero solution iffA^1, . . . , An are linearly dependent (as observed in the proof of Proposition 5.11), which, by Proposition 5.11, is equivalent to det(A) = 0.

As pleasing as Cramer’s rules are, it is usually impractical to solve systems of linear equations using the above expressions.

5.6. DETERMINANT OF A LINEAR MAP 141

5.6 Determinant of a Linear Map

We close this chapter with the notion of determinant of a linear mapf :E→E.

Given a vector space E of finite dimensionn, given a basis (u_1, . . . , un) ofE, for every linear mapf :E→ E, ifM(f) is the matrix off w.r.t. the basis (u_1, . . . , un), we can define det(f) = det(M(f)). If (v_1, . . . , vn) is any other basis ofE, and ifP is the change of basis matrix, by Corollary 3.5, the matrix off with respect to the basis (v_1, . . . , vn) isP−1M(f)P. Now, by proposition 5.9, we have

det(P−1M(f)P) = det(P−1) det(M(f)) det(P) = det(P−1) det(P) det(M(f)) = det(M(f)).

Thus, det(f) is indeed independent of the basis ofE.

Definition 5.7. Given a vector spaceE of finite dimension, for any linear mapf :E→E, we define the determinant det(f) off as the determinant det(M(f)) of the matrix off in any basis (since, from the discussion just before this definition, this determinant does not depend on the basis).

Then, we have the following proposition.

Proposition 5.14. Given any vector spaceE of finite dimensionn, a linear mapf :E→E is invertible iff det(f) = 0.

Proof. The linear mapf :E→E is invertible iff its matrixM(f) in any basis is invertible (by Proposition 3.2), iff det(M(f)) = 0, by Proposition 5.10.

Given a vector space of finite dimension n, it is easily seen that the set of bijective linear mapsf :E→ E such that det(f) = 1 is a group under composition. This group is a subgroup of the general linear group GL(E). It is called the special linear group (ofE), and it is denoted by SL(E), or whenE =Kn, by SL(n, K), or even by SL(n).

5.7 The Cayley–Hamilton Theorem

We conclude this chapter with an interesting and important application of Proposition 5.10, the Cayley–Hamilton theorem. The results of this section apply to matrices over any commutative ringK. First, we need the concept of the characteristic polynomial of a matrix.

Definition 5.8. IfK is any commutative ring, for everyn×n matrixA∈ Mn(K), the characteristic polynomialPA(X) ofA is the determinant

PA(X) = det(XI−A). The characteristic polynomialPA(X) is a polynomial inK[X], the ring of polynomials in the indeterminateX with coefficients in the ringK. For example, whenn = 2, if

A = a b ,c d

then PA(X) = X−a −b =X^2 (a +d)X +ad−bc.

−c X−d −

We can substitute the matrixA for the variableX in the polynomialPA(X), obtaining a matrixPA. If we write

PA(X) =Xn +c_1Xn−1 +· · · +cn, then

PA =An +c_1An−1 +· · · +cnI. We have the following remarkable theorem.

Theorem 5.15. (Cayley–Hamilton) IfK is any commutative ring, for everyn×n matrix A∈ Mn(K), if we let

PA(X) =Xn +c_1Xn−1 +· · · +cn be the characteristic polynomial ofA, then

PA =An +c_1An−1 +· · · +cnI = 0.

Proof. We can view the matrixB =XI−A as a matrix with coefficients in the polynomial ringK[X], and then we can form the matrixB which is the transpose of the matrix of cofactors of elements ofB. Each entry inB is an (n−1)×(n−1) determinant, and thus a polynomial of degree a mostn− 1, so we can writeB as

B =Xn−1B_0 +Xn−2B_1 +· · · +Bn−1,

for some matricesB_0, . . . , Bn−1 with coefficients inK. For example, whenn = 2, we have

B = X−a −b , B = X−d b =X1 0 + −d b . −c X−dc X−a 0 1c −a

By Proposition 5.10, we have

BB = det(B)I =PA(X)I.

On the other hand, we have

BB = (XI−A)(Xn−1B_0 +Xn−2B_1 +· · · +Xn−j−1Bj +· · · +Bn−1), and by multiplying out the right-hand side, we get

BB =XnD_0 +Xn−1D_1 +· · · +Xn−jDj +· · · +Dn,

with

D_0 =B_0

D_1 =B1−AB_0 .

Dj =Bj−ABj−1 .

Dn−1 =Bn−1−ABn−2 Dn =−ABn−1.

Since PA(X)I = (Xn +c_1Xn−1 +· · · +cn)I, the equality XnD_0 +Xn−1D_1 +· · · +Dn = (Xn +c_1Xn−1 +· · · +cn)I

is an equality between two matrices, so it 1requires that all corresponding entries are equal, and since these are polynomials, the coefficients of these polynomials must be identical, which is equivalent to the set of equations

I =B_0

c_1I =B1−AB_0 .

cjI =Bj−ABj−1 .

cn−1I =Bn−1 −ABn−2 cnI =−ABn−1,

for all j, with 1≤j≤n− 1. If we multiply the first equation byAn, the last byI, and generally the (j + 1)th byAn−j, when we add up all these new equations, we see that the right-hand side adds up to 0, and we get our desired equation

An +c_1An−1 +· · · +cnI = 0,

as claimed. As a concrete example, whenn = 2, the matrix

A

=

a b c d satisfies the equation A^2 (a +d)A + (ad−bc)I = 0.−

Most readers will probably find the proof of Theorem 5.15 rather clever but very mysterious and unmotivated. The conceptual difficulty is that we really need to understand how polynomials in one variable “act” on vectors, in terms of the matrixA. This can be done and yields a more “natural” proof. Actually, the reasoning is simpler and more general if we free ourselves from matrices and instead consider a finite-dimensional vector spaceE and some given linear mapf :E→ E. Given any polynomialp(X) =a_0Xn +a_1Xn−1 +· · ·+an with coefficients in the fieldK, we define the linear mapp(f):E→E by

p(f) =an +a_1fn−1 +· · · +anid,0f

wherefk =fæ · · · æf, thek-fold composition off with itself. Note that p(f)(u) =a_0fn(u) +a_1fn−1(u) +· · · +anu, for every vectoru∈E. Then, we define a new kind of scalar multiplication·:K[X]×E→E by polynomials as follows: for every polynomialp(X)∈K[X], for everyu∈E, p(X)·u =p(f)(u). It is easy to verify that this is a “good action,” which means that

p· (u +v) =p·u +p·v (p +q)·u =p·u +q·u (pq)·u =p· (q·u)

1·u =u,

for allp, q∈K[X] and allu, v∈E. With this new scalar multiplication,E is aK[X]-module. Ifp =λ is just a scalar inK (a polynomial of degree 0), then

λ·u = (λid)(u) =λu, which means thatK acts onE by scalar multiplication as before. Ifp(X) =X (the monomial X), then

X·u =f(u).

Now, if we pick a basis (e_1, . . . , en), if a polynomialp(X)∈K[X] has the property that p(X)·ei = 0, i = 1, . . . , n, then this means thatp(f)(ei) = 0 fori = 1, . . . , n, which means that the linear mapp(f) vanishes onE. We can also check, as we did in Section 5.2, that ifA andB are twon×n matrices and if (u_1, . . . , un) are anyn vectors, then

ë ëu1öö ëu1ö

A

ì ì

øø ø

ìB· í .÷÷ = (AB)·í .÷.· í un un

This suggests the plan of attack for our second proof of the Cayley–Hamilton theorem. For simplicity, we prove the theorem for vector spaces over a field. The proof goes through for a free module over a commutative ring.

Theorem 5.16. (Cayley–Hamilton) For every finite-dimensional vector space over a field K, for every linear mapf :E→E, for every basis (e_1, . . . , en), ifA is the matrix overf over the basis (e_1, . . . , en) and if

PA(X) =Xn +c_1Xn−1 +· · · +cn is the characteristic polynomial ofA, then

PA(f) =fn +c_1fn−1 +· · · +cnid = 0. Proof. Since the columns ofA consist of the vectorf(ej) expressed over the basis (e_1, . . . , en), we haven

f(ej) = ai jei, 1≤j≤n.

i=1

Using our action ofK[X] onE, the above equations can be expressed as

n

X·ej = ai j·ei, 1≤j≤n,

i=1

which yields

j−1 n

ai j·ei + (X−aj j)·ej +−ai j·ei = 0, 1≤j≤n.

i=1− i=j+1

Observe that the transpose of the characteristic polynomial shows up, so the above system can be written as

ëX−a1 1 −a2 1 · · · −an 1 ö ëe1ö ë0ö ì a1 2 X−a2 2 · · · −an 2 ÷ ìe2÷ ì0÷

ì − ÷ ì ÷

.

.

.

.

÷· ì. ÷ =ì ÷ .ì ì.÷

í a1 n −a2 n · · · X−a ø í ø í ø

n n en 0−

If we letB =XI−A , then as in the previous proof, ifB is the transpose of the matrix of cofactors ofB, we have

BB = det(B)I = det(XI−A )I = det(XI−A)I =PAI. But then, sinceëe1ö ë0ö

ìe_2÷ ì0÷

B

ì ÷

·

ì

.

÷

=

ì ÷

ì ,

í

ø

í

.

÷

ø

en 0

and sinceB is matrix whose entries are polynomials inK[X], it makes sense to multiply on the left byB and we get

ëe1ö ëe1ö ëe1ö ë_0ö ë_0ö

ìe_2÷ ìe_2÷ ìe_2÷ ì0÷ ì0÷

B

·

B

ì ÷ ì ÷ ì ÷ ì ÷

·

ì

.

÷

= (

BB

)

·

ì

.

÷

=

P

A

I

·

ì

.

÷

=

B

·

ì

.

÷

=

ì ÷ ì ;

í

ø

í

ø

í

ø

í

ø

í .

÷

ø en en en 0 0 that is, PA·ej = 0, j = 1, . . . , n, which proves thatPA(f) = 0, as claimed.

If K is a field, then the characteristic polynomial of a linear mapf :E→E is independent of the basis (e_1, . . . , en) chosen inE. To prove this, observe that the matrix off over another basis will be of the formP−1AP, for some inverible matrixP, and then

det( XI^1AP) = det(XP−1IP−P−1AP)−P−

= det(P−1(XI−A)P)

= det(P−1) det(XI−A) det(P) = det(XI−A).

Therefore, the characteristic polynomial of a linear map is intrinsic tof, and it is denoted byPf.

The zeros (roots) of the characteristic polynomial of a linear map f are called the eigenvalues off. They play an important role in theory and applications. We will come back to this topic later on.

5.8 Further Readings

Thorough expositions of the material covered in Chapters 2–4 and 5 can be found in Strang [101, 100], Lax [69], Lang [65], Artin [3], Mac Lane and Birkhoff [70], Hoffman and Kunze 5.8. FURTHER READINGS 147

[60], Bourbaki [12, 13], Van Der Waerden [108], Serre [92], Horn and Johnson [55], and Bertin [10]. These notions of linear algebra are nicely put to use in classical geometry, see Berger [6, 7], Tisseron [105] and Dieudonn´e [26].

Chapter 6 Gaussian Elimination, LU-Factorization, Cholesky Factorization, Reduced Row Echelon Form

6.1 Motivating Example: Curve Interpolation

Curve interpolation is a problem that arises frequently in computer graphics and in robotics (path planning). There are many ways of tackling this problem and in this section we will describe a solution using cubic splines. Such splines consist of cubic B´ezier curves. They are often used because they are cheap to implement and give more flexibility than quadratic B´ezier curves.

A cubic B´ezier curve C(t) (in R^2 or R^3) is specified by a list of four control points (b_0, b_2, b_2, b_3) and is given parametrically by the equation

C(t) = (1−t)3b_0 + 3(1−t)2t b_1 + 3(1−t)t^2b_2 +t^3b_3.

Clearly,C(0) =b_0,C(1) =b_3, and fort∈ [0,1], the pointC(t) belongs to the convex hull of the control pointsb_0, b_1, b_2, b_3. The polynomials

(1−t)3, 3(1−t)2t, 3(1−t)t^2, t^3

are the Bernstein polynomials of degree 3.

Typically, we are only interested in the curve segment corresponding to the values of t in the interval [0,1]. Still, the placement of the control points drastically affects the shape of the curve segment, which can even have a self-intersection; See Figures 6.1, 6.2, 6.3 illustrating various configuations.

149

b_1

b_2

b_0 b_3

Figure 6.1: A “standard” B´ezier curve

b_1

b_3

b0 b2

Figure 6.2: A B´ezier curve with an inflexion point 6.1. MOTIVATING EXAMPLE: CURVE INTERPOLATION 151

b_2 b_1

b_0 b_3

Figure 6.3: A self-intersecting B´ezier curve

Interpolation problems require finding curves passing through some given data points and possibly satisfying some extra constraints.

A B´ezier spline curveF is a curve which is made up of curve segments which are B´ezier curves, sayC_1, . . . , Cm (m≥ 2). We will assume thatF defined on [0, m], so that for i = 1, . . . , m,

F(t) =Ci(t−i + 1), i− 1≤t≤i.

Typically, some smoothness is required between any two junction points, that is, between any two pointsCi(1) andCi+1(0), fori = 1, . . . , m− 1. We require thatCi(1) =Ci+1(0) (C^0-continuity), and typically that the derivatives ofCi at 1 and ofCi+1 at 0 agree up to second order derivatives. This is calledC^2-continuity, and it ensures that the tangents agree as well as the curvatures.

There are a number of interpolation problems, and we consider one of the most common problems which can be stated as follows:

Problem: GivenN + 1 data pointsx_0, . . . , xN, find aC^2 cubic spline curveF, such that F(i) =xi, for alli, 0≤i≤N (N≥ 2).

A way to solve this problem is to findN + 3 auxiliary pointsd−1, . . . , dN+1 called de Boor control points from whichN B´ezier curves can be found. Actually,

d−1 =x_0 and dN+1 =xN so we only need to findN + 1 pointsd_0, . . . , dN.

It turns out that the C^2-continuity constraints on theN B´ezier curves yield onlyN− 1 equations, sod_0 anddN can be chosen arbitrarily. In practice,d_0 anddN are chosen according to various end conditions, such as prescribed velocities atx_0 andxN. For the time being, we will assume thatd_0 anddN are given.

Figure 6.4 illustrates an interpolation problem involvingN + 1 = 7 + 1 = 8 data points. The control pointsd_0 andd_7 were chosen arbitrarily.

d_2

d1 x_2

x_1 d7

d3x_3

d_0 d_6 x_6 x_4

d

4

x_5

d_5

x_0 = d−1 x_7 = d_8

Figure 6.4: AC^2 cubic interpolation spline curve passing through the pointsx_0, x_1,x_2, x_3, x_4, x_5,x_6, x7

3d0 ö

2 2 ì

It can be shown that d_1, . . . , dN−1 are given by the linear system ë7 1 öëd1 ö ë6x1−2 ÷ ì1 4 1 0÷ì d_2 ÷ ì 6x ÷

ì

...

...

...

÷ì ÷ ì

÷ì. ÷= ì. ÷ ì ÷ì ÷ ì ÷ ÷ ì ÷

í

. ì0 1 4 1÷ìdN−2ø í 6xN−2 ø 17øí 3dN2 dN−1 6xN−1− 2

It can be shown that the above matrix is invertible because it is strictly diagonally dominant.

Once the above system is solved, the B´ezier cubicsC_1, . . .,CN are determined as follows (we assumeN≥ 2): For 2≤i≤N− 1, the control points (bi, bi, bi, bi) ofCi are given by0 1 2 3

bi =xi−10

bi = 2di−1 + 1di1 3 3 bi = 1di−1 + 2di2 3 3

bi =xi.3

The control points (b^1, b^1, b^1, b^1) ofC_1 are given by0 1 2 3

b^1 =x_00

b^1 =d01

b^1 = 1d0 + 1d12 2 2

b^1 =x_1,3

and the control points (bN, bN, bN, bN ) ofCN are given by0 1 2 3

bN =xN−10

bN = 1dN−1 + 1dN1 2 2

bN =dN2

bN =xN.3

We will now describe various methods for solving linear systems. Since the matrix of the above system is tridiagonal, there are specialized methods which are more efficient than the general methods. We will discuss a few of these methods.

6.2 Gaussian Elimination andLU-Factorization

Let A be ann×n matrix, letb∈ Rn be ann-dimensional vector and assume thatA is invertible. Our goal is to solve the systemAx =b. SinceA is assumed to be invertible, we know that this system has a unique solution,x =A−1b. Experience shows that two counter-intuitive facts are revealed:

(1) One should avoid computing the inverse, A−1, ofA explicitly. This is because this would amount to solving then linear systems,Au(j) =ej, forj = 1, . . . , n, where ej = (0, . . . ,1, . . . ,0) is thejth canonical basis vector of Rn (with a 1 is thejth slot). By doing so, we would replace the resolution of a single system by the resolution ofn systems, and we would still have to multiplyA−1 byb.

(2) One does not solve (large) linear systems by computing determinants (using Cramer’s formulae). This is because this method requires a number of additions (resp. multiplications) proportional to (n + 1)! (resp. (n + 2)!).

The key idea on which most direct methods (as opposed to iterative methods, that look for an approximation of the solution) are based is that ifA is an upper-triangular matrix, which means thataij = 0 for 1≤j < i≤n (resp. lower-triangular, which means that aij = 0 for 1≤i < j≤n), then computing the solution,x, is trivial. Indeed, sayA is an upper-triangular matrix

ëa1 1 a1 2 · · · a1 n−2 a1 n−1 a1 n ö ì 0 a2 2 · · · a2 n−2 a2 n−1 a2 n ÷

ì ÷

ì 0 0 ... . . .÷ A

=

ì ÷ ì ÷. ì ... . . ÷ ì ÷

í 0 0 · · · 0 an−1 n−1 an−1 nø 0 0 · · · 0 0 an n

Then, det( A) =a1 1a2 2· · · an n = 0, which implies thatai i = 0 fori = 1, . . . , n, and we can solve the systemAx =b from bottom-up by back-substitution. That is, first we compute xn from the last equation, next plug this value ofxn into the next to the last equation and computexn−1 from it, etc. This yields

xn = a−1bnn n

xn−1 = a−1 (bn−1−an−1 nxn)n−1 n−1

.

x_1 = a−1(b1−a1 2x2− · · · −a1 nxn).1 1

Note that the use of determinants can be avoided to prove that if A is invertible then ai i = 0 fori = 1, . . . , n. Indeed, it can be shown directly (by induction) that an upper (or lower) triangular matrix is invertible iff all its diagonal entries are nonzero.

IfA is lower-triangular, we solve the system from top-down by forward-substitution.

Thus, what we need is a method for transforming a matrix to an equivalent one in uppertriangular form. This can be done by elimination. Let us illustrate this method on the following example:

2 x + y + z = 5

4x − 6y = −2

−2x+ 7y + 2z =9.

We can eliminate the variablex from the second and the third equation as follows: Subtract twice the first equation from the second and add the first equation to the third. We get the new system 2x + y + z = 5

−

8y − 2z = − 12

8y+ 3z = 14.

This time, we can eliminate the variabley from the third equation by adding the second equation to the third:

2x + y + z = 5

−

8y − 2z = − 12

z = 2.

This last system is upper-triangular. Using back-substitution, we find the solution: z = 2, y = 1,x = 1.

Observe that we have performed only row operations. The general method is to iteratively eliminate variables using simple row operations (namely, adding or subtracting a multiple of a row to another row of the matrix) while simultaneously applying these operations to the vectorb, to obtain a system,MAx =Mb, whereMA is upper-triangular. Such a method is called Gaussian elimination. However, one extra twist is needed for the method to work in all cases: It may be necessary to permute rows, as illustrated by the following example:

x + y + z = 1

x + y + 3z = 1

2x + 5y + 8z = 1.

In order to eliminatex from the second and third row, we subtract the first row from the second and we subtract twice the first row from the third:

x + y + z = 1

2z = 0

3y + 6z =−1.

Now, the trouble is that y does not occur in the second row; so, we can’t eliminatey from the third row by adding or subtracting a multiple of the second row to it. The remedy is simple: Permute the second and the third row! We get the system:

x + y + z = 1 3y + 6z =−1 2z = 0,

which is already in triangular form. Another example where some permutations are needed is:

z = 1 −2x + 7y + 2z = 1 4x −6y = −1.

First, we permute the first and the second row, obtaining

−

2x + 7y + 2z = 1

z = 1

4x − 6y = −1,

and then, we add twice the first row to the third, obtaining:

−

2x + 7y + 2z = 1

z = 1

8y + 4z = 1.

Again, we permute the second and the third row, getting

−

2x + 7y + 2z = 1

8y + 4z = 1

z = 1,

an upper-triangular system. Of course, in this example, z is already solved and we could have eliminated it first, but for the general method, we need to proceed in a systematic fashion.

We now describe the method of Gaussian Elimination applied to a linear system,Ax =b, whereA is assumed to be invertible. We use the variablek to keep track of the stages of elimination. Initially,k = 1.

(1) The first step is to pick some nonzero entry, ai 1, in the first column ofA. Such an entry must exist, sinceA is invertible (otherwise, the first column ofA would be the zero vector, and the columns ofA would not be linearly independent. Equivalently, we would have det(A) = 0). The actual choice of such an element has some impact on the numerical stability of the method, but this will be examined later. For the time being, we assume that some arbitrary choice is made. This chosen element is called the pivot of the elimination step and is denotedπ_1 (so, in this first step,π_1 =ai 1).

(2) Next, we permute the row ( i) corresponding to the pivot with the first row. Such a step is called pivoting. So, after this permutation, the first element of the first row is nonzero.

(3) We now eliminate the variable x_1 from all rows except the first by adding suitable multiples of the first row to these rows. More precisely we add−ai 1/π_1 times the first row to theith row, fori = 2, . . . , n. At the end of this step, all entries in the first column are zero except the first.

(4) Increment k by 1. Ifk =n, stop. Otherwise,k < n, and then iteratively repeat steps (1), (2), (3) on the (n−k + 1)× (n−k + 1) subsystem obtained by deleting the first k− 1 rows andk− 1 columns from the current system.

If we letA_1 =A andAk = (aki j) be the matrix obtained afterk− 1 elimination steps (2≤k≤n), then thekth elimination step is applied to the matrixAk of the form

ëak ak ak ö1 1 1 2 · · · · · · · · ·1

a

k

n

ì ak ÷

ì2 2 · · · · · · · · ·2 n÷

ì... . . ÷

A

k

=

ì ÷

ì .

ì

a

kk k

· · ·

a

kk n

÷

÷

ì ÷

í . . ø

akn k · · · akn n

Actually, note

aki j =aii j

for alli, j with 1≤i≤k− 2 andi≤j≤n, since the firstk− 1 rows remain unchanged after the (k− 1)th step.

We will prove later that det( Ak) =±det(A). Consequently,Ak is invertible. The fact thatAk is invertible iffA is invertible can also be shown without determinants from the fact that there is some invertible matrixMk such thatAk =MkA, as we will see shortly.

Since Ak is invertible, some entryaki k withk≤i≤n is nonzero. Otherwise, the last n−k + 1 entries in the firstk columns ofAk would be zero, and the firstk columns of Ak would yieldk vectors in Rk−1. But then, the firstk columns ofAk would be linearly dependent andAk would not be invertible, a contradiction.

So, one the entries aki k withk≤i≤n can be chosen as pivot, and we permute thekth row with theith row, obtaining the matrixαk = (αkj l). The new pivot isπk =αkk k, and we zero the entriesi =k + 1, . . . , n in columnk by adding−αki k/πk times rowk to rowi. At the end of this step, we haveAk+1. Observe that the firstk− 1 rows ofAk are identical to the firstk− 1 rows ofAk+1.

It is easy to figure out what kind of matrices perform the elementary row operations used during Gaussian elimination. The key point is that ifA =P B, whereA, B arem×n matrices andP is a square matrix of dimensionm, if (as usual) we denote the rows ofA and B byA_1, . . . , Am andB_1, . . . , Bm, then the formula

m

aij = pikbkj

k=1

giving the (i, j)th entry inA shows that theith row ofA is a linear combination of the rows ofB:

Ai =pi1B_1 +· · · +pimBm.

Therefore, multiplication of a matrix on the left by a square matrix performs row operations. Similarly, multiplication of a matrix on the right by a square matrix performs column operations

The permutation of the kth row with theith row is achieved by multiplyingA on the left by the transposition matrix P(i, k), which is the matrix obtained from the identity matrix by permuting rowsi andk, i.e.,

ë_1 ö

ì 1 ÷ ì0 1 ÷ ì ÷ ì ÷ ì 1 ÷ ì... ÷

P

(

i, k

) =

ì ÷ ì ÷ . ì_1 ÷ ì ÷ ì1 0 ÷ ì ÷ ì_1 ÷

í ø 1

Observe that det(P(i, k)) =−1. Furthermore,P(i, k) is symmetric (P(i, k) =P(i, k)), and P(i, k)−1 =P(i, k).

During the permutation step (2), if rowk and rowi need to be permuted, the matrixA is multiplied on the left by the matrixPk such thatPk =P(i, k), else we setPk =I. Addingβ times rowj to rowi is achieved by multiplyingA on the left by the elementary matrix,

Ei,j;β =I +βei j,

where 1 ifk =i andl =j(ei j)k l = 0 ifk =i orl =j,

i.e.,

ë_1 ö ë_1 ö

ì 1 ÷ ì 1 ÷ ì_1 ÷ ì1 β ÷ ì ÷ ì ÷ ì ÷ ì ÷ ì 1 ÷ ì 1 ÷ ì... ÷ ì... ÷

E

i,j

;

β

=

ì ÷ or Ei,j;β =ì ÷

ì ÷ ì ÷. ì_1 ÷ ì_1 ÷ ì ÷ ì ÷ ìβ 1 ÷ ì_1 ÷ ì ÷ ì ÷ ì_1 ÷ ì_1 ÷

í ø í ø 1 1

On the left, i > j, and on the right,i < j. Observe that the inverse ofEi,j;β =I +βei j is Ei,j;−β =I−βei j and that det(Ei,j;β) = 1. Therefore, during step 3 (the elimination step), the matrixA is multiplied on the left by a product,Ek, of matrices of the formEi,k;βi,k, with i > k.

Consequently, we see that

Ak+1 =EkPkAk, and then

Ak =Ek−1Pk−1· · ·E_1P_1A.

This justifies the claim made earlier, thatAk =MkA for some invertible matrixMk; we can pick

Mk =Ek−1Pk−1· · ·E_1P_1,

a product of invertible matrices.

The fact that det(P(i, k)) =−1 and that det(Ei,j;β) = 1 implies immediately the fact claimed above: We always have

det(Ak) =±det(A).

Furthermore, since Ak =Ek−1Pk−1· · ·E_1P_1A and since Gaussian elimination stops fork =n, the matrix

An =En−1Pn−1· · ·E_2P_2E_1P_1A

is upper-triangular. Also note that if we letM =En−1Pn−1· · ·E_2P_2E_1P_1, then det(M) =±1, and

det(A) =±det(An). The matricesP(i, k) andEi,j;β are called elementary matrices. We can summarize the above discussion in the following theorem:

Theorem 6.1. (Gaussian Elimination) LetA be ann×n matrix (invertible or not). Then there is some invertible matrix,M, so thatU =MA is upper-triangular. The pivots are all nonzero iffA is invertible.

Proof. We already proved the theorem whenA is invertible, as well as the last assertion. Now,A is singular iff some pivot is zero, say at stagek of the elimination. If so, we must haveaki k = 0, fori =k, . . . , n; but in this case,Ak+1 =Ak and we may pickPk =Ek =I.

Remark: Obviously, the matrixM can be computed as

M =En−1Pn−1· · ·E_2P_2E_1P_1,

but this expression is of no use. Indeed, what we need is M−1; when no permutations are needed, it turns out thatM−1 can be obtained immediately from the matricesEk’s, in fact, from their inverses, and no multiplications are necessary.

Remark: Instead of looking for an invertible matrix,M, so thatMA is upper-triangular, we can look for an invertible matrix,M, so thatMA is a diagonal matrix. Only a simple change to Gaussian elimination is needed. At every stage,k, after the pivot has been found and pivoting been performed, if necessary, in addition to adding suitable multiples of the kth row to the rows below rowk in order to zero the entries in columnk fori =k+ 1, . . . , n, also add suitable multiples of thekth row to the rows above rowk in order to zero the entries in columnk fori = 1, . . . , k− 1. Such steps are also achieved by multiplying on the left by elementary matricesEi,k;βi,k, except thati < k, so that these matrices are not lower-triangular matrices. Nevertheless, at the end of the process, we find thatAn =MA, is a diagonal matrix. This method is called the Gauss-Jordan factorization. Because it is more expansive than Gaussian elimination, this method is not used much in practice. However, Gauss-Jordan factorization can be used to compute the inverse of a matrix,A. Indeed, we find thejth column ofA−1 by solving the systemAx(j) =ej (whereej is thejth canonical basis vector of Rn). By applying Gauss-Jordan, we are led to a system of the form Djx(j) =Mjej, whereDj is a diagonal matrix, and we can immediately computex(j).

It remains to discuss the choice of the pivot, and also conditions that guarantee that no permutations are needed during the Gaussian elimination process. We begin by stating a necessary and sufficient condition for an invertible matrix to have anLU-factorization (i.e., Gaussian elimination does not require pivoting).

We say that an invertible matrix, A, has anLU-factorization if it can be written as A =LU, whereU is upper-triangular invertible andL is lower-triangular, withLi i = 1 for i = 1, . . . , n.

A lower-triangular matrix with diagonal entries equal to 1 is called a unit lower-triangular matrix. Given ann×n matrix,A = (ai j), for anyk, with 1≤k≤n, letA[1..k,1..k] denote the submatrix ofA whose entries areai j, where 1≤i, j≤k.

Proposition 6.2. LetA be an invertiblen×n-matrix. Then,A, has anLU-factorization, A =LU, iff every matrixA[1..k,1..k] is invertible fork = 1, . . . , n. Furthermore, whenA has anLU-factorization, we have

det(A[1..k,1..k]) =π_1π2· · ·πk, k = 1, . . . , n,

whereπk is the pivot obtained afterk−1 elimination steps. Therefore, thekth pivot is given

by ña_11 = det(A[1..1,1..1]) ifk = 1

πk =ò det(A[1..k,1..k])ifk = 2, . . . , n.ódet(A[1..k− 1,1..k− 1])

Proof. First, assume thatA =LU is anLU-factorization ofA. We can write

A = A[1..k,1..k] A2 = L_1 0 U_1 Q= L_1U_1 L_1Q ,A3 A4 P L4 0 U4 P U1 P Q +L4U4 whereL_1, L_4 are unit lower-triangular andU_1, U_4 are upper-triangular. Thus,

A[1..k,1..k] =L_1U_1,

and since U is invertible,U_1 is also invertible (the determinant ofU is the product of the diagonal entries inU, which is the product of the diagonal entries inU_1 andU_4). AsL_1 is invertible (since its diagonal entries are equal to 1), we see thatA[1..k,1..k] is invertible for k = 1, . . . , n.

Conversely, assume that A[1..k,1..k] is invertible, fork = 1, . . . , n. We just need to show that Gaussian elimination does not need pivoting. We prove by induction onk that thekth step does not need pivoting. This holds fork = 1, sinceA[1..1,1..1] = (a1 1), so,a1 1 = 0. Assume that no pivoting was necessary for the firstk−1 steps (2≤k≤n−1). In this case, we have

Ek−1· · ·E_2E_1A =Ak,

where L = Ek−1· · ·E_2E_1 is a unit lower-triangular matrix and Ak[1..k,1..k] is uppertriangular, so thatLA =Ak can be written as

L_1 0 A[1..k,1..k] A2 = U_1 B2 ,P L4 A3 A4 0 B4

whereL_1 is unit lower-triangular andU_1 is upper-triangular. But then, L_1A[1..k,1..k]) =U_1,

where L_1 is invertible (in fact, det(L_1) = 1), and since by hypothesisA[1..k,1..k] is invertible, U_1 is also invertible, which implies that (U_1)kk = 0, sinceU_1 is upper-triangular. Therefore, no pivoting is needed in stepk, establishing the induction step. Since det(L_1) = 1, we also have

det(U_1) = det(L_1A[1..k,1..k]) = det(L_1) det(A[1..k,1..k]) = det(A[1..k,1..k]), and sinceU_1 is upper-triangular and has the pivotsπ_1, . . . , πk on its diagonal, we get

det(A[1..k,1..k]) =π_1π2· · ·πk, k = 1, . . . , n, as claimed.

Remark: The use of determinants in the first part of the proof of Proposition 6.2 can be avoided if we use the fact that a triangular matrix is invertible iff all its diagonal entries are nonzero.

Corollary 6.3. (LU-Factorization) LetA be an invertiblen×n-matrix. If every matrix A[1..k,1..k] is invertible fork = 1, . . . , n , then Gaussian elimination requires no pivoting and yields anLU-factorization,A =LU.

Proof. We proved in Proposition 6.2 that in this case Gaussian elimination requires no pivoting. Then, since every elementary matrixEi,k;β is lower-triangular (since we always arrange that the pivot,πk, occurs above the rows that it operates on), sinceE−1 =Ei,k;−βi,k;β and theEks are products ofEi,k;βi,k’s, from

En−1· · ·E_2E_1A =U,

whereU is an upper-triangular matrix, we get

A =LU,

whereL = E− 1 E−1 E−1 is a lower-triangular matrix. Furthermore, as the diagonal

1 2 · · ·n−1

entries of eachEi,k;β are 1, the diagonal entries of eachEk are also 1.

The reader should verify that the example below is indeed anLU-factorization.

ë 2 1 1 0ö ë1 0 0 0öë2 1 1 0ö ì4 3 3 1÷ = ì2 1 0 0÷ì0 1 1 1÷

ì .í8 7 9 5÷ ì4 3 1 0÷ì0 0 2 2÷ ø í øí ø 6 7 9 8 3 4 1 1 0 0 0 2

One of the main reasons why the existence of an LU-factorization for a matrix,A, is interesting is that if we need to solve several linear systems,Ax =b, corresponding to the same matrix,A, we can do this cheaply by solving the two triangular systems

Lw =b, and Ux =w.

There is a certain asymmetry in the LU-decompositionA =LU of an invertible matrixA. Indeed, the diagonal entries ofL are all 1, but this is generally false forU. This asymmetry can be eliminated as follows: if

D = diag(u_11, u_22, . . . , unn)

is the diagonal matrix consisting of the diagonal entries inU (the pivots), then we if let U =D−1U, we can write

A =LDU ,

where L is lower- triangular,U is upper-triangular, all diagonal entries of bothL andU are 1, andD is a diagonal matrix of pivots. Such a decomposition is called anLDU-factorization. We will see shortly than ifA is symmetric, thenU =L .

As we will see a bit later, symmetric positive definite matrices satisfy the condition of Proposition 6.2. Therefore, linear systems involving symmetric positive definite matrices can be solved by Gaussian elimination without pivoting. Actually, it is possible to do better: This is the Cholesky factorization.

The following easy proposition shows that, in principle, A can be premultiplied by some permutation matrix, P, so thatP A can be converted to upper-triangular form without using any pivoting. Permutations are discussed in some detail in Section 20.3, but for now we just need their definition. A permutation matrix is a square matrix that has a single 1 in every row and every column and zeros everywhere else. It is shown in Section 20.3 that every permutation matrix is a product of transposition matrices (theP(i, k)s), and thatP is invertible with inverseP .

Proposition 6.4. LetA be an invertiblen×n-matrix. Then, there is some permutation matrix,P, so thatP A[1..k,1..k] is invertible for k = 1, . . . , n.

Proof. The casen = 1 is trivial, and so is the casen = 2 (we swap the rows if necessary). If n≥ 3, we proceed by induction. SinceA is invertible, its columns are linearly independent; so, in particular, its firstn−1 columns are also linearly independent. Delete the last column ofA. Since the remainingn−1 columns are linearly independent, there are alson−1 linearly independent rows in the correspondingn× (n− 1) matrix. Thus, there is a permutation of thesen rows so that the (n− 1)× (n− 1) matrix consisting of the firstn− 1 rows is invertible. But, then, there is a corresponding permutation matrix,P_1, so that the firstn−1 rows and columns ofP_1A form an invertible matrix,A . Applying the induction hypothesis to the (n− 1)× (n− 1) matrix,A , we see that there some permutation matrixP_2 (leaving thenth row fixed), so thatP_2P_1A[1..k,1..k] is invertible, fork = 1, . . . , n− 1. SinceA is invertible in the first place andP_1 andP_2 are invertible,P_1P_2A is also invertible, and we are done.

Remark: One can also prove Proposition 6.4 using a clever reordering of the Gaussian elimination steps suggested by Trefethen and Bau [106] (Lecture 21). Indeed, we know that ifA is invertible, then there are permutation matrices,Pi, and products of elementary matrices,Ei, so that

An =En−1Pn−1· · ·E_2P_2E_1P_1A,

where U =An is upper-triangular. For example, whenn = 4, we haveE_3P_3E_2P_2E_1P_1A =U. We can define new matricesE_1, E_2, E_3 which are still products of elementary matrices so that we have

E_3E_2E_1P_3P_2P_1A =U.

Indeed, if we letE_3 =E_3,E_2 =P_3E_2P_3−1, andE_1 =P_3P_2E_1P_2−1 P_3−1, we easily verify that eachEk is a product of elementary matrices and that

E_3E_2E_1P_3P_2P_1 =E_3(P_3E_2P3−1)(P_3P_2E_1P2−1 P3−1)P_3P_2P_1 =E_3P_3E_2P_2E_1P_1.

It can also be proved thatE_1, E_2, E_3 are lower triangular (see Theorem 6.5). In general, we let

Ek =Pn−1· · ·Pk+1EkPk−1 P−1 ,+1· · · n−1 and we have En−1· · ·E_1Pn−1· · ·P_1A =U,

where eachEj is a lower triangular matrix (see Theorem 6.5).

Using the above idea, we can prove the theorem below which also shows how to compute P, L andU using a simple adaptation of Gaussian elimination. We are not aware of a detailed proof of Theorem 6.5 in the standard texts. Although Golub and Van Loan [47] state a version of this theorem as their Theorem 3.1.4, they say that “The proof is a messy subscripting argument.” Meyer [77] also provides a sketch of proof (see the end of Section 3.10). In view of this situation, we offer a complete proof. It does involve a lot of subscripts and superscripts but, in our opinion, it contains some interesting techniques that go far beyond symbol manipulation.

Theorem 6.5. For every invertiblen×n-matrixA, the following hold:

(1) There is some permutation matrix, P, some upper-triangular matrix, U, and some unit lower-triangular matrix,L, so thatP A =LU (recall,Li i = 1 fori = 1, . . . , n). Furthermore, ifP =I, thenL andU are unique and they are produced as a result of Gaussian elimination without pivoting.

(2) If En−1. . . E_1A =U is the result of Gaussian elimination without pivoting, write as usualAk =Ek−1. . . E_1A (withAk = (akij)), and let ik =akik/akkk, with 1≤k≤n− 1 andk + 1≤i≤n. Then

ë 1 0 0 · · ·0ö

ì_21 1 0 · · · 0÷

ì ÷

L

=

ì 31 32 1 · · · 0÷

ì ÷,

ì. . . ... 0÷

í_1ø

n1 n2 n3 · · ·

where thekth column ofL is thekth column ofE−1, fork = 1, . . . , n− 1.

k

(3) If En−1Pn−1· · ·E_1P_1A =U is the result of Gaussian elimination with some pivoting, writeAk =Ek−1Pk−1· · ·E_1P_1A, and defineEkj , with 1≤j≤n−1 andj≤k≤n−1, such that, forj = 1, . . . , n− 2,

Ej =Ejj

Ekj =PkEk 1 k, fork =j + 1, . . . , n− 1,j P

and En−1 =En−1.n−1 Then,

Ekj =PkPk−1· · ·Pj+1EjPj+1· · ·Pk−1Pk U =En−1 En 1 n−1· · ·P_1A,n−1· · · 1 P

and if we set

P =Pn−1 · · ·P_1

L = (En−1)−1 (En−1)−1,1 · · · n−1

then P A =LU. Furthermore,

(Ekj)−1 =I +Ekj, 1≤j≤n− 1, j≤k≤n− 1, whereEkj is a lower triangular matrix of the form

ë0 · · ·0 0 · · ·0ö

ì.... . .. .÷ ì ÷

ì0 · · ·kj+1j 0 · · ·0÷

kj

=

ì 0 0 ÷,E ì0 · · ·. . · · ·0÷

ì ÷ ì. . ... .÷

í ø 0 · · ·knj 0 · · ·0 and E

kj

=

P

k

k

Ej ,1 1≤j≤n− 2, j + 1≤k≤n− 1,

where Pk =I or elsePk =P(k, i) for somei such thatk + 1≤i≤n; ifPk =I, this means that (Ekj)−1 is obtained from (Ek−1)−1 by permuting the entries on rowi andj

k in columnj. Because the matrices (Ekj)−1 are all lower triangular, the matrixL is also lower triangular.

In order to findL, define lower triangular matrices Λk of the form

ë 0 0 0 0 0 · · · · · ·0ö

ì

÷

ì

λ

k

0

0

0

0

.

. 0

÷

ì 21 ... 0 0 . . 0÷

ì λk

÷

ì

λ

k ÷ ì 31 32... 0 0 . . .÷

Λ

k

=

ì . . ÷

ìλkk+11 λkk+12 · · · λkk+1k 0 · · · · · ·0÷

ì ÷ ì ÷

ìλkk+21 λkk+22 · · · λkk+2k 0 ... · · · 0÷

ì ÷ ì . . ... . . . ... .÷

í ø λkn1 λkn2 · · · λknk 0 · · · · · ·0 to assemble the columns ofL iteratively as follows: let

( kk+1k, . . . , knk) be the lastn−k elements of thekth column ofE−1, and define Λk inductively by setting

k

ë 0 0 · · ·0ö

ì 1 0 · · · 0÷

Λ

1

=

ì 21 ÷

ì ,

í

.

.

...

.

÷

ø1 0 · · ·0n1

then fork = 2, . . . , n− 1, define

Λk =PkΛk−1, and

ë 0 0 0 0 0 · · · · · ·0ö

ì

÷

ì ÷

ì

λ

k

−

1

0

0

0

0

. . 0

÷

ì 21 ... 0 0 . . 0÷

ìλk−1 λk−1 ÷ ì31 32 ... ÷

0 0 . . .÷,k = (I + Λk)E−1 I =ì . . ÷Λ ì

k −ìλk−1 λk−1 λk −1

ì k k−1 ÷0 · · · · · ·0÷ ì k1 k2 · · · λk−1 kk+1k ... · · · 0÷ ìλk−1 λk−1 ÷

ì k+11 k+12 · · · k+1 k−1 ÷ ì . . ... . . . ... .÷

í ø

λ k−1 λk−1 λk−1 knk · · · · · ·0n1 n2 · · ·n k −1

withPk =I orPk =P(k, i) for somei > k. This means that in assemblingL, rowk and rowi of Λk−1 need to be permuted when a pivoting step permuting rowk and row i ofAk is required. Then

I

+ Λ

k

= (

E

k)−1 (Ekk)−1 1 · · ·Λk =k

E^1· · · Ekk,

fork = 1, . . . , n− 1, and therefore L =I + Λn−1.

Proof. (1) The only part that has not been proved is the uniqueness part (whenP =I). Assume thatA is invertible and thatA =L_1U_1 =L_2U_2, withL_1, L_2 unit lower-triangular andU_1, U_2 upper-triangular. Then, we have

L− 11 L_1 =U_2U1 .

2

However, it is obvious thatL−1 is lower-triangular and thatU1−1 is upper-triangular, and so,

L

−

1

2

L_1 is lower-triangular andU_2U_1−1 is upper-triangular. Since the diagonal entries of_2

L^1 =I, that is,U_1 =U_2, and so,1 andL_2 are 1, the above equality is only possible ifU_2U1−

L_1 =L_2.

(2) WhenP = I, we haveL = E1−1 E−1 E−1 , whereEk is the product ofn−k2 · · · n−1

elementary matrices of the formEi,k;− i, whereEi,k;− i subtracts i times rowk from rowi, with ik =akik/akkk, 1≤k≤n− 1, andk + 1≤i≤n. Then, it is immediately verified that

ë1 · · ·0 0 · · ·0ö

ì. ... . . . .÷

ì ÷

ì0 · · ·1 0 · · ·0÷

E

k

=

ì ÷

ì ,

ì

0

· · · −

k

+1

k

1

· · ·

0

÷

÷

ì. . . . ... .÷

í ø

0 · · · − nk 0 · · ·1

and that ë1 · · ·0 0 · · ·0ö

ì. ... . . . .÷ ì ÷

ì0 · · ·k+1k 1 · · ·0÷ E

k

−

1

=

ì 1 0 ÷ ì . ì

0

· · ·

. . · · ·0÷

÷ ì ÷

í. . ... .ø 0 · · ·nk 0 · · ·1 If we defineLk byë 1 0 0 0 0 . 0ö

ì 1 0 0 0 . 0÷

ì

÷

ì

21

÷

ì ... 0 0 . 0÷

ì

÷

ì

31 32÷

Lk =ì ...1 0 . 0÷ ,

ì . . ÷

ì 1 · · ·0÷

ì k+11 k+12 · · · k+1k ÷ ì ÷

í . . ... . 0 . 0ø n1 n2 · · ·1 · · ·1nk 0 fork = 1, . . . , n− 1, we easily check thatL_1 =E_1 , and that L^1 2≤k≤n− 1,k =Lk−1E− ,k

because multiplication on the right by Ek−1 adds i times columni to columnk (of the matrix Lk−1) withi > k, and columni ofLk−1 has only the nonzero entry 1 as itsith element. Since

L

k

=

E

−

1 E1

k , 1≤k≤n− 1,1 · · ·

we conclude thatL =Ln−1, proving our claim about the shape ofL.

(3) First, we prove by induction on k that

A kk· · ·EkPk· · ·P_1A, k = 1, . . . , n− 2.k+1 =E_1

Fork = 1, we haveA_2 =E_1P_1A =E^1P_1A, sinceE^1 =E_1, so our assertion holds trivially.1 1

Now, ifk≥ 2, Ak+1 =EkPkAk and, by the induction hypothesis,

Ak =Ek−1 Ek 1 k 1 k−1· · ·P_1A.k−1· · · 2 E1 P

BecausePk is either the identity or a transposition,P^2 =I, so by inserting occurrences ofk

PkPk as indicated below we can write

Ak+1 =EkPkAk

=

E

k

P

k

E

k−1Ek 1 k 1

2 E1 P

=EkPkEk

k−1· · ·P1Ak−1· · ·Pk)· · ·(PkPk)Ek−1(PkPk)Ek−1(PkPk)Pk−1· · ·P_1Ak−1(Pk 2 1

=

E

k

(

P

k

E

−1

k−1Pk)· · ·(PkEk 1 k)(PkEk 1 k)PkPk−1· · ·P_1A.k−1 2 P 1 P

Observe that Pk has been “moved” to the right of the elimination steps. However, by definition,

Ekj =PkEk 1 k, j = 1, . . . , k− 1j P

Ekk =Ek,

so we get Ak+1 =EkkEkk−1· · ·EkEkPk· · ·P_1A,2 1

establishing the induction hypothesis. Fork =n− 2, we get U =An−1 En 1 n−1· · ·P_1A,n−1 =En−1· · · 1 P

as claimed, and the factorizationP A =LU with P =Pn−1 · · ·P_1

L

= (

E

n−1)−1 (En−1)−1

1 · · ·n−1 is clear,

Since forj = 1, . . . , n− 2, we haveEj =Ej,j

Ekj =PkEk 1 k, k =j + 1, . . . , n− 1,j P

sinceEn−1 =En−1, andPk−1 =Pk, we get (Ej)−1 =Ej−1 forj = 1, . . . , n− 1, and forn−1 j

j = 1, . . . , n− 2, we have

(Ekj)−1 =Pk(Ek−1)−1Pk, k =j + 1, . . . , n− 1.j

Since (

E

k−1)−1 =I +k E

j − 1 j

andPk =P(k, i) is a transposition,P^2 =I, so we getk

(

E

kj

)

−

1 =Pk(Ek−1)−1Pk =Pk(I +k k 1 k 1 Ej−1)Pk =P^2 +PkEj Pk =I +PkEj Pk.j k

Therfore, we have

(Ek 1kj)−1 =I +PkEj Pk, 1≤j≤n− 2, j + 1≤k≤n− 1.

We prove forj = 1, . . . , n− 1, that fork =j, . . . , n− 1, eachEkj is a lower triagular matrix of the formë0 · · · 0 0 · · · 0ö

ì. ... . . . .÷

ì ÷

ì0 · · ·kj+1j 0 · · ·0÷

kj

= ì 0 0 ÷ ,E ì0 · · ·. . · · ·0÷

ì ÷

ì ÷

í. . ... .ø

0 · · ·knj 0 · · ·0

and that

E

kj

=

P

k

k

Ej ,1 1≤j≤n− 2, j + 1≤k≤n− 1,

withPk =I orPk =P(k, i) for somei such thatk + 1≤i≤n.

For eachj (1≤ j

1

≤n− 1) we proceed by induction onk =j, . . . , n− 1. Since (Ej)−1 =j Ej−1 and sinceEj− is of the above form, the base case holds.

For the induction step, we only need to consider the case wherePk =P(k, i) is a transposition, since the case where

P

k

=

I

is trivial. We have to figure out what P

k k 1

E

j Pk =

P

(

k, i

)

k 1

Ej P(k, i) is. However, since

ë0 · · ·0 0 · · ·0ö

ì.... . .. .÷ ì ÷

ì0 · · ·k−1 0 · · ·0÷ k−1 = ì 0 0 ÷,Ej ì0 · · ·j+1j · · ·0÷

ì ÷ ì ÷

í. . . . ... .ø 0

k

· · ·nj−1 0 · · ·0 and because

k

+ 1

≤

i

≤

n

and

j

≤

k

−

1, multiplying

k

E

j −1 on the right byP(k, i) will permute columnsi andk, which are columns of zeros, so

k 1 (k, i) =P(k, i)Ej ,1P(k, i)k

Ej P

and thus, (

E

kj

)

−

1 =I +P(k, i)k Ej ,1 which shows that E

kj

=

P

(

k, i

)

k

Ej .1 We also know that multiplying (

k

E j−1)−1 on the left byP(k, i) will permute rows i and k, which shows thatEkj has the desired form, as claimed. Since allEkj are strictly lower triangular, all (Ekj)−1 =I +Ekj are lower triangular, so the product

L

= (

E

n−1)−1 (En−1)−1

1 · · ·n−1 is also lower triangular.

From the beginning of part (3), we know that L = (En−1)−1 (En−1)−1.1 · · · n−1 We prove by induction onk that

I

+ Λ

k

= (

E

k)−1 (Ekk)−1 1 · · ·Λk =k

E^1· · · Ekk,

fork = 1, . . . , n− 1.

Ifk = 1, we haveE^1 =E_1 and_1

ë 1 0 · · ·0ö

ì

1

21 ÷ ì

1

· · ·

0

÷

E_1 =ì−. . ... .÷. í_1 ø n1 0 · · ·1−

We getë 1 0 · · · 0ö

ì 1 1 · · ·0÷

(

E

1

1

)−1 =ì 21 ÷ =I + Λ_1,ì . . ... .÷

í1n1 0 ø

· · ·1

Since (

E

− 1 −1 =I +1 1

E1, we also get Λ_1 =E1, and the base step holds.1 )

Since (Ekj)−1 =I +Ekj with

ë0 · · ·0 0 · · ·0ö

ì. ... . . . .÷ ì ÷

ì0 · · ·kj+1j 0 · · ·0÷

kj

= ì 0 0 ÷ ,E ì0 · · ·. . · · ·0÷

ì ÷ ì. . ... .÷

í ø 0 · · ·knj 0 · · ·0 as in part (2) for the computation involving the products ofLk’s, we get

(Ekk−1)−1 (Ek−1)−1 =I +E1−1 k−1, 2≤k≤n. (∗)1 · · · k−1 · · · Ek−1

Similarly, from the fact that

k 1 k

Ej P(k, i) =Ej−1 ifi≥k + 1 andj≤k− 1 and since

(

E

kj

)

−

1 =I +Pk k

Ej ,1 1≤j≤n− 2, j + 1≤k≤n− 1,

we get (

E

k)−1 (Ekk−1)−1 =I +Pk k

E1−1 k−1, 2≤k≤n− 1. (∗∗)1 · · · · · · Ek−1

By the induction hypothesis,

I + Λk−1 = (Ek−1)−1 (Ek−1)−1,1 · · · k−1

and from (∗), we get Λ

k

−

1

=

k

E1−1 k−1. · · · Ek−1 Using (∗∗), we deduce that

(Ek)−1 (Ekk−1)−1 =I +PkΛk−1.1 · · ·

SinceEkk =Ek, we obtain

(

E

k)−1 (Ekk−1)−1(Ekk)−1 = (I +PkΛk−1)E^1

k .1· · ·

However, by definition,

I

+ Λ

k

= (

I

+

P

k

Λ

k

−

1

) E

− 1 ,

k

which proves that I + Λk = (Ek)−1 (Ekk−1)−1(Ekk)−1, (†)1 · · ·

and finishes the induction step for the proof of this formula.

If we apply equation (∗) again withk + 1 in place ofk, we have

(

E

k)−1 (Ekk)−1 =I +k E^1· · · Ekk,1 · · ·

and together with (†), we obtain,

k

Λ k =E^1· · · Ekk,

also finishing the induction step for the proof of this formula. Fork =n−1 in (†), we obtain the desired equation:L =I + Λn−1.

Part (3) of Theorem 6.5 shows the remarkable fact that in assembling the matrix L while performing Gaussian elimination with pivoting, the only change to the algorithm is to make the same transposition on the rows ofL (really Λk, since the one’s are not altered) that we make on the rows ofA (reallyAk) during a pivoting step involving rowk and rowi. We can also assembleP by starting with the identity matrix and applying toP the same row transpositions that we apply toA and Λ. Here is an example illustrating this method. Consider the matrixë 1 2 −3 4 ö

A

=

ì ÷

ì 4 812 −8÷.í 2 3 2 1 ø

−3 −1 1 −4

We setP_0 =I_4, and we can also set Λ_0 = 0. The first step is to permute row 1 and row 2, using the pivot 4. We also apply this permutation toP_0:

ë 4 8 12 −8ö ë0 1 0 0ö

ì 1 2 −3 4 ÷ P_1 =ì1 0 0 0÷.1 = ì2 3 2 1 ÷ ì0 0 1 0÷Aí ø í ø

−3 −1 1 −4 0 0 0 1

Next, we subtract 1/4 times row 1 from row 2, 1/2 times row 1 from row 3, and add 3/4 times row 1 to row 4, and start assembling Λ:

ë4 8 12 −8ö ë 0 0 0 0ö ë0 1 0 0ö ì0 0 −6 6 ÷ Λ_1 =ì 1/4 0 0 0÷ P_1 =ì1 0 0 0÷.2 = ì0 −1 −4 5 ÷ ì1/2 0 0 0÷ ì0 0 1 0÷Aí ø í ø í ø 05 10 −10−3/4 0 0 0 0 0 0 1

Next we permute row 2 and row 4, using the pivot 5. We also apply this permutation to Λ andP:

ë4 8 12 − 8ö ë 0 0 0 0ö ë0 1 0 0ö ì0 5 10 −10÷ Λ2 = ì 3/4 0 0 0÷ P2 = ì0 0 0 1÷.3 = ì0 −1 −4 5 ÷ ì−/2 0 0 0÷ ì0 0 1 0÷A í ø í 1 ø í ø

00 −6 6 1/4 0 0 0 1 0 0 0

Next we add 1/5 times row 2 to row 3, and update Λ_2:

ë4 8 12 − 8ö ë 0 0 0 0ö ë0 1 0 0ö ì0 5 10 −10÷Λ2 = ì 3/4 0 0 0÷ P2 = ì0 0 0 1÷.3 = ì0 0 −2 3 ÷ ì−/2 −1/5 0 0÷ ì0 0 1 0÷A í ø í 1 ø í ø

0 0 −6 6 1/4 0 0 0 1 0 0 0

Next we permute row 3 and row 4, using the pivot−6. We also apply this permutation to Λ andP:

ë4 8 12 − 8ö ë 0 0 0 0ö ë0 1 0 0ö ì0 5 10 −10÷Λ3 = ì 3/4 0 0 0÷ P3 = ì0 0 0 1÷.4 = ì0 0 −6 6 ÷ ì−/4 0 0 0÷ ì1 0 0 0÷A í ø í 1 ø í ø

0 0 −2 3 1/2 −1/5 0 0 0 0 1 0 Finally, we subtract 1/3 times row 3 from row 4, and update Λ_3:

ë4 8 12 − 8ö ë 0 0 0 0ö ë0 1 0 0ö ì0 5 10 −10÷ Λ3 = ì 3/4 0 0 0÷ P3 = ì0 0 0 1÷.4 = ì0 0 −6 6 ÷ ì−/4 0 0 0÷ ì1 0 0 0÷A í ø í 1 ø í ø

0 0_0 1 1/2 −1/5 1/3 0 0 0 1 0 Consequently, adding the identity to Λ_3, we obtain

ë 1 0 0 0ö ë4 8 12 − 8ö ë0 1 0 0ö ì 3/4 1 0 0÷, U =ì0 5 10 −10÷, P =ì0 0 0 1÷.í−/4 0 1 0÷ ì0 0 −6 6 ÷ ì1 0 0 0÷L =ì

1 ø í ø í ø 1/2 −1/5 1/3 1 0 0_0 1 0 0 1 0 We check that ë0 1 0 0öë 1 2 −3 4ö ë 4 8 12 −8ö ì0 0 0 1÷ì 4 8 12 −8÷ =ì 3 −1 1 −4÷,í1 0 0 0÷ì 2 3 2 1 ø í−÷P A =ì ÷ ì

øí1 2 −3 4 ø 0 0 1 0−3 −1 1 −4 2 3_2 1 and that ë 1 0 0 0öë4 8 12 − 8ö ë 4 8 12 −8ö ì 3/4 1 0 0÷ì0 5 10 −10÷ =ì 3 −1 1 −4÷ =P A.í−/4 0 1 0÷ì0 0 −6 6 ÷ ì−÷LU =ì

1 øí ø í1 2 −3 4 ø 1/2 −1/5 1/3 1 0 0_0 1 2 3_2 1

Note that if one willing to overwrite the lower triangular part of the evolving matrix A, one can store the evolving Λ there, since these entries will eventually be zero anyway! There is also no need to save explicitly the permutation matrixP. One could instead record the permutation steps in an extra column (record the vector (π(1), . . . , π(n)) corresponding to the permutationπ applied to the rows). We let the reader write such a bold and spaceefficient version ofLU-decomposition!

As a corollary of Theorem 6.5(1), we can show the following result.

Proposition 6.6. If an invertible symmetric matrixA has anLU-decomposition, thenA has a factorization of the form

A =LDL ,

whereL is a lower-triangular matrix whose diagonal entries are equal to 1, and whereD consists of the pivots. Furthermore, such a decomposition is unique.

Proof. IfA has anLU-factorization, then it has anLDU factorization

A =LDU, whereL is lower-triangular,U is upper-triangular, and the diagonal entries of bothL and U are equal to 1. SinceA is symmetric, we have

LDU =A =A =U DL ,

with U lower-triangular andDL upper-triangular. By the uniqueness ofLU-factorization (part (1) of Theorem 6.5), we must haveL =U (andDU =DL ), thusU =L , as claimed.

Remark: It can be shown that Gaussian elimination + back-substitution requiresn^3/3 + O(n^2) additions,n^3/3 +O(n^2) multiplications andn^2/2 +O(n) divisions.

Let us now briefly comment on the choice of a pivot. Although theoretically, any pivot can be chosen, the possibility of roundoff errors implies that it is not a good idea to pick very small pivots. The following example illustrates this point. Consider the linear system

10−4x + y = 1 x + y = 2.

Since 10−4 is nonzero, it can be taken as pivot, and we get 10−4x + y = 1 (1− 10^4)y = 2− 10^4. Thus, the exact solution is

x = 104 = 104 2.104 1, y 4 − 1− 10 −

However, if roundoff takes place on the fourth digit, then 10^4 1 = 9999 and 10^4 2 = 9998− − will be rounded off both to 9990, and then, the solution isx = 0 andy = 1, very far from the exact solution wherex≈ 1 andy≈ 1. The problem is that we picked a very small pivot. If instead we permute the equations, the pivot is 1, and after elimination, we get the system

x + y = 2 (1− 10−4)y = 1− 2× 10−4.

This time, 1− 10−4 = 0.9999 and 1− 2× 10−4 = 0.9998 are rounded off to 0.999 and the solution isx = 1, y = 1, much closer to the exact solution.

To remedy this problem, one may use the strategy of partial pivoting. This consists of choosing during stepk (1≤k≤n− 1) one of the entriesaki k such that

|

aki k| = max|akp k|.

k≤p≤n

By maximizing the value of the pivot, we avoid dividing by undesirably small pivots. Remark: A matrix,A, is called strictly column diagonally dominant iff

n

|

aj j|>|ai j|, forj = 1, . . . , n

i=1, i=j

(resp. strictly row diagonally dominant iff

n

|

ai i|>|ai j|, fori = 1, . . . , n.)

j=1, j=i

It has been known for a long time (before 1900, say by Hadamard) that if a matrix, A, is strictly column diagonally dominant (resp. strictly row diagonally dominant), then it is invertible. (This is a good exercise, try it!) It can also be shown that ifA is strictly column diagonally dominant, then Gaussian elimination with partial pivoting does not actually require pivoting (See Problem 21.6 in Trefethen and Bau [106], or Question 2.19 in Demmel [25]).

Another strategy, called complete pivoting, consists in choosing some entryaki j, where k≤i, j≤n, such that

|

aki j| =kmaxn|akp q|.

≤p,q≤

However, in this method, if the chosen pivot is not in column k, it is also necessary to permute columns. This is achieved by multiplying on the right by a permutation matrix. However, complete pivoting tends to be too expensive in practice, and partial pivoting is the method of choice.

A special case where theLU-factorization is particularly efficient is the case of tridiagonal matrices, which we now consider.

6.3 Gaussian Elimination of Tridiagonal Matrices

Consider the tridiagonal matrix

ëb_1 c_1 ö

ìa_2 b_2 c_2 ÷

ì ÷

ì a_3 b_3 c_3 ÷

ì ... ... ...÷

A

=

ì ÷

ì ÷

÷

ì

.

ì

÷

í

a

n

−

2

b

n

−

2

c

n

−

2

÷

ì an−1 bn−1 cn−1øan bn

Define the sequence Proposition 6.7. IfA is the tridiagonal matrix above, thenδk = det(A[1..k,1..k]), for k = 1, . . . , n.

δ_0 = 1, δ_1 =b_1, δk =bkδk−1−akck−1δk−2, 2≤k≤n.

Proof. By expanding det(A[1..k,1..k]) with respect to its last row, the proposition follows by induction onk.

Theorem 6.8. IfA is the tridiagonal matrix above andδk = 0 fork = 1, . . . , n, thenA has the followingLU-factorization:

ö

÷ ì ÷ì ÷ ì ÷ì

δ0_1 ÷ìδ0 δ2 c2 ÷

ì ÷ì δ ÷ ì a3 ÷ì

ë 1 öëδ1

ìc1 ÷ ìa2δ1

÷ ì

δ1 1 ÷ì^1 δ_3

÷ ì ÷ì

÷ ì ÷ ì ÷ ì ÷ì ÷

÷ì

íí

c

3

÷ A = ì δ2... ... ÷ì δ2 ... ... ÷ . ì ÷ì ÷ ì δn−3_1 ÷ì δn−1 cn−1÷ì an−1δn−2 δn−2 1øì δn−2 δ ÷

n øanδn−1 δn−1

Proof. Sinceδk = det(A[1..k,1..k]) = 0 fork = 1, . . . , n, by Theorem 6.5 (and Proposition 6.2), we know thatA has a uniqueLU-factorization. Therefore, it suffices to check that the proposed factorization works. We easily check that

(LU)k k+1 = ck, 1≤k≤n− 1

(LU)k k−1 = ak, 2≤k≤n

(LU)k l = 0, |k−l| ≥ 2

(LU)1 1 = δ1 =b1δ0

(LU) akck−1δk−2 +δk =bk, 2≤k≤n,k k =δk−1

sinceδk =bkδk−1−akck−1δk−2.

It follows that there is a simple method to solve a linear system, Ax =d, whereA is tridiagonal (andδk = 0 fork = 1, . . . , n). For this, it is convenient to “squeeze” the diagonal matrix, , defined such that k k =δk/δk−1, into the factorization so thatA = (L)(−1U), and if we let

z

1

=

c1, zk =cδk

k δ−1, 2≤k≤n− 1, zn = δn =bn−anzn−1,b1 k δn−1

A = (L)(−1U) is written as

÷ì ÷

ì 1 z_3 ÷ì ÷ì ÷ì ÷

÷ ì

ë1 z1 ö

ì ÷

ë

c

1

ö

ì 1 z_2 ÷

ì ÷ ìz1 c2 ÷ì ÷ìa2 z2 c3 ÷ì ÷ì

÷ ÷ì ÷ì ÷ì ÷ì ÷ ÷ì ì÷

a

3

z

3

÷

ì

...

...

÷

A

=

ì ÷ì

ì ì ÷ .ì ... ... cn−1÷ì 1 zn−2 ÷ì an−1 zn−1 øì ÷í ì ÷

an znì 1 zn−1 ÷

ì ÷

í ø 1

÷ ì

As a consequence, the systemAx =d can be solved by constructing three sequences: First, the sequence

z

1 =c1, zk =bk−akzk−1 − 1, zn =bn−anzn−1,b

ck , k = 2, . . . , n

1

corresponding to the recurrenceδk =bkδk−1−akck−1δk−2 and obtained by dividing both sides of this equation byδk−1, next

w_1 =d1, wk =dk−akwk−1, k = 2, . . . , n,b1 bk−akzk−1

corresponding to solving the systemLw =d, and finally xn =wn, xk =wk−zkxk+1, k =n− 1, n− 2, . . . ,1,

corresponding to solving the system −1Ux =w.

Remark: It can be verified that this requires 3(n− 1) additions, 3(n− 1) multiplications, and 2n divisions, a total of 8n−6 operations, which is much less that theO(2n^3/3) required by Gaussian elimination in general.

We now consider the special case of symmetric positive definite matrices (SPD matrices). Recall that ann×n symmetric matrix,A, is positive definite iff

x Ax > 0 for allx∈ Rn withx = 0.

Equivalently, A is symmetric positive definite iff all its eigenvalues are strictly positive. The following facts about a symmetric positive definite matrice,A, are easily established (some left as an exercise):

(1) The matrixA is invertible. (Indeed, ifAx = 0, thenx Ax = 0, which impliesx = 0.)

(2) We haveai i> 0 fori = 1, . . . , n. (Just observe that forx =ei, theith canonical basis vector of Rn, we haveeiAei =ai i> 0.)

(3) For everyn×n invertible matrix,Z, the matrixZ AZ is symmetric positive definite iffA is symmetric positive definite.

Next, we prove that a symmetric positive definite matrix has a special LU-factorization of the formA =BB , whereB is a lower-triangular matrix whose diagonal elements are strictly positive. This is the Cholesky factorization.

6.4 SPD Matrices and the Cholesky Decomposition

First, we note that a symmetric positive definite matrix satisfies the condition of Proposition 6.2.

Proposition 6.9. IfA is a symmetric positive definite matrix, thenA[1..k,1..k] is symmetric positive definite, and thus, invertible, fork = 1, . . . , n.

Proof. SinceA is symmetric, eachA[1..k,1..k] is also symmetric. Ifw∈ Rk, with 1≤k≤n, we letx∈ Rn be the vector withxi =wi fori = 1, . . . , k andxi = 0 fori =k + 1, . . . , n. Now, sinceA is symmetric positive definite, we havex Ax > 0 for allx∈ Rn withx = 0. This holds in particular for all vectorsx obtained from nonzero vectorsw∈ Rk as defined earlier, and clearly

x Ax =w A[1..k,1..k]w,

which implies thatA[1..k,1..k] is positive definite Thus,A[1..k,1..k] is also invertible. Proposition 6.9 can be strengthened as follows: A symmetric matrixA is positive definite iff det(A[1..k,1..k])> 0 fork = 1, . . . , n.

The above fact is known as Sylvester’s criterion. We will prove it after establishing the Cholseky factorization.

LetA be a symmetric positive definite matrix and write

A = a1 1 W .W C

SinceA is symmetric positive definite,a1 1> 0, and we can computeα =√a1 1. The trick is that we can factorA uniquely as

A = a1 1 W = α 0 1 0 α W /α ,W C W/α I 0 C−W W /a1 1 0 I

i.e., as A =B_1A_1B_1 , whereB_1 is lower-triangular with positive diagonal entries. Thus,B_1

is invertible, and by fact (3) above,A_1 is also symmetric positive definite. 6.4. SPD MATRICES AND THE CHOLESKY DECOMPOSITION 179

Theorem 6.10. (Cholesky Factorization) LetA be a symmetric positive definite matrix. Then, there is some lower-triangular matrix,B, so thatA =BB . Furthermore,B can be chosen so that its diagonal elements are strictly positive, in which case,B is unique.

Proof. We proceed by induction onk. Fork = 1, we must havea1 1 > 0, and if we let α =√a1 1 andB = (α), the theorem holds trivially. Ifk≥ 2, as we explained above, again we must havea1 1> 0, and we can write

A = a1 1 W = α 0 1 0 α W /α =B1A1B1,W C W/α I 0 C−W W /a1 1 0 I

whereα =√a1 1, the matrixB_1 is invertible and

A

1

=

1 0

0 C−W W /a1 1

is symmetric positive definite. However, this implies that C−W W /a1 1 is also symmetric positive definite (considerx A_1x for everyx∈ Rn withx = 0 andx_1 = 0). Thus, we can apply the induction hypothesis toC−W W /a1 1, and we find a unique lower-triangular matrix,L, with positive diagonal entries, so that

C−W W /a1 1 =LL .

But then, we get

A

=

α 0 1 0 α W /α W/α I 0 C−W W /a1 1 0 I

α 0 1 0 α W /α= W/α I 0 LL 0 I

α 0 1 0 1 0 α W /α= W/α I 0 L 0 L 0 I = α 0 α W /α.W/α L 0 L

Therefore, if we let

B = α 0 ,W/α L

we have a unique lower-triangular matrix with positive diagonal entries andA =BB .

The proof of Theorem 6.10 immediately yields an algorithm to computeB fromA. For j = 1, . . . , n,

j−1 1/2

b

b

j k

2 ,j j = aj j− k=1

and fori =j + 1, . . . , n,

j−1

bi j = ai j− bi kbj k /bj j.

k=1

The above formulae are used to compute thejth column ofB from top-down, using the first j− 1 columns ofB previously computed, and the matrixA.

The Cholesky factorization can be used to solve linear systems,Ax = b, whereA is symmetric positive definite: Solve the two systemsBw =b andB x =w.

Remark: It can be shown that this methods requiresn^3/6 +O(n^2) additions,n^3/6 +O(n^2) multiplications,n^2/2+O(n) divisions, andO(n) square root extractions. Thus, the Cholesky method requires half of the number of operations required by Gaussian elimination (since Gaussian elimination requiresn^3/3 +O(n^2) additions,n^3/3 +O(n^2) multiplications, and n^2/2 +O(n) divisions). It also requires half of the space (onlyB is needed, as opposed to bothL andU). Furthermore, it can be shown that Cholesky’s method is numerically stable.

Remark: IfA =BB , whereB is any invertible matrix, thenA is symmetric positive definite.

Proof. Obviously,BB is symmetric, and sinceB is invertible,B is invertible, and from

x Ax =x BB x = (B x) B x,

it is clear thatx Ax > 0 ifx = 0.

We now give three more criteria for a symmetric matrix to be positive definite. Proposition 6.11. LetA be anyn×n symmetric matrix. The following conditions are equivalent:

(a) A is positive definite.

(b) All principal minors ofA are positive; that is: det(A[1..k,1..k])> 0 fork = 1, . . . , n (Sylvester’s criterion).

(c) A has anLU-factorization and all pivots are positive.

(d) A has anLDL -factorization and all pivots inD are positive.

Proof. By Proposition 6.9, ifA is symmetric positive definite, then each matrixA[1..k,1..k] is symmetric positive definite fork = 1, . . . , n. By the Cholsesky decomposition,A[1..k,1..k] = Q Q for some invertible matrixQ, so det(A[1..k,1..k]) = det(Q)2> 0. This shows that (a) implies (b).

If det(A[1..k,1..k])> 0 fork = 1, . . . , n, then eachA[1..k,1..k] is invertible. By Proposition 6.2, the matrixA has anLU-factorization, and since the pivotsπk are given by

ña_11 = det(A[1..1,1..1]) ifk = 1

πk =ò det(A[1..k,1..k])ifk = 2, . . . , n,ódet(A[1..k− 1,1..k− 1])

we see thatπk> 0 fork = 1, . . . , n. Thus (b) implies (c).

Assume A has an LU-factorization and that the pivots are all positive. Since A is symmetric, this implies thatA has a factorization of the form

A =LDL ,

withL lower-triangular with 1’s on its diagonal, and whereD is a diagonal matrix with positive entries on the diagonal (the pivots). This shows that (c) implies (d). Given a factorizationA =LDL with all pivots inD positive, if we form the diagonal matrix

√D = diag(√π_1, . . . ,√πn) and if we letB =L√D, then we have

Q =BB ,

withB lower-triangular and invertible. By the remark before Proposition 6.11,A is positive definite. Hence, (d) implies (a).

Criterion (c) yields a simple computational test to check whether a symmetric matrix is positive definite. There is one more criterion for a symmetric matrix to be positive definite: its eigenvalues must be positive. We will have to learn about the spectral theorem for symmetric matrices to establish this criterion.

For more on the stability analysis and efficient implementation methods of Gaussian elimination,LU-factoring and Cholesky factoring, see Demmel [25], Trefethen and Bau [106], Ciarlet [22], Golub and Van Loan [47], Meyer [77], Strang [100, 101], and Kincaid and Cheney [61].

6.5 Reduced Row Echelon Form

Gaussian elimination described in Section 6.2 can also be applied to rectangular matrices. This yields a method for determining whether a systemAx =b is solvable, and a description of all the solutions when the system is solvable, for any rectangularm×n matrixA.

It turns out that the discussion is simpler if we rescale all pivots to be 1, and for this we need a third kind of elementary matrix. For anyλ = 0, letEi,λ be then×n diagonal matrix ë_1 ö

ì ... ÷

ì ÷

ì_1 ÷

ì ÷

E

i,λ

=

ìλ ÷

ì ÷,

ì_1 ÷

ì ÷

ì ... ÷

í ø

1

with (Ei,λ)ii =λ (1≤i≤n). Note thatEi,λ is also given by

Ei,λ =I + (λ− 1)ei i,

and thatEi,λ is invertible with

E−1 =Ei,λ−1.i,λ

Now, after k− 1 elimination steps, if the bottom portion

(akkk, akk+1k, . . . , akmk)

of the kth column of the current matrixAk is nonzero so that a pivotπk can be chosen, after a permutation of rows if necessary, we also divide rowk byπk to obtain the pivot 1, and not only do we zero all the entriesi =k + 1, . . . , m in columnk, but also all the entries i = 1, . . . , k− 1, so that the only nonzero entry in columnk is a 1 in rowk. These row operations are achieved by multiplication on the left by elementary matrices.

Ifakkk =akk+1k =· · · =akmk = 0, we move on to columnk + 1.

The result is that after performing such elimination steps, we obtain a matrix that has a special shape known as a reduced row echelon matrix. Here is an example illustrating this process: Starting from the matrix

ë1 0 2 1 5ö A_1 = í1 1 5 2 7 ø 1 2 8 4 12 we perform the following steps

ë1 0 2 1 5ö

A1−→A_2 = í0 1 3 1 2ø,

0 2 6 3 7

by subtracting row 1 from row 2 and row 3;

ë1 0 2 1 5ö ë1 0 2 1 5ö ë1 0 2 1 5ö A2−→ í0 2 6 3 7ø−→ í0 1 3 3/2 7/2ø−→A_3 = í0 1 3 3/2 7/2 ø,

0 1 3 1 2 0 1 3 1 2 0 0 0 −1/2 −3/2 after choosing the pivot 2 and permuting row 2 and row 3, dividing row 2 by 2, and subtracting row 2 from row 3;

ë1 0 2 1 5ö ë1 0 2 0 2ö A3−→ í0 1 3 3/2 7/2ø−→A_4 = í0 1 3 0 −1ø, 0 0 0 1 3 0 0 0 1_3

after dividing row 3 by−1/2, subtracting row 3 from row 1, and subtracting (3/2)× row 3 from row 2.

It is clear that columns 1 ,2 and 4 are linearly independent, that column 3 is a linear combination of columns 1 and 2, and that column 5 is a linear combinations of columns 1,2,4.

In general, the sequence of steps leading to a reduced echelon matrix is not unique. For example, we could have chosen 1 instead of 2 as the second pivot in matrixA_2. Nevertherless, the reduced row echelon matrix obtained from any given matrix is unique; that is, it does not depend on the the sequence of steps that are followed during the reduction process. This fact is not so easy to prove rigorously, but we will do it later.

If we want to solve a linear system of equations of the form Ax =b, we apply elementary row operations to both the matrixA and the right-hand sideb. To do this conveniently, we form the augmented matrix (A, b), which is them×(n+ 1) matrix obtained by addingb as an extra column to the matrixA. For example if

ë1 0 2 1ö ë_5 ö

A = í1 1 5 2ø and b = í 7 ø,

1 2 8 4 12

then the augmented matrix is

ë1 0 2 1 5ö (A, b) = í1 1 5 2 7 ø. 1 2 8 4 12 Now, for any matrixM, since

M(A, b) = (MA, Mb),

performing elementary row operations on (A, b) is equivalent to simultaneously performing operations on bothA andb. For example, consider the system

x_1 + 2x_3 + x_4 = 5

x_1 + x_2 + 5x_3 + 2x_4 = 7

x_1 + 2x_2 + 8x_3 + 4x_4 = 12.

Its augmented matrix is the matrix

ë1 0 2 1 5ö (A, b) = í1 1 5 2 7ø 1 2 8 4 12 considered above, so the reduction steps applied to this matrix yield the system x_1 + 2x_3 = 2

x_2 + 3x_3 = −1

x_4 =3.

This reduced system has the same set of solutions as the original, and obviouslyx_3 can be chosen arbitrarily. Therefore, our system has infinitely many solutions given by x_1 = 2− 2x_3, x_2 =−1− 3x_3, x_4 = 3, wherex_3 is arbitrary.

The following proposition shows that the set of solutions of a systemAx =b is preserved by any sequence of row operations.

Proposition 6.12. Given anym×n matrixA and any vectorb∈ Rm, for any sequence of elementary row operationsE_1, . . . , Ek, ifP =Ek· · ·E_1 and (A , b ) =P(A, b), then the solutions ofAx =b are the same as the solutions ofA x =b .

Proof. Since each elementary row operationEi is invertible, so isP, and since (A , b ) = P(A, b), thenA =P A andb =P b. Ifx is a solution of the original systemAx =b, then multiplying both sides byP we getP Ax =P b; that is,A x =b , sox is a solution of the new system. Conversely, assume thatx is a solution of the new system, that isA x =b . Then, becauseA =P A,b =P B, andP is invertible, we get

Ax =P−1A x =P−1b =b,

sox is a solution of the original systemAx =b.

Another important fact is this:

Proposition 6.13. Given am×n matrixA, for any sequence of row operationsE_1, . . . , Ek, ifP =Ek· · ·E_1 andB =P A , then the subspaces spanned by the rows ofA and the rows of B are identical. Therefore,A andB have the same row rank. Furthermore, the matricesA andB also have the same (column) rank.

Proof. SinceB =P A, from a previous observation, the rows ofB are linear combinations of the rows ofA, so the span of the rows ofB is a subspace of the span of the rows ofA. SinceP is invertible,A =P−1B, so by the same reasoning the span of the rows ofA is a subspace of the span of the rows ofB. Therefore, the subspaces spanned by the rows ofA and the rows ofB are identical, which implies thatA andB have the same row rank.

Proposition 6.12 implies that the systems Ax = 0 andBx = 0 have the same solutions. SinceAx is a linear combinations of the columns ofA andBx is a linear combinations of the columns ofB, the maximum number of linearly independent columns inA is equal to the maximum number of linearly independent columns inB; that is,A andB have the same rank.

Remark: The subspaces spanned by the columns ofA andB can be different! However, their dimension must be the same.

Of course, we know from Proposition 4.29 that the row rank is equal to the column rank. We will see that the reduction to row echelon form provides another proof of this important fact. Let us now define precisely what is a reduced row echelon matrix.

Definition 6.1. Am×n matrixA is a reduced row echelon matrix iff the following conditions hold:

(a) The first nonzero entry in every row is 1. This entry is called a pivot. (b) The first nonzero entry of rowi + 1 is to the right of the first nonzero entry of rowi. (c) The entries above a pivot are zero.

If a matrix satisfies the above conditions, we also say that it is in reduced row echelon form, for short rref .

Note that condition (b) implies that the entries below a pivot are also zero. For example, the matrixë1 6 0 1ö

A = í0 0 1 2ø

0 0 0 0

is a reduced row echelon matrix.

The following proposition shows that every matrix can be converted to a reduced row echelon form using row operations.

Proposition 6.14. Given any m× n matrix A, there is a sequence of row operations E_1, . . . , Ek such that ifP =Ek· · ·E_1, thenU =P A is a reduced row echelon matrix.

Proof. We proceed by induction onm. Ifm = 1, then either all entries on this row are zero soA = 0, or ifaj is the first nonzero entry inA, letP = (aj−1) (a 1×1 matrix); clearly,P A is a reduced row echelon matrix.

Let us now assume that m≥ 2. IfA = 0 we are done, so let us assume thatA = 0. Since A = 0, there is a leftmost columnj which is nonzero, so pick any pivotπ =aij in thejth column, permute rowi and row 1 if necessary, multiply the new first row byπ−1, and clear out the other entries in columnj by subtracting suitable multiples of row 1. At the end of this process, we have a matrixA_1 that has the following shape:

ë0 · · ·0 1 ∗ · · · ∗÷

A_1 =ì0 · · ·0 0 ∗ · · · ∗÷,ì. . . . .÷

í0 · · ·0 0 ∗ · · · ∗

ö

ì

ø

where∗ stands for an arbitrary scalar, or more concisely,

A0 1 B ,1 =0 0 D

where D is a (m− 1)× (n−j) matrix. Ifj =n, we are done. Otherwise, by the induction hypothesis applied to D , there is a sequence of row operations that convertsD to a reduced row echelon matrixR , and these row operations do not affect the first row ofA_1, which means thatA_1 is reduced to a matrix of the form

R =0 1 B .0 0 R

Because R is a reduced row echelon matrix, the matrixR satisfies conditions (a) and (b) of the reduced row echelon form. Finally, the entries above all pivots inR can be cleared out by subtracting suitable multiples of the rows ofR containing a pivot. The resulting matrix also satisfies condition (c), and the induction step is complete.

Remark: There is a Matlab function named rref that converts any matrix to its reduced row echelon form.

If A is any matrix and ifR is a reduced row echelon form ofA, the second part of Proposition 6.13 can be sharpened a little. Namely, the rank ofA is equal to the number of pivots inR.

This is because the structure of a reduced row echelon matrix makes it clear that its rank is equal to the number of pivots.

Given a system of the form Ax =b, we can apply the reduction procedure to the augmented matrix (A, b) to obtain a reduced row echelon matrix (A , b ) such that the system A x =b has the same solutions as the original systemAx =b. The advantage of the reduced systemA x =b is that there is a simple test to check whether this system is solvable, and to find its solutions if it is solvable.

Indeed, if any row of the matrixA is zero and if the corresponding entry inb is nonzero, then it is a pivot and we have the “equation”

0 = 1,

which means that the system A x =b has no solution. On the other hand, if there is no pivot inb , then for every rowi in whichbi = 0, there is some columnj inA where the entry on rowi is 1 (a pivot). Consequently, we can assign arbitrary values to the variable xk if columnk does not contain a pivot, and then solve for the pivot variables.

For example, if we consider the reduced row echelon matrix

ë1 6 0 1 0ö

(A , b ) = í0 0 1 2 0ø,

0 0 0 0 1

there is no solution toA x =b because the third equation is 0 = 1. On the other hand, the reduced systemë1 6 0 1 1ö

(A , b ) = í0 0 1 2 3ø

0 0 0 0 0

has solutions. We can pick the variablesx_2, x_4 corresponding to nonpivot columns arbitrarily, and then solve forx_3 (using the second equation) andx_1 (using the first equation). The above reasoning proved the following theorem:

Theorem 6.15. Given any systemAx =b whereA is am×n matrix, if the augmented matrix (A, b) is a reduced row echelon matrix, then the systemAx =b has a solution iff there is no pivot inb. In that case, an arbitrary value can be assigned to the variablexj if column j does not contain a pivot.

Nonpivot variables are often called free variables.

Putting Proposition 6.14 and Theorem 6.15 together we obtain a criterion to decide whether a systemAx =b has a solution: Convert the augmented system (A, b) to a row reduced echelon matrix (A , b ) and check whetherb has no pivot.

Remark: When writing a program implementing row reduction, we may stop when the last column of the matrixA is reached. In this case, the test whether the systemAx =b is solvable is that the row-reduced matrixA has no zero row of indexi > r such thatbi = 0 (wherer is the number of pivots, andb is the row-reduced right-hand side).

If we have a homogeneous systemAx = 0, which means thatb = 0, of coursex = 0 is always a solution, but Theorem 6.15 implies that if the systemAx = 0 has more variables than equations, then it has some nonzero solution (we call it a nontrivial solution).

Proposition 6.16. Given any homogeneous systemAx = 0 ofm equations inn variables, ifm < n, then there is a nonzero vectorx∈ Rn such thatAx = 0.

Proof. Convert the matrixA to a reduced row echelon matrixA . We know thatAx = 0 iff A x = 0. Ifr is the number of pivots ofA , we must haver≤m, so by Theorem 6.15 we may assign arbitrary values ton−r > 0 nonpivot variables and we get nontrivial solutions.

Theorem 6.15 can also be used to characterize when a square matrix is invertible. First, note the following simple but important fact:

If a squaren×n matrixA is a row reduced echelon matrix, then eitherA is the identity or the bottom row of A is zero.

Proposition 6.17. LetA be a square matrix of dimensionn. The following conditions are equivalent:

(a) The matrixA can be reduced to the identity by a sequence of elementary row operations.

(b) The matrixA is a product of elementary matrices.

(c) The matrixA is invertible.

(d) The system of homogeneous equationsAx = 0 has only the trivial solutionx = 0.

Proof. First, we prove that (a) implies (b). If (a) can be reduced to the identity by a sequence of row operationsE_1, . . . , Ep, this means thatEp· · ·E_1A =I. Since eachEi is invertible, we get

A

=

E

−1 Ep 1

1 · · · , where eachEi−1 is also an elementary row operation, so (b) holds. Now if (b) holds, since elementary row operations are invertible,A is invertible, and (c) holds. IfA is invertible, we already observed that the homogeneous systemAx = 0 has only the trivial solutionx = 0, because fromAx = 0, we getA−1Ax =A−10; that is,x = 0. It remains to prove that (d) implies (a), and for this we prove the contrapositive: if (a) does not hold, then (d) does not hold.

Using our basic observation about reducing square matrices, if A does not reduce to the identity, thenA reduces to a row echelon matrixA whose bottom row is zero. SayA =P A, whereP is a product of elementary row operations. Because the bottom row ofA is zero, the systemA x = 0 has at mostn− 1 nontrivial equations, and by Proposition 6.16, this system has a nontrivial solutionx. But then,Ax =P−1A x = 0 withx = 0, contradicting the fact that the systemAx = 0 is assumed to have only the trivial solution. Therefore, (d) implies (a) and the proof is complete.

Proposition 6.17 yields a method for computing the inverse of an invertible matrixA: reduceA to the identity using elementary row operations, obtaining

Ep· · ·E_1A =I.

Multiplying both sides byA−1 we get

A−1 =Ep· · ·E_1.

From a practical point of view, we can build up the product Ep· · ·E_1 by reducing to row echelon form the augmentedn×2n matrix (A, In) obtained by adding then columns of the identity matrix toA. This is just another way of performing the Gauss–Jordan procedure.

Here is an example: let us find the inverse of the matrix

A =5 4 .6 5 We form the 2× 4 block matrix

(

A, I

) =

5 4 1 0

6 5 0 1

and apply elementary row operations to reduceA to the identity. For example:

(

A, I

) =

5 4 1 0 5 4 1 0

6 5 0 1 −→ 1 1 −1 1

by subtracting row 1 from row 2, 5 4 1 0 1 0 5 −4 1 1 −1 1 −→ 1 1 −1 1 by subtracting 4× row 2 from row 1, 1 0 5 −4 1 0 5 −4= (I, A−1),1 1 −1 1 −→ 0 1 −6 5

by subtracting row 1 from row 2. Thus

A

−

1

=

5

4

− .

−6 5

Proposition 6.17 can also be used to give an elementary proof of the fact that if a square matrixA has a left inverseB (resp. a right inverseB), so thatBA =I (resp. AB =I), thenA is invertible andA−1 =B. This is an interesting exercise, try it!

For the sake of completeness, we prove that the reduced row echelon form of a matrix is unique. The neat proof given below is borrowed and adapted from W. Kahan.

Proposition 6.18. LetA be anym×n matrix. IfU andV are two reduced row echelon matrices obtained fromA by applying two sequences of elementary row operationsE_1, . . . , Ep andF_1, . . . , Fq, so that

U =Ep· · ·E_1A and V =Fq· · ·F_1A, thenU =V andEp· · ·E_1 =Fq· · ·F_1. In other words, the reduced row echelon form of any matrix is unique.

Proof. Let C

=

E

p

· · ·

E

1

F

1

−1 Fq−1 · · · so that

U =CV and V =C−1U. We prove by induction onn thatU =V (andC =I). Let j denote thejth column of the identity matrixIn, and letuj =U j,vj =V j, cj =C j, andaj =Aj, be thejth column ofU,V ,C, andA respectively. First, I claim thatuj = 0 iffvj = 0, iffaj = 0.

Indeed, ifvj = 0, then (becauseU =CV )uj =Cvj = 0, and ifuj = 0, thenvj = C−1uj = 0. SinceA =Ep· · ·E_1U, we also getaj = 0 iffuj = 0.

Therefore, we may simplify our task by striking out columns of zeros from U, V , andA, since they will have corresponding indices. We still usen to denote the number of columns of A. Observe that becauseU andV are reduced row echelon matrices with no zero columns, we must haveu_1 =v_1 =1.

Claim . IfU andV are reduced row echelon matrices without zero columns such that U =CV , for allk≥ 1, ifk≤n, then k occurs inU iff k occurs inV , and if k does occurs inU, then

1. k occurs for the same indexjk in bothU andV ;

2. the firstjk columns ofU andV match;

3. the subsequent columns inU andV (of index> jk) whose elements beyond thekth all vanish also match;

4. the firstk columns ofC match the firstk columns ofIn.

We prove this claim by induction onk.

For the base casek = 1, we already know thatu_1 =v_1 =1. We also have

c_1 =C_1 =Cv_1 =u_1 =1.

Ifvj =λ_1 for someµ∈ R, then

uj =U_1 =CV_1 =Cvj =λC_1 =λ_1 =vj.

A similar argument using C−1 shows that ifuj =λ_1, thenvj =uj. Therefore, all the columns ofU andV proportional to_1 match, which establishes the base case. Observe that if_2 appears inU, then it must appear in bothU andV for the same index, and if not then U =V .

Next us now prove the induction step; this is only necessary if k+1 appears in bothU, in wich case, by (3) of the induction hypothesis, it appears in bothU andV for the same index, sayjk+1. Thusujk+1 =vjk+1 = k+1. It follows that

ck+1 =C k+1 =Cvjk+1 =ujk+1 = k+1,

so the firstk + 1 columns ofC match the firstk + 1 columns ofIn. Consider any subsequent columnvj (withj > jk+1) whose elements beyond the (k+ 1)th all vanish. Then,vj is a linear combination of columns ofV to the left ofvj, so

uj =Cvj =vj.

because the first k+ 1 columns ofC match the first column ofIn. Similarly, any subsequent columnuj (withj > jk+1) whose elements beyond the (k + 1)th all vanish is equal tovj. Therefore, all the subsequent columns inU andV (of index> jk+1) whose elements beyond the (k + 1)th all vanish also match, which completes the induction hypothesis.

We can now prove that U =V (recall that we may assume thatU andV have no zero columns). We noted earlier thatu_1 =v_1 =1, so there is a largestk≤n such that k occurs inU. Then, the previous claim implies that all the columns ofU andV match, which means thatU =V .

The reduction to row echelon form also provides a method to describe the set of solutions of a linear system of the formAx =b. First, we have the following simple result.

Proposition 6.19. LetA be anym× n matrix and letb∈ Rm be any vector. If the system Ax =b has a solution, then the setZ of all solutions of this system is the set

Z =x_0 + Ker (A) ={x_0 +x|Ax = 0},

where x0∈ Rn is any solution of the systemAx =b, which means thatAx_0 =b (x_0 is called a special solution), and where Ker (A) ={x∈ Rn Ax = 0}, the set of solutions of the

homogeneous system associated withAx =b.|

Proof. Assume that the systemAx =b is solvable and letx_0 andx_1 be any two solutions so thatAx_0 =b andAx_1 =b. Subtracting the first equation from the second, we get

A(x1−x_0) = 0,

which means that x1−x0∈ Ker (A). Therefore,Z⊆x_0 + Ker (A), wherex_0 is a special solution ofAx =b. Conversely, ifAx_0 =b, then for anyz∈ Ker (A), we haveAz = 0, and so

A(x_0 +z) =Ax_0 +Az =b + 0 =b, which shows thatx_0 + Ker (A)⊆Z. Therefore,Z =x_0 + Ker (A).

Given a linear system Ax =b, reduce the augmented matrix (A, b) to its row echelon form (A , b ). As we showed before, the systemAx =b has a solution iffb contains no pivot. Assume that this is the case. Then, if (A , b ) hasr pivots, which means thatA hasr pivots sinceb has no pivot, we know that the firstr columns ofIn appear inA .

We can permute the columns of A and renumber the variables inx correspondingly so that the firstr columns ofIn match the firstr columns ofA , and then our reduced echelon matrix is of the form (R, b ) with

Ir F

R = 0m−r,r 0m−r,n−r

and

b =d ,0m−r

whereF is ar× (n−r) matrix andd∈ Rr. Note thatR hasm−r zero rows. Then, because

d

,

0m

Ir F d = 0m−r−r,r 0m−r,n−r 0n−r

we see thatdx0 = 0n−r

is a special solution of Rx =b , and thus toAx =b. In other words, we get a special solution by assigning the firstr components ofb to the pivot variables and setting the nonpivot variables (the free variables) to zero.

We can also find a basis of the kernel (nullspace) of A usingF. Ifx = (u, v) is in the kernel ofA, withu∈ Rr andv∈ Rn−r, thenx is also in the kernel ofR, which means that Rx = 0; that is,

0

r

.

0m

Ir F u= u +F v= 0m−r−r,r 0m−r,n−r v 0m−r

Therefore,u =−F v, and Ker (A) consists of all vectors of the form

−F v = −F v,v In−r

for any arbitraryv∈ Rn−r. It follows that then−r columns of the matrix

N = −F

In−r

form a basis of the kernel of A. This is becauseN contains the identity matrixIn−r as a submatrix, so the columns ofN are linearly independent. In summary, ifN^1, . . . , Nn−r are the columns ofN, then the general solution of the equationAx =b is given by

x

=

0

d +xr+1N^1 +

n−r · · · +xnNn−r,

wherexr+1, . . . , xn are the free variables, that is, the nonpivot variables.

In the general case where the columns corresponding to pivots are mixed with the columns corresponding to free variables, we find the special solution as follows. Leti_1<· · ·< ir be the indices of the columns corresponding to pivots. Then, assignbk to the pivot variable xik fork = 1, . . . , r, and set all other variables to 0. To find a basis of the kernel, we form then−r vectorsNk obtained as follows. Letj_1<· · ·< jn−r be the indices of the columns corresponding to free variables. For every columnjk corresponding to a free variable (1≤k≤n−r), form the vectorNk defined so that the entriesNk, . . . , Nk are equal to thei1 ir negatives of the firstr entries in columnjk (flip the sign of these entries); letNk = 1, and setjk 1, . . . , Nn−rall other entries to zero. The presence of the 1 in positionjk guarantees thatN

are linearly independent.

An illustration of the above method, consider the problem of finding a basis of the subspaceV ofn×n matricesA∈ Mn(R) satisfying the following properties:

1. The sum of the entries in every row has the same value (sayc_1);

2. The sum of the entries in every column has the same value (sayc_2).

It turns out that c_1 =c_2 and that the 2n−2 equations corresponding to the above conditions are linearly independent. We leave the proof of these facts as an interesting exercise. By the duality theorem, the dimension of the spaceV of matrices satisying the above equations is n^2 (2n− 2). Let us consider the casen = 4. There are 6 equations, and the spaceV has−

dimension 10. The equations are

a_11 +a_12 +a_13 +a14−a21−a22−a23−a_24 = 0

a_21 +a_22 +a_23 +a24−a31−a32−a33−a_34 = 0

a_31 +a_32 +a_33 +a34−a41−a42−a43−a_44 = 0

a_11 +a_21 +a_31 +a41−a12−a22−a32−a_42 = 0

a_12 +a_22 +a_32 +a42−a13−a23−a33−a_43 = 0

a_13 +a_23 +a_33 +a43−a14−a24−a34−a_44 = 0,

and the corresponding matrix is

ë 1 1 1 1 −1 −1 −1 −1 0 0 0 0 0 0 0 0ö ì0 0 0 01 1 1 1 −1 −1 −1 −1 0 0 0 0÷

ì0 0 0 0 0 0 0 01 1 1 1 −1 −1 −1 −1÷ A

=

ì ÷

ì ÷.ì1 −1 0 0 1 −1 0 0 1 −1 0 0 1 −1 0 0 ÷ ì0 1 −1 0 0 1 −1 0 0 1 −1 0 0 1 −1 0 ÷

í ø 0 0 1 −1 0 0 1 −1 0 0 1 −1 0 0 1 −1 The result of performing the reduction to row echelon form yields the following matrix in rref:

ë1 0 0 0 0 − 1 − 1 − 1 0 − 1 − 1 − 1 2 1 1 1ö ì0 1 0 0 0 1 0 0 0 1 0 0 −1 0 −1 −1÷

ì

÷ ì

0

0

1

0

0

0

1

0

0

0

1

0

−

1

−

1

0

− 1

÷

U

=

ì ÷

ì0 0 0 1 0 0 0 1 0 0 0 1 −1 −1 −1 0 ÷ ì0 0 0 0 1 1 1 1 0 0 0 0 −1 −1 −1 −1÷ í ø 0 0 0 0 0 0 0 0 1 1 1 1 −1 −1 −1 −1 The list pivlist of indices of the pivot variables and the list freelist of indices of the free variables is given by

pivlist = (1,2,3,4,5,9),

freelist = (6,7,8,10,11,12,13,14,15,16). After applying the algorithm to find a basis of the kernel ofU, we find the following 16×10

matrix ë 1 1 1 1 1 1 −2 −1 −1 −1ö ì 1 0 0 −1 0 01 0 1 1 ÷

ì

÷

ì

−

1

0

0

−

1

0

1

1

0

1

÷ ì_0

÷ ì

−

1

0

0

−1 1 1 1 0÷

ì 0_0

÷ ì

−

1

0

0

0

1

1

1

1

÷

ì 1 −1 −0 0 0 0 0 0 0 ÷

ì− ÷ ì1 0 0 ÷

ì 0 1 0 0 0 0 0 0 0 0÷ ì 0 0 1 0 0 0 0 0 0 0÷ BK

=

ì ÷

ì .ì 0 0 0 −1 −1 −1 1 1 1 1÷

÷

ì 0 0 01 0 0 0 0 0 0÷

ì ÷

ì 0 0 0 0 1 0 0 0 0 0÷

ì ÷

ì 0 0 0 0 0 1 0 0 0 0÷

ì ÷

ì 0 0 0 0 0 0 1 0 0 0÷

ì ÷ ì ÷

ì 0 0 0 0 0 0 0 1 0 0÷ ì 0 0 0 0 0 0 0 0 1 0÷ í ø 0 0 0 0 0 0 0 0 0 1

The reader should check that that in each column j ofBK, the lowest 1 belongs to the row whose index is thejth element in freelist, and that in each columnj ofBK, the signs of the entries whose indices belong to pivlist are the fipped signs of the 6 entries in the column U corresponding to thejth index in freelist. We can now read off fromBK the 4×4 matrices that form a basis ofV : every column ofBK corresponds to a matrix whose rows have been concatenated. We get the following 10 matrices:

ë 1 −1 0 0ö ë 1 0 −1 0ö ë 1 0 0 −1ö ì 1 1 0 0÷, M_2 =ì 1 0 1 0÷, M_3 =ì 1 0 0 1 ÷M_1 =ì−0 0 0÷ ì−0 0 0÷ ì−0 0 0 ÷ í_0 ø í_0 ø í_0 ø 0 0 0 0 0 0 0 0 0 0 0 0 ë 1 −1 0 0ö ë 1 0 −1 0ö ë 1 0 0 −1ö ì 0 0 0 0÷, M_5 =ì 0 0 0 0÷, M_6 =ì 0 0 0 0 ÷M_4 = ì1 1 0 0÷ ì1 0 1 0÷ ì1 0 0 1 ÷ í− ø í ø í ø 0 0 00 0 0− 0 0 0− 0 0 0

ë 2 1 1 1ö ë 1 0 1 1ö ë 1 1 0 1ö ì− 0 0 0÷, M_8 =ì− 0 0 0÷, M_9 =ì− 0 0 0÷M_7 = ì^1 ÷ ì^1 ÷ ì^1 ÷ í 1 0 0 0ø í 1 0 0 0ø í 1 0 0 0ø 1 0 0 0 0 1 0 0 0 0 1 0 ë 1 1 1 0ö

ì− 0 0 0÷.10 = ì^1 ÷Mí 1 0 0 0ø

0 0 0 1

Recall that a magic square is a square matrix that satisfies the two conditions about the sum of the entries in each row and in each column to be the same number, and also the additional two constraints that the main descending and the main ascending diagonals add up to this common number. Furthermore, the entries are also required to be positive integers. Forn = 4, the additional two equations are

a_22 +a_33 +a44−a12−a13−a_14 = 0

a_41 +a_32 +a23−a11−a12−a_13 = 0,

and the 8 equations stating that a matrix is a magic square are linearly independent. Again, by running row elimination, we get a basis of the “generalized magic squares” whose entries are not restricted to be positive integers. We find a basis of 8 matrices. Forn = 3, we find a basis of 3 matrices.

A magic square is said to be normal if its entries are precisely the integers 1,2. . . , n^2. Then, since the sum of these entries is

1 + 2 + 3 +· · · +n^2 =n2(n2 + 1),2

and since each row (and column) sums to the same number, this common value (the magic sum) is

n(n^2 + 1).2

It is easy to see that there are no normal magic squares for n = 2. Forn = 3, the magic sum is 15, and forn = 4, it is 34. In the casen = 3, we have the additional condition that the rows and columns add up to 15, so we end up with a solution parametrized by two numbers

x1, x2; namely, ë x_1 +x2− 5 10−x_2 10−x_1 ö

í20− 2x1−x2 5 2x_1 +x2− 10ø.

x_1 x_2 15−x1−x_2

Thus, in order to find a normal magic square, we have the additional inequality constraints

x_1 +x_2> 5 x_1< 10 x_2< 10

2 x_1 +x_2< 20

2x_1 +x_2> 10 x_1> 0 x_2> 0 x_1 +x_2< 15,

and all 9 entries in the matrix must be distinct. After a tedious case analysis, we discover the remarkable fact that there is a unique normal magic square (up to rotations and reflections): ë2 7 6ö

í9 5 1ø.

4 3 8

It turns out that there are 880 different normal magic squares for n = 4, and 275,305,224 normal magic squares forn = 5 (up to rotations and reflections). Even forn = 4, it takes a fair amount of work to enumerate them all!

Instead of performing elementary row operations on a matrix A, we can perform elementary columns operations, which means that we multiplyA by elementary matrices on the right. As elementary row and column operations,P(i, k),Ei,j;β,Ei,λ perform the following actions:

1. As a row operation, P(i, k) permutes rowi and rowk.

2. As a column operation,P(i, k) permutes columni and columnk.

3. The inverse ofP(i, k) isP(i, k) itself.

4. As a row operation,Ei,j;β addsβ times rowj to rowi.

5. As a column operation,Ei,j;β addsβ times columni to columnj (note the switch in the indices).

6. The inverse of Ei,j;β isEi,j;−β.

7. As a row operation,Ei,λ multiplies rowi byλ.

8. As a column operation,Ei,λ multiplies columni byλ.

9. The inverse ofEi,λ isEi,λ−1.

We can define the notion of a reduced column echelon matrix and show that every matrix can be reduced to a unique reduced column echelon form. Now, given anym×n matrixA, if we first convertA to its reduced row echelon formR, it is easy to see that we can apply elementary column operations that will reduceR to a matrix of the form

Ir 0r,n−r ,0m−r,r 0m−r,n−r

where r is the number of pivots (obtained during the row reduction). Therefore, for every m×n matrixA, there exist two sequences of elementary matricesE_1, . . . , Ep andF_1, . . . , Fq, such that

EIr 0r,n−r .p· · ·E_1AF1· · ·Fq =0m−r,r 0m−r,n−r

The matrix on the right-hand side is called the rank normal form ofA. Clearly,r is the rank ofA. It is easy to see that the rank normal form also yields a proof of the fact thatA and its transposeA have the same rank.

6.6 Transvections and Dilatations

In this section, we characterize the linear isomorphisms of a vector space E that leave every vector in some hyperplane fixed. These maps turn out to be the linear maps that are represented in some suitable basis by elementary matrices of the formEi,j;β (transvections) orEi,λ (dilatations). Furthermore, the transvections generate the group SL(E), and the dilatations generate the group GL(E).

LetH be any hyperplane inE, and pick some (nonzero) vectorv∈E such thatv /∈H, so that

E =H⊕Kv.

Assume thatf :E→E is a linear isomorphism such thatf(u) =u for allu∈H, and that f is not the identity. We have

f(v) =h +αv, for someh∈H and someα∈K, withα = 0, because otherwise we would havef(v) =h =f(h) sinceh∈H, contradicting the injectivity off (v =h sincev /∈H). For anyx∈E, if we write

x =y +tv, for somey∈H and somet∈K,

then f(x) =f(y) +f(tv) =y +tf(v) =y +th +tαv, and sinceαx =αy +tαv, we get

f (x)−αx = (1−α)y +th f(x)−x =t(h + (α− 1)v). Observe that ifE is finite-dimensional, by picking a basis ofE consisting ofv and basis vectors ofH, then the matrix off is a lower triangular matrix whose diagonal entries are all 1 except the first entry which is equal toα. Therefore, det(f) =α.

Case 1.α = 1.

We havef(x) =αx iff (1−α)y +th = 0 iff

y = t h.α− 1

Then, if we letw =h + (α− 1)v, fory = (t/(α− 1))h, we have

x =y +tv = t h +tv = t 1(h + (α− 1)v) = t w,α− 1 α− α− 1

which shows thatf(x) =αx iffx∈Kw. Note thatw /∈H, sinceα = 1 andv /∈H. Therefore,

E =H⊕Kw,

andf is the identity onH and a magnification byα on the lineD =Kw.

Definition 6.2. Given a vector spaceE, for any hyperplaneH inE, any nonzero vector u∈E such thatu∈H, and any scalarα = 0,1, a linear mapf such thatf(x) =x for all x∈H andf(x) =αx for everyx∈D =Ku is called a dilatation of hyperplaneH, direction D , and scale factorα.

IfπH andπD are the projections ofE ontoH andD, then we have

f(x) =πH(x) +απD(x).

The inverse off is given by

f−1(x) =πH(x) +α−1πD(x).

Whenα =−1, we havef^2 = id, andf is a symmetry about the hyperplaneH in the directionD.

Case 2.α = 1. In this case,

f(x)−x =th,

that is, f(x)−x∈Kh for allx∈E. Assume that the hyperplaneH is given as the kernel of some linear formÕ, and leta =Õ(v). We havea = 0, sincev /H. For anyx∈E, we have

Õ (x−a−1Õ(x)v) =Õ(x)−a−1Õ(x)Õ(v) =Õ(x)−Õ(x) = 0, which shows thatx−a−1Õ(x)v∈H for allx∈E. Since every vector inH is fixed byf, we get

x−a−1Õ(x)v =f(x−a−1Õ(x)v) =f(x)−a−1Õ(x)f(v),

so f(x) =x +Õ(x)(f(a−1v)−a−1v).

Sincef(z)−z∈Kh for allz∈E, we conclude thatu =f(a−1v)−a−1v =βh for some β∈K, soÕ(u) = 0, and we have

f(x) =x +Õ(x)u, Õ(u) = 0. (∗)

A linear map defined as above is denoted byτÕ,u.

Conversely for any linear map f =τÕ,u given by equation (∗), whereÕ is a nonzero linear form andu is some vectoru∈E such thatÕ(u) = 0, ifu = 0 thenf is the identity, so assume thatu = 0. If so, we havef(x) =x iffÕ(x) = 0, that is, iffx∈H. We also claim that the inverse off is obtained by changingu to−u. Actually, we check the slightly more general fact that

τÕ,uæτÕ,v =τÕ,u+v.

Indeed, using the fact thatÕ(v) = 0, we have

τÕ,u(τÕ,v(x)) =τÕ,v(x) +Õ(τÕ,v(v))u =τÕ,v(x) + (Õ(x) +Õ(x)Õ(v))u =τÕ,v(x) +Õ(x)u

=x +Õ(x)v +Õ(x)u

=x +Õ(x)(u +v).

Forv =−u, we haveτÕ,u+v =ÕÕ,0 = id, soτ−1 =τÕ,−u, as claimed.Õ,u

Therefore, we proved that every linear isomorphism of E that leaves every vector in some hyperplaneH fixed and has the property thatf(x)−x∈H for allx∈E is given by a map τÕ,u as defined by equation (∗), whereÕ is some nonzero linear form definingH andu is some vector inH. We haveτÕ,u = id iffu = 0.

Definition 6.3. Given any hyperplaneH inE, for any nonzero nonlinear formÕ∈E∗ definingH (which means thatH = Ker (Õ)) and any nonzero vectoru∈H, the linear map τÕ,u given by

τÕ,u(x) =x +Õ(x)u, Õ(u) = 0,

for allx∈E is called a transvection of hyperplaneH and directionu. The mapτÕ,u leaves every vector inH fixed, andf(x)−x∈Ku for allx∈E.

The above arguments show the following result.

Proposition 6.20. Letf :E→E be a bijective linear map and assume thatf = id and thatf(x) =x for allx∈H, whereH is some hyperplane inE. If there is some nonzero vectoru∈E such thatu /∈H andf(u)− u∈H, thenf is a transvection of hyperplaneH; otherwise, f is a dilatation of hyperplane H.

Proof. Using the notation as above, for somev /∈H, we havef(v) =h +αv withα = 0, and writeu =y +tv withy∈H andt = 0 sinceu /∈H. Iff(u)−u∈H, from f(u)−u =t(h + (α− 1)v),

we get (α− 1)v∈H, and sincev /H, we must haveα = 1, and we proved thatf is a transvection. Otherwise,α = 0,1, and we proved thatf is a dilatation.

IfE is finite-dimensional, thenα = det(f), so we also have the following result.

Proposition 6.21. Letf :E→E be a bijective linear map of a finite-dimensional vector spaceE and assume thatf = id and thatf(x) =x for allx∈H, whereH is some hyperplane inE. If det(f) = 1, thenf is a transvection of hyperplaneH; otherwise,f is a dilatation of hyperplaneH.

Suppose that f is a dilatation of hyperplaneH and directionu, and say det(f) =α = 0,1. Pick a basis (u, e_2, . . . , en) ofE where (e_2, . . . , en) is a basis ofH. Then, the matrix off is of the formëα 0 · · · 0ö

ì0 1 0÷

ì... ÷

.

÷,ì

í . ø0 0 · · ·1

which is an elementary matrix of the formE1,α. Conversely, it is clear that every elementary matrix of the formEi,α withα = 0,1 is a dilatation.

Now, assume that f is a transvection of hyperplaneH and directionu∈H. Pick some v /H, and pick some basis (u, e_3, . . . , en) ofH, so that (v, u, e_3, . . . , en) is a basis ofE. Since f(v)−v∈Ku, the matrix off is of the form

ë1 0 · · ·0ö

ìα 1 0÷

ì... ÷

.

÷,ì

í . ø0 0 · · ·1

which is an elementary matrix of the formE2,1;α. Conversely, it is clear that every elementary matrix of the formEi,j;α (α = 0) is a transvection.

The following proposition is an interesting exercise that requires good mastery of the elementary row operationsEi,j;β.

Proposition 6.22. Given any invertiblen×n matrixA, there is a matrixS such that

SA = In− 1 0 =En,α,0 α

withα = det(A), and whereS is a product of elementary matrices of the formEi,j;β; that is,S is a composition of transvections.

Surprisingly, every transvection is the composition of two dilatations!

Proposition 6.23. If the fieldK is not of charateristic 2, then every transvectionf of hyperplaneH can be written asf =d2æd_1, whered_1, d_2 are dilatations of hyperplaneH, where the direction ofd_1 can be chosen arbitrarily.

Proof. Pick some dilalationd_1 of hyperplaneH and scale factorα = 0,1. Then,d_2 =fæd−1

1

1

leaves every vector inH fixed, and det(d_2) =α− = 1. By Proposition 6.21, the linear map d_2 is a dilatation of hyperplaneH, and we havef =d2æd_1, as claimed.

Observe that in Proposition 6.23, we can pick α =−1; that is, every transvection of hyperplaneH is the compositions of two symmetries about the hyperplaneH, one of which can be picked arbitrarily.

Remark: Proposition 6.23 holds as long asK ={0,1}.

The following important result is now obtained.

Theorem 6.24. LetE be any finite-dimensional vector space over a fieldK of characteristic not equal to 2. Then, the group SL(E) is generated by the transvections, and the group GL(E) is generated by the dilatations.

Proof. Consider anyf∈ SL(E), and letA be its matrix in any basis. By Proposition 6.22, there is a matrixS such that

SA = In− 1 0 =En,α,0 α

with α = det(A), and whereS is a product of elementary matrices of the formEi,j;β. Since det(A) = 1, we haveα = 1, and the resut is proved. Otherwise,En,α is a dilatation,S is a product of transvections, and by Proposition 6.23, every transvection is the composition of two dilatations, so the second result is also proved.

We conclude this section by proving that any two transvections are conjugate in GL(E). LetτÕ,u (u = 0) be a transvection and letg∈ GL(E) be any invertible linear map. We have

1 )(x) =g(g−1(x) +Õ(g−1(x))u)æτÕ,uæg−(g

=x +Õ(g−1(x))g(u). Let us find the hyperplane determined by the linear formx→Õ(g−1(x)). This is the set of vectorsx∈ E such thatÕ(g−1(x)) = 0, which holds iffg−1(x)

1

) =

g

(

H

) =

H

, and we have

g

(

u

)

∈

g

(

H

) =

H

, so

g

∈

H iffx∈g(H). Therefore, Ker (Õæg− æτÕ,uæg−1 is the transvection of hyperplaneH =g(H) and directionu =g(u) (withu∈H ).

Conversely, letτψ,u be some transvection (u = 0). Pick some vectorv, v such that Õ(v) =ψ(v ) = 1, so that

E =H⊕Kv =H⊕v .

There is a linear map g∈ GL(E) such thatg(u) =u ,g(v) =v , andg(H) =H . To defineg, pick a basis (v, u, e_2, . . . , en− 1) where (u, e_2, . . . , en− 1) is a basis ofH and pick a basis (v , u , e_2, . . . , en−1) where (u , e_2, . . . , en −1) is a basis ofH ; theng is defined so that g(v) =v ,g(u) =u , andg(ei) =g(ei), fori = 2, . . . , n− 1. Ifn = 2, thenei andei are missing. Then, we have

(gæτÕ,uæg−1)(x) =x +Õ(g−1(x))u .

Now,Õæg−1 also determines the hyperplaneH =g(H), so we haveÕæg−1 =λψ for some nonzeroλ inK. Sincev =g(v), we get

Õ(v) =Õæg−1(v ) =λψ(v ),

and sinceÕ(v) =ψ(v ) = 1, we must haveλ = 1. It follows that

(gæτÕ,uæg−1)(x) =x +ψ(x)u =τψ,u (x).

In summary, we proved almost all parts the following result.

Proposition 6.25. LetE be any finite-dimensional vector space. For every transvection τÕ,u (u = 0) and every linear mapg∈ GL(E), the mapgæτÕ,uæg−1 is the transvection of hyperplaneg(H) and directiong(u) (that is,gæτÕ,uæg−1 =τÕæg−1,g(u)). For every other transvectionτψ,u (u = 0) , there is someg∈ GL(E) suchτψ,u =gæτÕ,uæg−1; in other words any two transvections (= id) are conjugate in GL(E). Moreover, ifn≥ 3, then the linear isomorphimg as above can be chosen so thatg∈ SL(E).

Proof. We just need to prove that ifn≥ 3, then for any two transvectionsτÕ,u andτψ,u (u, u = 0), there is someg∈ SL(E) such thatτψ,u =gæτÕ,uæg−1. As before, we pick a basis (v, u, e_2, . . . , en−1) where (u, e_2, . . . , en−1) is a basis ofH, we pick a basis (v , u , e_2, . . . , en−1) where (u , e_2, . . . , en− 1) is a basis ofH , and we defineg as the unique linear map such that g(v) =v ,g(u) =u , andg(ei) =ei, fori = 1, . . . , n− 1. But, in this case, bothH and H =g(H) have dimension at least 2, so in any basis ofH includingu , there is some basis vectore_2 independent ofu , and we can rescalee_2 in such a way that the matrix ofg over the two bases has determinant +1.

6.7. SUMMARY 203

6.7 Summary

The main concepts and results of this chapter are listed below:

• One does not solve (large) linear systems by computing determinants.

• Upper-triangular (lower-triangular) matrices.

• Solving by back-substitution (forward-substitution).

• Gaussian elimination.

• Permuting rows.

• The pivot of an elimination step; pivoting.

• Transposition matrix; elementary matrix.

• The Gaussian elimination theorem (Theorem 6.1).

• Gauss-Jordan factorization.

•

LU-factorization; Necessary and sufficient condition for the existence of an

LU -factorization (Proposition 6.2).

• LDU-factorization.

• “P A =LU theorem” (Theorem 6.5).

• LDL -factorization of a symmetric matrix.

• Avoiding small pivots: partial pivoting; complete pivoting.

• Gaussian elimination of tridiagonal matrices.

• LU-factorization of tridiagonal matrices.

• Symmetric positive definite matrices (SPD matrices).

• Cholesky factorization (Theorem 6.10).

• Criteria for a symmetric matrix to be positive definite; Sylvester’s criterion.

• Reduced row echelon form.

• Reduction of a rectangular matrix to its row echelon form.

•

Using the reduction to row echelon form to decide whether a system Ax =b is solvable, and to find its solutions, using a special solution and a basis of the homogeneous system Ax = 0.

• Magic squares.

• transvections and dilatations.

Chapter 7 Vector Norms and Matrix Norms

7.1 Normed Vector Spaces

In order to define how close two vectors or two matrices are, and in order to define the convergence of sequences of vectors or matrices, we can use the notion of a norm. Recall that R+ ={x∈ R|x≥ 0}. Also recall that ifz =a +ib∈ C is a complex number, with a, b∈ R, thenz =a−ib and|z| =√a^2 +b^2 (|z| is the modulus ofz).

Definition 7.1. LetE be a vector space over a fieldK, whereK is either the field R of reals, or the field C of complex numbers. A norm onE is a function :E→ R+, assigning a nonnegative real number u to any vectoru∈E, and satisfying the following conditions for allx, y, z∈E:

(N1) x ≥ 0, and x = 0 iffx = 0. (positivity)

(N2) λx =|λ| x . (scaling)

(N3) x +y ≤ x + y . (triangle inequality)

A vector spaceE together with a norm is called a normed vector space.

From (N3), we easily get |x− y| ≤ x−y . Let us give some examples of normed vector spaces.

Example 7.1.

1. LetE = R, and x =|x|, the absolute value ofx.

2. LetE = C, and z =|z|, the modulus ofz.

205 3. LetE = Rn (orE = Cn). There are three standard norms. For every (x_1, . . . , xn)∈E, we have the norm x_1, defined such that,

x_1 =|x1| +· · · +|xn|,

we have the Euclidean norm x_2, defined such that,

x

2

=

|x1 2 2 1

2

| +· · · +|xn| ,

and the sup-norm x∞, defined such that,

x∞ = max{|xi| | 1≤i≤n}.

More generally, we define the p-norm (forp≥ 1) by

x

p

= (

|

x

1

p p

| +· · · +|xn| )1/p.

There are other norms besides the p-norms; we urge the reader to find such norms. Some work is required to show the triangle inequality for the p-norm. Proposition 7.1. IfE is a finite-dimensional vector space over R or C, for every real numberp≥ 1, the p-norm is indeed a norm.

Proof. The casesp = 1 andp =∞ are easy and left to the reader. Ifp > 1, then letq > 1 such that1 + 1 = 1.p q

We will make use of the following fact: for allα, β∈ R, ifα, β≥ 0, then

p βq

αβ

α

≤ p + q.(∗)

To prove the above inequality, we use the fact that the exponential functiont→et satisfies the following convexity inequality:

eθx+(1−θ)y θex + (1−y)ey,≤

for allx, y∈ R and allθ with 0≤θ≤ 1.

Since the caseαβ = 0 is trivial, let us assume thatα > 0 andβ > 0. If we replaceθ by 1/p,x byplogα andy byqlogβ, then we get

e

1p log α+1q log β^1 p q p log α + 1qeq log β,≤ pe

which simplifies top βq αβ

α

≤ p + q,

as claimed.

We will now prove that for any two vectorsu, v∈E, we have

n

i=1|uivi| ≤ u p v q. (∗∗)

Since the above is trivial ifu = 0 orv = 0, let us assume thatu = 0 andv = 0. Then, the inequality (∗) withα =|ui|/ u p andβ =|vi|/ v q yields

|uivi|

p vi| q,u p v q≤ |ui q

|

up q p + | uq

fori = 1, . . . , n, and by summing up these inequalities, we get

n

i=1|uivi| ≤ u p v q,

as claimed. To finish the proof, we simply have to prove that property (N3) holds, since (N1) and (N2) are clear. Now, fori = 1, . . . , n, we can write

(|ui| +|vi|)p =|ui|(|ui| +|vi|)p−1 +|vi|(|ui| +|vi|)p−1,

so that by summing up these equations we get

n n n

(|ui| +|vi|)p =|ui|(|ui| +|vi|)p−1 +|vi|(|ui| +|vi|)p−1,

i=1 i=1 i=1

and using the inequality (∗∗), we get

n n 1/q

(|ui| +|vi|)p ( u p + v p) (|ui| +|vi|)(p−1)q .

i=1≤ i=1

However, 1/p + 1/q = 1 impliespq =p +q, that is, (p− 1)q =p, so we have

n n 1/q

(|ui| +|vi|)p ( u p + v p) (|ui| +|vi|)p ,

i=1≤ i=1

which yields

n 1/p

(|ui| +|vi|)p u p + v p.

i=1≤

Since|ui +vi| ≤ |ui|+|vi|, the above implies the triangle inequality u +v p≤ u p + v p, as claimed.

Forp > 1 and 1/p + 1/q = 1, the inequality

n n 1/p n 1/q

u

i

v

i

| ≤

i

=1

|

u

i

p q

| i=1|vi|i=1|

is known as H¨older’s inequality. Forp = 2, it is the Cauchy–Schwarz inequality. Actually, if we define the Hermitian inner product−,− on Cn by

n

u, v = uivi,

i=1

whereu = (u_1, . . . , un) andv = (v_1, . . . , vn), then

n n

u, v| ≤ i=1|uivi| =|uivi|,| i=1

so H¨older’s inequality implies the inequality

|u, v| ≤ u p v q

also called H¨older’s inequality, which, forp = 2 is the standard Cauchy–Schwarz inequality. The triangle inequality for the p-norm,

n 1/p n 1/p n 1/q

(

|

u

i

+

v

i

|

)

pui p q

|

+|vi| ,

i=1≤ i=1| i=1

is known as Minkowski’s inequality.

When we restrict the Hermitian inner product to real vectors,u, v∈ Rn, we get the Euclidean inner product

n

u, v = uivi.

i=1

It is very useful to observe that if we represent (as usual)u = (u_1, . . . , un) andv = (v_1, . . . , vn) (in Rn) by column vectors, then their Euclidean inner product is given by

u, v =u v =v u,

and whenu, v∈ Cn, their Hermitian inner product is given by u, v =v∗u =u∗v.

In particular, whenu =v, in the complex case we get u^2 =u∗u,2 and in the real case, this becomes u^2 =u u.2

As convenient as these notations are, we still recommend that you do not abuse them; the notation u, v is more intrinsic and still “works” when our vector space is infinite dimensional.

The following proposition is easy to show.

Proposition 7.2. The following inequalities hold for allx∈ Rn (orx∈ Cn):

x∞≤ x1≤n x∞,

x∞≤ x_2≤√n x∞,

x2≤ x_1≤√n x_2.

Proposition 7.2 is actually a special case of a very important result: in a finite-dimensional vector space, any two norms are equivalent.

Definition 7.2. Given any (real or complex) vector spaceE, two normsa andb are equivalent iff there exists some positive realsC_1, C_2> 0, such that

u a≤C_1 u b and u b≤C_2 u a, for allu∈E.

Given any norm on a vector space of dimensionn, for any basis (e_1, . . . , en) ofE, observe that for any vectorx =x_1e_1 +· · · +xnen, we have

x = x_1e_1 +· · · +xnen ≤ |x1| e_1 +· · · +|xn| en ≤C(|x1| +· · · +|xn|) =C x_1, withC = max1≤i≤n ei and x_1 = x_1e_1 +· · · +xnen =|x1| +· · · +|xn|. The above implies that | u− v| ≤ u−v ≤C u−v_1,

which means that the mapu→ u is continuous with respect to the norm_1. LetSn−1 be the unit ball with respect to the norm_1, namely_1

Sn−1 ={x∈E| x_1 = 1}.1

Now,S

Borel (or equivalently, by Bolzano–Weiertrass),S

n−1 is a closed and bounded subset of a finite-dimensional vector space, so by Heine–1 n−1 is compact. On the other hand, it1

is a well known result of analysis that any continuous real-valued function on a nonempty compact set has a minimum and a maximum, and that they are achieved. Using these facts, we can prove the following important theorem:

Theorem 7.3. IfE is any real or complex vector space of finite dimension, then any two norms onE are equivalent.

Proof. It is enough to prove that any norm is equivalent to the 1-norm. We already proved that the functionx→ x is continuous with respect to the norm_1 and we observed that the unit ballSn−1 is compact. Now, we just recalled that because the functionf :x→ x is_1

continuous and becauseS

M, and because x is never zero onS

n−1 is compact, the functionf has a minimumm and a maximum1 n−1, we must havem > 0. Consequently, we just1

proved that if x_1 = 1, then

0< m≤ x ≤M,

so for any x∈E withx = 0, we get

m≤ x/ x1 ≤M,

which implies m x1≤ x ≤M x_1.

Since the above inequality holds trivially ifx = 0, we just proved that and_1 are equivalent, as claimed.

Next, we will consider norms on matrices.

7.2 Matrix Norms

For simplicity of exposition, we will consider the vector spaces Mn(R) and Mn(C) of square n×n matrices. Most results also hold for the spaces Mm,n(R) and Mm,n(C) of rectangular m×n matrices. Sincen×n matrices can be multiplied, the idea behind matrix norms is that they should behave “well” with respect to matrix multiplication.

Definition 7.3. A matrix norm on the space of squaren×n matrices in Mn(K), with K = R orK = C, is a norm on the vector space Mn(K) with the additional property that AB ≤ A B , for allA, B∈ Mn(K).

SinceI^2 =I, from I = I^2 I^2, we get I ≥ 1, for every matrix norm.≤

Before giving examples of matrix norms, we need to review some basic definitions about matrices. Given any matrixA = (aij)∈ Mm,n(C), the conjugateA ofA is the matrix such that

Aij =aij, 1≤i≤m, 1≤j≤n. The transpose ofA is then×m matrixA such that Aij =aji, 1≤i≤m, 1≤j≤n. The adjoint ofA is then×m matrixA∗ such that

A∗ = (A ) = (A) .

WhenA is a real matrix,A∗ =A . A matrixA∈ Mn(C) is Hermitian if

A∗ =A.

IfA is a real matrix (A∈ Mn(R)), we say thatA is symmetric if

A =A.

A matrixA∈ Mn(C) is normal if AA∗ =A∗A, and ifA is a real matrix, it is normal if

AA =A A.

A matrixU∈ Mn(C) is unitary if

UU∗ =U∗U =I.

A real matrixQ∈ Mn(R) is orthogonal if

QQ =Q Q =I.

Given any matrixA = (aij)∈ Mn(C), the trace tr(A) ofA is the sum of its diagonal elements

tr(A) =a_11 +· · · +ann. It is easy to show that the trace is a linear map, so that

tr(λA) =λtr(A)

and tr(A +B) = tr(A) + tr(B).

Moreover, ifA is anm×n matrix andB is ann×m matrix, it is not hard to show that

tr(AB) = tr(BA).

We also review eigenvalues and eigenvectors. We content ourselves with definition involving matrices. A more general treatment will be given later on (see Chapter 12). Definition 7.4. Given any square matrixA∈ Mn(C), a complex numberλ∈ C is an eigenvalue ofA if there is some nonzero vectoru∈ Cn, such that

Au =λu.

If λ is an eigenvalue ofA, then the nonzero vectorsu∈ Cn such thatAu =λu are called eigenvectors ofA associated withλ; together with the zero vector, these eigenvectors form a subspace of Cn denoted byEλ(A), and called the eigenspace associated withλ.

Remark: Note that Definition 7.4 requires an eigenvector to be nonzero. A somewhat unfortunate consequence of this requirement is that the set of eigenvectors is not a subspace, since the zero vector is missing! On the positive side, whenever eigenvectors are involved, there is no need to say that they are nonzero. The fact that eigenvectors are nonzero is implicitly used in all the arguments involving them, so it seems safer (but perhaps not as elegant) to stituplate that eigenvectors should be nonzero.

If A is a square real matrixA∈ Mn(R), then we restrict Definition 7.4 to real eigenvalues λ∈ R and real eigenvectors. However, it should be noted that although every complex matrix always has at least some complex eigenvalue, a real matrix may not have any real eigenvalues. For example, the matrix

A

=

0 −1 1_0

has the complex eigenvaluesi and−i, but no real eigenvalues. Thus, typically, even for real matrices, we consider complex eigenvalues.

Observe that λ∈ C is an eigenvalue ofA

iffAu =λu for some nonzero vectoru∈ Cn

iff (λI−A)u = 0

iff the matrixλI−A defines a linear map which has a nonzero kernel, that is, iffλI−A not invertible.

However, from Proposition 5.10,λI−A is not invertible iff

det(λI−A) = 0.

Now, det(λI− A) is a polynomial of degreen in the indeterminateλ, in fact, of the form

λn tr(A)λn−1 +· · · + (−1)n det(A).−

Thus, we see that the eigenvalues of A are the zeros (also called roots) of the above polynomial. Since every complex polynomial of degreen has exactlyn roots, counted with their multiplicity, we have the following definition:

Definition 7.5. Given any squaren×n matrixA∈ Mn(C), the polynomial

det(λI−A) =λn tr(A)λn−1 +· · · + (−1)n det(A)−

is called the characteristic polynomial ofA. Then (not necessarily distinct) rootsλ_1, . . . , λn of the characteristic polynomial are all the eigenvalues ofA and constitute the spectrum of A. We let

ρ(A) = max|λi

1 |≤i≤n

be the largest modulus of the eigenvalues ofA, called the spectral radius ofA. Proposition 7.4. For any matrix norm on Mn(C) or Mn(R), and for any squaren×n matrixA, we have

ρ(A)≤ A .

Proof. First, let us consider the case whereA is a complex matrix, since it is simpler. Letλ be some eigenvalue ofA for which|λ| is maximum, that is, such that|λ| =ρ(A). Ifu (= 0) is any eigenvector associated withλ and ifU is then×n matrix whose columns are allu, thenAu =λu implies

AU =λU, and since

|λ| U = λU = AU ≤ A U andU = 0, we have U = 0, and get

ρ(A) =|λ| ≤ A ,

as claimed.

If A is a real matrix, the problem is that even if there is a real eigenvalueλ such that ρ(A) =|λ|, corresponding eigenvectors may be complex. We use a trick based on the fact that for every matrixA (real or complex),

ρ(Ak) = (ρ(A))k,

which is left as a simple exercise.

Pick any complex normc on Cn and letc denote the corresponding induced norm on matrices. The restriction ofc to real matrices is a real norm that we also denote by c. Now, by Theorem 7.3, since Mn(R) has finite dimensionn^2, there is some constant C > 0 so that

A c≤C A , for all A∈ Mn(R).

Furthermore, for everyk≥ 1 and for every realn×n matrixA, by the previous part, ρ(Ak)≤ Ak , and because is a matrix norm, A≤ A k, so we havec

(ρ(A))k =ρ(Ak)≤ Ak C Ak C A k,c≤ ≤ for allk≥ 1. It follows that ρ(A)≤C1/k A , for all k≥ 1.

1/k = 1 (we have limk→∞ k log(C) = 0). Therek→∞C^1However becauseC > 0, we have lim

fore, we conclude that

ρ(A)≤ A , as desired.

Now, it turns out that ifA is a realn×n symmetric matrix, then the eigenvalues ofA are all real and there is some orthogonal matrixQ such that

A =Q diag(λ_1, . . . , λn)Q,

where diag( λ_1, . . . , λn) denotes the matrix whose only nonzero entries (if any) are its diagonal entries, which are the (real) eigenvalues ofA. Similarly, ifA is a complexn×n Hermitian matrix, then the eigenvalues ofA are all real and there is some unitary matrixU such that

A =U∗diag(λ_1, . . . , λn)U, where diag(λ_1, . . . , λn) denotes the matrix whose only nonzero entries (if any) are its diagonal entries, which are the (real) eigenvalues ofA.

We now return to matrix norms. We begin with the so-called Frobenius norm, which is just the norm_2 on Cn2, where then×n matrixA is viewed as the vector obtained by concatenating together the rows (or the columns) ofA. The reader should check that for anyn×n complex matrixA = (aij),

n 1/2

a

ij

2

|

= tr(A∗A) = tr(AA∗).

i,j=1|

Definition 7.6. The Frobenius normF is defined so that for every squaren×n matrix A∈ Mn(C),

n 1/2

A

F

=

|

a

ij

2

|

= tr(AA∗) = tr(A∗A).

i,j=1

The following proposition show that the Frobenius norm is a matrix norm satisfying other nice properties.

Proposition 7.5. The Frobenius normF on Mn(C) satisfies the following properties: (1) It is a matrix norm; that is, AB F≤ A F B F, for allA, B∈ Mn(C). (2) It is unitarily invariant, which means that for all unitary matricesU, V , we have

A F = UA F = AV F = UAV F. (3) ρ(A∗A)≤ A F≤√n ρ(A∗A), for allA∈ Mn(C).

Proof. (1) The only property that requires a proof is the fact AB F≤ A F B F. This follows from the Cauchy–Schwarz inequality:

n n 2

AB^2 = aikbkjF

i,j=1 k=1

n n n

2 2

≤ aih| k=1|bkj|i,j=1 h=1|

n n

=

|

a

ih

2 2

|

k,j

=1

|

bkj| = A^2 B^2.F F

i,h=1

(2) We have

A^2 = tr(A∗A) = tr(V V∗A∗A) = tr(V∗A∗AV ) = AV^2,F F and A^2 = tr(A∗A) = tr(A∗U∗UA) = UA^2.F F The identity

A F = UAV F

follows from the previous two.

(3) It is well known that the trace of a matrix is equal to the sum of its eigenvalues. Furthermore,A∗A is symmetric positive semidefinite (which means that its eigenvalues are nonnegative), soρ(A∗A) is the largest eigenvalue ofA∗A and

ρ(A∗A)≤ tr(A∗A)≤nρ(A∗A),

which yields (3) by taking square roots.

Remark: The Frobenius norm is also known as the Hilbert-Schmidt norm or the Schur norm. So many famous names associated with such a simple thing!

We now give another method for obtaining matrix norms using subordinate norms. First, we need a proposition that shows that in a finite-dimensional space, the linear map induced by a matrix is bounded, and thus continuous.

Proposition 7.6. For every norm on Cn (or Rn), for every matrixA∈ Mn(C) (or A∈ Mn(R)), there is a real constantCA> 0, such that

Au ≤CA u , for every vectoru∈ Cn (oru∈ Rn ifA is real). Proof. For every basis (e_1, . . . , en) of Cn (or Rn), for every vectoru =u_1e_1 +· · · +unen, we have

Au = u_1A(e_1) +· · ·unA(en)

u1| A(e1) +· · · +|un| A(en)≤ | (|u1| +· · · +|un|) =C_1 u_1,≤C_1

whereC_1 = max1≤i≤n A(ei) . By Theorem 7.3, the norms and_1 are equivalent, so there is some constantC_2> 0 so that u1≤C_2 u for allu, which implies that Au ≤CA u , whereCA =C_1C_2.

Proposition 7.6 says that every linear map on a finite-dimensional space is bounded. This implies that every linear map on a finite-dimensional space is continuous. Actually, it is not hard to show that a linear map on a normed vector spaceE is bounded iff it is continuous, regardless of the dimension ofE.

Proposition 7.6 implies that for every matrixA∈ Mn(C) (orA∈ Mn(R)),

supAx CA.

x ∈Cn x ≤

x =0

Now, since λu =|λ| u , for every nonzero vectorx, we have Ax= x A(x/ x )= A(x/ x ) x x (x/ x ) (x/ x ) , which implies that supAx = sup Ax .

x∈Cn x x∈Cn x =0 x =1

Similarly

supAx = sup Ax .

x∈Rn x x∈Rn

x =0 x =1

The above considerations justify the following definition.

Definition 7.7. If is any norm on Cn, we define the function on Mn(C) by

A = supAx = sup Ax .

x∈Cn x x∈Cn

x =0 x =1

The functionA→ A is called the subordinate matrix norm or operator norm induced by the norm .

It is easy to check that the functionA→ A is indeed a norm, and by definition, it satisfies the property

Ax ≤ A x , for allx∈ Cn. Consequently

ABx ≤ A Bx ≤ A B x , for allx∈ Cn, which implies that

AB ≤ A B

for allA, B∈ Mn(C), showing thatA→ A is a matrix norm. Observe that the subordinate matrix norm is also defined by

A = inf{λ∈ R| Ax ≤λ x , for allx∈ Cn .}

The definition also implies that

I = 1.

The above show that the Frobenius norm is not a subordinate matrix norm (why?). Remark: For any norm on Cn, we can define the functionR on Mn(R) by

AAx = sup Ax . R = supx x ∈Rnx∈Rn

x =0 x =1

The function A→ A R is a matrix norm on Mn(R), and

A R≤ A ,

for all real matricesA∈ Mn(R). However, it is possible to construct vector norms on Cn and real matricesA such that

A R< A .

In order to avoid this kind of difficulties, we define subordinate matrix norms over Mn(C). Luckily, it turns out that A R = A for the vector norms,1,2, and .

∞

We now determine explicitly what are the subordinate matrix norms associated with the vector norms_1,2, and .

∞

Proposition 7.7. For every square matrixA = (aij)∈ Mn(C), we have

n

A_1 = sup Ax_1 = max|aij

x |

x∈Cn j i=1_1=1

n

A = sup Ax = max|aij

∞ x | x

∈Cn ∞ i j=1 ∞=1

A_2 = sup Ax_2 = ρ(A∗A) = ρ(AA∗).

x∈Cn

x_2=1

Furthermore, A∗ 2 = A_2, the norm_2 is unitarily invariant, which means that

A_2 = UAV_2

for all unitary matricesU, V , and ifA is a normal matrix, then A_2 =ρ(A). Proof. For every vectoru, we have

Au_1 = aijuj≤ j |uj| i |aij| ≤ max |aij| u_1,

i jj i

which implies thatn

A1≤ max|aij|.

j i=1

It remains to show that equality can be achieved. For this letj_0 be some index such that

max |aij| = |aij0|,

j i i

and letui = 0 for alli =j_0 anduj0 = 1. In a similar way, we have

Au = max aijuj≤ max |aij| u ,

∞ ij ij ∞

which implies thatn A≤ max|aij|.

∞ i j=1

To achieve equality, leti_0 be some index such that

max |aij| = |ai0j|.

i j j

The reader should check that the vector given by

ai0j ifai0j = 0uj = |ai0j

1| ifai0j = 0

works.

We have

A 2 x∈CnCn

x∗

2 = sup Ax^2 = sup x∗A∗Ax.2 x∈x=1 x∗x=1 Since the matrixA∗A is symmetric, it has real eigenvalues and it can be diagonalized with respect to an orthogonal matrix. These facts can be used to prove that the functionx→x∗A∗Ax has a maximum on the spherex∗x = 1 equal to the largest eigenvalue ofA∗A,

namely,ρ(A∗A). We postpone the proof until we discuss optimizing quadratic functions. Therefore,

A_2 = ρ(A∗A).

Let use now prove thatρ(A∗A) =ρ(AA∗). First, assume thatρ(A∗A)> 0. In this case, there is some eigenvectoru(= 0) such that

A∗Au =ρ(A∗A)u,

and sinceρ(A∗A)> 0, we must haveAu = 0. SinceAu = 0,

AA∗(Au) =ρ(A∗A)Au

which means thatρ(A∗A) is an eigenvalue ofAA∗, and thus

ρ(A∗A)≤ρ(AA∗).

Because ( A∗)∗ =A, by replacingA byA∗, we get

ρ(AA∗)≤ρ(A∗A),

and soρ(A∗A) =ρ(AA∗).

Ifρ(A∗A) = 0, then we must haveρ(AA∗) = 0, since otherwise by the previous reasoning we would haveρ(A∗A) =ρ(AA∗)> 0. Hence, in all case

A^2 =ρ(A∗A) =ρ(AA∗) = A∗2.2 2

For any unitary matricesU andV , it is an easy exercise to prove thatV∗A∗AV andA∗A have the same eigenvalues, so

A^2 =ρ(A∗A) =ρ(V∗A∗AV ) = AV^2,2 2

and also

A^2 =ρ(A∗A) =ρ(A∗U∗UA) = UA^2.2 2

Finally, ifA is a normal matrix (AA∗ =A∗A), it can be shown that there is some unitary matrixU so that

A =U∗DU,

whereD = diag(λ_1, . . . , λn) is a diagonal matrix consisting of the eigenvalues ofA, and thus A∗A = (U∗DU)∗U∗DU =U∗D∗UU∗DU =U∗D∗DU.

However,

D

∗

D

= diag(

|

λ

1

2 2

| , . . . ,|λn| ), which proves that

ρ

(

A

∗

A

) =

ρ

(

D

∗

D

) = max

λ

i

2

|

= (ρ(A))2,

i |

so that A_2 =ρ(A).

The norm A_2 is often called the spectral norm. Observe that property (3) of proposition 7.5 says that

A2≤ A F≤√n A_2,

which shows that the Frobenius norm is an upper bound on the spectral norm. The Frobenius norm is much easier to compute than the spectal norm.

The reader will check that the above proof still holds if the matrix A is real, confirming the fact that A R = A for the vector norms_1,2, and

that the proof goes through for rectangular matrices, with the same formulae.∞

. It is also easy to verify

Similarly, the Frobenius norm is also a norm on rectangular matrices. For these norms, wheneverAB makes sense, we have

AB ≤ A B .

Remark: Let (E, ) and (F, ) be two normed vector spaces (for simplicity of notation, we use the same symbol for the norms onE andF; this should not cause any confusion). Recall that a functionf :E→F is continuous if for everya∈E, for every > 0, there is someη > 0 such that for allx∈E,

if x−a ≤η then f(x)−f(a) ≤.

It is not hard to show that a linear mapf :E→F is continuous iff there is some constant C > 0 such that

f(x) ≤C x for allx∈E.

If so, we say that f is bounded (or a linear bounded operator). We letL(E;F) denote the set of all continuous (equivalently, bounded) linear maps fromE toF. Then, we can define the operator norm (or subordinate norm) onL(E;F) as follows: for everyf∈ L(E;F),

f = supf(x) = sup f(x) ,

x∈E x x∈Ex =0 x =1

or equivalently by f = inf{λ∈ R| f(x) ≤λ x , for allx∈E}.

It is not hard to show that the mapf→ f is a norm onL(E;F) satisfying the property f(x) ≤ f x

for allx∈E, and that iff∈ L(E;F) andg∈ L(F;G), then

gæf ≤ g f .

Operator norms play an important role in functional analysis, especially when the spacesE andF are complete.

The following proposition will be needed when we deal with the condition number of a matrix.

Proposition 7.8. Let be any matrix norm and letB be a matrix such that B < 1.

(1) If is a subordinate matrix norm, then the matrixI +B is invertible and

(I +B)−1 1 .≤ 1− B

(2) If a matrix of the formI +B is singular, then B ≥ 1 for every matrix norm (not necessarily subordinate).

Proof. (1) Observe that (I +B)u = 0 impliesBu =−u, so

u = Bu .

Recall that Bu ≤ B u for every subordinate norm. Since B < 1, ifu = 0, then

Bu < u ,

which contradicts u = Bu . Therefore, we must haveu = 0, which proves thatI +B is injective, and thus bijective, i.e., invertible. Then, we have

(I +B)−1 +B(I +B)−1 = (I +B)(I +B)−1 =I,

so we get (I +B)−1 =I−B(I +B)−1, which yields

(I +B)−1 1 + B (I +B)−1 ,≤

and finally,

(I +B)−1 1 .≤ 1− B

(2) IfI +B is singular, then−1 is an eigenvalue ofB, and by Proposition 7.4, we get ρ(B)≤ B , which implies 1≤ρ(B)≤ B .

The following result is needed to deal with the convergence of sequences of powers of matrices.

Proposition 7.9. For every matrixA∈ Mn(C) and for every > 0, there is some subordinate matrix norm such that

A ≤ρ(A) +. Proof. By Theorem 12.4, there exists some invertible matrixU and some upper triangular matrixT such that

A =UT U−1,

and say thatëλ1 t12 t13 · · · t1n ö

ì 0 λ_2 t23 · · · t2n ÷

÷

T

=

ì

ì

í

ì .... . ÷,

ì

. . ÷

0

0

· · ·

λ

n

−

1

t

÷

n−1 nø

0 0 · · · 0 λn

whereλ_1, . . . , λn are the eigenvalues ofA. For everyδ = 0, define the diagonal matrix Dδ = diag(1, δ, δ^2, . . . , δ−1),

and consider the matrix

ëλ_1 δt_12 δ^2t13 · · · δn−1t1nö ì 0 λ_2 δt23 · · · δn−2t2n÷

÷

(UD

ì .... .÷ .δ)−1A(UDδ) =D− 1 T Dδ =ì

÷

í

.

.

÷δ ì

ì 0 0 · · · λn −1 δtn−1 nø0 0 · · · 0 λn

Now, define the function : Mn(C)→ R by

B = (UDδ)−1B(UDδ) ,

∞

for everyB∈ Mn(C). Then it is easy to verify that the above function is the matrix norm subordinate to the vector norm

v→ (UDδ)−1v .

∞

Furthermore, for every > 0, we can pickδ so that

n

j=i+1|δj−itij| ≤, 1≤i≤n− 1,

and by definition of the norm , we get

∞

A ≤ρ(A) +,

which shows that the norm that we have constructed satisfies the required properties.

Note that equality is generally not possible; consider the matrix

A =0 1 ,0 0

for whichρ(A) = 0< A , sinceA = 0.

7.3 Condition Numbers of Matrices

Unfortunately, there exist linear systemsAx =b whose solutions are not stable under small perturbations of eitherb orA. For example, consider the system

ë10 7 8 7ö ëx1ö ë_32ö

ì 7 5 6 5÷ ìx_2÷ = ì23÷

ì ÷ ì .í 8 6 10 9÷ ìx3ø í33÷

ø í ø

7 5 9 10 x_4 31

The reader should check that it has the solutionx = (1,1,1,1). If we perturb slightly the right-hand side, obtaining the new system

ë 10 7 8 7ö ëx_1 + x_1ö ë32.1ö

ì 7 5 6 5 ÷ ìx_2 + x_2÷ = ì22.9÷

ì ÷ ì ,í 8 6 10 9÷ ìx_3 + x_3ø í33.1÷

ø í ø

7 5 9 10 x_4 + x_4 30.9

the new solutions turns out to be x = (9.2,−12.6,4.5,−1.1). In other words, a relative error of the order 1/200 in the data (here,b) produces a relative error of the order 10/1 in the solution, which represents an amplification of the relative error of the order 2000.

Now, let us perturb the matrix slightly, obtaining the new system

ë 10 7 8.1 7.2öëx_1 + x_1ö ë32ö

ì7.08 5.04 6 5 ÷ìx_2 + x_2÷ = ì23÷

ì ÷ ì .í 8 5.98 9.98 9÷ìx_3 + x_3ø í33÷

øí ø

6.99 4.99 9 9.98 x_4 + x_4 31

This time, the solution is x = (−81,137,−34,22). Again, a small change in the data alters the result rather drastically. Yet, the original system is symmetric, has determinant 1, and has integer entries. The problem is that the matrix of the system is badly conditioned, a concept that we will now explain.

Given an invertible matrixA, first, assume that we perturbb tob+δb, and let us analyze the change between the two exact solutionsx andx +δx of the two systems

Ax =b

A(x +δx) =b +δb.

We also assume that we have some norm and we use the subordinate matrix norm on matrices. From

Ax =b

Ax +Aδx =b +δb, we get δx =A−1δb, and we conclude that

δx ≤ A−1 δb b ≤ A x .

Consequently, the relative error in the result δx / x is bounded in terms of the relative error δb / b in the data as follows:

δx A A−1 δb .x ≤ b

Now let us assume thatA is perturbed toA+δA, and let us analyze the change between the exact solutions of the two systems

Ax =b (A + A)(x + x) =b.

The second equation yieldsAx +Ax + A(x + x) =b, and by subtracting the first equation we get

x =−A−1A(x + x). It follows that

x ≤ A−1 A x + x , which can be rewritten as

x A A−1 A x + x ≤ A .

Observe that the above reasoning is valid even if the matrix A + A is singular, as long asx + x is a solution of the second system. Furthermore, if A

not unreasonable to expect that the ratio x / x + x is close to

be made more precise later.

is small enough, it is

x / x . This will

In summary, for each of the two perturbations, we see that the relative error in the result is bounded by the relative error in the data, multiplied the number A A−1 . In fact, this factor turns out to be optimal and this suggests the following definition:

Definition 7.8. For any subordinate matrix norm , for any invertible matrixA, the number

cond(A) = A A−1 is called the condition number ofA relative to .

The condition number cond( A) measures the sensitivity of the linear systemAx =b to variations in the datab andA; a feature referred to as the condition of the system. Thus, when we says that a linear system is ill-conditioned, we mean that the condition number of its matrix is large. We can sharpen the preceding analysis as follows:

Proposition 7.10. LetA be an invertible matrix and letx andx +δx be the solutions of the linear systems

Ax =b A(x +δx) =b +δb. Ifb = 0, then the inequality

δx cond(A) δb

x ≤ b

holds and is the best possible. This means that for a given matrixA, there exist some vectors b = 0 andδb = 0 for which equality holds.

Proof. We already proved the inequality. Now, because is a subordinate matrix norm, there exist some vectorsx = 0 andδb = 0 for which

A−1δb = A−1 δb and Ax = A x .

Proposition 7.11. LetA be an invertible matrix and letx andx + x be the solutions of the two systems

Ax =b (A + A)(x + x) =b.

Ifb = 0, then the inequality

x cond(A) A

x + x ≤ A

holds and is the best possible. This means that given a matrix A, there exist a vectorb = 0 and a matrix A = 0 for which equality holds. Furthermore, if A is small enough (for instance, if A < 1/ A−1 ), we have

x cond(A) A (1 +O( A ));x ≤ A

in fact, we have

x cond(A) A 1 .x ≤ A 1− A−1 A Proof. The first inequality has already been proved. To show that equality can be achieved, letw be any vector such thatw = 0 and

A−1w = A−1 w ,

and letβ = 0 be any real number. Now, the vectors x =−βA−1w x + x =w

b = (A +βI)w and the matrix A =βI sastisfy the equations

Ax =b

(A + A)(x + x) =b

x =|β| A−1w = A A−1 x + x . Finally, we can pickβ so that−β is not equal to any of the eigenvalues ofA, so that A + A =A +βI is invertible andb is is nonzero.

If A < 1/ A−1 , then

A−1A ≤ A−1 A < 1,

so by Proposition 7.8, the matrixI +A−1A is invertible and

(I +A−1A)−1 1 1 .≤ 1− A−1A ≤ 1− A−1 A Recall that we proved earlier that

x =−A−1A(x + x), and by addingx to both sides and moving the right-hand side to the left-hand side yields (I +A−1A)(x + x) =x,

and thus x + x = (I +A−1A)−1x,

which yields

x = ((I +A−1A)−1 I)x = (I +A−1A)−1(I− (I +A−1A))x =

−

(

I

+

A

−

1

A

)

−

1

−

A−1(A)x.

From this and

(I +A−1A)−1 1 ,≤ 1− A−1 A

we get

x A−1 A x , ≤ 1− A−1 A which can be written as

x cond(A) A 1 ,x ≤ A 1− A−1 A

which is the kind of inequality that we were seeking.

Remark: IfA andb are perturbed simultaneously, so that we get the “perturbed” system

(A + A)(x +δx) =b +δb,

it can be shown that if A < 1/ A−1 (andb = 0), then x cond(A) A+ δb;x ≤ 1− A−1 A A b

see Demmel [25], Section 2.2 and Horn and Johnson [55], Section 5.8.

We now list some properties of condition numbers and figure out what cond( A) is in the case of the spectral norm (the matrix norm induced by_2). First, we need to introduce a very important factorization of matrices, the singular value decomposition, for short, SVD.

It can be shown that given any n×n matrixA∈ Mn(C), there exist two unitary matrices U andV , and a real diagonal matrix Σ = diag(σ1, . . . , σn), withσ1≥σ2≥ · · · ≥σn≥ 0, such that

A =V ΣU∗.

The nonnegative numbersσ_1, . . . , σn are called the singular values ofA.

IfA is a real matrix, the matricesU andV are orthogonal matrices. The factorization A =V ΣU∗ implies that

A∗A =UΣ^2U∗ and AA∗ =V Σ^2V∗,

which shows that σ^2, . . . , σ^2 are the eigenvalues of bothA∗A andAA∗, that the columns ofU_1

are corresponding eivenvectors forA∗A, and that the columns ofV are corresponding eivenvectors forAA∗. In the case of a normal matrix ifλ_1, . . . , λn are the (complex) eigenvalues ofA, then

σi =|λi|, 1≤i≤n. Proposition 7.12. For every invertible matrixA∈ Mn(C), the following properties hold: (1)

cond(A)≥ 1,

cond(A) = cond(A−1)

cond(αA) = cond(A) for allα∈ C− {0}. (2) If cond_2(A) denotes the condition number ofA with respect to the spectral norm, then

cond_2(A) =σ1 ,σn

whereσ1≥ · · · ≥σn are the singular values ofA. (3) If the matrixA is normal, then

cond_2(A) =|λ1|,

|λn|

whereλ_1, . . . , λn are the eigenvalues ofA sorted so that|λ1| ≥ · · · ≥ |λn|. (4) IfA is a unitary or an orthogonal matrix, then

cond_2(A) = 1.

(5) The condition number cond_2(A) is invariant under unitary transformations, which means that

cond_2(A) = cond_2(UA) = cond_2(AV ),

for all unitary matricesU andV .

Proof. The properties in (1) are immediate consequences of the properties of subordinate matrix norms. In particular,AA−1 =I implies

1 = I ≤ A A−1 = cond(A).

(2) We showed earlier that A^2 =ρ(A∗A), which is the square of the modulus of the largest_2

eigenvalue ofA^2 σ^2, where∗A. Since we just saw that the eigenvalues ofA∗A areσ1≥ · · · ≥σ1, . . . , σn are the singular values ofA, we have

A_2 =σ_1.

Now, ifA is invertible, thenσ1≥ · · · ≥σn> 0, and it is easy to show that the eigenvalues of (

A

∗

A

)

−

1

are

σ

−

2

σ

2

1

, which shows that

n ≥ · · · ≥

A

−

1 =σ−1

2

,

n

and thus

condσ1.2(A) =σn

(3) This follows from the fact that A_2 =ρ(A) for a normal matrix.

(4) If A is a unitary matrix, then A∗A = AA∗ = I, so ρ(A∗A) = 1, and A_2 = ρ(A∗A) = 1. We also have A−1 2 = A∗ 2 = ρ(AA∗) = 1, and thus cond(A) = 1. (5) This follows immediately from the unitary invariance of the spectral norm.

Proposition 7.12 (4) shows that unitary and orthogonal transformations are very wellconditioned, and part (5) shows that unitary transformations preserve the condition number. In order to compute cond_2(A), we need to compute the top and bottom singular values ofA, which may be hard. The inequality

A2≤ A F≤√n A_2,

may be useful in getting an approximation of cond_2(A) = A_2 A−1 , if A−1 can be_2 determined.

Remark: There is an interesting geometric characterization of cond_2(A). Ifθ(A) denotes the least angle between the vectorsAu andAv asu andv range over all pairs of orthonormal vectors, then it can be shown that

cond_2(A) = cot(θ(A)/2)).

Thus, if A is nearly singular, then there will be some orthonormal pairu, v such thatAu andAv are nearly parallel; the angleθ(A) will the be small and cot(θ(A)/2)) will be large. For more details, see Horn and Johnson [55] (Section 5.8 and Section 7.4).

It should be noted that in general (if A is not a normal matrix) a matrix could have a very large condition number even if all its eigenvalues are identical! For example, if we consider then×n matrix

ë1 2 0 0 . . . 0 0ö ì0 1 2 0 . . . 0 0÷

ì ÷

ì0 0 1 2 . . . 0 0÷

÷

A

=

ì

ì

ì ... ... .... .÷,

ì

. . ÷

0

0

. . .

0

1

2

0

÷ ì ÷

ì0 0 . . . 0 0 1 2÷ í ø 0 0 . . . 0 0 0 1

it turns out that cond_2(A)≥ 2n−1. A classical example of matrix with a very large condition number is the Hilbert matrix H(n), then×n matrix with

1

H(n) =i +j− 1 .ij

For example, whenn = 5,ë1 1 1 1 1ö

2 3 4 5 ì1 1 1 1 1÷ ì2 3 4 5 6÷

H

(5)

=

ì1 1 1 1 1÷ ì .ì 3 4 5 6 7÷ ÷ ì1 1 1 1 1÷

í4 5 6 7 8ø

1 1 1 1 1 5 6 7 8 9

It can be shown that

cond_2(H(5))≈ 4.77× 10^5.

Hilbert introduced these matrices in 1894 while studying a problem in approximation theory. The Hilbert matrixH(n) is symmetric positive definite. A closed-form formula can be given for its determinant (it is a special form of the so-called Cauchy determinant). The inverse ofH(n) can also be computed explicitly! It can be shown that

cond_2(H(n)) =O((1 +√2)4n/√n).

Going back to our matrixë10 7 8 7 ö

ì 7 5 6 5 ÷,í 8 6 10 9÷A = ì

ø

7 5 9 10

which is a symmetric, positive, definite, matrix, it can be shown that its eigenvalues, which in this case are also its singular values becauseA is SPD, are

λ1≈ 30.2887> λ2≈ 3.858> λ3≈ 0.8431> λ4≈ 0.01015,

so that

condλ1 2984.2(A) =λ4≈

The reader should check that for the perturbation of the right-hand sideb used earlier, the relative errors δx / x and δx / x satisfy the inequality

δx cond(A) δb x ≤ b and comes close to equality. 7.4. AN APPLICATION OF NORMS: INCONSISTENT LINEAR SYSTEMS 231

7.4 An Application of Norms: Solving Inconsistent Linear Systems

The problem of solving an inconsistent linear system Ax =b often arises in practice. This is a system whereb does not belong to the column space ofA, usually with more equations than variables. Thus, such a system has no solution. Yet, we would still like to “solve” such a system, at least approximately.

Such systems often arise when trying to fit some data. For example, we may have a set of 3D data points

{p_1, . . . , pn},

and we have reason to believe that these points are nearly coplanar. We would like to find a plane that best fits our data points. Recall that the equation of a plane is

αx +βy +γz +δ = 0,

with (α, β, γ) = (0,0,0). Thus, every plane is either not parallel to thex-axis (α = 0) or not parallel to they-axis (β = 0) or not parallel to thez-axis (γ = 0).

Say we have reasons to believe that the plane we are looking for is not parallel to the z-axis. If we are wrong, in the least squares solution, one of the coefficients,α, β, will be very large. Ifγ = 0, then we may assume that our plane is given by an equation of the form

z =ax +by +d,

and we would like this equation to be satisfied for all thepi’s, which leads to a system ofn equations in 3 unknownsa, b, d, withpi = (xi, yi, zi);

ax_1 +by_1 +d =z_1 . .

axn +byn +d =zn.

However, if n is larger than 3, such a system generally has no solution. Since the above system can’t be solved exactly, we can try to find a solution (a, b, d) that minimizes the least-squares error

n

(axi +byi +d−zi)2.

i=1

This is what Legendre and Gauss figured out in the early 1800’s! In general, given a linear system

Ax =b, we solve the least squares problem: minimize Ax−b^2.2 Fortunately, everyn×m-matrixA can be written as

A =V DU

whereU andV are orthogonal andD is a rectangular diagonal matrix with non-negative entries (singular value decomposition, or SVD); see Chapter 16.

The SVD can be used to solve an inconsistent system. It is shown in Chapter 17 that there is a vectorx of smallest norm minimizing Ax−b_2. It is given by the (Penrose) pseudo-inverse ofA (itself given by the SVD).

It has been observed that solving in the least-squares sense may give too much weight to “outliers,” that is, points clearly outside the best-fit plane. In this case, it is preferable to minimize (the_1-norm)n

i=1|axi +byi +d−zi|.

This does not appear to be a linear problem, but we can use a trick to convert this minimization problem into a linear program (which means a problem involving linear constraints).

Note that|x| = max{x,−x}. So, by introducing new variablese_1, . . . , en, our minimization problem is equivalent to the linear program (LP):

minimize e_1 +· · · +en

subject to axi +byi +d−zi≤ei (axi +byi +d−zi)≤ei−

1 i≤n.≤

Observe that the constraints are equivalent to

ei≥ |axi +byi +d−zi|, 1≤i≤n.

For an optimal solution, we must have equality, since otherwise we could decrease some ei and get an even better solution. Of course, we are no longer dealing with “pure” linear algebra, since our constraints are inequalities.

We prefer not getting into linear programming right now, but the above example provides a good reason to learn more about linear programming!

7.5 Summary

The main concepts and results of this chapter are listed below: Norms and normed vector spaces.•

7.5. SUMMARY 233

• The triangle inequality.

The Euclidean norm; the p-norms.•

• H¨older’s inequality; the Cauchy–Schwarz inequality; Minkowski’s inequality. Hermitian inner product and Euclidean inner product.•

• Equivalent norms.

All norms on a finite-dimensional vector space are equivalent (Theorem 7.3).•

• Matrix norms.

Hermitian, symmetric and normal matrices. Orthogonal and unitary matrices.•

• The trace of a matrix.

Eigenvalues and eigenvectors of a matrix.•

• The characteristic polynomial of a matrix.

The spectral radiusρ(A) of a matrixA.•

• The Frobenius norm.

The Frobenius norm is a unitarily invariant matrix norm.•

• Bounded linear maps.

Subordinate matrix norms.•

•

Characterization of the subordinate matrix norms for the vector norms_1,2, and .

∞

• The spectral norm.

For every matrixA∈ Mn(C) and for every > 0, there is some subordinate matrix• norm such that A ≤ρ(A) + .

Condition numbers of matrices.•

• Perturbation analysis of linear systems.

The singular value decomposition (SVD).•

•

Properties of conditions numbers. Characterization of cond_2(A) in terms of the largest and smallest singular values ofA.

The Hilbert matrix: a very badly conditioned matrix.•

• Solving inconsistent linear systems by the method of least-squares; linear programming.

Chapter 8 Iterative Methods for Solving Linear Systems

8.1 Convergence of Sequences of Vectors and Matrices

In Chapter 6 we have discussed some of the main methods for solving systems of linear equations. These methods are direct methods, in the sense that they yield exact solutions (assuming infinite precision!).

Another class of methods for solving linear systems consists in approximating solutions using iterative methods. The basic idea is this: Given a linear systemAx =b (withA a square invertible matrix), find another matrixB and a vectorc, such that

1. The matrixI−B is invertible

2. The unique solutionx of the systemAx =b is identical to the unique solutionu of the system

u =Bu +c, and then, starting from any vectoru_0, compute the sequence (uk) given by uk+1 =Buk +c, k∈ N.

Under certain conditions (to be clarified soon), the sequence (uk) converges to a limitu which is the unique solution ofu =Bu +c, and thus ofAx =b.

Consequently, it is important to find conditions that ensure the convergence of the above sequences and to have tools to compare the “rate” of convergence of these sequences. Thus, we begin with some general results about the convergence of sequences of vectors and matrices.

Let (E, ) be a normed vector space. Recall that a sequence (uk) of vectorsuk∈E converges to a limitu∈E, if for every > 0, there some natural numberN such that uk−u ≤, for all k≥N.

235 We write u = lim uk.

k→∞

If E is a finite-dimensional vector space and dim(E) =n, we know from Theorem 7.3 that any two norms are equivalent, and if we choose the norm ∞, we see that the convergence of the sequence of vectorsuk is equivalent to the convergence of then sequences of scalars formed by the components of these vectors (over any basis). The same property applies to the finite-dimensional vector space Mm,n(K) ofm×n matrices (withK = R orK = C), which means that the convergence of a sequence of matricesAk = (a(k) ) is equivalent to theij

convergence of the

m

×

n

sequences of scalars (

a

(k)

ij ), withi, j fixed (1≤i≤m, 1≤j≤n).

The first theorem below gives a necessary and sufficient condition for the sequence ( Bk) of powers of a matrixB to converge to the zero matrix. Recall that the spectral radiusρ(B) of a matrixB is the maximum of the moduli|λi| of the eigenvalues ofB.

Theorem 8.1. For any square matrixB, the following conditions are equivalent: (1)k = 0, limk→∞B

(2) limk→∞Bkv = 0, for all vectorsv,

(3) ρ(B)< 1, (4) B < 1, for some subordinate matrix norm .

Proof. Assume (1) and let be a vector norm onE and be the corresponding matrix norm. For every vectorv∈E, because is a matrix norm, we have

Bkv ≤ Bk v ,

and since limk→∞Bk = 0 means that limk→∞ Bk = 0, we conclude that limk→∞ Bkv = 0, that is, limk→∞Bkv = 0. This proves that (1) implies (2).

Assume (2). If We hadρ(B)≥ 1, then there would be some eigenvectoru (= 0) and some eigenvalueλ such that

Bu =λu, |λ| =ρ(B)≥ 1,

but then the sequence (

B

ku) would not converge to 0, becauseBku =λku and|λk =|λk | ≥1. It follows that (2) implies (3).| Assume that (3) holds, that is,ρ(B)< 1. By Proposition 7.9, we can find > 0 small enough thatρ(B) + < 1, and a subordinate matrix norm such that

B ≤ρ(B) +, which is (4). 8.1. CONVERGENCE OF SEQUENCES OF VECTORS AND MATRICES 237

Finally, assume (4). Because is a matrix norm,

Bk B k,≤

and since B < 1, we deduce that (1) holds.

The following proposition is needed to study the rate of convergence of iterative methods. Proposition 8.2. For every square matrixB and every matrix norm , we have

lim Bk 1/k =ρ(B).

k→∞

Proof. We know from Proposition 7.4 thatρ(B)≤ B , and sinceρ(B) = (ρ(Bk))1/k, we deduce that

ρ(B)≤ Bk 1/k for allk≥ 1, and so

ρ(B)≤ lim Bk 1/k.

k→∞

Now, let us prove that for every > 0, there is some integerN( ) such that

Bk 1/k ρ(B) + for allk≥N( ),≤

which proves that

lim Bk 1/k ρ(B),

k→∞≤

and our proposition.

For any given > 0, letB be the matrix

B B =ρ(B) + .

Since B < 1, Theorem 8.1 implies that limk→∞Bk = 0. Consequently, there is some integerN( ) such that for allk≥N( ), we have

B

k

Bk = (ρ(B) + )k≤ 1, which implies that Bk 1/k ρ(B) +,≤ as claimed.

We now apply the above results to the convergence of iterative methods.

8.2 Convergence of Iterative Methods

Recall that iterative methods for solving a linear system Ax =b (withA invertible) consists in finding some matrixB and some vectorc, such thatI−B is invertible, and the unique solutionx ofAx =b is equal to the unique solutionu ofu =Bu +c. Then, starting from any vectoru_0, compute the sequence (uk) given by

uk+1 =Buk +c, k∈ N,

and say that the iterative method is convergent iff

lim uk =u,

k→∞

for every initial vectoru_0.

Here is a fundamental criterion for the convergence of any iterative methods based on a matrixB, called the matrix of the iterative method.

Theorem 8.3. Given a systemu =Bu+c as above, whereI−B is invertible, the following statements are equivalent:

(1) The iterative method is convergent.

(2) ρ(B)< 1.

(3) B < 1, for some subordinate matrix norm .

Proof. Define the vectorek (error vector) by

ek =uk−u,

whereu is the unique solution of the systemu =Bu +c. Clearly, the iterative method is convergent iff

lim ek = 0.

k →∞We claim that

ek =Bke_0, k≥ 0,

wheree_0 =u0−u.

This is proved by induction onk. The base casek = 0 is trivial. By the induction hypothesis,ek =Bke_0, and sinceuk+1 =Buk +c, we get

uk+1−u =Buk +c−u,

and becauseu =Bu +c andek =Bke_0 (by the induction hypothesis), we obtain uk+1−u =Buk−Bu =B(uk−u) =Bek =BBke_0 =Bk+1e_0, proving the induction step. Thus, the iterative method converges iff

lim Bke_0 = 0.

k→∞

Consequently, our theorem follows by Theorem 8.1.

8.2. CONVERGENCE OF ITERATIVE METHODS 239

The next proposition is needed to compare the rate of convergence of iterative methods. It shows that asymptotically, the error vectorek =Bke_0 behaves at worst like (ρ(B))k. Proposition 8.4. Let be any vector norm, letB be a matrix such thatI−B is invertible, and letu be the unique solution ofu =Bu +c.

(1) If (uk) is any sequence defined iteratively by

uk+1 =Buk +c, k∈ N,

then lim sup uk−u1/k =ρ(B).

k→∞ u0−u =1

(2) Let B_1 andB_2 be two matrices such thatI−B_1 andI−B_2 are invertibe, assume that bothu =B_1u+c_1 andu =B_2u+c_2 have the same unique solution u, and consider any two sequences (uk) and (vk) defined inductively by

uk+1 =B_1uk +c_1 vk+1 =B_2vk +c_2,

withu_0 =v_0. Ifρ(B_1)< ρ(B_2), then for any > 0, there is some integerN( ), such that for allk≥N( ), we have

supvk−u 1/k ρ(B2) . u0−u =1 uk−u ≥ ρ(B_1) +

Proof. Let be the subordinate matrix norm. Recall that

uk−u =Bke_0,

withe_0 =u0−u. For everyk∈ N, we have

(ρ(B_1))k =ρ(Bk)≤ Bk = sup Bke_0 ,1 1 e0 =1 1

which implies ρ(B_1) = sup Bke_0 1/k = Bk 1/k,

e_0 =11 1

and statement (1) follows from Proposition 8.2. Becauseu_0 =v_0, we have

uk−u =Bke_01 vk−u =Bke_0,2 withe_0 =u0−u =v0−u. Again, by Proposition 8.2, for every > 0, there is some natural numberN( ) such that ifk≥N( ), then

sup Bke_0 1/k ρ(B_1) +.

e_0 =1^1 ≤

Furthermore, for all k≥N( ), there exists a vectore_0 =e_0(k) such that eke_0 1/k = Bk 1/k ρ(B_2),0 = 1 and B2 2 ≥

which implies statement (2).

In light of the above, we see that when we investigate new iterative methods, we have to deal with the following two problems:

1. Given an iterative method with matrix B, determine whether the method is convergent. This involves determining whetherρ(B)< 1, or equivalently whether there is a subordinate matrix norm such that B < 1. By Proposition 7.8, this implies that I−B is invertible (since −B = B , Proposition 7.8 applies).

2. Given two convergent iterative methods, compare them. The iterative method which is faster is that whose matrix has the smaller spectral radius.

We now discuss three iterative methods for solving linear systems:

1. Jacobi’s method

2. Gauss-Seidel’s method

3. The relaxation method.

8.3 Description of the Methods of Jacobi, Gauss-Seidel, and Relaxation

The methods described in this section are instances of the following scheme: Given a linear systemAx =b, withA invertible, suppose we can writeA in the form

A =M−N,

withM invertible, and “easy to invert,” which means thatM is close to being a diagonal or a triangular matrix (perhaps by blocks). Then,Au =b is equivalent to

Mu =Nu +b,

that is,

u =M−1Nu +M−1b. Therefore, we are in the situation described in the previous sections withB =M−1N and c =M−1b. In fact, sinceA =M−N, we have

B =M−1N =M−1(M−A) =I−M−1A,

which shows thatI−B =M−1A is invertible. The iterative method associated with the matrixB =M−1N is given by

uk+1 =M−1Nuk +M−1b, k≥ 0, starting from any arbitrary vectoru_0. From a practical point of view, we do not invertM, and instead we solve iteratively the systems

Muk+1 =Nuk +b, k≥ 0.

Various methods correspond to various ways of choosing M andN fromA. The first two methods chooseM andN as disjoint submatrices ofA, but the relaxation method allows some overlapping ofM andN.

To describe the various choices ofM andN, it is convenient to writeA in terms of three submatricesD, E, F, as

A =D−E−F,

where the only nonzero entries in D are the diagonal entries inA, the only nonzero entries inE are entries inA below the the diagonal, and the only nonzero entries inF are entries inA above the diagonal. More explicitly, if

ë a11 a12 a13 · · · a1n

÷

ì

−

1 a1nö

ì a21 a22 a23 · · · a2n

÷

ì

−

1

a

2

n

÷

ì ÷

ì a31 a32 a33 · · · a3n−1 a3n÷

A

=

ì ÷

ì ÷ ,

ì . . . ... . . ÷

ì ÷

ì ÷

ìan−1 1 an−1 2 an−1 3 · · · an−1 n−1 an−1 n÷

í ø

an 1 an 2 an 3 · · · an n−1 an n

then ëa_11 0 0 · · · 0 0ö ì 0 a_22 0 · · · 0 0÷

ì ÷ ì ÷

ì 0 0 a33 · · · 0 0÷

ì ÷

D

=

ì ÷ ì ÷, ì . . . ... . . ÷ ì ÷

ì 0 0 0 · · · an−1 n−1 0÷

ì ÷

í ø 0 0 0 · · · 0 an n ë 0 0 0 · · · 0 0ö ë0 a_12 a13 · · · a1n

÷ ì ÷ ì

−

1 a1n ö ì a_21 0 0 · · · 0 0÷ ì0 0 a_23

÷ ì ÷ ì

· · ·

a2n−1 a2n ÷

ì ÷ ì ÷

ì a31 a32 0 · · · 0 0÷ ì0 0 0 ... a3n−1 a3n ÷ E

=

ì ÷,−F = ì ÷ ÷

ì ÷ ì

.− ì. . ... .... .÷ ì... ... . ÷

ì ÷ ì

÷ ì ÷

ì

.

.

.

÷

ì ... 0 0÷ ì0 0 0 · · · 0 an−1 n÷

ían−1 1 an−1 2 an−1 3 ø í ø

an 1 an 2 an 3 · · · an n−1 0 0 0 0 · · · 0 0 In Jacobi’s method, we assume that all diagonal entries inA are nonzero, and we pick M =D

N =E +F,

so that

B =M−1N =D−1(E +F) =I−D−1A.

As a matter of notation, we let

J =I−D−1A =D−1(E +F),

which is called Jacobi’s matrix. The corresponding method, Jacobi’s iterative method, computes the sequence (uk) using the recurrence

uk+1 =D−1(E +F)uk +D−1b, k≥ 0.

In practice, we iteratively solve the systems

Duk+1 = (E +F)uk +b, k≥ 0.

If we writeuk = (uk, . . . , ukn), we solve iteratively the following system:1

a_11uk+1 =−a_12uk a_13uk a1nukn +b11 2 − 3

a

23

u

k · · · −a2nukn +b_22+1 = −a_21uka_22uk

1 −3 · · · −.. . .

a

n

−

1

n

−

1

u

k+1 = −an−1 1uk

n an−1 n−2ukn−2kn−1 +b −1 a

n n

u

k

+1

−

1 k^1 · · · −k an n−1u −an−1 nukn +bn = −an 1u1 −an 2u2 · · · −nn

Observe that we can try to “speed up” the method by using the new value u

k+1

1 instead of

u

k in solving foruk+2 using the second equations, and more generally, useuk+1 , . . . , uk+1 1 k, . . . , uki2 1 i

−

1

in solving for

u

k

+1

in the

i

th equation. This observation leads to the − 1

instead ofu1 i

system

a

11

u

k+1 =−a_12uk a_13uk

1 3 a1nukn +b_1 a_22uk+1 = −a_21uk+1 2 −a_23uk · · · −a2nukn +b22 1 − 3 · · · −. . .

a

n

−

1

n

−

1

u

k+1= −an−1 1uk+1an−1 n−2uk+1

1

an

an−1 nukn +bn−1n−1 k+1 a· · · − n−2 k+1 − +bn,n nuk+1 = −an 1u1 −n 2u2 · · · −an n−1un−1

which, in matrix form, is written

Duk+1 =Euk+1 +F uk +b.

BecauseD is invertible andE is lower triangular, the matrixD−E is invertible, so the above equation is equivalent to

uk+1 = (D−E)−1F uk + (D−E)−1b, k≥ 0.

The above corresponds to choosingM andN to be

M =D−E N =F,

and the matrixB is given by

B =M−1N = (D−E)−1F.

SinceM =D−E is invertible, we know thatI−B =M−1A is also invertible. The method that we just described is the iterative method of Gauss-Seidel, and the matrixB is called the matrix of Gauss-Seidel and denoted byL1, with

L1 = (D−E)−1F.

One of the advantages of the method of Gauss-Seidel is that is requires only half of the memory used by Jacobi’s method, since we only need

u

k+1 , . . . , uk+1

1 i , uki+1, . . . , ukn−1

to compute uk+1. We also show that in certain important cases (for example, ifA is ai

tridiagonal matrix), the method of Gauss-Seidel converges faster than Jacobi’s method (in this case, they both converge or diverge simultaneously).

The new ingredient in the relaxation method is to incorporate part of the matrixD into N: we defineM andN by

M =D Eω−

ω

N = 1−ω D +F,

where ω = 0 is a real parameter to be suitably chosen. Actually, we show in Section 8.4 that for the relaxation method to converge, we must haveω∈ (0,2). Note that the caseω = 1 corresponds to the method of Gauss-Seidel.

If we assume that all diagonal entries ofD are nonzero, the matrixM is invertible. The matrixB is denoted byLω and called the matrix of relaxation, with

D

E

−

1 1 ω

−ω D +F = (D−ωE)−1((1−ω)D +ωF).Lω =ω−

The numberω is called the parameter of relaxation. Whenω > 1, the relaxation method is known as successive overrelaxation, abbreviated as SOR.

At first glance, the relaxation matrixLω seems at lot more complicated than the GaussSeidel matrixL1, but the iterative system associated with the relaxation method is very similar to the method of Gauss-Seidel, and is quite simple. Indeed, the system associated with the relaxation method is given by

D

E

u

k

+1

=

1

ω

−ω D +F uk +b,ω−

which is equivalent to

(D−ωE)uk+1 = ((1−ω)D +ωF)uk +ωb,

and can be written Duk+1 =Duk−ω(Duk−Euk+1−F uk−b).

Explicitly, this is the system

a_11uk+1 =a_11uk ω(a_11uk +a_12uk +a_13uk +· · · +a1n−2ukn−2 +a1n−1ukn−1 +a1nukn−b_1)1 1− 1 2 3

ω(a_21uk+1 +a_22uk +a_23uk +· · · +a2n−2ukn−2 +a2n−1ukn−1 +a2nukn−b_2)22uk+1 =a_22uka2 2− 1 2 3

.

an nuk+1 =an nukn−ω(an 1uk+1 +an 2uk+1 +· · · +an n−2uk+1 +an n−1uk+1 +an nukn−bn).n 1 2 n−2 n−1

What remains to be done is to find conditions that ensure the convergence of the relaxation method (and the Gauss-Seidel method), that is:

1. Find conditions onω, namely some intervalI⊆ R so thatω∈I impliesρ(Lω)< 1; we will prove thatω∈ (0,2) is a necessary condition.

2. Find if there exist some optimal valueω_0 ofω∈I, so that

ρ(Lω0) = infρ(Lω).

ω∈I

We will give partial answers to the above questions in the next section.

It is also possible to extend the methods of this section by using block decompositions of the formA =D−E−F, whereD, E, andF consist of blocks, and withD an invertible block-diagonal matrix.

8.4 Convergence of the Methods of Jacobi, Gauss-Seidel, and Relaxation

We begin with a general criterion for the convergence of an iterative method associated with a (complex) Hermitian, positive, definite matrix,A =M−N. Next, we apply this result to the relaxation method.

Proposition 8.5. LetA be any Hermitian, positive, definite matrix, written as

A =M−N, withM invertible. Then,M∗ +N is Hermitian, and if it is positive, definite, then

ρ (M−1N)< 1, so that the iterative method converges.

Proof. SinceM =A +N andA is Hermitian,A∗ =A, so we get

M∗ +N =A∗ +N∗ +N =A +N +N∗ =M +N∗ = (M∗ +N)∗, which shows thatM∗ +N is indeed Hermitian. BecauseA is symmetric, positive, definite, the function v→ (v∗Av)1/2 from Cn to R is a vector norm , and let also denote its subordinate matrix norm. We prove that

M−1N < 1,

which, by Theorem 8.1 proves thatρ(M−1N)< 1. By definition

M−1N = I−M−1A = sup v−M−1Av ,

v =1

which leads us to evaluate v^1Av when v = 1. If we writew =M−1Av, using the facts that

v

= 1,

v

=

A

−

1

−M−

Mw,A∗ =A, andA =M−N, we have

v−w w)∗A(v−w)

= v

2 = (v−2 v∗Aw−w∗Av +w∗Aw−

= 1−w∗M∗w−w∗Mw +w∗Aw

= 1−w∗(M∗ +N)w.

Now, since we assumed thatM∗ +N is positive definite, ifw = 0, thenw∗(M∗ +N)w > 0, and we conclude that

if v = 1 then v−M−1Av < 1. Finally, the function v→ v−M−1Av

is continuous as a composition of continuous functions, therefore it achieves its maximum on the compact subset{v∈ Cn v = 1}, which proves that|

sup v−M−1Av < 1,

v =1

and completes the proof.

Now, as in the previous sections, we assume that A is written asA = D−E−F, withD invertible, possibly in block form. The next theorem provides a sufficient condition (which turns out to be also necessary) for the relaxation method to converge (and thus, for the method of Gauss-Seidel to converge). This theorem is known as the Ostrowski-Reich theorem.

Theorem 8.6. IfA =D−E−F is Hermitian, positive, definite, and if 0< ω < 2, then the relaxation method converges. This also holds for a block decomposition ofA.

Proof. Recall that for the relaxation method,A =M−N with

M =D Eω−

ω

N = 1−ω D +F,

and becauseD∗ =D,E∗ =F (sinceA is Hermitian) andω = 0 is real, we have

M

∗

+

N

=

D ω ω

ω−

∗ E∗ + 1−ω D +F = 2−ω D.

If D consists of the diagonal entries ofA, then we know from Section 6.3 that these entries are all positive, and sinceω∈ (0,2), we see that the matrix ((2−ω)/ω)D is positive definite. IfD consists of diagonal blocks ofA, becauseA is positive, definite, by choosing vectorsz obtained by picking a nonzero vector for each block ofD and padding with zeros, we see that each block ofD is positive, definite, and thusD itself is positive definite. Therefore, in all cases,M∗ +N is positive, definite, and we conclude by using Proposition 8.5.

Remark: What if we allow the parameterω to be a nonzero complex numberω∈ C? In this case, we get

M

∗

+

N

=

D

∗

E

∗

+ 1

ω

−ω D +F =1 + 1− 1 D.ω− ω ω But, 1

+ 1

−

1 =

ω

+

ω

−

ωω

= 1

−

(

ω

−

1)(

ω

−

1)

= 1

− |

ω

−

1

2

| ,ωω ωω|ω 2 2

| |ω|so the relaxation method also converges forω∈ C, provided that

ω− 1|< 1.|

This condition reduces to 0< ω < 2 ifω is real.

Unfortunately, Theorem 8.6 does not apply to Jacobi’s method, but in special cases, Proposition 8.5 can be used to prove its convergence. On the positive side, if a matrix is strictly column (or row) diagonally dominant, then it can be shown that the method of Jacobi and the method of Gauss-Seidel both converge. The relaxation method also converges ifω∈ (0,1], but this is not a very useful result because the speed-up of convergence usually occurs forω > 1.

We now prove that, without any assumption on A =D−E−F, other than the fact thatA andD are invertible, in order for the relaxation method to converge, we must have ω∈ (0,2).

Proposition 8.7. Given any matrixA =D−E−F, withA andD invertible, for any ω = 0, we have

ρ(Lω)≥ |ω− 1|.

Therefore, the relaxation method (possibly by blocks) does not converge unlessω∈ (0,2). If

we allow ω to be complex, then we must have

ω− 1|< 1|

for the relaxation method to converge.

Proof. Observe that the productλ1· · ·λn of the eigenvalues ofLω, which is equal to det(Lω), is given by

det

1

ω

−ω D +F λ1· · ·λn = det(Lω) = D = (1 E

− ω)n. det ω−

It follows that

ρ

(

L

ω

)

≥ |

λ

1

· · ·

λ

n

1/n

|

=|ω− 1|. The proof is the same ifω∈ C.

We now consider the case where A is a tridiagonal matrix, possibly by blocks. In this case, we obtain precise results about the spectral radius ofJ andLω, and as a consequence, about the convergence of these methods. We also obtain some information about the rate of convergence of these methods. We begin with the caseω = 1, which is technically easier to deal with. The following proposition gives us the precise relationship between the spectral radiiρ(J) andρ(L1) of the Jacobi matrix and the Gauss-Seidel matrix.

Proposition 8.8. LetA be a tridiagonal matrix (possibly by blocks). Ifρ(J) is the spectral radius of the Jacobi matrix andρ(L1) is the spectral radius of the Gauss-Seidel matrix, then we have

ρ(L1) = (ρ(J))2.

Consequently, the method of Jacobi and the method of Gauss-Seidel both converge or both diverge simultaneously (even whenA is tridiagonal by blocks); when they converge, the method of Gauss-Seidel converges faster than Jacobi’s method.

Proof. We begin with a preliminary result. LetA(µ) with a tridiagonal matrix by block of

the form ë A_1 µ−1C_1 0 0· · · 0ö ìµB_1 A_2 µ−1

÷ ì

C

2

0

· · ·

0

÷

ì ... ... ... · · ·. ÷ A

(

µ

) =

ì 0 ÷ ì ÷ , ì

÷

í 0

.

· · ·

...

...

...

0

÷

ì 1Cp−1ø 0

· · ·

0 µBp−2 Ap−1 µ−

0 µBp−1 Ap

then · · · · · ·

det(A(µ)) = det(A(1)), µ = 0. To prove this fact, form the block diagonal matrix

P(µ) = diag(µI_1, µ^2I_2, . . . , µpIp),

whereIj is the identity matrix of the same dimension as the blockAj. Then, it is easy to see that

A(µ) =P(µ)A(1)P(µ)−1, and thus,

det(A(µ)) = det(P(µ)A(1)P(µ)−1) = det(A(1)).

Since the Jacobi matrix isJ =D−1(E +F), the eigenvalues ofJ are the zeros of the characteristic polynomial

pJ(λ) = det(λI−D−1(E +F)), and thus, they are also the zeros of the polynomial

qJ(λ) = det(λD−E−F) = det(D)pJ(λ).

Similarly, since the Gauss-Seidel matrix isL1 = (D−E)−1F, the zeros of the characteristic polynomial

pL1(λ) = det(λI− (D−E)−1F) are also the zeros of the polynomial

qL1(λ) = det(λD−λE−F) = det(D−E)pL1(λ). SinceA is tridiagonal (or tridiagonal by blocks), using our preliminary result withµ =λ = 0, we get

qL1(λ^2) = det(λ^2D−λ^2E−F) = det(λ^2D−λE−λF) =λnqJ(λ). By continuity, the above equation also holds forλ = 0. But then, we deduce that: 1. For anyβ = 0, ifβ is an eigenvalue ofL1, thenβ1/2 and−β1/2 are both eigenvalues of J, whereβ1/2 is one of the complex square roots ofβ.

2. For anyα = 0, ifα and−α are both eigenvalues ofJ, thenα^2 is an eigenvalue ofL1. The above immediately implies thatρ(L1) = (ρ(J))2.

We now consider the more general situation whereω is any real in (0,2).

Proposition 8.9. LetA be a tridiagonal matrix (possibly by blocks), and assume that the eigenvalues of the Jacobi matrix are all real. Ifω∈ (0,2), then the method of Jacobi and the method of relaxation both converge or both diverge simultaneously (even whenA is tridiagonal by blocks). When they converge, the functionω→ ρ(Lω) (forω∈ (0,2)) has a unique minimum equal toω0− 1 for

ω 2 ,0 = 1 + 1− (ρ(J))2

where 1< ω_0< 2 ifρ(J)> 0. We also haveρ(L1) = (ρ(J))2, as before.

Proof. The proof is very technical and can be found in Serre [92] and Ciarlet [22]. As in the proof of the previous proposition, we begin by showing that the eigenvalues of the matrix Lω are the zeros of the polynomnial

q λ +ω− 1D−λE−F = det D E pLω(λ),Lω(λ) = detω ω−

wherepLω(λ) is the characteristic polynomial ofLω. Then, using the preliminary fact from Proposition 8.8, it is easy to show that

q

L

ω

(

λ

2

) =

λ

n

q

λ^2 +ω

J λω− 1 , for allλ∈ C, withλ = 0. This time, we cannot extend the above equation toλ = 0. This leads us to consider the equation

λ^2 +ω− 1=α,λω

which is equivalent to

λ^2 αωλ +ω− 1 = 0,−

for all λ = 0. Sinceλ = 0, the above equivalence does not hold forω = 1, but this is not a problem since the caseω = 1 has already been considered in the previous proposition. Then, we can show the following:

1. For anyβ = 0, ifβ is an eigenvalue ofLω, then

β +ω−ω , − β1/2− 1

β1/2ω

1 β +ω

are eigenvalues ofJ.

2. For every α = 0, ifα and−α are eigenvalues ofJ, thenµ+(α, ω) andµ−(α, ω) are eigenvalues ofLω, whereµ+(α, ω) andµ−(α, ω) are the squares of the roots of the equation

λ^2 αωλ +ω− 1 = 0.−

It follows that

ρ(Lω) =λ max

|pJ(λ)=0{max(|µ+(α, ω)|,|µ−(α, ω)|)},

and since we are assuming thatJ has real roots, we are led to study the function M(α, ω) = max{|µ+(α, ω)|,|µ−(α, ω)|},

whereα∈ R andω∈ (0,2). Actually, becauseM(−α, ω) =M(α, ω), it is only necessary to consider the case whereα≥ 0.

Note that forα = 0, the roots of the equation

λ^2 αωλ +ω− 1 = 0.−

are

αω±√α^2ω^2 4ω + 4.2 −

In turn, this leads to consider the roots of the equation

ω^2α^2 4ω + 4 = 0,−

which are2(1±√1−α2),

α^2

forα = 0. Since we have

2(1 + 2√1−α^2) = 2(1 +√1− α^2)(1−√1−α^2)= 1−√1−α2α2 α2(1−√1−α2)

and

2(1 2−√1−α2) = 2(1 +√1−α2)(1−√1−α2) = 1 +√1−α2,

α^2 α^2(1 +√1−α^2)

these roots are

ω^22 , ω_1(α) =1−√1−α2.0(α) =1 +√1−α2

Observe that the expression forω_0(α) is exactly the expression in the statement of our proposition! The rest of the proof consists in analyzing the variations of the functionM(α, ω) by considering various cases forα. In the end, we find that the minimum ofρ(Lω) is obtained forω_0(ρ(J)). The details are tedious and we omit them. The reader will find complete proofs in Serre [92] and Ciarlet [22].

Combining the results of Theorem 8.6 and Proposition 8.9, we obtain the following result which gives precise information about the spectral radii of the matricesJ,L1, andLω.

Proposition 8.10. LetA be a tridiagonal matrix (possibly by blocks) which is Hermitian, positive, definite. Then, the methods of Jacobi, Gauss-Seidel, and relaxation, all converge forω∈ (0,2). There is a unique optimal relaxation parameter

2ω0 = 1 + 1− (ρ(J))2, such that ρ

(

L

ω

0

) =

inf

0<ω<2ρ(Lω) =ω0− 1. Furthermore, ifρ(J)> 0, then

ρ(Lω0)< ρ(L1) = (ρ(J))2< ρ(J),

and ifρ(J) = 0, thenω_0 = 1 andρ(L1) =ρ(J) = 0.

Proof. In order to apply Proposition 8.9, we have to check thatJ =D−1(E +F) has real eigenvalues. However, ifα is any eigenvalue ofJ and ifu is any corresponding eigenvector, then

D−1(E +F)u =αu implies that

(E +F)u =αDu,

and sinceA =D−E−F, the above shows that (D−A)u =αDu, that is,

Au = (1−α)Du.

Consequently, u∗Au = (1−α)u∗Du,

and sinceA andD are hermitian, positive, definite, we haveu∗Au > 0 andu∗Du > 0 if u = 0, which proves thatα∈ R. The rest follows from Theorem 8.6 and Proposition 8.9.

Remark: It is preferable to overestimate rather than underestimate the relaxation parameter when the optimum relaxation parameter is not known exactly.

8.5 Summary

The main concepts and results of this chapter are listed below:

• Iterative methods. SplittingA asA =M−N.

• Convergence of a sequence of vectors or matrices.

•

A criterion for the convergence of the sequence (Bk) of powers of a matrixB to zero in terms of the spectral radiusρ(B).

A characterization of the spectral radius ρ(B) as the limit of the sequence ( Bk 1/k).•

• A criterion of the convergence of iterative methods.

Asymptotic behavior of iterative methods.•

•

SplittingA asA =D−E−F, and the methods of Jacobi, Gauss-Seidel, and relaxation

(and SOR).

The Jacobi matrix,J =D−1(E +F).•

• The Gauss-Seidel matrix,L2 = (D−E)−1F.

The matrix of relaxation,Lω = (D−ωE)−1((1−ω)D +ωF).•

•

Convergence of iterative methods: a general result whenA =M−N is Hermitian, positive, definite.

A sufficient condition for the convergence of the methods of Jacobi, Gauss-Seidel, and• relaxation. The Ostrowski-Reich Theorem: A is symmetric, positive, definite, and

ω∈ (0,2).

A necessary condition for the convergence of the methods of Jacobi , Gauss-Seidel, and• relaxation:ω∈ (0,2).

The case of tridiagonal matrices (possibly by blocks). Simultaneous convergence or di• vergence of Jacobi’s method and Gauss-Seidel’s method, and comparison of the spectral

radii ofρ(J) andρ(L1):ρ(L1) = (ρ(J))2.

The case of tridiagonal, Hermitian, positive, definite matrices (possibly by blocks).• The methods of Jacobi, Gauss-Seidel, and relaxation, all converge.

In the above case, there is a unique optimal relaxation parameter for whichρ(Lω0)<• ρ(L1) = (ρ(J))2< ρ(J) (ifρ(J) = 0).

Chapter 9 Euclidean Spaces

Rien n’est beau que le vrai. —Hermann Minkowski

9.1 Inner Products, Euclidean Spaces

So far, the framework of vector spaces allows us to deal with ratios of vectors and linear combinations, but there is no way to express the notion of length of a line segment or to talk about orthogonality of vectors. A Euclidean structure allows us to deal with metric notions such as orthogonality and length (or distance).

This chapter covers the bare bones of Euclidean geometry. Deeper aspects of Euclidean geometry are investigated in Chapter 10. One of our main goals is to give the basic properties of the transformations that preserve the Euclidean structure, rotations and reflections, since they play an important role in practice. Euclidean geometry is the study of properties invariant under certain affine maps called rigid motions. Rigid motions are the maps that preserve the distance between points.

We begin by defining inner products and Euclidean spaces. The Cauchy–Schwarz inequality and the Minkowski inequality are shown. We define orthogonality of vectors and of subspaces, orthogonal bases, and orthonormal bases. We prove that every finite-dimensional Euclidean space has orthonormal bases. The first proof uses duality, and the second one the Gram–Schmidt orthogonalization procedure. TheQR-decomposition for invertible matrices is shown as an application of the Gram–Schmidt procedure. Linear isometries (also called orthogonal transformations) are defined and studied briefly. We conclude with a short section in which some applications of Euclidean geometry are sketched. One of the most important applications, the method of least squares, is discussed in Chapter 17.

For a more detailed treatment of Euclidean geometry, see Berger [6, 7], Snapper and Troyer [95], or any other book on geometry, such as Pedoe [85], Coxeter [24], Fresnel [38], Tisseron [105], or Cagnac, Ramis, and Commeau [17]. Serious readers should consult Emil

253 Artin’s famous book [2], which contains an in-depth study of the orthogonal group, as well as other groups arising in geometry. It is still worth consulting some of the older classics, such as Hadamard [51, 52] and Rouch´e and de Comberousse [86]. The first edition of [51] was published in 1898, and finally reached its thirteenth edition in 1947! In this chapter it is assumed that all vector spaces are defined over the field R of real numbers unless specified otherwise (in a few cases, over the complex numbers C).

First, we define a Euclidean structure on a vector space. Technically, a Euclidean structure over a vector spaceE is provided by a symmetric bilinear form on the vector space satisfying some extra properties. Recall that a bilinear formÕ:E×E→ R is definite if for everyu∈E,u = 0 implies thatÕ(u, u) = 0, and positive if for everyu∈E,Õ(u, u)≥ 0.

Definition 9.1. A Euclidean space is a real vector spaceE equipped with a symmetric bilinear formÕ:E×E→ R that is positive definite. More explicitly, Õ:E×E→ R satisfies the following axioms:

Õ (u_1 +u_2, v) = Õ(u_1, v) +Õ(u_2, v),

Õ(u, v_1 +v_2) = Õ(u, v_1) +Õ(u, v_2),

Õ(λu, v) = λÕ(u, v),

Õ(u, λv) = λÕ(u, v),

Õ(u, v) = Õ(v, u),

u = 0 implies thatÕ(u, u)> 0.

The real numberÕ(u, v) is also called the inner product (or scalar product) ofu andv. We also define the quadratic form associated withÕ as the function Φ:E→ R+ such that Φ(u) =Õ(u, u),

for allu∈E.

SinceÕ is bilinear, we haveÕ(0,0) = 0, and since it is positive definite, we have the stronger fact that

Õ(u, u) = 0 iff u = 0,

that is, Φ(u) = 0 iffu = 0.

Given an inner productÕ:E×E→ R on a vector spaceE, we also denoteÕ(u, v) by u·v or u, v or (u|v),

and Φ(u) by u .

Example 9.1. The standard example of a Euclidean space is Rn, under the inner product · defined such that

(x_1, . . . , xn)· (y_1, . . . , yn) =x_1y_1 +x_2y_2 +· · · +xnyn. This Euclidean space is denoted by En.

There are other examples.

Example 9.2. For instance, letE be a vector space of dimension 2, and let (e_1, e_2) be a basis ofE. Ifa > 0 andb^2 ac < 0, the bilinear form defined such that−

Õ(x_1e_1 +y_1e_2, x_2e_1 +y_2e_2) =ax_1x_2 +b(x_1y_2 +x_2y_1) +cy_1y_2 yields a Euclidean structure onE. In this case,

Φ(xe_1 +ye_2) =ax^2 + 2bxy +cy^2.

Example 9.3. LetC[a, b] denote the set of continuous functionsf : [a, b]→ R. It is easily checked thatC[a, b] is a vector space of infinite dimension. Given any two functionsf, g∈ C

[a, b], let

b

f, g = f(t)g(t)dt.

a

We leave as an easy exercise that−,− is indeed an inner product onC[a, b]. In the case wherea =−π andb =π (ora = 0 andb = 2π, this makes basically no difference), one should compute

sinpx,sinqx , sinpx,cosqx , and cospx,cosqx , for all natural numbersp, q≥ 1. The outcome of these calculations is what makes Fourier analysis possible!

Example 9.4. LetE = Mn(R) be the vector space of realn×n matrices. If we view a matrix A∈ Mn(R) as a “long” column vector obtained by concatenating together its columns, we can define the inner product of two matricesA, B∈ Mn(R) as

n

A, B = aijbij,

i,j=1

which can be conveniently written as

A, B = tr(A B) = tr(B A).

Since this can be viewed as the Euclidean product on Rn2, it is an inner product on Mn(R). The corresponding norm

A F = tr(A A) is the Frobenius norm (see Section 7.2).

Let us observe thatÕ can be recovered from Φ. Indeed, by bilinearity and symmetry, we have

Φ( u +v) = Õ(u +v, u +v)

= Õ(u, u +v) +Õ(v, u +v) = Õ(u, u) + 2Õ(u, v) +Õ(v, v) = Φ(u) + 2Õ(u, v) + Φ(v). Thus, we have

Õ(u, v) = 12[Φ(u +v)− Φ(u)− Φ(v)].

We also say thatÕ is the polar form of Φ.

IfE is finite-dimensional and ifÕ:E×E→ R is a bilinear form onE, given any basis (e_1, . . . , en) ofE, we can writex =n ei andy =n yjej, and we havei=1xi j=1

n n n

Õ(x, y) =Õ xiei, yjej = xiyjÕ(ei, ej).

i=1 j=1 i,j=1

If we letG be the matrixG = (Õ(ei, ej)), and ifx andy are the column vectors associated with (x_1, . . . , xn) and (y_1, . . . , yn), then we can write

Õ(x, y) =x Gy =y G x.

Furhermore, observe thatÕ is symmetric iffG =G , andÕ is positive definite iff the matrix G is positive definite, that is,

x Gx > 0 for allx∈ Rn, x = 0.

The matrixG associated with an inner product is called the Gram matrix of the inner product with respect to the basis (e_1, . . . , en).

Conversely, ifA is a symmetric positive definiten×n matrix, it is easy to check that the bilinear form

x, y =x Ay

is an inner product. If we make a change of basis from the basis ( e_1, . . . , en) to the basis (f_1, . . . , fn), and if the change of basis matrix isP (where thejth column ofP consists of the coordinates offj over the basis (e_1, . . . , en)), then with respect to coordinatesx andy over the basis (f_1, . . . , fn), we have

x, y =x Gy =x P GP y ,

so the matrix of our inner product over the basis (f_1, . . . , fn) isP GP. We summarize these facts in the following proposition.

Proposition 9.1. LetE be a finite-dimensional vector space, and let (e_1, . . . , en) be a basis ofE.

1. For any inner product−,− onE, ifG = (ei, ej ) is the Gram matrix of the inner product−,− w.r.t. the basis (e_1, . . . , en), thenG is symmetric positive definite.

2. For any change of basis matrix P, the Gram matrix of−,− with respect to the new basis isP GP.

3. IfA is anyn×n symmetric positive definite matrix, then

x, y =x Ay is an inner product onE. We will see later that a symmetric matrix is positive definite iff its eigenvalues are all positive.

One of the very important properties of an inner productÕ is that the mapu→ Φ(u) is a norm.

Proposition 9.2. LetE be a Euclidean space with inner productÕ, and let Φ be the corresponding quadratic form. For allu, v∈E, we have the Cauchy–Schwarz inequality

Õ(u, v)2 Φ(u)Φ(v),≤

the equality holding iffu andv are linearly dependent.

We also have the Minkowski inequality

Φ(u +v)≤ Φ(u) + Φ(v), the equality holding iffu andv are linearly dependent, where in addition ifu = 0 andv = 0, thenu =λv for someλ > 0.

Proof. For any vectorsu, v∈E, we define the functionT : R→ R such that

T(λ) = Φ(u +λv), for allλ∈ R. Using bilinearity and symmetry, we have

Φ( u +λv) = Õ(u +λv, u +λv)

= Õ(u, u +λv) +λÕ(v, u +λv)

= Õ(u, u) + 2λÕ(u, v) +λ^2Õ(v, v)

= Φ(u) + 2λÕ(u, v) +λ^2Φ(v).

Since Õ is positive definite, Φ is nonnegative, and thusT(λ)≥ 0 for allλ∈ R. If Φ(v) = 0, thenv = 0, and we also haveÕ(u, v) = 0. In this case, the Cauchy–Schwarz inequality is trivial, andv = 0 andu are linearly dependent.

Now, assume Φ(v)> 0. SinceT(λ)≥ 0, the quadratic equation

λ^2Φ(v) + 2λÕ(u, v) + Φ(u) = 0

cannot have distinct real roots, which means that its discriminant = 4(Õ(u, v)2 Φ(u)Φ(v))−

is null or negative, which is precisely the Cauchy–Schwarz inequality

Õ(u, v)2 Φ(u)Φ(v).≤

If

Õ(u, v)2 = Φ(u)Φ(v),

then the above quadratic equation has a double root λ_0, and we have Φ(u +λ_0v) = 0. If λ_0 = 0, thenÕ(u, v) = 0, and since Φ(v)> 0, we must have Φ(u) = 0, and thusu = 0. In this case, of course,u = 0 andv are linearly dependent. Finally, ifλ_0 = 0, since Φ(u +λ_0v) = 0 implies thatu+λ_0v = 0, thenu andv are linearly dependent. Conversely, it is easy to check that we have equality whenu andv are linearly dependent.

The Minkowski inequality

Φ(u +v)≤ Φ(u) + Φ(v)

is equivalent to Φ(u +v)≤ Φ(u) + Φ(v) + 2 Φ(u)Φ(v). However, we have shown that

2Õ(u, v) = Φ(u +v)− Φ(u)− Φ(v),

and so the above inequality is equivalent to

Õ(u, v)≤ Φ(u)Φ(v),

which is trivial when Õ(u, v)≤ 0, and follows from the Cauchy–Schwarz inequality when Õ(u, v)≥ 0. Thus, the Minkowski inequality holds. Finally, assume thatu = 0 andv = 0, and that

Φ(u +v) = Φ(u) + Φ(v). When this is the case, we have

Õ(u, v) = Φ(u)Φ(v),

and we know from the discussion of the Cauchy–Schwarz inequality that the equality holds iffu andv are linearly dependent. The Minkowski inequality is an equality whenu orv is null. Otherwise, ifu = 0 andv = 0, thenu =λv for someλ = 0, and since

Õ(u, v) =λÕ(v, v) = Φ(u)Φ(v),

by positivity, we must haveλ > 0. Note that the Cauchy–Schwarz inequality can also be written as

|Õ(u, v)| ≤ Φ(u) Φ(v).

Remark: It is easy to prove that the Cauchy–Schwarz and the Minkowski inequalities still hold for a symmetric bilinear form that is positive, but not necessarily definite (i.e.,Õ(u, v)≥ 0 for allu, v∈E). However,u andv need not be linearly dependent when the equality holds.

The Minkowski inequality

Φ(u +v)≤ Φ(u) + Φ(v)

shows that the map u→ Φ(u) satisfies the convexity inequality (also known as triangle inequality), condition (N3) of Definition 7.1, and sinceÕ is bilinear and positive definite, it also satisfies conditions (N1) and (N2) of Definition 7.1, and thus it is a norm onE. The norm induced byÕ is called the Euclidean norm induced byÕ.

Note that the Cauchy–Schwarz inequality can be written as

|u·v| ≤ u v ,

and the Minkowski inequality as

u +v ≤ u + v .

Remark: One might wonder if every norm on a vector space is induced by some Euclidean inner product. In general, this is false, but remarkably, there is a simple necessary and sufficient condition, which is that the norm must satisfy the parallelogram law:

u +v^2 + u−v^2 = 2( u^2 + v^2).

If−,− is an inner product, then we have

u +v^2 = u^2 + v^2 + 2u, v

u−v^2 = u^2 + v^2 2u, v ,−

and by adding and subtracting these identities, we get the parallelogram law and the equation

u, v = 14( u +v^2 u−v^2),−

which allows us to recover−,− from the norm.

Conversely, if is a norm satisfying the parallelogram law, and if it comes from an inner product, then this inner product must be given by

u, v = 14( u +v^2 u−v^2).−

We need to prove that the above form is indeed symmetric and bilinear.

Symmetry holds because u−v =−(u−v) = v−u . Let us prove additivity in the variableu. By the parallelogram law, we have

2( x +z^2 + y^2) = x +y +z^2 + x−y +z^2

which yields x +y +z^2 = 2( x +z^2 + y^2)− x−y +z^2 x +y +z^2 = 2( y +z^2 + x^2)− y−x +z^2,

where the second formula is obtained by swappingx andy. Then by adding up these equations, we get

x +y +z^2 = x^2 + y^2 + x +z^2 + y +z2 1 x−y +z2 1 y−x +z^2.− 2 − 2 Replacingz by−z in the above equation, we get

x +y^2 = x^2 + y^2 + x−z^2 + y−z2 1 x−y−z2 1 y−x−z^2,−z− 2 − 2

Since x− y +z =−(x−y +z) = y−x−z and y−x +z =−(y−x +z) = x−y−z , by subtracting the last two equations, we get

x +y, z = 14( x +y +z^2 x +y−z^2)−

= 14( x +z^2 x−z^2) + 14( y +z^2 y−z^2)− −

= x, z + y, z ,

as desired.

Proving that

λx, y =λ x, y for allλ∈ R

is a little tricky. The strategy is to prove the identity forλ∈ Z, then to promote it to Q, and then to R by continuity.

Since

−u, v = 14(−u +v^2 u−v^2)− −

= 14( u−v^2 u +v^2)−

=−u, v ,

the property holds forλ =−1. By linearity and by induction, for anyn∈ N withn≥ 1, writingn =n− 1 + 1, we get

λx, y =λ x, y for allλ∈ N,

and since the above also holds forλ =−1, it holds for allλ∈ Z. Forλ =p/q withp, q∈ Z andq = 0, we have

q (p/q)u, v = pu, v =p u, v , which shows that

(p/q)u, v = (p/q)u, v , and thus

λx, y =λ x, y for allλ∈ Q.

To finish the proof, we use the fact that a norm is a continuous mapx→ x . Then, the continuous function

t

1

→

0

t

tu, v defined on R− {0} agrees with u, von Q− {0}, so it is equal to u, v on R− { }. The caseλ = 0 is trivial, so we are done.

We now define orthogonality.

9.2 Orthogonality, Duality, Adjoint of a Linear Map

An inner product on a vector space gives the ability to define the notion of orthogonality. Families of nonnull pairwise orthogonal vectors must be linearly independent. They are called orthogonal families. In a vector space of finite dimension it is always possible to find orthogonal bases. This is very useful theoretically and practically. Indeed, in an orthogonal basis, finding the coordinates of a vector is very cheap: It takes an inner product. Fourier series make crucial use of this fact. WhenE has finite dimension, we prove that the inner product onE induces a natural isomorphism betweenE and its dual spaceE∗. This allows us to define the adjoint of a linear map in an intrinsic fashion (i.e., independently of bases). It is also possible to orthonormalize any basis (certainly when the dimension is finite). We give two proofs, one using duality, the other more constructive using the Gram–Schmidt orthonormalization procedure.

Definition 9.2. Given a Euclidean spaceE, any two vectorsu, v∈E are orthogonal, or perpendicular, ifu·v = 0. Given a family (ui)i∈I of vectors inE, we say that (ui)i∈I is orthogonal ifui·uj = 0 for alli, j∈I, wherei =j. We say that the family (ui)i∈I is orthonormal ifui·uj = 0 for alli, j∈I, wherei =j, and ui =ui·ui = 1, for alli∈I. For any subsetF ofE, the set

F⊥ ={v∈E|u·v = 0, for allu∈F},

of all vectors orthogonal to all vectors inF, is called the orthogonal complement ofF. Since inner products are positive definite, observe that for any vectoru∈E, we have

u·v = 0 for allv∈E iff u = 0.

It is immediately verified that the orthogonal complementF⊥ ofF is a subspace ofE.

Example 9.5. Going back to Example 9.3 and to the inner product

π

f, g = f(t)g(t)dt

−π

on the vector spaceC[−π, π], it is easily checked that

sin

px,

sin

qx

=

π ifp =q,p, q≥ 1, 0 ifp =q,p, q≥ 1,

cos

px,

cos

qx

=

π ifp =q,p, q≥ 1, 0 ifp =q,p, q≥ 0, and

sinpx,cosqx = 0,

for allp≥ 1 andq≥ 0, and of course, 1,1 = π dx = 2π.

−π

As a consequence, the family (sinpx)p≥1∪(cosqx)q≥0 is orthogonal. It is not orthonormal, but becomes so if we divide every trigonometric function by√π, and 1 by√2π.

We leave the following simple two results as exercises. Proposition 9.3. Given a Euclidean spaceE, for any family (ui)i∈I of nonnull vectors in E, if (ui)i∈I is orthogonal, then it is linearly independent.

Proposition 9.4. Given a Euclidean spaceE, any two vectorsu, v∈E are orthogonal iff

u +v^2 = u^2 + v^2.

One of the most useful features of orthonormal bases is that they afford a very simple method for computing the coordinates of a vector over any basis vector. Indeed, assume that (e_1, . . . , em) is an orthonormal basis. For any vector

x =x_1e_1 +· · · +xmem,

if we compute the inner productx·ei, we get

x·ei =x_1e1·ei +· · · +xiei·ei +· · · +xmem·ei =xi, since e

i

·

e

j

=

1 ifi =j,

0 ifi =j

is the property characterizing an orthonormal family. Thus, xi =x·ei,

which means that xiei = (x·ei)ei is the orthogonal projection of x onto the subspace generated by the basis vectorei. If the basis is orthogonal but not necessarily orthonormal,

then

x x·ei =x·ei.i = ei·ei ei 2

All this is true even for an infinite orthonormal (or orthogonal) basis (ei)i∈I. However, remember that every vectorx is expressed as a linear combination

x = xiei

i∈I

where the family of scalars ( xi)i∈I has finite support, which means thatxi = 0 for all i∈I−J, whereJ is a finite set. Thus, even though the family (sinpx)p≥1∪ (cosqx)q≥0 is orthogonal (it is not orthonormal, but becomes so if we divide every trigonometric function by

√

π

, and 1 by

√

2

π

; we won’t because it looks messy!), the fact that a function f

0

∈ C [−π, π] can be written as a Fourier series as

f(x) =a_0 + ∞ (ak coskx +bk sinkx)

k=1

does not mean that (sin px)p≥1∪ (cosqx)q≥ 0 is a basis of this vector space of functions, because in general, the families (ak) and (bk) do not have finite support! In order for this infinite linear combination to make sense, it is necessary to prove that the partial sums

n

a_0 + (ak coskx +bk sinkx)

k=1

of the series converge to a limit whenn goes to infinity. This requires a topology on the space.

A very important property of Euclidean spaces of finite dimension is that the inner product induces a canonical bijection (i.e., independent of the choice of bases) between the vector spaceE and its dualE∗.

Given a Euclidean spaceE, for any vectoru∈E, letÕu:E→ R be the map defined such that

Õu(v) =u·v, for allv∈E.

Since the inner product is bilinear, the mapÕu is a linear form inE∗. Thus, we have a map :E→E∗, defined such that

(u) =Õu.

Theorem 9.5. Given a Euclidean spaceE, the map :E→E∗ defined such that

(u) =Õu

is linear and injective. WhenE is also of finite dimension, the map :E→E∗ is a canonical isomorphism.

Proof. That :E→E∗ is a linear map follows immediately from the fact that the inner product is bilinear. IfÕu =Õv, thenÕu(w) =Õv(w) for allw∈E, which by definition ofÕu means that

u·w =v·w for allw∈E, which by bilinearity is equivalent to

(v−u)·w = 0

for all w∈E, which implies thatu =v, since the inner product is positive definite. Thus, :E→E∗ is injective. Finally, whenE is of finite dimensionn, we know thatE∗ is also of

dimensionn, and then :E→E∗ is bijective.

The inverse of the isomorphism :E→E∗ is denoted by :E∗→E.

As a consequence of Theorem 9.5, ifE is a Euclidean space of finite dimension, every linear formf∈E∗ corresponds to a uniqueu∈E such that

f(v) =u·v,

for everyv∈E. In particular, iff is not the null form, the kernel off, which is a hyperplane H, is precisely the set of vectors that are orthogonal tou.

Remarks:

(1) The “musical map” :E→E∗ is not surjective whenE has infinite dimension. The result can be salvaged by restricting our attention to continuous linear maps, and by assuming that the vector spaceE is a Hilbert space (i.e.,E is a complete normed vector space w.r.t. the Euclidean norm). This is the famous “little” Riesz theorem (or Riesz representation theorem).

(2) Theorem 9.5 still holds if the inner product on E is replaced by a nonde¿generate symmetric bilinear formÕ. We say that a symmetric bilinear formÕ:E×E→ R is nondegenerate if for everyu∈E,

if Õ(u, v) = 0 for allv∈E, then u = 0.

For example, the symmetric bilinear form on R^4 defined such that

Õ((x_1, x_2, x_3, x_4), (y_1, y_2, y_3, y_4)) =x_1y_1 +x_2y_2 +x_3y3−x_4y_4

is nondegenerate. However, there are nonnull vectorsu∈ R^4 such thatÕ(u, u) = 0, which is impossible in a Euclidean space. Such vectors are called isotropic.

The existence of the isomorphism :E→E∗ is crucial to the existence of adjoint maps. The importance of adjoint maps stems from the fact that the linear maps arising in physical problems are often self-adjoint, which means thatf =f∗. Moreover, self-adjoint maps can be diagonalized over orthonormal bases of eigenvectors. This is the key to the solution of many problems in mechanics, and engineering in general (see Strang [100]).

LetE be a Euclidean space of finite dimensionn, and letf :E→E be a linear map. For everyu∈E, the map

v→u·f(v)

is clearly a linear form inE∗, and by Theorem 9.5, there is a unique vector inE denoted by f∗(u) such that

f∗(u)·v =u·f(v),

for everyv∈E. The following simple proposition shows that the mapf∗ is linear. Proposition 9.6. Given a Euclidean spaceE of finite dimension, for every linear map f :E→E, there is a unique linear mapf∗:E→E such that

f∗(u)·v =u·f(v),

for allu, v∈E. The mapf∗ is called the adjoint off (w.r.t. to the inner product).

Proof. Givenu_1, u2∈E, since the inner product is bilinear, we have

(u_1 +u_2)·f(v) =u1·f(v) +u2·f(v),

for allv∈E, and (f∗(u_1) +f∗(u_2))·v =f∗(u_1)·v +f∗(u_2)·v, for allv∈E, and since by assumption,

f∗(u_1)·v =u1·f(v) and f∗(u_2)·v =u2·f(v), for allv∈E, we get

(f∗(u_1) +f∗(u_2))·v = (u_1 +u_2)·f(v), for allv∈E. Since is bijective, this implies that

f∗(u_1 +u_2) =f∗(u_1) +f∗(u_2).

Similarly, (λu)·f(v) =λ(u·f(v)), for allv∈E, and

(λf∗(u))·v =λ(f∗(u)·v), for allv∈E, and since by assumption,

f∗(u)·v =u·f(v),

for allv∈E, we get (λf∗(u))·v = (λu)·f(v), for allv∈E. Since is bijective, this implies that

f∗(λu) =λf∗(u).

Thus,f∗ is indeed a linear map, and it is unique, since is a bijection.

Linear maps f :E→E such thatf =f∗ are called self-adjoint maps. They play a very important role because they have real eigenvalues, and because orthonormal bases arise from their eigenvectors. Furthermore, many physical problems lead to self-adjoint linear maps (in the form of symmetric matrices).

Remark: Proposition 9.6 still holds if the inner product onE is replaced by a nondegenerate symmetric bilinear formÕ.

Linear maps such thatf−1 =f∗, or equivalently

f∗æf =fæf∗ = id,

also play an important role. They are linear isometries, or isometries. Rotations are special kinds of isometries. Another important class of linear maps are the linear maps satisfying the property

f∗æf =fæf∗, called normal linear maps. We will see later on that normal maps can always be diagonalized over orthonormal bases of eigenvectors, but this will require using a Hermitian inner product (over C).

Given two Euclidean spaces E andF, where the inner product onE is denoted by− ,−1 and the inner product onF is denoted by−,−2, given any linear mapf :E→F, it is immediately verified that the proof of Proposition 9.6 can be adapted to show that there is a unique linear mapf∗:F→E such that

f(u), v 2 = u, f∗(v) 1 for allu∈E and allv∈F. The linear mapf∗ is also called the adjoint off.

Remark: Given any basis forE and any basis forF, it is possible to characterize the matrix of the adjointf∗ off in terms of the matrix off, and the symmetric matrices defining the inner products. We will do so with respect to orthonormal bases. Also, since inner products are symmetric, the adjointf∗ off is also characterized by

f(u)·v =u·f∗(v),

for allu, v∈E.

We can also use Theorem 9.5 to show that any Euclidean space of finite dimension has an orthonormal basis.

Proposition 9.7. Given any nontrivial Euclidean spaceE of finite dimensionn≥ 1, there is an orthonormal basis (u_1, . . . , un) forE.

Proof. We proceed by induction onn. Whenn = 1, take any nonnull vectorv∈E, which exists, since we assumedE nontrivial, and let

u = v .v

Ifn≥ 2, again take any nonnull vectorv∈E, and let

u v .1 =v

Consider the linear form Õu1 associated withu_1. Sinceu_1 = 0, by Theorem 9.5, the linear formÕu1 is nonnull, and its kernel is a hyperplaneH. SinceÕu1(w) = 0 iffu1·w = 0, the hyperplaneH is the orthogonal complement of{u1}. Furthermore, sinceu_1 = 0 and the inner product is positive definite,u1·u_1 = 0, and thus,u_1 /∈H, which implies that E =H⊕Ru_1. However, sinceE is of finite dimensionn, the hyperplaneH has dimension n−1, and by the induction hypothesis, we can find an orthonormal basis (u_2, . . . , un) forH. Now, becauseH and the one dimensional space Ru_1 are orthogonal andE =H⊕Ru_1, it is clear that (u_1, . . . , un) is an orthonormal basis forE.

There is a more constructive way of proving Proposition 9.7, using a procedure known as the Gram–Schmidt orthonormalization procedure. Among other things, the Gram–Schmidt orthonormalization procedure yields theQR-decomposition for matrices, an important tool in numerical methods.

Proposition 9.8. Given any nontrivial Euclidean spaceE of finite dimensionn≥ 1, from any basis (e_1, . . . , en) forE we can construct an orthonormal basis (u_1, . . . , un) forE, with the property that for everyk, 1≤k≤n, the families (e_1, . . . , ek) and (u_1, . . . , uk) generate the same subspace.

Proof. We proceed by induction onn. Forn = 1, let

u e1 .1 =e1

Forn≥ 2, we also let

u_1 = e1 ,e1

and assuming that (u_1, . . . , uk) is an orthonormal system that generates the same subspace as (e_1, . . . , ek), for everyk with 1≤k < n, we note that the vector

k

uk+1 =ek+1− (ek+1

i=1 ·ui)ui

is nonnull, since otherwise, because ( u_1, . . . , uk) and (e_1, . . . , ek) generate the same subspace, (e_1, . . . , ek+1) would be linearly dependent, which is absurd, since (e_1, . . .,en) is a basis. Thus, the norm of the vectoruk+1 being nonzero, we use the following construction of the vectorsuk anduk:

u_1 =e_1, u_1 = u1 ,u1

and for the inductive step

k

u

(

e

k+1

uk+1 ,k+1 =ek+1− i=1 ·ui)ui, uk+1 =uk+1

where 1≤k≤n− 1. It is clear that uk+1 = 1, and since (u_1, . . . , uk) is an orthonormal system, we have

uk+1·ui =ek+1·ui− (ek+1·ui)ui·ui =ek+1·ui−ek+1·ui = 0,

for all i with 1≤i≤k. This shows that the family (u_1, . . . , uk+1) is orthonormal, and since (u_1, . . . , uk) and (e_1, . . . , ek) generates the same subspace, it is clear from the definition of uk+1 that (u_1, . . . , uk+1) and (e_1, . . . , ek+1) generate the same subspace. This completes the induction step and the proof of the proposition.

Note thatuk+1 is obtained by subtracting fromek+1 the projection ofek+1 itself onto the orthonormal vectorsu_1, . . . , uk that have already been computed. Then,uk+1 is normalized.

Remarks:

(1) TheQR-decomposition can now be obtained very easily, but we postpone this until Section 9.4.

(2) We could computeuk+1 using the formula

u k ek+1·ui ui,k+1 =ek+1− i=1 ui 2

and normalize the vectors uk at the end. This time, we are subtracting fromek+1 the projection ofek+1 itself onto the orthogonal vectorsu_1, . . . , uk. This might be preferable when writing a computer program.

(3) The proof of Proposition 9.8 also works for a countably infinite basis forE, producing a countably infinite orthonormal basis.

Example 9.6. If we consider polynomials and the inner product

1

f, g = f(t)g(t)dt,

−1

applying the Gram–Schmidt orthonormalization procedure to the polynomials

1, x, x^2, . . . , xn, . . . ,

which form a basis of the polynomials in one variable with real coefficients, we get a family of orthonormal polynomialsQn(x) related to the Legendre polynomials.

The Legendre polynomials Pn(x) have many nice properties. They are orthogonal, but their norm is not always 1. The Legendre polynomialsPn(x) can be defined as follows. Lettingfn be the function

fn(x) = (x^2 1)n,− we definePn(x) as follows:

P

0

(

x

) = 1

,

and

P

n

(

x

) =

1

2nn!f(n)(x),n

wheref(n) is thenth derivative offn.n

They can also be defined inductively as follows:

P_0(x) = 1,

P_1(x) = x,

P

n

+1

(

x

)

=

2n + 1 xPn(x) n

− n + 1Pn−1(x).n + 1

The polynomialsQn are related to the Legendre polynomialsPn as follows: 2

n

+ 1

Qn(x) =2 Pn(x).

As a consequence of Proposition 9.7 (or Proposition 9.8), given any Euclidean space of finite dimensionn, if (e_1, . . . , en) is an orthonormal basis forE, then for any two vectors u =u_1e_1 +· · · +unen andv =v_1e_1 +· · · +vnen, the inner productu·v is expressed as

n

u·v = (u_1e_1 +· · · +unen)· (v_1e_1 +· · · +vnen) = uivi,

i=1

and the norm u as

n 1/2

u = u_1e_1 +· · · +unen = u^2 .

i=1

The fact that a Euclidean space always has an orthonormal basis implies that any Gram matrixG can be written as

G =Q Q,

for some invertible matrix Q. Indeed, we know that in a change of basis matrix, a Gram matrixG becomesG =P GP. If the basis corresponding toG is orthonormal, thenG =I, soG = (P−1) P−1.

We can also prove the following proposition regarding orthogonal spaces.

Proposition 9.9. Given any nontrivial Euclidean spaceE of finite dimensionn≥ 1, for any subspaceF of dimensionk, the orthogonal complementF⊥ ofF has dimensionn−k, andE =F⊕F⊥. Furthermore, we haveF⊥⊥ =F.

Proof. From Proposition 9.7, the subspaceF has some orthonormal basis (u_1, . . . , uk). This linearly independent family (u_1, . . . , uk) can be extended to a basis (u_1, . . . , uk,vk+1, . . . , vn), and by Proposition 9.8, it can be converted to an orthonormal basis (u_1, . . . , un), which contains (u_1, . . . , uk) as an orthonormal basis ofF. Now, any vectorw =w_1u_1+· · ·+wnun∈ E is orthogonal toF iffw·ui = 0, for everyi, where 1≤i≤ k, iffwi = 0 for everyi, where 1≤i≤k. Clearly, this shows that (uk+1, . . . , un) is a basis ofF⊥, and thusE =F⊕F⊥, and F⊥ has dimensionn−k. Similarly, any vectorw =w_1u_1 +· · · +wnun∈E is orthogonal to F⊥ iffw·ui = 0, for everyi, wherek+ 1≤i≤ n, iffwi = 0 for everyi, wherek+ 1≤i≤n. Thus, (u_1, . . . , uk) is a basis ofF⊥⊥, andF⊥⊥ =F.

9.3 Linear Isometries (Orthogonal Transformations)

In this section we consider linear maps between Euclidean spaces that preserve the Euclidean norm. These transformations, sometimes called rigid motions, play an important role in geometry.

Definition 9.3. Given any two nontrivial Euclidean spacesE andF of the same finite dimensionn, a functionf :E→F is an orthogonal transformation, or a linear isometry, if it is linear and

f(u) = u , for allu∈E.

Remarks:

(1) A linear isometry is often defined as a linear map such that

f(v)−f(u) = v−u ,

for allu, v∈E. Since the mapf is linear, the two definitions are equivalent. The second definition just focuses on preserving the distance between vectors.

(2) Sometimes, a linear map satisfying the condition of Definition 9.3 is called a metric map, and a linear isometry is defined as a bijective metric map.

An isometry (without the word linear) is sometimes defined as a functionf :E→F (not necessarily linear) such that

f(v)−f(u) = v−u ,

for all u, v∈E, i.e., as a function that preserves the distance. This requirement turns out to be very strong. Indeed, the next proposition shows that all these definitions are equivalent whenE andF are of finite dimension, and for functions such thatf(0) = 0.

Proposition 9.10. Given any two nontrivial Euclidean spacesE andF of the same finite dimensionn, for every functionf :E→F, the following properties are equivalent:

(1) f is a linear map and f(u) = u , for allu∈E;

(2) f(v)−f(u) = v−u , for allu, v∈E, andf(0) = 0;

(3) f(u)·f(v) =u·v, for allu, v∈E.

Furthermore, such a map is bijective. Proof. Clearly, (1) implies (2), since in (1) it is assumed thatf is linear.

Assume that (2) holds. In fact, we shall prove a slightly stronger result. We prove that if

f(v)−f(u) = v−u

for allu, v∈E, then for any vectorτ∈E, the functiong:E→F defined such that

g(u) =f(τ +u)−f(τ)

for allu∈E is a linear map such thatg(0) = 0 and (3) holds. Clearly,g(0) =f(τ)−f(τ) = 0. Note that from the hypothesis

f(v)−f(u) = v−u

for allu, v∈E, we conclude that

g (v)−g(u) = f(τ +v)−f(τ)− (f(τ +u)−f(τ)) , = f(τ +v)−f(τ +u) ,

= τ +v− (τ +u) ,

= v−u ,

for allu, v∈E. Sinceg(0) = 0, by settingu = 0 in

g(v)−g(u) = v−u ,

we get g(v) = v

for allv∈E. In other words,g preserves both the distance and the norm. To prove thatg preserves the inner product, we use the simple fact that

2 + v^2 u−v^22u·v = u−

for allu, v∈E. Then, sinceg preserves distance and norm, we have

2g(u)·g(v) = g(u)2 + g(v)2 g(u)−g(v)2

2

+

v

2

−

u−v^2= u−= 2u·v,

and thusg(u)·g(v) =u·v, for allu, v∈E, which is (3). In particular, iff(0) = 0, by letting τ = 0, we haveg =f, andf preserves the scalar product, i.e., (3) holds.

Now assume that (3) holds. SinceE is of finite dimension, we can pick an orthonormal basis (e_1, . . . , en) forE. Sincef preserves inner products, (f(e_1), . . ., f(en)) is also orthonormal, and sinceF also has dimensionn, it is a basis ofF. Then note that for any u =u_1e_1 +· · · +unen, we have

ui =u·ei, for alli, 1≤i≤n. Thus, we have

n

f(u) = (f(u)·f(ei))f(ei),

i=1

and sincef preserves inner products, this shows that

n n

f(u) = (u·ei)f(ei) = uif(ei),

i=1 i=1

which shows thatf is linear. Obviously,f preserves the Euclidean norm, and (3) implies (1).

Finally, if f(u) =f(v), then by linearityf(v−u) = 0, so that f(v− u) = 0, and since f preserves norms, we must have v−u = 0, and thusu =v. Thus, f is injective, and sinceE andF have the same finite dimension,f is bijective.

Remarks:

(i) The dimension assumption is needed only to prove that (3) implies (1) when f is not known to be linear, and to prove thatf is surjective, but the proof shows that (1) implies thatf is injective.

(ii) The implication that (3) implies (1) holds if we also assume thatf is surjective, even ifE has infinite dimension.

In (2), whenf does not satisfy the conditionf(0) = 0, the proof shows thatf is an affine map. Indeed, taking any vectorτ as an origin, the mapg is linear, and

f(τ +u) =f(τ) +g(u) for allu∈E.

From section 19.7, this shows thatf is affine with associated linear mapg. This fact is worth recording as the following proposition.

Proposition 9.11. Given any two nontrivial Euclidean spacesE andF of the same finite dimensionn, for every functionf :E→F, if

f(v)−f(u) = v−u for allu, v∈E,

thenf is an affine map, and its associated linear mapg is an isometry.

In view of Proposition 9.10, we will drop the word “linear” in “linear isometry,” unless we wish to emphasize that we are dealing with a map between vector spaces. We are now going to take a closer look at the isometriesf :E→E of a Euclidean space of finite dimension.

9.4 The Orthogonal Group, Orthogonal Matrices

In this section we explore some of the basic properties of the orthogonal group and of orthogonal matrices.

Proposition 9.12. LetE be any Euclidean space of finite dimensionn, and letf :E→E be any linear map. The following properties hold:

(1) The linear mapf :E→E is an isometry iff

fæf∗ =f∗æf = id.

(2) For every orthonormal basis (e_1, . . . , en) ofE, if the matrix off isA, then the matrix off∗ is the transposeA ofA, andf is an isometry iffA satisfies the identities

A A =A A =In,

whereIn denotes the identity matrix of ordern, iff the columns ofA form an orthonormal basis ofE, iff the rows ofA form an orthonormal basis ofE.

Proof. (1) The linear mapf :E→E is an isometry iff

f(u)·f(v) =u·v,

for allu, v∈E, iff f∗(f(u))·v =f(u)·f(v) =u·v for allu, v∈E, which implies

(f∗(f(u))−u)·v = 0

for allu, v∈E. Since the inner product is positive definite, we must have

f∗(f(u))−u = 0

for allu∈E, that is, f∗æf =fæf∗ = id.

The converse is established by doing the above steps backward.

(2) If (e_1, . . . , en) is an orthonormal basis forE, letA = (ai j) be the matrix off, and let B = (bi j) be the matrix off∗. Sincef∗ is characterized by

f∗(u)·v =u·f(v) 9.4. THE ORTHOGONAL GROUP, ORTHOGONAL MATRICES 275

for allu, v∈E, using the fact that ifw =w_1e_1 +· · · +wnen we havewk =w·ek for allk, 1≤k≤n, lettingu =ei andv =ej, we get

bj i =f∗(ei)·ej =ei·f(ej) =ai j,

for all i, j, 1≤i, j≤n. Thus,B =A . Now, ifX andY are arbitrary matrices over the basis (e_1, . . . , en), denoting as usual thejth column ofX byXj, and similarly forY , a simple calculation shows that

X Y = (Xi Yj)1≤i,j≤n.· Then it is immediately verified that ifX =Y =A, then A A =A A =In iff the column vectors (A^1, . . . , An) form an orthonormal basis. Thus, from (1), we see that (2) is clear (also because the rows ofA are the columns ofA ).

Proposition 9.12 shows that the inverse of an isometry f is its adjointf∗. Recall that the set of all realn×n matrices is denoted by Mn(R). Proposition 9.12 also motivates the following definition.

Definition 9.4. A realn×n matrix is an orthogonal matrix if

A A =A A =In.

Remark: It is easy to show that the conditionsA A =In,A A =In, andA−1 =A , are equivalent. Given any two orthonormal bases (u_1, . . . , un) and (v_1, . . . , vn), ifP is the change of basis matrix from (u_1, . . . , un) to (v_1, . . . , vn), since the columns ofP are the coordinates of the vectorsvj with respect to the basis (u_1, . . . , un), and since (v_1, . . . , vn) is orthonormal, the columns ofP are orthonormal, and by Proposition 9.12 (2), the matrixP is orthogonal.

The proof of Proposition 9.10 (3) also shows that if f is an isometry, then the image of an orthonormal basis (u_1, . . . , un) is an orthonormal basis. Students often ask why orthogonal matrices are not called orthonormal matrices, since their columns (and rows) are orthonormal bases! I have no good answer, but isometries do preserve orthogonality, and orthogonal matrices correspond to isometries.

Recall that the determinant det( f) of a linear mapf :E→E is independent of the choice of a basis inE. Also, for every matrixA∈ Mn(R), we have det(A) = det(A ), and for any twon×n matricesA andB, we have det(AB) = det(A) det(B). Then, iff is an isometry, andA is its matrix with respect to any orthonormal basis,A A =A A =In implies that det(A)2 = 1, that is, either det(A) = 1, or det(A) =−1. It is also clear that the isometries of a Euclidean space of dimensionn form a group, and that the isometries of determinant +1 form a subgroup. This leads to the following definition.

Definition 9.5. Given a Euclidean spaceE of dimensionn, the set of isometriesf :E→E forms a subgroup of GL(E) denoted by O(E), or O(n) whenE = Rn, called the orthogonal group (ofE). For every isometryf, we have det(f) =±1, where det(f) denotes the determinant off. The isometries such that det(f) = 1 are called rotations, or proper isometries, or proper orthogonal transformations, and they form a subgroup of the special linear group SL(E) (and of O(E)), denoted by SO(E), or SO(n) whenE = Rn, called the special orthogonal group (ofE). The isometries such that det(f) =−1 are called improper isometries, or improper orthogonal transformations, or flip transformations .

As an immediate corollary of the Gram–Schmidt orthonormalization procedure, we obtain theQR-decomposition for invertible matrices.

9.5 QR-Decomposition for Invertible Matrices

Now that we have the definition of an orthogonal matrix, we can explain how the Gram– Schmidt orthonormalization procedure immediately yields theQR-decomposition for matrices.

Proposition 9.13. Given any realn×n matrixA, ifA is invertible, then there is an orthogonal matrixQ and an upper triangular matrixR with positive diagonal entries such thatA =QR.

Proof. We can view the columns ofA as vectorsA^1, . . . , An inEn. IfA is invertible, then they are linearly independent, and we can apply Proposition 9.8 to produce an orthonormal basis using the Gram–Schmidt orthonormalization procedure. Recall that we construct vectors Qk andQk as follows:

Q

1 =A^1, Q^1 = Q 1 Q 1 ,

and for the inductive step

k Qk+1 Qk+1 =Ak+1 (Ak+1 Qi)Qi, Qk+1 =Qk+1 ,− i=1 ·

where 1≤k≤n− 1. If we express the vectorsAk in terms of theQi andQi, we get the triangular system

A^1 = Q 1 Q^1,

.

Aj = (Aj Q^1)Q^1 +· · · + (Aj Qi)Qi +· · · + Qj Qj,· ·

.

An = (An Q^1)Q^1 +· · · + (An Qn−1)Qn−1 + Qn Qn.· ·

9.5. QR-DECOMPOSITION FOR INVERTIBLE MATRICES 277

Letting rk k = Qk , andri j =Aj Qi (the reversal ofi andj on the right-hand side is intentional!), where 1≤k≤n, 2≤j· n, and 1≤i≤j− 1, and lettingqi j be theith

component of

Q

j

≤

, we note that ai j, theith component ofAj, is given by

ai j =r1 jqi 1 +· · · +ri jqi i +· · · +rj jqi j =qi 1r1 j +· · · +qi iri j +· · · +qi jrj j.

If we let Q = (qi j), the matrix whose columns are the components of theQj, andR = (ri j), the above equations show thatA =QR, whereR is upper triangular. The diagonal entries rk k = Qk =Ak Qk are indeed positive.·

The reader should try the above procedure on some concrete examples for 2×2 and 3×3 matrices.

Remarks:

(1) Because the diagonal entries ofR are positive, it can be shown thatQ andR are unique.

(2) The QR-decomposition holds even whenA is not invertible. In this case,R has some zero on the diagonal. However, a different proof is needed. We will give a nice proof using Householder matrices (see Proposition 10.3, and also Strang [100, 101], Golub and Van Loan [47], Trefethen and Bau [106], Demmel [25], Kincaid and Cheney [61], or Ciarlet [22]).

Example 9.7. Consider the matrix

ë0 0 5ö

A = í0 4 1ø.

1 1 1

We leave as an exercise to show thatA =QR, with

ë0 0 1ö ë1 1 1ö Q = í0 1 0ø and R = í0 4 1ø. 1 0 0 0 0 5 Example 9.8. Another example ofQR-decomposition is

ë1 1 2ö ë1/√2 1/√2 0öë√2 1/√2 √2ö A = í0 0 1ø = í 0 0 1øí 0 1/√2 √2ø. 1 0 0 1/√2 −1/√2 0 0 0 1

The QR-decomposition yields a rather efficient and numerically stable method for solving systems of linear equations. Indeed, given a systemAx =b, whereA is ann×n invertible matrix, writingA =QR, sinceQ is orthogonal, we get

Rx =Q b,

and sinceR is upper triangular, we can solve it by Gaussian elimination, by solving for the last variablexn first, substituting its value into the system, then solving forxn−1, etc. The QR-decomposition is also very useful in solving least squares problems (we will come back to this later on), and for finding eigenvalues. It can be easily adapted to the case whereA is a rectangularm×n matrix with independent columns (thus,n≤m). In this case,Q is not quite orthogonal. It is anm×n matrix whose columns are orthogonal, andR is an invertible n×n upper triangular matrix with positive diagonal entries. For more onQR, see Strang [100, 101], Golub and Van Loan [47], Demmel [25], Trefethen and Bau [106], or Serre [92].

It should also be said that the Gram–Schmidt orthonormalization procedure that we have presented is not very stable numerically, and instead, one should use the modified Gram– Schmidt method. To computeQk+1, instead of projectingAk+1 ontoQ^1, . . . , Qk in a single step, it is better to performk projections. We computeQk+1 , Qk+1 , . . . , Qk+1 as follows:1 2 k

Qk+1 = Ak+1 (Ak+1 Q^1)Q^1,1 − (Qk+1·Qi+1)Qi+1,i+1 = Qk+1Qk+1

i −i ·

where 1≤i≤k−1. It is easily shown thatQk+1 =Qk+1. The reader is urged to code thisk

method.

9.6 Some Applications of Euclidean Geometry

Euclidean geometry has applications in computational geometry, in particular Voronoi diagrams and Delaunay triangulations. In turn, Voronoi diagrams have applications in motion planning (see O’Rourke [84]).

Euclidean geometry also has applications to matrix analysis. Recall that a real n×n matrixA is symmetric if it is equal to its transposeA . One of the most important properties of symmetric matrices is that they have real eigenvalues and that they can be diagonalized by an orthogonal matrix (see Chapter 13). This means that for every symmetric matrixA, there is a diagonal matrixD and an orthogonal matrixP such that

A =P DP .

Even though it is not always possible to diagonalize an arbitrary matrix, there are various decompositions involving orthogonal matrices that are of great practical interest. For example, for every real matrixA, there is theQR-decomposition, which says that a real matrix A can be expressed as

A =QR,

where Q is orthogonal andR is an upper triangular matrix. This can be obtained from the Gram–Schmidt orthonormalization procedure, as we saw in Section 9.5, or better, using Householder matrices, as shown in Section 10.2. There is also the polar decomposition, which says that a real matrixA can be expressed as

A =QS,

9.7. SUMMARY 279

where Q is orthogonal andS is symmetric positive semidefinite (which means that the eigenvalues ofS are nonnegative). Such a decomposition is important in continuum mechanics and in robotics, since it separates stretching from rotation. Finally, there is the wonderful singular value decomposition, abbreviated as SVD, which says that a real matrixA can be expressed as

A =V DU ,

where U andV are orthogonal andD is a diagonal matrix with nonnegative entries (see Chapter 16). This decomposition leads to the notion of pseudo-inverse, which has many applications in engineering (least squares solutions, etc). For an excellent presentation of all these notions, we highly recommend Strang [101, 100], Golub and Van Loan [47], Demmel [25], Serre [92], and Trefethen and Bau [106].

The method of least squares, invented by Gauss and Legendre around 1800, is another great application of Euclidean geometry. Roughly speaking, the method is used to solve inconsistent linear systemsAx = b, where the number of equations is greater than the number of variables. Since this is generally impossible, the method of least squares consists in finding a solutionx minimizing the Euclidean norm Ax−b^2, that is, the sum of the squares of the “errors.” It turns out that there is always a unique solutionx+ of smallest norm minimizing Ax−b^2, and that it is a solution of the square system

A Ax =A b,

called the system of normal equations. The solutionx+ can be found either by using theQR- decomposition in terms of Householder transformations, or by using the notion of pseudoinverse of a matrix. The pseudo-inverse can be computed using the SVD decomposition. Least squares methods are used extensively in computer vision More details on the method of least squares and pseudo-inverses can be found in Chapter 17.

9.7 Summary

The main concepts and results of this chapter are listed below:

• Bilinear forms; positive definite bilinear forms.

• inner products, scalar products, Euclidean spaces.

• quadratic form associated with a bilinear form.

• The Euclidean space En.

• The polar form of a quadratic form.

• Gram matrix associated with an inner product.

• The Cauchy–Schwarz inequality; the Minkowski inequality.

• The parallelogram law.

• Orthogonality, orthogonal complementF⊥; orthonormal family.

•

The musical isomorphisms :E→E∗ and :E∗→E (whenE is finite-dimensional); Theorem 9.5.

adjoint of a linear map (with respect to an inner product).• The

•

Existence of an orthonormal basis in a finite-dimensional Euclidean space (Proposition

9.7).

• The Gram–Schmidt orthonormalization procedure (Proposition 9.8).

• Linear isometries (orthogonal transformations, rigid motions). orthogonal group, orthogonal matrices.• The

•

The matrix representing the adjointf∗ of a linear mapf is the transpose of the matrix representingf.

• The orthogonal group O(n) and the special orthogonal group SO(n).

• QR-decomposition for invertible matrices.

Chapter 10 QR-Decomposition for Arbitrary Matrices

10.1 Orthogonal Reflections

Hyperplane reflections are represented by matrices called Householder matrices. These matrices play an important role in numerical methods, for instance for solving systems of linear equations, solving least squares problems, for computing eigenvalues, and for transforming a symmetric matrix into a tridiagonal matrix. We prove a simple geometric lemma that immediately yields theQR-decomposition of arbitrary matrices in terms of Householder matrices.

Orthogonal symmetries are a very important example of isometries. First let us review the definition of projections. Given a vector spaceE, letF andG be subspaces ofE that form a direct sumE =F⊕G. Since everyu∈E can be written uniquely asu =v +w, wherev∈F andw∈G, we can define the two projectionspF :E→ F andpG:E→G such thatpF(u) =v andpG(u) =w. It is immediately verified thatpG andpF are linear maps, and thatp^2F =pF,p^2G =pG,pFæpG =pGæpF = 0, andpF +pG = id.

Definition 10.1. Given a vector spaceE, for any two subspacesF andG that form a direct sumE =F⊕G, the symmetry (or reflection) with respect toF and parallel toG is the linear maps:E→E defined such that

s(u) = 2pF(u)−u, for everyu∈E.

BecausepF +pG = id, note that we also have s(u) =pF(u)−pG(u) and s(u) =u− 2pG(u),

281 s^2 = id,s is the identity onF, ands =−id onG. We now assume thatE is a Euclidean space of finite dimension.

Definition 10.2. LetE be a Euclidean space of finite dimensionn. For any two subspaces F andG, ifF andG form a direct sumE =F⊕G andF andG are orthogonal, i.e., F =G⊥, the orthogonal symmetry (or reflection) with respect toF and parallel toG is the linear maps:E→E defined such that

s(u) = 2pF(u)−u,

for every u∈E. WhenF is a hyperplane, we calls a hyperplane symmetry with respect to F (or reflection aboutF), and whenG is a plane (and thus dim(F) =n− 2), we calls a flip aboutF.

For any two vectorsu, v∈E, it is easily verified using the bilinearity of the inner product that

u +v^2 u−v^2 = 4(u·v).−

Then, since

u =pF(u) +pG(u) and

s(u) =pF(u)−pG(u),

since F andG are orthogonal, it follows that

pF(u)·pG(v) = 0,

and thus, s(u) = u ,

so thats is an isometry.

Using Proposition 9.8, it is possible to find an orthonormal basis ( e_1, . . . , en) ofE consisting of an orthonormal basis ofF and an orthonormal basis ofG. Assume thatF has dimensionp, so thatG has dimensionn−p. With respect to the orthonormal basis (e_1, . . . , en), the symmetrys has a matrix of the form

I p 0 .0 −In−p

Thus, det(s) = (−1)n−p, ands is a rotation iffn−p is even. In particular, whenF is a hyperplaneH, we havep =n− 1 andn−p = 1, so thats is an improper orthogonal transformation. WhenF ={0}, we haves =−id, which is called the symmetry with respect to the origin. The symmetry with respect to the origin is a rotation iffn is even, and an improper orthogonal transformation iffn is odd. Whenn is odd, we observe that every improper orthogonal transformation is the composition of a rotation with the symmetry 10.1. ORTHOGONAL REFLECTIONS 283

with respect to the origin. When G is a plane,p =n−2, and det(s) = (−1)2 = 1, so that a flip aboutF is a rotation. In particular, whenn = 3,F is a line, and a flip about the line F is indeed a rotation of measureπ.

Remark: Given any two orthogonal subspacesF, G forming a direct sumE =F⊕G, let f be the symmetry with respect toF and parallel toG, and letg be the symmetry with respect toG and parallel toF. We leave as an exercise to show that

fæg =gæf =−id.

WhenF =H is a hyperplane, we can give an explicit formula fors(u) in terms of any nonnull vectorw orthogonal toH. Indeed, from

u =pH(u) +pG(u),

sincepG(u)∈G andG is spanned byw, which is orthogonal toH, we have pG(u) =λw

for someλ∈ R, and we get

u·w =λ w^2, and thus) = (u·w) pG(uw 2 w. Since s(u) =u− 2pG(u), we get

s

(

u

) =

u

−

2 (

u·w) w 2 w.

Such reflections are represented by matrices called Householder matrices, and they play an important role in numerical matrix analysis (see Kincaid and Cheney [61] or Ciarlet [22]). Householder matrices are symmetric and orthogonal. It is easily checked that over an orthonormal basis (e_1, . . . , en), a hyperplane reflection about a hyperplaneH orthogonal to a nonnull vectorw is represented by the matrix

W W =In− 2W WH =In− 2W 2 W W,

whereW is the column vector of the coordinates ofw over the basis (e_1, . . . , en), andIn is the identityn×n matrix. Since

p

G

(

u

) = (

u

·

w

)

w,w 2

the matrix representingpG is

W W

W W,

and sincepH +pG = id, the matrix representingpH is

W W

In− W W.

These formulae can be used to derive a formula for a rotation of R^3, given the directionw of its axis of rotation and given the angleθ of rotation.

The following fact is the key to the proof that every isometry can be decomposed as a product of reflections.

Proposition 10.1. LetE be any nontrivial Euclidean space. For any two vectorsu, v∈E, if u = v , then there is a hyperplaneH such that the reflections aboutH mapsu tov, and ifu =v, then this reflection is unique.

Proof. Ifu =v, then any hyperplane containingu does the job. Otherwise, we must have H ={v−u}⊥, and by the above formula,

s

(

u

) =

u

−

2 (

u· (v−u)) + 2 u 2 2u·v (v−u),(v−u) 2 (v−u) =u −

(v−u)2 and since (v−u)2 = u^2 + v^2 2u·v− and u = v , we have

(v−u)2 = 2 u^2 2u·v,− and thus,s(u) =v.

If E is a complex vector space and the inner product is Hermitian, Proposition 10.1 is false. The problem is that the vectorv−u does not work unless the inner product u·v is real! The proposition can be salvaged enough to yield theQR-decomposition in terms of Householder transformations; see Gallier [42].

We now show that hyperplane reflections can be used to obtain another proof of the QR-decomposition.

10.2 QR-Decomposition Using Householder Matrices

First, we state the result geometrically. When translated in terms of Householder matrices, we obtain the fact advertised earlier that every matrix (not necessarily invertible) has a QR-decomposition.

Proposition 10.2. LetE be a nontrivial Euclidean space of dimensionn. For any orthonormal basis (e_1, . . .,en) and for anyn-tuple of vectors (v_1,. . .,vn), there is a sequence ofn isometriesh_1, . . . , hn such thathi is a hyperplane reflection or the identity, and if (r_1, . . . , rn) are the vectors given by

rj =hnæ · · · æh2æh_1(vj),

then every rj is a linear combination of the vectors (e_1, . . . , ej), 1≤j≤n. Equivalently, the matrixR whose columns are the components of therj over the basis (e1, . . . , en) is an upper triangular matrix. Furthermore, thehi can be chosen so that the diagonal entries ofR are nonnegative.

Proof. We proceed by induction onn. Forn = 1, we havev_1 =λe_1 for someλ∈ R. If λ≥ 0, we leth_1 = id, else ifλ < 0, we leth_1 =−id, the reflection about the origin. Forn≥ 2, we first have to findh_1. Let

r1,1 = v_1 .

Ifv_1 =r1,1e_1, we leth_1 = id. Otherwise, there is a unique hyperplane reflectionh_1 such that

h_1(v_1) =r1,1e_1,

defined such that 2 (u·w1) h_1(u) =u− w1 2 w_1

for allu∈E, where w_1 =r1,1e1−v_1.

The maph_1 is the reflection about the hyperplaneH_1 orthogonal to the vectorw_1 =r1,1e1−v1. Letting

r_1 =h_1(v_1) =r1,1e_1,

it is obvious thatr_1 belongs to the subspace spanned bye_1, andr1,1 = v_1 is nonnegative. Next, assume that we have foundk linear mapsh_1, . . . , hk, hyperplane reflections or the identity, where 1≤k≤n− 1, such that if (r_1, . . . , rk) are the vectors given by

rj =hkæ · · · æh2æh_1(vj),

then every rj is a linear combination of the vectors (e_1, . . . , ej), 1≤j≤k. The vectors (e_1, . . . , ek) form a basis for the subspace denoted byUk, the vectors (ek+1, . . . , en) form a basis for the subspace denoted by Uk, the subspaces Uk and Uk are orthogonal, and E =Uk⊕Uk. Let

uk+1 =hkæ · · · æh2æh_1(vk+1). We can write

uk+1 =uk+1 +uk+1, whereuk+1∈Uk anduk+1∈Uk. Let rk+1,k+1 = uk+1 .

Ifuk+1 =rk+1,k+1ek+1, we lethk+1 = id. Otherwise, there is a unique hyperplane reflection hk+1 such that

hk+1(uk+1) =rk+1,k+1ek+1,

defined such that

h

k

+1

(

u

) =

u

−

2 (

u

·

w

k

+1

)

wk+1wk+1 2

for allu∈E, where

wk+1 =rk+1,k+1ek+1−uk+1.

The map hk+1 is the reflection about the hyperplaneHk+1 orthogonal to the vectorwk+1 = rk+1,k+1ek+1−uk+1. However, sinceuk+1, ek+1∈Uk andUk is orthogonal toUk, the subspace Uk is contained inHk+1, and thus, the vectors (r_1, . . . , rk) anduk+1, which belong toUk, are invariant underhk+1. This proves that

hk+1(uk+1) =hk+1(uk+1) +hk+1(uk+1) =uk+1 +rk+1,k+1ek+1

is a linear combination of (e_1, . . . , ek+1). Letting

rk+1 =hk+1(uk+1) =uk+1 +rk+1,k+1ek+1,

sinceuk+1 =hkæ · · · æh2æh_1(vk+1), the vector

rk+1 =hk+1æ · · · æh2æh_1(vk+1)

is a linear combination of (e_1, . . . , ek+1). The coefficient ofrk+1 overek+1 isrk+1,k+1 = uk+1 , which is nonnegative. This concludes the induction step, and thus the proof.

Remarks:

(1) Since everyhi is a hyperplane reflection or the identity,

ρ =hnæ · · · æh2æh_1

is an isometry.

(2) If we allow negative diagonal entries inR, the last isometryhn may be omitted.

(3) Instead of picking rk,k = uk , which means that

wk =rk,kek−uk,

where 1≤k≤n, it might be preferable to pickrk,k =−uk if this makes wk 2 larger, in which case

wk =rk,kek +uk.

Indeed, since the definition ofhk involves division by wk 2, it is desirable to avoid division by very small numbers.

(4) The method also applies to any m-tuple of vectors (v_1, . . . , vm), wherem is not necessarily equal ton (the dimension ofE). In this case,R is an upper triangularn×m matrix we leave the minor adjustments to the method as an exercise to the reader (if m > n, the lastm−n vectors are unchanged).

Proposition 10.2 directly yields the QR-decomposition in terms of Householder transformations (see Strang [100, 101], Golub and Van Loan [47], Trefethen and Bau [106], Kincaid and Cheney [61], or Ciarlet [22]).

Theorem 10.3. For every realn×n matrixA, there is a sequenceH_1, . . .,Hn of matrices, where eachHi is either a Householder matrix or the identity, and an upper triangular matrix R such that

R =Hn· · ·H_2H_1A.

As a corollary, there is a pair of matrices Q, R, where Q is orthogonal and R is upper triangular, such thatA =QR (aQR-decomposition ofA). Furthermore,R can be chosen so that its diagonal entries are nonnegative.

Proof. Thejth column ofA can be viewed as a vectorvj over the canonical basis (e_1, . . . , en) of En (where (ej)i = 1 ifi =j, and 0 otherwise, 1≤i, j≤n). Applying Proposition 10.2 to (v_1, . . . , vn), there is a sequence ofn isometriesh1, . . . , hn such thathi is a hyperplane reflection or the identity, and if (r_1, . . . , rn) are the vectors given by

rj =hnæ · · · æh2æh_1(vj),

then every rj is a linear combination of the vectors (e_1, . . . , ej), 1≤j≤n. LettingR be the matrix whose columns are the vectorsrj, andHi the matrix associated with hi, it is clear that

R =Hn· · ·H_2H_1A,

whereR is upper triangular and everyHi is either a Householder matrix or the identity. However,hiæhi = id for alli, 1≤i≤n, and so

vj =h1æh2æ · · · æhn(rj)

for all j, 1≤j≤n. Butρ =h1æh2æ · · · æ hn is an isometry represented by the orthogonal matrixQ =H1H2· · ·Hn. It is clear thatA =QR, whereR is upper triangular. As we noted in Proposition 10.2, the diagonal entries ofR can be chosen to be nonnegative. Remarks:

(1) Letting Ak+1 =Hk· · ·H_2H_1A,

with A_1 =A, 1≤k≤n, the proof of Proposition 10.2 can be interpreted in terms of the computation of the sequence of matricesA_1, . . . , An+1 =R. The matrixAk+1 has the shape

ë uk+1 ö ì× × ×1 × × × ×÷ ì0 × . . . . . .÷

ì

÷

ì

ì k+1

0

0

× uk+1 ÷ ì k × × × ×÷ A

k

+1

=

ì0 0 0 uk+1 ÷ ,ì0 0 0 uk+1 × × × ×÷

÷

ì k+2 × × × ×÷

ì. . . . . . . . ÷ ì ÷

ì0 0 0 uk+1 ÷ í n−1 × × × ×ø 0

0

0

u

k+1

n × × × × where the (k + 1)th column of the matrix is the vector

uk+1 =hkæ · · · æh2æh_1(vk+1), and thus u

k

+1

=

u

k+1 , . . . , uk+1 1 k

and

uk+1 = uk+1, uk+1, . . . , uk+1 .k+1 k+2 n

If the last n− k−1 entries in columnk+ 1 are all zero, there is nothing to do, and we letHk+1 =I. Otherwise, we kill thesen−k− 1 entries by multiplyingAk+1 on the left by the Householder matrixHk+1 sending

k+1, . . . , uk+1 to (0, . . . ,0, rk+1,k+1,0, . . . ,0),k+1 n

wherer 0, . . . ,0, u

k+1, . . . , uk+1) .k+1,k+1 = (uk+1 n

(2) IfA is invertible and the diagonal entries ofR are positive, it can be shown thatQ andR are unique.

(3) If we allow negative diagonal entries inR, the matrixHn may be omitted (Hn =I).

(4) The method allows the computation of the determinant ofA. We have det(A) = (mr1,1· · ·rn,n,−1)

wherem is the number of Householder matrices (not the identity) among theHi. 10.3. SUMMARY 289

(5) The “condition number” of the matrix A is preserved (see Strang [101], Golub and Van Loan [47], Trefethen and Bau [106], Kincaid and Cheney [61], or Ciarlet [22]). This is very good for numerical stability.

(6) The method also applies to a rectangularm×n matrix. In this case,R is also an m×n matrix (and it is upper triangular).

10.3 Summary

The main concepts and results of this chapter are listed below:

Symmetry (or reflection) with respect toF and parallel toG.•

•

Orthogonal symmetry (or reflection) with respect toF and parallel toG; reflections, flips.

Hyperplane reflections and Householder matrices.•

• A key fact about reflections (Proposition 10.1).

• QR-decomposition in terms of Householder transformations (Theorem 10.3).

Chapter 11 Hermitian Spaces

11.1 Sesquilinear and Hermitian Forms, Pre-Hilbert Spaces and Hermitian Spaces

In this chapter we generalize the basic results of Euclidean geometry presented in Chapter 9 to vector spaces over the complex numbers. Such a generalization is inevitable, and not simply a luxury. For example, linear maps may not have real eigenvalues, but they always have complex eigenvalues. Furthermore, some very important classes of linear maps can be diagonalized if they are extended to the complexification of a real vector space. This is the case for orthogonal matrices, and, more generally, normal matrices. Also, complex vector spaces are often the natural framework in physics or engineering, and they are more convenient for dealing with Fourier series. However, some complications arise due to complex conjugation.

Recall that for any complex number z∈ C, ifz =x +iy wherex, y∈ R, we let z =x, the real part ofz, and z =y, the imaginary part ofz. We also denote the conjugate of z =x +iy byz =x−iy, and the absolute value (or length, or modulus) ofz by|z|. Recall

that

|

z

2

| =zz =x^2 +y^2.

There are many natural situations where a map Õ:E×E→ C is linear in its first argument and only semilinear in its second argument, which means thatÕ(u, µv) =µÕ(u, v), as opposed toÕ(u, µv) =µÕ(u, v). For example, the natural inner product to deal with functionsf : R→ C, especially Fourier series, is

π

f, g = f(x)g(x)dx,

−π

which is semilinear (but not linear) in g. Thus, when generalizing a result from the real case of a Euclidean space to the complex case, we always have to check very carefully that our proofs do not rely on linearity in the second argument. Otherwise, we need to revise our proofs, and sometimes the result is simply wrong!

291 Before defining the natural generalization of an inner product, it is convenient to define semilinear maps.

Definition 11.1. Given two vector spacesE andF over the complex field C, a function f :E→F is semilinear if

f(u +v) =f(u) +f(v), f(λu) =λf(u),

for allu, v∈E and allλ∈ C.

Remark: Instead of defining semilinear maps, we could have defined the vector spaceE as the vector space with the same carrier setE whose addition is the same as that ofE, but whose multiplication by a complex number is given by

(λ, u)→λu.

Then it is easy to check that a functionf :E→ C is semilinear ifff :E→ C is linear. We can now define sesquilinear forms and Hermitian forms.

Definition 11.2. Given a complex vector spaceE, a functionÕ:E×E→ C is a sesquilinear form if it is linear in its first argument and semilinear in its second argument, which means that

Õ (u_1 +u_2, v) =Õ(u_1, v) +Õ(u_2, v), Õ(u, v_1 +v_2) =Õ(u, v_1) +Õ(u, v_2), Õ(λu, v) =λÕ(u, v),

Õ(u, µv) =µÕ(u, v),

for allu, v,u_1, u_2,v_1, v2∈E, and allλ, µ∈ C. A functionÕ:E×E→ C is a Hermitian form if it is sesquilinear and if

Õ(v, u) =Õ(u, v) for all allu, v∈E.

Obviously,Õ(0, v) =Õ(u,0) = 0. Also note that ifÕ:E×E→ C is sesquilinear, we have

Õ

(

λu

+

µv, λu

+

µv

) =

|

λ

2 2

| Õ(u, u) +λµÕ(u, v) +λµÕ(v, u) +|µ| Õ(v, v), and ifÕ:E×E→ C is Hermitian, we have

Õ

(

λu

+

µv, λu

+

µv

) =

|

λ

2 2

| Õ(u, u) + 2 (λµÕ(u, v)) +|µ| Õ(v, v).

Note that restricted to real coefficients, a sesquilinear form is bilinear (we sometimes say R-bilinear). The function Φ:E→ C defined such that Φ(u) =Õ(u, u) for allu∈E is called the quadratic form associated withÕ.

The standard example of a Hermitian form on Cn is the mapÕ defined such that Õ((x_1, . . . , xn),(y_1, . . . , yn)) =x_1y_1 +x_2y_2 +· · · +xnyn.

This map is also positive definite, but before dealing with these issues, we show the following useful proposition.

Proposition 11.1. Given a complex vector spaceE, the following properties hold: (1) A sesquilinear formÕ:E×E→ C is a Hermitian form iffÕ(u, u)∈ R for allu∈E.

(2) IfÕ:E×E→ C is a sesquilinear form, then

4Õ(u, v) =Õ(u +v, u +v)−Õ(u−v, u−v) +iÕ(u +iv, u +iv)−iÕ(u−iv, u−iv), and

2Õ(u, v) = (1 +i)(Õ(u, u) +Õ(v, v))−Õ(u−v, u−v)−iÕ(u−iv, u−iv).

These are called polarization identities.

Proof. (1) IfÕ is a Hermitian form, then

Õ(v, u) =Õ(u, v) implies that

Õ(u, u) =Õ(u, u),

and thus Õ(u, u)∈ R. IfÕ is sesquilinear andÕ(u, u)∈ R for allu∈E, then Õ(u +v, u +v) =Õ(u, u) +Õ(u, v) +Õ(v, u) +Õ(v, v),

which proves that

Õ(u, v) +Õ(v, u) =α,

where α is real, and changingu toiu, we have i(Õ(u, v)−Õ(v, u)) =β,

whereβ is real, and thus

Õ

(

u, v

) =

α

−

iβ

and

Õ

(

v, u

) =

α

+

iβ

,

2 2

proving thatÕ is Hermitian.

(2) These identities are verified by expanding the right-hand side, and we leave them as an exercise.

Proposition 11.1 shows that a sesquilinear form is completely determined by the quadratic form Φ(u) =Õ(u, u), even ifÕ is not Hermitian. This is false for a real bilinear form, unless it is symmetric. For example, the bilinear formÕ: R^2 R^2 R defined such that× →

Õ((x_1, y_1),(x_2, y_2)) =x_1y2−x_2y_1

is not identically zero, and yet it is null on the diagonal. However, a real symmetric bilinear form is indeed determined by its values on the diagonal, as we saw in Chapter 9. As in the Euclidean case, Hermitian forms for whichÕ(u, u)≥ 0 play an important role.

Definition 11.3. Given a complex vector spaceE, a Hermitian formÕ:E×E→ C is positive ifÕ(u, u)≥ 0 for allu∈E, and positive definite ifÕ(u, u)> 0 for allu = 0. A pair E, Õ whereE is a complex vector space andÕ is a Hermitian form onE is called a pre-Hilbert space ifÕ is positive, and a Hermitian (or unitary) space ifÕ is positive definite.

We warn our readers that some authors, such as Lang [67], define a pre-Hilbert space as what we define as a Hermitian space. We prefer following the terminology used in Schwartz [89] and Bourbaki [14]. The quantityÕ(u, v) is usually called the Hermitian product ofu andv. We will occasionally call it the inner product ofu andv.

Given a pre-Hilbert space E, Õ , as in the case of a Euclidean space, we also denote Õ(u, v) by

u·v or u, v or (u|v), and Φ(u) by u .

Example 11.1. The complex vector space Cn under the Hermitian form Õ((x_1, . . . , xn),(y_1, . . . , yn)) =x_1y_1 +x_2y_2 +· · · +xnyn

is a Hermitian space.

Example 11.2. Letl^2 denote the set of all countably infinite sequencesx = (xi)i∈N of complex numbers such that

∞

i

=0

|xi|

2 i=0 2 is defined (i.e., the sequencen xi| converges as n

→ ∞

). It can be shown that the map

Õ

:

l

2 l^2 C defined such that| × →

Õ((xi)i∈N,(yi)i∈N) = ∞ xiyi

i=0

is well defined, andl^2 is a Hermitian space underÕ. Actually,l^2 is even a Hilbert space. Example 11.3. LetCpiece[a, b] be the set of piecewise bounded continuous functions f : [a, b]→ C under the Hermitian form

b

f, g = f(x)g(x)dx.

a

It is easy to check that this Hermitian form is positive, but it is not definite. Thus, under this Hermitian form,Cpiece[a, b] is only a pre-Hilbert space.

Example 11.4. LetC[a, b] be the set of complex-valued continuous functionsf : [a, b]→ C under the Hermitian formb

f, g = f(x)g(x)dx.

a

It is easy to check that this Hermitian form is positive definite. Thus,C[a, b] is a Hermitian space.

Example 11.5. LetE = Mn(C) be the vector space of complexn×n matrices. If we view a matrixA∈ Mn(C) as a “long” column vector obtained by concatenating together its columns, we can define the Hermitian product of two matricesA, B∈ Mn(C) as

n

A, B = aijbij,

i,j=1

which can be conveniently written as

A, B = tr(A∗B) = tr(B∗A).

Since this can be viewed as the standard Hermitian product onCn2, it is a Hermitian product on Mn(C). The corresponding norm

A F = tr(A∗A)

is the Frobenius norm (see Section 7.2).

IfE is finite-dimensional and ifÕ:E×E→ R is a sequilinear form onE, given any basis (e_1, . . . , en) ofE, we can writex =n andy =n yjej, and we havei=1xiei j=1

n n n

Õ(x, y) =Õ xiei, yjej = xiyjÕ(ei, ej).

i=1 j=1 i,j=1

If we letG be the matrixG = (Õ(ei, ej)), and ifx andy are the column vectors associated with (x_1, . . . , xn) and (y_1, . . . , yn), then we can write

Õ(x, y) =x Gy =y∗G x,

wherey corresponds to (y_1, . . . , yn).

Observe that inÕ(x, y) =y∗G x, the matrix involved is the transpose ofG = (Õ(ei, ej)).

Furthermore, observe thatÕ is Hermitian iffG =G∗, andÕ is positive definite iff the matrixG is positive definite, that is,

x Gx > 0 for allx∈ Cn, x = 0. The matrixG associated with a Hermitian product is called the Gram matrix of the Hermitian product with respect to the basis (e_1, . . . , en).

Remark: To avoid the transposition in the expression forÕ(x, y) =y∗G x, some authors (such as Hoffman and Kunze [60]), define the Gram matrixG = (gij) associated with−,−so that (gij) = (Õ(ej, ei)); that is,G =G .

Conversely, ifA is a Hermitian positive definiten×n matrix, it is easy to check that the Hermitian form

x, y =y∗Ax

is positive definite. If we make a change of basis from the basis (e_1, . . . , en) to the basis (f_1, . . . , fn), and if the change of basis matrix isP (where thejth column ofP consists of the coordinates offj over the basis (e_1, . . . , en)), then with respect to coordinatesx andy over the basis (f_1, . . . , fn), we have

x Gy =x P GP y ,

so the matrix of our inner product over the basis (f_1, . . . , fn) isP GP = (P)∗GP. We summarize these facts in the following proposition.

Proposition 11.2. LetE be a finite-dimensional vector space, and let (e_1, . . . , en) be a basis ofE.

1. For any Hermitian inner product−,− onE, ifG = (ei, ej ) is the Gram matrix of the Hermitian product−,− w.r.t. the basis (e_1, . . . , en), thenG is Hermitian positive definite.

2. For any change of basis matrixP, the Gram matrix of−,− with respect to the new basis is (P)∗GP.

3. IfA is anyn×n Hermitian positive definite matrix, then

x, y =y∗Ax

is a Hermitian product onE.

We will see later that a Hermitian matrix is positive definite iff its eigenvalues are all positive.

The following result reminiscent of the first polarization identity of Proposition 11.1 can be used to prove that two linear maps are identical.

Proposition 11.3. Given any Hermitian spaceE with Hermitian product−,−, for any linear mapf :E→E, if f(x), x = 0 for allx∈E, thenf = 0.

Proof. Compute f(x +y), x +y and f(x−y), x−y :

f(x +y), x +y = f(x), x + f(x), y + f(y), x + y, y f(x−y), x−y = f(x), x− f(x), y− f(y), x + y, y ;

then, subtract the second equation from the first, to obtain

f(x +y), x +y− f(x−y), x−y = 2(f(x), y + f(y), x ).

If f(u), u = 0 for allu∈E, we get

f(x), y + f(y), x = 0 for allx, y∈E.

Then, the above equation also holds if we replacex byix, and we obtain

i f(x), y−i f(y), x = 0, for allx, y∈E,

so we have

f(x), y + f(y), x = 0 f(x), y− f(y), x = 0,

which implies that f(x), y = 0 for allx, y∈E. Since−,− is positive definite, we have f(x) = 0 for allx∈E; that is,f = 0.

One should be careful not to apply Proposition 11.3 to a linear map on a real Euclidean space, because it is false! The reader should find a counterexample.

The Cauchy–Schwarz inequality and the Minkowski inequalities extend to pre-Hilbert spaces and to Hermitian spaces.

Proposition 11.4. Let E, Õ be a pre-Hilbert space with associated quadratic form Φ. For allu, v∈E, we have the Cauchy–Schwarz inequality

|Õ(u, v)| ≤ Φ(u) Φ(v).

Furthermore, if E, Õ is a Hermitian space, the equality holds iffu andv are linearly dependent.

We also have the Minkowski inequality

Φ(u +v)≤ Φ(u) + Φ(v).

Furthermore, if E, Õ is a Hermitian space, the equality holds iffu andv are linearly dependent, where in addition, ifu = 0 andv = 0, thenu =λv for some realλ such that λ > 0.

Proof. For allu, v∈E and allµ∈ C, we have observed that

Õ

(

u

+

µv, u

+

µv

) =

Õ

(

u, u

) + 2

(

µÕ

(

u, v

)) +

| µ

2

| Õ(v, v).

LetÕ(u, v) =ρeiθ, where|Õ(u, v)| =ρ (ρ≥ 0). LetF : R→ R be the function defined such that

F(t) = Φ(u +teiθv), for allt∈ R. The above shows that

F(t) =Õ(u, u) + 2t|Õ(u, v)| +t^2Õ(v, v) = Φ(u) + 2t|Õ(u, v)| +t^2Φ(v).

Since Õ is assumed to be positive, we haveF(t)≥ 0 for allt∈ R. If Φ(v) = 0, we must have Õ(u, v) = 0, since otherwise,F(t) could be made negative by choosingt negative and small enough. If Φ(v)> 0, in order forF(t) to be nonnegative, the equation

Φ(u) + 2t|Õ(u, v)| +t^2Φ(v) = 0

must not have distinct real roots, which is equivalent to

|

Õ

(

u, v

)

2

| ≤ Φ(u)Φ(v).

Taking the square root on both sides yields the Cauchy–Schwarz inequality. For the second part of the claim, ifÕ is positive definite, we argue as follows. Ifu andv are linearly dependent, it is immediately verified that we get an equality. Conversely, if

Õ

(

u, v

)

2

| = Φ(u)Φ(v),|

then the equation

Φ(u) + 2t|Õ(u, v)| +t^2Φ(v) = 0 has a double roott_0, and thus

Φ(u +t_0eiθv) = 0. SinceÕ is positive definite, we must have

u +t_0eiθv = 0,

which shows thatu andv are linearly dependent.

If we square the Minkowski inequality, we get

Φ(u +v)≤ Φ(u) + Φ(v) + 2 Φ(u) Φ(v).

However, we observed earlier that

Φ(u +v) = Φ(u) + Φ(v) + 2 (Õ(u, v)). Thus, it is enough to prove that

( Õ(u, v))≤ Φ(u) Φ(v), but this follows from the Cauchy–Schwarz inequality

|Õ(u, v)| ≤ Φ(u) Φ(v)

and the fact that z≤ |z|.

IfÕ is positive definite andu andv are linearly dependent, it is immediately verified that we get an equality. Conversely, if equality holds in the Minkowski inequality, we must have (Õ(u, v)) = Φ(u) Φ(v), which implies that |Õ(u, v)| = Φ(u) Φ(v),

since otherwise, by the Cauchy–Schwarz inequality, we would have (Õ(u, v))≤ |Õ(u, v)|< Φ(u) Φ(v).

Thus, equality holds in the Cauchy–Schwarz inequality, and (Õ(u, v)) =|Õ(u, v)|.

But then, we proved in the Cauchy–Schwarz case that u andv are linearly dependent. Since we also just proved thatÕ(u, v) is real and nonnegative, the coefficient of proportionality betweenu andv is indeed nonnegative.

As in the Euclidean case, if E, Õ is a Hermitian space, the Minkowski inequality

Φ(u +v)≤ Φ(u) + Φ(v)

shows that the map u→ Φ(u) is a norm onE. The norm induced byÕ is called the Hermitian norm induced byÕ. We usually denote Φ(u) by u , and the Cauchy–Schwarz inequality is written as

|u·v| ≤ u v .

Since a Hermitian space is a normed vector space, it is a topological space under the topology induced by the norm (a basis for this topology is given by the open ballsB_0(u, ρ) of centeru and radiusρ > 0, where

B_0(u, ρ) ={v∈E| v−u < ρ}.

IfE has finite dimension, every linear map is continuous; see Chapter 7 (or Lang [67, 68], Dixmier [27], or Schwartz [89, 90]). The Cauchy–Schwarz inequality

u·v| ≤ u v|

shows thatÕ:E×E→ C is continuous, and thus, that is continuous. If E, Õ is only pre-Hilbertian, u is called a seminorm. In this case, the condition u = 0 implies u = 0

is not necessarily true. However, the Cauchy–Schwarz inequality shows that if u = 0, then u·v = 0 for allv∈E.

Remark: As in the case of real vector spaces, a norm on a complex vector space is induced by some psotive definite Hermitian product−,− iff it satisfies the parallelogram law: u +v^2 + u−v^2 = 2( u^2 + v^2).

This time, the Hermitian product is recovered using the polarization identity from Proposition 11.1:

4u, v = u +v^2 u−v^2 +i u +iv^2 i u−iv^2.− −It is easy to check that u, u = u 2, and

v, u = u, v

iu, v =i u, v ,

so it is enough to check linearity in the variable u, and only for real scalars. This is easily done by applying the proof from Section 9.1 to the real and imaginary part of u, v ; the details are left as an exercise.

We will now basically mirror the presentation of Euclidean geometry given in Chapter 9 rather quickly, leaving out most proofs, except when they need to be seriously amended.

11.2 Orthogonality, Duality, Adjoint of a Linear Map

In this section we assume that we are dealing with Hermitian spaces. We denote the Hermitian inner product byu·v or u, v . The concepts of orthogonality, orthogonal family of vectors, orthonormal family of vectors, and orthogonal complement of a set of vectors are unchanged from the Euclidean case (Definition 9.2).

For example, the setC[−π, π] of continuous functionsf : [−π, π]→ C is a Hermitian space under the productπ

f, g = f(x)g(x)dx,

−π

and the family (eikx)k∈Z is orthogonal.

Proposition 9.3 and 9.4 hold without any changes. It is easy to show that

n^2 n

ui = ui 2 + 2 (ui·uj).

i=1 i=1 1≤i<j≤n

Analogously to the case of Euclidean spaces of finite dimension, the Hermitian product induces a canonical bijection (i.e., independent of the choice of bases) between the vector spaceE and the spaceE∗. This is one of the places where conjugation shows up, but in this case, troubles are minor.

Given a Hermitian spaceE, for any vectoru∈E, letÕlu:E→ C be the map defined such that

Õlu(v) =u·v, for allv∈E.

Similarly, for any vectorv∈E, letÕrv:E→ C be the map defined such that Õrv(u) =u·v, for allu∈E.

Since the Hermitian product is linear in its first argument u, the mapÕrv is a linear form inE∗, and since it is semilinear in its second argumentv, the mapÕlu is also a linear form inE∗. Thus, we have two maps l:E→E∗ and r:E→E∗, defined such that

l(u) =Õlu, andr(v) =Õrv.

Actually,Õlu =Õru and l = r. Indeed, for allu, v∈E, we have

l (u)(v) =Õlu(v) =u·v =v·u =Õru(v) = r(u)(v).

Therefore, we use the notationÕu for bothÕlu andÕru, and Theorem 11.5. letE be a Hermitian spaceE. The map for both l and r.

: E→E∗ defined such that (u) =Õlu =Õru for allu∈E

is semilinear and injective. WhenE is also of finite dimension, the map :E→E∗ is a canonical isomorphism.

Proof. That :E→E∗ is a semilinear map follows immediately from the fact that = r, and that the Hermitian product is semilinear in its second argument. IfÕu =Õv, then Õu(w) =Õv(w) for allw∈E, which by definition ofÕu andÕv means that

w·u =w·v

for allw∈E, which by semilinearity on the right is equivalent to

w· (v−u) = 0 for allw∈E,

which implies thatu =v, since the Hermitian product is positive definite. Thus, :E→E∗ is injective. Finally, whenE is of finite dimensionn,E∗ is also of dimensionn, and then

: E→E∗ is bijective. Since is semilinar, the map :E→E∗ is an isomorphism. The inverse of the isomorphism :E→E∗ is denoted by :E∗→E.

As a corollary of the isomorphism :E→E∗, ifE is a Hermitian space of finite dimension, then every linear formf∈E∗ corresponds to a uniquev∈E, such that

f(u) =u·v, for everyu∈E.

In particular, iff is not the null form, the kernel off, which is a hyperplaneH, is precisely the set of vectors that are orthogonal tov.

Remark: The “musical map” :E→E∗ is not surjective whenE has infinite dimension. This result can be salvaged by restricting our attention to continuous linear maps, and by assuming that the vector spaceE is a Hilbert space.

The existence of the isomorphism :E→E∗ is crucial to the existence of adjoint maps. Indeed, Theorem 11.5 allows us to define the adjoint of a linear map on a Hermitian space. LetE be a Hermitian space of finite dimensionn, and letf :E→E be a linear map. For everyu∈E, the map

v→u·f(v)

is clearly a linear form inE∗, and by Theorem 11.5, there is a unique vector inE denoted byf∗(u), such that

f∗(u)·v =u·f(v), that is,

f∗(u)·v =u·f(v), for everyv∈E. The following proposition shows that the mapf∗ is linear. Proposition 11.6. Given a Hermitian spaceE of finite dimension, for every linear map f :E→E there is a unique linear mapf∗:E→E such that

f∗(u)·v =u·f(v),

for allu, v∈E. The mapf∗ is called the adjoint off (w.r.t. to the Hermitian product).

Proof. Careful inspection of the proof of Proposition 9.6 reveals that it applies unchanged. The only potential problem is in proving thatf∗(λu) =λf∗(u), but everything takes place in the first argument of the Hermitian product, and there, we have linearity.

The fact that v·u =u·v implies that the adjointf∗ off is also characterized by

f(u)·v =u·f∗(v), for allu, v∈E. It is also obvious thatf∗∗ =f.

Given two Hermitian spaces E andF, where the Hermitian product onE is denoted by−,−1 and the Hermitian product onF is denoted by−,−2, given any linear map f :E→F, it is immediately verified that the proof of Proposition 11.6 can be adapted to show that there is a unique linear mapf∗:F→E such that

f(u), v 2 = u, f∗(v) 1

for allu∈E and allv∈F. The linear mapf∗ is also called the adjoint off. As in the Euclidean case, a linear mapf :E→ E (whereE is a finite-dimensional Hermitian space) is seff-adjoint iff =f∗. The mapf is positive semidefinite iff

f(x), x≥ 0 allx∈E;

positive definite iff f(x), x > 0 allx∈E, x = 0.

An interesting corollary of Proposition 11.3 is that a positive semidefinite linear map must be self-adjoint. In fact, we can prove a slightly more general result.

Proposition 11.7. Given any finite-dimensional Hermitian spaceE with Hermitian product −

,−, for any linear mapf :E→E, if f(x), x∈ R for allx∈E, thenf is self-adjoint. In particular, any positive semidefinite linear mapf :E→E is self-adjoint.

Proof. Since f(x), x∈ R for allx∈E, we have

f (x), x = f(x), x = x, f(x) = f∗(x), x ,

so we have (f−f∗)(x), x = 0 allx∈E,

and Proposition 11.3 implies that f−f∗ = 0.

Beware that Proposition 11.7 is false ifE is a real Euclidean space.

As in the Euclidean case, Theorem 11.5 can be used to show that any Hermitian space of finite dimension has an orthonormal basis. The proof is unchanged.

Proposition 11.8. Given any nontrivial Hermitian spaceE of finite dimensionn≥ 1, there is an orthonormal basis (u_1, . . . , un) forE.

The Gram–Schmidt orthonormalization procedure also applies to Hermitian spaces of finite dimension, without any changes from the Euclidean case!

Proposition 11.9. Given a nontrivial Hermitian spaceE of finite dimensionn≥ 1, from any basis (e_1, . . . , en) forE we can construct an orthonormal basis (u_1, . . . , un) forE with the property that for everyk, 1≤k≤n, the families (e_1, . . . , ek) and (u_1, . . . , uk) generate the same subspace.

Remark: The remarks made after Proposition 9.8 also apply here, except that in theQR- decomposition,Q is a unitary matrix.

As a consequence of Proposition 9.7 (or Proposition 11.9), given any Hermitian space of finite dimensionn, if (e_1, . . . , en) is an orthonormal basis forE, then for any two vectors u =u_1e_1 +· · · +unen andv =v_1e_1 +· · · +vnen, the Hermitian productu·v is expressed as

n

u·v = (u_1e_1 +· · · +unen)· (v_1e_1 +· · · +vnen) = uivi,

i=1

and the norm u as

n 1/2

u

=

u

1

e

1

+

· · ·

+

u

n

e

n

=

| u

i 2

| .

i=1

The fact that a Hermitian space always has an orthonormal basis implies that any Gram matrixG can be written as

G =Q∗Q,

for some invertible matrix Q. Indeed, we know that in a change of basis matrix, a Gram matrixG becomesG = (P)∗GP. If the basis corresponding toG is orthonormal, then G =I, soG = (P−1)∗P−1.

Proposition 9.9 also holds unchanged.

Proposition 11.10. Given any nontrivial Hermitian spaceE of finite dimensionn≥ 1, for any subspaceF of dimensionk, the orthogonal complementF⊥ ofF has dimension n−k, andE =F⊕F⊥. Furthermore, we haveF⊥⊥ =F.

11.3 Linear Isometries (Also Called Unitary Transformations)

In this section we consider linear maps between Hermitian spaces that preserve the Hermitian norm. All definitions given for Euclidean spaces in Section 9.3 extend to Hermitian spaces, except that orthogonal transformations are called unitary transformation, but Proposition 9.10 extends only with a modified condition (2). Indeed, the old proof that (2) implies (3) does not work, and the implication is in fact false! It can be repaired by strengthening condition (2). For the sake of completeness, we state the Hermitian version of Definition 9.3. 11.3. LINEAR ISOMETRIES (ALSO CALLED UNITARY TRANSFORMATIONS) 305

Definition 11.4. Given any two nontrivial Hermitian spacesE andF of the same finite dimensionn, a functionf :E→F is a unitary transformation, or a linear isometry, if it is linear and

f(u) = u , for allu∈E.

Proposition 9.10 can be salvaged by strengthening condition (2).

Proposition 11.11. Given any two nontrivial Hermitian spacesE andF of the same finite dimensionn, for every functionf :E→F, the following properties are equivalent:

(1) f is a linear map and f(u) = u , for allu∈E;

(2) f(v)−f(u) = v−u andf(iu) =if(u), for allu, v∈E.

(3) f(u)·f(v) =u·v, for allu, v∈E.

Furthermore, such a map is bijective.

Proof. The proof that (2) implies (3) given in Proposition 9.10 needs to be revised as follows. We use the polarization identity

2Õ(u, v) = (1 +i)( u^2 + v^2)− u−v^2 i u−iv^2.−

Sincef(iv) =if(v), we getf(0) = 0 by settingv = 0, so the functionf preserves distance and norm, and we get

2 Õ(f(u), f(v)) = (1 +i)( f(u)2 + f(v)2)− f(u)−f(v)2

−i f(u)−if(v)2 = (1 +i)( f(u)2 + f(v)2)− f(u)−f(v)2

−i f(u)−f(iv)2

= (1 +

i

)(

u

2 + v^2)− u−v^2 i u−iv2 −= 2Õ(u, v),

which shows thatf preserves the Hermitian inner product, as desired. The rest of the proof is unchanged.

Remarks: (i) In the Euclidean case, we proved that the assumption f(v)−f(u) = v−u for allu, v∈E andf(0) = 0 (2 ) implies (3). For this we used the polarization identity

2 u^2 + v^2 u−v^2.·v = u− In the Hermitian case the polarization identity involves the complex numberi. In fact, the implication (2 ) implies (3) is false in the Hermitian case! Conjugationz→z satisfies (2 ) since

|z2−z1| =|z2−z1| =|z2−z1|, and yet, it is not linear!

(ii) If we modify (2) by changing the second condition by now requiring that there be some τ∈E such that

f(τ +iu) =f(τ) +i(f(τ +u)−f(τ)) for allu∈E, then the functiong:E→E defined such that

g(u) =f(τ +u)−f(τ)

satisfies the old conditions of (2), and the implications (2)→ (3) and (3)→ (1) prove thatg is linear, and thus thatf is affine. In view of the first remark, some condition involvingi is needed onf, in addition to the fact thatf is distance-preserving.

11.4 The Unitary Group, Unitary Matrices

In this section, as a mirror image of our treatment of the isometries of a Euclidean space, we explore some of the fundamental properties of the unitary group and of unitary matrices. As an immediate corollary of the Gram–Schmidt orthonormalization procedure, we obtain theQR-decomposition for invertible matrices. In the Hermitian framework, the matrix of the adjoint of a linear map is not given by the transpose of the original matrix, but by its conjugate.

Definition 11.5. Given a complexm×n matrixA, the transposeA ofA is then×m matrixA = ai j defined such that

ai j =aj i,

and the conjugateA ofA is them×n matrixA = (bi j) defined such that

bi j =ai j

for alli, j, 1≤i≤m, 1≤j≤n. The adjointA∗ ofA is the matrix defined such that

A∗ = (A ) = A .

Proposition 11.12. LetE be any Hermitian space of finite dimensionn, and letf :E→E be any linear map. The following properties hold:

11.4. THE UNITARY GROUP, UNITARY MATRICES 307

(1) The linear mapf :E→E is an isometry iff fæf∗ =f∗æf = id.

(2) For every orthonormal basis (e_1, . . . , en) ofE, if the matrix off isA, then the matrix off∗ is the adjointA∗ ofA, andf is an isometry iffA satisfies the identities

A A∗ =A∗A =In, whereIn denotes the identity matrix of ordern, iff the columns ofA form an orthonormal basis ofE, iff the rows ofA form an orthonormal basis ofE.

Proof. (1) The proof is identical to that of Proposition 9.12 (1).

(2) If (e_1, . . . , en) is an orthonormal basis forE, letA = (ai j) be the matrix off, and let B = (bi j) be the matrix off∗. Sincef∗ is characterized by

f∗(u)·v =u·f(v) for allu, v∈E, using the fact that ifw =w_1e_1 +· · · +wnen, we havewk =w·ek, for allk, 1≤k≤n; lettingu =ei andv =ej, we get

bj i =f∗(ei)·ej =ei·f(ej) =f(ej)·ei =ai j,

for all i, j, 1≤i, j≤n. Thus,B =A∗. Now, ifX andY are arbitrary matrices over the basis (e_1, . . . , en), denoting as usual thejth column ofX byXj, and similarly forY , a simple calculation shows that

Y∗X = (Xj Yi)1≤i,j≤n.·

Then it is immediately verified that ifX =Y =A, thenA∗A =A A∗ =In iff the column vectors (A^1, . . . , An) form an orthonormal basis. Thus, from (1), we see that (2) is clear. Proposition 9.12 shows that the inverse of an isometryf is its adjointf∗. Proposition 9.12 also motivates the following definition.

Definition 11.6. A complexn×n matrix is a unitary matrix if

A A∗ =A∗A =In.

Remarks:

(1) The conditions A A∗ =In,A∗A =In, andA−1 =A∗ are equivalent. Given any two orthonormal bases (u_1, . . . , un) and (v_1, . . . , vn), ifP is the change of basis matrix from (u_1, . . . , un) to (v_1, . . . , vn), it is easy to show that the matrixP is unitary. The proof of Proposition 11.11 (3) also shows that iff is an isometry, then the image of an orthonormal basis (u_1, . . . , un) is an orthonormal basis.

(2) Using the explicit formula for the determinant, we see immediately that

det(A) = det(A).

Iff is unitary andA is its matrix with respect to any orthonormal basis, fromAA∗ =I, we get

det(

AA

∗

) = det(

A

) det(

A

∗

) = det(

A

)det(

A

) = det(

A

)det(

A

) = | det( A )

2

| , and so|det(A)| = 1. It is clear that the isometries of a Hermitian space of dimension n form a group, and that the isometries of determinant +1 form a subgroup. This leads to the following definition.

Definition 11.7. Given a Hermitian spaceE of dimensionn, the set of isometriesf :E→ E forms a subgroup of GL(E,C) denoted by U(E), or U(n) whenE = Cn, called the unitary group (ofE). For every isometryf we have|det(f)| = 1, where det(f) denotes the determinant off. The isometries such that det(f) = 1 are called rotations, or proper isometries, or proper unitary transformations, and they form a subgroup of the special linear group SL(E,C) (and of U(E)), denoted by SU(E), or SU(n) whenE = Cn, called the special unitary group (ofE). The isometries such that det(f) = 1 are called improper isometries, or improper unitary transformations, or flip transformations.

A very important example of unitary matrices is provided by Fourier matrices (up to a factor of√n), matrices that arise in the various versions of the discrete Fourier transform. For more on this topic, see the problems, and Strang [100, 102].

Now that we have the definition of a unitary matrix, we can explain how the Gram– Schmidt orthonormalization procedure immediately yields theQR-decomposition for matrices.

Proposition 11.13. Given anyn×n complex matrixA, ifA is invertible, then there is a unitary matrixQ and an upper triangular matrix R with positive diagonal entries such that A =QR.

The proof is absolutely the same as in the real case!

Due to space limitations, we will not study the isometries of a Hermitian space in this chapter. However, the reader will find such a study in the supplements on the web site (see http://www.cis.upenn.edu/ jean/gbooks/geom2.html).

11.5 Orthogonal Projections and Involutions

In this section, we assume that the field K is not a field of characteristic 2. Recall that a linear mapf :E→E is an involution ifff^2 = id, and is idempotent ifff^2 =f. We know from Proposition 4.7 that iff is idempotent, then

E = Im(f)⊕ Ker (f),

11.5. ORTHOGONAL PROJECTIONS AND INVOLUTIONS 309

and that the restriction of f to its image is the identity. For this reason, a linear involution is called a projection. The connection between involutions and projections is given by the following simple proposition.

Proposition 11.14. For any linear mapf :E→E, we havef^2 = id iff^1(id−f) is a projection iff

1

2

(id +f) is a projection; in this case, f is equal to the difference of the two projections 1 2 1(id−f).2(id +f) and_2

Proof. We have 1 2 = 14(id− 2f +f^2)2(id−f)

so

1 2 = 12(id−f) iff f^2 = id.2(id−f)

We also have

1 2 = 14(id + 2f +f^2),2(id +f)

so

1 2 = 12(id +f) iff f^2 = id.2(id +f)

Oviously,

f

=

1(id +f)1

− 2(id−f).2

LetU+ = Ker (1(id−f)) and letU− = Im(1(id−f)). Iff^2 = id, then2 2

(id +f)æ (id−f) = id−f^2 = id− id = 0,

which implies that

1 1

Im 2(id +f) ⊆ Ker 2(id−f) .

Conversely, ifu∈ Ker^1(id−f) , thenf(u) =u, so_2

1 f)(u) = 12(u +u) =u,2(id +

and thus1 1

Ker 2(id−f) ⊆ Im 2(id +f) .

Therefore,

1 1

U+ = Ker 2(id−f) = Im 2(id +f) ,

and so, f(u) =u onU+ andf(u) =−u onU−. The involutions ofE that are unitary transformations are characterized as follows.

Proposition 11.15. Letf∈ GL(E) be an involution. The following properties are equivalent:

(a) The mapf is unitary; that is,f∈ U(E).

(b) The subspaces^1(id−f)) andU+ = Im(1(id +f)) are orthogonal.U− = Im(2 2

Furthermore, ifE is finite-dimensional, then (a) and (b) are equivalent to (c) The map is self-adjoint; that is,f =f∗.

Proof. Iff is unitary, then from f(u), f(v) = u, v for allu, v∈E, we see that ifu∈U+ andv^1, we get∈U−

u, v = f(u), f(v) = u,−v =−u, v ,

so 2u, v = 0, which implies u, v = 0, that is,U+ andU− are orthogonal. Thus, (a) implies (b).

Conversely, if (b) holds, sincef(u) = u onU+ andf(u) =−u onU−, we see that f(u), f(v) = u, v ifu, v∈U+ or ifu, v∈ U−. SinceE =U+ and sinceU+ andU−⊕ U−

are orthogonal, we also have f(u), f(v) = u, v for allu, v∈E, and (b) implies (a). IfE is finite-dimensional, the adjointf∗ off exists, and we know thatf−1 =f∗. Since f is an involution,f^2 = id, which implies thatf∗ =f−1 =f.

A unitary involution is the identity onU+ = Im(1(id +f)), andf(v) =− v for all v

∈

U

−

= Im(

1

2

(id−f)). Furthermore,E is an orthogonal direct sumE =U+ U−1. We2 +. In the special case whereU+ ⊕ say thatf is an orthogonal reflection aboutU is a hyperplane, we say thatf is a hyperplane reflection. We already studied hyperplane reflections in the Euclidean case; see Chapter 10.

If f :E→E is a projection (f^2 =f), then

(id^2 = id− 4f + 4f^2 = id− 4f + 4f = id,− 2f)

so id− 2f is an involution. As a consequence, we get the following result.

Proposition 11.16. Iff :E→E is a projection (f^2 =f), then Ker (f) and Im(f) are orthogonal ifff∗ =f.

Proof. Apply Proposition 11.15 tog = id− 2f. Since id−g = 2f we have

1

U+ = Ker 2(id−g) = Ker (f)

and_1

U− = Im 2(id−g) = Im(f),

which proves the proposition. A projection such thatf =f∗ is called an orthogonal projection.

If ( a_1. . . , ak) arek linearly independent vectors in Rn, let us determine the matrixP of the orthogonal projection onto the subspace of Rn spanned by (a_1, . . . , ak). LetA be the n×k matrix whosejth column consists of the coordinates of the vectoraj over the canonical basis (e_1, . . . , en). Any vector in the subspace (a_1, . . . , ak) is a linear combination of the form Ax, for somex∈ Rk. Given anyy∈ Rn, the orthogonal projectionP y =Ax ofy onto the subspace spanned by (a_1, . . . , ak) is the vectorAx such thaty−Ax is orthogonal to the subspace spanned by (a_1, . . . , ak) (prove it). This means thaty−Ax is orthogonal to every aj, which is expressed by

A (y−Ax) = 0; that is,

A Ax =A y.

The matrixA A is invertible becauseA has full rankk, thus we get

x = (A A)−1A y,

and so

P y =Ax =A(A A)−1A y.

Therefore, the matrixP of the projection onto the subspace spanned by (a_1. . . , ak) is given by

P =A(A A)−1A . The reader should check thatP^2 =P andP =P.

11.6 Dual Norms

In the remark following the proof of Proposition 7.7, we explained that if ( E, ) and (F, ) are two normed vector spaces and if we letL(E;F) denote the set of all continuous (equivalently, bounded) linear maps fromE toF , then, we can define the operator norm (or subordinate norm) onL(E;F) as follows: for everyf∈ L(E;F),

f = supf(x) = sup f(x) .

x∈E x x∈E

x =0 x =1

In particular, ifF = C, thenL(E;F) =E is the dual space ofE, and we get the operator norm denoted by given by

∗ f = sup|f(x)|.

∗ x∈E

x =1

The norm is called the dual norm of onE .

∗

Let us now assume that E is a finite-dimensional Hermitian space, in which caseE =E∗. Theorem 11.5 implies that for every linear formf∈E∗, there is a unique vectory∈E so that

f(x) = x, y , for allx∈E, and so we can write

f = sup|x, y|.

∗ x∈E

x =1

The above suggests defining a normD onE.

Definition 11.8. IfE is a finite-dimensional Hermitian space and is any norm onE, for anyy∈E we let

y D = sup|x, y|,

x∈E

x =1

be the dual norm of (onE). IfE is a real Euclidean space, then the dual norm is defined by

y D = sup x, y x∈E

x =1

for ally∈E.

Beware that is generally not the Hermitian norm associated with the Hermitian innner product. The dual norm shows up in convex programming; see Boyd and Vandenberghe [15], Chapters 2, 3, 6, 9.

The fact thatD is a norm follows from the fact that is a norm and can also be checked directly. It is worth noting that the triangle inequality for∗ D comes “for free,” in the sense that it holds for any functionp:E→ R. Indeed, we have

pD(x +y) = sup|z, x +y|p(z)=1

= sup (|z, x + z, y|)

p(z)=1

sup (|z, x| +|z, y|)≤ p(z)=1

≤

p(z)=1 p(z)=1

sup|z, x| + sup|z, y|

=pD(x) +pD(y).

If p:E→ R is a function such that

(1) p(x)≥ 0 for allx∈E, andp(x) = 0 iffx = 0; (2) p(λx) =|λ|p(x), for allx∈E and allλ∈ C; (3) p is continuous, in the sense that for some basis (e_1, . . . , en) ofE, the function

(x_1, . . . , xn)→p(x_1e_1 +· · · +xnen)

from Cn to R is continuous;

then we say that p is a pre-norm. Obviously, every norm is a pre-norm, but a pre-norm may not satisfy the triangle inequality. However, we just showed that the dual norm of any pre-norm is actually a norm.

SinceE is finite dimensional, the unit sphereSn−1 ={x∈E| x = 1} is compact, so there is somex0∈Sn−1 such that

y D =|x_0, y|.

If x_0, y =ρeiθ, withρ≥ 0, then

|e−iθx_0, y| =|e−iθx_0, y| =|e−iθρeiθ =ρ,|

so

y D =ρ =|e−iθx_0, y|, with e−iθx_0 = x_0 = 1. On the other hand,

x, y≤ |x, y|,

so we get y D = sup|x, y| = sup x, y .

x∈E x∈E

x =1 x =1

Proposition 11.17. For allx, y∈E, we have

|

x, y| ≤ x y D D y .|x, y| ≤ x

Proof. Ifx = 0, then x, y = 0 and these inequalities are trivial. Ifx = 0, since x/ x = 1, by definition of y D, we have

|

x/ x , y| ≤ sup|z, y| = y D,

z =1

which yields |x, y| ≤ x y D. The second inequality holds because|x, y| =|y, x|. It is not hard to show that

y D = y1 ∞y D = y 1∞

y D = y_2.2

Thus, the Euclidean norm is autodual. More generally, ifp, q≥ 1 and 1/p + 1/q = 1, we have

y D = y q.p

It can also be shown that the dual of the spectral norm is the trace norm (or nuclear norm) from Section 16.3. We close this section by stating the following duality theorem. Theorem 11.18. IfE is a finite-dimensional Hermitian space, then for any norm on E, we have

y DD = y for ally∈E.

Proof. By Proposition 11.17, we have

|x, y| ≤ x D y ,

so we get y DD = sup|x, y| ≤ y , for ally∈E.

x D=1

It remains to prove that y ≤ y DD, for ally∈E.

Proofs of this fact can be found in Horn and Johnson [55] (Section 5.5), and in Serre [92] (Chapter 7). The proof makes use of the fact that a nonempty, closed, convex set has a supporting hyperplane through each of its boundary points, a result known as Minkowski’s lemma. This result is a consequence of the Hahn–Banach theorem; see Gallier [42]. We give the proof in the case whereE is a real Euclidean space. Some minor modifications have to be made when dealing with complex vector spaces and are left as an exercise.

Since the unit ballB ={z∈E| z ≤ 1} is closed and convex, the Minkowski lemma says for everyx such that x = 1, there is an affine mapg, of the form

g(z) = z, w− x, w

with w = 1, such thatg(x) = 0 andg(z)≤ 0 for allz such that z ≤ 1. Then, it is clear that

sup z, w = x, w , z =1

11.7. SUMMARY 315

and so w D = x, w . It follows that

x DD x, w = 1 = x≥ w/ w D, x =w D

for all x such that x = 1. By homogeneity, this is true for ally∈E, which completes the proof in the real case. WhenE is a complex vector space, we have to view the unit ballB as a closed convex set in R2n and we use the fact that there is real affine map of the form

g(z) = z, w− x, w

such thatg(x) = 0 andg(z)≤ 0 for allz with z = 1, so that w D = x, w . More details on dual norms and unitarily invariant norms can be found in Horn and Johnson [55] (Chapters 5 and 7).

11.7 Summary

The main concepts and results of this chapter are listed below:

Semilinear maps.•

• Sesquilinear forms; Hermitian forms.

Quadratic form associated with a sesquilinear form.•

• Polarization identities.

Positive and positive definite Hermitian forms; pre-Hilbert spaces, Hermitian spaces.•

• Gram matrix associated with a Hermitian product.

The Cauchy–Schwarz inequality and the Minkowski inequality.•

• Hermitian inner product, Hermitian norm.

The parallelogram law.•

•

The musical isomorphisms :E→E∗ and :E∗→E; Theorem 11.5 (E is finitedimensional).

The adjoint of a linear map (with respect to a Hermitian inner product).•

• Existence of orthonormal bases in a Hermitian space (Proposition 11.8). Gram–Schmidt orthonormalization procedure.•

• Linear isometries (unitary transformatios).

• The unitary group, unitary matrices.

• The unitary group U(n);

• The special unitary group SU(n).

• QR-Decomposition for invertible matrices.

• Orthogonal projections and involutions; orthogonal reflections.

• Dual norms.

Chapter 12 Eigenvectors and Eigenvalues

12.1 Eigenvectors and Eigenvalues of a Linear Map

Given a finite-dimensional vector spaceE, letf :E→E be any linear map. If, by luck, there is a basis (e_1, . . . , en) ofE with respect to whichf is represented by a diagonal matrix ëλ_1 0 . . . 0ö

ì 0 λ_2 ... .÷

D

=

ì ÷

ì ÷,

í . ... ... 0 ø

0 . . . 0 λn

then the action of f onE is very simple; in every “direction”ei, we have

f(ei) =λiei.

We can think of f as a transformation that stretches or shrinks space along the direction e_1, . . . , en (at least ifE is a real vector space). In terms of matrices, the above property translates into the fact that there is an invertible matrixP and a diagonal matrixD such that a matrixA can be factored as

A =P DP−1.

When this happens, we say that f (orA) is diagonalizable, theλis are called the eigenvalues off, and theeis are eigenvectors off. For example, we will see that every symmetric matrix can be diagonalized. Unfortunately, not every matrix can be diagonalized. For example, the matrix

A

1

=

1 1

0 1

can’t be diagonalized. Sometimes, a matrix fails to be diagonalizable because its eigenvalues do not belong to the field of coefficients, such as

1

A^0 − ,2 =1 0

317 whose eigenvalues are±i. This is not a serious problem becauseA_2 can be diagonalized over the complex numbers. However,A_1 is a “fatal” case! Indeed, its eigenvalues are both 1 and the problem is thatA_1 does not have enough eigenvectors to spanE.

The next best thing is that there is a basis with respect to which f is represented by an upper triangular matrix. In this case we say thatf can be triangularized. As we will see in Section 12.2, if all the eigenvalues off belong to the field of coefficientsK, thenf can be triangularized. In particular, this is the case ifK = C.

Now, an alternative to triangularization is to consider the representation of f with respect to two bases (e_1, . . . , en) and (f_1, . . . , fn), rather than a single basis. In this case, ifK = R orK = C, it turns out that we can even pick these bases to be orthonormal, and we get a diagonal matrix Σ with nonnegative entries, such that

f(ei) =σifi, 1≤i≤n.

The nonzero σis are the singular values off, and the corresponding representation is the singular value decomposition, or SVD. The SVD plays a very important role in applications, and will be considered in detail later.

In this section, we focus on the possibility of diagonalizing a linear map, and we introduce the relevant concepts to do so. Given a vector spaceE over a fieldK, letI denote the identity map onE.

Definition 12.1. Given any vector spaceE and any linear mapf :E→E, a scalarλ∈K is called an eigenvalue, or proper value, or characteristic value off if there is some nonzero vectoru∈E such that

f(u) =λu.

Equivalently, λ is an eigenvalue off if Ker (λI−f) is nontrivial (i.e., Ker (λI−f) ={ 0}). A vectoru∈E is called an eigenvector, or proper vector, or characteristic vector off if u = 0 and if there is someλ∈K such that

f(u) =λu;

the scalar λ is then an eigenvalue, and we say thatu is an eigenvector associated with λ. Given any eigenvalueλ∈K, the nontrivial subspace Ker (λI−f) consists of all the eigenvectors associated withλ together with the zero vector; this subspace is denoted by Eλ(f), orE(λ, f), or even byEλ, and is called the eigenspace associated withλ, or proper subspace associated withλ.

Note that distinct eigenvectors may correspond to the same eigenvalue, but distinct eigenvalues correspond to disjoint sets of eigenvectors.

Remark: As we emphasized in the remark following Definition 7.4, we require an eigenvector to be nonzero. This requirement seems to have more benefits than inconvenients, even though it may considered somewhat inelegant because the set of all eigenvectors associated with an eigenvalue is not a subspace since the zero vector is excluded.

Let us now assume thatE is of finite dimensionn. The next proposition shows that the eigenvalues of a linear mapf :E→E are the roots of a polynomial associated withf. Proposition 12.1. LetE be any vector space of finite dimensionn and letf be any linear mapf :E→E. The eigenvalues off are the roots (inK) of the polynomial

det(λI−f).

Proof. A scalarλ∈K is an eigenvalue off iff there is some nonzero vectoru = 0 inE such that

f(u) =λu iff

(λI−f)(u) = 0 iff (λI−f) is not invertible iff, by Proposition 5.14,

det(λI−f) = 0.

In view of the importance of the polynomial det(λI−f), we have the following definition.

Definition 12.2. Given any vector spaceE of dimensionn, for any linear mapf :E→E, the polynomialPf(X) =χf(X) = det(XI−f) is called the characteristic polynomial of f. For any square matrixA, the polynomialPA(X) =χA(X) = det(XI−A) is called the characteristic polynomial ofA.

Note that we already encountered the characteristic polynomial in Section 5.7; see Definition 5.8.

Given any basis ( e_1, . . . , en), ifA =M(f) is the matrix off w.r.t. (e_1, . . . , en), we can compute the characteristic polynomialχf(X) = det(XI−f) off by expanding the following determinant:

det(XI X−a1 1 −a1 2 . . . −a1 n a2 1 X−a2 2 . . . −a2 n .−A) = −. . ... . −an 1 −an 2 . . . X−an n If we expand this determinant, we find that

χn (a1 1 +· · · +an n)Xn−1 +· · · + (−1)n det(A).A(X) = det(XI−A) =X−

The sum tr( A) =a1 1+· · ·+an n of the diagonal elements ofA is called the trace ofA. Since we proved in Section 5.7 that the characteristic polynomial only depends on the linear map f, the above shows that tr(A) has the same value for all matricesA representingf. Thus, the trace of a linear map is well-defined; we have tr(f) = tr(A) for any matrixA representing f.

Remark: The characteristic polynomial of a linear map is sometimes defined as det(f−XI). Since

det(f−XI) = (−1)n det(XI−f),

this makes essentially no difference but the version det(XI−f) has the small advantage that the coefficient ofXn is +1.

If we write

χA(X) = det(XI−A) =Xn τ_1(A)Xn−1 +· · · + (−1)kτk(A)Xn−k +· · · + (−1)nτn(A),−

then we just proved that

τ_1(A) = tr(A) and τn(A) = det(A).

It is also possible to express τk(A) in terms of determinants of certain submatrices ofA. For any nonempty subset,I⊆ {1, . . . , n}, sayI ={i_1, . . . , ik}, letAI,I be thek×k submatrix ofA whosejth column consists of the elementsaihij, whereh = 1, . . . , k. Then, it can be shown that

τk(A) = det(AI,I).

I⊆{ 1,...,n}

|I|=k

If all the roots,λ_1, . . . , λn, of the polynomial det(XI−A) belong to the fieldK, then we can write

χA(X) = det(XI−A) = (X−λ_1)· · ·(X−λn),

where some of theλis may appear more than once. Consequently,

χA(X) = det(XIn σ_1(λ)Xn−1 +· · · + (−1)kσk(λ)Xn−k +· · · + (−1)nσn(λ),−A) =X−

where

thek σk(λ) = λi,

I⊆{ 1,...,n} i∈I |I|=k

th symmetric function of theλi’s. From this, it clear that σk(λ) =τk(A)

and, in particular, the product of the eigenvalues off is equal to det(A) = det(f), and the sum of the eigenvalues off is equal to the trace tr(A) = tr(f), off; for the record,

tr( f) =λ_1 +· · · +λn det(f) =λ1· · ·λn, whereλ_1, . . . , λn are the eigenvalues off (andA), where some of theλis may appear more than once. In particular,f is not invertible iff it admits 0 has an eigenvalue.

Remark: Depending on the fieldK, the characteristic polynomialχA(X) = det(XI−A) may or may not have roots inK. This motivates considering algebraically closed fields, which are fieldsK such that every polynomial with coefficients inK has all its root inK. For example, overK = R, not every polynomial has real roots. If we consider the matrix

A = cosθ −sinθ ,sinθ cosθ

then the characteristic polynomial det( XI−A) has no real roots unlessθ =kπ. However, over the field C of complex numbers, every polynomial has roots. For example, the matrix above has the roots cosθ±isinθ =e±iθ.

It is possible to show that every linear map f over a complex vector spaceE must have some (complex) eigenvalue without having recourse to determinants (and the characteristic polynomial). Letn = dim(E), pick any nonzero vectoru∈E, and consider the sequence

u, f(u), f^2(u), . . . , fn(u). Since the above sequence hasn + 1 vectors andE has dimensionn, these vectors must be linearly dependent, so there are some complex numbersc_0, . . . , cm, not all zero, such that cm(u) +c_1fm−1(u) +· · · +cmu = 0,0f

where m≤n is the largest integer such that the coefficient offm(u) is nonzero (m must exits since we have a nontrivial linear dependency). Now, because the field C is algebraically closed, the polynomial

c_0Xm +c_1Xm−1 +· · · +cm can be written as a product of linear factors as

c_0Xm +c_1Xm−1 +· · · +cm =c_0(X−λ_1)· · ·(X−λm) for some complex numbersλ_1, . . . , λm∈ C, not necessarily distinct. But then, sincec_0 = 0,

c_0fm(u) +c_1fm−1(u) +· · · +cmu = 0

is equivalent to (f−λ_1I)æ · · · æ (f−λmI)(u) = 0.

If all the linear maps f−λiI were injective, then (f−λ_1I)æ· · ·æ(f−λmI) would be injective, contradicting the fact thatu = 0. Therefore, some linear mapf−λiI must have a nontrivial kernel, which means that there is somev = 0 so that

f(v) =λiv; that is,λi is some eigenvalue off andv is some eigenvector off.

As nice as the above argument is, it does not provide a method for finding the eigenvalues off, and even if we prefer avoiding determinants as a much as possible, we are forced to deal with the characteristic polynomial det(XI−f).

Definition 12.3. LetA be ann×n matrix over a field,K. Assume that all the roots of the characteristic polynomialχA(X) = det(XI−A) ofA belong toK, which means that we can write

det(XI−A) = (X−λ_1)k1 (X−λm)km,· · ·

where λ_1, . . . , λm∈K are the distinct roots of det(XI−A) andk_1 +· · · +km =n. The integer,ki, is called the algebraic multiplicity of the eigenvalue λi and the dimension of the eigenspace,Eλi = Ker(λiI−A), is called the geometric multiplicity ofλi. We denote the algebraic multiplicity ofλi by alg(λi) and its geometric multiplicity by geo(λi).

By definition, the sum of the algebraic multiplicities is equal ton but the sum of the geometric multiplicities can be strictly smaller.

Proposition 12.2. LetA be ann×n matrix over a fieldK and assume that all the roots of the characteristic polynomialχA(X) = det(XI−A) ofA belong toK. For every eigenvalueλi ofA, the geometric multiplicity ofλi is always less than or equal to its algebraic multiplicity, that is,

geo(λi)≤ alg(λi).

Proof. To see this, ifni is the dimension of the eigenspace,Eλi, associated with the eigenvalue,λi, we can form a basis obtained by picking a basis ofEλi and completing this basis. With respect to this new basis, our matrix is of the form

A

=

λiIni B 0 D

and a simple determinant calculation shows that

det(XI−A) = det(XI−A ) = (X−λi)ni det(XIn−ni−D).

Therefore, (X−λi)ni divides the characteristic polynomial ofA , and thus, the characteristic polynomial ofA. It follows thatni is less than or equal to the algebraic multiplicity ofλi.

The following proposition shows an interesting property of eigenspaces.

Proposition 12.3. LetE be any vector space of finite dimensionn and letf be any linear map. Ifu_1, . . . , um are eigenvectors associated with pairwise distinct eigenvaluesλ_1, . . . , λm, then the family (u_1, . . . , um) is linearly independent.

Proof. Assume that (u_1, . . . , um) is linearly dependent. Then, there existsµ_1, . . . , µk∈K such that

µ_1ui1 +· · · +µkuik = 0,

where 1≤k≤m,µi = 0 for alli, 1≤i≤k,{i_1, . . . , ik} ⊆ {1, . . . , m}, and no proper subfamily of (ui1, . . . , uik) is linearly dependent (in other words, we consider a dependency relation withk minimal). Applyingf to this dependency relation, we get

µ_1λi1ui1 +· · · +µkλikuik = 0, and if we multiply the original dependency relation byλi1 and subtract it from the above, we get

µ_2(λi2−λi1)ui2 +· · · +µk(λik−λi1)uik = 0,

which is a linear dependency among a proper subfamily of (ui1, . . . , uik), a contradiction.

Thus, from Proposition 12.3, ifλ_1, . . . , λm are all the pairwise distinct eigenvalues off (wherem≤n), we have a direct sum

Eλ1⊕ · · · ⊕Eλm of the eigenspacesEλi. Unfortunately, it is not always the case that E =Eλ1⊕ · · · ⊕Eλm.

When E =Eλ1⊕ · · · ⊕Eλm,

we say that f is diagonalizable (and similarly for any matrix associated withf). Indeed, picking a basis in eachEλi, we obtain a matrix which is a diagonal matrix consisting of the eigenvalues, eachλi occurring a number of times equal to the dimension ofEλi. This happens if the algebraic multiplicity and the geometric multiplicity of every eigenvalue are equal. In particular, when the characteristic polynomial hasn distinct roots, thenf is diagonalizable. It can also be shown that symmetric matrices have real eigenvalues and can be diagonalized.

For a negative example, we leave as exercise to show that the matrix

1 1M =0 1

cannot be diagonalized, even though 1 is an eigenvalue. The problem is that the eigenspace of 1 only has dimension 1. The matrix

A

=

cosθ −sinθ sinθ cosθ

cannot be diagonalized either, because it has no real eigenvalues, unlessθ =kπ. However, over the field of complex numbers, it can be diagonalized.

12.2 Reduction to Upper Triangular Form

Unfortunately, not every linear map on a complex vector space can be diagonalized. The next best thing is to “triangularize,” which means to find a basis over which the matrix has zero entries below the main diagonal. Fortunately, such a basis always exist.

We say that a square matrixA is an upper triangular matrix if it has the following shape,

ëa1 1 a1 2 a1 3 . . . a1 n−1 a1 n ö

ì 0 a2 2 a2 3 . . . a2 n−1 a2 n ÷

ì ÷

ì

÷

ì

0

0

a

3 3

. . . a

3

n

−

1

a

3

n

÷

ì .... . ÷,ì

÷

í

.

.

.

÷

ì 0 0 0 . . . an−1 n−1 an−1 nø

0 0 0 . . . 0 an n

i.e.,ai j = 0 wheneverj < i, 1≤i, j≤n.

Theorem 12.4. Given any finite dimensional vector space over a fieldK, for any linear mapf :E→E, there is a basis (u_1, . . . , un) with respect to whichf is represented by an upper triangular matrix (in Mn(K)) iff all the eigenvalues off belong toK. Equivalently, for everyn×n matrixA∈ Mn(K), there is an invertible matrixP and an upper triangular matrixT (both in Mn(K)) such that

A =P T P−1

iff all the eigenvalues ofA belong toK.

Proof. If there is a basis (u_1, . . . , un) with respect to whichf is represented by an upper triangular matrixT in Mn(K), then since the eigenvalues off are the diagonal entries ofT, all the eigenvalues off belong toK.

For the converse, we proceed by induction on the dimension n ofE. Forn = 1 the result is obvious. Ifn > 1, since by assumptionf has all its eigenvalue inK, pick some eigenvalue λ1∈K off, and letu_1 be some corresponding (nonzero) eigenvector. We can findn− 1 vectors (v_2, . . . , vn) such that (u_1, v_2, . . . , vn) is a basis ofE, and letF be the subspace of dimensionn− 1 spanned by (v_2, . . . , vn). In the basis (u_1, v_2. . . , vn), the matrix off is of

the form ëλ_1 a1 2 . . . a1 nö

ì

÷

ì

0

a

2 2

. . . a

2

n

÷

U =ì .... ÷ ,

í . . ø

0 an 2 . . . an n

since its first column contains the coordinates of λ_1u_1 over the basis (u_1, v_2,. . . , vn). If we letp:E→F be the projection defined such thatp(u_1) = 0 andp(vi) =vi when 2≤i≤n, the linear mapg:F→F defined as the restriction ofpæf toF is represented by the 12.2. REDUCTION TO UPPER TRIANGULAR FORM 325

( n− 1)× (n− 1) matrixV = (ai j)2≤i,j≤n over the basis (v_2, . . . , vn). We need to prove that all the eigenvalues ofg belong toK. However, since the first column ofU has a single nonzero entry, we get

χU(X) = det(XI−U) = (X−λ_1) det(XI−V ) = (X−λ_1)χV(X),

where χU(X) is the characteristic polynomial ofU andχV(X) is the characteristic polynomial ofV . It follows thatχV(X) dividesχU(X), and since all the roots ofχU(X) are inK, all the roots ofχV(X) are also inK. Consequently, we can apply the induction hypothesis, and there is a basis (u_2, . . . , un) ofF such thatg is represented by an upper triangular matrix (bi j)1≤i,j≤n−1. However,

E =Ku1⊕F,

and thus (u_1, . . . , un) is a basis forE. Sincep is the projection fromE =Ku1⊕F ontoF andg:F→F is the restriction ofpæf toF, we have

f(u_1) =λ_1u_1

andi

f(ui+1) =a1 iu_1 + bi juj+1

j=1

for somea1 i∈K, when 1≤i≤n−1. But then the matrix off with respect to (u_1, . . . , un) is upper triangular.

For the matrix version, we assume that A is the matrix off with respect to some basis, Then, we just proved that there is a change of basis matrixP such thatA =P T P−1 where T is upper triangular.

If A =P T P−1 whereT is upper triangular, note that the diagonal entries ofT are the eigenvaluesλ_1, . . . , λn ofA. Indeed,A andT have the same characteristic polynomial. Also, ifA is a real matrix whose eigenvalues are all real, thenP can be chosen to real, and ifA is a rational matrix whose eigenvalues are all rational, thenP can be chosen rational. Since any polynomial over C has all its roots in C, Theorem 12.4 implies that every complexn×n matrix can be triangularized.

If E is a Hermitian space, the proof of Theorem 12.4 can be easily adapted to prove that there is an orthonormal basis (u_1, . . . , un) with respect to which the matrix off is upper triangular. This is usually known as Schur’s lemma.

Theorem 12.5. (Schur decomposition) Given any linear mapf :E→E over a complex Hermitian spaceE, there is an orthonormal basis (u_1, . . . , un) with respect to whichf is represented by an upper triangular matrix. Equivalently, for everyn×n matrixA∈ Mn(C), there is a unitary matrixU and an upper triangular matrixT such that

A =UT U∗. IfA is real and if all its eigenvalues are real, then there is an orthogonal matrixQ and a real upper triangular matrixT such that

A =QT Q .

Proof. During the induction, we chooseF to be the orthogonal complement of Cu_1 and we pick orthonormal bases. IfE is a real Euclidean space and if the eigenvalues off are all real, the proof also goes through with real matrices.

Using, Theorem 12.5, we can derive the fact that if A is a Hermitian matrix, then there is a unitary matrixU and a real diagonal matrixD such thatA =UDU∗. Indeed, since A∗ =A, we get

UT U∗ =UT∗U∗,

which implies that T =T∗. SinceT is an upper triangular matrix,T∗ is a lower triangular matrix, which implies thatT is a real diagonal matrix. In fact, applying this result to a (real) symmetric matrixA, we obtain the fact that all the eigenvalues of a symmetric matrix are real, and by applying Theorem 12.5 again, we conclude thatA =QDQ , whereQ is orthogonal andD is a real diagonal matrix. We will also prove this in Chapter 13.

When A has complex eigenvalues, there is a version of Theorem 12.5 involving only real matrices provided that we allowT to be block upper-triangular (the diagonal entries may be 2× 2 matrices or real entries).

Theorem 12.5 is not a very practical result but it is a useful theoretical result to cope with matrices that cannot be diagonalized. For example, it can be used to prove that every complex matrix is the limit of a sequence of diagonalizable matrices that have distinct eigenvalues!

12.3 Location of Eigenvalues

If A is ann×n complex (or real) matrixA, it would be useful to know, even roughly, where the eigenvalues ofA are located in the complex plane C. The Gershgorin discs provide some precise information about this.

Definition 12.4. For any complexn×n matrixA, fori = 1, . . . , n, let

n

Ri(A) =|ai j|j=1

j=i

and letn

G(A) ={z∈ C| |z−ai i| ≤Ri(A)}.

i=1

Each disc{z∈ C| |z−ai i| ≤Ri(A)} is called a Gershgorin disc and their unionG(A) is called the Gershgorin domain.

12.3. LOCATION OF EIGENVALUES 327

Although easy to prove, the following theorem is very useful:

Theorem 12.6. (Gershgorin’s disc theorem) For any complexn×n matrixA, all the eigenvalues ofA belong to the Gershgorin domainG(A). Furthermore the following properties hold:

(1) If A is strictly row diagonally dominant, that is

n

ai i|>|ai j|, fori = 1, . . . , n,| j=1, j=i

thenA is invertible. (2) If A is strictly row diagonally dominant, and ifai i> 0 fori = 1, . . . , n, then every eigenvalue ofA has a strictly positive real part.

Proof. Letλ be any eigenvalue ofA and letu be a corresponding eigenvector (recall that we must haveu = 0). Letk be an index such that

|

uk| = max|ui|.

1≤i≤n

SinceAu =λu, we haven

(λ−ak k)uk = ak juj,

j=1

j=k

which implies thatn n

|λ−ak k||uk| ≤j=1 |ak j||uj| ≤ |uk|j=1 |ak j|

j=k j=k

and sinceu = 0 and|uk| = max1≤i≤n|ui|, we must have|uk| = 0, and it follows that

n

λ−ak k| ≤ j=1 |ak j| =Rk(A),|

j=k

and thus λ∈ {z∈ C| |z−ak k| ≤Rk(A)} ⊆G(A),

as claimed.

(1) Strict row diagonal dominance implies that 0 does not belong to any of the Gershgorin discs, so all eigenvalues ofA are nonzero, andA is invertible.

(2) If A is strictly row diagonally dominant and ai i> 0 fori = 1, . . . , n, then each of the Gershgorin discs lies strictly in the right half-plane, so every eigenvalue ofA has a strictly positive real part.

In particular, Theorem 12.6 implies that if a symmetric matrix is strictly row diagonally dominant and has strictly positive diagonal entries, then it is positive definite. Theorem 12.6 is sometimes called the Gershgorin–Hadamard theorem.

SinceA andA have the same eigenvalues (even for complex matrices) we also have a version of Theorem 12.6 for the discs of radius

n

Cj(A) =|ai j|,

i=1i=j

whose domain is denoted byG(A ). Thus we get the following:

Theorem 12.7. For any complexn×n matrixA, all the eigenvalues ofA belong to the intersection of the Gershgorin domains, G(A)∩G(A ). Furthermore the following properties hold:

(1) If A is strictly column diagonally dominant, that is

n

ai i|>|ai j|, forj = 1, . . . , n,| i=1, i=j

thenA is invertible. (2) If A is strictly column diagonally dominant, and ifai i> 0 fori = 1, . . . , n, then every eigenvalue ofA has a strictly positive real part.

There are refinements of Gershgorin’s theorem and eigenvalue location results involving other domains besides discs; for more on this subject, see Horn and Johnson [55], Sections 6.1 and 6.2.

Remark: Neither strict row diagonal dominance nor strict column diagonal dominance are necessary for invertibility. Also, if we relax all strict inequalities to inequalities, then row diagonal dominance (or column diagonal dominance) is not a sufficient condition for invertibility.

12.4 Summary

The main concepts and results of this chapter are listed below: Diagonal matrix.•

• Eigenvalues, eigenvectors; the eigenspace associated with an eigenvalue. The characteristic polynomial.•

12.4. SUMMARY 329

• The trace.

• algebraic and geometric multiplicity.

• Eigenspaces associated with distinct eigenvalues form a direct sum (Proposition 12.3).

• Reduction of a matrix to an upper-triangular matrix.

Schur decomposition.•

•

The Gershgorin’s discs can be used to locate the eigenvalues of a complex matrix; see Theorems 12.6 and 12.7.

Chapter 13 Spectral Theorems in Euclidean and Hermitian Spaces

13.1 Introduction

The goal of this chapter is to show that there are nice normal forms for symmetric matrices, skew-symmetric matrices, orthogonal matrices, and normal matrices. The spectral theorem for symmetric matrices states that symmetric matrices have real eigenvalues and that they can be diagonalized over an orthonormal basis. The spectral theorem for Hermitian matrices states that Hermitian matrices also have real eigenvalues and that they can be diagonalized over a complex orthonormal basis. Normal real matrices can be block diagonalized over an orthonormal basis with blocks having size at most two, and there are refinements of this normal form for skew-symmetric and orthogonal matrices.

13.2 Normal Linear Maps

We begin by studying normal maps, to understand the structure of their eigenvalues and eigenvectors. This section and the next two were inspired by Lang [65], Artin [3], Mac Lane and Birkhoff [70], Berger [6], and Bertin [10].

Definition 13.1. Given a Euclidean spaceE, a linear mapf :E→E is normal if

fæf∗ =f∗æf.

A linear mapf :E→E is self-adjoint iff =f∗, skew-self-adjoint iff =−f∗, and orthogonal iffæf∗ =f∗æf = id.

Obviously, a self-adjoint, skew-self-adjoint, or orthogonal linear map is a normal linear map. Our first goal is to show that for every normal linear mapf :E→E, there is an orthonormal basis (w.r.t.−,−) such that the matrix off over this basis has an especially

331 nice form: It is a block diagonal matrix in which the blocks are either one-dimensional matrices (i.e., single entries) or two-dimensional matrices of the form

λ µ . −µ λ

This normal form can be further refined iff is self-adjoint, skew-self-adjoint, or orthogonal. As a first step, we show thatf andf∗ have the same kernel whenf is normal. Proposition 13.1. Given a Euclidean spaceE, iff :E→E is a normal linear map, then Kerf = Kerf∗.

Proof. First, let us prove that

f(u), f(v) = f∗(u), f∗(v)

for allu, v∈E. Sincef∗ is the adjoint off andfæf∗ =f∗æf, we have

f (u), f(u) = u,(f∗æf)(u), = u,(fæf∗)(u), = f∗(u), f∗(u).

Since−,− is positive definite,

f(u), f(u) = 0 iff f(u) = 0, f∗(u), f∗(u) = 0 iff f∗(u) = 0,

and since f(u), f(u) = f∗(u), f∗(u), we have

f(u) = 0 iff f∗(u) = 0. Consequently, Kerf = Kerf∗.

The next step is to show that for every linear map f :E→E there is some subspaceW of dimension 1 or 2 such thatf(W)⊆W. When dim(W) = 1, the subspaceW is actually an eigenspace for some real eigenvalue off. Furthermore, whenf is normal, there is a subspaceW of dimension 1 or 2 such thatf(W)⊆W andf∗(W)⊆W. The difficulty is that the eigenvalues off are not necessarily real. One way to get around this problem is to complexify both the vector spaceE and the inner product−,−.

Every real vector space E can be embedded into a complex vector spaceEC, and every linear mapf :E→E can be extended to a linear mapfC:EC→EC.

Definition 13.2. Given a real vector spaceE, letEC be the structureE×E under the addition operation

( u_1, u_2) + (v_1, v_2) = (u_1 +v_1, u_2 +v_2),

and let multiplication by a complex scalarz =x +iy be defined such that

(x +iy)· (u, v) = (xu−yv, yu +xv).

The spaceEC is called the complexification ofE.

It is easily shown that the structureEC is a complex vector space. It is also immediate that

(0, v) =i(v, 0),

and thus, identifyingE with the subspace ofEC consisting of all vectors of the form (u, 0), we can write

(u, v) =u +iv. Observe that if (e_1, . . . , en) is a basis ofE (a real vector space), then (e_1, . . . , en) is also a basis ofEC (recall thatei is an abreviation for (ei,0)).

A linear mapf :E→E is extended to the linear mapfC:EC→EC defined such that fC(u +iv) =f(u) +if(v).

For any basis ( e_1, . . . , en) ofE, the matrixM(f) representingf over (e_1, . . . , en) is identical to the matrixM(fC) representingfC over (e_1, . . . , en), where we view (e_1, . . . , en) as a basis ofEC. As a consequence, det(zI−M(f)) = det(zI−M(fC)), which means thatf andfC have the same characteristic polynomial (which has real coefficients). We know that every polynomial of degreen with real (or complex) coefficients always hasn complex roots (counted with their multiplicity), and the roots of det(zI−M(fC)) that are real (if any) are the eigenvalues off.

Next, we need to extend the inner product onE to an inner product onEC. The inner product− ,− on a Euclidean spaceE is extended to the Hermitian positive definite form−,−C onEC as follows:

u_1 +iv_1, u_2 +iv2 C = u_1, u_2 + v_1, v_2 +i(u_2, v1− u_1, v_2 ).

It is easily verified that−,−C is indeed a Hermitian form that is positive definite, and it is clear that−,−C agrees with−,− on real vectors. Then, given any linear map f :E→E, it is easily verified that the mapf∗ defined such thatC

f∗(u +iv) =f∗(u) +if∗(v)C

for allu, v∈E is the adjoint offC w.r.t.−,−C.

Assuming again that E is a Hermitian space, observe that Proposition 13.1 also holds. We deduce the following corollary.

Proposition 13.2. Given a Hermitian spaceE, for any normal linear mapf :E→E, we have Ker (f)∩ Im(f) = (0).

Proof. Assumev∈ Ker (f)∩ Im(f) = (0), which means thatv =f(u) for someu∈E, and f(v) = 0. By Proposition 13.1, Ker (f) = Ker (f∗), sof(v) = 0 implies thatf∗(v) = 0. Consequently,

0 = f∗(v), u

= v, f(u)

= v, v ,

and thus, v = 0.

We also have the following crucial proposition relating the eigenvalues off andf∗.

Proposition 13.3. Given a Hermitian spaceE, for any normal linear mapf :E→E, a vectoru is an eigenvector off for the eigenvalueλ (in C) iffu is an eigenvector off∗ for the eigenvalueλ.

Proof. First, it is immediately verified that the adjoint off−λid isf∗−λid. Furthermore, f−λid is normal. Indeed,

( f−λid)æ (f−λid)∗ = (f−λid)æ (f∗−λid), =fæf∗−λf−λf∗ +λλid, =f∗æf−λf∗−λf +λλid, = (f∗−λid)æ (f−λid), = (f−λid)∗æ (f−λid).

Applying Proposition 13.1 tof−λid, for every nonnull vectoru, we see that (f−λid)(u) = 0 iff (f∗−λid)(u) = 0,

which is exactly the statement of the proposition.

The next proposition shows a very important property of normal linear maps: Eigenvectors corresponding to distinct eigenvalues are orthogonal.

Proposition 13.4. Given a Hermitian spaceE, for any normal linear mapf :E→E, if u andv are eigenvectors off associated with the eigenvaluesλ andµ (in C) whereλ =µ, then u, v = 0.

Proof. Let us compute f(u), v in two different ways. Sincev is an eigenvector off forµ, by Proposition 13.3,v is also an eigenvector off∗ forµ, and we have

f(u), v = λu, v =λ u, v and f(u), v = u, f∗(v) = u,µv =µ u, v ,

where the last identity holds because of the semilinearity in the second argument, and thus λ u, v =µ u, v ,

that is,

(λ−µ)u, v = 0, which implies that u, v = 0, sinceλ =µ.

We can also show easily that the eigenvalues of a self-adjoint linear map are real. Proposition 13.5. Given a Hermitian spaceE, all the eigenvalues of any self-adjoint linear mapf :E→E are real.

Proof. Letz (in C) be an eigenvalue off and letu be an eigenvector forz. We compute f(u), u in two different ways. We have

f(u), u = zu, u =z u, u , and sincef =f∗, we also have

f(u), u = u, f∗(u) = u, f(u) = u, zu =z u, u . Thus,

z u, u =z u, u ,

which implies thatz =z, sinceu = 0, andz is indeed real.

There is also a version of Proposition 13.5 for a (real) Euclidean spaceE and a self-adjoint mapf :E→E.

Proposition 13.6. Given a Euclidean spaceE, iff :E→E is any self-adjoint linear map, then every eigenvalueλ offC is real and is actually an eigenvalue of f (which means that there is some real eigenvectoru∈E such thatf(u) =λu). Therefore, all the eigenvalues of f are real.

Proof. LetEC be the complexification ofE,−,−C the complexification of the inner product −,− onE, andfC:EC→EC the complexification off :E→E. By definition offC and −,−C, iff is self-adjoint, we have

fC(u_1 +iv_1), u_2 +iv2 C = f(u_1) +if(v_1), u_2 +iv2 C

= f(u_1), u_2 + f(v_1), v_2 +i(u_2, f(v_1)− f(u_1), v_2 ) = u_1, f(u_2) + v_1, f(v_2) +i(f(u_2), v1− u_1, f(v_2) ) = u_1 +iv_1, f(u_2) +if(v_2) C

= u_1 +iv_1, fC(u_2 +iv_2) C,

which shows thatfC is also self-adjoint with respect to−,−C.

As we pointed out earlier, f andfC have the same characteristic polynomial det(zI−fC) = det(zI−f), which is a polynomial with real coefficients. Proposition 13.5 shows that the zeros of det(zI−fC) = det(zI−f) are all real, and for each real zeroλ of det(zI−f), the linear mapλid−f is singular, which means that there is some nonzerou∈E such that f(u) =λu. Therefore, all the eigenvalues off are real.

Given any subspaceW of a Euclidean spaceE, recall that the orthogonal complement W⊥ ofW is the subspace defined such that

W⊥ ={u∈E| u, w = 0, for allw∈W}.

Recall from Proposition 9.9 that E =W⊕ W⊥ (this can be easily shown, for example, by constructing an orthonormal basis ofE using the Gram–Schmidt orthonormalization procedure). The same result also holds for Hermitian spaces; see Proposition 11.10.

As a warm up for the proof of Theorem 13.10, let us prove that every self-adjoint map on a Euclidean space can be diagonalized with respect to an orthonormal basis of eigenvectors.

Theorem 13.7. (Spectral theorem for self-adjoint linear maps on a Euclidean space) Given a Euclidean spaceE of dimensionn, for every self-adjoint linear mapf :E→E, there is an orthonormal basis (e_1, . . . , en) of eigenvectors off such that the matrix off w.r.t. this basis is a diagonal matrixëλ1 . . . ö

ì λ_2 . . .÷

ì... ÷

.

÷ ,ì

í . . ø. . . λn

withλi∈ R.

Proof. We proceed by induction on the dimensionn ofE as follows. Ifn = 1, the result is trivial. Assume now thatn≥ 2. From Proposition 13.6, all the eigenvalues off are real, so pick some eigenvalueλ∈ R, and letw be some eigenvector forλ. By dividingw by its norm, we may assume thatw is a unit vector. LetW be the subspace of dimension 1 spanned byw. Clearly,f(W)⊆W. We claim thatf(W⊥)⊆W⊥, whereW⊥ is the orthogonal complement ofW.

Indeed, for anyv∈W⊥, that is, if v, w = 0, becausef is self-adjoint andf(w) =λw, we have

f (v), w = v, f(w) = v, λw

=λ v, w = 0 since v, w = 0. Therefore,

f(W⊥)⊆W⊥.

Clearly, the restriction off toW⊥ is self-adjoint, and we conclude by applying the induction hypothesis toW⊥ (whose dimension isn− 1).

We now come back to normal linear maps. One of the key points in the proof of Theorem 13.7 is that we found a subspaceW with the property thatf(W)⊆W implies thatf(W⊥)⊆ W⊥. In general, this does not happen, but normal maps satisfy a stronger property which ensures that such a subspace exists.

The following proposition provides a condition that will allow us to show that a normal linear map can be diagonalized. It actually holds for any linear map. We found the inspiration for this proposition in Berger [6].

Proposition 13.8. Given a Hermitian spaceE, for any linear mapf :E→E and any subspaceW ofE, iff(W)⊆ W, thenf∗ W⊥ ⊆W⊥. Consequently, iff(W)⊆W and f∗(W)⊆W, thenf W⊥ ⊆W⊥ andf∗ W⊥ ⊆W⊥.

Proof. Ifu∈W⊥, then

w, u = 0 for allw∈W . However,

f(w), u = w, f∗(u),

and f(W)⊆W implies thatf(w)∈W. Sinceu∈W⊥, we get 0 = f(w), u = w, f∗(u),

which shows that w, f∗(u) = 0 for allw∈W, that is,f∗(u)∈W⊥. Therefore, we have f∗(W⊥)⊆W⊥.

We just proved that if f(W)⊆W, thenf∗ W⊥ ⊆W⊥. If we also havef∗(W)⊆W, then by applying the above fact tof∗, we getf∗∗(W⊥)⊆W⊥, and sincef∗∗ =f, this is justf(W⊥)⊆W⊥, which proves the second statement of the proposition.

It is clear that the above proposition also holds for Euclidean spaces.

Although we are ready to prove that for every normal linear map f (over a Hermitian space) there is an orthonormal basis of eigenvectors (see Theorem 13.11 below), we now return to real Euclidean spaces.

Iff :E→E is a linear map andw =u +iv is an eigenvector offC:EC→EC for the eigenvaluez =λ +iµ, whereu, v∈E andλ, µ∈ R, since

fC(u +iv) =f(u) +if(v) and fC(u +iv) = (λ +iµ)(u +iv) =λu−µv +i(µu +λv), we have f(u) =λu−µv and f(v) =µu +λv, from which we immediately obtain

fC(u−iv) = (λ−iµ)(u−iv),

which shows thatw =u−iv is an eigenvector offC forz =λ−iµ. Using this fact, we can prove the following proposition.

Proposition 13.9. Given a Euclidean spaceE, for any normal linear mapf :E→E, if w =u+iv is an eigenvector offC associated with the eigenvaluez =λ+iµ (whereu, v∈E andλ, µ∈ R), ifµ = 0 (i.e.,z is not real) then u, v = 0 and u, u = v, v , which implies thatu andv are linearly independent, and ifW is the subspace spanned byu andv, then f(W) =W andf∗(W) =W. Furthermore, with respect to the (orthogonal) basis (u, v), the restriction off toW has the matrix

λ µ . −µ λ

If µ = 0, thenλ is a real eigenvalue off, and eitheru orv is an eigenvector off forλ. If W is the subspace spanned byu ifu = 0, or spanned byv = 0 ifu = 0, thenf(W)⊆W and f∗(W)⊆W.

Proof. Sincew =u +iv is an eigenvector offC, by definition it is nonnull, and eitheru = 0 orv = 0. From the fact stated just before Proposition 13.9,u−iv is an eigenvector offC for λ−iµ. It is easy to check thatfC is normal. However, ifµ = 0, thenλ +iµ =λ−iµ, and from Proposition 13.4, the vectorsu +iv andu−iv are orthogonal w.r.t.−,−C, that is,

u +iv, u−iv C = u, u− v, v + 2i u, v = 0.

Thus, we get u, v independent. Since = 0 and u, u = v, v , and sinceu = 0 orv = 0,u andv are linearly

f(u) =λu−µv and f(v) =µu +λv and since by Proposition 13.3u +iv is an eigenvector off∗ forλ−iµ, we haveC

f∗(u) =λu +µv and f∗(v) =−µu +λv,

and thusf(W) =W andf∗(W) =W, whereW is the subspace spanned byu andv. Whenµ = 0, we have

f(u) =λu and f(v) =λv,

and since u = 0 orv = 0, eitheru orv is an eigenvector off forλ. IfW is the subspace spanned byu ifu = 0, or spanned byv ifu = 0, it is obvious thatf(W)⊆ W and f∗(W)⊆W. Note thatλ = 0 is possible, and this is why⊆ cannot be replaced by =.

The beginning of the proof of Proposition 13.9 actually shows that for every linear map f :E→E there is some subspaceW such thatf(W)⊆W, whereW has dimension 1 or 2. In general, it doesn’t seem possible to prove thatW⊥ is invariant underf. However, this happens whenf is normal.

We can finally prove our first main theorem.

Theorem 13.10. (Main spectral theorem) Given a Euclidean spaceE of dimensionn, for every normal linear mapf :E→E, there is an orthonormal basis (e_1, . . . , en) such that the matrix off w.r.t. this basis is a block diagonal matrix of the form

ëA_1 . . .ö

ì

÷

ì

A

2

. . .

÷

ì .... ÷

í . . ø

. . . Ap

such that each blockAj is either a one-dimensional matrix (i.e., a real scalar) or a twodimensional matrix of the form

Aj = λj −µj ,µj λj

whereλj, µj∈ R, withµj> 0.

Proof. We proceed by induction on the dimensionn ofE as follows. Ifn = 1, the result is trivial. Assume now thatn≥ 2. First, since C is algebraically closed (i.e., every polynomial has a root in C), the linear mapfC:EC→EC has some eigenvaluez =λ +iµ (where λ, µ∈ R). Letw =u +iv be some eigenvector offC forλ +iµ (whereu, v∈E). We can now apply Proposition 13.9.

If µ = 0, then eitheru orv is an eigenvector off forλ∈ R. LetW be the subspace of dimension 1 spanned bye_1 =u/ u ifu = 0, or bye_1 =v/ v otherwise. It is obvious thatf(W)⊆W andf∗(W)⊆W. The orthogonalW⊥ ofW has dimensionn− 1, and by Proposition 13.8, we havef W⊥ ⊆W⊥. But the restriction off toW⊥ is also normal, and we conclude by applying the induction hypothesis toW⊥.

If µ = 0, then u, v = 0 and u, u = v, v , and ifW is the subspace spanned byu/ u andv/ v , thenf(W) =W andf∗(W) =W. We also know that the restriction off toW has the matrix

λ µ

−µ λ

with respect to the basis ( u/ u , v/ v ). Ifµ < 0, we letλ_1 =λ,µ_1 =−µ,e_1 =u/ u , and e_2 =v/ v . Ifµ > 0, we letλ_1 =λ,µ_1 =µ,e_1 =v/ v , ande_2 =u/ u . In all cases, it is easily verified that the matrix of the restriction off toW w.r.t. the orthonormal basis (e_1, e_2) is

A_1 = λ1 −µ1 ,µ1 λ1

whereλ_1, µ1∈ R, withµ_1> 0. However,W⊥ has dimensionn−2, and by Proposition 13.8, f W⊥ ⊆W⊥. Since the restriction off toW⊥ is also normal, we conclude by applying the induction hypothesis toW⊥.

After this relatively hard work, we can easily obtain some nice normal forms for the matrices of self-adjoint, skew-self-adjoint, and orthogonal linear maps. However, for the sake of completeness (and since we have all the tools to so do), we go back to the case of a Hermitian space and show that normal linear maps can be diagonalized with respect to an orthonormal basis. The proof is a slight generalization of the proof of Theorem 13.6.

Theorem 13.11. (Spectral theorem for normal linear maps on a Hermitian space) Given a Hermitian spaceE of dimensionn, for every normal linear mapf :E→E there is an orthonormal basis (e_1, . . . , en) of eigenvectors off such that the matrix off w.r.t. this basis is a diagonal matrixëλ1 . . . ö

ì

÷

ì

λ

2

. . .

÷

ì .... ÷ ,

í . . ø

. . . λn

whereλj∈ C.

Proof. We proceed by induction on the dimensionn ofE as follows. Ifn = 1, the result is trivial. Assume now thatn≥ 2. Since C is algebraically closed (i.e., every polynomial has a root in C), the linear mapf :E→E has some eigenvalueλ∈ C, and letw be some unit eigenvector forλ. LetW be the subspace of dimension 1 spanned byw. Clearly,f(W)⊆W. By Proposition 13.3,w is an eigenvector off∗ forλ, and thusf∗(W)⊆W. By Proposition 13.8, we also havef(W⊥)⊆W⊥. The restriction off toW⊥ is still normal, and we conclude by applying the induction hypothesis toW⊥ (whose dimension isn− 1).

Thus, in particular, self-adjoint, skew-self-adjoint, and orthogonal linear maps can be diagonalized with respect to an orthonormal basis of eigenvectors. In this latter case, though, an orthogonal map is called a unitary map. Also, Proposition 13.5 shows that the eigenvalues of a self-adjoint linear map are real. It is easily shown that skew-self-adjoint maps have eigenvalues that are pure imaginary or null, and that unitary maps have eigenvalues of absolute value 1.

Remark: There is a converse to Theorem 13.11, namely, if there is an orthonormal basis (e_1, . . . , en) of eigenvectors off, thenf is normal. We leave the easy proof as an exercise.

13.3 Self-Adjoint, Skew-Self-Adjoint, and Orthogonal Linear Maps

We begin with self-adjoint maps.

Theorem 13.12. Given a Euclidean spaceE of dimensionn, for every self-adjoint linear mapf :E→E, there is an orthonormal basis (e_1, . . . , en) of eigenvectors off such that the matrix off w.r.t. this basis is a diagonal matrix

ëλ1 . . . ö

ì λ_2 . . .÷

ì... ÷

.

÷ ,ì

í . . ø. . . λn

whereλi∈ R.

Proof. We already proved this; see Theorem 13.6. However, it is instructive to give a more direct method not involving the complexification of−,− and Proposition 13.5.

Since C is algebraically closed,fC has some eigenvalueλ +iµ, and letu +iv be some eigenvector offC forλ+iµ, whereλ, µ∈ R andu, v∈E. We saw in the proof of Proposition 13.9 that

f(u) =λu−µv and f(v) =µu +λv. Sincef =f∗,

f(u), v = u, f(v) for allu, v∈E. Applying this to

f(u) =λu−µv and f(v) =µu +λv,

we get f(u), v = λu−µv, v =λ u, v−µ v, v and

u, f(v) = u, µu +λv =µ u, u +λ u, v , and thus we get

λ u, v−µ v, v =µ u, u +λ u, v , that is,

µ(u, u + v, v ) = 0,

which impliesµ = 0, since eitheru = 0 orv = 0. Therefore,λ is a real eigenvalue off. Now, going back to the proof of Theorem 13.10, only the case whereµ = 0 applies, and the induction shows that all the blocks are one-dimensional.

Theorem 13.12 implies that ifλ_1, . . . , λp are the distinct real eigenvalues off, andEi is the eigenspace associated withλi, then

E =E1⊕ · · · ⊕Ep, whereEi andEj are orthogonal for alli =j.

Remark: Another way to prove that a self-adjoint map has a real eigenvalue is to use a little bit of calculus. We learned such a proof from Herman Gluck. The idea is to consider the real-valued function Φ:E→ R defined such that

Φ(u) = f(u), u

for everyu∈E. This function isC∞, and if we representf by a matrixA over some orthonormal basis, it is easy to compute the gradient vector

Φ(X) = ∂Φ (X), . . . ,∂Φ (X)∂x1 ∂xn

of Φ atX. Indeed, we find that

Φ(X) = (A +A )X,

whereX is a column vector of sizen. But sincef is self-adjoint,A =A , and thus

Φ(X) = 2AX.

The next step is to find the maximum of the function Φ on the sphere

Sn−1 ={(x_1, . . . , xn)∈ Rn x^2 +· · · +x^2 = 1}.| 1 n

SinceSn−1 is compact and Φ is continuous, and in factC∞, Φ takes a maximum at someX onSn−1. But then it is well known that at an extremumX of Φ we must have

dΦX(Y ) = Φ(X), Y = 0

for all tangent vectorsY toSn−1 atX, and so Φ(X) is orthogonal to the tangent plane at X, which means that

Φ(X) =λX for someλ∈ R. Since Φ(X) = 2AX, we get

2AX =λX,

and thusλ/2 is a real eigenvalue ofA (i.e., off).

Next, we consider skew-self-adjoint maps. Theorem 13.13. Given a Euclidean spaceE of dimensionn, for every skew-self-adjoint linear mapf :E→E there is an orthonormal basis (e_1, . . . , en) such that the matrix off w.r.t. this basis is a block diagonal matrix of the form

ëA_1 . . .ö

ì

÷

ì

A

2

. . .

÷

ì .... ÷

í . . ø

. . . Ap

such that each blockAj is either 0 or a two-dimensional matrix of the form

Aj =0 −µj ,µj 0

whereµj ∈ R, withµj> 0. In particular, the eigenvalues offC are pure imaginary of the form±iµj or 0.

Proof. The case wheren = 1 is trivial. As in the proof of Theorem 13.10,fC has some eigenvaluez =λ +iµ, whereλ, µ∈ R. We claim thatλ = 0. First, we show that f(w), w = 0 for allw∈E. Indeed, sincef =−f∗, we get f(w), w = w, f∗(w) = w,−f(w) =−w, f(w) =−f(w), w ,

since−,− is symmetric. This implies that f(w), w = 0.

Applying this tou andv and using the fact that f(u) =λu−µv and f(v) =µu +λv,

we get 0 = f(u), u = λu−µv, u =λ u, u−µ u, v and

0 = f(v), v = µu +λv, v =µ u, v +λ v, v ,

from which, by addition, we get

λ(v, v + v, v ) = 0.

Sinceu = 0 orv = 0, we haveλ = 0.

Then, going back to the proof of Theorem 13.10, unless µ = 0, the case whereu andv are orthogonal and span a subspace of dimension 2 applies, and the induction shows that all the blocks are two-dimensional or reduced to 0.

Remark: One will note that iff is skew-self-adjoint, thenifC is self-adjoint w.r.t.−,−C. By Proposition 13.5, the mapifC has real eigenvalues, which implies that the eigenvalues of fC are pure imaginary or 0.

Finally, we consider orthogonal linear maps.

Theorem 13.14. Given a Euclidean spaceE of dimensionn, for every orthogonal linear mapf :E→E there is an orthonormal basis (e_1, . . . , en) such that the matrix off w.r.t. this basis is a block diagonal matrix of the form

ëA_1 . . .ö

ì

÷

ì

A

2

. . .

÷

ì .... ÷

í . . ø. . . Ap

such that each blockAj is either 1,−1, or a two-dimensional matrix of the form cosθj − sinθjAj = sinθj cosθj

where 0< θj< π. In particular, the eigenvalues offC are of the form cosθj±isinθj, 1, or 1.−

Proof. The case wheren = 1 is trivial. As in the proof of Theorem 13.10,fC has some eigenvaluez =λ +iµ, whereλ, µ∈ R. It is immediately verified thatfæf∗ =f∗æf = id implies thatfCæf∗ =f∗ fC = id, so the mapfC is unitary. In fact, the eigenvalues offCC Cæ

have absolute value 1. Indeed, ifz (in C) is an eigenvalue offC, andu is an eigenvector for z, we have

fC(u), fC(u) = zu, zu =zz u, u and

fC(u), fC(u) = u,(f∗ fC)(u) = u, u ,Cæfrom which we get

zz u, u = u, u .

Since u = 0, we havezz = 1, i.e.,|z| = 1. As a consequence, the eigenvalues offC are of the form cosθ±isinθ, 1, or−1. The theorem then follows immediately from Theorem 13.10, where the conditionµ > 0 implies that sinθj> 0, and thus, 0< θj< π.

It is obvious that we can reorder the orthonormal basis of eigenvectors given by Theorem 13.14, so that the matrix off w.r.t. this basis is a block diagonal matrix of the form ëA_1 . . .ö

ì

÷

ì

.

...

.

.

÷

ì ÷

ì

÷

í

. . . A

r

÷

ì Iq ø. . .− Ip

where each blockAj is a two-dimensional rotation matrixAj =±I_2 of the form

A

j

=

cosθj − sinθj sinθj cosθj

with 0< θj< π.

The linear map f has an eigenspaceE(1, f) = Ker (f− id) of dimensionp for the eigenvalue 1, and an eigenspaceE(−1, f) = Ker (f + id) of dimensionq for the eigenvalue−1. If det(f) = +1 (f is a rotation), the dimensionq ofE(−1, f) must be even, and the entries in −Iq can be paired to form two-dimensional blocks, if we wish. In this case, every rotation in SO(n) has a matrix of the form

ëA_1 . . .ö

ì

÷

í

.

...

.

÷

ì ÷

ì . . . Am ø. . . In−2m

where the firstm blocksAj are of the form

A

j

=

cosθj − sinθj sinθj cosθj

with 0 < θj≤π.

Theorem 13.14 can be used to prove a version of the Cartan–Dieudonn´e theorem.

Theorem 13.15. LetE be a Euclidean space of dimensionn≥ 2. For every isometry f∈ O(E), ifp = dim(E(1, f)) = dim(Ker (f− id)), thenf is the composition ofn−p reflections, andn−p is minimal.

Proof. From Theorem 13.14 there arer subspacesF_1, . . . , Fr, each of dimension 2, such that

E =E(1, f)⊕E(−1, f)⊕F1⊕ · · · ⊕Fr,

and all the summands are pairwise orthogonal. Furthermore, the restriction ri off to each Fi is a rotationri =±id. Each 2D rotationri can be written a the compositionri =siæsi of two reflectionssi andsi about lines inFi (forming an angleθi/2). We can extendsi and si to hyperplane reflections inE by making them the identity onFi⊥ . Then,

s ræsræ · · · æs1æs1 agrees withf onF1⊕ · · · ⊕Fr and is the identity onE(1, f)⊕E(−1, f). IfE(−1, f)

has an orthonormal basis of eigenvectors (v_1, . . . , vq), lettingsj be the reflection about the hyperplane (vj)⊥, it is clear that

sqæ · · · æs1 agrees withf onE(−1, f) and is the identity onE(1, f)⊕F1⊕ · · · ⊕Fr. But then,

f =sqæ · · · æs1æsræsræ · · · æs1æs1, the composition of 2r +q =n−p reflections. If

f =stæ · · · æs_1, fort reflectionssi, it is clear that

t

F = E(1, si)⊆E(1, f),

i=1

where E(1, si) is the hyperplane defining the reflectionsi. By the Grassmann relation, if we intersectt≤n hyperplanes, the dimension of their intersection is at leastn−t. Thus, n−t≤p, that is,t≥n−p, andn−p is the smallest number of reflections composingf.

The theorems of this section and of the previous section can be immediately applied to matrices.

13.4 Normal and Other Special Matrices

First, we consider real matrices. Recall the following definitions.

Definition 13.3. Given a realm×n matrixA, the transposeA ofA is then×m matrix A = (ai j) defined such that

ai j =aj i for alli, j, 1≤i≤m, 1≤j≤n. A realn×n matrixA is

normal if• A A =A A,

•

symmetric if A =A,

•

skew-symmetric if A =−A,

•

orthogonal if

A A =A A =In. Recall from Proposition 9.12 that whenE is a Euclidean space and (e_1, . . .,en) is an orthonormal basis forE, ifA is the matrix of a linear mapf :E→E w.r.t. the basis (e_1, . . . , en), thenA is the matrix of the adjointf∗ off. Consequently, a normal linear map has a normal matrix, a self-adjoint linear map has a symmetric matrix, a skew-self-adjoint linear map has a skew-symmetric matrix, and an orthogonal linear map has an orthogonal matrix. Similarly, ifE andF are Euclidean spaces, (u_1, . . . , un) is an orthonormal basis for E, and (v_1, . . . , vm) is an orthonormal basis forF, if a linear mapf :E→F has the matrix A w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm), then its adjointf∗ has the matrixA w.r.t. the bases (v_1, . . . , vm) and (u_1, . . . , un).

Furthermore, if ( u_1, . . . , un) is another orthonormal basis forE andP is the change of basis matrix whose columns are the components of theui w.r.t. the basis (e_1, . . . , en), then P is orthogonal, and for any linear mapf :E→E, ifA is the matrix off w.r.t (e_1, . . . , en) andB is the matrix off w.r.t. (u_1, . . . , un), then

B =P AP.

As a consequence, Theorems 13.10 and 13.12–13.14 can be restated as follows.

Theorem 13.16. For every normal matrixA there is an orthogonal matrixP and a block diagonal matrixD such thatA =P D P , whereD is of the form

ëD_1 . . .ö

ì

÷

ì

D

2

. . .

÷

D =ì .... ÷

í . . ø

. . . Dp

such that each blockDj is either a one-dimensional matrix (i.e., a real scalar) or a twodimensional matrix of the form

Dj = λj −µj ,µj λj

whereλj, µj∈ R, withµj> 0.

Theorem 13.17. For every symmetric matrixA there is an orthogonal matrixP and a diagonal matrixD such thatA =P D P , whereD is of the form

ëλ1 . . . ö

ì λ_2 . . .÷

D

=

ì... ÷

.

÷ ,ì

í . . ø. . . λn

where λi∈ R. Theorem 13.18. For every skew-symmetric matrixA there is an orthogonal matrixP and a block diagonal matrixD such thatA =P D P , whereD is of the form

ëD_1 . . .ö

ì

÷

ì

D

2

. . .

÷

D =ì .... ÷

í . . ø. . . Dp

such that each blockDj is either 0 or a two-dimensional matrix of the form

Dj =0 −

µj

0 ,µj

whereµj∈ R, withµj> 0. In particular, the eigenvalues ofA are pure imaginary of the form±iµj, or 0.

Theorem 13.19. For every orthogonal matrixA there is an orthogonal matrixP and a block diagonal matrixD such thatA =P D P , whereD is of the form

ëD_1 . . .ö

ì

÷

ì

D

2

. . .

÷

D =ì .... ÷

í . . ø. . . Dp

such that each blockDj is either 1,−1, or a two-dimensional matrix of the form

D

j

=

cosθj − sinθj sinθj cosθj

where 0< θj< π. In particular, the eigenvalues ofA are of the form cosθj±isinθj, 1, or −1.

We now consider complex matrices.

Definition 13.4. Given a complexm×n matrixA, the transposeA ofA is then×m matrixA = ai j defined such that

ai j =aj i for alli, j, 1≤i≤m, 1≤j≤n. The conjugateA ofA is them×n matrixA = (bi j) defined such that

bi j =ai j

for alli, j, 1≤i≤m, 1≤j≤n. Given anm×n complex matrixA, the adjointA∗ ofA is the matrix defined such that

A∗ = (A ) = (A) . A complexn×n matrixA is

•

normal if AA∗ =A∗A,

•

Hermitian if A∗ =A,

•

skew-Hermitian if A∗ =−A,

•

unitary if AA∗ =A∗A =In.

Recall from Proposition 11.12 that when E is a Hermitian space and (e_1, . . .,en) is an orthonormal basis forE, ifA is the matrix of a linear mapf :E→E w.r.t. the basis (e_1, . . . , en), thenA∗ is the matrix of the adjointf∗ off. Consequently, a normal linear map has a normal matrix, a self-adjoint linear map has a Hermitian matrix, a skew-self-adjoint linear map has a skew-Hermitian matrix, and a unitary linear map has a unitary matrix. Similarly, ifE andF are Hermitian spaces, (u_1, . . . , un) is an orthonormal basis forE, and (v_1, . . . , vm) is an orthonormal basis forF, if a linear mapf :E→F has the matrixA w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm), then its adjointf∗ has the matrixA∗ w.r.t. the bases (v_1, . . . , vm) and (u_1, . . . , un).

Furthermore, if ( u_1, . . . , un) is another orthonormal basis forE andP is the change of basis matrix whose columns are the components of theui w.r.t. the basis (e_1, . . . , en), then P is unitary, and for any linear mapf :E→E, ifA is the matrix off w.r.t (e_1, . . . , en) and B is the matrix off w.r.t. (u_1, . . . , un), then

B =P∗AP.

Theorem 13.11 can be restated in terms of matrices as follows. We can also say a little more about eigenvalues (easy exercise left to the reader).

Theorem 13.20. For every complex normal matrixA there is a unitary matrixU and a diagonal matrixD such thatA =UDU∗. Furthermore, ifA is Hermitian, thenD is a real matrix; ifA is skew-Hermitian, then the entries inD are pure imaginary or null; and ifA is unitary, then the entries inD have absolute value 1.

We now have all the tools to present the important singular value decomposition (SVD) and the polar form of a matrix. However, we prefer to first illustrate how the material of this section can be used to discretize boundary value problems, and we give a brief introduction to the finite elements method.

13.5 Summary

The main concepts and results of this chapter are listed below:

•

Normal linear maps, self-adjoint linear maps, skew-self-adjoint linear maps, and orthogonal linear maps.

Properties of the eigenvalues and eigenvectors of a normal linear map.•

•

The complexification of a real vector space, of a linear map, and of a Euclidean inner product.

The eigenvalues of a self-adjoint map in a Hermitian space are real.•

• The eigenvalues of a self-adjoint map in a Euclidean space are real.

Every self-adjoint linear map on a Euclidean space has an orthonormal basis of eigen• vectors.

Every normal linear map on a Euclidean space can be block diagonalized (blocks of• size at most 2× 2) with respect to an orthonormal basis of eigenvectors.

Every normal linear map on a Hermitian space can be diagonalized with respect to an• orthonormal basis of eigenvectors.

The spectral theorems for self-adjoint, skew-self-adjoint, and orthogonal linear maps• (on a Euclidean space).

The spectral theorems for normal, symmetric, skew-symmetric, and orthogonal (real)• matrices.

The spectral theorems for normal, Hermitian, skew-Hermitian, and unitary (complex)• matrices.

Chapter 14 Bilinear Forms and Their Geometries

14.1 Bilinear Forms

In this chapter, we study the structure of a K-vector spaceE endowed with a nondegenerate bilinear formÕ:E×E→K (for any fieldK), which can be viewed as a kind of generalized inner product. Unlike the case of an inner product, there may be nonzero vectorsu∈E such thatÕ(u, u) = 0, so the mapu→Õ(u, u) can no longer be interpreted as a notion of square length (also,Õ(u, u) may not be real and positive!). However, the notion of orthogonality survives: we say thatu, v∈ E are orthogonal iffÕ(u, v) = 0. Under some additional conditions onÕ, it is then possible to splitE into orthogonal subspaces having some special properties. It turns out that the special cases whereÕ is symmetric (or Hermitian) or skewsymmetric (or skew-Hermitian) can be handled uniformly using a deep theorem due to Witt (the Witt decomposition theorem (1936)).

We begin with the very general situation of a bilinear form Õ:E×F→K, whereK is an arbitrary field, possibly of characteristric 2. Actually, even though at first glance this may appear to be an uncessary abstraction, it turns out that this situation arises in attempting to prove properties of a bilinear mapÕ:E×E→K, because it may be necessary to restrict Õ to different subspacesU andV ofE . This general approach was pioneered by Chevalley [20], E. Artin [2], and Bourbaki [11]. The third source was a major source of inspiration, and many proofs are taken from it. Other useful references include Snapper and Troyer [95], Berger [7], Jacobson [57], Grove [50], Taylor [104], and Berndt [9].

Definition 14.1. Given two vector spacesE andF over a fieldK, a mapÕ:E×F→ K is a bilinear form iff the following conditions hold: For allu, u_1, u2∈E, allv, v_1, v2∈F, for allλ µ∈K, we have

Õ (u_1 +u_2, v) =Õ(u_1, v) +Õ(u_2, v) Õ(u, v_1 +v_2) =Õ(u, v_1) +Õ(u, v_2) Õ(λu, v) =λÕ(u, v)

Õ(u, µv) =µÕ(u, v).

351 A bilinear form as in Definition 14.1 is sometimes called a pairing. The first two conditions imply thatÕ(0, v) =Õ(u,0) = 0 for allu∈E and allv∈F.

IfE =F, observe that

Õ(λu +µv, λu +µv) =λÕ(u, λu +µv) +µÕ(v, λu +µv) =λ^2Õ(u, u) +λµÕ(u, v) +λµÕ(v, u) +µ^2Õ(v, v).

If we letλ =µ = 1, we get

Õ(u +v, u +v) =Õ(u, u) +Õ(u, v) +Õ(v, u) +Õ(v, v).

IfÕ is symmetric, which means that

Õ(u, v) =Õ(v, u) for allu, v∈E,

then 2Õ(u, v) =Õ(u +v, u +v)−Õ(u, u)−Õ(v, v). The function Φ defined such that

Φ(u) =Õ(u, u) u∈E,

is called the quadratic form associated withÕ. If the fieldK is not of characteristic 2, then Õ is completely determined by its quadratic form Φ. The symmetric bilinear formÕ is called the polar form of Φ. This suggests the following definition.

Definition 14.2. A function Φ:E→K is a quadratic form onE if the following conditions hold:

(1) We have Φ(λu) =λ^2Φ(u), for allu∈E and allλ∈E.

(2) The mapÕ given byÕ (u, v) = Φ(u+v)−Φ(u)−Φ(v) is bilinear. Obviously, the map Õ is symmetric.

Since Φ(x +x) = Φ(2x) = 4Φ(x), we have

Õ (u, u) = 2Φ(u) u∈E.

If the field K is not of characteristic 2, thenÕ = 1Õ is the unique symmetric bilinear form_2

such that thatÕ(u, u) = Φ(u) for allu 1Õ is called the polar form

∈E. The bilinear formÕ =2

of Φ. In this case, there is a bijection between the set of bilinear forms onE and the set of quadratic forms onE.

IfK is a field of characteristic 2, thenÕ is alternating, which means that

Õ (u, u) = 0 for allu∈E. Thus, Φ cannot be recovered from the symmetric bilinear formÕ . However, there is some (nonsymmetric) bilinear formψ such that Φ(u) =ψ(u, u) for allu∈E. Thus, quadratic forms are more general than symmetric bilinear forms (except in characteristic = 2).

In general, ifK is a field of any characteristic, the identity

Õ(u +v, u +v) =Õ(u, u) +Õ(u, v) +Õ(v, u) +Õ(v, v) shows that ifÕ is alternating (that is,Õ(u, u) = 0 for allu∈E), then, Õ(v, u) =−Õ(u, v) for allu, v∈E; we say thatÕ is skew-symmetric. Conversely, if the fieldK is not of characteristic 2, then a skew-symmetric bilinear map is alternating, sinceÕ(u, u) =−Õ(u, u) impliesÕ(u, u) = 0.

An important consequence of bilinearity is that a pairing yields a linear map from E into F∗ and a linear map fromF intoE∗ (whereE∗ = HomK(E, K), the dual ofE, is the set of linear maps fromE toK, called linear forms).

Definition 14.3. Given a bilinear mapÕ:E×F→K, for everyu∈E, letlÕ(u) be the linear form inF∗ given by

lÕ(u)(y) =Õ(u, y) for ally∈F , and for everyv∈F, letrÕ(v) be the linear form inE∗ given by

rÕ(v)(x) =Õ(x, v) for allx∈E.

BecauseÕ is bilinear, the mapslÕ:E→F∗ andrÕ:F→E∗ are linear. Definition 14.4. A bilinear mapÕ:E×F→K is said to be nondegenerate iff the following conditions hold:

(1) For everyu∈E, ifÕ(u, v) = 0 for allv∈F, thenu = 0, and

(2) For everyv∈F, ifÕ(u, v) = 0 for allu∈E, thenv = 0.

The following proposition shows the importance of lÕ andrÕ.

Proposition 14.1. Given a bilinear mapÕ:E×F→K, the following properties hold: (a) The maplÕ is injective iff property (1) of Definition 14.4 holds.

(b) The maprÕ is injective iff property (2) of Definition 14.4 holds. (c) The bilinear formÕ is nondegenerate and ifflÕ andrÕ are injective.

(d) If the bilinear form Õ is nondegenerate and ifE andF have finite dimensions, then dim(E) = dim(F), andlÕ:E→F∗ andrÕ:F→E∗ are linear isomorphisms. Proof. (a) Assume that (1) of Definition 14.4 holds. IflÕ(u) = 0, thenlÕ(u) is the linear form whose value is 0 for ally; that is,

lÕ(u)(y) =Õ(u, y) = 0 for ally∈F ,

and by (1) of Definition 14.4, we must haveu = 0. Therefore,lÕ is injective. Conversely, if lÕ is injective, and if

lÕ(u)(y) =Õ(u, y) = 0 for ally∈F ,

thenlÕ(u) is the zero form, and by injectivity oflÕ, we getu = 0; that is, (1) of Definition 14.4 holds.

(b) The proof is obtained by swapping the arguments of Õ.

(c) This follows from (a) and (b).

(d) If E andF are finite dimensional, then dim(E) = dim(E∗) and dim(F) = dim(F∗). SinceÕ is nondegenerate,lÕ:E→F∗ andrÕ:F→E∗ are injective, so dim(E)≤ dim(F∗) = dim(F) and dim(F)≤ dim(E∗) = dim(E), which implies that

dim(E) = dim(F),

and thus,lÕ:E→F∗ andrÕ:F→E∗ are bijective. As a corollary of Proposition 14.1, we have the following characterization of a nondegenerate bilinear map. The proof is left as an exercise.

Proposition 14.2. Given a bilinear mapÕ:E×F→K, ifE andF have the same finite dimension, then the following properties are equivalent:

(1) The maplÕ is injective. (2) The maplÕ is surjective.

(3) The map rÕ is injective. (4) The maprÕ is surjective. (5) The bilinear formÕ is nondegenerate.

Observe that in terms of the canonical pairing betweenE∗ andE given by

f, u =f(u), f∈E∗, u∈E,

(and the canonical pairing betweenF∗ andF), we have

Õ (u, v) = lÕ(u), v = rÕ(v), u . Proposition 14.3. Given a bilinear mapÕ:E×F→ K, ifÕ is nondegenerate andE and F are finite-dimensional, then dim(E) = dim(F) =n, and for every basis (e_1, . . . , en) ofE, there is a basis (f_1, . . . , fn) ofF such thatÕ(ei, fj) =δij, for alli, j = 1, . . . , n.

Proof. SinceÕ is nondegenerate, by Proposition 14.1 we have dim(E) = dim(F) =n, and by Proposition 14.2, the linear maprÕ is bijective. Then, if (e∗, . . . , e∗n) is the dual basis (in_1

E∗) of the basis (e^1 (e∗) form a basis ofF,1, . . . , en), the vectors (f_1, . . . , fn) given byfi =r−

Õ

and we have

Õ(ei, fj) = rÕ(fj), ei = e∗, ej =δij, as claimed.

IfE =F andÕ is symmetric, then we have the following interesting result.

Theorem 14.4. Given any bilinear formÕ:E×E→K with dim(E) =n, ifÕ is symmetric andK does not have characteristic 2 , then there is a basis (e_1, . . . , en) ofE such that Õ(ei, ej) = 0, for alli =j.

Proof. We proceed by induction onn≥ 0, following a proof due to Chevalley. The base casen = 0 is trivial. For the induction step, assume thatn≥ 1 and that the induction hypothesis holds for all vector spaces of dimensionn− 1. IfÕ(u, v) = 0 for allu, v∈E, then the statement holds trivially. Otherwise, sinceK does not have characteristic 2, by a previous remark, there is some nonzero vectore1∈E such thatÕ(e_1, e_1) = 0. We claim that the set

H ={v∈E|Õ(e_1, v) = 0} has dimensionn− 1, and thate_1/H.

This is because

H = Ker (lÕ(e_1)),

where lÕ(e_1) is the linear form inE∗ determined bye_1. SinceÕ(e_1, e_1) = 0, we havee_1/∈H, the linear formlÕ(e_1) is not the zero form, and thus its kernel is a hyperplaneH (a subspace of dimensionn− 1). Since dim(H) =n− 1 ande_1/H, we have the direct sum

E =H⊕Ke_1.

By the induction hypothesis applied to H, we get a basis (e_2, . . . , en) of vectors inH such thatÕ(ei, ej) = 0, for alli =j with 2≤i, j≤n. SinceÕ(e_1, v) = 0 for allv∈H and since Õ is symmetric, we also haveÕ(v, e_1) = 0 for allv∈H, so we obtain a basis (e_1, . . . , en) of E such thatÕ(ei, ej) = 0, for alli =j.

IfE andF are finite-dimensional vector spaces and if (e_1, . . . , em) is a basis ofE and (f_1, . . . , fn) is a basis ofF then the bilinearity ofÕ yields

m n m n

Õ xiei, yjfj = xiÕ(ei, fj)yj.

i=1 j=1 i=1 j=1

This shows thatÕ is completely determined by them×n matrixM = (Õ(ei, ej)), and in matrix form, we have

Õ(x, y) =x My =y M x,

wherex andy are the column vectors associated with (x_1, . . . , xm)∈Km and (y_1, . . . , yn)∈Kn. We callM the matrix ofÕ with respect to the bases (e1, . . . , em) and (f1, . . . , fn).

Ifm = dim(E) = dim(F) =n, then it is easy to check thatÕ is nondegenerate iffM is invertible iff det(M) = 0.

As we will see later, most bilinear forms that we will encounter are equivalent to one whose matrix is of the following form:

1. In,−In. 2. Ifp +q =n, withp, q≥ 1, I

p,q

=

Ip 0 0 −Iq 3. Ifn = 2m, J

m.m

=

0 Im Im 0− 4. Ifn = 2m,

Am,m =Im.mJm.m =0 Im .Im 0

If we make changes of bases given by matrices P andQ, so thatx =P x andy =Qy , then the new matrix expressingÕ isP MQ. In particular, ifE =F and the same basis is used, then the new matrix isP MP. This shows that ifÕ is nondegenerate, then the determinant ofÕ is determined up to a square element.

Observe that if Õ is a symmetric bilinear form (E =F) and ifK does not have characteristic 2, then by Theorem 14.4, there is a basis ofE with respect to which the matrixM representingÕ is a diagonal matrix. IfK = R orK = C, this allows us to classify completely the symmetric bilinear forms. Recall that Φ(u) =Õ(u, u) for allu∈E.

Proposition 14.5. Given any bilinear formÕ:E×E→ K with dim(E) = n, ifÕ is symmetric andK does not have characteristic 2 , then there is a basis (e_1, . . . , en) ofE such thatn r

Φ xiei = λix^2,

i=1 i=1

for someλi∈ K− {0} and withr≤n. Furthermore, ifK = C, then there is a basis (

e

1

, . . . , e

n

)

ofE such that

n r

Φ xiei = x^2,

i=1 i=1

and ifK = R, then there is a basis (e_1, . . . , en) ofE such that

n p p+q

Φ xiei = x^2 x^2,

i=1 i=1− i=p+1

with 0≤p, q andp +q≤n.

Proof. The first statement is a direct consequence of Theorem 14.4. IfK = C, then every λi has a square rootµi, and if replaceei byei/µi, we obtained the desired form.

If K = R, then there are two cases:

1. Ifλi> 0, letµi be a positive square root ofλi and replaceei byei/µi.

2. Ifλi< 0, etµi be a positive square root of−λi and replaceei byei/µi.

In the nondegenerate case, the matrices corresponding to the complex and the real case are,In,−In, andIp,q. Observe that the second statement of Proposition 14.4 holds in any field in which every element has a square root. In the caseK = R, we can show that(p, q) only depends onÕ.

For any subspace U ofE, we say thatÕ is positive definite onU iffÕ(u, u)> 0 for all nonzerou∈U, and we say thatÕ is negative definite onU iffÕ(u, u)< 0 for all nonzero u∈U. Then, let

r = max{dim(U)|U⊆E, Õ is positive definite onU}

and let

s = max{dim(U)|U⊆E, Õ is negative definite onU} Proposition 14.6. (Sylvester’s inertia law) Given any symmetric bilinear formÕ:E×E→R with dim(E) =n, for any basis (e1, . . . , en) ofE such that

n p p+q

Φ xiei = x^2 x^2,

i=1 i=1− i=p+1

with 0≤p, q andp +q≤n, the integersp, q depend only onÕ; in fact,p =r andq =s, withr ands as defined above.

Proof. If we letU be the subspace spanned by (e_1, . . . , ep), thenÕ is positive definite on U, sor≥p. Similarly, if we letV be the subspace spanned by (ep+1, . . . , ep+q), thenÕ is negative definite onV , sos≥q.

Next, if W_1 is any subspace of maximum dimension such thatÕ is positive definite on W_1, and if we letV be the subspace spanned by (ep+1, . . . , en), thenÕ(u, u)≤ 0 onV , so W1∩V = (0), which implies that dim(W_1) + dim(V )≤n, and thus,r +n−p≤n; that is,r≤p. Similarly, ifW_2 is any subspace of maximum dimension such thatÕ is negative definite onW_2, and if we letU be the subspace spanned by (e_1, . . . , ep, ep+q+1, . . . , en), then Õ(u, u)≥ 0 onU , soW2∩U = (0), which implies thats +n−q≤n; that is,s≤q. Therefore,p =r andq =s, as claimed

These last two results can be generalized to ordered fields. For example, see Snapper and Troyer [95], Artin [2], and Bourbaki [11].

14.2 Sesquilinear Forms

In order to accomodate Hermitian forms, we assume that some involutive automorphism, λ→λ, of the fieldK is given. This automorphism ofK satisfies the following properties: (λ +µ) =λ +µ

(λµ) =λ µ

λ =λ.

If the automorphism λ→λ is the identity, then we are in the standard situation of a bilinear form. WhenK = C (the complex numbers), then we usually pick the automorphism of C to be conjugation; namely, the map

a +ib→a−ib.

Definition 14.5. Given two vector spacesE andF over a fieldK with an involutive automorphismλ→λ, a mapÕ:E×F→ K is a (right) sesquilinear form iff the following conditions hold: For allu, u_1, u2∈E, allv, v_1, v2∈F, for allλ µ∈K, we have

Õ (u_1 +u_2, v) =Õ(u_1, v) +Õ(u_2, v) Õ(u, v_1 +v_2) =Õ(u, v_1) +Õ(u, v_2) Õ(λu, v) =λÕ(u, v)

Õ(u, µv) =µÕ(u, v).

Again,Õ(0, v) =Õ(u,0) = 0. IfE =F, then we have Õ(λu +µv, λu +µv) =λÕ(u, λu +µv) +µÕ(v, λu +µv) =λλÕ(u, u) +λµÕ(u, v) +λµÕ(v, u) +µµÕ(v, v). If we letλ =µ = 1 and thenλ = 1, µ =−1, we get

Õ(u +v, u +v) =Õ(u, u) +Õ(u, v) +Õ(v, u) +Õ(v, v) Õ(u−v, u−v) =Õ(u, u)−Õ(u, v)−Õ(v, u) +Õ(v, v), so by subtraction, we get

2(Õ(u, v) +Õ(v, u)) =Õ(u +v, u +v)−Õ(u−v, u−v) foru, v∈E.

If we replacev byλv (withλ = 0), we get

2(λÕ(u, v) +λÕ(v, u)) =Õ(u +λv, u +λv)−Õ(u−λv, u−λv),

and by combining the above two equations, we get

2(λ−λ)Õ(u, v) =λÕ(u+v, u+v)−λÕ(u−v, u−v)−Õ(u+λv, u+λv) +Õ(u−λv, u−λv).

If the automorphism λ→λ is not the identity, then there is someλ∈K such thatλ−λ = 0, and ifK is not of characteristic 2, then we see that the sesquilinear formÕ is completely determined by its restriction to the diagonal (that is, the set of values{Õ(u, u)|u∈E}). In the special case whereK = C, we can pickλ =i, and we get

4Õ(u, v) =Õ(u +v, u +v)−Õ(u−v, u−v) +iÕ(u +λv, u +λv)−iÕ(u−λv, u−λv).

Remark: If the automorphismλ→λ is the identity, then in generalÕ is not determined by its value on the diagonal, unlessÕ is symmetric.

In the sesquilinear setting, it turns out that the following two cases are of interest: 1. We have Õ(v, u) =Õ(u, v), for allu, v∈E,

in which case we say thatÕ is Hermitian. In the special case whereK = C and the involutive automorphism is conjugation, we see thatÕ(u, u)∈ R, foru∈E. 2. We have Õ(v, u) =−Õ(u, v), for allu, v∈E, in which case we say thatÕ is skew-Hermitian.

We observed that in characteristic different from 2, a sesquilinear form is determined by its restriction to the diagonal. For Hermitian and skew-Hermitian forms, we have the following kind of converse.

Proposition 14.7. IfÕ is a nonzero Hermitian or skew-Hermitian form and ifÕ(u, u) = 0 for allu∈E, thenK is of characteristic 2 and the automorphismλ→λ is the identity. Proof. We give the prooof in the Hermitian case, the skew-Hermitian case being left as an exercise. Assume thatÕ is alternating. From the identity

Õ(u +v, u +v) =Õ(u, u) +Õ(u, v) +Õ(u, v) +Õ(v, v), we get Õ(u, v) =−Õ(u, v) for allu, v∈E.

SinceÕ is not the zero form, there exist some nonzero vectorsu, v∈E such thatÕ(u, v) = 1. For anyλ∈K, we have

λÕ(u, v) =Õ(λu, v) =−Õ(λu, v) =−λ Õ(u, v),

and sinceÕ(u, v) = 1, we get

λ =−λ for allλ∈K.

For λ = 1, we get 1 =−1, which means thatK has characterictic 2. But then λ =−λ =λ for allλ∈K,

so the automorphismλ→λ is the identity.

The definition of the linear mapslÕ andrÕ requires a small twist due to the automorphism λ→λ.

Definition 14.6. Given a vector spaceE over a fieldK with an involutive automorphism λ→λ, we define theK-vector spaceE asE with its abelian group structure, but with scalar multiplication given by

( λ, u)→λu.

Given twoK-vector spacesE andF, a semilinear mapf :E→F is a function, such that for allu, v∈E, for allλ∈K, we have

f(u +v) =f(u) +f(v)

f(λu) =λf(u).

Because λ =λ, observe that a functionf :E→F is semilinear iff it is a linear map f :E→F. TheK-vector spacesE andE are isomorphic, since any basis (ei)i∈I ofE is also a basis ofE.

The mapslÕ andrÕ are defined as follows:

For everyu∈E, letlÕ(u) be the linear form inF∗ defined so that

lÕ(u)(y) =Õ(u, y) for ally∈F ,

and for everyv∈F, letrÕ(v) be the linear form inE∗ defined so that

rÕ(v)(x) =Õ(x, v) for allx∈E.

The reader should check that because we used Õ(u, y) in the definition oflÕ(u)(y), the functionlÕ(u) is indeed a linear form inF∗. It is also easy to check thatlÕ is a linear maplÕ:E→F∗, and thatrÕ is a linear maprÕ:F→E∗ (equivalently,lÕ:E→F∗ and rÕ:F→E∗ are semilinear).

The notion of a nondegenerate sesquilinear form is identical to the notion for bilinear forms. For the convenience of the reader, we repeat the definition.

Definition 14.7. A sesquilinear mapÕ:E×F→K is said to be nondegenerate iff the following conditions hold:

(1) For everyu∈E, ifÕ(u, v) = 0 for allv∈F, thenu = 0, and

(2) For everyv∈F, ifÕ(u, v) = 0 for allu∈E, thenv = 0.

Proposition 14.1 translates into the following proposition. The proof is left as an exercise. Proposition 14.8. Given a sesquilinear mapÕ:E×F→K, the following properties hold: (a) The maplÕ is injective iff property (1) of Definition 14.7 holds. (b) The maprÕ is injective iff property (2) of Definition 14.7 holds. (c) The sesquilinear formÕ is nondegenerate and ifflÕ andrÕ are injective.

(d) If the sesquillinear formÕ is nondegenerate and ifE andF have finite dimensions, then dim(E) = dim(F), andlÕ:E→F∗ andrÕ:F→E∗ are linear isomorphisms.

Propositions 14.2 and 14.3 also generalize to sesquilinear forms. We also have the following version of Theorem 14.4, whose proof is left as an exercise.

Theorem 14.9. Given any sesquilinear formÕ:E×E→K with dim(E) =n, ifÕ is Hermitian andK does not have characteristic 2 , then there is a basis (e_1, . . . , en) ofE such thatÕ(ei, ej) = 0, for alli =j.

As in Section 14.1, ifE andF are finite-dimensional vector spaces and if (e_1, . . . , em) is a basis ofE and (f_1, . . . , fn) is a basis ofF then the sesquilinearity ofÕ yields

m n m n

Õ xiei, yjfj = xiÕ(ei, fj)yj.

i=1 j=1 i=1 j=1

This shows thatÕ is completely determined by them×n matrixM = (Õ(ei, ej)), and in matrix form, we have

Õ(x, y) =x My =y∗M x,

where x andy are the column vectors associated with (x_1, . . . , xm)∈Km and (y_1, . . . , yn)∈ Kn, andy∗ =y . We callM the matrix ofÕ with respect to the bases (e_1, . . . , em) and (f_1, . . . , fn).

Ifm = dim(E) = dim(F) =n, thenÕ is nondegenerate iffM is invertible iff det(M) = 0.

Observe that if Õ is a Hermitian form (E =F) and ifK does not have characteristic 2, then by Theorem 14.9, there is a basis ofE with respect to which the matrixM representing Õ is a diagonal matrix. IfK = C, then these entries are real, and this allows us to classify completely the Hermitian forms.

Proposition 14.10. Given any Hermitian formÕ:E×E→ C with dim(E) =n, there is a basis (e_1, . . . , en) ofE such that

n p p+q

Φ xiei = x^2 x^2,

i=1 i=1− i=p+1

with 0≤p, q andp +q≤n. The proof of Proposition 14.10 is the same as the real case of Proposition 14.5. Sylvester’s inertia law (Proposition 14.6) also holds for Hermitian forms:p andq only depend onÕ.

14.3 Orthogonality

In this section, we assume that we are dealing with a sesquilinear form Õ:E×F→K. We allow the automorphismλ→λ to be the identity, in which caseÕ is a bilinear form. This way, we can deal with properties shared by bilinear forms and sesquilinear forms in a uniform fashion. Orthogonality is such a property.

Definition 14.8. Given a sesquilinear formÕ:E×F→K, we say that two vectorsu∈E andv∈F are orthogonal (or conjugate) ifÕ(u, v) = 0. Two subsetsE⊆E andF⊆F are orthogonal ifÕ(u, v) = 0 for allu∈E and allv∈F . Given a subspaceU ofE, the right orthogonal space ofU, denotedU⊥, is the subspace ofF given by

U⊥ ={v∈F|Õ(u, v) = 0 for allu∈U},

and given a subspaceV ofF, the left orthogonal space ofV , denotedV⊥, is the subspace of E given by

V⊥ ={u∈E|Õ(u, v) = 0 for allv∈V}.

When E and F are distinct, there is little chance of confusing the right orthogonal subspaceU⊥ of a subspaceU ofE and the left orthogonal subspaceV⊥ of a subspaceV of F. However, ifE =F, thenÕ(u, v) = 0 does not necessarily imply thatÕ(v, u) = 0, that is, orthogonality is not necessarily symmetric. Thus, if bothU andV are subsets ofE, there is a notational ambiguity ifU =V . In this case, we may writeU⊥r for the right orthogonal andU⊥l for the left orthogonal.

The above discussion brings up the following point: When is orthogonality symmetric? IfÕ is bilinear, it is shown in E. Artin [2] (and in Jacobson [57]) that orthogonality is symmetric iff eitherÕ is symmetric orÕ is alternating (Õ(u, u) = 0 for allu∈E). IfÕ is sesquilinear, the answer is more complicated. In addition to the previous two cases, there is a third possibility:

Õ(u, v) = Õ(v, u) for allu, v∈E, where is some nonzero element inK. We say thatÕ is -Hermitian. Observe that

Õ(u, u) = Õ(u, u),

so ifÕ is not alternating, thenÕ(u, u) = 0 for someu, and we must have = 1. The most common cases are

1. = 1, in which caseÕ is Hermitian, and

2. =−1, in which caseÕ is skew-Hermitian.

IfÕ is alternating andK is not of characteristic 2, then the automorphismλ→λ must be the identity ifÕ is nonzero. If so,Õ is skew-symmetric, so =−1.

In summary, ifÕ is either symmetric, alternating, or -Hermitian, then orthogonality is symmetric, and it makes sense to talk about the orthogonal subspaceU⊥ ofU. Observe that ifÕ is -Hermitian, then

rÕ =lÕ.

This is because

lÕ(u)(y) =Õ(u, y) rÕ(u)(y) =Õ(y, u) = Õ(u, y),

sorÕ =lÕ.

IfE andF are finite-dimensional with bases (e_1, . . . , em) and (f_1, . . . , fn), and ifÕ is represented by them×n matrixM, thenÕ is -Hermitian iff

M =M∗,

where M∗ = (M) (as usual). This captures the following kinds of familiar matrices:

1. Symmetric matrices ( = 1)

2. Skew-symmetric matrices ( =−1)

3. Hermitian matrices ( = 1)

4. Skew-Hermitian matrices ( =−1).

Going back to a sesquilinear formÕ:E×F→K, for any subspaceU ofE, it is easy to check that

U⊆ (U⊥)⊥, and that for any subspaceV ofF, we have

V⊆ (V⊥)⊥.

For simplicity of notation, we write U⊥⊥ instead of (U⊥)⊥ (andV⊥⊥ instead of (V⊥)⊥). Given any two subspacesU_1 andU_2 ofE, ifU1⊆U_2, thenU2⊥ U1⊥ (and similarly for any⊆

two subspacesV1⊆V_2 ofF). As a consequence, it is easy to show that

U⊥ =U⊥⊥⊥, V⊥ =V⊥⊥⊥.

Observe thatÕ is nondegenerate iffE⊥ ={0} andF⊥ ={0}. Furthermore, since

Õ(u +x, v) =Õ(u, v)

Õ(u, v +y) =Õ(u, v)

for anyx∈ F⊥ and anyy∈ E⊥, we see that we obtain by passing to the quotient a sesquilinear form

[ Õ]: (E/F⊥)× (F/E⊥)→K

which is nondegenerate.

Proposition 14.11. For any sesquilinear formÕ:E×F→K, the spaceE/F⊥ is finitedimensional iff the spaceF/E⊥ is finite-dimensional; if so, dim(E/F⊥) = dim(F/E⊥).

Proof. Since the sesquilinear form [Õ]: (E/F⊥)× (F/E⊥)→K is nondegenerate, the maps l[Õ]: (E/F⊥)→ (F/E⊥)∗ andr[Õ]: (F/E⊥)→ (E/F⊥)∗ are injective. If dim(E/F⊥) = m, then dim(E/F⊥) = dim((E/F⊥)∗), so by injectivity ofr[Õ], we have dim(F/E⊥) = dim((F/E⊥))≤m. A similar reasoning using the injectivity ofl[Õ] applies if dim(F/E⊥) =n, and we get dim(E/F⊥) = dim((E/F⊥))≤ n. Therefore, dim(E/F⊥) = m is finite iff dim(F/E⊥) =n is finite, in which casem =n.

If U is a subspace of a spaceE, recall that the codimension ofU is the dimension of E/U, which is also equal to the dimension of any subspaceV such thatE is a direct sum of U andV (E =U⊕V ).

Proposition 14.11 implies the following useful fact.

Proposition 14.12. LetÕ:E×F→ K be any nondegenerate sesquilinear form. A subspace U ofE has finite dimension iffU⊥ has finite codimension inF. If dim(U) is finite, then codim(U⊥) = dim(U), andU⊥⊥ =U.

Proof. SinceÕ is nondegenerateE⊥ ={0} andF⊥ ={0}, so the first two statements follow from proposition 14.11 applied to the restriction ofÕ toU×F. SinceU⊥ andU⊥⊥ are orthogonal, and since codim(U⊥) if finite, dim(U⊥⊥) is finite and we have dim(U⊥⊥) = codim(U⊥) = dim(U). SinceU⊆U⊥⊥, we must haveU =U⊥⊥.

Proposition 14.13. LetÕ:E×F→K be any sesquilinear form. Given any two subspaces U andV ofE, we have

(U +V )⊥ =U⊥∩V⊥.

Furthermore, ifÕ is nondegenerate and ifU andV are finite-dimensional, then

(U∩V )⊥ =U⊥ +V⊥.

Proof. Ifw∈ (U +V )⊥, thenÕ(u +v, w) = 0 for allu∈U and allv∈ V . In particular, withv = 0, we haveÕ(u, w) = 0 for allu∈U, and withu = 0, we haveÕ(v, w) = 0 for all v∈V , sow∈U⊥∩V⊥. Conversely, ifw∈U⊥∩V⊥, thenÕ(u, w) = 0 for allu∈U and Õ(v, w) = 0 for allv∈V . By bilinearity,Õ(u +v, w) =Õ(u, w) +Õ(v, w) = 0, which shows thatw∈ (U +V )⊥. Therefore, the first identity holds.

Now, assume that Õ is nondegenerate and thatU andV are finite-dimensional, and let W =U⊥ +V⊥. Using the equation that we just established and the fact thatU andV are finite-dimensional, by Proposition 14.12, we get

W⊥ =U⊥⊥∩V⊥⊥ =U∩V.

We can apply Proposition 14.11 to the restriction ofÕ toU×W (sinceU⊥⊆ W and W⊥⊆U), and we get

dim(U/W⊥) = dim(U/(U∩V )) = dim(W/U⊥) = codim(U⊥)− codim(W),

and since codim(U⊥) = dim(U), we deduce that

dim(U∩V ) = codim(W).

However, by Proposition 14.12, we have dim( U∩V ) = codim((U∩V )⊥), so codim(W) = codim((U∩V )⊥), and sinceW⊆W⊥⊥ = (U∩V )⊥, we must haveW = (U∩V )⊥, as claimed.

In view of Proposition 14.11, we can make the following definition.

Definition 14.9. LetÕ:E×F→K be any sesquilinear form. IfE/F⊥ andF/E⊥ are finite-dimensional, then their common dimension is called the rank of the formÕ. IfE/F⊥ andF/E⊥ have infinite dimension, we say thatÕ has infinite rank.

Not surprisingly, the rank ofÕ is related to the ranks oflÕ andrÕ.

Proposition 14.14. LetÕ:E×F→K be any sesquilinear form. IfÕ has finite rankr, thenlÕ andrÕ have the same rank, which is equal tor.

Proof. Because for everyu∈E,

lÕ(u)(y) =Õ(u, y) for ally∈F ,

and for everyv∈F,

rÕ(v)(x) =Õ(x, v) for allx∈E,

it is clear that the kernel of lÕ:E→F∗ is equal toF⊥ and that, the kernel ofrÕ:F→E∗ is equal toE⊥. Therefore, rank(lÕ) = dim(ImlÕ) = dim(E/F⊥) =r, and similarly rank(lÕ) = dim(F/E⊥) =r.

Remark: If the sesquilinear formÕ is represented by the matrixm×n matrixM with respect to the bases (e_1, . . . , em) inE and (f_1, . . . , fn) inF, it can be shown that the matrix representinglÕ with respect to the bases (e_1, . . . , em) and (f∗, . . . , f∗) isM∗, and that the_1

matrix representingrÕ with respect to the bases (f_1, . . . , fn) and (e∗, . . . , e∗m) isM. It follows

1

that the rank ofÕ is equal to the rank ofM.

14.4 Adjoint of a Linear Map

Let E_1 andE_2 be twoK-vector spaces, and letÕ_1:E1×E1→K be a sesquilinear form onE_1 andÕ_2:E2×E2→K be a sesquilinear form onE_2. It is also possible to deal with the more general situation where we have four vector spacesE_1, F_1, E_2, F_2 and two sesquilinear forms Õ_1:E1×F1→K andÕ_2:E2×F2→K, but we will leave this generalization as an exercise. We also assume thatlÕ1 andrÕ1 are bijective, which implies that thatÕ_1 is nondegenerate. This is automatic if the spaceE_1 is finite dimensional andÕ_1 is nondegenerate.

Given any linear mapf :E1→E_2, for any fixedu∈E_2, we can consider the linear form inE∗ given by

1

x→Õ_2(f(x), u), x∈E_1.

SincerÕ1:E1→E∗ is bijective, there is a uniquey∈E_1 (because the vector spacesE_1 and_1

E_1 only differ by scalar multiplication), so that

Õ_2(f(x), u) =Õ_1(x, y), for allx∈E_1.

If we denote this uniquey∈E_1 byf∗l(u), then we have

Õ_2(f(x), u) =Õ_1(x, f∗l(u)), for allx∈E_1, and allu∈E_2. Thus, we get a functionf∗l:E2→E_1. We claim that this function is a linear map. For any v_1, v2∈E_2, we have

Õ_2(f(x), v_1 +v_2) =Õ_2(f(x), v_1) +Õ_2(f(x), v_2) =Õ_1(x, f∗l(v_1)) +Õ_1(x, f∗l(v_2)) =Õ_1(x, f∗l(v_1) +f∗l(v_2)) =Õ_1(x, f∗l(v_1 +v_2)),

14.4. ADJOINT OF A LINEAR MAP 367

for allx∈E_1. SincerÕ1 is injective, we conclude that

f∗l(v_1 +v_2) =f∗l(v_1) +f∗l(v_2).

For anyλ∈K, we have

Õ_2(f(x), λv) =λÕ_2(f(x), v) =λÕ_1(x, f∗l(v)) =Õ_1(x, λf∗l(v)) =Õ_1(x, f∗l(λv)),

for allx∈E_1. SincerÕ1 is injective, we conclude that

f∗l(λv) =λf∗l(v).

Therefore,f∗l is linear. We call it the left adjoint off.

Now, for any fixedu∈E_2, we can consider the linear form inE∗ given by_1

x→Õ_2(u, f(x)) x∈E_1.

SincelÕ1:E1→E∗ is bijective, there is a uniquey∈E_1 so that_1

Õ_2(u, f(x)) =Õ_1(y, x), for allx∈E_1.

If we denote this uniquey∈E_1 byf∗r(u), then we have

Õ_2(u, f(x)) =Õ_1(f∗r(u), x), for allx∈E_1, and allu∈E_2.

Thus, we get a functionf∗r :E2→E_1. As in the previous situation, it easy to check that f∗r is linear. We call it the right adjoint off. In summary, we make the following definition.

Definition 14.10. LetE_1 andE_2 be twoK-vector spaces, and letÕ_1:E1×E1→K and Õ_2:E2×E2→K be two sesquilinear forms. Assume thatlÕ1 andrÕ1 are bijective, so thatÕ_1 is nondegnerate. For every linear mapf :E1→E_2, there exist unique linear maps f∗l:E2→E_1 andf∗r :E2→E_1, such that

Õ_2(f(x), u) =Õ_1(x, f∗l(u)), for allx∈E_1, and allu∈E_2 Õ_2(u, f(x)) =Õ_1(f∗r(u), x), for allx∈E_1, and allu∈E_2.

The mapf∗l is called the left adjoint off, and the mapf∗r is called the right adjoint off.

If E_1 andE_2 are finite-dimensional with bases (e_1, . . . , em) and (f_1, . . . , fn), then we can work out the matricesA∗l andA∗r corresponding to the left adjointf∗l and the right adjoint f∗r off. Assuming thatf is represented by then× m matrixA,Õ_1 is represented by the m×m matrixM_1, andÕ_2 is represented by then×n matrixM_2, we find that

A∗l = (M_1)−1A∗M_2 A∗r = (M_1 )−1A∗M_2.

IfÕ_1 andÕ_2 are symmetric bilinear forms, thenf∗l = f∗r. This also holds ifÕ is

-Hermitian. Indeed, since

Õ_2(u, f(x)) =Õ_1(f∗r(u), x), we get Õ_2(f(x), u) = Õ_1(x, f∗r(u)), and sinceλ→λ is an involution, we get

Õ_2(f(x), u) =Õ_1(x, f∗r(u)).

Since we also have Õ_2(f(x), u) =Õ_1(x, f∗l(u)), we obtain Õ_1(x, f∗r(u)) =Õ_1(x, f∗l(u)) for allx∈E_1, and allu∈E_2, and sinceÕ_1 is nondegenerate, we conclude thatf∗l =f∗r. Wheneverf∗l =f∗r, we use the simpler notationf∗.

Iff :E1→E_2 andg:E1→E_2 are two linear maps, we have the following properties:

( f +g)∗l =f∗l +g∗l id∗l = id

(λf)∗l =λf∗l,

and similarly for right adjoints. IfE_3 is another space,Õ_3 is a sesquilinear form onE_3, and iflÕ2 andrÕ2 are bijective, then for any linear mapsf :E1→E_2 andg:E2→E_3, we have

(gæf)∗l =f∗l g∗l,æ

and similarly for right adjoints. Furthermore, ifE_1 =E_2 andÕ:E×E→K is -Hermitian, for any linear mapf :E→E (recall that in this casef∗l =f∗r =f∗). we have f∗∗ = f.

14.5 Isometries Associated with Sesquilinear Forms

The notion of adjoint is a good tool to investigate the notion of isometry between spaces equipped with sesquilinear forms. First, we define metric maps and isometries.

Definition 14.11. If (E_1, Õ_1) and (E_2, Õ_2) are two pairs of spaces and sesquilinear maps Õ_1:E1×E2→K andÕ_2:E2×E2→K, a metric map from (E_1, Õ_1) to (E_2, Õ_2) is a linear mapf :E1→E2 such that

Õ_1(u, v) =Õ_2(f(u), f(v)) for allu, v∈E_1.

We say thatÕ_1 andÕ_2 are equivalent iff there is a metric mapf :E1→E_2 which is bijective. Such a metric map is called an isometry.

The problem of classifying sesquilinear forms up to equivalence is an important but very difficult problem. Solving this problem depends intimately on properties of the fieldK, and a complete answer is only known in a few cases. The problem is easily solved forK = R, K = C. It is also solved for finite fields and forK = Q (the rationals), but the solution is surprisingly involved!

It is hard to say anything interesting if Õ_1 is degenerate and if the linear mapf does not have adjoints. The next few propositions make use of natural conditions onÕ_1 that yield a useful criterion for being a metric map.

Proposition 14.15. With the same assumptions as in Definition 14.10, iff :E1→E_2 is a bijective linear map, then we have

Õ_1(x, y) =Õ_2(f(x), f(y)) for allx, y∈E_1 iff f−1 =f∗l =f∗r.

Proof. We have Õ_1(x, y) =Õ_2(f(x), f(y)) iff

Õ_1(x, y) =Õ_2(f(x), f(y)) =Õ_1(x, f∗l(f(y)) iff

Õ_1(x,(id−f∗l f)(y)) = 0 for all∈E_1 and ally∈E_2.æ

SinceÕ_1 is nondegenerate, we must have

f∗l f = id,æ

which implies thatf−1 =f∗l. similarly,

Õ_1(x, y) =Õ_2(f(x), f(y)) iff Õ_1(x, y) =Õ_2(f(x), f(y)) =Õ_1(f∗r(f(x)), y) iff

Õ_1((id−f∗r f)(x), y) = 0 for all∈E_1 and ally∈E_2.æ

Since Õ_1 is nondegenerate, we must have

f∗r f = id,æ

which implies thatf−1 = f∗r. Therefore, f−1 = f∗l = f∗r. For the converse, do the computations in reverse.

As a corollary, we get the following important proposition.

Proposition 14.16. IfÕ:E×E→K is a sesquilinear map, and iflÕ andrÕ are bijective, for every bijective linear mapf :E→E, then we have

Õ(f(x), f(y)) =Õ(x, y) for allx, y∈E iff f−1 =f∗l =f∗r.

We also have the following facts.

Proposition 14.17. (1) IfÕ:E×E→K is a sesquilinear map and iflÕ is injective, then for every linear mapf :E→E, if

Õ(f(x), f(y)) =Õ(x, y) for allx, y∈E, (∗) thenf is injective.

(2) IfE is finite-dimensional and ifÕ is nondegenerate, then the linear mapsf :E→E satisfying (∗) form a group. The inverse off is given byf−1 =f∗.

Proof. (1) Iff(x) = 0, then

Õ(x, y) =Õ(f(x), f(y)) =Õ(0, f(y)) = 0 for ally∈E.

SincelÕ is injective, we must havex = 0, and thusf is injective.

(2) IfE is finite-dimensional, since a linear map satisfying (∗) is injective, it is a bijection. By Proposition 14.16, we havef−1 =f∗. We also have

Õ(f(x), f(y)) =Õ((f∗æf)(x), y) =Õ(x, y) =Õ((fæf∗)(x), y) =Õ(f∗(x), f∗(y)), which shows thatf∗ satisfies (∗). IfÕ(f(x), f(y)) =Õ(x, y) for allx, y∈E andÕ(g(x), g(y)) =Õ(x, y) for allx, y∈E, then we have

Õ((gæf)(x),(gæf)(y)) =Õ(f(x), f(y)) =Õ(x, y) for allx, y∈E. Obviously, the identity map idE satisfies (∗). Therefore, the set of linear maps satisfying (∗) is a group.

The above considerations motivate the following definition.

Definition 14.12. LetÕ:E×E→K be a sesquilinear map, and assume thatE is finitedimensional and thatÕ is nondegenerate. A linear mapf :E→E is an isometry ofE (with respect toÕ) iff

Õ(f(x), f(y)) =Õ(x, y) for allx, y∈E. The set of all isometries ofE is a group denoted by Isom(Õ).

If Õ is symmetric, then the group Isom(Õ) is denoted O(Õ) and called the orthogonal group ofÕ. IfÕ is alternating, then the group Isom(Õ) is denoted Sp(Õ) and called the symplectic group ofÕ. IfÕ is -Hermitian, then the group Isom(Õ) is denoted U (Õ) and called the -unitary group ofÕ. When = 1, we drop and just say unitary group.

If (e_1, . . . , en) is a basis ofE,Õ is the represented by then×n matrixM, andf is represented by then×n matrixA, then we find thatf∈ Isom(Õ) iff

A∗M A =M iff A MA =M,

andA−1 is given byA−1 = (M )−1A∗M = (M)−1A∗M.

More specifically, we define the following groups, using the matricesIp,q, Jm.m andAm.m defined at the end of Section 14.1.

(1) K = R. We have

O(n) ={A∈ Matn(R)|A A =In}

O(p, q) ={A∈ Matp+q(R)|A Ip,qA =Ip,q}Sp(2n,R) ={A∈ Mat2n(R)|A Jn,nA =Jn,n}SO(n) ={A∈ Matn(R)|A A =In, det(A) = 1}SO(p, q) ={A∈ Matp+q(R)|A Ip,qA =Ip,q, det(A) = 1}. The group O(n) is the orthogonal group, Sp(2n,R) is the real symplectic group, and SO(n) is the special orthogonal group. We can define the group

A∈ Mat2n(R)|A An,nA =An,n},

but it is isomorphic to

O

{

(n, n).

(2) K = C. We have

U(n) ={A∈ Matn(C)|A∗A =In}

U(p, q) ={A∈ Matp+q(C)|A∗Ip,qA =Ip,q}Sp(2n,C) ={A∈ Mat2n(C)|A∗Jn,nA =Jn,n}

SU(n) ={A∈ Matn(C)|A∗A =In, det(A) = 1}SU(p, q) ={A∈ Matp+q(C)|A∗Ip,qA =Ip,q, det(A) = 1}.

The group U(n) is the unitary group, Sp(2n,C) is the complex symplectic group, and SU(n) is the special unitary group.

It can be shown that ifA∈ Sp(2n,R) or ifA∈ Sp(2n,C), then det(A) = 1.

14.6 Totally Isotropic Subspaces. Witt Decomposition

In this section, we deal with -Hermitian forms,Õ:E×E→K. In general,E may have subspacesU such thatU∩U⊥ = (0), or worse, such thatU⊆U⊥ (that is,Õ is zero onU). We will see that such subspaces play a crucial in the decomposition ofE into orthogonal subspaces.

Definition 14.13. Given an -Hermitian formsÕ:E×E→K, a nonzero vectoru∈E is said to be isotropic ifÕ(u, u) = 0. It is convenient to consider 0 to be isotropic. Given any subspaceU ofE, the subspace rad(U) =U∩U⊥ is called the radical ofU. We say that

(i) U is degenerate if rad(U) = (0) (equivalently if there is some nonzero vectoru∈U such thatx∈U⊥). Otherwise, we say thatU is nondegenerate.

(ii) U is totally isotropic ifU⊆U⊥ (equivalently if the restriction ofÕ toU is zero).

By definition, the trivial subspace U = (0) (={0}) is nondegenerate. Observe that a subspaceU is nondegenerate iff the restriction ofÕ toU is nondegenerate. A degenerate subspace is sometimes called an isotropic subspace. Other authors say that a subspaceU is isotropic if it contains some (nonzero) isotropic vector. A subspace which has no nonzero isotropic vector is often called anisotropic. The space of all isotropic vectors is a cone often called the light cone (a terminology coming from the theory of relativity). This is not to be confused with the cone of silence (from Get Smart)! It should also be noted that some authors (such as Serre) use the term isotropic instead of totally isotropic. The apparent lack of standard terminology is almost as bad as in graph theory!

It is clear that any direct sum of pairwise orthogonal totally isotropic subspaces is totally isotropic. Thus, every totally isotropic subspace is contained in some maximal totally isotropic subspace.

First, let us show that in order to sudy an -Hermitian form on a spaceE, it suffices to restrict our attention to nondegenerate forms.

Proposition 14.18. Given an -Hermitian formÕ:E×E→K onE, we have: (a) IfU andV are any two orthogonal subspaces ofE, then

rad(U +V ) = rad(U) + rad(V ).

(b) rad(rad(E)) = rad(E). (c) IfU is any subspace supplementary to rad(E), so that

E = rad(E)⊕U,

thenU is nondegenerate, and rad(E) andU are orthogonal. Proof. (a) IfU andV are orthogonal, thenU⊆V⊥ andV⊆U⊥. We get

rad( U +V ) = (U +V )∩ (U +V )⊥

= (U +V )∩U⊥∩V⊥

=U∩U⊥∩V⊥ +V∩U⊥∩V⊥ =U∩U⊥ +V∩V⊥

= rad(U) + rad(V ).

(b) By definition, rad(E) =E⊥, and obviouslyE =E⊥⊥, so we get

rad(rad(E)) =E⊥∩E⊥⊥ =E⊥∩E =E⊥ = rad(E).

(c) IfE = rad(E)⊕U, by definition of rad(E), the subspaces rad(E) andU are orthogonal. From (a) and (b), we get

rad(E) = rad(E) + rad(U).

Since rad(U) =U∩U⊥⊆U and since rad(E)⊕U is a direct sum, we have a direct sum

rad(E) = rad(E)⊕ rad(U),

which implies that rad(U) = (0); that is,U is nondegenerate.

Proposition 14.18(c) shows that the restriction of Õ to any supplementU of rad(E) is nondegenerate andÕ is zero on rad(U), so we may restrict our attention to nondegenerate forms.

The following is also a key result.

Proposition 14.19. Given an -Hermitian formÕ:E×E→K onE, ifU is a finitedimensional nondegenerate subspace ofE, thenE =U⊕U⊥.

Proof. By hypothesis, the restrictionÕU ofÕ toU is nondegenerate, so the semilinear map rÕU :U→U∗ is injective. SinceU is finite-dimensional,rÕU is actually bijective, so for every v∈E, if we consider the linear form inU∗ given byu→Õ(u, v) (u∈U), there is a unique v0∈U such that

Õ(u, v_0) =Õ(u, v) for allu∈U;

that is,Õ(u, v−v_0) = 0 for allu∈U, sov−v0∈U⊥. It follows thatv =v_0 +v−v_0, with v0∈U andv0−v∈U⊥, and since U is nondegenerateU∩U⊥ = (0), andE =U⊕U⊥. As a corollary of Proposition 14.19, we get the following result.

Proposition 14.20. Given an -Hermitian formÕ:E×E→K onE, ifÕ is nondegenerate and ifU is a finite-dimensional subspace ofE, then rad(U) = rad(U⊥), and the following conditions are equivalent:

(i) U is nondegenerate.

(ii) U⊥ is nondegenerate.

(iii) E =U⊕U⊥.

Proof. By definition, rad(U⊥) =U⊥ ∩U⊥⊥, and sinceÕ is nondegenerate andU is finitedimensional,U⊥⊥ =U, so rad(U⊥) =U⊥∩U⊥⊥ =U∩U⊥ = rad(U).

By Proposition 14.19, (i) implies (iii). If E =U⊕ U⊥, then rad(U) =U∩U⊥ = (0), soU is nondegenerate and (iii) implies (i). Since rad(U⊥) = rad(U), (iii) also implies (ii). Now, ifU⊥ is nondegenerate, we haveU⊥∩U⊥⊥ = (0), and sinceU⊆U⊥⊥, we get

U∩U⊥⊆U⊥⊥∩U⊥ = (0),

which shows thatU is nondegenerate, proving the implication (ii) =⇒ (i).

IfE is finite-dimensional, we have the following results.

Proposition 14.21. Given an -Hermitian formÕ:E×E→ K on a finite-dimensional spaceE, ifÕ is nondegenerate, then for every subspaceU ofE we have

(i) dim(U) + dim(U⊥) = dim(E).

(ii) U⊥⊥ =U.

Proof. (i) SinceÕ is nondegenerate andE is finite-dimensional, the semilinear maplÕ:E→E∗ is bijective. By transposition, the inclusioni:U→E yields a surjectionr:E∗→U∗

(withr(f) =fæi for everyf∈E∗; the mapfæi is the restriction of the linear formf to U). It follows that the semilinear maprælÕ:E→U∗ given by

(rælÕ)(x)(u) =Õ(x, u) x∈E, u∈U

is surjective, and its kernel isU⊥. Thus, we have

dim(U∗) + dim(U⊥) = dim(E),

and since dim(U) = dim(U∗) becauseU is finite-dimensional, we get

dim(U) + dim(U⊥) = dim(U∗) + dim(U⊥) = dim(E).

(ii) Applying the above formula toU⊥, we deduce that dim(U) = dim(U⊥⊥). Since

U⊆U⊥⊥, we must haveU⊥⊥ =U.

Remark: We already proved in Proposition 14.12 that ifU is finite-dimensional, then codim(U⊥) = dim(U) andU⊥⊥ =U, but it doesn’t hurt to give another proof. Observe that (i) implies that

dim(U) + dim(rad(U))≤ dim(E).

We can now proceed with the Witt decomposition, but before that, we quickly take care of the structure theorem for alternating bilinear forms (the case whereÕ(u, u) = 0 for all u∈E). For an alternating bilinear form, the spaceE is totally isotropic. For example in dimension 2, the matrix

B =0 1 −1 0 defines the alternating form given by

Õ((x_1, y_1),(x_2, y_2)) =x_1y2−x_2y_1.

This case is surprisingly general.

Proposition 14.22. LetÕ:E×E→K be an alternating bilinear form onE. Ifu, v∈E are two (nonzero) vectors such thatÕ(u, v) =λ = 0, thenu andv are linearly independent. If we letu_1 =λ−1u andv_1 =v, thenÕ(u_1, v_1) = 1, and the restriction ofÕ to the plane spanned byu_1 andv_1 is represented by the matrix

0 1 . −1 0

Proof. Ifu andv were linearly dependent, asu, v = 0, we could writev =µu for someµ = 0, but then, sinceÕ is alternating, we would have

λ =Õ(u, v) =Õ(u, µu) =µÕ(u, u) = 0,

contradicting the fact thatλ = 0. The rest is obvious.

Proposition 14.22 yields a plane spanned by two vectors u_1, v_1 such thatÕ(u_1, u_1) = Õ(v_1, v_1) = 0 andÕ(u_1, v_1) = 1. Such a plane is called a hyperbolic plane. IfE is finitedimensional, we obtain the following theorem.

Theorem 14.23. LetÕ:E×E→ K be an alternating bilinear form on a spaceE of finite dimensionn. Then, there is a direct sum decomposition ofE into pairwise orthogonal subspaces

E =W1⊕ · · · ⊕Wr⊕ rad(E),

where eachWi is a hyperbolic plane and rad(E) =E⊥. Therefore, there is a basis ofE of the form

(u_1, v_1, . . . , ur, vr, w_1, . . . , wn−2r), with respect to which the matrix representingÕ is a block diagonal matrixM of the form ëJ 0 ö

ì J ÷

ì... ÷

M

=

ì ÷

ì ÷ ,

ìJ ÷

í ø

0 0n−2r

with J =0 1 . −1 0

Proof. IfÕ = 0, thenE =E⊥ and we are done. Otherwise, there are two nonzero vectors u, v∈E such thatÕ(u, v) = 0, so by Proposition 14.22, we obtain a hyperbolic planeW_2 spanned by two vectorsu_1, v_1 such thatÕ(u_1, v_1) = 1. The subspaceW_1 is nondegenerate (for example, det(J) =−1), so by Proposition 14.20, we get a direct sum

E =W1⊕W_1.

By Proposition 14.13, we also have

E⊥ = (W1⊕W⊥ ) =W1⊥ W1⊥⊥ = rad(W_1 ). 1 ∩

By the induction hypothesis applied toW⊥ , we obtain our theorem.1

The following corollary follows immediately. Proposition 14.24. LetÕ:E×E→K be an alternating bilinear form on a spaceE of finite dimensionn.

(1) The rank ofÕ is even.

(2) IfÕ is nondegenerate, then dim(E) =n is even.

(3) Two alternating bilinear formsÕ_1:E1×E1→K andÕ_2:E2×E2→K are equivalent iff dim(E_1) = dim(E_2) andÕ_1 andÕ_2 have the same rank.

The only part that requires a proof is part (3), which is left as an easy exercise. IfÕ is nondegenerate, thenn = 2r, and a basis ofE as in Theorem 14.23 is called a symplectic basis. The spaceE is called a hyperbolic space (or symplectic space). Observe that if we reorder the vectors in the basis

(u_1, v_1, . . . , ur, vr, w_1, . . . , wn−2r)

to obtain the basis (u_1, . . . , ur, v_1, . . . vr, w_1, . . . , wn−2r), then the matrix representingÕ becomes

ë 0 Ir 0ö í− Ir 0 0 ø. 0 0 0n−2r

This particularly simple matrix is often preferable, especially when dealing with the matrices (symplectic matrices) representing the isometries ofÕ (in which casen = 2r). We now return to the Witt decomposition. From now on,Õ:E×E→K is an -Hermitian form. The following assumption will be needed:

Property (T). For everyu∈E, there is someα∈K such thatÕ(u, u) =α + α. Property (T) is always satisfied ifÕ is alternating, or ifK is of characteristic = 2 and =±1, withα = 1Õ(u, u).2

The following (bizarre) technical lemma will be needed.

Lemma 14.25. LetÕ be an -Hermitian form onE and assume thatÕ satisfies property (T). For any totally isotropic subspaceU = (0) ofE, for everyx∈E not orthogonal toU, and for everyα∈K, there is somey∈U so that

Õ(x +y, x +y) =α + α.

Proof. By property (T), we haveÕ(x, x) =β + β for someβ∈K. For anyy∈U, sinceÕ is -Hermitian,Õ(y, x) = Õ(x, y), and sinceU is totally isotropicÕ(y, y) = 0, so we have

Õ (x +y, x +y) =Õ(x, x) +Õ(x, y) +Õ(y, x) +Õ(y, y)

=β + β +Õ(x, y) + Õ(x, y)

=β +Õ(x, y) + (β +Õ(x, y).

Sincex is not orthogonal toU, the functiony→Õ(x, y) +β is not the constant function. Consequently, this function takes the valueα for somey∈U, which proves the lemma.

Definition 14.14. LetÕ be an -Hermitian form onE. A Witt decomposition ofE is a triple (U, U , W), such that

(i) E =U⊕U⊕W (a direct sum)

(ii) U andU are totally isotropic

(iii) W is nondegenerate and orthogonal toU⊕U . Furthermore, ifE is finite-dimensional, then dim(U) = dim(U ) and in a suitable basis, the matrix representingÕ is of the form

ë 0 A 0ö í A 0 0ø 0 0 B

We say thatÕ is a neutral form if it is nondegenerate,E is finite-dimensional, and ifW = (0).

Sometimes, we use the notationU_1 ⊥ U_2 to indicate that in a direct sumU1⊕U_2,⊕

the subspaces U_1 and U_2 are orthogonal. Then, in Definition 14.14, we can write that E = (U⊕U )⊥W.⊕

As a warm up for Proposition 14.27, we prove an analog of Proposition 14.22 in the case of a symmetric bilinear form.

Proposition 14.26. LetÕ:E×E→K be a nondegenerate symmetric bilinear form withK a field of characteristic different from 2. For any nonzero isotropic vectoru, there is another nonzero isotropic vectorv such thatÕ(u, v) = 2, andu andv are linearly independent. In the basis (u, v/2), the restriction ofÕ to the plane spanned byu andv/2 is of the form

0 1 .1 0

Proof. SinceÕ is nondegenerate, there is some nonzero vectorz such that (rescalingz if necessary)Õ(u, z) = 1. If

v = 2z−Õ(z, z)u, then sinceÕ(u, u) = 0 andÕ(u, z) = 1, note that

Õ(u, v) =Õ(u,2z−Õ(z, z)u) = 2Õ(u, z)−Õ(z, z)Õ(u, u) = 2,

and

Õ (v, v) =Õ(2z−Õ(z, z)u,2z−Õ(z, z)u)

= 4Õ(z, z)− 4Õ(z, z)Õ(u, z) +Õ(z, z)2Õ(u, u) = 4Õ(z, z)− 4Õ(z, z) = 0.

Ifu andz were linearly dependent, asu, z = 0, we could writez =µu for someµ = 0, but then, we would have

Õ(u, z) =Õ(u, µu) =µÕ(u, u) = 0,

contradicting the fact thatÕ(u, z) = 0. Thenu andv = 2z− Õ(z, z)u are also linearly independent, since otherwisez could be expressed as a multiple ofu. The rest is obvious.

Proposition 14.26 yields a plane spanned by two vectors u_1, v_1 such thatÕ(u_1, u_1) = Õ(v_1, v_1) = 0 andÕ(u_1, v_1) = 1. Such a plane is called an Artinian plane. Proposition 14.26 also shows that nonzero isotropic vectors come in pair.

Remark: Some authors refer to the above plane as a hyperbolic plane. Berger (and others) point out that this terminology is undesirable because the notion of hyperbolic plane already exists in differential geometry and refers to a very different object.

We leave it as an exercice to figure out that the group of isometries of the Artinian plane, the set of all 2× 2 matricesA such that

A0 1 A =0 1 ,1 0 1 0

consists of all matrices of the form

λ 0 or0 λ , λ∈K− {0}.0 λ−1 λ−1 0

In particular, if K = R, then this group denoted O(1,1) has four connected components. The first step in showing the existence of a Witt decomposition is this.

Proposition 14.27. LetÕ be an -Hermitian form onE, assume thatÕ is nondegenerate and satisfies property (T), and letU be any totally isotropic subspace ofE of finite dimension dim(U) =r.

(1) If U is any totally isotropic subspace of dimensionr and ifU∩U⊥ = (0), thenU⊕U is nondegenerate, and for any basis (u_1, . . . , ur) ofU, there is a basis (u_1, . . . , ur) ofU such thatÕ(ui, uj) =δij, for alli, j = 1, . . . , r.

(2) If W is any totally isotropic subspace of dimension at mostr and ifW∩U⊥ = (0), then there exists a totally isotropic subspaceU with dim(U ) =r such that W⊆U andU∩U⊥ = (0).

Proof. (1) LetÕ be the restriction ofÕ toU×U . SinceU∩U⊥ = (0), for anyv∈U , ifÕ(u, v) = 0 for allu∈U, thenv = 0. Thus,Õ is nondegenerate (we only have to check on the left sinceÕ is -Hermitian). Then, the assertion about bases follows from the version of Proposition 14.3 for sesquilinear forms. SinceU is totally isotropic,U⊆U⊥, and since U∩U⊥ = (0), we must haveU∩U = (0), which show that we have a direct sumU⊕U .

It remains to prove thatU +U is nondegenerate. Observe that

H = (U +U )∩ (U +U )⊥ = (U +U )∩U⊥∩U⊥. SinceU is totally isotropic,U⊆U⊥, and sinceU∩U⊥ = (0), we have

( U +U )∩U⊥ = (U∩U⊥) + (U∩U⊥) =U + (0) =U, thusH =U∩U⊥. SinceÕ is nondegenerate,U∩U⊥ = (0), soH = (0) andU +U is nondegenerate.

(2) We proceed by descending induction on s = dim(W). The base cases =r is trivial. For the induction step, it suffices to prove that ifs < r, then there is a totally isotropic subspaceW containingW such that dim(W ) =s + 1 andW∩U⊥ = (0).

Since s = dim(W) < dim(U), the restriction of Õ to U×W is degenerate. Since W∩U⊥ = (0), we must haveU∩W⊥ = (0). We claim that

W⊥⊆W +U⊥.

If we had W⊥⊆W +U⊥,

then becauseU andW are finite-dimensional andÕ is nondegenerate, by Proposition 14.12, U⊥⊥ =U andW⊥⊥ =W, so by taking orthogonals,W⊥⊆W +U⊥ would yield (W +U⊥)⊥⊆W⊥⊥,

that is, W⊥∩U⊆W,

thusW⊥∩U⊆W∩U, and sinceU is totally isotropic,U⊆U⊥, which yields

W⊥∩U⊆W∩U⊆W∩U⊥ = (0), contradicting the fact thatU∩W⊥ = (0).

Therefore, there is some u∈W⊥ such thatu /W +U⊥. SinceU⊆U⊥, we can add tou any vectorz∈W⊥∩U⊆U⊥ so thatu+z∈W⊥ andu+z /W +U⊥ (ifu+z∈W +U⊥, sincez∈U⊥, thenu∈W +U⊥, a contradiction). SinceW⊥∩U = (0) is totally isotropic andu /W +U⊥ = (W⊥∩U)⊥, we can invoke Lemma 14.25 to find az∈W⊥∩U such that Õ(u +z, u +z) = 0. If we writex =u +z, thenx /∈W +U⊥, soW =W +Kx is a totally isotopic subspace of dimensions + 1. Furthermore, we claim thatW∩U⊥ = 0.

Otherwise, we would have y =w +λx∈U⊥, for somew∈W and someλ∈K, and then we would haveλx =−w +y∈W +U⊥. Ifλ = 0, thenx∈W +U⊥, a contradiction. Therefore,λ = 0,y =w, and sincey∈U⊥ andw∈W, we havey∈W∩U⊥ = (0), which means thaty = 0. Therefore,W is the required subspace and this completes the proof.

Here are some consequences of Proposition 14.27. If we setW = (0) in Proposition 14.27(2), then we get:

Proposition 14.28. LetÕ be an -Hermitian form onE which is nondegenerate and satisfies property (T). For any totally isotropic subspaceU ofE of finite dimensionr, there exists a totally isotropic subspaceU of dimensionr such thatU∩U = (0) andU⊕U is nondegenerate.

Proposition 14.29. Any two -Hermitian neutral forms satisfying property (T) defined on spaces of the same dimension are equivalent.

Note that under the conditions of Proposition 14.28, ( U, U ,(U⊕U )⊥) is a Witt decomposition forE. By Proposition 14.27(1), the blockA in the matrix ofÕ is the identity matrix.

The following proposition shows that every subspaceU ofE can be embedded into a nondegenerate subspace.

Proposition 14.30. LetÕ be an -Hermitian form onE which is nondegenerate and satisfies property (T). For any subspaceU ofE of finite dimension, if we write

U =V ⊥W,⊕

for some orthogonal complementW ofV = rad(U), and if we letr = dim(rad(U)), then there exists a totally isotropic subspaceV of dimensionr such thatV∩V = (0), and (V⊕V ) ⊥ W = V ⊕U is nondegenerate. Furthermore, any isometryf fromU into another space⊕ (E , Õ ) whereÕ is an -Hermitian form satisfying the same assumptions as Õ can be extended to an isometry on (V⊕V )⊥W.⊕

Proof. SinceW is nondegenerate,W⊥ is also nondegenerate, andV⊆W⊥. Therefore, we can apply Proposition 14.28 to the restriction ofÕ toW⊥ and toV to obtain the requiredV . We know thatV⊕V is nondegenerate and orthogonal toW, which is also nondegenerate,

so (V⊕V )⊥W =V⊕U is nondegenerate.⊕

We leave the second statement about extendingf as an exercise (use the fact thatf(U) =

f (V )⊥f(W), whereV_1 =f(V ) is totally isotropic of dimensionr, to find another totally⊕

isotropic susbpaceV_1 of dimensionr such thatV1∩V_1 = (0) andV1⊕V_1 is orthogonal to f(W)).

The subspace ( V⊕V )⊥W =V⊕U is often called a nondegenerate completion ofU.⊕

The subspaceV⊕V is called an Artinian space. Proposition 14.27 show thatV⊕V has a basis (u_1, v_1, . . . , ur, vr) consisting of vectorsui∈V andvj∈ V such thatÕ(ui, uj) =δij. The subspace spanned by (ui, vi) is an Artinian plane, soV⊕V it is the orthogonal direct sum ofr Artinian planes. Such a space is often denoted by Ar2r.

We now sharpen Proposition 14.27.

Theorem 14.31. LetÕ be an -Hermitian form onE which is nondegenerate and satisfies property (T). LetU_1 andU_2 be two totally isotropic maximal subspaces ofE, withU_1 orU_2 of finite dimension. WriteU =U1∩ U_2, letS_1 be a supplement ofU inU_1 andS_2 be a supplement ofU inU_2 (so thatU_1 =U⊕S_1,U_2 =U⊕S_2), and letS =S_1 +S_2. Then, there exist two subspacesW andD ofE such that:

(a) The subspacesS,U +W, andD are nondegenerate and pairwise orthogonal.

(b) We have a direct sum E =S⊥ (U⊕W)⊥D.⊕ ⊕

(c) The subspaceD contains no nonzero isotropic vector (D is anisotropic). (d) The subspaceW is totally isotropic.

Furthermore, U_1 and U_2 are both finite dimensional, and we have dim(U_1) = dim(U_2), dim(W) = dim(U), dim(S_1) = dim(S_2), and codim(D) = 2 dim(F_1).

Proof. First observe that ifX is a totally isotropic maximal subspace ofE, then any isotropic vectorx∈E orthogonal toX must belong toX, since otherwise,X +Kx would be a totally isotropic subspace strictly containingX, contradicting the maximality ofX. As a consequence, ifxi is any isotropic vector such thatxi∈Ui⊥ (fori = 1,2), thenxi∈Ui.

We claim that

S1∩S⊥ = (0) and S2∩S⊥ = (0).2 1

Assume that y∈S_1 is orthogonal toS_2. SinceU_1 =U⊕S_1 andU_1 is totally isotropic,y is orthogonal toU_1, and thus orthogonal toU, so thaty is orthogonal toU_2 =U⊕S_2. Since S1⊆U_1 andU_1 is totally isotropic,y is an isotropic vector orthogonal toU_2, which by a previous remark implies thaty∈U_2. Then, sinceS1⊆U_1 andU⊕S_1 is a direct sum, we have

y∈S1∩U_2 =S1∩U1∩U_2 =S1∩U = (0).

Therefore S1∩S⊥ = (0). A similar proof show thatS2∩S⊥ = (0). IfU_1 is finite-dimensional2 1

(the case where U_2 is finite-dimensional is similar), then S_1 is finite-dimensional, so by Proposition 14.12,S⊥ has finite codimension. SinceS2∩S⊥ = (0), and since any supplement

1 1

ofS⊥ has finite dimension, we must have

1

dim(S_2)≤ codim(S⊥ ) = dim(S_1).

1

By a similar argument, dim(S_1)≤ dim(S_2), so we have

dim(S_1) = dim(S_2).

By Proposition 14.27(1), we conclude thatS =S_1 +S_2 is nondegenerate.

By Proposition 14.20, the subspace N = S⊥ = (S_1 +S_2)⊥ is nondegenerate. Since U_1 =U⊕S_1,U_2 =U⊕ S_2, andU_1, U_2 are totally isotropic,U is orthogonal toS_1 and to S_2, soU⊆N. SinceU is totally isotropic, by Proposition 14.28 applied toN, there is a totally isotropic subspaceW ofN such that dim(W) = dim(U),U∩W = (0), andU +W is nondegenerate. Consequently, (d) is satisfied byW.

To satisfy (a) and (b), we pickD to be the orthogonal ofU⊕W inN. Then,N = (U⊕W)⊥D andE =S⊥N, soE =S⊥ (U⊕W)⊥D.⊕ ⊕ ⊕ ⊕

As to (c), since D is orthogonalU⊕W,D is orthogonal toU, and sinceD⊆N andN is orthogonal toS_1 +S_2,D is orthogonal toS_1, soD is orthogonal toU_1 =U⊕S1. Ify∈D is any isotropic vector, sincey∈U1⊥ , by a previous remark,y∈U_1, soy∈D∩U_1. But, D⊆N withN∩(S_1 +S_2) = (0), andD∩(U +W) = (0), soD∩(U +S_1) =D∩U_1 = (0), which yieldsy = 0. The statements about dimensions are easily obtained.

We obtain the following corollaries. Theorem 14.32. LetÕ be an -Hermitian form onE which is nondegenerate and satisfies property (T).

(1) Any two totally isotropic maximal spaces of finite dimension have the same dimension.

(2) For any totally isotropic maximal subspace U of finite dimensionr, there is another totally isotropic maximal subspaceU of dimensionr such thatU∩U = (0), and U⊕U is nondegenerate. Furthermore, ifD = (U⊕U )⊥, then (U, U , D) is a Witt

decomposition ofE, and there are no nonzero isotropic vectors in D (D is anisotropic).

(3) IfE has finite dimensionn≥ 1, thenE has a Witt decomposition (U, U , D) as in (2). There is a basis ofE such that

(a) ifÕ is alternating ( =−1 andλ =λ for allλ∈K), thenn = 2m andÕ is represented by a matrix of the form

0 Im

Im 0−

(b) ifÕ is symmetric ( = +1 andλ =λ for allλ∈K), thenÕ is represented by a matrix of the formë 0 Ir 0ö

íIr 0 0ø,

0 0 P

where eithern = 2r andP does not occur, orn > 2r andP is a definite symmetric matrix.

(c) ifÕ is -Hermitian (the involutive automorphismλ→λ is not the identity), then Õ is represented by a matrix of the form

ë 0 Ir 0ö

íIr 0 0ø,

0 0 P

where either n = 2r andP does not occur, orn > 2r andP is a definite matrix such thatP∗ =P.

Proof. Part (1) follows from Theorem 14.31. By Proposition 14.28, we obtain a totally isotropic subspaceU of dimensionr such thatU∩U = (0). By applying Theorem 14.31 toU_1 =U andU_2 =U , we getU =W = (0), which proves (2). Part (3) is an immediate consequence of (2).

As a consequence of Theorem 14.32, we make the following definition.

Definition 14.15. LetE be a vector space of finite dimensionn, and letÕ be an -Hermitian form onE which is nondegenerate and satisfies property (T). The index (or Witt index)ν ofÕ, is the common dimension of all totally isotropic maximal subspaces ofE. We have 2ν≤n.

Neutral forms only exist if n is even, in which case,ν =n/2. Forms of indexν = 0 have no nonzero isotropic vectors.

negative definite symmetric forms.

WhenK = R, this is satisfied by positive definite or WhenK = C, this is satisfied by positive definite or

negative definite Hermitian forms. The vector space of a neutral Hermitian form ( = +1) is an Artinian space, and the vector space of a neutral alternating form is a hyperbolic space. If the fieldK is algebraically closed, we can describe all nondegenerate quadratic forms. Proposition 14.33. IfK is algebraically closed andE has dimensionn, then for every nondegenerate quadratic form Φ, there is a basis (e_1, . . . , en) such that Φ is given by

n m xixm+i ifn = 2mΦ xiei = i=1

m xixm+i +x^2 ifn = 2m + 1.i−1 i=1 2m+1

Proof. We work with the polar form Õ of Φ. Let U_1 and U_2 be some totally isotropic subspaces such thatU1∩U_2 = (0) given by Theorem 14.32, and letq be their common dimension. Then,W =U = (0). Since we can pick bases (e_1, . . . eq) inU_1 and (eq+1, . . . , e2q) inU_2 such thatÕ(ei, ei+q) = 0, fori, j = 1, . . . , q, it suffices to proves that dim(D)≤ 1. If x, y∈D withx = 0, from the identity

Φ(y−λx) = Φ(y)−λÕ(x, y) +λ^2Φ(x)

and the fact that Φ( x) = 0 sincex∈D andx = 0, we see that the equation Φ(y−λy) = 0 has at least one solution. Since Φ(z) = 0 for every nonzeroz∈D, we gety =λx, and thus dim(D)≤ 1, as claimed.

We also have the following proposition which has applications in number theory.

Proposition 14.34. If Φ is any nondegenerate quadratic form such that there is some nonzero vectorx∈E with Φ(x) = 0, then for everyα∈K, there is somey∈E such that Φ(y) =α.

The proof is left as an exercise. We now turn to the Witt extension theorem.

14.7 Witt’s Theorem

Witt’s theorem was referred to as a “scandal” by Emil Artin. What he meant by this is that one had to wait until 1936 (Witt [110]) to formulate and prove a theorem at once so simple in its statement and underlying concepts, and so useful in various domains (geometry, arithmetic of quadratic forms).1

Besides Witt’s original proof (Witt [110]), Chevalley’s proof [20] seems to be the “best” proof that applies to the symmetric as well as the skew-symmetric case. The proof in Bourbaki [11] is based on Chevalley’s proof, and so are a number of other proofs. This is the one we follow (slightly reorganized). In the symmetric case, Serre’s exposition is hard to beat (see Serre [93], Chapter IV).

Theorem 14.35. (Witt, 1936) LetE andE be two finite-dimensional spaces respectively equipped with two nondegenerate -Hermitan formsÕ andÕ satisfying condition (T), and assume that there is an isometry between (E, Õ) and (E , Õ ). For any subspaceU ofE, every injective metric linear mapf fromU intoE extends to an isometry fromE toE .

Proof. Since (E, Õ) and (E , Õ ) are isometric, we may assume thatE =E andÕ =Õ (if h:E→E is an isometry, thenh−1 f is an injective metric map fromU intoE. Theæ

details are left to the reader). We begin with the following observation. IfU_1 andU_2 are two subspaces ofE such thatU1∩U_2 = (0) and if we have metric linear mapsf_1:U1→E andf_2:U2→E such that

Õ(f_1(u_1), f_2(u_2)) =Õ(u_1, u_2) forui∈Ui (i = 1,2), (∗) then the linear mapf :U1⊕U2 →E given byf(u_1 +u_2) =f_1(u_1) +f_2(u_2) extendsf_1 and f_2 and is metric. Indeed, sincef_1 andf_2 are metric and using (∗), we have

Õ (f_1(u_1) +f_2(u_2), f_1(v_1) +f_2(v_2)) =Õ(f_1(u_1), f_1(v_1)) +Õ(f_1(u_1), f_2(v_2)) +Õ(f_2(u_2), f_1(v_1)) +Õ(f_2(u_2), f_2(v_2)) =Õ(u_1, v_1) +Õ(u_1, v_2) +Õ(u_2, v_1) +Õ(u_2, v_2) =Õ(u_1 +u_2, v_2 +v_2).

Furthermore, iff_1 andf_2 are injective, then so iff.

We now proceed by induction on the dimension r ofU. The caser = 0 is trivial. For the induction step,r≥ 1 soU = (0), and letH be any hyperplane inU. Letf :U→E be an injective metric linear map. By the induction hypothesis, the restrictionf_0 off toH extends to an isometryg_0 ofE. Ifg_0 extendsf, we are done. Otherwise,H is the subspace of elements ofU left fixed byg−1 f. If the theorem holds in this situation, namely the0 æ

1 Curiously, some references to Witt’s paper claim its date of publication to be 1936, but others say 1937. The answer to this mystery is that Volume 176 of Crelle Journal was published in four issues. The cover page of volume 176 mentions the year 1937, but Witt’s paper is dated May 1936. This is not the only paper of Witt appearing in this volume!

subspace of U left fixed byf is a hyperplaneH inU, then we have an isometryg_1 ofE extendingg− 1 f, andg0æg_1 is an isometry ofE extendingf. Therefore, we are reduced to0 æ

the following situation:

Case (H). The subspace ofU left fixed byf is a hyperplaneH inU.

In this case, the setD ={f(u)−u|u∈U} is a line inU (a one-dimensional subspace). For allu, v∈U, we have

Õ(f(u), f(v)−v) =Õ(f(u), f(v))−Õ(f(u), v) =Õ(u, v)−Õ(f(u), v) =Õ(u−f(u), v), that is Õ(f(u), f(v)−v) =Õ(u−f(u), v) for allu, v∈U, (∗∗)

and if u∈H, which means thatf(u) =u, we getu∈D⊥. Therefore,H⊆D⊥. SinceÕ is nondegenerate, we have dim(D) + dim(D⊥) = dim(E), and since dim(D) = 1, the subspace D⊥ is a hyperplane inE.

Hypothesis (V). We can find a subspaceV ofE orthogonal toD and such that V∩U =V∩f(U) = (0).

Then, we have

Õ(f(u), v) =Õ(u, v) for allu∈U and allv∈V ,

since Õ(f(u), v)−Õ(u, v) =Õ(f(u)−u, v) = 0, withf(u)−u∈D andv∈V orthogonal to D. By the remark at the beginning of the proof, withf_1 =f andf_2 the inclusion ofV into E, we can extendf to an injective metric map onU⊕V leaving all vectors inV fixed. In this case, the set{f(w)−w|w∈U⊕V} is still the lineD. We show below that the fact thatf can be extended toU⊕V implies thatf can be extended to the whole ofE.

We are reduced to proving that a subspaceV as above exists. We distinguish between two cases.

Case (a).U⊆D⊥.

In this case, formula (∗∗) show thatf(U) is not contained inD⊥ (check this!). Consequently,

U∩D⊥ =f(U)∩D⊥ =H.

We can pick V to be any supplement ofH inD⊥, and the above formula shows thatV∩U = V∩f(U) = (0). SinceU⊕ V contains the hyperplaneD⊥ (sinceD⊥ =H⊕V andH⊆U), andU⊕V =D⊥ (sinceU is not contained inD⊥ andV⊆D⊥), we must haveE =U⊕V , and as we showed as a consequence of hypothesis (V),f can be extended to an isometry of U⊕V =E.

Case (b).U⊆D⊥.

In this case, formula (∗∗) shows thatf(U)⊆D⊥ soU +f(U)⊆D⊥, and sinceD = {f(u)−u|u∈U}, we haveD⊆D⊥; that is, the lineD is isotropic.

We show that case (b) can be reduced to the situation whereU = D⊥ andf is an isometry ofU. For this, we show that there exists a subspaceV ofD⊥, such that

D⊥ =U⊕V =f(U)⊕V.

This is obvious ifU =f(U). Otherwise, letx∈U withx /H, and lety∈f(U) withy /∈H. Sincef(H) =H (pointwise),f is injective, and H is a hyperplane inU , we have

U =H⊕Kx, f(U) =H⊕Ky.

We claim that x +y /∈U. Otherwise, sincey =x +y−x, withx +y, x∈U and since y∈f(U), we would havey∈U∩f(U) =H, a contradiction. Similarly,x +y /f(U). It follows that

U +f(U) =U⊕K(x +y) =f(U)⊕K(x +y).

Now, pickW to be any supplement ofU +f(U) inD⊥ so thatD⊥ = (U +f(U))⊕W, and let

V =K(x +y) +W.

Then, sincex∈U, y∈f(U),W⊆D⊥, andU +f(U)⊆D⊥, we haveV⊆D⊥. We also have

U⊕V =U⊕K(x +y)⊕W = (U +f(U))⊕W =D⊥ and f(U)⊕V =f(U)⊕K(x +y)⊕W = (U +f(U))⊕W =D⊥, so as we showed as a consequence of hypothesis (V),f can be extended to an isometry of the hyperplaneD⊥, andD is still the line{f(w)−w|w∈U⊕V}.

The above argument shows that we are reduced to the situation where U =D⊥ is a hyperplane inE andf is an isometry ofU. If we pick anyv /∈U, thenE =U⊕Kv, and if we can find somev1∈E such that

Õ(f(u), v_1) =Õ(u, v) for allu∈U Õ(v_1, v_1) =Õ(v, v),

then as we showed at the beginning of the proof, we can extendf to a metric mapg of U +Kv =E such thatg(v) =v_1.

To findv_1, let us prove that for everyv∈E, there is somev∈E such that

Õ(f(u), v ) =Õ(u, v) for allu∈U. (†)

This is because the linear formu→Õ(f−1(u), v) (u∈U) is the restriction of a linear form ψ∈E∗, and sinceÕ is nondegenerate, there is some (unique)v∈E, such that

ψ(x) =Õ(x, v ) for allx∈E, which implies that Õ(u, v ) =Õ(f−1(u), v) for allu∈U,

and since f is an automorphism ofU, that (†) holds. Furthermore, observe that formula (†) still holds if we add tov a vectory inD, sincef(U) =U =D⊥. Therefore, for any v_1 =v +y withy∈D, if we extendf to a linear map ofE by settingg(v) =v_1, then by (†) we have

Õ(g(u), g(v)) =Õ(u, v) for allu∈U.

We still need to pick y∈ D so thatv_1 =v +y satisfiesÕ(v_1, v_1) =Õ(v, v). However, since v /U =D⊥, the vectorv is not orthogonalD, and by lemma 14.25, there is somey∈D such that

Õ(v +y, v +y) =Õ(v, v).

Then, if we let v_1 =v +y, as we showed at the beginning of the proof, we can extendf to a metric mapg ofU +Kv =E by settingg(v) =v_1. SinceÕ is nondegenerate,g is an isometry.

The first corollary of Witt’s theorem is sometimes called the Witt’s cancellation theorem.

Theorem 14.36. (Witt Cancellation Theorem) Let (E_1, Õ_1) and (E_2, Õ_2) be two pairs of finite-dimensional spaces and nondegenerate -Hermitian forms satisfying condition (T), and assume that (E_1, Õ_1) and (E_2, Õ_2) are isometric. For any subspaceU ofE_1 and any subspace V ofE_2, if there is an isometryf :U→V , then there is an isometryg:U⊥→V⊥.

Proof. Iff :U→ V is an isometry betweenU andV , by Witt’s theorem (Theorem 14.36), the linear mapf extends to an isometryg betweenE_1 andE_2. We claim thatg mapsU⊥ intoV⊥. This is because ifv∈U⊥, we haveÕ_1(u, v) = 0 for allu∈U, so

Õ_2(g(u), g(v)) =Õ_1(u, v) = 0 for allu∈U,

and since g is a bijection betweenU andV , we haveg(U) =V , so we see thatg(v) is orthogonal toV for everyv∈U⊥; that is,g(U⊥)⊆V⊥. Sinceg is a metric map and since Õ_1 is nondegenerate, the restriction ofg toU⊥ is an isometry fromU⊥ toV⊥.

A pair ( E, Õ) whereE is finite-dimensional andÕ is a nondegenerate -Hermitian form is often called an -Hermitian space. When = 1 andÕ is symmetric, we use the term Euclidean space or quadratic space. When =−1 andÕ is alternating, we use the term symplectic space. When = 1 and the automorphismλ→ λ is not the identity we use the term Hermitian space, and when =−1, we use the term skew-Hermitian space.

We also have the following result showing that the group of isometries of an -Hermitian space is transitive on totally isotropic subspaces of the same dimension.

Theorem 14.37. LetE be a finite-dimensional vector space and letÕ be a nondegenerate

-Hermitian form onE satisfying condition (T). Then for any two totally isotropic subspaces

U andV of the same dimension, there is an isometryf∈ Isom(Õ) such thatf(U) =V .

Furthermore, every linear automorphism ofU is induced by an isometry ofE. Remark: Witt’s cancelation theorem can be used to define an equivalence relation on - Hermitian spaces and to define a group structure on these equivalence classes. This way, we obtain the Witt group, but we will not discuss it here.

14.8 Symplectic Groups

In this section, we are dealing with a nondegenerate alternating form Õ on a vector spaceE of dimensionn. As we saw earlier,n must be even, sayn = 2m. By Theorem 14.23, there is a direct sum decomposition ofE into pairwise orthogonal subspaces

E =W_1⊥ ⊥Wm,⊕ · · · ⊕

where eachWi is a hyperbolic plane. EachWi has a basis (ui, vi), withÕ(ui, ui) =Õ(vi, vi) = 0 andÕ(ui, vi) = 1, fori = 1, . . . , m. In the basis

(u_1, . . . , um, v_1, . . . , vm),

Õ is represented by the matrix

Jm,m =0 Im .

−Im 0

The symplectic group Sp(2m, K) is the group of isometries ofÕ. The maps in Sp(2m, K) are called symplectic maps. With respect to the above basis, Sp(2m, K) is the group of 2m× 2m matricesA such that

A Jm,mA =Jm,m.

Matrices satisfying the above identity are called symplectic matrices. In this section, we show that Sp(2m, K) is a subgroup of SL(2m, K) (that is, det(A) = +1 for allA∈ Sp(2m, K)), and we show that Sp(2m, K) is generated by special linear maps called symplectic transvections.

First, we leave it as an easy exercise to show that Sp(2, K) = SL(2, K). The reader should also prove that Sp(2m, K) has a subgroup isomorphic to GL(m, K). Next we characterize the symplectic mapsf that leave fixed every vector in some given hyperplaneH, that is,

f(v) =v for allv∈H.

Since Õ is nondegenerate, by Proposition 14.21, the orthogonalH⊥ ofH is a line (that is, dim(H⊥) = 1). For everyu∈E and everyv∈H, sincef is an isometry andf(v) =v for allv∈H, we have

Õ (f(u)−u, v) =Õ(f(u), v)−Õ(u, v) =Õ(f(u), v)−Õ(f(u), f(v)) =Õ(f(u), v−f(v))) =Õ(f(u),0) = 0,

which shows thatf(u)−u∈H⊥ for allu∈E. Therefore,f− id is a linear map fromE into the lineH⊥ whose kernel containsH, which means that there is some nonzero vector w∈H⊥ and some linear formψ such that

f(u) =u +ψ(u)w, u∈E.

Sincef is an isometry, we must haveÕ(f(u), f(v)) =Õ(u, v) for allu, v∈E, which means that

Õ (u, v) =Õ(f(u), f(v))

=Õ(u +ψ(u)w, v +ψ(v)w)

=Õ(u, v) +ψ(u)Õ(w, v) +ψ(v)Õ(u, w) +ψ(u)ψ(v)Õ(w, w) =Õ(u, v) +ψ(u)Õ(w, v)−ψ(v)Õ(w, u),

which yields

ψ(u)Õ(w, v) =ψ(v)Õ(w, u) for allu, v∈E.

SinceÕ is nondegenerate, we can pick somev_0 such thatÕ(w, v_0) = 0, and we get ψ(u)Õ(w, v_0) =ψ(v_0)Õ(w, u) for allu∈E; that is,

ψ(u) =λÕ(w, u) for allu∈E,

for someλ∈K. Therefore,f is of the form

f(u) =u +λÕ(w, u)w, for allu∈E.

It is also clear that every f of the above form is a symplectic map. Ifλ = 0, thenf = id. Otherwise, ifλ = 0, thenf(u) =u iffÕ(w, u) = 0 iffu∈ (Kw)⊥ =H, whereH is a hyperplane. Thus,f fixes every vector in the hyperplaneH. Note that sinceÕ is alternating, Õ(w, w) = 0, which means thatw∈H.

In summary, we have characterized all the symplectic maps that leave every vector in some hyperplane fixed, and we make the following definition.

Definition 14.16. Given a nondegenerate alternating formÕ on a spaceE, a symplectic transvection (of directionw) is a linear mapf of the form

f(u) =u +λÕ(w, u)w, for allu∈E,

for some nonzerow∈E and someλ∈K. Ifλ = 0, the subspace of vectors left fixed byf is the hyperplaneH = (Kw)⊥. The mapf is also denotedτu,λ.

Observe that

τu,λæτu,µ =τu,λ+µ

andτu,λ = id iffλ = 0. The above shows that det(τu,λ) = 1, since whenλ = 0, we have τu,λ = (τu,λ/2)2.

Our next goal is to show that if u andv are any two nonzero vectors inE, then there is a simple symplectic mapf such thatf(u) =v.

Proposition 14.38. Given any two nonzero vectorsu, v∈E, there is a symplectic map f such thatf(u) =v, andf is either a symplectic transvection, or the composition of two symplectic transvections.

Proof. There are two cases.

Case 1.Õ(u, v) = 0.

In this case,u =v, sinceÕ(u, u) = 0. Let us look for a symplectic transvection of the formτv−u,λ. We want

v =u +λÕ(v−u, u)(v−u) =u +λÕ(v, u)(v−u),

which yields (λÕ(v, u)− 1)(v−u) = 0.

Since Õ(u, v) = 0 andÕ(v, u) =−Õ(u, v), we can pickλ =Õ(v, u)−1 andτv−u,λ mapsu tov. Case 2.Õ(u, v) = 0.

If u =v, useτu,0 = id. Now, assumeu =v. We claim that it is possible to pick some w∈E such thatÕ(u, w) = 0 andÕ(v, w) = 0. Indeed, if (Ku)⊥ = (Kv)⊥, then pick any nonzero vectorw not in the hyperplane (Ku)⊥. Othwerwise, (Ku)⊥ and (Kv)⊥ are two distinct hyperplanes, so neither is contained in the other (they have the same dimension), so pick any nonzero vector w_1 such that w1 ∈ (Ku)⊥ and w_1 / (Kv)⊥, and pick any nonzero vectorw_2 such thatw2∈ (Kv)⊥ andw2 /∈ (Ku)⊥. If we letw =w_1 +w_2, then Õ(u, w) =Õ(u, w_2) = 0, andÕ(v, w) =Õ(v, w_1) = 0. From case 1, we have some symplectic transvectionτw−u,λ1 such thatτw−u,λ1(u) =w, and some symplectic transvectionτv−w,λ2 such thatτv−w,λ2(w) =v, so the compositionτv−w,λ2æτw−u,λ1 mapsu tov.

Next, we would like to extend Proposition 14.38 to two hyperbolic planesW_1 andW_2.

Proposition 14.39. Given any two hyperbolic planesW_1 andW_2 given by bases (u_1, v_1) and (u_2, v_2) (withÕ(ui, ui) =Õ(vi, vi) = 0 andÕ(ui, vi) = 1, fori = 1,2), there is a symplectic mapf such thatf(u_1) =u_2,f(v_1) =v_2, andf is the composition of at most four symplectic transvections.

Proof. From Proposition 14.38, we can mapu_1 tou_2, using a mapf which is the composition of at most two symplectic transvections. Sayv_3 =f(v_1). We claim that there is a mapg such thatg(u_2) =u_2 andg(v_3) =v_2, andg is the composition of at most two symplectic transvections. If so,gæf maps the pair (u_1, v_1) to the pair (u_2, v_2), andgæf consists of at most four symplectic transvections. Thus, we need to prove the following claim:

Claim . If (u, v) and (u, v ) are hyperbolic bases determining two hyperbolic planes, then there is a symplectic mapg such thatg(u) =u,g(v) =v , andg is the composition of at most two symplectic transvections. There are two case.

Case 1.Õ(v, v ) = 0.

In this case, there is a symplectic transvectionτv−v,λ such thatτv−v,λ(v) =v . We also have

Õ(u, v−v) =Õ(u, v )−Õ(u, v) = 1− 1 = 0.

Therefore, τv−v,λ(u) =u, andg =τv−v,λ does the job.

Case 2.Õ(v, v ) = 0.

First, check that (u, u +v) is a also hyperbolic basis. Furthermore,

Õ(v, u +v) =Õ(v, u) +Õ(v, v) =Õ(v, u) =−1 = 0. Thus, there is a symplectic transvectionτv,λ1 such thatτu,λ1(v) =u +v andτu,λ1(u) =u. We also have

Õ(u +v, v ) =Õ(u, v ) +Õ(v, v ) =Õ(u, v ) = 1 = 0, so there is a symplectic transvectionτv−u−v,λ2 such thatτv−u−v,λ2(u +v) =v . Since Õ(u, v−u−v) =Õ(u, v )−Õ(u, u)−Õ(u, v) = 1− 0− 1 = 0, we haveτv −u−v,λ2(u) =u. Then, the compositiong =τv−u−v,λ2æτu,λ1 is such thatg(u) =u andg(v) =v .

We will use Proposition 14.39 in an inductive argument to prove that the symplectic transvections generate the symplectic group. First, make the following observation: IfU is a nondegenerate subspace ofE, so that

E =U⊥U⊥,⊕

and ifτ is a transvection ofH⊥, then we can form the linear map idU⊥τ whose restriction to

U

is the identity and whose restriction to

U

⊥

is

τ

, and id

U

⊥

⊕

τ is a transvection ofE.⊕

Theorem 14.40. The symplectic group Sp(2m, K) is generated by the symplectic transvections. For every transvectionf∈ Sp(2m, K), we have det(f) = 1.

Proof. LetG be the subgroup of Sp(2m, K) generated by the tranvections. We need to prove thatG = Sp(2m, K). Let (u_1, v_1, . . . , um, vm) be a symplectic basis ofE, and letf∈ Sp(2m, K) be any symplectic map. Then,f maps (u_1, v_1, . . . , um, vm) to another symplectic basis (u_1, v_1, . . . , um, vm). If we prove that there is someg∈G such thatg(ui) =ui and g(vi) =vi fori = 1, . . . , m, thenf =g andG = Sp(2m, K).

We use induction oni to prove that there is somegi∈G so thatgi maps (u_1, v_1, . . . , ui, vi) to (u_1, v_1, . . . , ui, vi).

The base casei = 1 follows from Proposition 14.39.

For the induction step, assume that we have some gi∈G mapping (u_1, v_1, . . . , ui, vi) to (u_1, v_1, . . . , ui, vi), and let (ui+1, vi+1, . . . , um, vm) be the image of (ui+1, vi+1, . . . , um, vm) bygi. IfU is the subspace spanned by (u_1, v_1, . . . , um, vm), then each hyperbolic plane Wi+k given by (ui+k, vi+k) and each hyperbolic planeWi+k given by (ui+k, vi+k) belongs to U⊥. Using the remark before the theorem and Proposition 14.39, we can find a transvectionτ mappingWi+1 ontoWi+1 and leaving every vector inU fixed. Then,τægi maps (u_1, v_1, . . . , ui+1, vi+1) to (u_1, v_1, . . . , ui+1, vi+1), establishing the induction step.

For the second statement, since we already proved that every transvection has a determinant equal to +1, this also holds for any composition of transvections inG, and since G = Sp(2m, K), we are done.

It can also be shown that the center of Sp(2m, K) is reduced to the subgroup{id,−id}. The projective symplectic group PSp(2m, K) is the quotient group PSp(2m, K)/{id,− id}. All symplectic projective groups are simple, except PSp(2,F_2),PSp(2,F_3), and PSp (4,F_2), see Grove [50].

The orders of the symplectic groups over finite fields can be determined. For details, see Artin [2], Jacobson [57] and Grove [50].

An interesting property of symplectic spaces is that the determinant of a skew-symmetric matrixB is the square of some polynomial Pf(B) called the Pfaffian; see Jacobson [57] and Artin [2]. We leave considerations of the Pfaffian to the exercises.

We now take a look at the orthogonal groups.

14.9 Orthogonal Groups

In this section, we are dealing with a nondegenerate symmetric bilinear from Õ over a finitedimensional vector spaceE of dimensionn over a field of characateristic not equal to 2. Recall that the orthogonal group O(Õ) is the group of isometries ofÕ; that is, the group of linear mapsf :E→E such that

Õ(f(u), f(v)) =Õ(u, v) for allu, v∈E.

The elements of O(Õ) are also called orthogonal transformations. IfM is the matrix ofÕ in any basis, then a matrixA represents an orthogonal transformation iff

A MA =M.

SinceÕ is nondegenerate,M is invertible, so we see that det(A) =±1. The subgroup

SO(Õ) ={f∈ O(Õ)| det(f) = 1}

is called the special orthogonal group (ofÕ), and its members are called rotations (or proper orthogonal transformations). Isometriesf∈ O(Õ) such that det(f) =−1 are called improper orthogonal transformations, or sometimes reversions.

IfH is any nondegenerate hyperplane inE, thenD =H⊥ is a nondegenerate line and we have

E =H⊥H⊥.⊕

For any nonzero vectoru∈D =H⊥ Consider the mapτu given by

τu(v) =v− 2Õ(v, u)u for allv∈E.Õ(u, u)

If we replaceu byλu withλ = 0, we have

τÕ(v, λu) λu =v− 2λÕ(v, u)λu =v− 2Õ(v, u)u,λu(v) =v− 2Õ(λu, λu) λ2Õ(u, u) Õ(u, u)

which shows that τu depends only on the lineD, and thus only the hyperplaneH. Therefore, denote byτH the linear mapτu determined as above by any nonzero vectoru∈H⊥. Note that ifv∈H, then

τH(v) =v, and ifv∈D, then

τH(v) =−v. A simple computation shows that

Õ(τH(u), τH(v)) =Õ(u, v) for allu, v∈E, soτH∈ O(Õ), and by picking a basis consisting ofu and vectors inH, that det(τH) =−1. It is also clear thatτ^2 = id.H

Definition 14.17. IfH is any nondegenerate hyperplane inE, for any nonzero vector u∈H⊥, the linear mapτH given by

τH(v) =v− 2Õ(v, u)u for allv∈EÕ(u, u)

is an involutive isometry ofE called the reflection through (or about) the hyperplaneH. Remarks:

1. It can be shown that if f∈ O(Õ) leaves every vector in some hyperplaneH fixed, then eitherf = id orf =τH; see Taylor [104] (Chapter 11). Thus, there is no analog to symplectic transvections in the orthogonal group.

2. IfK = R andÕ is the usual Euclidean inner product, the matrices corresponding to hyperplane reflections are called Householder matrices.

Our goal is to prove that O(Õ) is generated by the hyperplane reflections. The following proposition is needed.

Proposition 14.41. LetÕ be a nondegenerate symmetric bilinear form on a vector space E. For any two nonzero vectorsu, v∈E, ifÕ(u, u) =Õ(v, v) andv− u is nonisotropic, then the hyperplane reflectionτH =τv−u mapsu tov, withH = (K(v−u))⊥.

Proof. Sincev−u is not isotropic,Õ(v−u, v−u) = 0, and we have

τv−u(u) =u− 2 Õ(u, v−u) )(v−u)Õ(v− u, v−u

=u Õ(u, v)−Õ(u, u) )(v−u)− 2Õ(v, v)− 2Õ(u, v) +Õ(u, u 2(

Õ

(

u, v

)

−

Õ

(

u, u

))

=u− 2(Õ(u, u)− 2Õ(u, v))(v−u) =v, which proves the proposition.

We can now obtain a cheap version of the Cartan–Dieudonn´e theorem.

Theorem 14.42. (Cartan–Dieudonn´e, weak form) LetÕ be a nondegenerate symmetric bilinear form on aK-vector spaceE of dimensionn (char(K) = 2). Then, every isometry f∈ O(Õ) withf = id is the composition of at most 2n− 1 hyperplane reflections.

Proof. We proceed by induction onn. Forn = 0, this is trivial (since O(Õ) ={id}). Next, assume thatn≥ 1. SinceÕ is nondegenerate, we know that there is some nonisotropic vectoru∈E. There are three cases.

Case 1.f(u) =u.

SinceÕ is nondegenrate andu is nonisotropic, the hyperplaneH = (Ku)⊥ is nondegenerate,E =H⊥ (Ku)⊥, and sincef(u) =u, we must havef(H) =H. The restrictionf of⊕

off toH is an isometry ofH. By the induction hypothesis, we can write f =τkæ · · · æτ_1,

whereτi is some hyperplane reflection about a hyperplaneLi inH, withk≤ 2n− 3. We can extend eachτi to a reflectionτi about the hyperplaneLi⊥Ku so thatτi(u) =u, and clearly,⊕

f =τkæ · · · æτ_1.

Case 2.f(u) =−u.

Ifτ is the hyperplane reflection about the hyperplaneH = (Ku)⊥, theng =τæf is an isometry ofE such thatg(u) =u, and we are back to Case (1). Sinceτ^2 = 1 We obtain

f =τæτkæ · · · æτ1 whereτ and theτi are hyperplane reflections, withk≥ 2n−3, and we get a total of 2n−2 hyperplane reflections.

Case 3.f(u) =u andf(u) =−u.

Note thatf(u)−u andf(u) +u are orthogonal, since

Õ(f(u)−u, f(u) +u) =Õ(f(u), f(u)) +Õ(f(u), u)−Õ(u, f(u))−Õ(u, u) =Õ(u, u)−Õ(u, u) = 0.

We also have Õ(u, u) =Õ((f(u) +u− (f(u)−u))/2,(f(u) +u− (f(u)−u))/2) = 1Õ(f(u) +u, f(u) +u) + 1Õ(f(u)−u, f(u)−u),

4 4

sof(u) +u andf(u)−u cannot be both isotropic, sinceu is not isotropic. Iff(u)−u is not isotopic, then the reflectionτf(u)−u is such that

τf(u)−u(u) =f(u), and sinceτ^2 = id, ifg =τf(u)−uæf, theng(u) =u, and we are back to case (1). Wef(u)−u

obtain

f =τf(u)−uæτkæ · · · æτ1 whereτf(u)−u and theτi are hyperplane reflections, withk≥ 2n− 3, and we get a total of 2n− 2 hyperplane reflections.

Iff(u) +u is not isotropic, then the reflectionτf(u)+u is such that

τf(u)+u(u) =−f(u), and sinceτ^2 = id, ifg =τf(u)+uæf, theng(u) =−u, and we are back to case (2). Wef(u)+u

obtain

f =τf(u)−uæτæτkæ · · · æτ1 whereτ, τf(u)−u and theτi are hyperplane reflections, withk≥ 2n−3, and we get a total of 2n− 1 hyperplane reflections. This proves the induction step.

The bound 2 n−1 is not optimal. The strong version of the Cartan–Dieudonn´e theorem says that at mostn reflections are needed, but the proof is harder. Here is a neat proof due to E. Artin (see [2], Chapter III, Section 4).

Case 1 remains unchanged. Case 2 is slightly different: f(u)−u is not isotropic. Since Õ(f(u) +u, f(u)−u) = 0, as in the first subcase of Case (3),g =τf(u)−uæf is such that g(u) =u and we are back to Case 1. This only costs one more reflection.

The new (bad) case is:

Case 3’.f(u)−u is nonzero and isotropic for all nonisotropicu∈ E. In this case, what saves us is thatE must be an Artinian space of dimensionn = 2m and thatf must be a rotation (f∈ SO(Õ)).

If we acccept this fact, then pick any hyperplane reflection τ. Then, sincef is a rotation, g =τæf is not a rotation because det(g) = det(τ) det(f) = (−1)(+1) =−1, sog(u)−u is not isotropic for all nonisotropicu∈E, we are back to Case 2, and using the induction hypothesis, we get

τæf =τkæ. . . , τ_1,

where each τi is a hyperplane reflection, andk≤ 2m. Sinceτæf is not a rotation, actually k≤ 2m−1, and thenf =τæτkæ. . . , τ_1, the composition of at mostk+ 1≤ 2m hyperplane reflections.

Therefore, except for the fact that in Case 3’, E must be an Artinian space of dimension n = 2m and thatf must be a rotation, which has not been proven yet, we proved the following theorem.

Theorem 14.43. (Cartan–Dieudonn´e, strong form) LetÕ be a nondegenerate symmetric bilinear form on aK-vector spaceE of dimensionn (char(K) = 2). Then, every isometry f∈ O(Õ) withf = id is the composition of at mostn hyperplane reflections.

To fill in the gap, we need two propositions.

Proposition 14.44. Let (E, Õ) be an Artinian space of dimension 2m, and letU be a totally isotropic subspace of dimensionm. For any isometryf∈ O(Õ), we have det(f) = 1 (f is a rotation).

Proof. We know that we can find a basis (u_1, . . . , um, v_1, . . . , vm) ofE such (u_1, . . . , um) is a basis ofU andÕ is represented by the matrix

0 Im .Im 0

Sincef(U) =U, the matrix representingf is of the form

A = B C .0 D

The conditionA Am,mA =Am,m translates as

B 0 0 Im B C= 0 Im C D Im 0 0 D Im 0

that is, B 0 0 D= 0 B D

D B C D +D C= 0 Im ,C D B C Im 0 which implies thatB D =I, and so

det( A) = det(B) det(D) = det(B ) det(D) = det(B D) = det(I) = 1, as claimed

Proposition 14.45. LetÕ be a nondegenerate symmetric bilinear form on a spaceE of dimensionn, and letf be any isometryf∈ O(Õ) such thatf(u)−u is nonzero and isotropic for every nonisotropic vectoru∈E. Then, E is an Artinian space of dimensionn = 2m, andf is a rotation (f∈ SO(Õ)).

Proof. We follow E. Artin’s proof (see [2], Chapter III, Section 4). First, consider the case n = 2. Since we are assuming thatE has some nonzero isotropic vector, by Proposition 14.26,E is an Artinian plane and there is a basis in whichÕ is represented by the matrix

0 1 ,1 0

we haveÕ((x_1, x_2),(x_1, x_2)) = 2x_1x_2, and the matrices representing isometries are of the

form λ 0 or0 λ , λ∈K− {0}.0 λ−1 λ−1 0

In the second case,

0 λ λ= λ ,λ−1 0 1 1

butu = (λ,1) is a nonisotropic vector such thatf(u)−u = 0. Therefore, we must be in the first case, and det(f) = +1.

Let us now assume that n≥ 3. Lety be some nonzero isotropic vector. Sincen≥ 3, the orthogonal space (Ky)⊥ has dimension at least 2, and we know that rad(Ky) = rad((Ky)⊥), which implies that (Ky)⊥ contains some nonisotropic vector, sayx. We haveÕ(x, y) = 0, so Õ(x +y, x +y) = Õ(x, x) = 0, for =±1. Then, by hypothesis, the vectors f(x)−x, f(x+y)−(x+y) =f(x)−x+(f(y)−y), andf(x−y)−(x−y) =f(x)−x−(f(y)−y) are isotropic. The last two vectors can be written asf(x)−x) + (f(y)−y) with =±1, so we have

0 =Õ(f(x)−x) + (f(y)−y), f(x)−x) + (f(y)−y))

= 2Õ(f(x)−x, f(y)−y)) +2Õ(f(y)−y, f(y)−y). If we write the two equations corresponding to =±1, and then add them up, we get

Õ(f(y)−y, f(y)−y) = 0.

Therefore, we proved that f(u)−u is isotropic for everyu∈E. If we letW = Im(f− id), then every vector inW is isotropic, and thusW is totally isotropic (recall that we assumed that char(K) = 2, soÕ is determined by Φ). For anyu∈E and anyv∈W⊥, sinceW is totally isotropic, we have

Õ(f(u)−u, f(v)−v) = 0, and sincef(u)−u∈W andv∈W⊥, we haveÕ(f(u)−u, v) = 0, and so

0 = Õ(f(u)−u, f(v)−v)

=Õ(f(u), f(v))−Õ(u, f(v))−Õ(f(u)−u, v) =Õ(u, v)−Õ(u, f(v))

=Õ(u, v−f(v)),

for all u∈E. SinceÕ is nonsingular, this means thatf(v) =v, for allv∈W⊥. However, by hypothesis, no nonisotropic vector is left fixed, which implies thatW⊥ is also totally isotropic. In summary, we proved thatW⊆W⊥ andW⊥⊆W⊥⊥ =W, that is,

W =W⊥.

Since, dim(W) + dim(W⊥) =n, we conclude thatW is a totally isotropic subspace ofE such that

dim(W) =n/2.

By Proposition 14.27, the spaceE is an Artinian space of dimensionn = 2m. SinceW =W⊥ andf(W⊥) =W⊥, by Proposition 14.44, the isometryf is a rotation.

Remarks:

1. Another way to finish the proof of Proposition 14.45 is to prove that iff is an isometry, then

Ker (f− id) = (Im(f− id))⊥. After having proved thatW = Im(f− id) is totally isotropic, we get

Ker (f− id) = Im(f− id),

which implies that (f− id)2 = 0. From this, we deduce that det(f) = 1. For details, see Jacobson [57] (Chapter 6, Section 6).

2. Iff =τHkæ · · · æτH1, where theHi are hyperplanes, then it can be shown that

dim(H1∩H2∩ · · · ∩Hs)≥n−s.

Now, since each Hi is left fixed byτHi, we see that every vector inH1∩ · · · ∩Hs is left fixed byf. In particular, ifs < n, thenf has some nonzero fixed point. As a consequence, an isometry without fixed points requiresn hyperplane reflections.

Witt’s Theorem can be sharpened to isometries in SO(Õ), but some condition onU is needed.

Theorem 14.46. (Witt–Sharpened Version) LetE be a finite-dimensional space equipped with a nondegenerate symmetric bilinear formsÕ. For any subspaceU ofE, every linear injective metric mapf fromU intoE extends to an isometryg ofE with a prescribed value

1 of det(g) iff± dim(U) + dim(rad(U))< dim(E) =n.

If

dim(U) + dim(rad(U)) = dim(E) =n,

and det(f) =−1, then there is nog∈ SO(Õ) extendingf.

Proof. Ifg_1 andg_2 are two extensions off such that det(g_1) det(g_2) =−1, thenh =g−1 g21 æ is an isometry such that det(h) =−1, andh leaves every vector ofU fixed. Conversely, ifh is an isometry such that det(h) =−1, andh(u) =u for allu∈U, then for any extesniong_1 off, the mapg_2 =hæg_1 is another extension off such that det(g_2) =−det(g_1). Therefore, we need to show that a maph as above exists.

If dim( U) + dim(rad(U))< dim(E), consider the nondegenerate completionU ofU given by Proposition 14.30. We know that dim(U) = dim(U) + dim(rad(U))< n, and sinceU is nondegenerate, we have

E =U⊥U⊥,⊕

withU⊥ = (0). Pick any isometryτ ofU⊥ such that det(τ) =−1, and extend it to an isometryh ofE whose restriction toU is the identity.

If dim(U) + dim(rad(U)) = dim(E) =n, thenU =V ⊥W withV = rad(U) and since dim(U) = dim(U) + dim(rad(U)) =n, we have⊕

E =U = (V⊕V )⊥W,⊕

where V⊕V = Ar2r =W⊥ is an Artinian space. Any isometryh ofE which is the identity onU and with det(h) =−1 is the identity onW, and thus it must mapW⊥ = Ar2r =V⊕V into itself, and the restrictionh ofh to Ar2r has det(h ) =−1. However,h is the identity onV = rad(U), a totally isotopic subspace of Ar2r of dimensionr, and by Proposition 14.44, we have det(h ) = +1, a contradiction.

It can be shown that the center of O(Õ) is{id,−id}. For further properties of orthogonal groups, see Grove [50], Jacobson [57], Taylor [104], and Artin [2].

Chapter 15 Variational Approximation of Boundary-Value Problems; Introduction to the Finite Elements Method

15.1 A One-Dimensional Problem: Bending of a Beam

Consider a beam of unit length supported at its ends in 0 and 1, stretched along its axis by a forceP, and subjected to a transverse loadf(x)dx per elementdx, as illustrated in Figure 15.1.

− P dx 01 P

f(x)dx

Figure 15.1: Vertical deflection of a beam

The bending momentu(x) at the absissax is the solution of a boundary problem (BP) of the form

−

u (x) +c(x)u(x) =f(x), 0< x < 1 u(0) =α

u(1) =β,

401 wherec(x) =P/(EI(x)), whereE is the Young’s modulus of the material of which the beam is made andI(x) is the principal moment of inertia of the cross-section of the beam at the abcissax, and withα =β = 0. For this problem, we may assume thatc(x)≥ 0 for all x∈ [0,1].

Remark: The vertical deflectionw(x) of the beam and the bending momentu(x) are related by the equation

u(x) =−EId2w.

dx^2

If we seek a solution u^2([0,1]), that is, a function whose first and second derivatives∈C

exist and are continuous, then it can be shown that the problem has a unique solution (assumingc andf to be continuous functions on [0,1]).

Except in very rare situations, this problem has no closed-form solution, so we are led to seek approximations of the solutions.

One one way to proceed is to use the finite difference method, where we discretize the problem and replace derivatives by differences. Another way is to use a variational approach. In this approach, we follow a somewhat surprising path in which we come up with a so-called “weak formulation” of the problem, by using a trick based on integrating by parts!

First, let us observe that we can always assume thatα =β = 0, by looking for a solution of the formu(x)−(α(1−x) +βx). This turns out to be crucial when we integrate by parts. There are a lot of subtle mathematical details involved to make what follows rigorous, but we here, we will take a “relaxed” approach.

First, we need to specify the space of “weak solutions.” This will be the vector spaceV of continuous functionsf on [0,1], withf(0) =f(1) = 0, and which are piecewise continuously differentiable on [0,1]. This means that there is a finite number of pointsx_0, . . . , xN+1 with x_0 = 0 andxN+1 = 1, such thatf (xi) is undefined fori = 1, . . . , N, but otherwisef is defined and continuous on each interval (xi, xi+1) fori = 0, . . . , N.1 The spaceV becomes a Euclidean vector space under the inner product

1

f, g V = (f(x)g(x) +f (x)g (x))dx,

0

for allf, g∈V . The associated norm is

1 1/2

f V = (f(x)2 +f (x)2)dx .

0

Assume thatu is a solution of our original boundary problem (BP), so that u (x) +c(x)u(x) =f(x), 0< x < 1− u(0) = 0

u(1) = 0.

1We also assume that f (x) has a limit when x tends to a boundary of (xi, xi+1).

Multiply the differential equation by any arbitrary test functionv∈V , obtaining −u (x)v(x) +c(x)u(x)v(x) =f(x)v(x), (∗) and integrate this equation! We get

1 1 1

−

u (x)v(x)dx + c(x)u(x)v(x)dx = f(x)v(x)dx. (

0 0 0 †)

Now, the trick is to use integration by parts on the first term. Recall that (u v) =u v +u v ,

and to be careful about discontinuities, write

1N xi+1

u (x)v(x)dx = u (x)v(x)dx.

0 i=0xi

Using integration by parts, we have

xi+1 xi+1 xi+1

u (x)v (x)dx

xi xi

u (x)v(x)dx = (u (x)v(x))dx− xi

xi+1

= [u (x)v(x)]x=xi+1 u (x)v (x)dxx=xi − xi

xi+1

=

u

(

x

i

+1

)

v

(

x

i

+1

)

−

u

(

x

i

)

v

(

x

i

)

−

xi

u (x)v (x)dx.

It follows that

1N xi+1

u (x)v(x)dx = u (x)v(x)dx

0 i=0xi

N xi+1

=

u

(

x

i

+1

)

v

(

x

i

+1

)

−

u

(

x

i

) v ( x

i

) −

xii=0

u (x)v (x)dx

1

=

u

(1)

v

(1)

−

u

(0)

v

(0)

−

0

u (x)v (x)dx.

However, the test functionv satisfies the boundary conditionsv(0) =v(1) = 0 (recall that v∈V ), so we get

1 1

u (x)v (x)dx.

0

u (x)v(x)dx =− 0

Consequently, the equation (†) becomes

1 1 1

u (x)v (x)dx + c(x)u(x)v(x)dx = f(x)v(x)dx,

0 0 0

or1 1

(u v +cuv)dx = fvdx, for allv∈V. (∗∗)

0 0

Thus, it is natural to introduce the bilinear forma:V×V→ R given by

1

a(u, v) = (u v +cuv)dx, for allu, v∈V ,

0

and the linear formf :V→ R given by

1

f(v) = f(x)v(x)dx, for allv∈V .

0

Then, (∗∗) becomes a(u, v) =f(v), for allv∈V.

We also introduce the energy functionJ given by

J(v) = 1a(v, v)−f(v) v∈V.

2

Then, we have the following theorem.

Theorem 15.1. Letu be any solution of the boundary problem (BP). (1) Then we have

a(u, v) =f(v), for allv∈V, (WF) where_1

a(u, v) = (u v +cuv)dx, for allu, v∈V ,

0

and

1

f(v) = f(x)v(x)dx, for allv∈V .

0

(2) Ifc(x)≥ 0 for allx∈ [0,1], then a functionu∈V is a solution of (WF) iffu minimizesJ(v) , that is,

J(u) = infJ(v),

v∈V

with

J(v) = 1a(v, v)−f(v) v∈V.

2

Furthermore,u is unique.

Proof. We already proved (1).

To prove (2), first we show that

v^2 2a(v, v), for allv∈V.V≤

For this, it suffices to prove that

1

v^2 2 (f (x))2dx, for allv∈V.V≤ 0

However, by Cauchy-Schwarz for functions, for everyx∈ [0,1], we have

x 1 11/2

v

(

x

)

|

=

v

(

t

)

dt

≤

0

|

v

(

t

)

|

dt

≤

0

|

v

( t )

2

| dt ,| 0

and so1 1

v^2 = ((v(x))2 + (v (x))2)dx≤ 2 (v (x))2dx≤ 2a(v, v),V

0 0

since_1

a(v, v) = ((v )2 +cv^2)dx.

0

Next, it is easy to check that

J(u +v)−J(u) =a(u, v)−f(v) + 1a(v, v), for allu, v∈V .

2

Then, ifu is a solution of (WF), we deduce that

) = 1a(v, v)≥ 4 v V≥ 0 for allv∈V.−J(u^1J(u +v)

2

sincea(u, v)−f(v) = 0 for allv∈V . Therefore,J achieves a minimun foru. We also have

J

(

u

+

θv

)

−

J

(

u

) =

θ

(

a

(

u, v

)

−

f

(

v

)) +

θ

2

a(v, v) for allθ∈ R,2

and soJ(u +θv)−J(u)≥ 0 for allθ∈ R. Consequently, ifJ achieves a minimum foru, thena(u, v) =f(v), which means thatu is a solution of (WF).

Finally, assuming thatc(x)≥ 0, we claim that ifv∈V andv = 0, thena(v, v)> 0. This is because ifa(v, v) = 0, since

v^2 2a(v, v) for allv∈V,V≤

we would have v V = 0, that is,v = 0. Then, ifv = 0, from

J(u +v) ) = 1a(v, v) for allv∈V−J(u

2

we see thatJ(u +v)> J(u), so the minimumu is unique Theorem 15.1 shows that every solutionu of our boundary problem (BP) is a solution (in fact, unique) of the equation (WF).

The equation (WF) is called the weak form or variational equation associated with the boundary problem. This idea to derive these equations is due to Ritz and Galerkin.

Now, the natural question is whether the variational equation (WF) has a solution, and whether this solution, if it exists, is also a solution of the boundary problem (it must belong toC^2([0,1]), which is far from obvious). Then, (BP) and (WF) would be equivalent.

Some fancy tools of analysis can be used to prove these assertions. The first difficulty is that the vector spaceV is not the right space of solutions, because in order for the variational problem to have a solution, it must be complete. So, we must construct a completion of the vector spaceV . This can be done and we get the Sobolev spaceH^1(0,1). Then, the question_0

of the regularity of the “weak solution” can also be tackled.

We will not worry about all this. Instead, let us find approximations of the problem (WF). Instead of using the infinite-dimensional vector spaceV , we consider finite-dimensional subspacesVa (with dim(Va) =n) ofV , and we consider the discrete problem:

Find a function u(a) Va, such that∈

a(u(a), v) =f(v), for allv∈Va. (DWF)

SinceVa is finite dimensional (of dimensionn), let us pick a basis of functions (w_1, . . . , wn) inVa, so that every functionu∈Va can we written as

u =u_1w_1 +· · · +unwn.

Then, the equation (DWF) holds iff a(u, wj) =f(wj), j = 1, . . . , n,

and by pluggingu_1w_1 +· · · +unwn foru, we get a system ofk linear equations

n

a(wi, wj)ui =f(wj), 1≤j≤n.

i=1

Because

a

(

v, v

)

1

≥ 2 v Va, the bilinear forma is symmetric positive definite, and thus the matrix (a(wi, wj)) is symmetric positive definite, and thus invertible. Therefore, (DWF) has a solution given by a linear system!

From a practical point of view, we have to compute the integrals

1

aij =a(wi, wj) = (wiwj +cwiwj)dx,

0

and_1

bj =f(wj) = f(x)wj(x)dx.

0

However, if the basis functions are simple enough, this can be done “by hand.” Otherwise, numerical integration methods must be used, but there are some good ones.

Let us also remark that the proof of Theorem 15.1 also shows that the unique solution of (DWF) is the unique minimizer ofJ over all functions inVa. It is also possible to compare the approximate solutionu(a) Va with the exact solutionu∈V .∈

Theorem 15.2. Supposec(x)≥ 0 for allx∈ [0,1]. For every finite-dimensional subspace Va (dim(Va) =n) ofV , for every basis (w_1, . . . , wn) ofVa, the following properties hold:

(1) There is a unique functionu(a) Va such that∈

a(u(a), v) =f(v), for allv∈Va, (DWF)

and ifu(a) =u_1w_1 +· · · +unwn, then u = (u_1, . . . , un) is the solution of the linear system

Au =b, (∗) withA = (aij) = (a(wi, wj)) andbj =f(wj), 1≤i, j≤n. Furthermore, the matrix A = (aij) is symmetric positive definite.

(2) The unique solutionu(a) Va of (DWF) is the unique minimizer ofJ overVa, that is,∈

J(u(a)) = inf J(v),

v∈Va

(3) There is a constantC independent ofVa and of the unique solutionu∈V of (WF), such that

u−u(a) C inf u−v V.V≤ v∈Va

We proved (1) and (2), but we will omit the proof of (3) which can be found in Ciarlet [22].

Let us now give examples of the subspacesVa used in practice. They usually consist of piecewise polynomial functions.

Pick an integerN≥ 1 and subdivide [0,1] intoN + 1 intervals [xi, xi+1], where 1 xi =hi, h =N + 1, i = 0, . . . , N + 1.

We will use the following fact: every polynomialP(x) of degree 2m + 1 (m≥ 0) is completely determined by its values as well as the values of its firstm derivatives at two distinct pointsα, β∈ R.

There are various ways to prove this. One way is to use the Bernstein basis, because thekth derivative of a polynomial is given by a formula in terms of its control points. For example, form = 1, every degree 3 polynomial can be written as

P(x) = (1^3b_0 + 3(1−x)2x b_1 + 3(1−x)x^2b_2 +x^3b_3,−x)

withb_0, b_1, b_2, b3∈ R, and we showed that P (0) = 3(b1−b_0) P (1) = 3(b3−b_2). GivenP(0) andP(1), we determineb_0 andb_3, and fromP (0) andP (1), we determineb_1 andb_2.

In general, for a polynomial of degreem written as

m

P(x) = bjBm (x)j

j=0

in terms of the Bernstein basis (Bm (x), . . . , Bm(x)) with0 m

Bm (x) = m (1−x)m−jxj,j j

it can be shown that thekth derivative ofP at zero is given by

k

P(k)(0) =m(mk (−1)k−ibi ,− 1)· · ·(m−k + 1)ii=0

and there is a similar formula forP(k)(1).

Actually, we need to use the Bernstein basis of polynomialsBm [r, s], wherek

m−j x−rj Bm r, s](x) = m s−x ,j [ j s−r s−r

withr < s, in which case

P(k)(0) =m(m− 1)· · ·(m−k + 1) k k (−1)k−ibi ,(s−r)k ii=0

with a similar formula forP(k)(1). In our case, we setr =xi, s =xi+1. Now, if the 2m + 2 values

P (0), P(1)(0), . . . , P(m)(0), P(1), P(1)(1), . . . , P(m)(1) are given, we obtain a triangular system that determines uniquely the 2m+ 2 control points b_0, . . . , b2m+1.

Recall thatCm([0,1]) denotes the set ofCm functionsf on [0,1], which means that f, f(1), . . . , f(m) exist are are continuous on [0,1].

We define the vector spaceVm as the subspace ofCm([0,1]) consisting of all functionsfN

such that

1. f(0) =f(1) = 0. 2. The restriction off to [xi, xi+1] is a polynomial of degree 2m + 1, fori = 0, . . . , N.

Observe that the functions inV 0 are the piecewise affine functionsf withf(0) =f(1) =N

0; an example is shown in Figure 15.2.

y

0 ih 1x

Figure 15.2: A piecewise affine function

This space has dimension N, and a basis consists of the “hat functions”wi, where the only two nonflat parts of the graph ofwi are the line segments from (xi−1,0) to (xi,1), and from (xi,1) to (xi+1,0), fori = 1, . . . , N, see Figure 15.3.

The basis functions wi have a small support, which is good because in computing the integrals givinga(wi, wj), we find that we get a tridiagonal matrix. They also have the nice property that every functionv∈V 0 has the following expression on the basis (wi):N

N

v(x) = v(ih)wi(x), x∈ [0,1]. i=1

y

wi

(i− 1)h ih (i+1)h x

Figure 15.3: A basis “hat function”

In general, it it not hard to see thatVm has dimensionmN + 2(m− 1).N

Going back to our problem (the bending of a beam), assuming thatc andf are constant functions, it is not hard to show that the linear system (∗) becomes

ë 2 + 2ch^2 1 +ch2 öë u_1 ö ëfö

6

ì ÷ ì

3 − 2ch^2 1 +ch2 ÷ì u_2 ÷ ìf÷ ì−1 +ch^2 2 +3 − 6 ÷ì ÷ 1

ì^6 ÷ì ÷ ì ÷ ì ... ... ... ÷ì ÷=hì ÷ ì ÷ì .h ì ÷ì . ÷ ì .÷ ÷ ì ÷

ì ch^2 2 + 2ch^2 1 +ch2 ÷ì ÷ ì ÷

ì 1 +6 3 − 6 ÷ìuN−1÷ ìf÷

í −ch2 2 + 2ch2øí ø í ø −1 +6 3 uN f

We can also find a basis of 2N + 2 cubic functions forV 1 consisting of functions with small support. This basis consists of theN functionsw0 N 1 and of theN + 2 functionsw uniquely determined by the following conditions:

w^0(xj) =δij, 1≤j≤N (w^0) (xj) = 0, 0≤j≤N + 1

w^1(xj) = 0, 1≤j≤N (w^1) (xj) =δij, 0≤j≤N + 1,

withδij = 1 iffi =j andδij = 0 ifi =j. Some of these functions are displayed in Figure 15.4. The functionw^0 is given explicitly by

w^0(x) = 1(x− (i− 1)h)2((2i + 1)h− 2x), (i− 1)h≤x≤ih,

h^3

w^0(x) = 1((i + 1)h−x)2(2x− (2i− 1)h), ih≤x≤ (i + 1)h,

h^3

fori = 1, . . . , N. The functionw^1 is given explicitly by

1 w^1(x) =−h2(ih−x)(x− (i− 1)h)2, (i− 1)h≤x≤ih,

and w^1(x) = 1((i + 1)h−x)2(x−ih), ih≤x≤ (i + 1)h,

h^2

forj = 0, . . . , N + 1. Furthermore, for every functionv∈V 1, we haveN

N N+1

v(x) = v(ih)w^0(x) + v jih)w^1(x), x∈ [0,1].

i=1 j=0

If we order these basis functions as

w^1, w^0, w^1, w^0, w^1, . . . , w^0 , w^1, w^1 ,0 1 1 2 2 N N N+1

we find that if c = 0, the matrixA of the system (∗) is tridiagonal by blocks, where the blocks are 2× 2, 2× 1, or 1× 2 matrices, and with single entries in the top left and bottom right corner. A different order of the basis vectors would mess up the tridiagonal block structure ofA. We leave the details as an exercise.

Let us now take a quick look at a two-dimensional problem, the bending of an elastic membrane.

y

w

0 i

w1 w1 w10 j N+1 x0 ih jh 1 Figure 15.4: The basis functionsw^0 andw^1

15.2 A Two-Dimensional Problem: An Elastic Membrane

Consider an elastic membrane attached to a round contour whose projection on the ( x_1, x_2)plane is the boundary Γ of an open, connected, bounded region & in the (x_1, x_2)-plane, as illustrated in Figure 15.5. In other words, we view the membrane as a surface consisting of the set of points (x, z) given by an equation of the form

z =u(x),

withx = (x_1, x_2)∈ &, whereu: &→ R is some sufficiently regular function, and we think ofu(x) as the vertical displacement of this membrane.

We assume that this membrane is under the action of a vertical forceτf(x)dx per surface element in the horizontal plane (whereτ is the tension of the membrane). The problem is 15.2. A TWO-DIMENSIONAL PROBLEM: AN ELASTIC MEMBRANE 413

τf(x)dx

g(y) u(x) x_2 dx& x

y Γx_1

Figure 15.5: An elastic membrane

to find the vertical displacementu as a function ofx, forx∈ &. It can be shown (under some assumptions on &, Γ, andf), thatu(x) is given by a PDE with boundary condition, of the form

−

u(x) =f(x), x∈ & u(x) =g(x), x∈ Γ,

whereg: Γ→ R represents the height of the contour of the membrane. We are looking for a functionu inC^2(&)∩C^1(&). The operator is the Laplacian, and it is given by

u(x) =∂2u(x) +∂2u(x).∂x2 ∂x2

1 2

This is an example of a boundary problem, since the solutionu of the PDE must satisfy the conditionu(x) =g(x) on the boundary of the domain &. The above equation is known as Poisson’s equation, and whenf = 0 as Laplace’s equation.

It can be proved that if the dataf, g and Γ are sufficiently smooth, then the problem has a unique solution.

To get a weak formulation of the problem, first we have to make the boundary condition homogeneous, which means thatg(x) = 0 on Γ. It turns out thatg can be extended to the whole of & as some sufficiently smooth functionh, so we can look for a solution of the form u−h, but for simplicity, let us assume that the contour of & lies in a plane parallel to the (x_1, x_2)- plane, so thatg = 0. We letV be the subspace ofC^2(&)∩C^1(&) consisting of functionsv such thatv = 0 on Γ.

As before, we multiply the PDE by a test function v∈V , getting

−u(x)v(x) =f(x)v(x),

and we “integrate by parts.” In this case, this means that we use a version of Stokes formula known as Green’s first identity, which says that

&

−

u v dx = (gradu)· (gradv)dx− (gradu)

Γ

· n vdσ

&

(wheren denotes the outward pointing unit normal to the surface). Becausev = 0 on Γ, the integralΓ drops out, and we get an equation of the form

a(u, v) =f(v) for allv∈V, wherea is the bilinear form given by

a(u, v) =∂u ∂v + ∂u ∂v dx & ∂x_1 ∂x_1 ∂x_2 ∂x_2

andf is the linear form given by f(v) = fvdx.

&

We get the same equation as in section 15.2, but over a set of functions defined on a two-dimensional domain. As before, we can choose a finite-dimensional subspaceVa ofV and consider the discrete problem with respect toVa. Again, if we pick a basis (w_1, . . . , wn) ofVa, a vectoru =u_1w_1 +· · ·+unwn is a solution of the Weak Formulation of our problem iff u = (u_1, . . . , un) is a solution of the linear system

Au =b,

withA = (a(wi, wj)) andb = (f(wj)). However, the integrals that give the entries inA and b are much more complicated.

An approach to deal with this problem is the method of finite elements. The idea is to also discretize the boundary curve Γ. If we assume that Γ is a polygonal line, then we can triangulate the domain &, and then we consider spaces of functions which are piecewise defined on the triangles of the triangulation of &. The simplest functions are piecewise affine and look like tents erected above groups of triangles. Again, we can define base functions with small support, so that the matrixA is tridiagonal by blocks.

The finite element method is a vast subject and it is presented in many books of various degrees of difficulty and obscurity. Let us simply state three important requirements of the finite element method:

1. “Good” triangulations must be found. This in itself is a vast research topic. Delaunay

triangulations are good candidates.

2. “Good” spaces of functions must be found; typically piecewise polynomials and splines.

3. “Good” bases consisting of functions will small support must be found, so that integrals can be easily computed and sparse banded matrices arise.

We now consider boundary problems where the solution varies with time.

15.3 Time-Dependent Boundary Problems: The Wave Equation

Consider a homogeneous string (or rope) of constant cross-section, of length L, and stretched (in a vertical plane) between its two ends which are assumed to be fixed and located along thex-axis atx = 0 and atx =L.

Figure 15.6: A vibrating string

The string is subjected to a transverse force τf(x)dx per element of lengthdx (where τ is the tension of the string). We would like to investigate the small displacements of the string in the vertical plane, that is, how it vibrates.

Thus, we seek a functionu(x, t) defined fort≥ 0 andx∈ [0, L], such thatu(x, t) represents the vertical deformation of the string at the abscissax and at timet. It can be shown thatu must satisfy the following PDE

1 ∂2u ∂2u(x, t) =f(x, t), 0< x < L, t > 0,c2 ∂t2 (x, t)− ∂x2

withc = τ/ρ, whereρ is the linear density of the string, known as the one-dimensional wave equation.

Furthermore, the initial shape of the string is known at t = 0, as well as the distribution of the initial velocities along the string; in other words, there are two functionsui,0 andui,1 such that

u(x,0) =ui,0(x), 0≤x≤L,

∂u(x,0) =ui,1(x), 0≤x≤L.∂t

For example, if the string is simply released from its given starting position, we haveui,1 = 0. Lastly, because the ends of the string are fixed, we must have

u(0, t) =u(L, t) = 0, t≥ 0.

Consequently, we look for a functionu: R+× [0, L]→ R satisfying the following conditions:

1 ∂2u (x, t) ∂2u

− ∂x2(x, t) =f(x, t), 0< x < L, t > 0,c2 ∂t2

u(0, t) =u(L, t) = 0, t≥ 0 (boundary condition),

u(x,0) =ui,0(x), 0≤x≤L (intitial condition), ∂u(x,0) =ui,1(x), 0≤x≤L (intitial condition).∂t

This is an example of a time-dependent boundary-value problem, with two initial conditions.

To simplify the problem, assume thatf = 0, which amounts to neglecting the effect of gravity. In this case, our PDE becomes

1 ∂2u (x, t) ∂2u

− ∂x2(x, t) = 0, 0< x < L, t > 0,c2 ∂t2

Let us try our trick of multiplying by a test functionv depending only onx,C^1 on [0, L], and such thatv(0) =v(L) = 0, and integrate by parts. We get the equation

L∂2u L

2 ∂2u(x, t)v(x)dx = 0.

0 ∂t2 (x, t)v(x)dx−c∂x20

For the first term, we get

L ∂2uL ∂2

∂t

2

(x, t)v(x)dx =∂t2[u(x, t)v(x)]dx

0 0

d

2 L

=dt2 u(x, t)v(x)dx

0

d

2

=dt2 u, v , where u, v is the inner product inL^2([0, L]). The fact that it is legitimate to move∂2/∂t^2 outside of the integral needs to be justified rigorously, but we won’t do it here.

For the second term, we get

L ∂2u(x, t)v(x)dx =∂u x=L L

− ∂x(x, t)v(x) +∂u(x, t)dv(x)dx,− 0 ∂x2 x=0 0 ∂x dx

and becausev∈V , we havev(0) =v(L) = 0, so we obtain

L∂2u L

(x, t)v(x)dx =∂u(x, t)dv(x)dx.− 0 ∂x2 0 ∂x dx

Our integrated equation becomes

d

2

L

2 ∂u(x, t)dv(x)dx = 0, for allv∈V and allt≥ 0.dt2 u, v +c∂x dx0

It is natural to introduce the bilinear forma:V×V→ R given by

L

a(u, v) =∂u(x, t)∂v(x, t)dx,

0 ∂x ∂x

where, for every t∈ R+, the functionsu(x, t) and (v, t) belong toV . Actually, we have to replaceV by the subspace of the Sobolev spaceH^1(0, L) consisting of the functions such_0

thatv(0) =v(L) = 0. Then, the weak formulation (variational formulation) of our problem is this:

Find a functionu∈V such that

d^2

2 u, v +a(u, v) = 0, for allv∈V and allt≥ 0dt

u(x,0) =ui,0(x), 0≤x≤L (intitial condition), ∂u(x,0) =ui,1(x), 0≤x≤L (intitial condition).∂t

It can be shown that there is a positive constant α > 0 such that a(u, u)2 1 for allv∈V≥α uH0

(Poincar´e’s inequality), which shows thata is positive definite onV . The above method is known as the method of Rayleigh-Ritz.

A study of the above equation requires some sophisticated tools of analysis which go far beyond the scope of these notes. Let us just say that there is a countable sequence of solutions with separated variables of the form

u(1) kπx cos kπct , u(2) = sin kπx sin kπct , k∈ N+,k = sinL L k L L

called modes (or normal modes). Complete solutions of the problem are series obtained by combining the normal modes, and they are of the form

u(x, t) = ∞ sin kπx Ak cos kπct +Bk sin kπct , k=1 L L L

where the coefficientsAk, Bk are determined from the Fourier series ofui,0 andui,1.

We now consider discrete approximations of our problem. As before, consider a finite dimensional subspaceVa ofV and assume that we have approximationsua,0 andua,1 ofui,0 andui,1. If we pick a basis (w_1, . . . , wn) ofVa, then we can write our unknown function u(x, t) as

u(x, t) =u_1(t)w_1 +· · · +un(t)wn,

whereu_1, . . . , un are functions oft. Then, if we write u = (u_1, . . . , un), the discrete version of our problem is

Ad2u u = 0,

dt2 +K

u(x,0) =ua,0(x), 0≤x≤L,

∂u(x,0) =ua,1(x), 0≤x≤L,∂t

where A = (wi, wj ) andK = (a(wi, wj)) are two symmetric matrices, called the mass matrix and the stiffness matrix, respectively. In fact, becausea and the inner product −,− are positive definite, these matrices are also positive definite.

We have made some progress since we now have a system of ODE’s, and we can solve it by analogy with the scalar case. So, we look for solutions of the form Ucosωt (or Usinωt), where U is ann-dimensional vector. We find that we should have

(K−ω^2A)Ucosωt = 0,

which implies thatω must be a solution of the equation

KU =ω^2AU.

Thus, we have to find someλ such that

KU =λAU,

a problem known as a generalized eigenvalue problem, since the ordinary eigenvalue problem forK is

K U =λU. Fortunately, becauseA is SPD, we can reduce this generalized eigenvalue problem to a standard eigenvalue problem. A good way to do so is to use a Cholesky decomposition ofA as

A =LL ,

whereL is a lower triangular matrix (see Theorem 6.10). BecauseA is SPD, it is invertible, soL is also invertible, and

KU =λAU =λLL U yields

L−1KU =λL U, which can also be written as

L−1K(L )−1L U =λL U.

Then, if we make the change of variable Y =L U,

using the fact (L )−1 = (L−1) , the above equation is equivalent to

L−1K(L−1) Y =λY,

a standard eigenvalue problem for the matrix K =L−1K(L−1) . Furthermore, we know from Section 6.3 that sinceK is SPD andL−1 is invertible, the matrixK =L−1K(L−1) is also SPD.

Consequently, K has positive real eigenvalues (ω^1, . . . , ω^2) (not necessarily distinct) and1 n

it can be diagonalized with respect to an orthonormal basis of eigenvectors, say Y^1, . . . ,Yn. Then, since Y =L U, the vectors

Ui = (L )−1Yi, i = 1, . . . , n,

are linearly independent and are solutions of the generalized eigenvalue problem; that is, KUi =ω^2AUi, i = 1, . . . , n.

More is true. Because the vectors Y^1, . . . ,Yn are orthonormal, and because Yi =L Ui, from

(Yi) Yj =δij, we get

(Ui) LL Uj =δij, 1≤i, j≤n, and sinceA =LL , this yields

(Ui) AUj =δij, 1≤i, j≤n. This suggests defining the functionsUi Va by∈

n

Ui = Uikwk.

k=1

Then, it immediate to check that a(Ui, Uj) = (Ui) AUj =δij,

which means that the functions (U^1, . . . , Un) form an orthormal basis ofVa for the inner producta. The functionsUi Va are called modes (or modal vectors).∈

As a final step, let us look again for a solution of our discrete weak formulation of the problem, this time expressing the unknown solutionu(x, t) over the modal basis (U^1, . . . , Un), sayn

u = uj(t)Uj,

j=1

where eachuj is a function oft. Because

n n n n n

u = uj(t)Uj = uj(t) Ujwk = uj(t)Uj wk,k k

j=1 j=1 k=1 k=1 j=1

if we write u = (u_1, . . . , un) withuk =n uj(t)Uj fork = 1, . . . , n, we see thatj=1 k

n

u = ujUj,

j=1

so using the fact that KUj =ω^2AUj, j = 1, . . . , n, the equation

Ad2u u = 0 dt2 +K

yieldsn

[(uj) +ω^2uj]AUj = 0.

j=1

SinceA is invertible and since (U^1, . . . ,Un) are linearly independent, the vectors (AU^1, . . . , AUn) are linearly independent, and consequently we get the system ofn ODEs’ (uj) +ω^2uj = 0, 1≤j≤n.

Each of these equation has a well-known solution of the form

uj =Aj cosωjt +Bj sinωjt.

Therefore, the solution of our approximation problem is given by

n

u = (Aj cosωjt +Bj sinωjt)Uj,

j=1

and the constantsAj, Bj are obtained from the intial conditions u(x,0) =ua,0(x), 0≤x≤L,

∂u(x,0) =ua,1(x), 0≤x≤L,∂t

by expressingua,0 andua,1 on the modal basis (U^1, . . . , Un). Furthermore, the modal functions (U^1, . . . , Un) form an orthonormal basis ofVa for the inner producta. If we use the vector spaceV 0 of piecewise affine functions, we find that the matricesAN

andK are familar! Indeed,

ë 2 −1 0 0 0ö ì 1 2 − 1 0 0÷

= 1

ì− ... ... ...÷

.

÷A ì

ì . ÷

h ì 0 0 −1 2 −1÷ í0 0 ø 0 −1 2

and ë4 1 0 0 0ö ì1 4 1 0 0÷

÷

K =ìh ì ... ... ... .÷.6 ì.÷ ì0 0 1 4 1÷ í ø 0 0 0 1 4

To conclude this section, let us discuss briefly the wave equation for an elastic membrane, as described in Section 15.2. This time, we look for a functionu: R+× &→ R satisfying the following conditions:

1 ∂2u (x, t)− u(x, t) =f(x, t), x∈ &, t > 0,c2 ∂t2

u(x, t) = 0, x∈ Γ, t≥ 0 (boundary condition), u(x,0) =ui,0(x), x∈ & (intitial condition), ∂u(x,0) =ui,1(x), x∈ & (intitial condition).∂t

Assuming that f = 0, we look for solutions in the subspaceV of the Sobolev spaceH^1(&)0 consisting of functionsv such thatv = 0 on Γ. Multiplying by a test functionv∈V and using Green’s first identity, we get the weak formulation of our problem:

Find a functionu∈V such that

d^2

dt2 u, v +a(u, v) = 0, for allv∈V and allt≥ 0 u(x,0) =ui,0(x), x∈ & (intitial condition), ∂u(x,0) =ui,1(x), x∈ & (intitial condition),∂t

wherea:V×V→ R is the bilinear form given by

a(u, v) =∂u ∂v + ∂u ∂v dx, & ∂x_1 ∂x_1 ∂x_2 ∂x_2 and u, v = uvdx.

&

As usual, we find approximations of our problem by using finite dimensional subspaces Va ofV . Picking some basis (w_1, . . . , wn) ofVa, and triangulating &, as before, we obtain the equation

Ad2u +Ku = 0,

dt^2

u(x,0) =ua,0(x), x∈ Γ,

∂u(x,0) =ua,1(x), x∈ Γ,∂t

whereA = (wi, wj ) andK = (a(wi, wj)) are two symmetric positive definite matrices.

In principle, the problem is solved, but, it may be difficult to find good spaces Va, good triangulations of &, and good bases ofVa, to be able to compute the matricesA andK, and to ensure that they are sparse.

Chapter 16 Singular Value Decomposition and Polar Form

16.1 Singular Value Decomposition for Square Matrices

In this section we assume that we are dealing with a real Euclidean space E. Letf :E→E be any linear map. In general, it may not be possible to diagonalizef. We show that every linear map can be diagonalized if we are willing to use two orthonormal bases. This is the celebrated singular value decomposition (SVD). A close cousin of the SVD is the polar form of a linear map, which shows how a linear map can be decomposed into its purely rotational component (perhaps with a flip) and its purely stretching part.

The key observation is thatf∗æf is self-adjoint, since

(f∗æf)(u), v = f(u), f(v) = u,(f∗æf)(v).

Similarly,fæf∗ is self-adjoint.

The fact that f∗æf andfæf∗ are self-adjoint is very important, because it implies that f∗æf andfæf∗ can be diagonalized and that they have real eigenvalues. In fact, these eigenvalues are all nonnegative. Indeed, ifu is an eigenvector off∗æf for the eigenvalueλ, then

(f∗æf)(u), u = f(u), f(u) and

(f∗æf)(u), u =λ u, u , and thus

λ u, u = f(u), f(u),

which implies thatλ≥ 0, since−,− is positive definite. A similar proof applies tofæf∗. Thus, the eigenvalues off∗æf are of the formσ^2, . . . , σ^2 or 0, whereσi> 0, and similarly_1

423 forfæf∗. The situation is even better, since we will show shortly thatf∗æf andfæf∗ have the same eigenvalues.

Remark: Given any two linear mapsf :E→F andg:F→E, where dim(E) =n and dim(F) =m, it can be shown that

(−λ)m det(gæf−λ In) = (−λ)n det(fæg−λ Im),

and thusgæf andfæg always have the same nonzero eigenvalues!

Definition 16.1. The square rootsσi> 0 of the positive eigenvalues off∗æf (andfæf∗) are called the singular values off.

Definition 16.2. A self-adjoint linear mapf :E→E whose eigenvalues are nonnegative is called positive semidefinite (or positive), and iff is also invertible,f is said to be positive definite. In the latter case, every eigenvalue off is strictly positive.

We just showed that f∗æf andfæf∗ are positive semidefinite self-adjoint linear maps. This fact has the remarkable consequence that every linear map has two important decompositions:

1. The polar form.

2. The singular value decomposition (SVD).

The wonderful thing about the singular value decomposition is that there exist two orthonormal bases (u_1, . . . , un) and (v_1, . . . , vn) such that, with respect to these bases,f is a diagonal matrix consisting of the singular values off, or 0. Thus, in some sense,f can always be diagonalized with respect to two orthonormal bases. The SVD is also a useful tool for solving overdetermined linear systems in the least squares sense and for data analysis, as we show later on.

First, we show some useful relationships between the kernels and the images of f,f∗, f∗æf, andfæf∗. Recall that iff :E→F is a linear map, the image Imf off is the subspacef(E) ofF, and the rank off is the dimension dim(Imf) of its image. Also recall that (Theorem 4.11)

dim (Kerf) + dim (Imf) = dim (E), and that (Propositions 9.9 and 11.10) for every subspaceW ofE,

dim ( W) + dim (W⊥) = dim (E). Proposition 16.1. Given any two Euclidean spacesE andF, whereE has dimensionn andF has dimensionm, for any linear mapf :E→F, we have

Ker f = Ker (f∗æf), Kerf∗ = Ker (fæf∗),

Kerf = (Imf∗)⊥, Kerf∗ = (Imf)⊥, dim(Imf) = dim(Imf∗),

andf,f∗,f∗æf, andfæf∗ have the same rank.

Proof. To simplify the notation, we will denote the inner products onE andF by the same symbol−,− (to avoid subscripts). Iff(u) = 0, then (f∗æf)(u) =f∗(f(u)) =f∗(0) = 0, and so Kerf⊆ Ker (f∗æf). By definition off∗, we have

f(u), f(u) = (f∗æf)(u), u

for allu∈E. If (f∗æf)(u) = 0, since−,− is positive definite, we must havef(u) = 0, and so Ker (f∗æf)⊆ Kerf. Therefore,

Kerf = Ker (f∗æf).

The proof that Kerf∗ = Ker (fæf∗) is similar. By definition off∗, we have

f(u), v = u, f∗(v) for allu∈E and allv∈F . (∗)

This immediately implies that

Kerf = (Imf∗)⊥ and Kerf∗ = (Imf)⊥.

Let us explain why Kerf = (Imf∗)⊥, the proof of the other equation being similar.

Because the inner product is positive definite, for every u∈E, we have u∈ Kerf

ifff(u) = 0

iff f(u), v = 0 for allv,

by (∗) iff u, f∗(v) = 0 for allv,

iffu∈ (Imf∗)⊥.

Since

dim(Imf) =n− dim(Kerf) and

dim(Imf∗) =n− dim((Imf∗)⊥), from Kerf = (Imf∗)⊥ we also have dim(Kerf) = dim((Imf∗)⊥), from which we obtain dim(Imf) = dim(Imf∗). Since dim(Ker (f∗æf)) + dim(Im (f∗æf)) = dim(E), Ker (f∗æf) = Kerf and Kerf = (Imf∗)⊥, we get

dim((Imf∗)⊥) + dim(Im (f∗æf)) = dim(E).

Since dim((Imf∗)⊥) + dim(Imf∗) = dim(E), we deduce that dim(Imf) = dim(Im (f∗æf)). A similar proof shows that

dim(Imf∗) = dim(Im (fæf∗)).

Consequently,f,f∗,f∗æf, andfæf∗ have the same rank.

We will now prove that every square matrix has an SVD. Stronger results can be obtained if we first consider the polar form and then derive the SVD from it (there are uniqueness properties of the polar decomposition). For our purposes, uniqueness results are not as important so we content ourselves with existence results, whose proofs are simpler. Readers interested in a more general treatment are referred to [42].

The early history of the singular value decomposition is described in a fascinating paper by Stewart [97]. The SVD is due to Beltrami and Camille Jordan independently (1873, 1874). Gauss is the grandfather of all this, for his work on least squares (1809, 1823) (but Legendre also published a paper on least squares!). Then come Sylvester, Schmidt, and Hermann Weyl. Sylvester’s work was apparently “opaque.” He gave a computational method to find an SVD. Schmidt’s work really has to do with integral equations and symmetric and asymmetric kernels (1907). Weyl’s work has to do with perturbation theory (1912). Autonne came up with the polar decomposition (1902, 1915). Eckart and Young extended SVD to rectangular matrices (1936, 1939).

Theorem 16.2. (Singular value decomposition) For every realn×n matrixA there are two orthogonal matricesU andV and a diagonal matrixD such thatA =V DU , whereD is of the formëσ1 . . . ö

ì

÷

ì

σ

2

. . .

÷

D =ì .... ÷ ,

í . . ø

. . . σn

where σ_1, . . . , σr are the singular values off, i.e., the (positive) square roots of the nonzero eigenvalues ofA A andA A , andσr+1 =· · · =σn = 0. The columns ofU are eigenvectors ofA A, and the columns ofV are eigenvectors ofA A .

Proof. SinceA A is a symmetric matrix, in fact, a positive semidefinite matrix, there exists an orthogonal matrixU such that

A A =UD^2U ,

withD = diag(σ_1, . . . , σr,0, . . . ,0), whereσ^2, . . . , σ^2 are the nonzero eigenvalues ofA A,1

and wherer is the rank ofA; that is,σ_1, . . . , σr are the singular values ofA. It follows that

U A AU = (AU) AU =D^2,

and if we letfj be thejth column ofAU forj = 1, . . . , n, then we have f^2δij, 1≤i, j≤ri, fj =σ

and fj = 0, r + 1≤j≤n. If we define (v_1, . . . , vr) by

v

j

=

σ

1

j fj, 1≤j≤r, then we have

vi, vj =δij, 1≤i, j≤r,

so complete (v_1, . . . , vr) into an orthonormal basis (v_1, . . . , vr, vr+1, . . . , vn) (for example, using Gram–Schmidt). Now, sincefj =σjvj forj = 1. . . , r, we have

vi, fj =σjvi, vj =σjδi,j, 1≤i≤n, 1≤j≤r

and sincefj = 0 forj =r + 1, . . . , n,

vi, fj = 0 1≤i≤n, r + 1≤j≤n.

IfV is the matrix whose columns arev_1, . . . , vn, thenV is orthogonal and the above equations prove that

V AU =D, which yieldsA =V DU , as required.

The equation A =V DU implies that

A A =UD^2U , AA =V D^2V ,

which shows thatA A andAA have the same eigenvalues, that the columns ofU are eigenvectors ofA A, and that the columns ofV are eigenvectors ofAA . Theorem 16.2 suggests the following definition.

Definition 16.3. A triple (U, D, V ) such thatA =V D U , whereU andV are orthogonal andD is a diagonal matrix whose entries are nonnegative (it is positive semidefinite) is called a singular value decomposition (SVD) ofA.

The proof of Theorem 16.2 shows that there are two orthonormal bases ( u_1, . . . , un) and (v_1, . . . , vn), where (u_1, . . . , un) are eigenvectors ofA A and (v_1, . . . , vn) are eigenvectors ofAA . Furthermore, (u_1, . . . , ur) is an orthonormal basis of ImA , (ur+1, . . . , un) is an orthonormal basis of KerA, (v_1, . . . , vr) is an orthonormal basis of ImA, and (vr+1, . . . , vn) is an orthonormal basis of KerA .

Using a remark made in Chapter 3, if we denote the columns ofU byu_1, . . . , un and the columns ofV byv_1, . . . , vn, then we can write

A =V D U =σ_1v_1u_1 +· · · +σrvrur.

As a consequence, if r is a lot smaller thann (we writer n), we see thatA can be reconstructed fromU andV using a much smaller number of elements. This idea will be used to provide “low-rank” approximations of a matrix. The idea is to keep only thek top singular values for some suitablek r for whichσk+1, . . . σr are very small.

Remarks:

(1) In Strang [101] the matrices U, V, D are denoted byU =Q_2,V =Q_1, andD = Σ, and an SVD is written asA =Q_1ΣQ_2 . This has the advantage thatQ_1 comes beforeQ_2 in A =Q_1ΣQ_2 . This has the disadvantage thatA maps the columns ofQ_2 (eigenvectors ofA A) to multiples of the columns ofQ_1 (eigenvectors ofA A ).

(2) Algorithms for actually computing the SVD of a matrix are presented in Golub and Van Loan [47], Demmel [25], and Trefethen and Bau [106], where the SVD and its applications are also discussed quite extensively.

(3) The SVD also applies to complex matrices. In this case, for every complexn×n matrix A, there are two unitary matricesU andV and a diagonal matrixD such that A =V D U∗,

where D is a diagonal matrix consisting of real entriesσ_1, . . . , σn, whereσ_1, . . . , σr are the singular values ofA, i.e., the positive square roots of the nonzero eigenvalues of A∗A andA A∗, andσr+1 =. . . =σn = 0.

A notion closely related to the SVD is the polar form of a matrix.

Definition 16.4. A pair (R, S) such thatA =RS withR orthogonal andS symmetric positive semidefinite is called a polar decomposition ofA.

Theorem 16.2 implies that for every realn×n matrixA, there is some orthogonal matrix R and some positive semidefinite symmetric matrixS such that

A =RS.

This is easy to show and we will prove it below. Furthermore, R, S are unique ifA is invertible, but this is harder to prove.

For example, the matrixë1 1 1 1 ö

= 1ì1 1 −1 −1÷Aì1 −1 1 −1÷

2 í ø

1 −1 −1 1

is both orthogonal and symmetric, andA =RS withR =A andS =I, which implies that some of the eigenvalues ofA are negative.

Remark: In the complex case, the polar decomposition states that for every complexn×n matrixA, there is some unitary matrixU and some positive semidefinite Hermitian matrix H such that

A =UH.

It is easy to go from the polar form to the SVD, and conversely.

Given an SVD decompositionA =V D U , letR =V U andS =UD U . It is clear thatR is orthogonal and thatS is positive semidefinite symmetric, and

RS =V U UD U =V D U =A.

Going the other way, given a polar decomposition A =R_1S, whereR_1 is orthogonal andS is positive semidefinite symmetric, there is an orthogonal matrixR_2 and a positive semidefinite diagonal matrixD such thatS =R_2D R_2 , and thus

A =R_1R_2D R_2 =V D U ,

whereV =R_1R_2 andU =R_2 are orthogonal. The eigenvalues and the singular values of a matrix are typically not related in any obvious way. For example, then×n matrix

ë 1 2 0 0 . . . 0 0ö

ì0 1 2 0 . . . 0 0÷

ì0 0 1 2 . . . 0 0÷

ì ÷

ì ... ... ...÷

.

.

÷A =ì

ì. . ÷

ì0 0 . . . 0 1 2 0÷

ì ÷

ì0 0 . . . 0 0 1 2÷

í ø

0 0 . . . 0 0 0 1

has the eigenvalue 1 with multiplicityn, but its singular values,σ1≥ · · · ≥σn, which are the positive square roots of the eigenvalues of the matrixB =A A with

ë1 2 0 0 . . . 0 0ö ì2 5 2 0 . . . 0 0÷

ì ÷ ì

÷ ì

0

2

5

2

. . .

0 0

÷

ì ... ... .... .÷B =ì. . ÷

ì0 0 . . . 2 5 2 0÷

ì ÷

ì0 0 . . . 0 2 5 2÷ í ø 0 0 . . . 0 0 2 5 have a wide spread, sinceσ1 = cond2(A)≥ 2n−1.σn

IfA is a complexn× n matrix, the eigenvaluesλ_1, . . . , λn and the singular values σ1≥σ2≥ · · · ≥σn ofA are not unrelated, since

2

1· · · nσ^2 σ^2 = det(A∗A) =|det(A)|

and

|λ1| · · · |λn| =|det(A)|, so we have

|λ1| · · · |λn| =σ1· · ·σn.

More generally, Hermann Weyl proved the following remarkable theorem: Theorem 16.3. (Weyl’s inequalities, 1949 ) For any complexn×n matrix,A, ifλ_1, . . . , λn∈ C are the eigenvalues ofA andσ_1, . . . , σn∈ R+ are the singular values ofA, listed so that |λ1| ≥ · · · ≥ |λn| andσ1≥ · · · ≥σn≥ 0, then

|

λ1| · · · |λn| =σ1· · ·σn and

λ1| · · · |λk| ≤σ1· · ·σk, for k = 1, . . . , n− 1.|

A proof of Theorem 16.3 can be found in Horn and Johnson [56], Chapter 3, Section 3.3, where more inequalities relating the eigenvalues and the singular values of a matrix are given.

Theorem 16.2 can be easily extended to rectangular m×n matrices, as we show in the next section (for various versions of the SVD for rectangular matrices, see Strang [101] Golub and Van Loan [47], Demmel [25], and Trefethen and Bau [106]).

16.2 Singular Value Decomposition for Rectangular Matrices

Here is the generalization of Theorem 16.2 to rectangular matrices.

Theorem 16.4. (Singular value decomposition) For every realm×n matrixA, there are two orthogonal matricesU (n×n) andV (m×m) and a diagonalm×n matrixD such that A =V D U , whereD is of the form

ëσ1 . . . ö

ì σ_2 . . .÷

ì... ÷

.

÷ ëσ_1 . . . 0 . . . 0öì

ì . . ÷

ì . . . σ÷ ì

÷ ì

÷

ì

n

÷

or

D

=

ì

σ

2

. . .

0

. . .

0

÷

D =ì .... 0 . 0÷, ì 0 . . . . 0÷ í. . øì ÷

ì ÷ . . . σm 0 . . . 0ì . . .... ÷

í ø

0 . . . . 0

where σ_1, . . . , σr are the singular values off, i.e. the (positive) square roots of the nonzero eigenvalues ofA A andA A , andσr+1 =. . . =σp = 0, wherep = min(m, n). The columns ofU are eigenvectors ofA A, and the columns ofV are eigenvectors ofA A .

Proof. As in the proof of Theorem 16.2, sinceA A is symmetric positive semidefinite, there exists ann×n orthogonal matrixU such that

A A =UΣ^2U ,

with Σ = diag( σ_1, . . . , σr,0, . . . ,0), whereσ^2, . . . , σ^2 are the nonzero eigenvalues ofA A,1

and wherer is the rank ofA. Observe thatr≤ min{m, n}, andAU is anm×n matrix. It follows that

U A AU = (AU) AU = Σ^2,

and if we letfj∈ Rm be thejth column ofAU forj = 1, . . . , n, then we have

fi, fj =σ^2δij, 1≤i, j≤r and fj = 0, r + 1≤j≤n. If we define (v_1, . . . , vr) by

v

j

=

σ

1

j fj, 1≤j≤r, then we have

vi, vj =δij, 1≤i, j≤r,

so complete (v_1, . . . , vr) into an orthonormal basis (v_1, . . . , vr, vr+1, . . . , vm) (for example, using Gram–Schmidt).

Now, sincefj =σjvj forj = 1. . . , r, we have

vi, fj =σjvi, vj =σjδi,j, 1≤i≤m, 1≤j≤r and sincefj = 0 forj =r + 1, . . . , n, we have

vi, fj = 0 1≤i≤m, r + 1≤j≤n. IfV is the matrix whose columns arev_1, . . . , vm, thenV is anm×m orthogonal matrix and ifm≥n, we letëσ1 . . . ö

ì

÷

ì

σ

2

. . .

÷

ì .... ÷

ì . . ÷

ì ÷

D

=

ì ÷

Σ = ì . . . σn÷ ,0m−n ì 0 . . . . 0÷

ì ÷

ì .... ÷

í. . ø

0 . . . . 0

else ifn≥m, then we let

÷

ì

ëσ_1 . . . 0 . . . 0ö ì σ_2 . . . 0 . . . 0÷ D

=

ì... ÷

.

0

. 0

÷ .ì

í . . ø. . . σm 0 . . . 0

In either case, the above equations prove that V AU =D,

which yieldsA =V DU , as required.

The equationA =V DU implies that

A A =UD DU =Udiag(σ^2, . . . , σ^2,0, . . . ,0)U_1

n−r

and AA =V DD V =V diag(σ^2, . . . , σ^2,0, . . . ,0)V ,1

m−r

which shows thatA A andAA have the same nonzero eigenvalues, that the columns ofU are eigenvectors ofA A, and that the columns ofV are eigenvectors ofAA .

A triple (U, D, V ) such thatA =V D U is called a singular value decomposition (SVD) ofA.

Even though the matrixD is anm×n rectangular matrix, since its only nonzero entries are on the descending diagonal, we still say thatD is a diagonal matrix.

If we view A as the representation of a linear mapf :E→F, where dim(E) =n and dim(F) =m, the proof of Theorem 16.4 shows that there are two orthonormal bases (u_1, . . ., un) and (v_1, . . . , vm) forE andF, respectively, where (u_1, . . . , un) are eigenvectors off∗æf and (v_1, . . . , vm) are eigenvectors offæf∗. Furthermore, (u_1, . . . , ur) is an orthonormal basis of Imf∗, (ur+1, . . . , un) is an orthonormal basis of Kerf, (v_1, . . . , vr) is an orthonormal basis of Imf, and (vr+1, . . . , vm) is an orthonormal basis of Kerf∗.

The SVD of matrices can be used to define the pseudo-inverse of a rectangular matrix; we will do so in Chapter 17. The reader may also consult Strang [101], Demmel [25], Trefethen and Bau [106], and Golub and Van Loan [47].

One of the spectral theorems states that a symmetric matrix can be diagonalized by an orthogonal matrix. There are several numerical methods to compute the eigenvalues of a symmetric matrixA. One method consists in tridiagonalizing A, which means that there exists some orthogonal matrixP and some symmetric tridiagonal matrixT such that A =P T P . In fact, this can be done using Householder transformations. It is then possible to compute the eigenvalues ofT using a bisection method based on Sturm sequences. One can also use Jacobi’s method. For details, see Golub and Van Loan [47], Chapter 8, Demmel [25], Trefethen and Bau [106], Lecture 26, or Ciarlet [22]. Computing the SVD of a matrixA is more involved. Most methods begin by finding orthogonal matricesU andV and a bidiagonal matrixB such thatA =V BU . This can also be done using Householder transformations. Observe thatB B is symmetric tridiagonal. Thus, in principle, the previous method to diagonalize a symmetric tridiagonal matrix can be applied. However, it is unwise to compute B B explicitly, and more subtle methods are used for this last step. Again, see Golub and Van Loan [47], Chapter 8, Demmel [25], and Trefethen and Bau [106], Lecture 31.

The polar form has applications in continuum mechanics. Indeed, in any deformation it is important to separate stretching from rotation. This is exactly whatQS achieves. The orthogonal partQ corresponds to rotation (perhaps with an additional reflection), and the symmetric matrixS to stretching (or compression). The real eigenvaluesσ_1, . . . , σr ofS are the stretch factors (or compression factors) (see Marsden and Hughes [73]). The fact that S can be diagonalized by an orthogonal matrix corresponds to a natural choice of axes, the principal axes.

The SVD has applications to data compression, for instance in image processing. The idea is to retain only singular values whose magnitudes are significant enough. The SVD can also be used to determine the rank of a matrix when other methods such as Gaussian elimination produce very small pivots. One of the main applications of the SVD is the computation of the pseudo-inverse. Pseudo-inverses are the key to the solution of various optimization problems, in particular the method of least squares. This topic is discussed in the next chapter (Chapter 17). Applications of the material of this chapter can be found in Strang [101, 100]; Ciarlet [22]; Golub and Van Loan [47], which contains many other references; Demmel [25]; and Trefethen and Bau [106].

16.3 Ky Fan Norms and Schatten Norms

The singular values of a matrix can be used to define various norms on matrices which have found recent applications in quantum information theory and in spectral graph theory. Following Horn and Johnson [56] (Section 3.4) we can make the following definitions:

Definition 16.5. For any matrixA∈ Mm,n(C), letq = min{m, n}, and ifσ1≥ · · · ≥σq are the singular values ofA, for anyk with 1≤k≤q, let

Nk(A) =σ_1 +· · · +σk,

called the Ky Fank-norm ofA.

More generally, for anyp≥ 1 and anyk with 1≤k≤q, let

Np +· · · +σp)1/p,k;p(A) = (σ1 k

called the Ky Fanp-k-norm ofA. Whenk =q,Nq;p is also called the Schattenp-norm.

Observe that when k = 1,N_1(A) =σ_1, and the Ky Fan normN_1 is simply the spectral norm from Chapter 7, which is the subordinate matrix norm associated with the Euclidean norm. Whenk =q, the Ky Fan normNq is given by

Nq(A) =σ_1 +· · · +σq = tr((A∗A)1/2)

and is called the trace norm or nuclear norm. Whenp = 2 andk =q, the Ky FanNq;2 norm is given by

Nk;2(A) = (σ^2 +· · · +σ^2)1/2 = tr(A∗A) = A F,1

which is the Frobenius norm ofA.

It can be shown that Nk andNk;p are unitarily invariant norms, and that whenm =n, they are matrix norms; see Horn and Johnson [56] (Section 3.4, Corollary 3.4.4 and Problem 3).

16.4. SUMMARY 435

16.4 Summary

The main concepts and results of this chapter are listed below:

•

For any linear mapf :E→E on a Euclidean spaceE, the mapsf∗æf andfæf∗ are self-adjoint and positive semidefinite.

The singular values of a linear map.•

• Positive semidefinite and positive definite self-adjoint maps.

Relationships between Imf, Kerf, Imf∗, and Kerf∗.•

• The singular value decomposition theorem for square matrices (Theorem 16.2). The SVD of matrix.•

• The polar decomposition of a matrix.

The Weyl inequalities.•

• The singular value decomposition theorem form×n matrices (Theorem 16.4). Ky Fank-norms, Ky Fanp-k-norms, Schattenp-norms.•

Chapter 17 Applications of SVD and Pseudo-inverses

De tous les principes qu’on peut proposer pour cet objet, je pense qu’il n’en est pas de plus g´en´eral, de plus exact, ni d’une application plus facile, que celui dont nous avons fait usage dans les recherches p´ec´edentes, et qui consiste `a rendre minimum la somme des carr´es des erreurs. Par ce moyen il s’´etablit entre les erreurs une sorte d’´equilibre qui, empˆechant les extrˆemes de pr´evaloir, est tr`es propre `a faire connaitre l’´etat du syst`eme le plus proche de la v´erit´e.

—Legendre, 1805, Nouvelles M´ethodes pour la d´etermination des Orbites des Com`etes

17.1 Least Squares Problems and the Pseudo-inverse

This chapter presents several applications of SVD. The first one is the pseudo-inverse, which plays a crucial role in solving linear systems by the method of least squares. The second application is data compression. The third application is principal component analysis (PCA), whose purpose is to identify patterns in data and understand the variance–covariance structure of the data. The fourth application is the best affine approximation of a set of data, a problem closely related to PCA.

The method of least squares is a way of “solving” an overdetermined system of linear equations

Ax =b,

i.e., a system in which A is a rectangularm×n matrix with more equations than unknowns (whenm > n). Historically, the method of least squares was used by Gauss and Legendre to solve problems in astronomy and geodesy. The method was first published by Legendre in 1805 in a paper on methods for determining the orbits of comets. However, Gauss had already used the method of least squares as early as 1801 to determine the orbit of the asteroid

437 Ceres, and he published a paper about it in 1810 after the discovery of the asteroid Pallas. Incidentally, it is in that same paper that Gaussian elimination using pivots is introduced.

The reason why more equations than unknowns arise in such problems is that repeated measurements are taken to minimize errors. This produces an overdetermined and often inconsistent system of linear equations. For example, Gauss solved a system of eleven equations in six unknowns to determine the orbit of the asteroid Pallas. As a concrete illustration, suppose that we observe the motion of a small object, assimilated to a point, in the plane. From our observations, we suspect that this point moves along a straight line, say of equation y =dx +c. Suppose that we observed the moving point at three different locations (x_1, y_1), (x_2, y_2), and (x_3, y_3). Then we should have

c +dx_1 =y_1,

c +dx_2 =y_2,

c +dx_3 =y_3.

If there were no errors in our measurements, these equations would be compatible, and c andd would be determined by only two of the equations. However, in the presence of errors, the system may be inconsistent. Yet we would like to findc andd!

The idea of the method of least squares is to determine (c, d) such that it minimizes the sum of the squares of the errors, namely,

(c +dx1−y_1)2 + (c +dx2−y_2)2 + (c +dx3−y_3)2.

In general, for an overdeterminedm×n systemAx =b, what Gauss and Legendre discovered is that there are solutionsx minimizing

Ax−b^2 (where u^2 =u^2+· · ·+u^2n, the square of the Euclidean norm of the vectoru = (u_1, . . . , un)),1

and that these solutions are given by the squaren×n system

A Ax =A b,

called the normal equations. Furthermore, when the columns ofA are linearly independent, it turns out thatA A is invertible, and sox is unique and given by

x = (A A)−1A b.

Note thatA A is a symmetric matrix, one of the nice features of the normal equations of a least squares problem. For instance, the normal equations for the above problem are 3 x_1 +x_2 +x_3 c= y_1 +y_2 +y_3

x

1

y

1

+

x

2

y

2

+

x

3

y

3 1

+

x

2 +x^2 d .x_1 +x_2 +x_3 x^2

2 3

In fact, given any real m×n matrixA, there is always a uniquex+ of minimum norm that minimizes Ax−b^2, even when the columns ofA are linearly dependent. How do we prove this, and how do we findx+?

Theorem 17.1. Every linear systemAx =b, whereA is anm×n matrix, has a unique least squares solutionx+ of smallest norm.

Proof. Geometry offers a nice proof of the existence and uniqueness ofx+. Indeed, we can interpretb as a point in the Euclidean (affine) space Rm, and the image subspace ofA (also called the column space ofA) as a subspaceU of Rm (passing through the origin). Then, we claim thatx minimizes Ax−b^2 iffAx is the orthogonal projectionp ofb onto the subspace U, which is equivalent to−→ =b−Ax being orthogonal toU.

First of all, if U⊥ is the vector space orthogonal toU, the affine spaceb +U⊥ intersects U in a unique pointp (this follows from Lemma 19.15 (2)). Next, for any pointy∈U, the vectorspy and−→ are orthogonal, which implies that−→

by 2 =bp 2 +py 2.−→ −→−→

Thus,p is indeed the unique point inU that minimizes the distance fromb to any point in U.

To show that there is a uniquex+ of minimum norm minimizing the (square) error Ax−b^2, we use the fact that

Rn = KerA⊕ (KerA)⊥.

Indeed, everyx∈ Rn can be written uniquely asx =u +v, whereu∈ KerA andv∈(KerA)⊥, and sinceu andv are orthogonal,

x^2 = u^2 + v^2.

Furthermore, since u∈ KerA, we haveAu = 0, and thusAx =p iffAv =p, which shows that the solutions ofAx =p for whichx has minimum norm must belong to (KerA)⊥. However, the restriction ofA to (KerA)⊥ is injective. This is because ifAv_1 =Av_2, where v_1, v2∈ (KerA)⊥, thenA(v2−v_2) = 0, which impliesv2−v1 ∈ KerA, and sincev_1, v2∈ (KerA)⊥, we also havev2−v1∈ (KerA)⊥, and consequently,v2−v_1 = 0. This shows that there is a uniquex of minimum norm minimizing Ax−b^2, and that it must belong to (KerA)⊥.

The proof also shows that x minimizes Ax−b^2 iff−→ =b−Ax is orthogonal toU, which can be expressed by saying thatb−Ax is orthogonal to every column ofA. However, this is equivalent to

A (b−Ax) = 0, i.e., A Ax =A b. Finally, it turns out that the minimum norm least squares solutionx+ can be found in terms of the pseudo-inverseA+ ofA, which is itself obtained from any SVD ofA. Definition 17.1. Given anym×n matrixA, ifA =V DU is an SVD ofA with

D = diag(λ_1, . . . , λr,0, . . . ,0),

whereD is anm×n matrix andλi> 0, if we let

D+ = diag(1/λ_1, . . . ,1/λr,0, . . . ,0),

ann×m matrix, the pseudo-inverse ofA is defined by

A+ =UD+V .

Actually, it seems thatA+ depends on the specific choice ofU andV in an SVD (U, D, V ) forA, but the next theorem shows that this is not so.

Theorem 17.2. The least squares solution of smallest norm of the linear systemAx =b, whereA is anm×n matrix, is given by

x+ =A+b =UD+V b.

Proof. First, assume thatA is a (rectangular) diagonal matrixD, as above. Then, sincex minimizes Dx−b^2 iffDx is the projection ofb onto the image subspaceF ofD, it is fairly obvious thatx+ =D+b. Otherwise, we can write

A =V DU ,

whereU andV are orthogonal. However, sinceV is an isometry,

Ax−b = V DU x−b = DU x−V b .

Letting y =U x, we have x = y , sinceU is an isometry, and sinceU is surjective, Ax−b is minimized iff Dy−V b is minimized, and we have shown that the least

solution is

y+ =D+V b.

Sincey =U x, with x = y , we get

x+ =UD+V b =A+b.

Thus, the pseudo-inverse provides the optimal solution to the least squares problem.

By Lemma 17.2 and Theorem 17.1,A+b is uniquely defined by everyb, and thusA+ depends only onA.

LetA =UΣV be an SVD forA. It is easy to check that

AA+A = A, A+AA+ = A+,

and bothAA+ andA+A are symmetric matrices. In fact,

AA+ =UΣV V Σ+U =UΣΣ+U =U Ir 0 U0 0n−r

and A+A =V Σ+U UΣV =V Σ+ΣV =V Ir 0 V .0 0n−r

We immediately get

(AA+)2 = AA+, (A+A)2 = A+A,

so both AA+ andA+A are orthogonal projections (since they are both symmetric). We claim thatAA+ is the orthogonal projection onto the range ofA andA+A is the orthogonal projection onto Ker(A)⊥ = Im(A ), the range ofA .

Obviously, we have range(AA+)⊆ range(A), and for anyy =Ax∈ range(A), since AA+A =A, we have

AA+y =AA+Ax =Ax =y,

so the image ofAA+ is indeed the range ofA. It is also clear that Ker(A)⊆ Ker(A+A), and sinceAA+A =A, we also have Ker(A+A)⊆ Ker(A), and so

Ker(A+A) = Ker(A).

SinceA+A is Hermitian, range(A+A) = Ker(A+A)⊥ = Ker(A)⊥, as claimed. It will also be useful to see that range(A) = range(AA+) consists of all vectorsy∈ Rn such that

U y = z ,0

withz∈ Rr.

Indeed, ify =Ax, then

U y =U Ax =U UΣV x = ΣV x =Σr 0 V x = z ,0 0n−r 0 where Σr is ther×r diagonal matrix diag(σ_1, . . . , σr). Conversely, ifU y = (z), then

y

=

U

(

z

0

), and_0

AA+y =U Ir 0 U y0 0n−r

Ir 0 U U z=U0 0n−r 0

Ir 0 z=U 0 0n−r 0

=U z =y,0

which shows thaty belongs to the range ofA.

Similarly, we claim that range(A+A) = Ker(A)⊥ consists of all vectorsy∈ Rn such that

V y = z ,0

withzr.∈ R

Ify =A+Au, then

for somez y =A+Au =V Ir 0 V u =V z ,0 0n−r 0

z), theny =V (z), and so∈ Rr. Conversely, ifV y = (0 0

A

+AV z =V Ir 0 V V z 0 0 0n−r 0 Ir 0 z=V 0 0n−r 0

=V z =y,0

which shows thaty∈ range(A+A).

If A is a symmetric matrix, then in general, there is no SVD UΣV ofA withU =V . However, ifA is positive semidefinite, then the eigenvalues ofA are nonnegative, and so the nonzero eigenvalues ofA are equal to the singular values ofA and SVDs ofA are of the form

A =UΣU . Analogous results hold for complex matrices, but in this case, U andV are unitary matrices andAA+ andA+A are Hermitian orthogonal projections.

If A is a normal matrix, which means thatAA = A A, then there is an intimate relationship between SVD’s ofA and block diagonalizations ofA. As a consequence, the pseudo-inverse of a normal matrixA can be obtained directly from a block diagonalization ofA.

IfA is a (real) normal matrix, then we know from Theorem 13.16 thatA can be block diagonalized with respect to an orthogonal matrixU as

A =UΛU ,

where Λ is the (real) block diagonal matrix

Λ = diag(B_1, . . . , Bn),

consisting either of 2× 2 blocks of the form

λ j −µjBj = µj λj

withµj = 0, or of one-dimensional blocksBk = (λk). Then we have the following proposition:

Proposition 17.3. For any (real) normal matrixA and any block diagonalizationA = UΛU ofA as above, the pseudo-inverse ofA is given by

A+ =UΛ+U ,

where Λ+ is the pseudo-inverse of Λ. Furthermore, if

Λ =Λr 0 ,0 0

where Λr has rankr, then

Λ

+

=

Λ−1 0

r .0 0

Proof. Assume thatB_1, . . . , Bp are 2×2 blocks and thatλ2p+1, . . . , λn are the scalar entries. We know that the numbersλj±iµj, and theλ2p+k are the eigenvalues ofA. Letρ2j−1 = ρ2j = λ^2 +µ^2 forj = 1, . . . , p,ρ2p+j =λj forj = 1, . . . , n−2p, and assume that the blocks are ordered so thatρ1≥ρ2≥ · · · ≥ρn. Then it is easy to see that

UU =U U =UΛU UΛ U =UΛΛ U , with ΛΛ = diag(ρ^2, . . . , ρ^2),1 n

so the singular valuesσ1≥σ2≥ · · · ≥σn ofA, which are the nonnegative square roots of the eigenvalues ofAA , are such that

σj =ρj, 1≤j≤n. We can define the diagonal matrices Σ = diag(σ_1, . . . , σr,0, . . . ,0), wherer = rank(A),σ1≥ · · · ≥σr> 0 and

Θ = diag(σ−1 B_1, . . . , σ−1Bp,1, . . . ,1),1 2p

so that Θ is an orthogonal matrix and Λ = ΘΣ = (B_1, . . . , Bp, λ2p+1, . . . , λr,0, . . . ,0). But then we can write

A =UΛU =UΘΣU ,

and we if letV =UΘ, sinceU is orthogonal and Θ is also orthogonal,V is also orthogonal andA =V ΣU is an SVD for A. Now we get

A+ =UΣ+V =UΣ+Θ U .

However, since Θ is an orthogonal matrix, Θ = Θ−1, and a simple calculation shows that Σ+Θ = Σ+Θ−1 = Λ+,

which yields the formula

A+ =UΛ+U .

Also observe that if we write

Λr = (B_1, . . . , Bp, λ2p+1, . . . , λr),

then Λr is invertible and

Λ

+

=

Λ−1 0

r .0 0

Therefore, the pseudo-inverse of a normal matrix can be computed directly from any block diagonalization ofA, as claimed.

The following properties, due to Penrose, characterize the pseudo-inverse of a matrix. We have already proved that the pseudo-inverse satisfies these equations. For a proof of the converse, see Kincaid and Cheney [61].

Lemma 17.4. Given anym×n matrixA (real or complex), the pseudo-inverseA+ ofA is the uniquen×m matrix satisfying the following properties:

AA+A =A,

A+AA+ =A+,

(AA+) =AA+,

(A+A) =A+A.

17.2. DATA COMPRESSION AND SVD 445

IfA is anm×n matrix of rankn (and som≥n), it is immediately shown that the QR-decomposition in terms of Householder transformations applies as follows: There aren m×m matricesH_1, . . . , Hn, Householder matrices or the identity, and an upper triangularm×n matrixR of rankn such that

A =H1· · ·HnR.

Then, because eachHi is an isometry, Ax−b = Rx−Hn· · ·H_1b ,

and the least squares problemAx =b is equivalent to the system Rx =Hn· · ·H_1b.

Now, the system Rx =Hn· · ·H_1b

is of the formR1 x = c ,0m−n d

whereR_1 is an invertiblen×n matrix (sinceA has rankn),c∈ Rn, andd∈ Rm−n, and the least squares solution of smallest norm is

x+ =R− 1 c.

1

SinceR_1 is a triangular matrix, it is very easy to invertR_1.

The method of least squares is one of the most effective tools of the mathematical sciences. There are entire books devoted to it. Readers are advised to consult Strang [101], Golub and Van Loan [47], Demmel [25], and Trefethen and Bau [106], where extensions and applications of least squares (such as weighted least squares and recursive least squares) are described. Golub and Van Loan [47] also contains a very extensive bibliography, including a list of books on least squares.

17.2 Data Compression and SVD

Among the many applications of SVD, a very useful one is data compression, notably for images. In order to make precise the notion of closeness of matrices, we review briefly the notion of matrix norm. We assume that the reader is familiar with the concept of vector nroem and a matrix norm. The concept of a norm is defined in Chapter 7 and the reader may want to review it before reading any further.

Given an m×n matrix of rankr, we would like to find a best approximation ofA by a matrixB of rankk≤r (actually,k < r) so that A−B_2 (or A−B F) is minimized. Proposition 17.5. LetA be anm×n matrix of rankr and letV DU =A be an SVD for A. Writeui for the columns ofU,vi for the columns ofV , andσ1≥σ2≥ · · · ≥σp for the singular values ofA (p = min(m, n)). Then a matrix of rankk < r closest toA (in the2norm) is given by

k

Ak = σiviui =V diag(σ_1, . . . , σk)U

i=1

and A−Ak 2 =σk+1.

Proof. By construction,Ak has rankk, and we have

p

A−Ak 2 = σiviui = V diag(0, . . . ,0, σ

2

k+1, . . . , σp)U_2 =σk+1.

i=k+1

It remains to show that A−B2≥σk+1 for all rankk matricesB. LetB be any rankk matrix, so its kernel has dimensionp−k. The subspaceVk+1 spanned by (v_1, . . . , vk+1) has dimensionk + 1, and because the sum of the dimensions of the kernel ofB and ofVk+1 is (p−k) +k + 1 =p + 1, these two subspaces must intersect in a subspace of dimension at least 1. Pick any unit vectorh in Ker(B)∩Vk+1. Then sinceBh = 0, we have

A

2

2 σ^2 U h^2 =σ^2 ,−B^2 (A−B)h^2 = Ah^2 = V DU h2≥ +1 2 +1≥ 2 2

which proves our claim.

Note thatAk can be stored using (m +n)k entries, as opposed tomn entries. When k m, this is a substantial gain.

A nice example of the use of Proposition 17.5 in image compression is given in Demmel [25], Chapter 3, Section 3.2.3, pages 113–115; see the Matlab demo.

An interesting topic that we have not addressed is the actual computation of an SVD. This is a very interesting but tricky subject. Most methods reduce the computation of an SVD to the diagonalization of a well-chosen symmetric matrix (which is notA A). Interested readers should read Section 5.4 of Demmel’s excellent book [25], which contains an overview of most known methods and an extensive list of references.

17.3 Principal Components Analysis (PCA)

Suppose we have a set of data consisting ofn pointsX_1, . . . , Xn, with eachXi∈ Rd viewed as a row vector.

Think of the Xi’s as persons, and ifXi = (xi 1, . . . , xi d), eachxi j is the value of some feature (or attribute) of that person. For example, theXi’s could be mathematicians,d = 2, and the first component,xi 1, ofXi could be the year thatXi was born, and the second component,xi 2, the length of the beard ofXi in centimeters. Here is a small data set: Name year length Carl Friedrich Gauss 1777 0 Camille Jordan 1838 12 Adrien-Marie Legendre 1752 0 Bernhard Riemann 1826 15 David Hilbert 1862 2 Henri Poincar´ 1854 5 Emmy Noether 1882 0 Karl Weierstrass 1815 0 Eugenio Beltrami 1835 2 Hermann Schwarz 1843 20

We usually form the n×d matrixX whoseith row isXi, with 1≤i≤n. Then the jth column is denoted byCj (1≤j≤d). It is sometimes called a feature vector, but this terminology is far from being universally accepted. In fact, many people in computer vision call the data pointsXi feature vectors!

The purpose of principal components analysis, for short PCA, is to identify patterns in data and understand the variance–covariance structure of the data. This is useful for the following tasks:

1. Data reduction: Often much of the variability of the data can be accounted for by a smaller number of principal components.

2. Interpretation: PCA can show relationships that were not previously suspected. Given a vector (a sample of measurements)x = (x_1, . . . , xn)∈ Rn, recall that the mean (or average)x ofx is given byn xix = i=1

n .

We letx−x denote the centered data point

x−x = (x1−x, . . . , xn−x).

In order to measure the spread of thexi’s around the mean, we define the sample variance (for short, variance) var(x) (ors^2) of the samplex by

n (xi−x)2

var(x) =i=1 .n− 1

There is a reason for using n− 1 instead ofn. The above definition makes var(x) an unbiased estimator of the variance of the random variable being sampled. However, we don’t need to worry about this. Curious readers will find an explanation of these peculiar definitions in Epstein [33] (Chapter 14, Section 14.5), or in any decent statistics book.

Given two vectorsx = (x_1, . . . , xn) andy = (y_1, . . . , yn), the sample covariance (for short, covariance) ofx andy is given by

cov(x, y) =i=1 n (xi−x)(yi−y). n− 1

The covariance ofx andy measures howx andy vary from the mean with respect to each other. Obviously, cov(x, y) = cov(y, x) and cov(x, x) = var(x).

Note that

cov(x, y) = (x−x) (y−y).n− 1

We say thatx andy are uncorrelated iff cov(x, y) = 0.

Finally, given an n×d matrixX ofn pointsXi, for PCA to be meaningful, it will be necessary to translate the origin to the centroid (or center of gravity)µ of theXi’s, defined by

µ = 1(X_1 +· · · +Xn).

n

Observe that ifµ = (µ_1, . . . , µd), thenµj is the mean of the vectorCj (thejth column of X).

We letX−µ denote the matrix whoseith row is the centered data pointXi−µ (1≤i≤n). Then, the sample covariance matrix (for short, covariance matrix) ofX is thed×d symmetric matrix

1 Σ =n− 1(X−µ) (X−µ) = (cov(Ci, Cj)).

Remark:

The factor

1

n−1 is irrelevant for our purposes and can be ignored. Here is the matrixX−µ in the case of our bearded mathematicians: Since

µ_1 = 1828.4, µ_2 = 5.6,

we get Name year length Carl Friedrich Gauss −51.4 −5.6 Camille Jordan 9.6 6.4 Adrien-Marie Legendre −76.4 −5.6 Bernhard Riemann−2.4 9.4 David Hilbert 33.6 −3.6 Henri Poincar´ 25.6 −0.6 Emmy Noether 53.6 −5.6 Karl Weierstrass 13.4 −5.6 Eugenio Beltrami 6.6 − 3.6 Hermann Schwarz 14.6 14.4

We can think of the vectorCj as representing the features ofX in the directionej (the jth canonical basis vector in Rd, namelyej = (0, . . . ,1, . . .0), with a 1 in thejth position).

If v∈ Rd is a unit vector, we wish to consider the projection of the data pointsX_1, . . . , Xn onto the line spanned byv. Recall from Euclidean geometry that ifx∈ Rd is any vector andv∈ Rd is a unit vector, the projection ofx onto the line spanned byv is

x, v v.

Thus, with respect to the basisv, the projection ofx has coordinate x, v . Ifx is represented by a row vector andv by a column vector, then

x, v =xv.

Therefore, the vectorY∈ Rn consisting of the coordinates of the projections ofX_1, . . . , Xn onto the line spanned byv is given byY =Xv, and this is the linear combination Xv =v_1C_1 +· · · +vdCd

of the columns ofX (withv = (v_1, . . . , vd)).

Observe that becauseµj is the mean of the vectorCj (thejth column ofX), we get Y =Xv =v_1µ_1 +· · · +vdµd,

and so the centered pointY−Y is given by

Y−Y =v_1(C1−µ_1) +· · · +vd(Cd−µd) = (X−µ)v.

Furthermore, ifY =Xv andZ =Xw, then

((X−µ)v) (X−µ)wcov(Y, Z) = n− 1

=

v

1

n− 1(X−µ) (X−µ)w

= v Σw,

where Σ is the covariance matrix ofX. SinceY−Y has zero mean, we have

1 var(Y ) = var(Y−Y ) =vn− 1(X−µ) (X−µ)v.

The above suggests that we should move the origin to the centroidµ of theXi’s and consider the matrixX−µ of the centered data pointsXi−µ.

From now on, beware that we denote the columns ofX−µ byC_1, . . . , Cd and thatY denotes the centered pointY = (X−µ)v =d vjCj, wherev is a unit vector.j=1

Basic idea of PCA : The principal components ofX are uncorrelated projectionsY of the data pointsX_1,. . .,Xn onto some directionsv (where thev’s are unit vectors) such that var(Y ) is maximal.

This suggests the following definition:

Definition 17.2. Given ann×d matrixX of data pointsX_1, . . . , Xn, ifµ is the centroid of theXi’s, then a first principal component of X (first PC) is a centered pointY_1 = (X−µ)v_1, the projection ofX_1, . . . , Xn onto a directionv_1 such that var(Y_1) is maximized, wherev_1 is a unit vector (recall thatY_1 = (X−µ)v_1 is a linear combination of theCj’s, the columns of X−µ).

More generally, if Y_1, . . . , Yk arek principal components ofX along some unit vectors v_1, . . . , vk, where 1≤k < d, a (k+1)th principal component ofX ((k+1)th PC) is a centered pointYk+1 = (X−µ)vk+1, the projection ofX_1, . . . , Xn onto some directionvk+1 such that var(Yk+1) is maximized, subject to cov(Yh, Yk+1) = 0 for allh with 1≤h≤k, and where vk+1 is a unit vector (recall thatYh = (X−µ)vh is a linear combination of theCj’s). The vh are called principal directions.

The following lemma is the key to the main result about PCA:

Lemma 17.6. IfA is a symmetricd×d matrix with eigenvaluesλ1≥λ2≥ · · · ≥λd and if (u_1, . . . , ud) is any orthonormal basis of eigenvectors ofA, whereui is a unit eigenvector associated withλi, then

maxx Ax =λ1x=0 x x

(with the maximum attained forx =u_1) and

x Ax

x

=0

,x

max⊥ x x =λk+1

∈{u1,...,uk}

(with the maximum attained forx =uk+1), where 1≤k≤d− 1. Proof. First, observe that

maxx Ax = max x Ax|x x = 1},

x=0 x xx{

and similarly,

x Ax

x

=0

,x

max⊥ x x = max x Ax| (x∈ {u_1, . . . , uk}⊥)∧ (x x = 1) .

∈{u1,...,uk} x

SinceA is a symmetric matrix, its eigenvalues are real and it can be diagonalized with respect to an orthonormal basis of eigenvectors, so let (u_1, . . . , ud) be such a basis. If we write

d

x = xiui,

i=1

a simple computation shows that

Ifx x = 1, then

d

x Ax = λix^2.

i=1

d x^2 = 1, and since we assumed thatλ1≥λ2≥ · · · ≥λd, we geti=1

d d

x Ax = λix^2 λ_1 x^2 =λ_1.

i=1≤ i=1

Thus,

max x Ax|x x = 1 ≤λ_1,

x

and since this maximum is achieved fore_1 = (1,0, . . . ,0), we conclude that max x Ax|x x = 1 =λ_1.

x

Next, observe thatx∈ {u_1, . . . , uk}⊥ andx x = 1 iffx_1 =· · · =xk = 0 andd xi = 1.i=1 Consequently, for such anx, we have

d d

x Ax = λix^2 λk+1 x^2 =λk+1.

i=k+1≤ i=k+1

Thus,

max x Ax| (x∈ {u_1, . . . , uk}⊥)∧ (x x = 1) ≤λk+1,

x

and since this maximum is achieved forek+1 = (0, . . . ,0,1,0, . . . ,0) with a 1 in positionk+1, we conclude that

max x Ax| (x∈ {u_1, . . . , uk}⊥)∧ (x x = 1) =λk+1,

x

as claimed.

The quantity x Ax

x x

is known as the Rayleigh–Ritz ratio and Lemma 17.6 is often known as part of the Rayleigh–

Ritz theorem.

Lemma 17.6 also holds if A is a Hermitian matrix and if we replacex Ax byx∗Ax and x x byx∗x. The proof is unchanged, since a Hermitian matrix has real eigenvalues and is diagonalized with respect to an orthonormal basis of eigenvectors (with respect to the Hermitian inner product).

We then have the following fundamental result showing how the SVD ofX yields the PCs:

Theorem 17.7. (SVD yields PCA) LetX be ann×d matrix of data pointsX_1, . . . , Xn, and letµ be the centroid of theXi’s. IfX−µ =V DU is an SVD decomposition ofX−µ and if the main diagonal ofD consists of the singular valuesσ1≥σ2≥ · · · ≥σd, then the centered pointsY_1, . . . , Yd, where

Yk = (X−µ)uk =kth column ofV D anduk is thekth column ofU, ared principal components ofX. Furthermore,

2

var(

Y

k

) =

σ

n− 1

and cov(Yh, Yk) = 0, wheneverh =k and 1≤k, h≤d.

Proof. Recall that for any unit vectorv, the centered projection of the pointsX_1, . . . , Xn onto the line of directionv isY = (X−µ)v and that the variance ofY is given by

var(Y ) =v^1 1(X−µ) (X−µ)v.n−

SinceX−µ =V DU , we get

1 var(Y ) = v(n− 1)(X−µ) (X−µ)v

=

v

1

(n− 1)UDV V DU v

= v U^1 D^2U v.(n− 1)

Similarly, ifY = (X−µ)v andZ = (X−µ)w, then the covariance ofY andZ is given by

cov( Y, Z) =v U^1 D^2U w.(n− 1)

Obviously,U(n−1)D^2U is a symmetric matrix whose eigenvalues are σ2 σ21 1 d , and the columns ofU form an orthonormal basis of unit eigenvectors.n−1≥ · · · ≥ n−1

We proceed by induction onk. For the base case,k = 1, maximizing var(Y ) is equivalent to maximizing

v

U

1

(n− 1)D^2U v,

wherev is a unit vector. By Lemma 17.6, the maximum of the above quantity is the largest eigenvalue of

U

1_1

(

n

−

1)

D^2U , namely σ2 , and it is achieved foru_1, the first columnn ofU. Now we getn−1

Y_1 = (X−µ)u_1 =V DU u_1,

and since the columns ofU form an orthonormal basis,U u_1 =e_1 = (1,0, . . . ,0), and soY_1 is indeed the first column ofV D.

By the induction hypothesis, the centered pointsY_1, . . . , Yk, whereYh = (X−µ)uh and u_1, . . . , uk are the firstk columns ofU, arek principal components ofX. Because

cov(Y, Z) =v U^1 D^2U w,(n− 1)

where Y = (X−µ)v andZ = (X−µ)w, the condition cov(Yh, Z) = 0 forh = 1, . . . , k is equivalent to the fact that w belongs to the orthogonal complement of the subspace spanned by{u_1, . . . , uk}, and maximizing var(Z) subject to cov(Yh, Z) = 0 forh = 1, . . . , k is equivalent to maximizing

w

U

1

(n− 1)D^2U w,

wherew is a unit vector orthogonal to the subspace spanned by{u_1, . . . , uk}. By Lemma 17.6, the maximum of the above quantity is the (

k

+ 1)th eigenvalue of

U

1

(n−1)D^2U , namely

σ

2

k+1 , and it is achieved foruk+1, the (k + 1)th columnn ofU. Now we getn−1

Yk+1 = (X−µ)uk+1 =V DU uk+1,

and since the columns ofU form an orthonormal basis,U uk+1 =ek+1, andYk+1 is indeed the (k + 1)th column ofV D, which completes the proof of the induction step.

The d columnsu_1, . . . , ud ofU are usually called the principal directions ofX−µ (and X). We note that not only do we have cov(Yh, Yk) = 0 wheneverh =k, but the directions u_1, . . . , ud along which the data are projected are mutually orthogonal.

We know from our study of SVD that σ^2, . . . , σ^2 are the eigenvalues of the symmetric_1

positive semidefinite matrix (X−µ) (X−µ) and thatu_1, . . . , ud are corresponding eigenvectors. Numerically, it is preferable to use SVD onX−µ rather than to compute explicitly (X−µ) (X−µ) and then diagonalize it. Indeed, the explicit computation ofA A from a matrixA can be numerically quite unstable, and good SVD algorithms avoid computing A A explicitly.

In general, since an SVD of X is not unique, the principal directionsu_1, . . . , ud are not unique. This can happen when a data set has some rotational symmetries, and in such a case, PCA is not a very good method for analyzing the data set.

17.4 Best Affine Approximation

A problem very close to PCA (and based on least squares) is to best approximate a data set ofn pointsX_1, . . . , Xn, withXi∈ Rd, by ap-dimensional affine subspaceA of Rd, with 1≤p≤d− 1 (the terminology rankd−p is also used).

First, considerp =d− 1. ThenA =A_1 is an affine hyperplane (in Rd), and it is given by an equation of the form

a_1x_1 +· · · +adxd +c = 0.

By best approximation, we mean that (a_1, . . . , ad, c) solves the homogeneous linear system ëa1ö ë_0öëx1 1 · · · x1 d 1ö

ì

ì

.

.

.

.

÷

ì

.

÷ ì.÷

÷= ì ÷

í

ø

ìa ÷ ì

d

ø

í

0

÷

x

n

1

· · ·

x

n d

1

í ø

c 0

in the least squares sense, subject to the condition thata = (a_1, . . . , ad) is a unit vector, that is,a a = 1, whereXi = (xi 1,· · ·, xi d).

If we form the symmetric matrix

ëx1 1 · · · x1 d 1ö ëx1 1 · · · x1 d 1ö

ì. . . .÷ ì. . . .÷

í ø í ø

xn 1 · · · xn d 1 xn 1 · · · xn d 1

involved in the normal equations, we see that the bottom row (and last column) of that matrix is

nµ1 · · · nµd n,

wherenµj =n xi j isn times the mean of the columnCj ofX.i=1

Therefore, if (a_1, . . . , ad, c) is a least squares solution, that is, a solution of the normal equations, we must have

nµ_1a_1 +· · · +nµdad +nc = 0,

that is,

a_1µ_1 +· · · +adµd +c = 0, which means that the hyperplaneA_1 must pass through the centroidµ of the data points X_1, . . . , Xn. Then we can rewrite the original system with respect to the centered data Xi−µ, and we find that the variablec drops out and we get the system

(X−µ)a = 0,

wherea = (a_1, . . . , ad).

Thus, we are looking for a unit vectora solving (X−µ)a = 0 in the least squares sense, that is, somea such thata a = 1 minimizing

a (X−µ) (X−µ)a.

Compute some SVD V DU ofX− µ, where the main diagonal ofD consists of the singular valuesσ1≥σ2≥ · · · ≥σd ofX−µ arranged in descending order. Then

a (X−µ) (X−µ)a =a UD^2U a,

where D^2 = diag(σ^2, . . . , σ^2) is a diagonal matrix, so picka to be the last column inU_1

(corresponding to the smallest eigenvalueσ^2 of (X−µ) (X−µ)). This is a solution to our best fit problem.

Therefore, if Ud−1 is the linear hyperplane defined bya, that is,

Ud−1 ={u∈ Rd u, a = 0},|

where a is the last column inU for some SVD V DU ofX−µ, we have shown that the affine hyperplaneA_1 =µ +Ud−1 is a best approximation of the data setX_1, . . . , Xn in the least squares sense.

Is is easy to show that this hyperplane A_1 =µ +Ud−1 minimizes the sum of the square distances of eachXi to its orthogonal projection ontoA_1. Also, sinceUd−1 is the orthogonal complement ofa, the last column ofU, we see thatUd−1 is spanned by the firstd−1 columns ofU, that is, the firstd− 1 principal directions ofX−µ.

All this can be generalized to a best (d−k)-dimensional affine subspaceAk approximating X_1, . . . , Xn in the least squares sense (1≤k≤d− 1). Such an affine subspaceAk is cut out byk independent hyperplanesHi (with 1≤i≤k), each given by some equation

ai 1x_1 +· · · +ai dxd +ci = 0.

If we writeai = (ai 1,· · ·, ai d), to say that theHi are independent means thata_1, . . . , ak are linearly independent. In fact, we may assume thata_1, . . . , ak form an orthonormal system. Then, finding a best (d−k)-dimensional affine subspace Ak amounts to solving the homogeneous linear system

ëa1ö

ëX 1 0 · · · 0 0 0ö ìc_1÷ ë0ö

ì ÷

ì. . . ... . . .÷ ì. ÷= ì.÷,í 1ø ì ÷ í ø

0

0

0

· · ·

0

X

ìa ÷

k ø 0íck

in the least squares sense, subject to the conditionsaiaj =δi j, for alli, j with 1≤i, j≤k, where the matrix of the system is a block diagonal matrix consisting ofk diagonal blocks (X,1), where 1 denotes the column vector (1, . . . ,1)∈ Rn.

Again, it is easy to see that each hyperplaneHi must pass through the centroid µ of X_1, . . . , Xn, and by switching to the centered dataXi−µ we get the system ëX−µ 0 · · · 0 ö ëa1ö ë_0ö

ì. . ... . ÷ ì. ÷ = ì.÷,í ø í ø í ø

0 0 · · · X−µ ak 0

withaiaj =δi j for alli, j with 1≤i, j≤k.

If V DU =X−µ is an SVD decomposition, it is easy to see that a least squares solution of this system is given by the lastk columns ofU, assuming that the main diagonal ofD consists of the singular valuesσ1≥σ2 ≥ · · · ≥ σd ofX−µ arranged in descending order. But now the (d−k)-dimensional subspaceUd−k cut out by the hyperplanes defined bya_1, . . . , ak is simply the orthogonal complement of (a_1, . . . , ak), which is the subspace spanned by the firstd−k columns ofU.

So the best (d−k)-dimensional affine subpsaceAk approximatingX_1, . . . , Xn in the least squares sense is

Ak =µ +Ud−k,

where Ud−k is the linear subspace spanned by the firstd−k principal directions ofX−µ, that is, the firstd−k columns ofU. Consequently, we get the following interesting interpretation of PCA (actually, principal directions):

Theorem 17.8. LetX be ann× d matrix of data pointsX_1, . . . , Xn, and letµ be the centroid of theXi’s. IfX−µ =V DU is an SVD decomposition ofX−µ and if the main diagonal ofD consists of the singular valuesσ1≥σ2≥ · · · ≥σd, then a best (d−k)-dimensional affine approximationAk ofX_1, . . . , Xn in the least squares sense is given by

Ak =µ +Ud−k,

whereUd−k is the linear subspace spanned by the firstd−k columns ofU, the firstd−k

principal directions ofX−µ (1≤k≤d− 1).

There are many applications of PCA to data compression, dimension reduction, and pattern analysis. The basic idea is that in many cases, given a data setX_1, . . . , Xn, with Xi∈ Rd, only a “small” subset ofm < d of the features is needed to describe the data set accurately.

If u_1, . . . , ud are the principal directions ofX−µ, then the firstm projections of the data (the firstm principal components, i.e., the firstm columns ofV D) onto the firstm principal directions represent the data without much loss of information. Thus, instead of using the original data pointsX_1, . . . , Xn, withXi∈ Rd, we can use their projections onto the firstm principal directionsY_1, . . . , Ym, whereYi∈ Rm andm < d, obtaining a compressed version of the original data set.

For example, PCA is used in computer vision for face recognition. Sirovitch and Kirby (1987) seem to be the first to have had the idea of using PCA to compress facial images. They introduced the term eigenpicture to refer to the principal directions,ui. However, an explicit face recognition algorithm was given only later, by Turk and Pentland (1991). They renamed eigenpictures as eigenfaces.

For details on the topic of eigenfaces, see Forsyth and Ponce [37] (Chapter 22, Section 22.3.2), where you will also find exact references to Turk and Pentland’s papers.

Another interesting application of PCA is to the recognition of handwritten digits. Such an application is described in Hastie, Tibshirani, and Friedman, [53] (Chapter 14, Section 14.5.1).

Chapter 18 Quadratic Optimization Problems

18.1 Quadratic Optimization: The Positive Definite Case

In this chapter, we consider two classes of quadratic optimization problems that appear frequently in engineering and in computer science (especially in computer vision): 1. Minimizing

f(x) = 1x Ax +x b

2

over allx∈ Rn, or subject to linear or affine constraints.

2. Minimizing

f(x) = 1x Ax +x b

2

over the unit sphere.

In both cases,A is a symmetric matrix. We also seek necessary and sufficient conditions for f to have a global minimum.

Many problems in physics and engineering can be stated as the minimization of some energy function, with or without constraints. Indeed, it is a fundamental principle of mechanics that nature acts so as to minimize energy. Furthermore, if a physical system is in a stable state of equilibrium, then the energy in that state should be minimal. For example, a small ball placed on top of a sphere is in an unstable equilibrium position. A small motion causes the ball to roll down. On the other hand, a ball placed inside and at the bottom of a sphere is in a stable equilibrium position, because the potential energy is minimal.

The simplest kind of energy function is a quadratic function. Such functions can be conveniently defined in the form

P(x) =x Ax−x b,

459 whereA is a symmetricn×n matrix, andx, b, are vectors in Rn, viewed as column vectors. Actually, for reasons that will be clear shortly, it is preferable to put a factor 1 in front of the quadratic term, so that^2

P(x) = 1x Ax−x b.

2

The question is, under what conditions (onA) doesP(x) have a global minimum, preferably unique?

We give a complete answer to the above question in two stages:

1. In this section, we show that ifA is symmetric positive definite, thenP(x) has a unique global minimum precisely when

Ax =b. 2. In Section 18.2, we give necessary and sufficient conditions in the general case, in terms of the pseudo-inverse ofA.

We begin with the matrix version of Definition 16.2.

Definition 18.1. A symmetric positive definite matrix is a matrix whose eigenvalues are strictly positive, and a symmetric positive semidefinite matrix is a matrix whose eigenvalues are nonnegative.

Equivalent criteria are given in the following proposition.

Proposition 18.1. Given any Euclidean spaceE of dimensionn, the following properties hold:

(1) Every self-adjoint linear mapf :E→E is positive definite iff

x, f(x) > 0

for allx∈E withx = 0.

(2) Every self-adjoint linear mapf :E→E is positive semidefinite iff

x, f(x)≥ 0

for allx∈E.

Proof. (1) First, assume thatf is positive definite. Recall that every self-adjoint linear map has an orthonormal basis (e_1, . . . , en) of eigenvectors, and letλ_1, . . . , λn be the corresponding eigenvalues. With respect to this basis, for everyx =x_1e_1 +· · · +xnen = 0, we have

n n n n n

x, f(x) = xiei, f xiei = xiei, λixiei = λix^2,

i=1 i=1 i=1 i=1 i=1

which is strictly positive, sinceλi> 0 fori = 1, . . . , n, andx^2> 0 for somei, sincex = 0. Conversely, assume that

x, f(x) > 0

for all x = 0. Then forx =ei, we get

ei, f(ei) = ei, λiei =λi,

and thusλi> 0 for alli = 1, . . . , n.

(2) As in (1), we have

n

x, f(x) = λix^2,

i=1

and sinceλi≥ 0 fori = 1, . . . , n becausef is positive semidefinite, we have x, f(x)≥ 0, as claimed. The converse is as in (1) except that we get onlyλi≥ 0 since ei, f(ei)≥ 0. Some special notation is customary (especially in the field of convex optinization) to express that a symmetric matrix is positive definite or positive semidefinite. Definition 18.2. Given anyn×n symmetric matrixA we writeA 0 ifA is positive semidefinite and we writeA 0 ifA is positive definite.

It should be noted that we can define the relation

A B

between any two n×n matrices (symmetric or not) iffA−B is symmetric positive semidefinite. It is easy to check that this relation is actually a partial order on matrices, called the positive semidefinite cone ordering; for details, see Boyd and Vandenberghe [15], Section 2.4.

If A is symmetric positive definite, it is easily checked thatA−1 is also symmetric positive definite. Also, ifC is a symmetric positive definitem×m matrix andA is anm×n matrix of rankn (and som≥n), thenA CA is symmetric positive definite.

We can now prove that

P(x) = 1x Ax−x b

2

has a global minimum whenA is symmetric positive definite.

Proposition 18.2. Given a quadratic function

P(x) = 1x Ax−x b,

2

ifA is symmetric positive definite, thenP(x) has a unique global minimum for the solution of the linear systemAx =b. The minimum value ofP(x) is

P (A−1b) =1b A−1b.−2

Proof. SinceA is positive definite, it is invertible, since its eigenvalues are all strictly positive. Letx =A−1b, and computeP(y)−P(x) for anyy∈ Rn. SinceAx =b, we get

) = 1y Ay−y b− 2x Ax +x b−P(x^1P(y)

2

= 1y Ay−y Ax + 1x Ax

2 2

= 12(y−x) A(y−x).

SinceA is positive definite, the last expression is nonnegative, and thus

P(y)≥P(x) for all y∈ Rn, which proves that x = A−1b is a global minimum of P(x). A simple computation yields

P(A−1b) =1b A−1b.−2

Remarks: (1) The quadratic functionP(x) is also given by

P(x) = 1x Ax−b x,

2

but the definition usingx b is more convenient for the proof of Proposition 18.2. (2) IfP(x) contains a constant termc∈ R, so that

P(x) = 1x Ax−x b +c,

2

the proof of Proposition 18.2 still shows thatP(x) has a unique global minimum for x =A−1b, but the minimal value is

P

(

A

−

1b) =1

−2b A−1b +c.

Thus, when the energy functionP(x) of a system is given by a quadratic function

P(x) = 1x Ax−x b,

2

where A is symmetric positive definite, finding the global minimum ofP(x) is equivalent to solving the linear systemAx =b. Sometimes, it is useful to recast a linear problemAx =b as a variational problem (finding the minimum of some energy function). However, very often, a minimization problem comes with extra constraints that must be satisfied for all admissible solutions. For instance, we may want to minimize the quadratic function

) = 1 y^2 +y^2Q(y_1, y2 1 22

subject to the constraint

2y1−y_2 = 5.

The solution for whichQ(y_1, y_2) is minimum is no longer (y_1, y_2) = (0,0), but instead, (y_1, y_2) = (2,−1), as will be shown later.

Geometrically, the graph of the function defined byz =Q(y_1, y_2) in R^3 is a paraboloid of revolutionP with axis of revolutionOz. The constraint

2y1−y_2 = 5

corresponds to the vertical plane H parallel to thez-axis and containing the line of equation 2y1−y_2 = 5 in thexy-plane. Thus, the constrained minimum ofQ is located on the parabola that is the intersection of the paraboloidP with the planeH.

A nice way to solve constrained minimization problems of the above kind is to use the method of Lagrange multipliers. But first, let us define precisely what kind of minimization problems we intend to solve.

Definition 18.3. The quadratic constrained minimization problem consists in minimizing a quadratic function

Q(y) = 1y C−1y−b y

2

subject to the linear constraints

A y =f,

where C−1 is anm×m symmetric positive definite matrix,A is anm×n matrix of rankn (so thatm≥n), and whereb, y∈ Rm (viewed as column vectors), andf∈ Rn (viewed as a column vector).

The reason for using C−1 instead ofC is that the constrained minimization problem has an interpretation as a set of equilibrium equations in which the matrix that arises naturally isC (see Strang [100]). SinceC andC−1 are both symmetric positive definite, this doesn’t make any difference, but it seems preferable to stick to Strang’s notation.

The method of Lagrange consists in incorporating the n constraintsA y =f into the quadratic functionQ(y), by introducing extra variablesλ = (λ_1, . . . , λn) called Lagrange multipliers, one for each constraint. We form the Lagrangian

L(y, λ) =Q(y) +λ (A y ) = 1y C−1y− (b−Aλ) y−λ f.−f

2

We shall prove that our constrained minimization problem has a unique solution given by the system of linear equations

C−1y +Aλ =b, A y =f,

which can be written in matrix form as

C−1 A y= b .A 0 λ f

Note that the matrix of this system is symmetric. Eliminatingy from the first equation

C−1y +Aλ =b,

we get y =C(b−Aλ), and substituting into the second equation, we get

A C(b−Aλ) =f,

that is, A CAλ =A Cb−f.

However, by a previous remark, since C is symmetric positive definite and the columns of A are linearly independent,A CA is symmetric positive definite, and thus invertible. Note that this way of solving the system requires solving for the Lagrange multipliers first.

Lettinge =b−Aλ, we also note that the system

C^1 A y= b−

A 0 λ f

is equivalent to the system

e =b−Aλ, y =Ce,

A y =f.

The latter system is called the equilibrium equations by Strang [100]. Indeed, Strang shows that the equilibrium equations of many physical systems can be put in the above form. This includes spring-mass systems, electrical networks, and trusses, which are structures built from elastic bars. In each case,y,e,b,C,λ,f, andK =A CA have a physical interpretation. The matrixK =A CA is usually called the stiffness matrix. Again, the reader is referred to Strang [100].

In order to prove that our constrained minimization problem has a unique solution, we proceed to prove that the constrained minimization ofQ(y) subject toA y =f is equivalent to the unconstrained maximization of another function−P(λ). We getP(λ) by minimizing the LagrangianL(y, λ) treated as a function ofy alone. SinceC−1 is symmetric positive definite and

L(y, λ) = 1y C−1y− (b−Aλ) y−λ f,

2

by Proposition 18.2 the global minimum (with respect toy) ofL(y, λ) is obtained for the solutiony of

C−1y =b−Aλ, that is, when

y =C(b−Aλ), and the minimum ofL(y, λ) is

1

min L(y, λ) =−2(Aλ−b) C(Aλ−b)−λ f.

y

Letting

P(λ) = 12(Aλ−b) C(Aλ−b) +λ f,

we claim that the solution of the constrained minimization of Q(y) subject toA y =f is equivalent to the unconstrained maximization of−P(λ). Of course, since we minimized L(y, λ) with respect toy, we have

L(y, λ)≥ −P(λ) for ally and allλ. However, when the constraintA y =f holds,L(y, λ) =Q(y), and thus for any admissibley, which means thatA y =f, we have

min Q(y)≥ max P(λ).

y λ −

In order to prove that the unique minimum of the constrained problemQ(y) subject to A y =f is the unique maximum of−P(λ), we computeQ(y) +P(λ).

Proposition 18.3. The quadratic constrained minimization problem of Definition 18.3 has a unique solution (y, λ) given by the system

C−1 A y= b .A 0 λ f

Furthermore, the component λ of the above solution is the unique value for which−P(λ) is maximum.

Proof. As we suggested earlier, let us computeQ(y) +P(λ), assuming that the constraint A y =f holds. Eliminatingf, sinceb y =y b andλ A y =y Aλ, we get

Q(y) +P(λ) = 1y C−1y−b y + 12(Aλ−b) C(Aλ−b) +λ f

2

= 12(C−1y +Aλ−b) C(C−1y +Aλ−b).

SinceC is positive definite, the last expression is nonnegative. In fact, it is null iff

C−1y +Aλ−b = 0, that is, C−1y +Aλ =b.

But then the unique constrained minimum of Q(y) subject toA y = f is equal to the unique maximum of−P(λ) exactly whenA y =f andC−1y +Aλ =b, which proves the proposition.

Remarks:

(1) There is a form of duality going on in this situation. The constrained minimization of Q(y) subject to A y = f is called the primal problem, and the unconstrained maximization of−P(λ) is called the dual problem. Duality is the fact stated slightly loosely as

min Q(y) = max P(λ).

y λ −

Recalling thate =b−Aλ, since

P(λ) = 12(Aλ−b) C(Aλ−b) +λ f,

we can also write

P(λ) = 1e Ce +λ f.

2

This expression often represents the total potential energy of a system. Again, the optimal solution is the one that minimizes the potential energy (and thus maximizes −P(λ)).

(2) It is immediately verified that the equations of Proposition 18.3 are equivalent to the equations stating that the partial derivatives of the LagrangianL(y, λ) are null:

∂L = 0, i = 1, . . . , m,∂yi

∂L = 0, j = 1, . . . , n.∂λj

Thus, the constrained minimum ofQ(y) subject toA y =f is an extremum of the LagrangianL(y, λ). As we showed in Proposition 18.3, this extremum corresponds to simultaneously minimizingL(y, λ) with respect toy and maximizingL(y, λ) with respect toλ. Geometrically, such a point is a saddle point forL(y, λ).

(3) The Lagrange multipliers sometimes have a natural physical meaning. For example, in the spring-mass system they correspond to node displacements. In some general sense, Lagrange multipliers are correction terms needed to satisfy equilibrium equations and the price paid for the constraints. For more details, see Strang [100].

Going back to the constrained minimization ofQ(y_1, y_2) = 1(y^2 +y^2) subject to2 1 2

2y1−y_2 = 5,

the Lagrangian is

L(y ) = 1 y^2 +y^2 +λ(2y1−y2− 5),1, y_2, λ1 22

and the equations stating that the Lagrangian has a saddle point are

y_1 + 2λ = 0,

y2−λ = 0,

2y1−y2− 5 = 0.

We obtain the solution (y_1, y_2, λ) = (2,−1,−1).

Much more should be said about the use of Lagrange multipliers in optimization or variational problems. This is a vast topic. Least squares methods and Lagrange multipliers are used to tackle many problems in computer graphics and computer vision; see Trucco and Verri [107], Metaxas [76], Jain, Katsuri, and Schunck [58], Faugeras [35], and Foley, van Dam, Feiner, and Hughes [36]. For a lucid introduction to optimization methods, see Ciarlet [22].

18.2 Quadratic Optimization: The General Case

In this section, we complete the study initiated in Section 18.1 and give necessary and sufficient conditions for the quadratic function 1x Ax+x b to have a global minimum. We begin with the following simple fact:2

Proposition 18.4. IfA is an invertible symmetric matrix, then the function

f(x) = 1x Ax +x b

2

has a minimum value iffA 0, in which case this optimal value is obtained for a unique value ofx, namelyx∗ =−A−1b, and with

f

(

A

−

1b) =1

−2b A−1b. Proof. Observe that

1 1b) A(x +A−1b) = 1x Ax +x b + 1b A−1b.2(x +A−

2 2

Thus,

f(x^1) = 1x Ax +x b = 12(x +A−1b) A(x +A−1b)− 2b A−1b.

2

If A has some negative eigenvalue, say−λ (withλ > 0), if we pick any eigenvectoru ofA associated withλ, then for anyα∈ R withα = 0, if we letx =αu−A−1b, then since Au =−λu, we get

f

(

x

) = 1

2(

x

+

A

−

1b) A(x +A−1b)1

−

2

b A−1b

= 1

αu

Aαu

1

−

2

b A−1b

2

=1α^2λ u2 1b A−1b,−2 2− 2

and since α can be made as large as we want andλ > 0, we see thatf has no minimum. Consequently, in order forf to have a minimum, we must haveA 0. In this case, since (x +A−1b) A(x +A−1b)≥ 0, it is clear that the minimum value off is achieved when x +A−1b = 0, that is,x =−A−1b.

Let us now consider the case of an arbitrary symmetric matrixA.

Proposition 18.5. IfA is a symmetric matrix, then the function

f(x) = 1x Ax +x b

2

has a minimum value iffA 0 and (I−AA+)b = 0, in which case this minimum value is

p∗ =1b A+b.−2

Furthermore, ifA =U ΣU is an SVD ofA, then the optimal value is achieved by allx∈ Rn of the form

x =−A+b +U^0 ,z

for anyz∈ Rn−r, wherer is the rank ofA.

Proof. The case thatA is invertible is taken care of by Proposition 18.4, so we may assume thatA is singular. IfA has rankr < n, then we can diagonalizeA as

A =UΣr 0 U,0 0

whereU is an orthogonal matrix and where Σr is anr×r diagonal invertible matrix. Then we have

f(x) = 1x UΣr 0 Ux +x U Ub 2 0 0

= 12(Ux)Σr 0 Ux + (Ux) Ub.0 0

If we write Ux = y and Ub = c ,z d

withy, c∈ Rr andz, d∈ Rn−r, we get

f(x) = 12(Ux)Σr 0 Ux + (Ux) Ub0 0

= 1

2(

y

, z

)

Σr 0 y + (y , z ) c 0 0 z d = 1y Σry +y c +z d.

2

Fory = 0, we get

f(x) =z d,

so ifd = 0, the functionf has no minimum. Therefore, iff has a minimum, thend = 0. However,d = 0 means that

Ub = c ,0

and we know from Section 17.1 thatb is in the range ofA (here,U isU ), which is equivalent to (I−AA+)b = 0. Ifd = 0, then

f(x) = 1y Σry +y c,

2

and since Σr is invertible, by Proposition 18.4, the functionf has a minimum iff Σr 0, which is equivalent toA 0.

Therefore, we have proved that if f has a minimum, then (I−AA+)b = 0 andA 0. Conversely, if (I−AA+)b = 0 andA 0, what we just did proves thatf does have a minimum.

When the above conditions hold, the minimum is achieved ify =−Σ− 1 c,z = 0 and

r

d = 0, that is, forx∗ given by

Ux

∗

=

−

Σ

−

1

c

r

c ,0 and Ub =0 from which we deduce that

Σ− 1 Σ− 1 0 c =−UΣ− 1 0 Ub =−A+b−Ur c =−Ur rx∗ = c c

0 0 0 0 0 0 and the minimum value off is

f(x∗) =1b A+b.−2

For anyx∈ Rn of the form

x =−A+b +U^0 ,z

for any

z

∈

R

n−r, our previous calculations show thatf(x) =1 −2b A+b.

The case in which we add either linear constraints of the form C x = 0 or affine constraints of the formC x =t (wheret = 0) can be reduced to the unconstrained case using a QR-decomposition ofC orN. Let us show how to do this for linear constraints of the form C x = 0.

If we use aQR decomposition ofC, by permuting the columns, we may assume that

C =Q R S Π,0 0

whereR is anr×r invertible upper triangular matrix andS is anr× (m−r) matrix (C has rankr). Then, if we let

x =Q y ,z

wherey∈ Rr andz∈ Rn−r, thenC x = 0 becomes

Π R 0 Qx = Π R 0 y = 0,S 0 S 0 z

which impliesy = 0, and every solution ofC x = 0 is of the form

x =Q^0 .z

Our original problem becomes

1

minimize y + (y , z )Qb2(y , z )QAQz

subject to y = 0, y∈ Rr, z∈ Rn−r.

Thus, the constraintC x = 0 has been eliminated, and if we write G_11 G12QAQ = G21 G22 and Qb = b1 , b1∈ Rr, b2∈ Rn−r,b2

our problem becomes minimize 1z G_22z +z b_2, z∈ Rn−r,

2

the problem solved in Proposition 18.5.

Constraints of the form C x =t (wheret = 0) can be handled in a similar fashion. In this case, we may assume thatC is ann×m matrix with full rank (so thatm≤n) and t∈ Rm. Then we use aQR-decomposition of the form

C =P R ,0

whereP is an orthogonal matrix andR is anm×m invertible upper triangular matrix. If we write

x =P y ,z

wherey∈ Rm andz∈ Rn−m, the equationC x =t becomes

(R ,0)P x =t,

that is, (R ,0) y =t,z

which yields

R y =t.

Since R is invertible, we gety = (R )−1t, and then it is easy to see that our original problem reduces to an unconstrained problem in terms of the matrixP AP; the details are left as an exercise.

18.3 Maximizing a Quadratic Function on the Unit Sphere

In this section we discuss various quadratic optimization problems mostly arising from computer vision (image segmentation and contour grouping). These problems can be reduced to the following basic optimization problem: Given ann×n real symmetric matrixA

maximize x Ax

subject to x x = 1, x∈ Rn.

In view of Proposition 17.6, the maximum value of x Ax on the unit sphere is equal to the largest eigenvalueλ_1 of the matrixA, and it is achieved for any unit eigenvectoru_1 associated withλ_1.

A variant of the above problem often encountered in computer vision consists in minimizingx Ax on the ellipsoid given by an equation of the form

x Bx = 1,

whereB is a symmetric positive definite matrix. SinceB is positive definite, it can be diagonalized as

B =QDQ ,

whereQ is an orthogonal matrix andD is a diagonal matrix,

D = diag(d_1, . . . , dn),

withdi> 0, fori = 1, . . . , n. If we define the matricesB1/2 andB−1/2 by

B1/2 =Qdiag d_1, . . . , dn Q

and B−1/2 =Qdiag 1/ d_1, . . . ,1/ dn Q ,

it is clear that these matrices are symmetric, thatB−1/2BB−1/2 =I, and thatB1/2 and B−1/2 are mutual inverses. Then, if we make the change of variable

x =B−1/2y,

the equationx Bx = 1 becomesy y = 1, and the optimization problem

maximize x Ax

subject to x Bx = 1, x∈ Rn,

is equivalent to the problem

maximize y B−1/2AB−1/2y subject to y y = 1, y∈ Rn,

wherey =B1/2x and whereB−1/2AB−1/2 is symmetric.

The complex version of our basic optimization problem in whichA is a Hermitian matrix also arises in computer vision. Namely, given ann×n complex Hermitian matrixA,

maximize x∗Ax

subject to x∗x = 1, x∈ Cn.

Again by Proposition 17.6, the maximum value of x∗Ax on the unit sphere is equal to the largest eigenvalueλ_1 of the matrixA and it is achieved for any unit eigenvectoru_1 associated withλ_1.

It is worth pointing out that ifA is a skew-Hermitian matrix, that is, ifA∗ =−A, then x∗Ax is pure imaginary or zero.

Indeed, sincez =x∗Ax is a scalar, we havez∗ =z (the conjugate ofz), so we have

x∗Ax = (x∗Ax)∗ =x∗A∗x =−x∗Ax,

sox∗Ax +x∗Ax = 2Re(x∗Ax) = 0, which means thatx∗Ax is pure imaginary or zero. In particular, ifA is a real matrix and ifA is skew-symmetric, then

x Ax = 0.

Thus, for any real matrix (symmetric or not),

x Ax =x H(A)x,

whereH(A) = (A +A )/2, the symmetric part ofA.

There are situations in which it is necessary to add linear constraints to the problem of maximizing a quadratic function on the sphere. This problem was completely solved by Golub [46] (1973). The problem is the following: Given ann×n real symmetric matrixA and ann×p matrixC,

minimize x Ax

subject to x x = 1, C x = 0, x∈ Rn.

Golub shows that the linear constraintC x = 0 can be eliminated as follows: If we use aQR decomposition ofC, by permuting the columns, we may assume that

C =Q R S Π,0 0

whereR is anr×r invertible upper triangular matrix andS is anr×(p−r) matrix (assuming C has rankr). Then if we let

x =Q y ,z

wherey∈ Rr andz∈ Rn−r, thenC x = 0 becomes

Π R 0 Qx = Π R 0 y = 0,S 0 S 0 z which impliesy = 0, and every solution ofC x = 0 is of the form

x =Q^0 .z

Our original problem becomes

minimize

(

y

, z

)

QAQ y z subject to z z = 1, z∈ Rn−r, y = 0, y∈ Rr. Thus, the constraintC x = 0 has been eliminated, and if we write

QAQ = G11 G12 ,G12 G22

our problem becomes minimize z G_22z

subject to z z = 1, z∈ Rn−r, a standard eigenvalue problem. Observe that if we let

J =0 0 ,0 In−r

then

JQAQ J =0 0 ,0 G22

and if we set

P =Q JQ, then

P AP =Q JQAQ JQ.

Now, Q JQAQ JQ andJQAQ J have the same eigenvalues, soP AP andJQAQ J also have the same eigenvalues. It follows that the solutions of our optimization problem are among the eigenvalues ofK =P AP, and at leastr of those are 0. Using the fact thatCC+ is the projection onto the range ofC, whereC+ is the pseudo-inverse ofC, it can also be shown that

P =I−CC+,

the projection onto the kernel ofC . In particular, whenn≥p andC has full rank (the columns ofC are linearly independent), then we know thatC = (C C)−1C and P =I^1C .−C(C C)−

This fact is used by Cour and Shi [23] and implicitly by Yu and Shi [111].

The problem of adding affine constraints of the form N x =t, wheret = 0, also comes up in practice. At first glance, this problem may not seem harder than the linear problem in whicht = 0, but it is. This problem was extensively studied in a paper by Gander, Golub, and von Matt [43] (1989).

Gander, Golub, and von Matt consider the following problem: Given an ( n+m)×(n+m) real symmetric matrixA (withn > 0), an (n+m)×m matrixN with full rank, and a nonzero vectort∈ Rm with (N )†t < 1 (where (N )† denotes the pseudo-inverse ofN ),

minimize x Ax

subject to x x = 1, N x =t, x∈ Rn+m.

The condition (N )†t < 1 ensures that the problem has a solution and is not trivial. The authors begin by proving that the affine constraintN x =t can be eliminated. One way to do so is to use aQR decomposition ofN. If

N =P R ,0

whereP is an orthogonal matrix andR is anm×m invertible upper triangular matrix, then if we observe that

x Ax =x P P AP P x, N x = (R ,0)P x =t, x x =x P P x = 1,

and if we write

P

AP

= B Γ Γ C and

P x = y ,z

then we get

x Ax =y By + 2z Γy +z Cz, R y =t,

y y +z z = 1.

Thus

y = (R )−1t, and if we write s^2 = 1−y y > 0 and

b = Γy, we get the simplified problem

minimize z Cz + 2z b

subject to z z =s^2, z∈ Rm.

Unfortunately, if b = 0, Proposition 17.6 is no longer applicable. It is still possible to find the minimum of the functionz Cz + 2z b using Lagrange multipliers, but such a solution is too involved to be presented here. Interested readers will find a thorough discussion in Gander, Golub, and von Matt [43].

18.4 Summary

The main concepts and results of this chapter are listed below:

• Quadratic optimization problems; quadratic functions.

• Symmetric positive definite and positive semidefinite matrices.

• The positive semidefinite cone ordering.

• Existence of a global minimum whenA is symmetric positive definite.

• Constrained quadratic optimization problems.

• Lagrange multipliers; Lagrangian.

• Primal and dual problems.

• Quadratic optimization problems: the case of a symmetric invertible matrixA.

• Quadratic optimization problems: the general case of a symmetric matrixA.

• Adding linear constraints of the formC x = 0.

• Adding affine constraints of the formC x =t, witht = 0.

• Maximizing a quadratic function over the unit sphere.

• Maximizing a quadratic function over an ellipsoid.

• Maximizing a Hermitian quadratic form.

• Adding linear constraints of the formC x = 0.

• Adding affine constraints of the formN x =t, witht = 0.

Chapter 19 Basics of Affine Geometry

L’alg`ebre n’est qu’une g´eom´etrie ´ecrite; la g´eom´etrie n’est qu’une alg`ebre figur´ee. —Sophie Germain

19.1 Affine Spaces

Geometrically, curves and surfaces are usually considered to be sets of points with some special properties, living in a space consisting of “points.” Typically, one is also interested in geometric properties invariant under certain transformations, for example, translations, rotations, projections, etc. One could model the space of points as a vector space, but this is not very satisfactory for a number of reasons. One reason is that the point corresponding to the zero vector (0), called the origin, plays a special role, when there is really no reason to have a privileged origin. Another reason is that certain notions, such as parallelism, are handled in an awkward manner. But the deeper reason is that vector spaces and affine spaces really have different geometries. The geometric properties of a vector space are invariant under the group of bijective linear maps, whereas the geometric properties of an affine space are invariant under the group of bijective affine maps, and these two groups are not isomorphic. Roughly speaking, there are more affine maps than linear maps.

Affine spaces provide a better framework for doing geometry. In particular, it is possible to deal with points, curves, surfaces, etc., in an intrinsic manner, that is, independently of any specific choice of a coordinate system. As in physics, this is highly desirable to really understand what is going on. Of course, coordinate systems have to be chosen to finally carry out computations, but one should learn to resist the temptation to resort to coordinate systems until it is really necessary.

Affine spaces are the right framework for dealing with motions, trajectories, and physical forces, among other things. Thus, affine geometry is crucial to a clean presentation of kinematics, dynamics, and other parts of physics (for example, elasticity). After all, a rigid motion is an affine map, but not a linear map in general. Also, given anm×n matrixA and a vectorb∈ Rm, the setU ={x∈ Rn Ax =b} of solutions of the systemAx =b is an|

477 affine space, but not a vector space (linear space) in general.

Use coordinate systems only when needed!

This chapter proceeds as follows. We take advantage of the fact that almost every affine concept is the counterpart of some concept in linear algebra. We begin by defining affine spaces, stressing the physical interpretation of the definition in terms of points (particles) and vectors (forces). Corresponding to linear combinations of vectors, we define affine combinations of points (barycenters), realizing that we are forced to restrict our attention to families of scalars adding up to 1. Corresponding to linear subspaces, we introduce affine subspaces as subsets closed under affine combinations. Then, we characterize affine subspaces in terms of certain vector spaces called their directions. This allows us to define a clean notion of parallelism. Next, corresponding to linear independence and bases, we define affine independence and affine frames. We also define convexity. Corresponding to linear maps, we define affine maps as maps preserving affine combinations. We show that every affine map is completely defined by the image of one point and a linear map. Then, we investigate briefly some simple affine maps, the translations and the central dilatations. At this point, we give a glimpse of affine geometry. We prove the theorems of Thales, Pappus, and Desargues. After this, the definition of affine hyperplanes in terms of affine forms is reviewed. The section ends with a closer look at the intersection of affine subspaces.

Our presentation of affine geometry is far from being comprehensive, and it is biased toward the algorithmic geometry of curves and surfaces. For more details, the reader is referred to Pedoe [85], Snapper and Troyer [95], Berger [6, 7], Coxeter [24], Samuel [87], Tisseron [105], and Hilbert and Cohn-Vossen [54].

Suppose we have a particle moving in 3D space and that we want to describe the trajectory of this particle. If one looks up a good textbook on dynamics, such as Greenwood [49], one finds out that the particle is modeled as a point, and that the position of this pointx is determined with respect to a “frame” in R^3 by a vector. Curiously, the notion of a frame is rarely defined precisely, but it is easy to infer that a frame is a pair (O,(e_1, e_2, e_3)) consisting of an originO (which is a point) together with a basis of three vectors (e_1, e_2, e_3). For example, the standard frame in R^3 has originO = (0,0,0) and the basis of three vectors e_1 = (1,0,0),e_2 = (0,1,0), ande_3 = (0,0,1). The position of a pointx is then defined by the “unique vector” fromO tox.

But wait a minute, this definition seems to be defining frames and the position of a point without defining what a point is! Well, let us identify points with elements of R^3. If so, given any two pointsa = (a_1, a_2, a_3) andb = (b_1, b_2, b_3), there is a unique free vector, denoted by −→, froma tob, the vector−→ = (b1−a_1, b2−a_2, b3−a_3). Note that

b =a +−→ab,

addition being understood as addition in R^3. Then, in the standard frame, given a point x = (x_1, x_2, x_3), the position ofx is the vector−→Ox = (x_1, x_2, x_3), which coincides with the point itself. In the standard frame, points and vectors are identified. Points and free vectors are illustrated in Figure 19.1.

b

−→

a O

Figure 19.1: Points and free vectors

What if we pick a frame with a different origin, say & = (ω_1, ω_2, ω_3), but the same basis vectors (e_1, e_2, e_3)? This time, the pointx = (x_1, x_2, x_3) is defined by two position vectors:

Ox = (x_1, x_2, x_3)

−→

in the frame (O,(e_1, e_2, e_3)) and

in the frame (&,(e_1, e_2, e_3)). This is because

&x = (x1−ω_1, x2−ω_2, x3−ω_3)

−→

Ox =O& +&x andO& = (ω_1, ω_2, ω_3).

−→ −→ −→ −→

We note that in the second frame (& ,(e_1, e_2, e_3)), points and position vectors are no longer identified. This gives us evidence that points are not vectors. It may be computationally convenient to deal with points using position vectors, but such a treatment is not frame invariant, which has undesirable effets.

Inspired by physics, we deem it important to define points and properties of points that are frame invariant. An undesirable side effect of the present approach shows up if we attempt to define linear combinations of points. First, let us review the notion of linear combination of vectors. Given two vectorsu andv of coordinates (u_1, u_2, u_3) and (v_1, v_2, v_3) with respect to the basis (e_1, e_2, e_3), for any two scalarsλ, µ, we can define the linear combinationλu+µv as the vector of coordinates

(λu_1 +µv_1, λu_2 +µv_2, λu_3 +µv_3). If we choose a different basis (e_1, e_2, e_3) and if the matrixP expressing the vectors (e_1, e_2, e_3) over the basis (e_1, e_2, e_3) is

ëa_1 b_1 c_1ö P = ía_2 b_2 c_2ø, a_3 b_3 c_3 which means that the columns ofP are the coordinates of theej over the basis (e_1, e_2, e_3), since

u_1e_1 +u_2e_2 +u_3e_3 =u_1e_1 +u_2e_2 +u_3e_3 and v_1e_1 +v_2e_2 +v_3e_3 =v_1e_1 +v_2e_2 +v_3e_3,

it is easy to see that the coordinates ( u_1, u_2, u_3) and (v_1, v_2, v_3) ofu andv with respect to the basis (e_1, e_2, e_3) are given in terms of the coordinates (u_1, u_2, u_3) and (v_1, v_2, v_3) ofu and v with respect to the basis (e_1, e_2, e_3) by the matrix equations

ëu1ö ëu1ö ëv1ö ëv1ö

íu_2ø =P íu_2ø and ív_2ø =P ív_2ø.

u_3 u_3 v_3 v_3

From the above, we get

ëu1ö ëu1ö ëv1ö ëv1ö

íu_2ø =P−1 íu_2ø and ív_2ø =P−1 ív_2ø,

u_3 u_3 v_3 v_3

and by linearity, the coordinates

(λu_1 +µv_1, λu_2 +µv_2, λu_3 +µv_3)

ofλu +µv with respect to the basis (e_1, e_2, e_3) are given by

ëλu_1 +µv_1ö ëu_1ö ëv_1ö ëλu_1 +µv_1ö íλu_2 +µv_2ø =λP−1 íu_2ø +µP−1ív_2ø =P−1 íλu_2 +µv_2ø. λu_3 +µv_3 u_3 v_3 λu_3 +µv_3

Everything worked out because the change of basis does not involve a change of origin. On the other hand, if we consider the change of frame from the frame (O,(e_1, e_2, e_3)) to the frame (&,(e_1, e_2, e_3)), where−→& = (ω_1, ω_2, ω_3), given two pointsa,b of coordinates (a_1, a_2, a_3) and (b_1, b_2, b_3) with respect to the frame (O,(e_1, e_2, e_3)) and of coordinates (a_1, a_2, a_3) and (b_1, b_2, b_3) with respect to the frame (&,(e_1, e_2, e_3)), since

(a_1, a_2, a_3) = (a1−ω_1, a2−ω_2, a3−ω_3)

and (b_1, b_2, b_3) = (b1−ω_1, b2−ω_2, b3−ω_3), the coordinates ofλa +µb with respect to the frame (O,(e_1, e_2, e_3)) are

(λa_1 +µb_1, λa_2 +µb_2, λa_3 +µb_3), but the coordinates (λa_1 +µb_1, λa_2 +µb_2, λa_3 +µb_3) ofλa +µb with respect to the frame (&,(e_1, e_2, e_3)) are

(λa_1 +µb1− (λ +µ)ω_1, λa_2 +µb2− (λ +µ)ω_2, λa_3 +µb3− (λ +µ)ω_3), which are different from (λa_1 +µb1−ω_1, λa_2 +µb2−ω_2, λa_3 +µb3−ω_3),

unlessλ +µ = 1.

Thus, we have discovered a major difference between vectors and points: The notion of linear combination of vectors is basis independent, but the notion of linear combination of points is frame dependent. In order to salvage the notion of linear combination of points, some restriction is needed: The scalar coefficients must add up to 1.

A clean way to handle the problem of frame invariance and to deal with points in a more intrinsic manner is to make a clearer distinction between points and vectors. We duplicate R^3 into two copies, the first copy corresponding to points, where we forget the vector space structure, and the second copy corresponding to free vectors, where the vector space structure is important. Furthermore, we make explicit the important fact that the vector spaceR^3 acts on the set of points R^3: Given any pointa = (a_1, a_2, a_3) and any vectorv = (v_1, v_2, v_3), we obtain the point

a +v = (a_1 +v_1, a_2 +v_2, a_3 +v_3),

which can be thought of as the result of translating a tob using the vectorv. We can imagine thatv is placed such that its origin coincides witha and that its tip coincides withb. This action +: R^3 R^3 R^3 satisfies some crucial properties. For example,× →

a + 0 = a,

(a +u) +v = a + (u +v),

and for any two pointsa, b, there is a unique free vector−→ such that

b =a +−→ab.

It turns out that the above properties, although trivial in the case of R^3, are all that is needed to define the abstract notion of affine space (or affine structure). The basic idea is to consider two (distinct) setsE andE, whereE is a set of points (with no structure) and E is a vector space (of free vectors) acting on the setE.

Did you say “A fine space”?

Intuitively, we can think of the elements ofE as forces moving the points inE, considered as physical particles. The effect of applying a force (free vector)u∈E to a pointa∈E is a translation. By this, we mean that for every forceu∈E, the action of the forceu is to “move” every pointa∈E to the pointa+u∈E obtained by the translation corresponding tou viewed as a vector. Since translations can be composed, it is natural thatE is a vector space.

For simplicity, it is assumed that all vector spaces under consideration are defined over the fieldR of real numbers. Most of the definitions and results also hold for an arbitrary field K, although some care is needed when dealing with fields of characteristic different from zero (see the problems). It is also assumed that all families (λi)i∈I of scalars have finite support. Recall that a family (λi)i∈I of scalars has finite support ifλi = 0 for alli∈I−J, where J is a finite subset ofI. Obviously, finite families of scalars have finite support, and for simplicity, the reader may assume that all families of scalars are finite. The formal definition of an affine space is as follows.

Definition 19.1. An affine space is either the degenerate space reduced to the empty set, or a triple E, E,+ consisting of a nonempty setE (of points), a vector spaceE (of translations, or free vectors), and an action +:E×E→E, satisfying the following conditions.

(A1) a + 0 =a, for everya∈E. (A2) (a +u) +v =a + (u +v), for everya∈E, and everyu, v∈E. (A3) For any two pointsa, b∈E, there is a uniqueu∈E such thata +u =b.

The unique vectoru∈E such thata +u =b is denoted byab, or sometimes by ab, or−→

even byb−a. Thus, we also write

b =a +−→

(or b =a +ab, or evenb =a + (b−a)).

The dimension of the affine space E, E,+ is the dimension dim(E) of the vector space

E. For simplicity, it is denoted by dim(E).

Conditions (A1) and (A2) say that the (abelian) groupE acts onE, and condition (A3) says thatE acts transitively and faithfully onE. Note that

−−−−−→

a(a +v) =v

for all a∈E and allv∈E, since−−−−−→) is the unique vector such thata+v =a+−−−−−→). Thus,b =a +v is equivalent to−→ =v. Figure 19.2 gives an intuitive picture of an affine space. It is natural to think of all vectors as having the same origin, the null vector.

The axioms defining an affine space E, E,+ can be interpreted intuitively as saying thatE andE are two different ways of looking at the same object, but wearing different sets of glasses, the second set of glasses depending on the choice of an “origin” inE. Indeed, E −→

b= a+ u u a c= a+ w w v

Figure 19.2: Intuitive picture of an affine space

we can choose to look at the points in E, forgetting that every pair (a, b) of points defines a unique vector−→ inE, or we can choose to look at the vectorsu inE, forgetting the points inE. Furthermore, if we also pick any pointa inE, a point that can be viewed as an origin inE, then we can recover all the points inE as the translated pointsa +u for allu∈E. This can be formalized by defining two maps betweenE andE.

For everya∈E, consider the mapping fromE toE given by

u→a +u, whereu∈E, and consider the mapping fromE toE given by b→−→ab, whereb∈E. The composition of the first mapping with the second is

u→a +u→−−−−−→),

which, in view of (A3), yieldsu. The composition of the second with the first mapping is

b→−→→a +−→ab,

which, in view of (A3), yieldsb. Thus, these compositions are the identity fromE toE and the identity fromE toE, and the mappings are both bijections.

When we identify E withE via the mappingb→−→, we say that we considerE as the vector space obtained by takinga as the origin inE , and we denote it byEa. BecauseEa is a vector space, to be consistent with our notational conventions we should use the notation Ea (using an arrow), instead ofEa. However, for simplicity, we stick to the notationEa.

Thus, an affine space E, E,+ is a way of defining a vector space structure on a set of pointsE, without making a commitment to a fixed origin inE. Nevertheless, as soon as we commit to an origina inE, we can viewE as the vector spaceEa. However, we urge the reader to think ofE as a physical set of points and ofE as a set of forces acting onE, rather than reducingE to some isomorphic copy of Rn. After all, points are points, and not vectors! For notational simplicity, we will often denote an affine space E, E,+ by (E, E), or even byE. The vector spaceE is called the vector space associated withE.

One should be careful about the overloading of the addition symbol +. Addition is well-defined on vectors, as inu +v; the translatea +u of a pointa∈E by a vectoru∈E is also well-defined, but addition of pointsa +b does not make sense . In this respect, the notationb−a for the unique vectoru such thatb =a +u is somewhat confusing, since it suggests that points can be subtracted (but not added!).

Any vector space E has an affine space structure specified by choosingE =E, and letting + be addition in the vector spaceE. We will refer to the affine structure E, E,+ on a vector spaceE as the canonical (or natural) affine structure onE. In particular, the vector space Rn can be viewed as the affine space Rn,Rn,+ , denoted by An. In general, ifK is any field, the affine space Kn, Kn,+ is denoted by AnK. In order to distinguish between the double role played by members of Rn, points and vectors, we will denote points by row vectors, and vectors by column vectors. Thus, the action of the vector space Rn over the set Rn simply viewed as a set of points is given by

ëu1ö

(a_1, . . . , an) +ì .÷ = (a_1 +u_1, . . . , an +un).í ø

un

We will also use the convention that if x = (x_1, . . . , xn)∈ Rn, then the column vector associated withx is denoted by x (in boldface notation). Abusing the notation slightly, if a∈ Rn is a point, we also writea∈ An. The affine space An is called the real affine space of

dimensionn. In most cases, we will considern = 1,2,3.

19.2 Examples of Affine Spaces

Let us now give an example of an affine space that is not given as a vector space (at least, not in an obvious fashion). Consider the subsetL of A^2 consisting of all points (x, y) satisfying the equation

x +y− 1 = 0.

The setL is the line of slope−1 passing through the points (1,0) and (0,1) shown in Figure 19.3.

The lineL can be made into an official affine space by defining the action +:L×R→L of R onL defined such that for every point (x,1−x) onL and anyu∈ R,

(x,1−x) +u = (x +u,1−x−u). 19.2. EXAMPLES OF AFFINE SPACES 485

L

Figure 19.3: An affine space: the line of equationx +y− 1 = 0

It is immediately verified that this action makes L into an affine space. For example, for any two pointsa = (a_1,1−a_1) andb = (b_1,1−b_1) onL, the unique (vector)u∈ R such that b =a +u isu =b1−a1. Note that the vector space R is isomorphic to the line of equation x +y = 0 passing through the origin.

Similarly, consider the subsetH of A^3 consisting of all points (x, y, z) satisfying the equation

x +y +z− 1 = 0.

The setH is the plane passing through the points (1,0,0), (0,1,0), and (0,0,1). The plane H can be made into an official affine space by defining the action +:H×R^2 H of R^2 on H

defined such that for every point (

x, y,

1

−

x

−

y

) on

H

and any

u

→ R2,v ∈

(x, y,1−x−y) + u = (x +u, y +v,1−x−u−y−v).v

For a slightly wilder example, consider the subsetP of A^3 consisting of all points (x, y, z) satisfying the equation

x^2 +y^2 z = 0.−

The setP is a paraboloid of revolution, with axisOz. The surfaceP can be made into an official affine space by defining the action +:P×R^2 P of R^2 onP defined such that for every point (

x, y, x

2

+

y

2

) on

P

and any

u

R

2

→

,v ∈

(x, y, x^2 +y^2) + u = (x +u, y +v,(x +u)2 + (y +v)2).v

E

E

−→

b

ab

−→aca c −→

−→

Figure 19.4: Points and corresponding vectors in affine geometry

This should dispell any idea that affine spaces are dull. Affine spaces not already equipped with an obvious vector space structure arise in projective geometry.

19.3 Chasles’s Identity

Given any three pointsa, b, c∈E, sincec =a +ac,b =a +−→, andc =b +−→, we get−→

c =b +−→ = (a +−→) +−→ =a + (ab +−→)

by (A2), and thus, by (A3), −→

+

−→

=

ac,

−→

which is known as Chasles’s identity, and illustrated in Figure 19.4. Sincea =a +aa and by (A1)a =a + 0, by (A3) we get−→

aa = 0.

−→

Thus, lettinga =c in Chasles’s identity, we get

ba =−−→ab.−→

Given any four pointsa, b, c, d∈E, since by Chasles’s identity

−→

+

−→

=

−→ + −→ =

ac,

−→ we have the parallelogram law

ab =dc iff −→bc =−→ad.−→ −→

19.4 Affine Combinations, Barycenters

A fundamental concept in linear algebra is that of a linear combination. The corresponding concept in affine geometry is that of an affine combination, also called a barycenter. However, there is a problem with the naive approach involving a coordinate system, as we saw in Section 19.1. Since this problem is the reason for introducing affine combinations, at the risk of boring certain readers, we give another example showing what goes wrong if we are not careful in defining linear combinations of points.

Consider R^2 as an affine space, under its natural coordinate system with originO = (0,0) and basis vectors^1 and^0 . Given any two pointsa = (a_1, a_2) andb = (b_1, b_2), it is0 1

natural to define the affine combinationλa +µb as the point of coordinates

(λa_1 +µb_1, λa_2 +µb_2).

Thus, whena = (−1,−1) andb = (2,2), the pointa +b is the pointc = (1,1).

Let us now consider the new coordinate system with respect to the origin c = (1,1) (and the same basis vectors). This time, the coordinates ofa are (−2,−2), the coordinates ofb are (1,1), and the pointa +b is the pointd of coordinates (−1,−1). However, it is clear that the pointd is identical to the originO = (0,0) of the first coordinate system.

Thus, a+b corresponds to two different points depending on which coordinate system is used for its computation!

This shows that some extra condition is needed in order for affine combinations to make sense. It turns out that if the scalars sum up to 1, the definition is intrinsic, as the following lemma shows.

Lemma 19.1. Given an affine spaceE, let (ai)i∈I be a family of points inE, and let (λi)i∈I be a family of scalars. For any two pointsa, b∈E, the following properties hold:

(1) Ifi∈Iλi = 1, then

a

+

λ

aa

i−→ =b + λi−→.

i∈I i∈I

(2) Ifi∈Iλi = 0, then

λ

aa

i−→ = λi−→i.

i∈I i∈I

Proof. (1) By Chasles’s identity (see Section 19.3), we have

a

+

λ

aa

i−→ =a + λi(−→ +−→i)

i∈I i∈I

=a + λi −→ + λi−→i

i∈I i∈I

=a +−→ + λi−→ sincei∈Iλi = 1

i∈I

=b + λi−→i sinceb =a +−→ab.

i∈I

(2) We also have

λ

aa

i−→ = λi(−→ +−→i)

i∈I i∈I

= λi −→ + λi−→i

i∈I i∈I

= λi−→i,

i∈I

sincei∈Iλi = 0. Thus, by Lemma 19.1, for any family of points (ai)i∈I inE, for any family (λi)i∈I of scalars such thati∈Iλi = 1, the point

x

=

a

+

λ

aa

i−→

i∈I

is independent of the choice of the origina∈E. This property motivates the following definition.

Definition 19.2. For any family of points (ai)i∈I inE, for any family (λi)i∈I of scalars such thati∈Iλi = 1, and for anya∈E, the point

a

+

λ

aa

i−→

i∈I

(which is independent of a∈E, by Lemma 19.1) is called the barycenter (or barycentric combination, or affine combination) of the pointsai assigned the weightsλi, and it is denoted by

λiai.

i∈I

In dealing with barycenters, it is convenient to introduce the notion of a weighted point, which is just a pair (a, λ), wherea∈E is a point, andλ∈ R is a scalar. Then, given a family of weighted points ((ai, λi))i∈I, wherei∈Iλi = 1, we also say that the pointi∈Iλiai is the barycenter of the family of weighted points ((ai, λi))i∈I.

Note that the barycenterx of the family of weighted points ((ai, λi))i∈I is the unique point such that

ax

=

λ

aa

i

−→

for every

a∈E, −→

i∈I

and settinga =x, the pointx is the unique point such that

λ

xa

i−→ = 0.

i∈I

In physical terms, the barycenter is the center of mass of the family of weighted points ((ai, λi))i∈I (where the masses have been normalized, so thati∈Iλi = 1, and negative masses are allowed).

Remarks:

(1) Since the barycenter of a family (( ai, λi))i∈I of weighted points is defined for families (λi)i∈I of scalars with finite support (and such thati∈Iλi = 1), we might as well assume thatI is finite. Then, for allm≥ 2, it is easy to prove that the barycenter ofm weighted points can be obtained by repeated computations of barycenters of two weighted points.

(2) This result still holds, provided that the fieldK has at least three distinct elements, but the proof is trickier!

(3) When

i

∈

I

λ

i

= 0, the vector

i

∈

I

λ

aa

i −→ does not depend on the pointa, and we may denote it byi∈Iλiai . This observation will be used to define a vector space in which linear combinations of both points and vectors make sense, regardless of the value of i∈Iλi.

Figure 19.5 illustrates the geometric construction of the barycenters g_1 andg_2 of the weighted points a, 1 , b, 1 , and c, 1 , and (a,−1), (b,1), and (c,1).4 4 2

The pointg_1 can be constructed geometrically as the middle of the segment joiningc to the middle 1a + 1b of the segment (a, b), since2 2

g1 = 1^1a + 1b + 1c.

2 2 2 2

The pointg_2 can be constructed geometrically as the point such that the middle 1b + 1c of the segment (b, c) is the middle of the segment (a, g_2), since2 2

g_2 =−a + 2^1b + 1c .2 2 c

g_1

a b

c g_2

a b

Figure 19.5: Barycenters,g_1 = 1a + 1b + 1c, g_2 =−a +b +c4 4 2

Later on, we will see that a polynomial curve can be defined as a set of barycenters of a fixed number of points. For example, let (a, b, c, d) be a sequence of points in A^2. Observe that

(1−t)3 + 3t(1−t)2 + 3t^2(1−t) +t^3 = 1,

since the sum on the left-hand side is obtained by expanding (t + (1−t))3 = 1 using the binomial formula. Thus,

(1−t)3a + 3t(1−t)2b + 3t^2(1−t)c +t^3d

is a well-defined affine combination. Then, we can define the curveF : A→ A^2 such that F(t) = (1−t)3a + 3t(1−t)2b + 3t^2(1−t)c +t^3d.

Such a curve is called a B´ezier curve, and (a, b, c, d) are called its control points. Note that the curve passes througha andd, but generally not throughb andc. It can be sbown that any pointF(t) on the curve can be constructed using an algorithm performing affine interpolation steps (the de Casteljau algorithm).

19.5 Affine Subspaces

In linear algebra, a (linear) subspace can be characterized as a nonempty subset of a vector space closed under linear combinations. In affine spaces, the notion corresponding to the notion of (linear) subspace is the notion of affine subspace. It is natural to define an affine subspace as a subset of an affine space closed under affine combinations.

Definition 19.3. Given an affine space E, E,+ , a subsetV ofE is an affine subspace (of

E, E,+ ) if for every family of weighted points ((ai, λi))i∈I inV such thati∈Iλi = 1, the barycenteri∈Iλiai belongs toV .

An affine subspace is also called a flat by some authors. According to Definition 19.3, the empty set is trivially an affine subspace, and every intersection of affine subspaces is an affine subspace.

As an example, consider the subsetU of R^2 defined by

U = (x, y)∈ R^2 ax +by =c ,|

i.e., the set of solutions of the equation

ax +by =c, where it is assumed thata = 0 orb = 0. Given anym points (xi, yi)∈U and anym scalars λi such thatλ_1 +· · · +λm = 1, we claim that

m

λi(xi, yi)∈U.

i=1

Indeed, (xi, yi)∈U means that axi +byi =c,

and if we multiply both sides of this equation byλi and add up the resultingm equations, we getm m

(λiaxi +λibyi) = λic,

i=1 i=1

and sinceλ_1 +· · · +λm = 1, we get

m m m

a λixi +b λiyi = λi c =c,

i=1 i=1 i=1

which shows thatm m m

λixi, λiyi = λi(xi, yi)∈U.

i=1 i=1 i=1

Thus,U is an affine subspace of A^2. In fact, it is just a usual line in A^2. It turns out thatU is closely related to the subset of R^2 defined by U = (x, y)∈ R^2 ax +by = 0 ,|

i.e., the set of solutions of the homogeneous equation ax +by = 0

U

−→

Figure 19.6: An affine lineU and its direction

obtained by setting the right-hand side ofax+by =c to zero. Indeed, for anym scalarsλi, the same calculation as above yields that

m

λi(xi, yi)∈U,

i=1

this time without any restriction on theλi, since the right-hand side of the equation is null. Thus,U is a subspace of R^2. In fact,U is one-dimensional, and it is just a usual line in R^2. This line can be identified with a line passing through the origin of A^2, a line that is parallel to the lineU of equationax +by =c, as illustrated in Figure 19.6.

Now, if (x_0, y_0) is any point inU, we claim that

U = (x_0, y_0) +U, where (x_0, y_0) +U = (x_0 +u_1, y_0 +u_2)| (u_1, u_2)∈U .

First, (x_0, y_0) +U⊆U, sinceax_0 +by_0 =c andau_1 +bu_2 = 0 for all (u_1, u_2)∈U. Second, if (x, y)∈U, thenax+by =c, and since we also haveax_0 +by_0 =c, by subtraction, we get a(x−x_0) +b(y−y_0) = 0, which shows that (x− x_0, y−y_0)∈U, and thus (x, y)∈ (x_0, y_0) +U. Hence, we also have U⊆ (x_0, y_0) +U, andU = (x0, y_0) +U.

The above example shows that the affine lineU defined by the equation ax +by =c is obtained by “translating” the parallel lineU of equation ax +by = 0 passing through the origin. In fact, given any point (x_0, y_0)∈U, U = (x_0, y_0) +U. More generally, it is easy to prove the following fact. Given anym×n matrixA and any vectorb∈ Rm, the subsetU of Rn defined by

U

=

{

x

∈

R

n

|Ax =b}

is an affine subspace of An.

Actually, observe that Ax =b should really be written asAx =b, to be consistent with our convention that points are represented by row vectors. We can also use the boldface notation for column vectors, in which case the equation is written asAx =b. For the sake of minimizing the amount of notation, we stick to the simpler (yet incorrect) notationAx =b. If we consider the corresponding homogeneous equationAx = 0, the set

U

=

{x∈ R |

n Ax = 0}

is a subspace of Rn, and for anyx0∈U, we have

U =x_0 +U.

This is a general situation. Affine subspaces can be characterized in terms of subspaces of E. LetV be a nonempty subset ofE. For every family (a_1, . . . , an) inV , for any family (λ_1, . . . , λn) of scalars, and for every pointa∈V , observe that for everyx∈E,

n

x

=

a

+

λ

aa

i−→

i=1

is the barycenter of the family of weighted points

n

(

a

1

, λ

1

)

, . . . ,

(

a

n

, λ

n

)

,

a,

1 −

i=1

λi ,

sincen n

λi = 1.

i=1

λi + 1− i=1

Given any pointa∈E and any subsetV ofE, leta +V denote the following subset ofE:

a +V = a +v|v∈V .

E

E

−→

a −→ V = a+−→

Figure 19.7: An affine subspaceV and its directionV

Lemma 19.2. Let E, E,+ be an affine space.

(1) A nonempty subsetV ofE is an affine subspace iff for every pointa∈V , the set

V

a

=

ax

{−→|x∈V} is a subspace ofE. Consequently,V =a +Va. Furthermore,

V = xy|x, y∈V

{−→ }

is a subspace ofE andVa =V for alla∈E. Thus,V =a +V .

(2) For any subspaceV ofE and for anya∈E, the setV =a +V is an affine subspace.

Proof. The proof is straightforward, and is omitted. It is also given in Gallier [41].

In particular, when E is the natural affine space associated with a vector spaceE, Lemma 19.2 shows that every affine subspace ofE is of the formu+U, for a subspaceU ofE. The subspaces ofE are the affine subspaces ofE that contain 0.

The subspace V associated with an affine subspaceV is called the direction ofV . It is also clear that the map +:V×V→V induced by +:E×E→E confers to V, V,+ an affine structure. Figure 19.7 illustrates the notion of affine subspace.

By the dimension of the subspaceV , we mean the dimension ofV .

An affine subspace of dimension 1 is called a line, and an affine subspace of dimension 2 is called a plane.

An affine subspace of codimension 1 is called a hyperplane (recall that a subspaceF of a vector spaceE has codimension 1 iff there is some subspaceG of dimension 1 such that E =F⊕G, the direct sum ofF andG, see Strang [101] or Lang [65]).

We say that two affine subspacesU andV are parallel if their directions are identical. Equivalently, sinceU =V , we haveU =a+U andV =b+U for anya∈U and anyb∈V , and thusV is obtained fromU by the translation−→.

In general, when we talk aboutn pointsa_1, . . . , an, we mean the sequence (a_1, . . . , an), and not the set{a_1, . . . , an} (theai’s need not be distinct).

By Lemma 19.2, a line is specified by a pointa∈E and a nonzero vectorv∈E, i.e., a line is the set of all points of the forma +λv, forλ∈ R.

We say that three pointsa, b, c are collinear if the vectors−→ andac are linearly depen−→

dent. If two of the pointsa, b, c are distinct, saya =b, then there is a uniqueλ∈ R such

that

ac =λ−→, and we define the ratioac =λ.−→ −→

ab

A plane is specified by a point a∈E and two linearly independent vectorsu, v∈E, i.e., a plane is the set of all points of the forma +λu +µv, forλ, µ∈ R.

We say that four pointsa, b, c, d are coplanar if the vectors−→ab,ac, and−→ are linearly−→

dependent. Hyperplanes will be characterized a little later.

Lemma 19.3. Given an affine space E, E,+ , for any family (ai)i∈I of points inE, the

setV of barycentersi∈Iλiai (wherei∈Iλi = 1) is the smallest affine subspace containing

(ai)i∈I.

Proof. If (ai)i∈I is empty, thenV =∅, because of the conditioni∈Iλi = 1. If (ai)i∈I is nonempty, then the smallest affine subspace containing (ai)i∈I must contain the setV of barycentersi∈Iλiai, and thus, it is enough to show thatV is closed under affine combina

tions, which is immediately verified.

Given a nonempty subset S ofE, the smallest affine subspace ofE generated byS is often denoted by S . For example, a line specified by two distinct pointsa andb is denoted by a, b , or even (a, b), and similarly for planes, etc.

Remarks:

(1) Since it can be shown that the barycenter of n weighted points can be obtained by repeated computations of barycenters of two weighted points, a nonempty subsetV ofE is an affine subspace iff for every two pointsa, b∈V , the setV contains all barycentric combinations ofa andb. IfV contains at least two points, thenV is an affine subspace iff for any two distinct pointsa, b∈V , the setV contains the line determined bya andb, that is, the set of all points (1−λ)a +λb,λ∈ R.

(2) This result still holds if the fieldK has at least three distinct elements, but the proof is trickier!

19.6 Affine Independence and Affine Frames

Corresponding to the notion of linear independence in vector spaces, we have the notion of affine independence. Given a family (ai)i∈I of points in an affine spaceE, we will reduce the notion of (affine) independence of these points to the (linear) independence of the families (aa )j∈(I−{i}) of vectors obtained by choosing anyai as an origin. First, the following lemma−−→

shows that it is sufficient to consider only one of these families.

Lemma 19.4. Given an affine space E, E,+ , let (ai)i∈I be a family of points inE. If the family (aa )j∈(I−{i}) is linearly independent for somei∈I, then (aa )j∈(I−{i}) is linearly−−→ −−→independent for every i∈I.

Proof. Assume that the family (aa )j∈(I−{i}) is linearly independent for some specifici∈I. Let

k

∈

I

with

k

=

i

−−→

, and assume that there are some scalars (λj)j∈(I−{k}) such that

λjaa = 0.−−→

j∈(I−{k})

Since aa =a ai + aa ,−−→ −−→ −−→ we have

λ

j

a

a = λjaa + λa j−−→aj,−−→ −−→

j∈(I−{k}) j∈(I−{k}) j∈(I−{k})

= λjaa + λjaa ,−−→ −−→

j∈(I−{k}) j∈(I−{i,k})

=

λ

j

a

a λ a

j −−→ak,−−→

j− j∈(I−{k})∈(I−{i,k})

and thus

j

λ

j

a

a λ a

j −−→ak = 0.−−→− j∈(I−{k})∈(I−{i,k})

Since the family (aa )j∈(I−{i}) is linearly independent, we must haveλj = 0 for allj∈(I− {i, k}) and −−→

j∈(I−{k})λj = 0, which implies thatλj = 0 for allj∈ (I− {k}). We define affine independence as follows.

Definition 19.4. Given an affine space E, E,+ , a family (ai)i∈I of points inE is affinely independent if the family (aa )j∈(I−{i}) is linearly independent for somei∈I.−−→

Definition 19.4 is reasonable, because by Lemma 19.4, the independence of the family (aa )j∈(I−{i}) does not depend on the choice ofai. A crucial property of linearly independent−−→

vectors (u1, . . . , um) is that if a vectorv is a linear combination

m

v = λiui

i=1

of theui, then theλi are unique. A similar result holds for affinely independent points. E −→

a_2

a

a

−−→ a

0

a

1

a

a

−−→

Figure 19.8: Affine independence and linear independence

Lemma 19.5. Given an affine space E, E,+ , let (a_0, . . . , am) be a family ofm+ 1 points inE. Letx∈E, and assume thatx =m λiai, wherem λi = 1. Then, the family

(

λ

0

, . . . , λ

m

)

such that

x

=

m

i=0 i=0

λiai is unique iff the family (aa , . . . ,aa ) is linearly independent.i=0 −−→ −−→ Proof. The proof is straightforward and is omitted. It is also given in Gallier [41].

Lemma 19.5 suggests the notion of affine frame. Affine frames are the affine analogues of bases in vector spaces. Let E, E,+ be a nonempty affine space, and let (a_0, . . . , am) be a family ofm + 1 points inE. The family (a_0, . . . , am) determines the family ofm vectors (aa , . . . , aa ) inE. Conversely, given a pointa_0 inE and a family ofm vectors−−→ −−→

(u_1, . . . , um) inE, we obtain the family ofm+ 1 points (a_0, . . . , am) inE, whereai =a_0+ui, 1≤i≤m.

Thus, for any m≥ 1, it is equivalent to consider a family ofm + 1 points (a_0, . . . , am) inE, and a pair (a_0,(u1, . . . , um)), where theui are vectors inE. Figure 19.8 illustrates the notion of affine independence.

Remark: The above observation also applies to infinite families (ai)i∈I of points inE and families (ui)i∈I−{0} of vectors inE, provided that the index setI contains 0. When (aa , . . . ,aa ) is a basis ofE then, for everyx∈E, sincex =a_0 + ax, there is−−→ −−→ −→a unique family (x1, . . . , xm) of scalars such that

x =a_0 +x_1aa +· · · +xmaa .−−→ −−→

The scalars (x_1, . . . , xm) may be considered as coordinates with respect to (a_0,(aa , . . . ,aa )). Since−−→ −−→

m m m

x =a_0 +−−→ xi a_0 + xiai,

i=1

xiaa iff x = 1− i=1 i=1 x∈E can also be expressed uniquely as

m

x = λiai

i=0

with

m λi = 1, and whereλ_0 = 1m

− i=1xi, andλi =xi for 1≤i≤m. The scalarsi=0

(λ_0, . . . , λm) are also certain kinds of coordinates with respect to (a0, . . . , am). All this is summarized in the following definition.

Definition 19.5. Given an affine space E, E,+ , an affine frame with origina_0 is a family (a_0, . . . , am) ofm + 1 points inE such that the list of vectors (aa , . . . ,aa ) is a basis of−−→ −−→ E. The pair (a_0,(aa , . . . ,aa )) is also called an affine frame with origina_0. Then, every−−→ −−→x∈E can be expressed as

x

=

a

0

+

x

1

a

a +· · · +xmaa

−−→ −−→

for a unique family (x_1, . . . , xm) of scalars, called the coordinates ofx w.r.t. the affine frame (a_0,(aa , . . ., aa )). Furthermore, everyx∈E can be written as−−→ −−→

x =λ_0a_0 +· · · +λmam for some unique family (λ_0, . . . , λm) of scalars such thatλ_0+· · ·+λm = 1 called the barycentric coordinates ofx with respect to the affine frame (a_0, . . . , am).

The coordinates (x_1, . . . , xm) and the barycentric coordinates (λ_0, . . .,λm) are related by the equations

λ

0

= 1

m

−

i

=1

xi andλi =xi, for 1≤i≤m. An affine frame is called an affine basis by some authors. A family (ai)i∈I of points inE is affinely dependent if it is not affinely independent. We can also characterize affinely dependent families as follows.

Lemma 19.6. Given an affine space E, E,+ , let (ai)i∈I be a family of points inE. The family (ai)i∈I is affinely dependent iff there is a family (λi)i∈I such thatλj = 0 for some

j

∈

I

,

i

∈

I

λ

i

= 0

, and

i

∈

I

λ

xa

i−→ = 0 for everyx∈E.

Proof. By Lemma 19.5, the family (ai)i∈I is affinely dependent iff the family of vectors (aa )j∈(I−{i}) is linearly dependent for somei∈I. For anyi∈ I, the family (aa )j∈(I−{i})−−→ −−→ is linearly dependent iff there is a family (λj)j∈(I−{i}) such thatλj = 0 for somej, and such that

λjaa = 0.−−→

j∈(I−{i})

Then, for anyx∈E, we have

λjaa = λj(xa xai)−−→ −→−−→j∈(I−{i}) j∈(I−{i})

=xa λj −→,

j

λ

j−→− j∈(I−{i})∈(I−{i})

xa

a_2

a_0 a_0 a_1

a_3

a_0 a_1 a_0 a_2

a_1

Figure 19.9: Examples of affine frames and their convex hulls

and letting

λ

i

=

−

j

∈

(

I

−{

i

}

)

λ

j

, we get

i

∈

I

λ

xa

i−→ = 0, withi∈Iλi = 0 andλj = 0 for somej∈I. The converse is obvious by settingx =ai for somei such thatλi = 0, since i∈Iλi = 0 implies thatλj = 0, for somej =i.

Even though Lemma 19.6 is rather dull, it is one of the key ingredients in the proof of beautiful and deep theorems about convex sets, such as Carath´eodory’s theorem, Radon’s theorem, and Helly’s theorem.

A family of two points (

a, b

) in

E

ab

is affinely independent iff −→ = 0, iffa =b. Ifa =b, the affine subspace generated bya andb is the set of all points (1−λ)a+λb, which is the unique line passing througha andb. A family of three points (a, b, c) inE is affinely independent iff−→ andac are linearly independent, which means thata,b, andc are not on the same line−→

(they are not collinear). In this case, the affine subspace generated by (a, b, c) is the set of all points (1−λ−µ)a +λb +µc, which is the unique plane containinga,b, andc. A family of four points (a, b, c, d) inE is affinely independent iff−→,ac, and−→ are linearly independent,−→

which means thata,b,c, andd are not in the same plane (they are not coplanar). In this case,a,b,c, andd are the vertices of a tetrahedron. Figure 19.9 shows affine frames and their convex hulls for|I| = 0,1,2,3.

Given n+1 affinely independent points (a_0, . . . , an) inE, we can consider the set of points λ_0a_0+· · ·+λnan, whereλ_0+· · ·+λn = 1 andλi≥ 0 (λi∈ R). Such affine combinations are called convex combinations. This set is called the convex hull of (a_0, . . . , an) (orn-simplex spanned by (a_0, . . . , an)). Whenn = 1, we get the segment betweena_0 anda_1, including a_0 anda_1. Whenn = 2, we get the interior of the triangle whose vertices area_0, a_1, a_2, including boundary points (the edges). Whenn = 3, we get the interior of the tetrahedron whose vertices area_0, a_1, a_2, a_3, including boundary points (faces and edges). The set

a

0

+

λ

1

a

a +· · · +λnaa

{ −−→ −−→| where 0≤λi≤ 1 (λi∈ R)} is called the parallelotope spanned by (a_0, . . . , an). WhenE has dimension 2, a parallelotope is also called a parallelogram, and whenE has dimension 3, a parallelepiped. More generally, we say that a subsetV ofE is convex if for any two pointsa, b∈V , we havec∈V for every pointc = (1−λ)a +λb, with 0≤λ≤ 1 (λ∈ R).

Points are not vectors! The following example illustrates why treating points as vectors may cause problems. Leta, b, c be three affinely independent points in A^3.

Any point x in the plane (a, b, c) can be expressed as

x =λ_0a +λ_1b +λ_2c,

where λ_0 +λ_1 +λ_2 = 1. How can we computeλ_0, λ_1, λ_2? Lettinga = (a_1, a_2, a_3),b = (b_1, b_2, b_3),c = (c_1, c_2, c_3), andx = (x_1, x_2, x_3) be the coordinates ofa, b, c, x in the standard frame of A^3, it is tempting to solve the system of equations

ëa1 b1 c1ö ëλ0ö ëx1ö

í a_2 b_2 c_2ø íλ_1ø = íx_2ø.

a_3 b_3 c_3 λ_2 x_3

However, there is a problem when the origin of the coordinate system belongs to the plane (a, b, c), since in this case, the matrix is not invertible! What we should really be doing is to solve the system

λ_0Oa +λ_1−→ +λ_2−→ =−→Ox,

whereO is any point not in the plane (a, b, c). An alternative is to use certain well-chosen cross products.

It can be shown that barycentric coordinates correspond to various ratios of areas and volumes; see the problems.

19.7 Affine Maps

Corresponding to linear maps we have the notion of an affine map. An affine map is defined as a map preserving affine combinations.

Definition 19.6. Given two affine spaces E, E,+ and E , E ,+ , a functionf :E→E is an affine map iff for every family ((ai, λi))i∈I of weighted points inE such thati∈Iλi = 1, we have

f λiai = λif(ai).

i∈I i∈I

In other words,f preserves barycenters.

Affine maps can be obtained from linear maps as follows. For simplicity of notation, the same symbol + is used for both affine spaces (instead of using both + and + ).

Given any pointa∈E, any pointb∈E , and any linear maph:E→E , we claim that the mapf :E→E defined such that

f(a +v) =b +h(v)

is an affine map. Indeed, for any family (λi)i∈I of scalars withi∈Iλi = 1 and any family (vi)i∈I, since

λi(a +vi) =a + λia(a +vi−−−−−→) =a + λivi

i∈I i∈I i∈I

and

λ

i

(

b

+

h

(

v

i

)) =

b

+

λ

i

b

(

b

+

−−−−−−−→

h(vi)) =b + λih(vi),

i∈I i∈I i∈I

we have

f λi(a +vi) = f a + λivi

i∈I i∈I

= b +h λivi

i∈I

= b + λih(vi)

i∈I

= λi(b +h(vi))

i∈I

= λif(a +vi).

i∈I

Note that the conditioni∈Iλi = 1 was implicitly used (in a hidden call to Lemma 19.1) in deriving that

λi(a +vi) =a + λivi

i∈I i∈I

and

λi(b +h(vi)) =b + λih(vi).

i∈I i∈I

As a more concrete example, the map

x_1 1 2 x1 + 3 x_2 → 0 1 x_2 1

d c

d c

a b ab

Figure 19.10: The effect of a shear

defines an affine map in A^2. It is a “shear” followed by a translation. The effect of this shear on the square (a, b, c, d) is shown in Figure 19.10. The image of the square (a, b, c, d) is the parallelogram (a , b , c , d ).

Let us consider one more example. The map

x_1 1 1 x1 + 3 x_2 → 1 3 x_2 0 is an affine map. Since we can write

1 1=√2 √2/2 −√2/2 1 2 ,1 3 2/2 √2/2 0 1

this affine map is the composition of a shear, followed by a rotation of angle π/4, followed by a magnification of ratio√2, followed by a translation. The effect of this map on the square (a, b, c, d) is shown in Figure 19.11. The image of the square (a, b, c, d) is the parallelogram (a , b , c , d ).

The following lemma shows the converse of what we just showed. Every affine map is determined by the image of any point and a linear map.

Lemma 19.7. Given an affine mapf :E→E , there is a unique linear mapf :E→E such that

f(a +v) =f(a) +f(v), for everya∈E and everyv∈E.

Proof. Leta∈E be any point inE. We claim that the map defined such that

f(v) =−−−−−−−−−→

for everyv∈E is a linear mapf :E→E . Indeed, we can write a +λv =λ(a +v) + (1−λ)a,

c

d

d c b

a ba

Figure 19.11: The effect of an affine map

−−−−−→

sincea +λv =a +λa(a +v) + (1−λ)aa, and also

−→

a +u +v = (a +u) + (a +v)−a,

since a +u +v =a +−−−−−→) +−−−−−→−−→aa. Sincef preserves barycenters, we get f(a +λv) =λf(a +v) + (1−λ)f(a).

If we recall thatx =i∈Iλiai is the barycenter of a family ((ai, λi))i∈I of weighted points (withi∈Iλi = 1) iff

bx = λi−→ for everyb∈E,

i∈I

we get

−−−−−→ −−−−−−−−−→

f(a)f(a +λv) =λf(

−−−−−−−−−−→ −−−−−−−−−→a)f(a +v) + (1−λ)f(a)f(a) =λf(a)f(a +v),

showing that f(λv) =λf(v). We also have

f(a +u +v) =f(a +u) +f(a +v)−f(a),

from which we get

−−−−−−−−−−−−→) =−−−−−−−−−→) +−−−−−−−−−→, showing thatf(u +v) =f(u) +f(v). Consequently,f is a linear map. For any other point b∈E, since

b

+

v

=

a

+

−→

+

v

=

a

+

a

(

a

+ −−−−−→ v)−−→aa +−→ab,

b +v = (a +v)−a +b, and sincef preserves barycenters, we get f(b +v) =f(a +v)−f(a) +f(b), which implies that

−−−−−−−−→ = −−−−−−−−→−−−−−−→) +−−−−−→, = −−−−−→) +−−−−−−−−→,

=

−−−−−−−−−→

f(a)f(a +v).

Thus,

−−−−−−−−→(b)f(b +v) = f(a)f(a−−−−−−−−−→

f +v), which shows that the definition off does not depend on the choice ofa∈ E. The fact thatf is unique is obvious: We must havef(v) =

−−−−−−−−−→

f(a)f(a +v).

The unique linear map f :E→E given by Lemma 19.7 is called the linear map associated with the affine mapf.

Note that the condition

f(a +v) =f(a) +f(v),

for everya∈E and everyv∈E, can be stated equivalently as

f

(

x

) =

f

(

a

) +

f

(

ax

)

,

or

f

(

a

)

f

(

−−−−−→

x) =f(ax),

−→ −→

for alla, x∈E. Lemma 19.7 shows that for any affine mapf :E→E , there are points a∈E,b∈E , and a unique linear mapf :E→E , such that

f(a +v) =b +f(v),

for allv∈E (just letb =f(a), for anya∈E). Affine maps for whichf is the identity map are called translations. Indeed, iff = id,

f(x) = f(a) +f(ax) =f(a) +ax =x +xa + af(a−−−→) +ax −→ −→ −→ −→ = x +xa +−−−→a)−−→xa =x +−−−→a),−→af( af(

and so

xf

(

−−−→−−−→

x) = af(a), which shows that

f

is the translation induced by the vector

−−−→

af(a) (which does not depend ona).

Since an affine map preserves barycenters, and since an affine subspace V is closed under barycentric combinations, the imagef(V ) ofV is an affine subspace inE . So, for example, the image of a line is a point or a line, and the image of a plane is either a point, a line, or a plane.

It is easily verified that the composition of two affine maps is an affine map. Also, given affine mapsf :E→E andg:E→E , we have

g (f(a +v)) =g f(a) +f(v) =g(f(a)) +g f(v) , which shows thatgæf =gæf. It is easy to show that an affine mapf :E→E is injective ifff :E→E is injective, and thatf :E→E is surjective ifff :E→E is surjective. An affine mapf :E→E is constant ifff :E→E is the null (constant) linear map equal to 0 for allv∈E.

If E is an affine space of dimensionm and (a_0, a_1, . . . , am) is an affine frame forE, then for any other affine spaceF and for any sequence (b_0, b_1, . . . , bm) ofm+ 1 points inF, there is a unique affine mapf :E→F such thatf(ai) =bi, for 0≤i≤m. Indeed,f must be such that

f(λ_0a_0 +· · · +λmam) =λ_0b_0 +· · · +λmbm,

whereλ_0+· · ·+λm = 1, and this defines a unique affine map on all ofE, since (a_0, a_1, . . . , am) is an affine frame forE.

Using affine frames, affine maps can be represented in terms of matrices. We explain how an affine mapf :E→E is represented with respect to a frame (a_0, . . . , an) inE, the more general case where an affine mapf :E→F is represented with respect to two affine frames (a_0, . . . , an) inE and (b_0, . . . , bm) inF being analogous. Since

f(a_0 +x) =f(a_0) +f(x)

for allx∈E, we have 0 0−−−−−−−−→) =0−−−−→0) +f(x). Sincex,0−−−−→0), and_0−−−−−−−−→0 +x), can be expressed as

x = x_1aa +· · · +xnaa ,−−→ −−→ a

0

f

(

a

−−−−→ = ba

1−−→a_1 +· · · +bnaa ,0) −−→ a

0

0

−−−−−−−−→

=

y

a

1−−→a_1 +· · · +ynaa ,−−→

ifA = (ai j) is then×n matrix of the linear mapf over the basis (aa , . . . ,aa ), lettingx,−−→ −−→ y, andb denote the column vectors of components (x_1, . . . , xn), (y_1, . . . , yn), and (b_1, . . . , bn),

a

0

f

(

a

0

+

−−−−−−−−→) = a_0f(−−−−→

x a0) +f(x)

is equivalent to y =Ax +b.

Note that b = 0 unlessf(a_0) =a_0. Thus,f is generally not a linear transformation, unless it has a fixed point, i.e., there is a pointa_0 such thatf(a_0) =a_0. The vectorb is the “translation part” of the affine map. Affine maps do not always have a fixed point. Obviously, nonnull translations have no fixed point. A less trivial example is given by the affine map

x_1 1 0 x1 + 1 .x2 → 0 −1 x2 0 This map is a reflection about thex-axis followed by a translation along thex-axis. The affine map

x1 1 −√3 x1 + 1

x2 → √3/4 1/4 x2 1

can also be written as

x_1 2 0 1/2 −√3/2 x1 + 1 x_2 → 0 1/2 √3/2 1/2 x_2 1

which shows that it is the composition of a rotation of angle π/3, followed by a stretch (by a factor of 2 along thex-axis, and by a factor of 1 along they-axis), followed by a translation.2

It is easy to show that this affine map has a unique fixed point. On the other hand, the affine map

x_1 8/5 −6/5 x1 + 1 x_2 → 3/10_2/5 x_2 1 has no fixed point, even though

8/5 −6/5= 2 0 4/5 −3/5 3/10_2/5 0 1/2 3/5_4/5 ,

and the second matrix is a rotation of angleθ such that cosθ = 4 and sinθ = 3. For more5 5 on fixed points of affine maps, see the problems.

There is a useful trick to convert the equation y =Ax +b into what looks like a linear equation. The trick is to consider an (n + 1)× (n + 1) matrix. We add 1 as the (n + 1)th component to the vectorsx,y, andb, and form the (n + 1)× (n + 1) matrix

A b 0 1

so thaty =Ax +b is equivalent to

y= A b x .1 0 1 1

This trick is very useful in kinematics and dynamics, where A is a rotation matrix. Such affine maps are called rigid motions.

Iff :E→ E is a bijective affine map, given any three collinear pointsa, b, c inE, witha =b, where, say,c = (1−λ)a +λb, sincef preserves barycenters, we havef(c) = (1−λ)f(a) +λf(b), which shows thatf(a), f(b), f(c) are collinear inE . There is a converse to this property, which is simpler to state when the ground field isK = R. The converse states that given any bijective functionf :E→E between two real affine spaces of the same dimensionn≥ 2, iff maps any three collinear points to collinear points, thenf is affine. The proof is rather long (see Berger [6] or Samuel [87]).

19.8. AFFINE GROUPS 507

Given three collinear pointsa, b, c, wherea =c, we haveb = (1−β)a +βc for some uniqueβ, and we define the ratio of the sequencea, b,c, as

ratio(

a, b, c

) =

β

(1

−

β

) =

−→

bc,

−→

provided that β = 1, i.e.,b =c. Whenb =c, we agree that ratio(a, b, c) =∞. We warn our readers that other authors define the ratio ofa, b, c as−ratio(a, b, c) =−→. Since affine maps−→

preserve barycenters, it is clear that affine maps preserve the ratio of three points.

19.8 Affine Groups

We now take a quick look at the bijective affine maps. Given an affine space E, the set of affine bijectionsf :E→E is clearly a group, called the affine group ofE, and denoted by GA(E). Recall that the group of bijective linear maps of the vector spaceE is denoted by GL(E). Then, the mapf→f defines a group homomorphismL: GA(E)→ GL(E). The kernel of this map is the set of translations onE.

The subset of all linear maps of the form λidE , whereλ∈ R− {0}, is a subgroup of GL(E), and is denoted by R∗idE (whereλid−→ (u) =λu, and R∗ = R− {0}). The

subgroup

DIL

(

E

) =

L

−

1

(

R

∗

id

E

) of

−→

GA

−→

(E) is particularly interesting. It turns out that it −→is the disjoint union of the translations and of the dilatations of ratioλ = 1. The elements of DIL(E) are called affine dilatations.

Given any pointa∈E, and any scalarλ∈ R, a dilatation or central dilatation (or homothety) of centera and ratioλ is a mapHa,λ defined such that

Ha,λ(x) =a +λax,

−→

for everyx∈E.

Remark: The terminology does not seem to be universally agreed upon. The terms affine dilatation and central dilatation are used by Pedoe [85]. Snapper and Troyer use the term dilation for an affine dilatation and magnification for a central dilatation [95]. Samuel uses homothety for a central dilatation, a direct translation of the French “homoth´etie” [87]. Since dilation is shorter than dilatation and somewhat easier to pronounce, perhaps we should use that!

Observe that Ha,λ(a) =a, and whenλ = 0 andx =a,Ha,λ(x) is on the line defined by a andx, and is obtained by “scaling”ax byλ.−→

Figure 19.12 shows the effect of a central dilatation of centerd. The triangle (a, b, c) is magnified to the triangle (a , b , c ). Note how every line is mapped to a parallel line.

Whenλ = 1,Ha,1 is the identity. Note thatHa,λ =λidE . Whenλ = 0, it is clear that Ha,λ is an affine bijection. It is immediately verified that−→

Ha,λæHa,µ =Ha,λµ. a

a

d b b

c

c

Figure 19.12: The effect of a central dilatation

We have the following useful result.

Lemma 19.8. Given any affine spaceE, for any affine bijectionf∈ GA(E), iff =λidE , for someλ∈ R∗ withλ = 1, then there is a unique pointc∈E such thatf =Hc,λ.−→ Proof. The proof is straightforward, and is omitted. It is also given in Gallier [41].

Clearly, if f = id−→ , the affine mapf is a translation. Thus, the group of affine dilatations DIL(E) is the disjoint union of the translations and of the dilatations of ratioλ = 0,1. Affine dilatations can be given a purely geometric characterization.

Another point worth mentioning is that affine bijections preserve the ratio of volumes of parallelotopes. Indeed, given any basisB = (u_1, . . . , um) of the vector spaceE associated with the affine spaceE, given anym + 1 affinely independent points (a_0, . . . , am), we can compute the determinant detB(aa , . . . ,aa ) w.r.t. the basisB. For any bijective affine mapf :E→E, since−−→ −−→

detB f(aa ), . . . , f(aa ) = detf detB(aa , . . . ,aa )−−→ −−→ −−→ −−→

and the determinant of a linear map is intrinsic (i.e., depends only onf, and not on the particular basisB), we conclude that the ratio

detB f(aa ), . . . , f(aa )−−→ −−→= detfdetB(a a , . . . ,a a )−−→ −−→

is independent of the basis B. Since detB(aa , . . . ,aa ) is the volume of the parallelotope−−→ −−→

spanned by (a_0, . . . , am), where the parallelotope spanned by any pointa and the vectors (u_1, . . . , um) has unit volume (see Berger [6], Section 9.12), we see that affine bijections preserve the ratio of volumes of parallelotopes. In fact, this ratio is independent of the choice of the parallelotopes of unit volume. In particular, the affine bijectionsf∈ GA(E) such that detf = 1 preserve volumes. These affine maps form a subgroup SA(E) of GA(E) called the special affine group ofE. We now take a glimpse at affine geometry.

19.9 Affine Geometry: A Glimpse

In this section we state and prove three fundamental results of affine geometry. Roughly speaking, affine geometry is the study of properties invariant under affine bijections. We now prove one of the oldest and most basic results of affine geometry, the theorem of Thales.

Lemma 19.9. Given any affine spaceE, ifH_1, H_2, H_3 are any three distinct parallel hyperplanes, andA andB are any two lines not parallel toHi, lettingai =Hi∩A andbi =Hi∩B, then the following ratios are equal:

a a −−→3 =ρ.a a =1−−→

−−→1−−→2

a d =ρ, thend =a_3.a aConversely, for any pointd on the lineA, if −→

−−→

Proof. Figure 19.13 illustrates the theorem of Thales. We sketch a proof, leaving the details as an exercise. SinceH_1, H_2,H_3 are parallel, they have the same directionH, a hyperplane inE. Letu∈E−H be any nonnull vector such thatA =a_1+Ru. SinceA is not parallel to H, we haveE =H⊕Ru, and thus we can define the linear mapp:E→ Ru, the projection on Ru parallel toH. This linear map induces an affine mapf :E→A, by definingf such that

f(b_1 +w) =a_1 +p(w),

for allw∈ E. Clearly,f(b_1) =a_1, and sinceH_1, H_2, H_3 all have directionH, we also have f(b_2) =a_2 andf(b_3) =a_3. Sincef is affine, it preserves ratios, and thus

a

a −−→.a

−−→

−−→1

a =−−→2

The converse is immediate. We also have the following simple lemma, whose proof is left as an easy exercise.

Lemma 19.10. Given any affine spaceE, given any two distinct pointsa, b∈E, and for any affine dilatationf different from the identity, ifa =f(a),D = a, b is the line passing througha andb, andD is the line parallel toD and passing througha , the following are equivalent:

a_1 b_1

H_1

H_2

a_2 b_2

a_3 b_3

H_3

AB

Figure 19.13: The theorem of Thales

c D b a

c b a D Figure 19.14: Pappus’s theorem (affine version)

(i) b =f(b); (ii) Iff is a translation, thenb is the intersection ofD with the line parallel to a, a passing throughb;

Iff is a dilatation of centerc, thenb =D∩ c, b .

The first case is the parallelogram law, and the second case follows easily from Thales’ theorem.

We are now ready to prove two classical results of affine geometry, Pappus’s theorem and Desargues’s theorem. Actually, these results are theorems of projective geometry, and we are stating affine versions of these important results. There are stronger versions that are best proved using projective geometry.

Lemma 19.11. Given any affine planeE, any two distinct linesD andD , then for any distinct pointsa, b, c onD anda , b , c onD , ifa, b, c, a ,b ,c are distinct from the intersection ofD andD (ifD andD intersect) and if the lines a, b and a , b are parallel, and the lines b, c and b , c are parallel, then the lines a, c and a , c are parallel.

Proof. Pappus’s theorem is illustrated in Figure 19.14. IfD andD are not parallel, letd be their intersection. Letf be the dilatation of centerd such thatf(a) =b, and letg be the dilatation of centerd such thatg(b) =c. Since the lines a, b and a , b are parallel, and the lines b, c and b , c are parallel, by Lemma 19.10 we havea =f(b ) andb =g(c ). However, we observed that dilatations with the same center commute, and thusfæg =gæf, and thus, lettingh =gæf, we getc =h(a) anda =h(c ). Again, by Lemma 19.10, the lines a, c and a , c are parallel. IfD andD are parallel, we use translations instead of dilatations.

There is a converse to Pappus’s theorem, which yields a fancier version of Pappus’s theorem, but it is easier to prove it using projective geometry. It should be noted that in axiomatic presentations of projective geometry, Pappus’s theorem is equivalent to the commutativity of the ground fieldK (in the present case,K = R). We now prove an affine version of Desargues’s theorem.

Lemma 19.12. Given any affine spaceE, and given any two triangles (a, b, c) and (a , b , c ), wherea, b, c, a , b , c are all distinct, if a, b and a , b are parallel and b, c and b , c are parallel, then a, c and a , c are parallel iff the lines a, a , b, b , and c, c are either parallel or concurrent (i.e., intersect in a common point).

Proof. We prove half of the lemma, the direction in which it is assumed that a, c and a , c are parallel, leaving the converse as an exercise. Since the lines a, b and a , b are parallel, the pointsa, b, a , b are coplanar. Thus, either a, a and b, b are parallel, or they have some intersectiond. We consider the second case where they intersect, leaving the other case as an easy exercise. Letf be the dilatation of centerd such thatf(a) =a . By Lemma 19.10, we getf(b) =b . Iff(c) =c , again by Lemma 19.10 twice, the lines b, c and b , c are parallel, and the lines a, c and a , c are parallel. From this it follows thatc =c . Indeed, recall that b, c and b , c are parallel, and similarly a, c and a , c are parallel. Thus, the lines b , c and b , c are identical, and similarly the lines a , c and a , c are identical. Since−→ and−→ are linearly independent, these lines have a unique intersection, which must bec =c .

The direction where it is assumed that the lines a, a , b, b and c, c , are either parallel or concurrent is left as an exercise (in fact, the proof is quite similar).

Desargues’s theorem is illustrated in Figure 19.15.

There is a fancier version of Desargues’s theorem, but it is easier to prove it using projective geometry. It should be noted that in axiomatic presentations of projective geometry, Desargues’s theorem is related to the associativity of the ground fieldK (in the present case,K = R). Also, Desargues’s theorem yields a geometric characterization of the affine dilatations. An affine dilatationf on an affine spaceE is a bijection that maps every line D to a linef(D) parallel toD. We leave the proof as an exercise.

19.10 Affine Hyperplanes

We now consider affine forms and affine hyperplanes. In Section 19.5 we observed that the setL of solutions of an equation

ax +by =c 19.10. AFFINE HYPERPLANES 513

a

a

d b b

c

c

Figure 19.15: Desargues’s theorem (affine version)

is an affine subspace of A^2 of dimension 1, in fact, a line (provided thata andb are not both null). It would be equally easy to show that the setP of solutions of an equation

ax +by +cz =d

is an affine subspace of A^3 of dimension 2, in fact, a plane (provided thata, b, c are not all null). More generally, the setH of solutions of an equation

λ_1x_1 +· · · +λmxm =µ

is an affine subspace ofAm, and ifλ_1, . . . , λm are not all null, it turns out that it is a subspace of dimensionm− 1 called a hyperplane.

We can interpret the equation

λ_1x_1 +· · · +λmxm =µ

in terms of the mapf : Rm R defined such that→

f(x_1, . . . , xm) =λ_1x_1 +· · · +λmxm−µ

for all (x_1, . . . , xm)∈ Rm. It is immediately verified that this map is affine, and the setH of solutions of the equation

λ_1x_1 +· · · +λmxm =µ

is the null set, or kernel, of the affine mapf : Am R, in the sense that→

H =f−1(0) ={x∈ Am f(x) = 0},|

wherex = (x_1, . . . , xm).

Thus, it is interesting to consider affine forms, which are just affine mapsf :E→ R from an affine space to R. Unlike linear formsf∗, for which Kerf∗ is never empty (since it always contains the vector 0), it is possible thatf−1(0) =∅ for an affine formf. Given an affine mapf :E→ R, we also denotef−1(0) by Kerf, and we call it the kernel off. Recall that an (affine) hyperplane is an affine subspace of codimension 1. The relationship between affine hyperplanes and affine forms is given by the following lemma.

Lemma 19.13. LetE be an affine space. The following properties hold: (a) Given any nonconstant affine formf :E→ R, its kernelH = Kerf is a hyperplane.

(b) For any hyperplane H inE, there is a nonconstant affine formf :E→ R such that H = Kerf. For any other affine formg:E→ R such thatH = Kerg, there is some λ∈ R such thatg =λf (withλ = 0).

(c) Given any hyperplane H inE and any (nonconstant) affine formf :E→ R such that H = Kerf, every hyperplaneH parallel toH is defined by a nonconstant affine form g such thatg(a) =f(a)−λ, for alla∈E and someλ∈ R.

Proof. The proof is straightforward, and is omitted. It is also given in Gallier [41].

When E is of dimension n, given an affine frame (a_0,(u_1, . . . , un)) of E with origin a_0, recall from Definition 19.5 that every point ofE can be expressed uniquely asx = a_0 +x_1u_1 +· · · +xnun, where (x_1, . . . , xn) are the coordinates ofx with respect to the affine frame (a_0,(u1, . . . , un)).

Also recall that every linear form f∗ is such thatf∗(x) =λ_1x_1 +· · · +λnxn, for every x =x_1u_1 +· · · +xnun and someλ_1, . . . , λn∈ R. Since an affine formf :E→ R satisfies the propertyf(a0 +x) =f(a_0) +f(x), denotingf(a_0 +x) byf(x_1, . . . , xn), we see that we have

f(x_1, . . . , xn) =λ_1x_1 +· · · +λnxn +µ,

whereµ =f(a_0)∈ R andλ_1, . . . , λn∈ R. Thus, a hyperplane is the set of points whose coordinates (x_1, . . . , xn) satisfy the (affine) equation

λ_1x_1 +· · · +λnxn +µ = 0.

19.11 Intersection of Affine Spaces

In this section we take a closer look at the intersection of affine subspaces. This subsection can be omitted at first reading.

First, we need a result of linear algebra. Given a vector spaceE and any two subspacesM andN, there are several interesting linear maps. We have the canonical injectionsi:M→M+N andj:N→M+N, the canonical injectionsin1:M→M⊕N andin2:N→M⊕N, 19.11. INTERSECTION OF AFFINE SPACES 515

and thus, injections f :M∩N→ M⊕N andg:M∩N→ M⊕N, wheref is the composition of the inclusion map fromM∩N toM within_1, andg is the composition of the inclusion map fromM∩N toN within_2. Then, we have the mapsf +g:M∩N→M⊕N, and i−j:M⊕N→M +N.

Lemma 19.14. Given a vector spaceE and any two subspacesM andN, with the definitions above,

0−→M∩N f+gM⊕N i−j M +N−→ 0−→ −→

is a short exact sequence, which means thatf +g is injective,i−j is surjective, and that Im (f +g) = Ker (i−j). As a consequence, we have the Grassmann relation

dim(M) + dim(N) = dim(M +N) + dim (M∩N).

Proof. It is obvious thati−j is surjective and thatf +g is injective. Assume that (i− j)(u+v) = 0, whereu∈M, andv∈N. Then,i(u) =j(v), and thus, by definition ofi and j, there is somew∈M∩N, such thati(u) =j(v) =w∈M∩N. By definition off and g,u =f(w) andv =g(w), and thus Im (f +g) = Ker (i−j), as desired. The second part of the lemma follows from standard results of linear algebra (see Artin [3], Strang [101], or Lang [65]).

We now prove a simple lemma about the intersection of affine subspaces. Lemma 19.15. Given any affine spaceE, for any two nonempty affine subspacesM and N, the following facts hold:

(1) M∩N =∅ iff−→∈M +N for somea∈M and someb∈N.

(2) M∩ N consists of a single point iff−→∈M +N for somea∈M and someb∈N, and M∩N ={0}.

(3) IfS is the least affine subspace containingM andN, thenS =M +N +K−→ (the vector spaceE is defined over the fieldK).

Proof. (1) Pick anya∈M and anyb∈N, which is possible, sinceM andN are nonempty. SinceM =ax x∈ M} andN ={−→|y∈N}, ifM∩N =∅, for anyc∈M∩N we have{−→|

−→ =ac−−→, withac M and−→bc∈ N, and thus,−→∈M +N. Conversely, assume that−→ −→

−→∈ M and someb∈N. Then−→ =ax +−→, for somex∈M and∈M +N for somea∈ −→somey∈N. But we also have

−→ =ax +xy +−→yb,−→ −→

and thus we get 0 =xy +−→−−→, that is,xy = 2−→. Thus,b is the middle of the segment−→ −→

[x, y], and sinceyx = 2−→,x = 2b−y is the barycenter of the weighted points (b,2) and−→

(y,−1). Thusx also belongs toN, sinceN being an affine subspace, it is closed under barycenters. Thus,x∈M∩N, andM∩N =∅.

(2) Note that in general, ifM∩N =∅, then

M∩N =M∩N,

because

M∩N =ab a, b∈M∩N} =ab a, b∈M} ∩ {−→ab|a, b∈N} =M∩N.{−→| {−→|

SinceM∩N =c +M∩N for anyc∈M∩N, we have

M∩N =c +M∩N for anyc∈M∩N.

From this it follows that ifM∩N =∅, thenM∩N consists of a single point iffM∩N ={0}. This fact together with what we proved in (1) proves (2).

(3) This is left as an easy exercise.

Remarks:

(1) The proof of Lemma 19.15 shows that ifM∩N =∅, then−→∈M +N for alla∈M and allb∈N.

(2) Lemma 19.15 implies that for any two nonempty affine subspaces M andN, ifE = M⊕N, thenM∩N consists of a single point. Indeed, ifE =M⊕N, then−→∈E for alla∈M and allb∈N, and sinceM∩N ={0}, the result follows from part (2) of the lemma.

We can now state the following lemma. Lemma 19.16. Given an affine spaceE and any two nonempty affine subspacesM andN, ifS is the least affine subspace containingM andN, then the following properties hold:

(1) IfM∩N =∅, then

dim(M) + dim(N)< dim(E) + dim(M +N)

and dim(S) = dim(M) + dim(N) + 1− dim(M∩N).

(2) IfM∩N =∅, then

dim(S) = dim(M) + dim(N)− dim(M∩N).

Proof. The proof is not difficult, using Lemma 19.15 and Lemma 19.14, but we leave it as an exercise.

19.12 Problems

Problem 19.1. Given a triangle (a, b, c), give a geometric construction of the barycenter of the weighted points (a, 1), (b, 1), and (c, 1). Give a geometric construction of the barycenter4 3 4 3 2

of the weighted points (a,2), (b,2), and (c,−2).

Problem 19.2. Given a tetrahedron (a, b, c, d) and any two distinct pointsx, y∈ {a, b, c, d}, let letmx,y be the middle of the edge (x, y). Prove that the barycenterg of the weighted points (a, 1), (b, 1), (c, 1), and (d, 1) is the common intersection of the line segments (ma,b, mc,d),

(

m

4

4 4 4

, mb,d), and (ma,d, mb,c). Show that if gd is the barycenter of the weighted pointsa,c

(a, 1),(b, 1),(c, 1), theng is the barycenter of (d, 1) and (gd, 3).3 3 3 4 4

Problem 19.3. LetE be a nonempty set, andE a vector space and assume that there is a function Φ:E×E→E, such that if we denote Φ(a, b) by−→, the following properties hold:

(1)−→ +−→ =ac, for alla, b, c∈E;−→

(2) For everya∈E, the map Φa:E→E defined such that for everyb∈E, Φa(b) =ab,−→ is a bijection.

Let Ψa:E→E be the inverse of Φa:E→E.

Prove that the function +:E×E→E defined such that

a +u = Ψa(u)

for all a∈E and allu∈E makes (E, E,+) into an affine space.

Note. We showed in the text that an affine space (E, E,+) satisfies the properties stated above. Thus, we obtain an equivalent characterization of affine spaces.

Problem 19.4. Given any three pointsa,b,c in the affine plane A^2, letting (a_1, a_2), (b_1, b_2), and (c_1, c_2) be the coordinates ofa, b, c, with respect to the standard affine frame for A^2, prove thata, b, c are collinear iff

a_1 b_1 c_1

a_2 b_2 c_2 = 0,

1 1 1

i.e., the determinant is null.

Letting (a_0, a_1, a_2), (b_0, b_1, b_2), and (c_0, c_1, c_2) be the barycentric coordinates ofa, b, c with respect to the standard affine frame for A^2, prove thata, b, c are collinear iff

a_0 b_0 c_0

a_1 b_1 c_1 = 0. a_2 b_2 c_2

Given any four pointsa, b, c, d in the affine spaceA^3, letting (a_1, a_2, a_3), (b_1, b_2, b_3), (c_1, c_2, c_3), and (d_1, d_2, d_3) be the coordinates ofa, b, c, d, with respect to the standard affine frame for A^3, prove thata, b, c, d are coplanar iff

a_1 b_1 c_1 d_1

a_2 b_2 c_2 d2 = 0,a3 b3 c3 d3

1 1 1 1

i.e., the determinant is null.

Letting ( a_0, a_1, a_2, a_3), (b_0, b_1, b_2, b_3), (c_0, c_1, c_2, c_3), and (d_0, d_1, d_2, d_3) be the barycentric coordinates ofa, b, c, d, with respect to the standard affine frame for A^3, prove thata, b, c, d are coplanar iff

a_0 b_0 c_0 d_0

a1 b1 c1 d1 = 0.a_2 b_2 c_2 d_2

a_3 b_3 c_3 d_3

Problem 19.5. The functionf : A→ A^3 given by

t→ (t, t^2, t^3)

defines what is called a twisted cubic curve. Given any four pairwise distinct valuest_1, t_2, t_3, t_4, prove that the pointsf(t_1), f(t_2), f(t_3), andf(t_4) are not coplanar.

Hint. Have you heard of the Vandermonde determinant?

Problem 19.6. For any two distinct pointsa, b∈ A^2 of barycentric coordinates (a_0, a_1, a_2) and (b_0, b_1, b_2) with respect to any given affine frame (O, i, j), show that the equation of the line a, b determined bya andb is

a_0 b_0 x

a_1 b_1 y = 0, a_2 b_2 z

or, equivalently,

(a_1b2−a_2b_1)x + (a_2b0−a_0b_2)y + (a_0b1−a_1b_0)z = 0, where (x, y, z) are the barycentric coordinates of the generic point on the line a, b . Prove that the equation of a line in barycentric coordinates is of the form ux +vy +wz = 0,

whereu =v orv =w oru =w. Show that two equations ux +vy +wz = 0 and u x +v y +w z = 0 represent the same line in barycentric coordinates iff (u , v , w ) =λ(u, v, w) for someλ∈ R (withλ = 0).

A triple (u, v, w) whereu = v orv = w oru = w is called a system of tangential coordinates of the line defined by the equation

ux +vy +wz = 0.

Problem 19.7. Given two linesD andD in A^2 defined by tangential coordinates (u, v, w) and (u , v , w ) (as defined in Problem 19.6), let

u v w

d = u v w =vw−wv +wu−uw +uv−vu .

1 1 1

(a) Prove thatD andD have a unique intersection point iffd = 0, and that when it exists, the barycentric coordinates of this intersection point are

1(vw−wv , wu−uw , uv−vu ).d

(b) Letting (O, i, j) be any affine frame for A^2, recall that whenx +y +z = 0, for any pointa, the vector

xaO +yai +zaj −→ −→ −→ is independent ofa and equal to

yOi +z−→ = (y, z).

The triple (x, y, z) such thatx+y +z = 0 is called the barycentric coordinates of the vector yOi +z−→ w.r.t. the affine frame (O, i, j).

Given any affine frame (O, i, j), prove that foru =v orv =w oru =w, the line of equation

ux +vy +wz = 0

in barycentric coordinates (x, y, z) (wherex+y +z = 1) has for direction the set of vectors of barycentric coordinates (x, y, z) such that

ux +vy +wz = 0

(wherex +y +z = 0).

Prove thatD andD are parallel iffd = 0. In this case, ifD =D , show that the common direction ofD andD is defined by the vector of barycentric coordinates

(vw−wv , wu−uw , uv−vu ).

(c) Given three lines D,D , andD , at least two of which are distinct and defined by tangential coordinates (u, v, w), (u , v , w ), and (u , v , w ), prove thatD,D , andD are parallel or have a unique intersection point iff

u v w

u v w = 0.

u v w

Problem 19.8. Let (A, B, C) be a triangle in A^2. LetM, N, P be three points respectively on the linesBC,CA, andAB, of barycentric coordinates (0, m , m ), (n,0, n ), and (p, p ,0), w.r.t. the affine frame (A, B, C).

(a) Assuming thatM =C,N =A, andP =B, i.e.,m n p = 0, show that

MB NC

−−→

−−→

−→

P A = m np −m n p.−−→ NA −−→

(b) Prove Menelaus’s theorem: The pointsM, N, P are collinear iff

m np +m n p = 0.

WhenM =C,N =A, andP =B, this is equivalent to

−−→ −−→ P A = 1.−−→ NA P B

(c) Prove Ceva’s theorem: The linesAM, BN, CP have a unique intersection point or are parallel iff

m np−m n p = 0. WhenM =C,N =A, andP =B, this is equivalent to

MB −−→ P A

MC =−1.

−−→

NAP B

−−→ −−→

Problem 19.9. This problem uses notions and results from Problems 19.6 and 19.7. In view of (a) and (b) of Problem 19.7, it is natural to extend the notion of barycentric coordinates of a point in A^2 as follows. Given any affine frame (a, b, c) in A^2, we will say that the barycentric coordinates (x, y, z) of a pointM, wherex +y +z = 1, are the normalized barycentric coordinates ofM. Then, any triple (x, y, z) such thatx+y+z = 0 is also called a system of barycentric coordinates for the point of normalized barycentric coordinates

1 (x, y, z).x +y +z

With this convention, the intersection of the two linesD andD is either a point or a vector, in both cases of barycentric coordinates

( vw−wv , wu−uw , uv−vu ). When the above is a vector, we can think of it as a point at infinity (in the direction of the line defined by that vector).

Let ( D_0, D_0), (D_1, D_1), and (D_2, D_2) be three pairs of six distinct lines, such that the four lines belonging to any union of two of the above pairs are neither parallel nor concurrent (have a common intersection point). IfD_0 andD_0 have a unique intersection point, letM be this point, and ifD_0 andD_0 are parallel, letM denote a nonnull vector defining the common direction ofD_0 andD_0. In either case, let (m, m , m ) be the barycentric coordinates ofM, as explained at the beginning of the problem. We callM the intersection ofD_0 andD_0. Similarly, defineN = (n, n , n ) as the intersection ofD_1 andD_1, andP = (p, p , p ) as the intersection ofD_2 andD_2.

Prove thatm n p

m n p = 0

m n p

iff either

(i) (D_0, D_0), (D_1, D_1), and (D_2, D_2) are pairs of parallel lines; or

(ii) the lines of some pair ( Di, Di) are parallel, each pair (Dj, Dj) (withj =i) has a unique intersection point, and these two intersection points are distinct and determine a line parallel to the lines of the pair (Di, Di); or

(iii) each pair (Di, Di) (i = 0,1,2) has a unique intersection point, and these pointsM, N, P are distinct and collinear.

Problem 19.10. Prove the following version of Desargues’s theorem. LetA, B, C,A , B , C be six distinct points of A^2. If no three of these points are collinear, then the linesAA ,BB , andCC are parallel or collinear iff the intersection pointsM, N, P (in the sense of Problem 19.7) of the pairs of lines (BC, B C ), (CA, C A ), and (AB, A B ) are collinear in the sense of Problem 19.9.

Problem 19.11. Prove the following version of Pappus’s theorem. LetD andD be distinct lines, and letA, B, C andA , B , C be distinct points respectively onD andD . If these points are all distinct from the intersection ofD andD (if it exists), then the intersection points (in the sense of Problem 19.7) of the pairs of lines (BC , CB ), (CA , AC ), and (AB , BA ) are collinear in the sense of Problem 19.9.

Problem 19.12. The purpose of this problem is to prove Pascal’s theorem for the nondegenerate conics. In the affine plane A^2, a conic is the set of points of coordinates (x, y) such that

αx^2 +βy^2 + 2γxy + 2δx + 2λy +µ = 0, whereα = 0 orβ = 0 orγ = 0. We can write the equation of the conic as ëα γ δöëxö

(x, y,1) íγ β λøíyø = 0. δ λ µ 1

If we now use barycentric coordinates ( x, y, z) (wherex +y +z = 1), we can write ëxö ë1 0 0öëxö

íyø = í0 1 0øíyø.

1 1 1 1 z

Let ëα γ δö ë1 0 0ö ëxö B = íγ β λø, C = í0 1 0ø, X = íyø. δ λ µ 1 1 1 z (a) LettingA =C BC, prove that the equation of the conic becomes

X AX = 0.

Prove thatA is symmetric, that det(A) = det(B), and thatX AX is homogeneous of degree 2. The equationX AX = 0 is called the homogeneous equation of the conic.

We say that a conic of homogeneous equation X AX = 0 is nondegenerate if det(A) = 0, and degenerate if det(A) = 0. Show that this condition does not depend on the choice of the affine frame.

(b) Given an affine frame (A, B, C), prove that any conic passing throughA, B, C has an equation of the form

ayz +bxz +cxy = 0.

Prove that a conic containing more than one point is degenerate iff it contains three distinct collinear points. In this case, the conic is the union of two lines.

(c) Prove Pascal’s theorem. Given any six distinct pointsA, B, C,A , B ,C , if no three of the above points are collinear, then a nondegenerate conic passes through these six points iff the intersection pointsM, N, P (in the sense of Problem 19.7) of the pairs of lines (BC , CB ), (CA , AC ) and (AB , BA ) are collinear in the sense of Problem 19.9.

Hint. Use the affine frame (A, B, C), and let (a, a , a ), (b, b , b ), and (c, c , c ) be the barycentric coordinates ofA , B , C respectively, and show thatM, N, P have barycentric coordinates

(bc, cb , c b), (c a, c a , c a ), (ab , a b , a b ).

Problem 19.13. The centroid of a triangle (a, b, c) is the barycenter of (a, 1), (b, 1), (c, 1).3 3 3 If an affine map takes the vertices of triangle 1 ={(0,0),(6,0),(0,9)} to the vertices of triangle 2 ={(1,1),(5,4),(3,1)}, does it also take the centroid of 1 to the centroid of 2? Justify your answer.

Problem 19.14. LetE be an affine space over R, and let (a_1, . . . , an) be anyn≥ 3 points inE. Let (λ_1, . . . , λn) be anyn scalars in R, withλ_1 +· · · +λn = 1. Show that there must be somei, 1≤i≤n, such thatλi = 1. To simplify the notation, assume thatλ_1 = 1. Show that the barycenterλ_1a_1 +· · · +λnan can be obtained by first determining the barycenterb of then−1 pointsa_2, . . . , an assigned some appropriate weights, and then the barycenter of a_1 andb assigned the weightsλ_1 andλ_2 +· · · +λn. From this, show that the barycenter of anyn≥ 3 points can be determined by repeated computations of barycenters of two points. Deduce from the above that a nonempty subsetV ofE is an affine subspace iff wheneverV contains any two pointsx, y∈V , thenV contains the entire line (1−λ)x +λy,λ∈ R.

Problem 19.15. Assume thatK is a field such that 2 = 1 + 1 = 0, and letE be an affine space overK. In the case whereλ_1 +· · · +λn = 1 andλi = 1, for 1≤i≤n andn≥ 3, show that the barycentera_1 +a_2 +· · ·+an can still be computed by repeated computations of barycenters of two points.

Finally, assume that the field K contains at least three elements (thus, there is some µ∈K such thatµ = 0 andµ = 1, but 2 = 1 + 1 = 0 is possible). Prove that the barycenter of anyn≥ 3 points can be determined by repeated computations of barycenters of two points. Prove that a nonempty subsetV ofE is an affine subspace iff wheneverV contains any two pointsx, y∈V , thenV contains the entire line (1−λ)x +λy,λ∈K. Hint. When 2 = 0,λ_1 +· · · +λn = 1 andλi = 1, for 1≤i≤n, show thatn must be odd, and that the problem reduces to computing the barycenter of three points in two steps involving two barycenters. Since there is someµ∈K such thatµ = 0 andµ = 1, note that µ−1 and (1−µ)−1 both exist, and use the fact that

1

−

µ + 1

1

−

µ

= 1. −µ

Problem 19.16. (i) Let (a, b, c) be three points in A^2, and assume that (a, b, c) are not collinear. For any pointx∈ A^2, ifx =λ_0a+λ_1b+λ_2c, where (λ_0, λ_1, λ_2) are the barycentric coordinates ofx with respect to (a, b, c), show that

λ

0

= det(

xb,−→), λ1 = det(ax,ac), λ2 = det(ab, ax

−→ −→ −→). det(ab,ac) det(−→ab,ac) det(ab,ac)−→ −→ −→

Conclude thatλ_0, λ_1, λ_2 are certain signed ratios of the areas of the triangles (a, b, c), (x, a, b), (x, a, c), and (x, b, c).

(ii) Let ( a, b, c) be three points in A^3, and assume that (a, b, c) are not collinear. For any pointx in the plane determined by (a, b, c), ifx =λ_0a+λ_1b+λ_2c, where (λ_0, λ_1, λ_2) are the barycentric coordinates ofx with respect to (a, b, c), show that

λ

0

=

−→

×−→

ax

ac,

λ

1

=

−→

×−→

ac −→×−→ax

ac, λ2 = ac.−→×−→−→×−→ab−→×−→

Given any pointO not in the plane of the triangle (a, b, c), prove that

λ1 = det(Oa,−→Ox,−→), λ2 = det(Oa,−→Ob,−→Ox), det(Oa,−→Ob,Oc) det(Oa,−→Ob,Oc) −→ −→−→ −→ and λ0 = det(Ox,−→Ob,−→).

det(Oa,−→Ob,Oc)

−→ −→

(iii) Let ( a, b, c, d) be four points in A^3, and assume that (a, b, c, d) are not coplanar. For any pointx∈ A^3, ifx =λ_0a +λ_1b +λ_2c +λ_3d, where (λ_0, λ_1, λ_2, λ_3) are the barycentric coordinates ofx with respect to (a, b, c, d), show that

λ

1

= det(

ax,ac,−→), λ2 = det(ab,ax,−→), λ3 = det(ab,ac,ax

−→ −→ −→ −→ −→),

det(ab,ac,−→) det(ab, ac,−→) det(ab, ac,−→)

−→ −→ −→

and λ0 = det(xb,−→bc,−→).

det(ab,ac,−→)

−→

Conclude thatλ_0, λ_1, λ_2, λ_3 are certain signed ratios of the volumes of the five tetrahedra (a, b, c, d), (x, a, b, c), (x, a, b, d), (x, a, c, d), and (x, b, c, d).

(iv) Let ( a_0, . . . , am) bem+1 points inAm, and assume that they are affinely independent. For any pointx∈ Am, ifx =λ_0a_0 +· · · +λmam, where (λ_0, . . . , λm) are the barycentric coordinates ofx with respect to (a_0, . . . , am), show that

λ

i

= det(

a

a , . . . ,a a−−−→,a x,a a−−−→, . . . ,aa −→ −−→

det(aa

)−−→ −,a ai,aa , . . . ,aa )−−→, . . . ,a a−−−→ −−→ −−−→ −−→−

for everyi, 1≤i≤m, and

λ

0

=

det(

xa ,a a_2, . . . ,aa

−→ −−→ −−→) .det(aa , . . . ,aa , . . . ,aa )−−→ −−→ −−→

Conclude thatλi is the signed ratio of the volumes of the simplexes (a_0, . . .,x, . . . am) and (a_0, . . . , ai, . . . am), where 0≤i≤m.

Problem 19.17. With respect to the standard affine frame for the plane A^2, consider the three geometric transformationsf_1, f_2, f_3 defined by

x = −4x− 4 y + 3, y =√3 1y +√31 √3

4 4 x− 4 4 , 1

x

+

√

3 3, y =− 4 x− 4y +√3 x

=

−

4

4

y

√3 1 − 4 4 , 1

x, y

= 1

y

+

√

3

x =2 2 2 .

(a) Prove that these maps are affine. Can you describe geometrically what their action is (rotation, translation, scaling)?

(b) Given any polygonal lineL, define the following sequence of polygonal lines:

S_0 = L,

Sn+1 = f_1(Sn)∪f_2(Sn)∪f_3(Sn).

ConstructS_1 starting from the line segmentL = ((−1,0),(1,0)).

Can you figure out whatSn looks like in general? (You may want to write a computer program.) Do you think thatSn has a limit?

Problem 19.18. In the plane A^2, with respect to the standard affine frame, a point of coordinates (x, y) can be represented as the complex numberz =x +iy. Consider the set of geometric transformations of the form

z→az +b,

where a, b are complex numbers such thata = 0.

(a) Prove that these maps are affine. Describe what these maps do geometrically. (b) Prove that the above set of maps is a group under composition. (c) Consider the set of geometric transformations of the form

z→az +b or z→az +b,

where a, b are complex numbers such thata = 0, and wherez =x−iy ifz =x +iy. Describe what these maps do geometrically. Prove that these maps are affine and that this

set of maps is a group under composition.

Problem 19.19. Given a groupG, a subgroupH ofG is called a normal subgroup ofG iff xHx−1 =H for allx∈G (wherexHx−1 ={xhx−1 h∈H}).|

(i) Given any two subgroupsH andK of a groupG, let

HK ={hk|h∈H, k∈K}.

Prove that everyx∈HK can be written in a unique way asx =hk forh∈H andk∈K iffH∩K ={1}, where 1 is the identity element ofG.

(ii) If H andK are subgroups ofG, andH is a normal subgroup ofG, prove thatHK is a subgroup ofG. Furthermore, ifG =HK andH∩K ={1}, prove thatG is isomorphic toH×K under the multiplication operation

(h_1, k_1)· (h_2, k_2) = (h_1k_1h_2k− 1 , k_1k_2).1

When G =HK, whereH, K are subgroups ofG,H is a normal subgroup ofG, and H∩ K ={1}, we say thatG is the semidirect product ofH andK.

(iii) Let (E, E) be an affine space. Recall that the affine group ofE, denoted by GA(E), is the set of affine bijections ofE, and that the linear group ofE, denoted by GL(E), is the group of bijective linear maps ofE. The mapf→f defines a group homomorphism L: GA(E)→ GL(E), and the kernel of this map is the set of translations onE, denoted as T(E). Prove thatT(E) is a normal subgroup of GA(E).

(iv) For anya∈E, let

GAa(E) ={f∈ GA(E)|f(a) =a},

the set of affine bijections leaving a fixed. Prove that that GAa(E) is a subgroup of GA(E), and that GAa(E) is isomorphic to GL(E). Prove that GA(E) is isomorphic to the direct product ofT(E) and GAa(E).

Hint. Note that ifu =−−−→ andtu is the translation associated with the vectoru, then tuæf∈ GAa(E) (where the translationtu is defined such thattu(a) =a+u for everya∈E).

(v) Given a group G, let Aut(G) denote the set of homomorphismsf :G→G. Prove that the set Aut(G) is a group under composition (called the group of automorphisms ofG). Given any two groupsH andK and a homomorphismθ:K→ Aut(H), we defineH×θK as the setH×K under the multiplication operation

(h_1, k_1)· (h_2, k_2) = (h_1θ(k_1)(h_2), k_1k_2).

Prove thatH×θK is a group.

Hint. The inverse of (h, k) is (θ(k−1)(h−1), k−1).

Prove that the groupH×θK is the semidirect product of the subgroups {

(h,1)|h∈H} and{(1, k)|k∈K}. The groupH×θK is also called the semidirect product of H and K relative to θ.

Note . It is natural to identify{(h,1)|h∈H} withH and{(1, k)|k∈K} withK. IfG is the semidirect product of two subgroupsH andK as defined in (ii), prove that

the mapγ:K→ Aut(H) defined by conjugation such that

γ(k)(h) =khk−1

is a homomorphism, and that G is isomorphic toH×γK.

(vi) Define the mapθ: GL(E)→ Aut(E) as follows: θ(f) =f, wheref∈ GL(E)

(note thatθ can be viewed as an inclusion map). Prove that GA(E) is isomorphic to the

semidirect productE×θ GL(E).

(vii) Let SL(E) be the subgroup of GL(E) consisting of the linear maps such that

det(f) = 1 (the special linear group ofE), and let SA(E) be the subgroup of GA(E) (the

special affine group ofE) consisting of the affine mapsf such thatf∈ SL(E). Prove that

SA(E) is isomorphic to the semidirect productE×θ SL(E), whereθ: SL(E)→ Aut(E) is

defined as in (vi).

(viii) Assume that (E, E) is a Euclidean affine space. Let SO(E) be the special orthogonal

group ofE (the isometries with determinant +1), and let SE(E) be the subgroup of SA(E)

(the special Euclidean group ofE) consisting of the affine isometriesf such thatf∈ SO(E).

Prove that SE(E) is isomorphic to the semidirect productE×θSO(E), whereθ: SO(E)→Aut(E) is defined as in (vi).

Problem 19.20. The purpose of this problem is to study certain affine maps of A^2. (1) Consider affine maps of the form

x_1 cosθ − sinθ x1 + b1 .x2 → sinθ cosθ x2 b2

Prove that such maps have a unique fixed point c ifθ = 2kπ, for all integersk. Show that these are rotations of centerc, which means that with respect to a frame with originc (the unique fixed point), these affine maps are represented by rotation matrices.

(2) Consider affine maps of the form

x_1 λcosθ −λsinθ x1 + b1 .x2 → µsinθ µcosθ x2 b2

Prove that such maps have a unique fixed point iff ( λ +µ) cosθ = 1 +λµ. Prove that if λµ = 1 andλ > 0, there is some angleθ for which either there is no fixed point, or there are infinitely many fixed points.

(3) Prove that the affine map

x_1 8/5 −6/5 x1 + 1 x_2 → 3/10_2/5 x_2 1 has no fixed point.

(4) Prove that an arbitrary affine map x_1 a_1 a_2 x1 + b_1 x_2 → a_3 a_4 x_2 b_2 has a unique fixed point iff the matrix a1− 1 a2 a3 a4− 1 is invertible.

Problem 19.21. Let (E, E) be any affine space of finite dimension. For every affine map f :E→E, let Fix(f) ={a∈E|f(a) =a} be the set of fixed points off. (i) Prove that if Fix(f) =∅, then Fix(f) is an affine subspace ofE such that for every b∈ Fix(f),

Fix(f) =b + Ker (f− id).

(ii) Prove that Fix( f) contains a unique fixed point iff Ker (f− id) ={0}, i.e.,f(u) =u iffu = 0.

Hint. Show that

&f(a) for any two points &, a∈E.

−−−→ −−−−→f(&) +f(−→)−−→a,−−→ =&

Problem 19.22. Given two affine spaces (E, E) and (F, F), letA(E, F) be the set of all affine mapsf :E→F.

(i) Prove that the setA(E, F) (viewingF as an affine space) is a vector space under the operationsf +g andλf defined such that

(f +g)(a) = f(a) +g(a), (λf)(a) = λf(a),

for alla∈E.

(ii) Define an action

+:A(E, F)× A(E, F)→ A(E, F)

ofA(E, F) onA(E, F) as follows: For everya∈E, everyf∈ A(E, F), and everyh∈ A

(E, F),

(f +h)(a) =f(a) +h(a).

Prove that (A(E, F),A(E, F),+) is an affine space.

Hint. Show that for any two affine mapsf, g∈ A(E, F), the map−→ defined such that

−−−−−→

fg(a) = f(a)g(a)

(for everya∈E) is affine, and thus−→∈ A(E, F). Furthermore,−→ is the unique map in A

(E, F) such that

f +−→ =g.

(iii) IfE has dimensionm andF has dimensionn, prove thatA(E, F) has dimension n +mn =n(m + 1).

Problem 19.23. Let (c_1, . . . , cn) ben≥ 3 points in Am (wherem≥ 2). Investigate whether there is a closed polygon withn vertices (a_1, . . . , an) such thatci is the middle of the edge (ai, ai+1) for everyi with 1≤i≤n− 1, andcn is the middle of the edge (an, a_0). Hint. The parity (odd or even) ofn plays an important role. Whenn is odd, there is a unique solution, and whenn is even, there are no solutions or infinitely many solutions. Clarify under which conditions there are infinitely many solutions.

Problem 19.24. Given an affine spaceE of dimensionn and an affine frame (a_0, . . . , an) for E, letf :E→E andg:E→E be two affine maps represented by the two (n+ 1)×(n+ 1) matrices

A band B c

0 1 0 1

w.r.t. the frame (a_0, . . . , an). We also say thatf andg are represented by (A, b) and (B, c). (1) Prove that the compositionfæg is represented by the matrix

AB Ac +b.0 1

We also say thatfæg is represented by (A, b)(B, c) = (AB, Ac +b).

(2) Prove thatf is invertible iffA is invertible and that the matrix representingf−1 is A−1 A−1b.0 − 1

We also say thatf−1 is represented by (A, b)−1 = (A−1,−A−1b). Prove that ifA is an orthogonal matrix, the matrix associated withf−1 is

A −A b.0 1

Furthermore, denoting the columns ofA byA_1, . . . , An, prove that the vectorA b is the column vector of components

(A1·b, . . . , An·b)

(where· denotes the standard inner product of vectors).

(3) Given two affine frames (a_0, . . . , an) and (a_0, . . . , an) forE, any affine mapf :E→E

has a matrix representation (A, b) w.r.t. (a_0, . . . , an) and (a_0, . . . , an) defined such that

b =0−−−−→0) is expressed over the basis (−−→ , . . . ,−−→ ), andai j is theith coefficient of0 1 0 n

f(aa ) over the basis (−−→a_1, . . . ,−−→). Given any three frames (a_0, . . . , an), (a_0, . . . , an),−−→0 0 n

and (a_0, . . . , an), for any two affine mapsf :E→E andg:E→E, iff has the matrix

representation (A, b) w.r.t. (a_0, . . . , an) and (a_0, . . . , an) andg has the matrix representation

(B, c) w.r.t. (a_0, . . . , an) and (a_0, . . . , an), prove thatgæf has the matrix representation

(B, c)(A, b) w.r.t. (a_0, . . . , an) and (a_0, . . . , an).

(4) Given two affine frames (a_0, . . . , an) and (a_0, . . . , an) forE, there is a unique affine

maph:E→E such thath(ai) =ai fori = 0, . . . , n, and we let (P, ω) be its associated

matrix representation with respect to the frame (a_0, . . . , an). Note thatω =−−→0, and that

0

p

i j

is the

i

th coefficient of

−−→ over the basis (aa , . . . , aa

a ). Observe that (P, ω) is also0aj −−→ −−→

the matrix representation of idE w.r.t. the frames (a_0, . . . , an) and (a_0, . . . , an), in that order. For any affine mapf :E→E, iff has the matrix representation (A, b) over the frame (a_0, . . . , an) and the matrix representation (A , b ) over the frame (a_0, . . . , an), prove that

(A , b ) = (P, ω)−1(A, b)(P, ω).

Given any two affine maps f :E→ E andg:E→E, wheref is invertible, for any affine frame (a_0, . . . , an) forE, if (a_0, . . . , an) is the affine frame image of (a_0, . . . , an) underf (i.e., f(ai) =ai fori = 0, . . . , n), letting (A, b) be the matrix representation off w.r.t. the frame (a_0, . . . , an) and (B, c) be the matrix representation ofg w.r.t. the frame (a_0, . . . , an) (not the frame (a_0, . . . , an)), prove thatgæf is represented by the matrix (A, b)(B, c) w.r.t. the frame (a_0, . . . , an).

Remark: Note that this is the opposite of what happens iff andg are both represented by matrices w.r.t. the “fixed” frame (a_0, . . . , an), wheregæf is represented by the matrix (B, c)(A, b). The frame (a_0, . . . , an) can be viewed as a “moving” frame. The above has applications in robotics, for example to rotation matrices expressed in terms of Euler angles, or “roll, pitch, and yaw.”

Problem 19.25. (a) LetE be a vector space, and letU andV be two subspaces ofE so that they form a direct sumE =U⊕V . Recall that this means that every vectorx∈E can be written asx =u+v, for some uniqueu∈U and some uniquev∈V . Define the function pU :E→U (resp. pV :E→V ) so thatpU(x) =u (resp. pV(x) =v), wherex =u +v, as explained above. Check that thatpU andpV are linear.

(b) Now assume that E is an affine space (nontrivial), and letU andV be affine subspaces such thatE =U⊕V . Pick any &∈V , and defineqU :E→U (resp. qV :E→V , with &∈U) so that

qU(a) =pU (&a) (resp. qV(a) =pV (&a)), for everya∈E.−→ −→−→ −→

Prove thatqU does not depend on the choice of &∈V (resp. qV does not depend on the choice of &∈U). Define the mappU :E→U (resp.pV :E→V ) so that

pU(a) =a−qV(a) (resp. pV(a) =a−qU(a)), for everya∈E.

Prove that pU (resp.pV) is affine.

The mappU (resp. pV) is called the projection ontoU parallel toV (resp. projection

ontoV parallel toU).

(c) Let (a_0, . . . , an) ben+1 affinely independent points inAn and let (a_0, . . . , an) denote

the convex hull of (a_0, . . . , an) (ann-simplex). Prove that iff : An An is an affine map

sending (a_0, . . . , an) inside itself, i.e.,→

f((a_0, . . . , an))⊆ (a_0, . . . , an),

then, f has some fixed pointb∈ (a_0, . . . , an), i.e.,f(b) =b.

Hint: Proceed by induction onn. First, treat the casen = 1. The affine map is determined byf(a_0) andf(a_1), which are affine combinations ofa_0 anda_1. There is an explicit formula for some fixed point off. For the induction step, composef with some suitable projections.

Chapter 20 Polynomials, Ideals and PID’s

20.1 Multisets

This chapter contains a review of polynomials and their basic properties. First, multisets are defined. Polynomials in one variable are defined next. The notion of a polynomial function in one argument is defined. Polynomials in several variable are defined, and so is the notion of a polynomial function in several arguments. The Euclidean division algorithm is presented, and the main consequences of its existence are derived. Ideals are defined, and the characterization of greatest common divisors of polynomials in one variables (gcd’s) in terms of ideals is shown. We also prove the Bezout identity. Next, we consider the factorization of polynomials in one variables into irreducible factors. The unique factorization of polynomials in one variable into irreducible factors is shown. Roots of polynomials and their multiplicity are defined. It is shown that a nonnull polynomial in one variable and of degreem over an integral domain has at mostm roots. The chapter ends with a brief treatment of polynomial interpolation: Lagrange, Newton, and Hermite interpolants are introduced.

In this chapter, it is assumed that all rings considered are commutative. Recall that a (commutative) ringA is an integral domain (or an entire ring) if 1 = 0, and ifab = 0, then eithera = 0 orb = 0, for alla, b∈A. This second condition is equivalent to saying that if a = 0 andb = 0, thenab = 0. Also, recall thata = 0 is not a zero divisor ifab = 0 whenever b = 0. Observe that a field is an integral domain.

Our goal is to define polynomials in one or more indeterminates (or variables) X_1, . . . , Xn, with coefficients in a ringA. This can be done in several ways, and we choose a definition that has the advantage of extending immediately from one to several variables. First, we need to review the notion of a (finite) multiset.

Definition 20.1. Given a setI, a (finite) multiset overI is any functionM :I→ N such thatM(i) = 0 for finitely manyi∈I. The multisetM such thatM(i) = 0 for alli∈I is the empty multiset, and it is denoted by 0. IfM(i) =k = 0, we say thati is a member of M of multiplicityk. The unionM_1 +M_2 of two multisetsM_1 andM_2 is defined such that (M_1+M_2)(i) =M_1(i) +M_2(i), for everyi∈I. IfI is finite, sayI ={1, . . . , n}, the multiset

531 M such thatM(i) =ki for everyi, 1≤i≤n, is denoted byk1· 1 +· · · +kn·n, or more simply, by (k_1, . . . , kn), and deg(k1· 1 +· · · +kn·n) =k_1 +· · · +kn is the size or degree of M. The set of all multisets overI is denoted by N(I), and whenI ={1, . . . , n}, by N(n).

Intuitively, the order of the elements of a multiset is irrelevant, but the multiplicity of each element is relevant, contrary to sets. Everyi∈I is identified with the multisetMi such thatMi(i) = 1 andMi(j) = 0 forj =i. WhenI ={1}, the set N(1) of multisetsk·1 can be identified with N and{1}∗. We will denotek· 1 simply byk.

However, beware that when n≥ 2, the set N(n) of multisets cannot be identified with the set of strings in{1, . . . , n}∗, because multiset union is commutative, but concatenation of strings in{1, . . . , n}∗ is not commutative whenn≥ 2. This is because in a multiset k1· 1 +· · · +kn·n, the order is irrelevant, whereas in a string, the order is relevant. For example, 2· 1 + 3· 2 = 3· 2 + 2· 1, but 11222 = 22211, as strings over{1,2}.

Nevertherless, N(n) and the set Nn of orderedn-tuples under component-wise addition are isomorphic under the map

k1· 1 +· · · +kn·n→ (k_1, . . . , kn).

Thus, since the notation (k_1, . . . , kn) is less cumbersome thatk1· 1 +· · · +kn·n, it will be preferred. We just have to remember that the order of theki is really irrelevant. But whenI is infinite, beware that N(I) and the set NI of orderedI-tuples are not isomorphic.

We are now ready to define polynomials.

20.2 Polynomials

We begin with polynomials in one variable.

Definition 20.2. Given a ringA, we define the setPA(1) of polynomials overA in one variable as the set of functionsP : N→A such thatP(k) = 0 for finitely manyk∈ N. The polynomial such thatP(k) = 0 for allk∈ N is the null (or zero) polynomial and it is denoted by 0. We define addition of polynomials, multiplication by a scalar, and multiplication of polynomials, as follows: Given any three polynomialsP, Q, R∈ PA(1), lettingak =P(k), bk =Q(k), andck =R(k), for everyk∈ N, we defineR =P +Q such that

ck =ak +bk,

R =λP such that ck =λak, whereλ∈A,

andR =P Q such that ck = aibj. i+j=k

We define the polynomial ek such thatek(k) = 1 andek(i) = 0 fori =k. We also denote e_0 by 1 whenk = 0. Given a polynomialP, theak =P(k)∈A are called the coefficients ofP. IfP is not the null polynomial, there is a greatestn≥ 0 such thatan = 0 (and thus, ak = 0 for allk > n) called the degree ofP and denoted by deg(P). Then,P is written uniquely as

P =a_0e_0 +a_1e_1 +· · · +anen. WhenP is the null polynomial, we let deg(P) =−∞.

There is an injection ofA intoPA(1) given by the mapa→a1 (recall that 1 denotese_0). There is also an injection of

N

into

P

A

(1) given by the map

k

→

e

k

. Observe that e

k

= e

k 1

(with e^0 =e_0 = 1). In order to alleviate the notation, we often denotee_1 byX, and we call_1

X a variable (or indeterminate). Then,ek =ek is denoted byXk. Adopting this notation,1

given a nonnull polynomialP of degreen, ifP(k) =ak,P is denoted by

P =a_0 +a_1X +· · · +anXn,

or by P =anXn +an−1Xn−1 +· · · +a_0,

if this is more convenient (the order of the terms does not matter anyway). Sometimes, it will also be convenient to write a polynomial as

P =a_0Xn +a_1Xn−1 +· · · +an.

The setPA(1) is also denoted byA[X] and a polynomialP may be denoted byP(X). In denoting polynomials, we will use both upper-case and lower-case letters, usually,P, Q, R, S,p, q, r, s, but alsof, g, h, etc., if needed (as long as no ambiguities arise).

Given a nonnull polynomial P of degreen, the nonnull coefficientan is called the leading coefficient ofP. The coefficienta_0 is called the constant term ofP. A polynomial of the formakXk is called a monomial. We say thatakXk occurs inP ifak = 0. A nonzero polynomialP of degreen is called a monic polynomial (or unitary polynomial, or monic) if an = 1, wherean is its leading coefficient, and such a polynomial can be written as

P =Xn +an−1Xn−1 +· · · +a_0 or P =Xn +a_1Xn−1 +· · · +an.

The choice of the variableX to denotee_1 is standard practice, but there is nothing special aboutX. We could have chosenY ,Z, or any other symbol, as long as no ambiguities arise.

Formally, the definition ofPA(1) has nothing to do withX. The reason for usingX is simply convenience. Indeed, it is more convenient to write a polynomial asP =a_0 +a_1X + · · · +anXn rather than asP =a_0e_0 +a_1e_1 +· · · +anen.

We have the following simple but crucial proposition.

Proposition 20.1. Given two nonnull polynomialsP(X) =a_0+a_1X+· · ·+amXm of degree m andQ(X) =b_0 +b_1X +· · · +bnXn of degreen, if eitheram orbn is not a zero divisor, thenambn = 0, and thus,P Q = 0 and

deg( P Q) = deg(P) + deg(Q). In particular, ifA is an integral domain, thenA[X] is an integral domain.

Proof. Since the coefficient ofXm+n inP Q isambn, and since we assumed that eitheram or an is not a zero divisor, we haveambn = 0, and thus,P Q = 0 and

deg(P Q) = deg(P) + deg(Q).

Then, it is obvious thatA[X] is an integral domain.

It is easily verified that A[X] is a commutative ring, with multiplicative identity 1X^0 = 1. It is also easily verified thatA[X] satisfies all the conditions of Definition 2.9, butA[X] is not a vector space, sinceA is not necessarily a field.

A structure satisfying the axioms of Definition 2.9 when K is a ring (and not necessarily a field) is called a module. As we mentioned in Section 4.2, we will not study modules because they fail to have some of the nice properties that vector spaces have, and thus, they are harder to study. For example, there are modules that do not have a basis.

However, when the ring A is a field,A[X] is a vector space. But even whenA is just a ring, the family of polynomials (Xk)k∈N is a basis ofA[X], since every polynomialP(X) can be written in a unique way asP(X) =a_0 +a_1X +· · · +anXn (withP(X) = 0 whenP(X) is the null polynomial). Thus,A[X] is a free module.

Next, we want to define the notion of evaluating a polynomialP(X) at someα∈A. For this, we need a proposition.

Proposition 20.2. LetA, B be two rings and leth:A→ B be a ring homomorphism. For anyβ∈B, there is a unique ring homomorphismÕ:A[X]→B extendingh such that Õ(X) =β, as in the following diagram (where we denote byh+β the maph+β:A∪{X} →B such that (h +β)(a) =h(a) for alla∈A and (h +β)(X) =β):

ι A[X]

}

A∪ {XLL

Õ h+β

LLLLLLLL B Proof. LetÕ(0) = 0, and for every nonull polynomialP(X) =a_0 +a_1X +· · · +anXn, let Õ(P(X)) =h(a_0) +h(a_1)β +· · · +h(an)βn.

It is easily verified thatÕ is the unique homomorphismÕ:A[X]→B extendingh such that Õ(X) =β.

Taking A =B in Proposition 20.2 andh:A→A the identity, for everyβ∈A, there is a unique homomorphismÕβ:A[X]→A such thatÕβ(X) =β, and for every polynomial P(X), we writeÕβ(P(X)) asP(β) and we callP(β) the value ofP(X) atX =β. Thus, we can define a functionPA:A→A such thatPA(β) =P(β), for allβ∈A. This function is called the polynomial function induced by P .

More generally, PB can be defined for any (commutative) ringB such thatA⊆B. In general, it is possible thatPA = QA for distinct polynomialsP, Q. We will see shortly conditions for which the mapP→PA is injective. In particular, this is true forA = R (in general, any infinite integral domain). We now define polynomials inn variables.

Definition 20.3. Givenn≥ 1 and a ringA, the setPA(n) of polynomials overA inn variables is the set of functionsP : N(n) A such thatP(k_1, . . . , kn) = 0 for finitely many

(

k

1

, . . . , k

n

)

∈

N

(

n

)

→

. The polynomial such that P(k_1, . . . , kn) = 0 for all (k_1, . . . , kn) is the null (or zero) polynomial and it is denoted by 0. We define addition of polynomials, multiplication by a scalar, and multiplication of polynomials, as follows: Given any three polynomialsP, Q, R∈ P A(n), lettinga(k1,...,kn) =P(k_1, . . . , kn),b(k1,...,kn) =Q(k_1, . . . , kn), c(k1,...,kn) =R(k_1, . . . , kn), for every (k_1, . . . , kn)∈ N(n), we defineR =P +Q such that

c(k_1,...,kn) =a(k_1,...,kn) +b(k_1,...,kn),

R =λP, whereλ∈A, such that

c(k_1,...,kn) =λa(k_1,...,kn),

andR =P Q, such that

c(k_1,...,kn) = a(i_1,...,in)b(j_1,...,jn).

(i_1,...,in)+(j_1,...,jn)=(k_1,...,kn)

For every ( k_1, . . . , kn)∈ N(n), we lete(k1,...,kn) be the polynomial such that e(k1,...,kn)(k_1, . . . , kn) = 1 and e(k1,...,kn)(h_1, . . . , hn) = 0,

for ( h_1, . . . , hn) = (k_1, . . . , kn). We also denotee(0,...,0) by 1. Given a polynomialP, the a(k1,...,kn) =P(k_1, . . . , kn)∈A, are called the coefficients ofP. IfP is not the null polynomial, there is a greatestd≥ 0 such thata(k1,...,kn) = 0 for some (k_1, . . . , kn)∈ N(n), withd = k_1 +· · · +kn, called the total degree ofP and denoted by deg(P). Then, P is written uniquely as

P = a(k_1,...,kn)e(k_1,...,kn). (k_1,...,kn)∈N(n)

When P is the null polynomial, we let deg(P) =−∞. There is an injection ofA intoPA(n) given by the mapa→a1 (where 1 denotese(0,...,0)). There is also an injection of N(n) intoPA(n) given by the map (h_1, . . . , hn)→e(h1,...,hn). Note thate(h1,...,hn)e(k1,...,kn) =e(h1+k1,...,hn+kn). In order to alleviate the notation, letX_1, . . . , Xn ben distinct variables and denotee(0,...,0,1,0...,0), where 1 occurs in the positioni, byXi (where 1≤i≤n). With this convention, in view ofe(h1,...,hn)e(k1,...,kn) =e(h1+k1,...,hn+kn), the polynomiale(k1,...,kn) is denoted byXk1 Xkn (withe(0,...,0) =X^0 X^0 = 1) and it is called1· · · n 1· · ·a primitive monomial. Then,P is also written as

P = a(k1,...,kn)Xk1 Xk .1· · · n

(k_1,...,kn)∈N(n)

We also denotePA(n) byA[X_1, . . . , Xn]. A polynomialP∈A[X_1, . . . , Xn] is also denoted byP(X_1, . . . , Xn).

As in the case n = 1, there is nothing special about the choice ofX_1, . . . , Xn as variables (or indeterminates). It is just a convenience. After all, the construction ofPA(n) has nothing to do withX_1, . . . , Xn.

Given a nonnull polynomial P of degreed, the nonnull coefficientsa(k1,...,kn) = 0 such thatd =k_1 +· · · +kn are called the leading coefficients ofP. A polynomial of the form a(k1,...,kn)Xk1 kn is called a monomial. Note that deg(a(k1,...,kn)Xk1 Xk ) =k_1+· · ·+kn.1· · ·Xn 1· · · n Given a polynomial

P = a(k1,...,kn)Xk1 Xk ,1· · · n

(k_1,...,kn)∈N(n)

a monomiala(k1,...,kn)Xk1 Xkn occurs in the polynomialP ifa(k1,...,kn) = 0.1· · · n

A polynomial

P

=

a

(

k

1

,...,k

n

)

X

k1 Xkn

1· · ·n

(k_1,...,kn)∈N(n)

is homogeneous of degreed if deg(Xk1 Xkn ) =d,1· · · n

for every monomiala(k1,...,kn)Xk1 Xkn occurring inP. IfP is a polynomial of total degree1· · · n

d, it is clear thatP can be written uniquely as

P =P(0) +P(1) +· · · +P(d), whereP(i) is the sum of all monomials of degreei occurring inP, where 0≤i≤d.

It is easily verified that A[X_1, . . . , Xn] is a commutative ring, with multiplicative identity 1X^0 X^0 = 1. It is also easily verified thatA[X] is a module. WhenA is a field,A[X] is1· · ·

a vector space.

Even whenA is just a ring, the family of polynomials

k1 Xk )(k1,...,kn)∈N(n)(X1· · · n

is a basis ofA[X_1, . . . , Xn], since every polynomialP(X_1, . . . , Xn) can be written in a unique way as

P(X_1, . . . , Xn) = a(k1,...,kn)Xk1 Xk .1· · · n

(k_1,...,kn)∈N(n)

Thus,A[X_1, . . . , Xn] is a free module.

Remark: The construction of Definition 20.3 can be immediately extended to an arbitrary setI, and not justI ={1, . . . , n}. It can also be applied to monoids more general that N(I). Proposition 20.2 is generalized as follows.

Proposition 20.3. LetA, B be two rings and leth:A→B be a ring homomorphism. For anyβ = (β_1, . . . , βn)∈Bn, there is a unique ring homomorphismÕ:A[X_1, . . . , Xn]→B extendingh such thatÕ(Xi) =βi, 1≤i≤n, as in the following diagram (where we denote byh +β the maph +β:A∪ {X_1, . . . , Xn} →B such that (h +β)(a) =h(a) for alla∈A and (h +β)(Xi) =βi, 1≤i≤n):

A∪ {X_1, . . . , Xι A[X_1, . . . , Xn]

TTTT Õ

h+β

n}TTTTTTTTTTTTT B

Proof. LetÕ(0) = 0, and for every nonull polynomial

P(X_1, . . . , Xn) = a(k1,...,kn)Xk1 Xk ,1· · · n

(k_1,...,kn)∈N(n)

let Õ(P(X_1, . . . , Xn)) = h(a(k1,...,kn))βk1 βk.1· · · n It is easily verified thatÕ is the unique homomorphismÕ:A[X_1, . . . , Xn]→B extendingh such thatÕ(Xi) =βi.

Taking A =B in Proposition 20.3 andh:A→A the identity, for everyβ_1, . . . , βn∈A, there is a unique homomorphismÕ:A[X_1, . . . , Xn]→ A such thatÕ(Xi) = βi, and for every polynomialP(X_1, . . . , Xn), we writeÕ(P(X_1, . . . , Xn)) asP(β_1, . . . , βn) and we call P(β_1, . . . , βn) the value ofP(X_1, . . . , Xn) atX_1 =β_1, . . . , Xn =βn. Thus, we can define a functionPA:An A such thatPA(β_1, . . . , βn) =P(β_1, . . . , βn), for allβ_1, . . . , βn∈A. This→

function is called the polynomial function induced by P.

More generally, PB can be defined for any (commutative) ringB such thatA⊆B. As in the case of a single variable, it is possible thatPA =QA for distinct polynomialsP, Q. We will see shortly that the mapP→PA is injective whenA = R (in general, any infinite integral domain).

Given any nonnull polynomialP(X_1, . . . , Xn) =(k1,...,kn)∈N(n)a(k1,...,kn)Xk1 Xkn in1· · · n A[X_1, . . . , Xn], wheren≥ 2,P(X_1, . . . , Xn) can be uniquely written as

P(X_1, . . . , Xn) = Qkn(X_1, . . . , Xn−1)Xk ,n

where each polynomial Qkn(X_1, . . . , Xn−1) is inA[X_1, . . . , Xn−1]. Thus, even ifA is a field, A[X_1, . . . , Xn−1] is not a field, which confirms that it is useful (and necessary!) to consider polynomials over rings that are not necessarily fields.

It is not difficult to show that A[X_1, . . . , Xn] andA[X_1, . . . , Xn−1][Xn] are isomorphic rings. This way, it is often possible to prove properties of polynomials in several variables X_1, . . . , Xn, by induction on the numbern of variables. For example, given two nonnull polynomialsP(X_1, . . . , Xn) of total degreep andQ(X_1, . . . , Xn) of total degreeq, since we assumed thatA is an integral domain, we can prove that

deg(P Q) = deg(P) + deg(Q), and thatA[X_1, . . . , Xn] is an integral domain.

Next, we will consider the division of polynomials (in one variable).

20.3 Euclidean Division of Polynomials

We know that every natural number n≥ 2 can be written uniquely as a product of powers of prime numbers and that prime numbers play a very important role in arithmetic. It would be nice if every polynomial could be expressed (uniquely) as a product of “irreducible” factors. This is indeed the case for polynomials over a field. The fact that there is a division algorithm for the natural numbers is essential for obtaining many of the arithmetical properties of the natural numbers. As we shall see next, there is also a division algorithm for polynomials in A[X], whenA is a field.

Proposition 20.4. LetA be a ring, letf(X), g(X)∈A[X] be two polynomials of degree m = deg(f) andn = deg(g) withf(X) = 0 , and assume that the leading coefficientam of f(X) is invertible. Then, there exist unique polynomialsq(X) andr(X) inA[X] such that

g =fq +r and deg(r)< deg(f) =m.

Proof. We first prove the existence ofq andr. Let

f =am +am−1Xm−1 +· · · +a_0,mX

and g =bnXn +bn−1Xn−1 +· · · +b_0.

Ifn < m, then letq = 0 andr =g. Since deg(g)< deg(f) andr =g, we have deg(r)< deg(f).

20.3. EUCLIDEAN DIVISION OF POLYNOMIALS 539

Ifn≥m, we proceed by induction onn. Ifn = 0, theng =b_0,m = 0,f =a_0 = 0, and we let

q

=

a

−

1

b0 andr = 0. Since deg(r) = deg(0) =−∞ and deg(f) = deg(a_0) = 0 because

0

a_0 = 0, we have deg(r)< deg(f).

Ifn≥ 1, sincen≥m, note that

g_1(X) =g(X)−bna−1Xn−mf(X)m

=bnXn +bn−1Xn−1 +· · · +b0−bna−1Xn−m(amXm +am−1Xm−1 +· · · +a_0)m

is a polynomial of degree deg( g_1)< n, since the termsbnXn andbna−1Xn−mamXm of degreem

n cancel out. Now, since deg(g_1)< n, by the induction hypothesis, we can findq_1 andr such that

g_1 =fq_1 +r and deg(r)< deg(f) =m, and thus,

g_1(X) =g(X)−bna−1Xn−mf(X) =f(X)q_1(X) +r(X),

m

from which, lettingq(X) =bna−1Xn−m +q_1(X), we getm

g =fq +r and deg(r)< m = deg(f).

We now prove uniqueness. If

g =fq_1 +r_1 =fq_2 +r_2,

with deg(r_1)< deg(f) and deg(r_2)< deg(f), we get

f(q1−q_2) =r2−r_1.

Ifq2−q_1 = 0, since the leading coefficientam off is invertible, by Proposition 20.1, we have

deg(r2−r_1) = deg(f(q1−q_2)) = deg(f) + deg(q2−q_1),

and so, deg(r2−r_1)≥ deg(f), which contradicts the fact that deg(r_1)< deg(f) and deg(r_2)< deg(f). Thus,q_1 =q2, and then alsor_1 =r_2.

It should be noted that the proof of Proposition 20.4 actually provides an algorithm for finding the quotient q and the remainder r of the division ofg byf. This algorithm is called the Euclidean algorithm, or division algorithm. Note that the division ofg byf is always possible whenf is a monic polynomial, since 1 is invertible. Also, whenA is a field, am = 0 is always invertible, and thus, the division can always be performed. We say thatf dividesg whenr = 0 in the result of the divisiong =fq +r. We now draw some important consequences of the existence of the Euclidean algorithm.

20.4 Ideals, PID’s, and Greatest Common Divisors

First, we introduce the fundamental concept of an ideal.

Definition 20.4. Given a ringA, an ideal ofA is any nonempty subset I ofA satisfying the following two properties:

(ID1) Ifa, b∈ I, thenb−a∈ I.

(ID2) Ifa∈ I, thenax∈ I for everyx∈A.

An ideal I is a principal ideal if there is somea∈ I, called a generator, such that I ={ax|x∈A}.

The equality I ={ax| x∈ A} is also written as I = aA or as I = (a). The ideal I = (0) ={0} is called the null ideal (or zero ideal).

An ideal I is a maximal ideal if I =A and for every ideal J =A, if I⊆ J, then J = I. An ideal I is a prime ideal if I =A and ifab∈ I, thena∈ I orb∈ I, for alla, b∈A. Equivalently, I is a prime ideal if I =A and ifa, b∈A−I, thenab∈A−I, for alla, b∈A. In other words,A−I is closed under multiplication and 1∈A−I.

Note that if I is an ideal, then I =A iff 1∈ I. Since by definition, an ideal I is nonempty, there is somea∈ I, and by (ID1) we get 0 =a−a∈ I. Then, for everya∈ I, since 0∈ I, by (ID1) we get−a∈ I. Thus, an ideal is an additive subgroup ofA. Because of (ID2), an ideal is also a subring.

Observe that if A is a field, thenA only has two ideals, namely, the trivial ideal (0) and A itself. Indeed, if I = (0), because every nonnull element has an inverse, then 1∈ I, and thus, I =A.

Given a, b∈A, we say thatb is a multiple ofa and thata dividesb ifb =ac for some c∈A; this is usually denoted bya|b. Note that the principal ideal (a) is the set of all multiples ofa, and thata dividesb iffb is a multiple ofa iffb∈ (a) iff (b)⊆ (a).

Note that every a∈A divides 0. However, it is customary to say thata is a zero divisor iffac = 0 for somec = 0. With this convention, 0 is a zero divisor unlessA ={0} (the trivial ring), andA is an integral domain iff 0 is the only zero divisor inA.

Given a, b∈A witha, b = 0, if (a) = (b) then there existc, d∈A such thata =bc and b =ad . From this, we geta =adc andb =bcd, that is,a(1−dc ) = 0 andb(1−cd) = 0. IfA is an integral domain, we getdc = 1 andcd = 1, that is,c is invertible with inversed. Thus, whenA is an integral domain, we haveb =ad, withd invertible. The converse is obvious, if b =ad withd invertible, then (a) = (b).

As a summary, if A is an integral domain, for anya, b∈A witha, b = 0, we have (a) = (b) iff there exists some invertibled∈A such thatb =ad. An invertible elementu∈A is also called a unit.

Given two ideals I and J, their sum

I +J ={a +b|a∈ I, b∈ J}

is clearly an ideal. Given any nonempty subsetJ ofA, the set

{a_1x_1 +· · · +anxn|x_1, . . . , xn∈A, a_1, . . . , an∈J, n≥ 1}

is easily seen to be an ideal, and in fact, it is the smallest ideal containingJ. It is usually denoted by (J).

Ideals play a very important role in the study of rings. They tend to show up everywhere. For example, they arise naturally from homomorphisms.

Proposition 20.5. Given any ring homomorphismh:A→B, the kernel Kerh ={a∈A|h(a) = 0} ofh is an ideal.

Proof. Givena, b∈A, we havea, b∈ Kerh iffh(a) =h(b) = 0, and sinceh is a homomorphism, we get

h(b−a) =h(b)−h(a) = 0, and

h(ax) =h(a)h(x) = 0 for allx∈A, which shows that Kerh is an ideal.

There is a sort of converse property. Given a ring A and an ideal I⊆ A, we can define the quotient ringA/I, and there is a surjective homomorphismπ:A→A/I whose kernel is precisely I.

Proposition 20.6. Given any ringA and any ideal I⊆A, the equivalence relation≡I defined bya≡Ib iffb−a∈ I is a congruence, which means that if a1≡Ib_1 anda2≡Ib2, then

1. a_1 +a2≡Ib_1 +b_2, and

2. a_1a2≡Ib_1b_2.

Then, the setA/I of equivalence classes modulo I is a ring under the operations

[a] + [b] = [a +b] [a][b] = [ab].

The map π:A→A/I such thatπ(a) = [a] is a surjective homomorphism whose kernel is precisely I.

Proof. Everything is straightforward. For example, ifa1≡Ib_1 anda2≡Ib_2, thenb1−a1∈ I andb2−a2∈ I. Since I is an ideal, we get

(b1−a_1)b_2 =b_1b2−a_1b2∈ I

and (b2−a_2)a_1 =a_1b2−a_1a2∈ I. Since I is an ideal, and thus, an additive group, we get

b_1b2−a_1a2∈ I,

i.e.,a_1a2≡Ib_1b_2. The equality Kerπ = I holds because I is an ideal.

Example 20.1.

1. In the ring Z, for everyp∈ Z, the subrouppZ is an ideal, and Z/pZ is a ring, the ring of residues modulop. This ring is a field iffp is a prime number.

2. The quotient of the polynomial ring R[X] by a prime ideal I is an integral domain.

3. The quotient of the polynomial ring R[X] by a maximal ideal I is a field. For example, if I = (X^2 + 1), the principal ideal generated byX^2 + 1 (which is indeed a maximal ideal sinceX^2 + 1 has no real roots), then R[X]/(X^2 + 1)= C.∼

The following proposition yields a characterization of prime ideals and maximal ideals in terms of quotients.

Proposition 20.7. Given a ringA, for any ideal I⊆A, the following properties hold.

(1) The ideal I is a prime ideal iffA/I is an integral domain.

(2) The ideal I is a maximal ideal iffA/I is a field.

Proof. (1) Assume that I is a prime ideal. Since I is prime, I =A, and thus,A/I is not the trivial ring (0). If [a][b] = 0, since [a][b] = [ab], we haveab∈ I, and since I is prime, then eithera∈ I orb∈ I, so that either [a] = 0 or [b] = 0. Thus,A/I is an integral domain.

Conversely, assume thatA/I is an integral domain. SinceA/I is not the trivial ring, I =A. Assume thatab∈ I. Then, we have

π(ab) =π(a)π(b) = 0,

which implies that eitherπ(a) = 0 orπ(b) = 0, sinceA/I is an integral domain (where π:A→A/I is the quotient map). Thus, eithera∈ I orb∈ I, and I is a prime ideal.

(2) Assume that I is a maximal ideal. As in (1),A/I is not the trivial ring (0). Let [a] = 0 inA/I. We need to prove that [a] has a multiplicative inverse. Since [a] = 0, we havea / I. Let Ia be the ideal generated by I anda. We have

I⊆ Ia and I = Ia,

sincea / I, and since I is maximal, this implies that

Ia =A.

However, we know that Ia ={ax +h|x∈A, h∈ I}, and thus, there is somex∈A so that

ax +h = 1,

which proves that [a][x] = 1, as desired.

Conversely, assume that A/I is a field. Again, sinceA/I is not the trivial ring, I =A. Let J be any proper ideal such that I⊆ J, and assume that I = J. Thus, there is some j∈ J− I, and since Kerπ = I, we haveπ(j) = 0. SinceA/I is a field andπ is surjective, there is somek∈A so thatπ(j)π(k) = 1, which implies that

jk− 1 =i

for somei∈ J, and since I⊂ J and J is an ideal, it follows that 1 =jk−i∈ J, showing that J =A, a contradiction. Therefore, I = J, and I is a maximal ideal.

As a corollary, we obtain the following useful result. It emphasizes the importance of maximal ideals.

Corollary 20.8. Given any ringA, every maximal ideal I inA is a prime ideal.

Proof. If I is a maximal ideal, then, by Proposition 20.7, the quotient ringA/I is a field. However, a field is an integral domain, and by Proposition 20.7 (again), I is a prime ideal.

Observe that a ring A is an integral domain iff (0) is a prime ideal. This is an example of a prime ideal which is not a maximal ideal, as immediately seen inA = Z, where (p) is a maximal ideal for every prime numberp.

A less obvious example of a prime ideal which is not a maximal ideal, is the ideal (X) in the ring of polynomials Z[X]. Indeed, (X,2) is also a prime ideal, but (X) is properly contained in (X,2).

Definition 20.5. An integral domain in which every ideal is a principal ideal is called a principal ring or principal ideal domain, for short, a PID.

The ring Z is a PID. This is a consequence of the existence of a (Euclidean) division algorithm. As we shall see next, whenK is a field, the ringK[X] is also a principal ring.

However, when n≥ 2, the ringK[X_1, . . . , Xn] is not principal. For example, in the ring K[X, Y ], the ideal (X, Y ) generated byX andY is not principal. First, since (X, Y ) is the set of all polynomials of the formXq_1 +Y q_2, whereq_1, q2∈K[X, Y ], except when Xq_1 +Y q_2 = 0, we have deg(Xq_1 +Y q_2)≥ 1. Thus, 1/∈ (X, Y ). Now if there was somep∈ K[X, Y ] such that (X, Y ) = (p), since 1/∈ (X, Y ), we must have deg(p)≥ 1. But we would also haveX =pq_1 andY =pq_2, for someq_1, q2∈K[X, Y ]. Since deg(X) = deg(Y ) = 1, this is impossible.

Even though K[X, Y ] is not a principal ring, a suitable version of unique factorization in terms of irreducible factors holds. The ringK[X, Y ] (and more generallyK[X_1, . . . , Xn]) is what is called a unique factorization domain, for short, UFD, or a factorial ring.

From this point until Definition 20.10, we consider polynomials in one variable over a fieldK.

Remark: Although we already proved part (1) of Proposition 20.9 in a more general situation above, we reprove it in the special case of polynomials. This may offend the purists, but most readers will probably not mind.

Proposition 20.9. LetK be a field. The following properties hold:

(1) For any two nonzero polynomialsf, g∈K[X], (f) = (g) iff there is someλ = 0 inK such thatg =λf.

(2) For every nonnull ideal I inK[X], there is a unique monic polynomialf∈K[X] such that I = (f).

Proof. (1) If (f) = (g), there are some nonzero polynomialsq_1, q2∈K[X] such thatg =fq_1 andf = gq_2. Thus, we havef = fq_1q_2, which impliesf(1−q1q_2) = 0. SinceK is a field, by Proposition 20.1,K[X] has no zero divisor, and since we assumedf = 0, we must haveq_1q_2 = 1. However, if eitherq_1 orq_2 is not a constant, by Proposition 20.1 again, deg(q_1q_2) = deg(q_1) + deg(q_2)≥ 1, contradictingq_1q_2 = 1, since deg(1) = 0. Thus, both q_1, q2∈K− {0}, and (1) holds withλ =q_1. In the other direction, it is obvious thatg =λf implies that (f) = (g).

(2) Since we are assuming that I is not the null ideal, there is some polynomial of smallest degree in I, and sinceK is a field, by suitable multiplication by a scalar, we can make sure that this polynomial is monic. Thus, letf be a monic polynomial of smallest degree in I. By (ID2), it is clear that (f)⊆ I. Now, letg∈ I. Using the Euclidean algorithm, there exist uniqueq, r∈K[X] such that

g =qf +r and deg(r)< deg(f).

If r = 0, there is someλ = 0 inK such thatλr is a monic polynomial, and sinceλr = λg−λqf, withf, g∈ I, by (ID1) and (ID2), we haveλr∈ I, where deg(λr)< deg(f) and λr is a monic polynomial, contradicting the minimality of the degree off. Thus,r = 0, and g∈ (f). The uniqueness of the monic polynomialf follows from (1).

Proposition 20.9 shows thatK[X] is a principal ring whenK is a field.

We now investigate the existence of a greatest common divisor (gcd) for two nonzero polynomials. Given any two nonzero polynomialsf, g∈K[X], recall thatf dividesg if g =fq for someq∈K[X].

Definition 20.6. Given any two nonzero polynomialsf, g∈ K[X], a polynomiald∈K[X] is a greatest common divisor off andg (for short, a gcd off andg) ifd dividesf andg and wheneverh∈K[X] dividesf andg, thenh dividesd. We say thatf andg are relatively prime if 1 is a gcd off andg.

Note thatf andg are relatively prime iff all of their gcd’s are constants (scalars inK), or equivalently, iff, g have no divisorq of degree deg(q)≥ 1.

In particular, note thatf andg are relatively prime whenf is a nonzero constant polynomial (a scalarλ = 0 inK) andg is any nonzero polynomial.

We can characterize gcd’s of polynomials as follows. Proposition 20.10. LetK be a field and letf, g∈K[X] be any two nonzero polynomials. For every polynomiald∈K[X], the following properties are equivalent:

(1) The polynomiald is a gcd off andg.

(2) The polynomiald dividesf andg and there existu, v∈K[X] such that

d =uf +vg.

(3) The ideals (f),(g), and (d) satisfy the equation

(d) = (f) + (g).

In addition,d = 0, andd is unique up to multiplication by a nonzero scalar inK.

Proof. Given any two nonzero polynomialsu, v∈K[X], observe thatu dividesv iff (v)⊆ (u). Now, (2) can be restated as (f)⊆ (d), (g)⊆ (d), andd∈ (f) + (g), which is equivalent to (d) = (f) + (g), namely (3).

If (2) holds, sinced =uf +vg, wheneverh∈K[X] dividesf andg, thenh dividesd, andd is a gcd off andg.

Assume that d is a gcd off andg. Then, sinced dividesf andd dividesg, we have (f)⊆ (d) and (g)⊆ (d), and thus (f) + (g)⊆ (d), and (f) + (g) is nonempty sincef and g are nonzero. By Proposition 20.9, there exists a monic polynomiald1∈K[X] such that (d_1) = (f) + (g). Then,d_1 divides bothf andg, and sinced is a gcd off andg, thend_1 dividesd, which shows that (d)⊆ (d_1) = (f) + (g). Consequently, (f) + (g) = (d), and (3) holds.

Since (d) = (f) + (g) and f and g are nonzero, the last part of the proposition is obvious.

As a consequence of Proposition 20.10, two nonzero polynomialsf, g∈K[X] are relatively prime iff there existu, v∈K[X] such that

uf +vg = 1.

The identity

d =uf +vg

of part (2) of Proposition 20.10 is often called the Bezout identity.

We derive more useful consequences of Proposition 20.10.

Proposition 20.11. LetK be a field and letf, g∈K[X] be any two nonzero polynomials. For every gcdd∈K[X] off andg, the following properties hold:

(1) For every nonzero polynomialq∈K[X], the polynomialdq is a gcd offq andgq. (2) For every nonzero polynomialq∈K[X], ifq dividesf andg, thend/q is a gcd off/q andg/q.

Proof. (1) By Proposition 20.10 (2),d dividesf andg, and there existu, v∈K[X], such that

d =uf +vg. Then,dq dividesfq andgq, and

dq =ufq +vgq.

By Proposition 20.10 (2),dq is a gcd offq andgq. The proof of (2) is similar.

The following proposition is used often.

Proposition 20.12. (Euclid’s proposition) LetK be a field and letf, g, h∈K[X] be any nonzero polynomials. Iff dividesgh andf is relatively prime tog, thenf dividesh. Proof. From Proposition 20.10,f andg are relatively prime iff there exist some polynomials u, v∈K[X] such that

uf +vg = 1.

Then, we have

ufh +vgh =h,

and sincef dividesgh, it divides bothufh andvgh, and so,f dividesh. Proposition 20.13. LetK be a field and letf, g_1, . . . , gm∈K[X] be some nonzero polynomials. Iff andgi are relatively prime for alli, 1≤i≤m, thenf andg1· · ·gm are relatively prime.

Proof. We proceed by induction onm. The casem = 1 is trivial. Leth =g2· · ·gm. By the induction hypothesis,f andh are relatively prime. Letd be a gcd off andg1h. We claim thatd is relatively prime tog_1. Otherwise,d andg_1 would have some nonconstant gcdd_1 which would divide bothf andg_1, contradicting the fact thatf andg_1 are relatively prime. Now, by Proposition 20.12, sinced dividesg_1h andd andg_1 are relatively prime,d divides h =g2· · ·gm. But then,d is a divisor off andh, and sincef andh are relatively prime,d must be a constant, andf andg1· · ·gm are relatively prime.

Definition 20.6 is generalized to any finite number of polynomials as follows.

Definition 20.7. Given any nonzero polynomialsf_1, . . . , fn∈K[X], wheren≥ 2, a polynomiald∈K[X] is a greatest common divisor off_1, . . . , fn (for short, a gcd of f_1, . . . , fn) ifd divides eachfi and wheneverh∈K[X] divides eachfi, thenh dividesd. We say that f_1, . . . , fn are relatively prime if 1 is a gcd off_1, . . . , fn.

It is easily shown that Proposition 20.10 can be generalized to any finite number of polynomials, and similarly for its relevant corollaries. The details are left as an exercise. Proposition 20.14. LetK be a field and letf_1, . . . , fn∈ K[X] be anyn≥ 2 nonzero polynomials. For every polynomiald∈K[X], the following properties are equivalent: (1) The polynomiald is a gcd off_1, . . . , fn.

(2) The polynomiald divides eachfi and there existu_1, . . . , un∈K[X] such that d =u_1f_1 +· · · +unfn.

(3) The ideals (fi), and (d) satisfy the equation (d) = (f_1) +· · · + (fn).

In addition,d = 0, andd is unique up to multiplication by a nonzero scalar inK.

As a consequence of Proposition 20.14, some polynomialsf_1, . . . , fn∈K[X] are relatively prime iff there existu_1, . . . , un∈K[X] such that

u_1f_1 +· · · +unfn = 1.

The identity

u_1f_1 +· · · +unfn = 1

of part (2) of Proposition 20.14 is also called the Bezout identity.

We now consider the factorization of polynomials of a single variable into irreducible factors.

20.5 Factorization and Irreducible Factors inK[X]

Definition 20.8. Given a fieldK, a polynomialp∈K[X] is irreducible or indecomposable or prime if deg(p)≥ 1 and ifp is not divisible by any polynomialq∈K[X] such that 1≤ deg(q)< deg(p). Equivalently,p is irreducible if deg(p)≥ 1 and ifp =q_1q_2, then either q1∈K orq2∈K (and of course,q_1 = 0,q_2 = 0).

Example 20.2. Every polynomialaX +b of degree 1 is irreducible. Over the field R, the polynomialX^2 + 1 is irreducible (why?), butX^3 + 1 is not irreducible, since

X^3 + 1 = (X + 1)(X^2 X + 1).−

The polynomialX^2 X + 1 is irreducible over R (why?). It would seem thatX^4 + 1 is−

irreducible over R, but in fact,

X^4 + 1 = (X^2 −√2X + 1)(X^2 +√2X + 1).

However, in view of the above factorization,X^4 + 1 is irreducible over Q.

It can be shown that the irreducible polynomials over R are the polynomials of degree 1, or the polynomials of degree 2 of the formaX^2 +bX +c, for whichb^2 4ac < 0 (i.e., those−

having no real roots). This is not easy to prove! Over the complex numbers C, the only irreducible polynomials are those of degree 1. This is a version of a fact often referred to as the “Fundamental theorem of Algebra”, or, as the French sometimes say, as “d’Alembert’s theorem”!

We already observed that for any two nonzero polynomials f, g∈K[X],f dividesg iff (g)⊆ (f). In view of the definition of a maximal ideal given in Definition 20.4, we now prove that a polynomialp∈K[X] is irreducible iff (p) is a maximal ideal inK[X].

Proposition 20.15. A polynomialp∈K[X] is irreducible iff (p) is a maximal ideal in K[X].

Proof. SinceK[X] is an integral domain, for all nonzero polynomialsp, q∈K[X], deg(pq) = deg(p) + deg(q), and thus, (p) =K[X] iff deg(p)≥ 1. Assume thatp∈K[X] is irreducible. Since every ideal inK[X] is a principal ideal, every ideal inK[X] is of the form (q), for someq∈K[X]. If (p)⊆ (q), with deg(q)≥ 1, thenq dividesp, and sincep∈K[X] is irreducible, this implies thatp =λq for someλ = 0 inK, and so, (p) = (q). Thus, (p) is a maximal ideal. Conversely, assume that (p) is a maximal ideal. Then, as we showed above, deg(p)≥ 1, and ifq dividesp, with deg(q)≥ 1, then (p)⊆ (q), and since (p) is a maximal ideal, this implies that (p) = (q), which means thatp =λq for someλ = 0 inK, and so,p is irreducible.

Let p∈K[X] be irreducible. Then, for every nonzero polynomialg∈K[X], eitherp and g are relatively prime, orp dividesg. Indeed, ifd is any gcd ofp andg, ifd is a constant, then p andg are relatively prime, and if not, becausep is irreducible, we haved =λp for some λ = 0 inK, and thus,p dividesg. As a consequence, ifp, q∈K[X] are both irreducible, then eitherp andq are relatively prime, orp =λq for someλ = 0 inK. In particular, if p, q∈K[X] are both irreducible monic polynomials andp =q, thenp andq are relatively prime.

We now prove the (unique) factorization of polynomials into irreducible factors. Theorem 20.16. Given any fieldK, for every nonzero polynomial

d +ad−1Xd−1 +· · · +a_0

f =adX

of degree d = deg(f)≥ 1 inK[X], there exists a unique set{p_1, k_1, . . . , pm, km} such that k1 pkmf =adp1· · · m,

where thepi∈K[X] are distinct irreducible monic polynomials, theki are (not necessarily distinct) integers, and m≥ 1,ki≥ 1.

Proof. First, we prove the existence of such a factorization by induction ond = deg(f). Clearly, it is enough to prove the result for monic polynomialsf of degreed = deg(f)≥ 1. Ifd = 1, thenf =X +a_0, which is an irreducible monic polynomial.

Assume d≥ 2, and assume the induction hypothesis for all monic polynomials of degree < d. Consider the setS of all monic polynomialsg such that deg(g)≥ 1 andg divides f. Sincef∈S, the setS is nonempty, and thus,S contains some monic polynomialp_1 of minimal degree. Since deg(p_1)≥ 1, the monic polynomialp_1 must be irreducible. Otherwise we would havep_1 =g_1g_2 , for some monic polynomialsg_1, g_2 such that deg(p_1)> deg(g_1)≥ 1 and deg(p_1)> deg(g_2)≥ 1, and sincep_1 dividef, theng_1 would dividef, contradicting the minimality of the degree ofp_1. Thus, we havef =p_1q, for some irreducible monic polynomialp_1, withq also monic. Since deg(p_1)≥ 1, we have deg(q)< deg(f), and we can apply the induction hypothesis toq. Thus, we obtain a factorization of the desired form.

We now prove uniqueness. Assume that

f

=

a

d

p

k1 pkm

1· · ·m, and f =adqh1 qh .1· · · n Thus, we have

adpk1 pkm =adqh1 qh .1· · · m 1· · · n

We prove that m =n,pi =qi andhi =ki, for alli, with 1≤i≤n.

The proof proceeds by induction onh_1 +· · · +hn.

Ifh_1 +· · ·+hn = 1, thenn = 1 andh_1 = 1. Then, sinceK[X] is an integral domain, we have

pk1 pkm =q_1,1· · · m

and sinceq_1 and thepi are irreducible monic, we must havem = 1 andp_1 =q_1. Ifh_1 +· · · +hn≥ 2, sinceK[X] is an integral domain and sinceh1≥ 1, we have

pk1 pkm =q_1q,1· · · m

with q =qh1−1 qh ,1 · · · n

where ( h1− 1) +· · · +hn≥ 1 (andqh1−1 = 1 ifh_1 = 1). Now, ifq_1 is not equal to any of_1

thepi, by a previous remark,q_1 andpi are relatively prime, and by Proposition 20.13,q_1

andpk1 pkm are relatively prime. But this contradicts the fact thatq_1 dividespk1 pkm .1 · · · m 1 · · · m Thus,q_1 is equal to one of thepi. Without loss of generality, we can assume thatq_1 =p_1. Then, sinceK[X] is an integral domain, we have

p k1−1 pkm =qh1−1 qh ,1 · · ·m 1 · · ·n wherepk1−1 = 1 ifk_1 = 1, andqh1−1 = 1 ifh_1 = 1. Now, (h1− 1) +· · ·+hn< h_1 +· · ·+hn,1 1

and we can apply the induction hypothesis to conclude thatm =n,pi =qi andhi =ki, with 1≤i≤n.

The above considerations about unique factorization into irreducible factors can be extended almost without changes to more general rings known as Euclidean domains. In such rings, some abstract version of the division theorem is assumed to hold.

Definition 20.9. A Euclidean domain (or Euclidean ring) is an integral domainA such that there exists a functionÕ:A→ N with the following property: For alla, b∈A with b = 0, there are someq, r∈A such that

a =bq +r and Õ(r)< Õ(b).

Note that the pair (q, r) is not necessarily unique.

Actually, unique factorization holds in principal ideal domains (PID’s), see Theorem 21.12. As shown below, every Euclidean domain is a PID, and thus, unique factorization holds for Euclidean domains.

Proposition 20.17. Every Euclidean domainA is a PID.

Proof. Let I be a nonnull ideal inA. Then, the set

{Õ(a)|a∈ I}

is nonempty, and thus, has a smallest elementm. Letb be any (nonnull) element of I such thatm =Õ(b). We claim that I = (b). Given anya∈ I, we can write

a =bq +r for someq, r∈A, withÕ(r)< Õ(b). Sinceb∈ I and I is an ideal, we also havebq∈ I, and sincea, bq∈ I and I is an ideal, thenr∈ I withÕ(r)< Õ(b) =m, contradicting the minimality ofm. Thus,r = 0 anda∈ (b). But then,

I⊆ (b),

and sinceb∈ I, we get I = (b), andA is a PID.

As a corollary of Proposition 20.17, the ring Z is a Euclidean domain (using the function Õ(a) =|a|) and thus, a PID. IfK is a field, the functionÕ onK[X] defined such that

Õ

(

f

) =

0 iff = 0, deg(f) + 1 iff = 0,

shows thatK[X] is a Euclidean domain.

Example 20.3. A more interesting example of a Euclidean domain is the ring Z[i] of Gaussian integers, i.e., the subring of C consisting of all complex numbers of the forma +ib, wherea, b∈ Z. Using the functionÕ defined such that

Õ(a +ib) =a^2 +b^2,

we leave it as an interesting exercise to prove that Z[i] is a Euclidean domain.

Not every PID is a Euclidean ring.

Remark: Given any integer,d∈ Z, such thatd = 0,1 andd does not have any square factor greater than one, the quadratic field, Q(√d), is the field consisting of all complex numbers of the forma +ib√−d ifd < 0, and of all the real numbers of the forma +b√d ifd > 0, witha, b∈ Q. The subring of Q(√d) consisting of all elements as above for whicha, b∈ Z is denoted by Z[√d]. We define the ring of integers of the field Q(√d) as the subring of Q(√d) consisting of the following elements:

(1) Ifd≡ 2 (mod 4) ord≡ 3 (mod 4), then all elements of the forma +ib√−d (ifd < 0) or all elements of the forma +b√d (ifd > 0), witha, b∈ Z;

(2) If d≡ 1 (mod 4), then all elements of the form (a+ib√−d)/2 (ifd < 0) or all elements of the form (a+b√d)/2 (ifd > 0), witha, b∈ Z and witha, b either both even or both odd.

Observe that whend≡ 2 (mod 4) ord≡ 3 (mod 4), the ring of integers of Q(√d) is equal to Z[√d]. For more on quadratic fields and their rings of integers, see Stark [96] (Chapter 8) or Niven, Zuckerman and Montgomery [83] (Chapter 9). It can be shown that the rings of integers, Z[√−d], whered = 19, 43, 67, 163, are PID’s, but not Euclidean rings.

Actually the rings of integers of Q(√d) that are Euclidean domains are completely determined but the proof is quite difficult. It turns out that there are twenty one such rings corresponding to the integers:−11,−7,−3,−2,−1, 2,3,5,6,7,11, 13,17,19,21, 29,33,37,41,57 and 73, see Stark [96] (Chapter 8).

It is possible to characterize a larger class of rings (in terms of ideals), factorial rings (or unique factorization domains), for which unique factorization holds (see Section 21.1). We now consider zeros (or roots) of polynomials.

20.6 Roots of Polynomials

We go back to the general case of an arbitrary ring for a little while.

Definition 20.10. Given a ringA and any polynomialf∈A[X], we say that someα∈A is a zero off, or a root off, iff(α) = 0. Similarly, given a polynomialf∈A[X_1, . . . , Xn], we say that (α_1, . . . , αn)∈An is a a zero off, or a root off, iff(α_1, . . . , αn) = 0.

Whenf∈A[X] is the null polynomial, everyα∈A is trivially a zero off. This case being trivial, we usually assume that we are considering zeros of nonnull polynomials. Example 20.4. Considering the polynomialf(X) =X^2 1, both +1 and−1 are zeros of f

(

X

). Over the field of reals, the polynomial

g

(

X

) =

X

2

−

+ 1 has no zeros. Over the field C of complex numbers,g(X) =X^2 + 1 has two rootsi and−i, the square roots of−1, which are “imaginary numbers.”

We have the following basic proposition showing the relationship between polynomial division and roots.

Proposition 20.18. Letf∈ A[X] be any polynomial andα∈A any element ofA. If the result of dividingf byX−α isf = (X−α)q +r, thenr = 0 ifff(α) = 0, i.e.,α is a root off iffr = 0.

Proof. We havef = (X−α)q +r, with deg(r)< 1 = deg(X−α). Thus,r is a constant in K, and sincef(α) = (α−α)q(α) +r, we getf(α) =r, and the proposition is trivial.

We now consider the issue of multiplicity of a root.

Proposition 20.19. Letf∈A[X] be any nonnull polynomial andh≥ 0 any integer. The following conditions are equivalent.

(1) f is divisible by (X−α)h but not by (X−α)h+1.

(2) There is someg∈A[X] such thatf = (X−α)hg andg(α) = 0.

Proof. Assume (1). Then, we havef = (X−α)hg for someg∈A[X]. If we hadg(α) = 0, by Proposition 20.18,g would be divisible by (X−α), and thenf would be divisible by (X−α)h+1, contradicting (1).

Assume (2), that is,f = (X−α)hg andg(α) = 0. Iff is divisible by (X−α)h+1, then we havef = (X−α)h+1g_1, for someg1∈A[X]. Then, we have

(X−α)hg = (X−α)h+1g_1,

and thus (X−α)h(g− (X−α)g_1) = 0,

and since the leading coefficient of ( X−α)h is 1 (show this by induction), by Proposition 20.1, (X−α)h is not a zero divisor, and we getg− (X−α)g_1 = 0, i.e.,g = (X−α)g_1, and sog(α) = 0, contrary to the hypothesis.

As a consequence of Proposition 20.19, for every nonnull polynomial f∈A[X] and every α∈A, there is a unique integerh≥ 0 such thatf is divisible by (X−α)h but not by (X−α)h+1. Indeed, sincef is divisible by (X−α)h, we haveh≤ deg(f). Whenh = 0,α is not a root off, i.e.,f(α) = 0. The interesting case is whenα is a root off.

Definition 20.11. Given a ringA and any nonnull polynomialf∈A[X], given anyα∈A, the uniqueh≥ 0 such thatf is divisible by (X−α)h but not by (X−α)h+1 is called the order, or multiplicity, of α. We haveh = 0 iffα is not a root off, and whenα is a root off, ifh = 1, we callα a simple root, ifh = 2, a double root, and generally, a root of multiplicity h≥ 2 is called a multiple root.

Observe that Proposition 20.19 (2) implies that if A⊆B, whereA andB are rings, for every nonnull polynomialf∈A[X], ifα∈A is a root off, then the multiplicity ofα with respect tof∈A[X] and the multiplicity ofα with respect tof considered as a polynomial inB[X], is the same.

We now show that if the ringA is an integral domain, the number of roots of a nonzero polynomial is at most its degree.

Proposition 20.20. Letf, g∈ A[X] be nonnull polynomials, letα∈A, and leth≥ 0 and k≥ 0 be the multiplicities ofα with respect tof andg. The following properties hold. (1) Ifl is the multiplicity ofα with respect to (f +g), thenl≥ min(h, k). Ifh =k, then l = min(h, k).

(2) If m is the multiplicity ofα with respect tofg, thenm≥h +k. IfA is an integral domain, thenm =h +k.

Proof. (1) We havef(X) = (X−α)hf_1(X),g(X) = (X−α)kg_1(X), withf_1(α) = 0 and g_1(α) = 0. Clearly,l≥ min(h, k). Ifh =k, assumeh < k. Then, we have

f(X) +g(X) = (Xhf_1(X) + (X−α)kg_1(X) = (X−α)h(f_1(X) + (X−α)k−hg_1(X)),−α)

and since (f_1(X) + (X−α)k−hg_1(X))(α) =f_1(α) = 0, we havel =h = min(h, k). (2) We have

f(X)g(X) = (X−α)h+kf_1(X)g_1(X),

withf_1(α) = 0 andg_1(α) = 0. Clearly, m≥ h +k. IfA is an integral domain, then f_1(α)g_1(α) = 0, and som =h +k.

Proposition 20.21. LetA be an integral domain. Letf be any nonnull polynomialf∈A[X] and letα_1, . . . , αm∈A bem≥ 1 distinct roots off of respective multiplicitiesk_1, . . . , km. Then, we have

f(X) = (X−α_1)k1 (X−αm)kmg(X),· · ·

whereg∈A[X] andg(αi) = 0 for alli, 1≤i≤m.

Proof. We proceed by induction onm. The casem = 1 is obvious in view of Definition 20.11 (which itself, is justified by Proposition 20.19). Ifm≥ 2, by the induction hypothesis, we have

f(X) = (X−α_1)k1 (X−αm−1)km−1g_1(X),· · ·

whereg1∈A[X] andg_1(αi) = 0, for 1≤i≤m− 1. SinceA is an integral domain and αi =αj fori =j, sinceαm is a root off, we have

0 = (αk1 (αm−αm−1)km−1g_1(αm),m−α_1)· · ·

which implies that g_1(αm) = 0. Now, by Proposition 20.20 (2), sinceαm is not a root of the polynomial (X−α_1)k1 (X−αm−1)km−1 and sinceA is an integral domain,αm must be a· · ·

root of multiplicitykm ofg_1, which means that

g_1(X) = (X−αm)kmg(X), withg(αm) = 0. Sinceg_1(αi) = 0 for 1≤i≤m− 1 andA is an integral domain, we must also haveg(αi) = 0, for 1≤i≤m− 1. Thus, we have

f(X) = (Xk1 (X−αm)kmg(X),−α_1)· · ·

whereg∈A[X], andg(αi) = 0 for 1≤i≤m.

As a consequence of Proposition 20.21, we get the following important result.

Theorem 20.22. LetA be an integral domain. For every nonnull polynomialf∈A[X], if the degree off isn = deg(f) andk_1, . . . , km are the multiplicities of all the distinct roots of f (wherem≥ 0), thenk_1 +· · · +km≤n.

Proof. Immediate from Proposition 20.21.

Since fields are integral domains, Theorem 20.22 holds for nonnull polynomials over fields and in particular, for R andC. An important consequence of Theorem 20.22 is the following.

Proposition 20.23. LetA be an integral domain. For any two polynomialsf, g∈A[X], if deg(f)≤n, deg(g)≤n, and if there aren + 1 distinct elementsα_1, α_2, . . . , αn+1∈A (with αi =αj fori =j) such thatf(αi) =g(αi) for alli, 1≤i≤n + 1, thenf =g.

Proof. Assumef =g, then, (f−g) is nonnull, and sincef(αi) =g(αi) for alli, 1≤i≤n+1, the polynomial (f−g) hasn + 1 distinct roots. Thus, (f−g) hasn + 1 distinct roots and is of degree at mostn, which contradicts Theorem 20.22.

Proposition 20.23 is often used to show that polynomials coincide. We will use it to show some interpolation formulae due to Lagrange and Hermite. But first, we characterize the multiplicity of a root of a polynomial. For this, we need the notion of derivative familiar in analysis. Actually, we can simply define this notion algebraically.

First, we need to rule out some undesirable behaviors. Given a fieldK, as we saw in Example 2.4, we can define a homomorphismχ: Z→K given by

χ(n) =n· 1,

where 1 is the multiplicative identity ofK. Recall that we definen·a by

n·a =a +· · · +a

n

ifn≥ 0 (with 0·a = 0) and n·a =−(−n)·a

ifn < 0. We say that the fieldK is of characteristic zero if the homomorphismχ is injective. Then, for anya∈K witha = 0, we haven·a = 0 for alln = 0

The fields Q, R, and C are of characteristic zero. In fact, it is easy to see that every field of characteristic zero contains a subfield isomorphic to Q. Thus, finite fields can’t be of characteristic zero.

Remark: If a field is not of characteristic zero, it is not hard to show that its characteristic, that is, the smallestn≥ 2 such thatn·1 = 0, is a prime numberp. The characteristicp ofK is the generator of the principal idealpZ, the kernel of the homomorphismχ: Z→K. Thus, every finite field is of characteristic some primep. Infinite fields of nonzero characteristic also exist.

Definition 20.12. LetA be a ring. The derivativef , or Df, or D^1f, of a polynomial f∈A[X] is defined inductively as follows:

f = 0, iff = 0, the null polynomial,

f = 0, iff =a, a = 0, a∈A,

f =nanXn−1 + (n− 1)an−1Xn−2 +· · · + 2a_2X +a_1,

iff =anXn +an−1Xn−1 +· · · +a_0, withn = deg(f)≥ 1.

If A =K is a field of characteristic zero, if deg(f)≥ 1, the leading coefficientnan off is nonzero, and thus,f is not the null polynomial. Thus, ifA =K is a field of characteristic zero, whenn = deg(f)≥ 1, we have deg(f ) =n− 1.

For rings or for fields of characteristicp≥ 2, we could havef = 0, for a polynomialf of degree≥ 1.

The following standard properties of derivatives are recalled without proof (prove them as an exercise).

Given any two polynomials,f, g∈A[X], we have

(f +g) =f +g , (fg) =f g +fg .

For example, iff = (X−α)kg andk≥ 1, we have

f =k(X−α)k−1g + (X−α)kg .

We can now give a criterion for the existence of simple roots. The first proposition holds for any ring.

Proposition 20.24. LetA be any ring. For every nonnull polynomialf∈A[X],α∈A is a simple root off iffα is a root off andα is not a root off .

Proof. Sinceα∈ A is a root off, we havef = (X−α)g for someg∈A[X]. Now,α is a simple root off iffg(α) = 0. However, we havef=g + (X−α)g , and sof (α) =g(α). Thus,α is a simple root off ifff (α) = 0.

We can improve the previous proposition as follows.

Proposition 20.25. LetA be any ring. For every nonnull polynomialf∈A[X], letα∈A be a root of multiplicityk≥ 1 off. Then,α is a root of multiplicity at least k− 1 off . If A is a field of characteristic zero, then α is a root of multiplicityk− 1 off . Proof. Sinceα∈A is a root of multiplicityk off, we havef = (X−α)kg for someg∈A[X] andg(α) = 0. Since

f =k(X−α)k−1g + (X−α)kg = (X−α)k−1(kg + (X−α)g ),

it is clear that the multiplicity of α w.r.t.f is at leastk−1. Now, (kg+(X−α)g )(α) =kg(α), and ifA is of characteristic zero, sinceg(α) = 0, thenkg(α) = 0. Thus,α is a root of multiplicityk− 1 off .

As a consequence, we obtain the following test for the existence of a root of multiplicity k for a polynomialf:

Given a field K of characteristic zero, for any nonnull polynomialf∈K[X], anyα∈K is a root of multiplicityk≥ 1 off iffα is a root off,D^1f,D^2f, . . . ,Dk−1f, but not a root of Dkf.

We can now return to polynomial functions and tie up some loose ends. Given a ring A, recall that every polynomialf∈A[X_1, . . . , Xn] induces a functionfA:An A defined such

that

f

A

(

α

1

, . . . , α

n

) =

f

(

α

1

, . . . , α

n

), for every (

α

1

, . . . , α

n

)

∈

A

n

→

. We now give a sufficient condition for the mappingf→fA to be injective.

Proposition 20.26. LetA be an integral domain. For every polynomialf∈A[X_1, . . . , Xn], ifA_1, . . . , An aren infinite subsets ofA such thatf(α_1, . . . , αn) = 0 for all (α_1, . . . , αn)∈A1×· · ·×An, thenf = 0, i.e.,f is the null polynomial. As a consequence, ifA is an infinite

integral domain, then the mapf→fA is injective.

Proof. We proceed by induction onn. Assumen = 1. Iff∈ A[X_1] is nonnull, letm = deg(f) be its degree. SinceA_1 is infinite andf(α_1) = 0 for allα1∈A_1, thenf has an infinite number of roots. But sincef is of degreem, this contradicts Theorem 20.22. Thus,f = 0.

Ifn≥ 2, we can viewf∈A[X_1, . . . , Xn] as a polynomial

f =gmXm +gm−1Xm−1 +· · · +g_0,n n

where the coefficients gi are polynomials inA[X_1, . . . , Xn−1]. Now, for every (α_1, . . . , αn−1)∈ A1× · · · ×An−1,f(α_1, . . . , αn−1, Xn) determines a polynomialh(Xn)∈A[Xn], and sinceAn is infinite andh(αn) =f(α_1, . . . , αn−1, αn) = 0 for allαn∈ An, by the induction hypothesis, we havegi(α_1, . . . , αn−1) = 0. Now, sinceA_1, . . . , An−1 are infinite, using the induction hypothesis again, we getgi = 0, which shows thatf is the null polynomial. The second part of the proposition follows immediately from the first, by lettingAi =A.

When A is an infinite integral domain, in particular an infinite field, since the mapf→fA is injective, we identify the polynomialf with the polynomial functionfA, and we write fA simply asf.

The following proposition can be very useful to show polynomial identities. Proposition 20.27. LetA be an infinite integral domain andf, g_1, . . . , gm∈A[X_1, . . . , Xn] be polynomials. If thegi are nonnull polynomials and if

f(α_1, . . . , αn) = 0 whenevergi(α_1, . . . , αn) = 0 for alli, 1≤i≤m, for every (α_1, . . . , αn)∈An, then

f = 0,

i.e.,f is the null polynomial.

Proof. Iff is not the null polynomial, since thegi are nonnull andA is an integral domain, thenfg1· · ·gm is nonnull. By Proposition 20.26, only the null polynomial maps to the zero function, and thus there must be some (α_1, . . . , αn)∈An, such that

f(α_1, . . . , αn)g_1(α_1, . . . , αn)· · ·gm(α_1, . . . , αn) = 0, but this contradicts the hypothesis.

Proposition 20.27 is often called the principle of extension of algebraic identities. Another perhaps more illuminating way of stating this proposition is as follows: For any polynomial g∈A[X_1, . . . , Xn], let

V (g) ={(α_1, . . . , αn)∈An g(α_1, . . . , αn) = 0},|

the set of zeros ofg. Note thatV (g_1)∪ · · · ∪V (gm) =V (g1· · ·gm). Then, Proposition 20.27 can be stated as:

Iff(α_1, . . . , αn) = 0 for every (α_1, . . . , αn)∈An V (g1· · ·gm), thenf = 0.−

In other words, if the algebraic identity f(α_1, . . . , αn) = 0 holds on the complement of V (g_1)∪ · · · ∪V (gm) =V (g1· · ·gm), thenf(α_1, . . . , αn) = 0 holds everywhere inAn. With this second formulation, we understand better the terminology “principle of extension of algebraic identities.”

Remark: LettingU(g) =A−V (g), the identityV (g_1)∪· · ·∪V (gm) =V (g1· · · gm) translates toU(g_1)∩ · · · ∩U(gm) =U(g1· · ·gm). This suggests to define a topology onA whose basis of open sets consists of the setsU(g).

sets of the formV (g) are closed sets.

In this topology (called the Zariski topology), the

Also, when g_1, . . . , gm∈A[X_1, . . . , Xn] andn≥ 2, understanding the structure of the closed sets of the formV (g1)∩· · ·∩V (gm) is quite difficult, and it is the object of algebraic geometry (at least, its classical part).

Whenf∈A[X_1, . . . , Xn] andn≥ 2, one should not apply Proposition 20.26 abusively. For example, let

f(X, Y ) =X^2 +Y 2 1,−

considered as a polynomial in R[X, Y ]. Since R is an infinite field and since 1

−

t

2 2t = (1−t^2)2 (2t)2

f 1 +t2, 1 +t^2 (1 +t^2)2 + (1 +t^2)2− 1 = 0, for everyt∈ R, it would be tempting to say thatf = 0. But what’s wrong with the above reasoning is that there are no two infinite subsetsR_1, R_2 of R such thatf(α_1, α_2) = 0 for all (α_1, α_2)∈ R^2. For everyα1∈ R, there are at most twoα2∈ R such thatf(α_1, α_2) = 0. What the example shows though, is that a nonnull polynomialf∈A[X_1, . . . , Xn] where n≥ 2 can have an infinite number of zeros. This is in contrast with nonnull polynomials in one variables over an infinite field (which have a number of roots bounded by their degree).

We now look at polynomial interpolation.

20.7 Polynomial Interpolation (Lagrange, Newton, Hermite)

Let K be a field. First, we consider the following interpolation problem: Given a sequence (α_1, . . . , αm+1) of pairwise distinct scalars inK and any sequence (β_1, . . . , βm+1) of scalars inK, where theβj are not necessarily distinct, find a polynomialP(X) of degree≤m such that

P(α_1) =β_1, . . . , P(αm+1) =βm+1.

First, observe that if such a polynomial exists, then it is unique. Indeed, this is a consequence of Proposition 20.23. Thus, we just have to find any polynomial of degree≤m. Consider the following so-called Lagrange polynomials:

L ) = (X−α_1)· · ·(X−αi−1)(X−αi+1)· · ·(X−αm+1).i(X

(αi−α_1)· · ·(αi−αi−1)(αi−αi+1)· · ·(αi−αm+1)

Note that L(αi) = 1 and thatL(αj) = 0, for allj =i. But then,

P(X) =β_1L_1 +· · · +βm+1Lm+1

is the unique desired polynomial, since clearly, P(αi) =βi. Such a polynomial is called a Lagrange interpolant. Also note that the polynomials (L_1, . . . , Lm+1) form a basis of the vector space of all polynomials of degree≤m. Indeed, if we had

λ_1L_1(X) +· · · +λm+1Lm+1(X) = 0,

setting X toαi, we would getλi = 0. Thus, theLi are linearly independent, and by the previous argument, they are a set of generators. We we call (L_1, . . . , Lm+1) the Lagrange basis (of orderm + 1).

It is known from numerical analysis that from a computational point of view, the Lagrange basis is not very good. Newton proposed another solution, the method of divided differences. Consider the polynomialP(X) of degree≤m, called the Newton interpolant, P(X) =λ_0 +λ_1(X−α_1) +λ_2(X−α_1)(X−α_2) +· · · +λm(X−α_1)(X−α_2)· · ·(X−αm).

Then, the λi can be determined by successively settingX to,α_1, α_2, . . . , αm+1. More precisely, we define inductively the polynomialsQ(X) andQ(α_1, . . . , αi, X), for 1≤i≤m, as follows:

Q(α

Q(α Q(X) =P(X)

Q(X)−Q(α_1)Q1(α1, X) = X−α1

Q(α_1, X)−Q(α_1, α_2)Q(α1, α2, X) = X−α2

. . .

Q(α_1, . . . , αi−1, X)−Q(α_1, . . . , αi−1, αi),1, . . . , αi, X) = X−αi

. . .

Q(α_1, . . . , αm−1, X)−Q(α_1, . . . , αm−1, αm).1, . . . , αm, X) = X−αm

By induction oni, 1≤i≤m− 1, it is easily verified that Q(X) =P(X),

Q(α_1, . . . , αi, X) =λi +λi+1(X−αi+1) +· · · +λm(X−αi+1)· · ·(X−αm), Q(α_1, . . . , αm, X) =λm.

From the above expressions, it is clear that λ_0 =Q(α_1),

λi =Q(α_1, . . . , αi, αi+1), λm =Q(α_1, . . . , αm, αm+1).

The expression Q(α_1, α_2, . . . , αi+1) is called thei-th difference quotient. Then, we can compute theλi in terms ofβ_1 =P(α_1), . . . , βm+1 =P(αm+1), using the inductive formulae for theQ(α_1, . . . , αi, X) given above, initializing theQ(αi) such thatQ(αi) =βi.

The above method is called the method of divided differences and it is due to Newton.

An astute observation may be used to optimize the computation. Observe that if Pi(X) is the polynomial of degree≤i taking the valuesβ_1, . . . , βi+1 at the pointsα_1, . . . , αi+1, then the coefficient ofXi inPi(X) isQ(α_1, α_2, . . . , αi+1), which is the value ofλi in the Newton interpolant

Pi(X) =λ_0 +λ_1(X−α_1) +λ_2(X−α_1)(X−α_2) +· · · +λi(X−α_1)(X−α_2)· · ·(X−αi).

As a consequence,Q(α_1, α_2, . . . , αi+1) does not depend on the specific ordering of theαj and there are better ways of computing it. For example,Q(α_1, α_2, . . . , αi+1) can be computed using Q(α1, . . . , αi+1) =Q(α_2, . . . , αi+1)−Q(α_1, . . . , αi).αi+1−α1

Then, the computation can be arranged into a triangular array reminiscent of Pascal’s triangle, as follows:

Initially,Q(αj) =βj, 1≤j≤m + 1, and

Q (α_1) Q(α_1, α_2)

Q(α_2) Q(α_1, α_2, α_3) Q(α_2, α_3) . . .

Q(α_3) Q(α_2, α_3, α_4) Q(α_3, α_4) . . .

Q(α_4) . . .

. . .

In this computation, each successive column is obtained by forming the difference quotients of the preceeding column according to the formula

Q(α Q(αk+1, . . . , αi+k)−Q(αk, . . . , αi+k−1).k, . . . , αi+k) = αi+k−αk

Theλi are the elements of the descending diagonal.

Observe that if we performed the above computation starting with a polynomial Q(X) of degreem, we could extend it by considering new given pointsαm+2,αm+3, etc. Then, from what we saw above, the (m+ 1)th column consists ofλm in the expression ofQ(X) as a Newton interpolant and the (m + 2)th column consists of zeros. Such divided differences are used in numerical analysis.

Newton’s method can be used to compute the value P(α) at someα of the interpolant P(X) taking the valuesβ_1, . . . , βm+1 for the (distinct) argumentsα_1, . . . , αm+1. We also mention that inductive methods for computingP(α) without first computing the coefficients of the Newton interpolant exist, for example, Aitken’s method. For this method, the reader is referred to Farin [34].

It has been observed that Lagrange interpolants oscillate quite badly as their degree increases, and thus, this makes them undesirable as a stable method for interpolation. A standard example due to Runge, is the function

f(x) =1 ,1 +x2

in the interval [−5, +5]. Assuming a uniform distribution of points on the curve in the interval [−5, +5], as the degree of the Lagrange interpolant increases, the interpolant shows wilder and wilder oscillations around the pointsx =−5 andx = +5. This phenomenon becomes quite noticeable beginning for degree 14, and gets worse and worse. For degree 22, things are quite bad! Equivalently, one may consider the function

f(x) =1 ,1 + 25x2

in the interval [−1, +1].

We now consider a more general interpolation problem which will lead to the Hermite polynomials.

We consider the following interpolation problem:

Given a sequence (α_1, . . . , αm+1) of pairwise distinct scalars inK, integersn_1, . . . , nm+1 wherenj≥ 0, andm + 1 sequences (β^0, . . . , βnj ) of scalars inK, lettingj

n =n_1 +· · · +nm+1 +m, find a polynomialP of degree≤n, such that

P(αm+1 D^1P(α

0, . . . P(αm+1) =β0 ,1) =β1 1, . . . D^1P(αm+1) =β^1m+1,1) =β_1

. . .

DiP(α_1) =βi, . . . DiP(αm+1) =βim+1,1

Dn1P(α_1 . . . nm+1.1) =βn , . . . Dnm+1P(αm+1) =βm+1

Note that the above equations constitute n+ 1 constraints, and thus, we can expect that there is a unique polynomial of degree≤n satisfying the above problem. This is indeed the case and such a polynomial is called a Hermite polynomial. We call the above problem the Hermite interpolation problem.

Proposition 20.28. The Hermite interpolation problem has a unique solution of degree≤n, wheren =n_1 +· · · +nm+1 +m.

Proof. First, we prove that the Hermite interpolation problem has at most one solution. Assume thatP andQ are two distinct solutions of degree≤n. Then, by Proposition 20.25 and the criterion following it,P−Q has among its rootsα_1 of multiplicity at leastn_1+1, . . ., αm+1 of multiplicity at leastnm+1 + 1. However, by Theorem 20.22, we should have

n_1 + 1 +· · · +nm+1 + 1 =n_1 +· · · +nm+1 +m + 1≤n,

which is a contradiction, since n =n_1 +· · · +nm+1 +m. Thus,P =Q. We are left with proving the existence of a Hermite interpolant. A quick way to do so is to use Proposition 5.13, which tells us that given a square matrixA over a fieldK, the following properties hold:

For every column vectorB, there is a unique column vectorX such thatAX =B iff the only solution toAX = 0 is the trivial vectorX = 0 iffD(A) = 0.

If we let P =y_0 +y_1X +· · · +ynXn, the Hermite interpolation problem yields a linear system of equations in the unknowns (y_0, . . . , yn) with some associated (n+1)×(n+1) matrix A. Now, the systemAY = 0 has a solution iffP has among its rootsα_1 of multiplicity at leastn_1 + 1, . . .,αm+1 of multiplicity at leastnm+1 + 1. By the previous argument, sinceP has degree≤n, we must haveP = 0, that is,Y = 0. This concludes the proof.

Proposition 20.28 shows the existence of unique polynomials Hij(X) of degree≤n such that DiHij(αj) = 1 and DkHij(αl) = 0, fork =i orl =j, 1≤j, l≤m + 1, 0≤i, k≤nj. The polynomialsHij are called Hermite basis polynomials.

One problem with Proposition 20.28 is that it does not give an explicit way of computing the Hermite basis polynomials. We first show that this can be done explicitly in the special casesn_1 =. . . =nm+1 = 1, andn_1 =. . . =nm+1 = 2, and then suggest a method using a generalized Newton interpolant.

Assume thatn_1 = . . . = nm+1 = 1. We tryH^0 = (a(X−αj) +b)L^2, andH^1 = (c(X−αj) +d)L^2, whereLj is the Lagrange interpolant determined earlier. Since

DH^0 =aL^2 + 2(a(X−αj) +b)LjDLj,

requiring that H^0(αj) = 1,H^0(αk) = 0, DH^0(αj) = 0, and DH^0(αk) = 0, fork =j, implies b = 1 anda =−2DLj(αj). Similarly, from the requirementsH^1(αj) = 0,H^1(αk) = 0, DH^1(αj) = 1, and DH^1(αk) = 0,k =j, we getc = 1 andd = 0.

Thus, we have the Hermite polynomials

H^0 = (1− 2DLj(αj)(X−αj))L^2, H^1 = (X−αj)L^2.

In the special case wherem = 1,α_1 = 0, andα_2 = 1, we leave as an exercise to show that the Hermite polynomials are

H^0 = 2X^3 3X^2 + 1,0 −

H^0 =−2X^3 + 3X^2,1

H^1 =X^3 2X^2 +X,0 −X2.1 =X3H1

−

As a consequence, the polynomial P of degree 3 such that P(0) = x_0, P(1) = x_1, P (0) =m_0, andP (1) =m_1, can be written as

P(X) =x_0(2X^3 3X^2 + 1) +m_0(X^3 2X^2 +X) +m_1(X^3 X^2) +x_1(−2X^3 + 3X^2).− − −

If we want the polynomialP of degree 3 such thatP(a) =x_0,P(b) =x_1,P (a) =m_0, andP (b) =m_1, whereb =a, then we have

P(X) =x^3 3t^2 + 1) + (b−a)m_0(t^3 2t^2 +t) + (b−a)m_1(t^3 t^2) +x_1(−2t^3 + 3t^2),0(2t− − −

whereX−a.

Observe the presence of the extra factor (

a

t = b−b−a) in front ofm_0 andm_1, the formula would be false otherwise!

We now consider the case wheren_1 =. . . =nm+1 = 2. Let us try

Hij(X) = (ai(X−αj)2 +bi(X−αj) +ci)L^3,

where 0≤i≤ 2. Sparing the readers some (tedious) computations, we find:

H^0(X) = 6(DL−

3 2Lj(αj) (X−αj)2 3DLj(αj)(X−αj) + 1 L^3(X),j(αj))2 2D−

H

1(X) = 9(DLj(αj))2(X−αj)2 3DLj(αj)(X−αj) L^3

−

H^2(X

(X),

) = 12(X−αj)2L^3(X).

Going back to the general problem, it seems to us that a kind of Newton interpolant will be more manageable. Let

P^0(X) = 1,0

P^0 (X) = (X−α_1)n1+1 (X−αj)nj+1, 1≤j≤mj

Pi · · · )n2+1 (X−αm+1)nm+1+1, 1≤i≤n_1,0(X) = (X−α_1)i(X−α_2

P· · ·nj+1(X−αj+1)i(X−αj+2)nj+2+1 (X−αm+1)nm+1+1,ij(X) = (X−α_1)n1+1 (X−αj)· · ·1≤j≤m− 1, · · ·

1 i≤nj+1,

Pim(X) = (X−α_1)n1+1 ≤(X−αm)nm+1(X−αm+1)i, 1≤i≤nm+1,· · ·

and let j=m,i=nj+1

P(X) = λijPij(X).

j=0,i=0

We can think of P(X) as a generalized Newton interpolant. We can compute the derivatives DkPij, for 1≤k≤nj+1, and if we look for the Hermite basis polynomialsHij(X) such that DiHij(αj) = 1 and DkHij(αl) = 0, fork =i orl =j, 1≤j, l≤m + 1, 0≤i, k≤nj, we find that we have to solve triangular systems of linear equations. Thus, as in the simple casen_1 =. . . =nm+1 = 0, we can solve successively for theλij. Obviously, the computations are quite formidable and we leave such considerations for further study.

Chapter 21 UFD’s, Noetherian Rings, Hilbert’s Basis Theorem

21.1 Unique Factorization Domains (Factorial Rings)

We saw in Section 20.5 that if K is a field, then every nonnull polynomial inK[X] can be factored as a product of irreducible factors, and that such a factorization is essentially unique. The same property holds for the ringK[X_1, . . . , Xn] wheren≥ 2, but a different proof is needed.

The reason why unique factorization holds for K[X_1, . . . , Xn] is that ifA is an integral domain for which unique factorization holds in some suitable sense, then the property of unique factorization lifts to the polynomial ringA[X]. Such rings are called factorial rings, or unique factorization domains. The first step if to define the notion of irreducible element in an integral domain, and then to define a factorial ring. If will turn out that in a factorial ring, any nonnull elementa is irreducible (or prime) iff the principal ideal (a) is a prime ideal.

Recall that given a ring A, a unit is any invertible element (w.r.t. multiplication). The set of units ofA is denoted byA∗. It is a multiplicative subgroup ofA, with identity 1. Also, givena, b∈A, recall thata dividesb ifb =ac for somec∈A; equivalently,a dividesb iff (b)⊆ (a). Any nonzeroa∈A is divisible by any unitu, sincea =u(u−1a). The relation “a dividesb,” often denoted bya|b, is reflexive and transitive, and thus, a preorder onA−{0}.

Definition 21.1. LetA be an integral domain. Some elementa∈A is irreducible ifa = 0, a /∈A∗ (a is not a unit), and whenevera =bc, then eitherb orc is a unit (whereb, c∈A). Equivalently,a∈A is reducible ifa = 0, ora∈A∗ (a is a unit), ora =bc whereb, c /∈A∗ (a, b are both noninvertible) andb, c = 0.

Observe that if a∈A is irreducible andu∈ A is a unit, thenua is also irreducible. Generally, ifa∈A,a = 0, andu is a unit, thena andua are said to be associated. This is the equivalence relation on nonnull elements ofA induced by the divisibility preorder.

565 The following simple proposition gives a sufficient condition for an elementa∈A to be irreducible.

Proposition 21.1. LetA be an integral domain. For anya∈A witha = 0, if the principal ideal (a) is a prime ideal, thena is irreducible.

Proof. If (a) is prime, then (a) =A anda is not a unit. Assume thata =bc. Then,bc∈ (a), and since (a) is prime, eitherb∈ (a) orc∈ (a). Consider the case whereb∈ (a), the other case being similar. Then,b =ax for somex∈A. As a consequence,

a =bc =axc,

and sinceA is an integral domain anda = 0, we get

1 =xc,

which proves thatc =x−1 is a unit.

It should be noted that the converse of Proposition 21.1 is generally false. However, it holds for factorial rings, defined next.

Definition 21.2. A factorial ring or unique factorization domain (UFD) (or unique factorization ring) is an integral domainA such that the following two properties hold: (1) For every nonnulla∈A, ifa /A∗ (a is not a unit), thena can be factored as a product

a =a1· · ·am

where eachai∈A is irreducible (m≥ 1).

(2) For every nonnulla∈A, ifa /∈A∗ (a is not a unit) and if

a =a1· · ·am =b1· · ·bn

whereai ∈A andbj∈A are irreducible, thenm =n and there is a permutationσ of {1, . . . , m} and some unitsu_1, . . . , um∈A∗ such thatai =uibσ(i) for alli, 1≤i≤m.

Example 21.1. The ring Z of integers if a typical example of a UFD. Given a fieldK, the polynomial ringK[X] is a UFD. More generally, we will show later that every PID is a UFD (see Theorem 21.12). Thus, in particular, Z[X] is a UFD. However, we leave as an exercise to prove that the ideal (2X, X^2) generated by 2X andX^2 is not principal, and thus, Z[X] is not a PID.

First, we prove that condition (2) in Definition 21.2 is equivalent to the usual “Euclidean” condition.

There are integral domains that are not UFD’s. For example, the subring Z[√−5] of C consisting of the complex numbers of the forma +bi√5 wherea, b∈ Z is not a UFD.

Indeed, we have

9 = 3· 3 = (2 +i√5)(2−i√5),

and it can be shown that 3, 2 +i√5, and 2−i√5 are irreducible, and that the units are±1. The uniqueness condition (2) fails and Z[√−5] is not a UFD.

Remark: Ford∈ Z withd < 0, it is known that the ring of integers of Q(√d) is a UFD iffd is one of the nine primes,d =−1,−2,−3,−7,−11,−19,−43,−67 and−163. This is a hard theorem that was conjectured by Gauss but not proved until 1966, independently by Stark and Baker. Heegner had published a proof of this result in 1952 but there was some doubt about its validity. After finding his proof, Stark reexamined Heegner’s proof and concluded that it was essentially correct after all. In sharp contrast, whend is a positive integer, the problem of determining which of the rings of integers of Q(√d) are UFD’s, is still open. It can also be shown that ifd < 0, then the ring Z[√d] is a UFD iffd =−1 ord =−2. If d≡ 1 (mod 4), then Z[√d] is never a UFD. For more details about these remarkable results, see Stark [96] (Chapter 8).

Proposition 21.2. LetA be an integral domain satisfying condition (1) in Definition 21.2. Then, condition (2) in Definition 21.2 is equivalent to the following condition: (2 ) Ifa∈A is irreducible anda divides the productbc, whereb, c∈A andb, c = 0, then eithera dividesb ora dividesc.

Proof. First, assume that (2) holds. Letbc =ad, whered∈A,d = 0. Ifb is a unit, then c =adb−1,

andc is divisible bya. A similar argument applies toc. Thus, we may assume thatb andc are not units. In view of (1), we can write

b =p1· · ·pm and c =pm+1· · ·qm+n, wherepi∈A is irreducible. Sincebc =ad,a is irreducible, andb, c are not units,d cannot be a unit. In view of (1), we can write

d =q1· · ·qr, whereqi∈A is irreducible. Thus, p1· · ·pmpm+1· · ·pm+n =aq1· · ·qr, where all the factors involved are irreducible. By (2), we must have a =ui0pi0 for some unitui0∈A and some indexi_0, 1≤ i0≤m+n. As a consequence, if 1≤i0≤m, thena dividesb, and ifm + 1≤i0≤m +n, thena dividesc. This proves that (2 ) holds. Let us now assume that (2 ) holds. Assume that

a =a1· · ·am =b1· · ·bn,

whereai∈A andbj∈A are irreducible. Without loss of generality, we may assume that m≤n. We proceed by induction onm. Ifm = 1,

a_1 =b1· · ·bn,

and since a_1 is irreducible,u =b1· · ·bi−1bi+1bn must be a unit for somei, 1≤i≤n. Thus, (2) holds withn = 1 anda_1 =biu. Assume thatm > 1 and that the induction hypothesis holds form− 1. Since

a_1a2· · ·am =b1· · ·bn,

a_1 dividesb1 · · ·bn, and in view of (2 ),a_1 divides somebj. Sincea_1 andbj are irreducible, we must havebj =uja_1, whereuj∈A is a unit. SinceA is an integral domain, a_1a2· · ·am =b1· · ·bj−1uja_1bj+1· · ·bn

implies that a2· · ·am = (ujb_1)· · ·bj−1bj+1· · ·bn,

and by the induction hypothesis, m− 1 =n− 1 andai =vibτ(i) for some unitsvi∈A and some bijectionτ between{2, . . . , m} and{1, . . . , j−1, j + 1, . . . , m}. However, the bijection τ extends to a permutationσ of{1, . . . , m} by lettingσ(1) =j, and the result holds by lettingv_1 =uj−1.

As a corollary of Proposition 21.2. we get the converse of Proposition 21.1. Proposition 21.3. LetA be a factorial ring. For anya∈A witha = 0, the principal ideal (a) is a prime ideal iffa is irreducible.

Proof. In view of Proposition 21.1, we just have to prove that ifa∈A is irreducible, then the principal ideal (a) is a prime ideal. Indeed, ifbc∈ (a), thena dividesbc, and by Proposition 21.2, property (2 ) implies that eithera dividesb ora dividesc, that is, eitherb∈ (a) or c∈ (a), which means that (a) is prime.

Because Proposition 21.3 holds, in a UFD, an irreducible element is often called a prime.

In a UFD A, every nonzero elementa∈A that is not a unit can be expressed as a producta =a1· · ·an of irreducible elementsai, and by property (2), the numbern of factors only depends ona, that is, it is the same for all factorizations into irreducible factors. We agree that this number is 0 for a unit.

Remark: IfA is a UFD, we can state the factorization properties so that they also applies to units:

(1) For every nonnulla∈A,a can be factored as a product

a =ua1· · ·am whereu∈A∗ (u is a unit) and eachai∈A is irreducible (m≥ 0). (2) For every nonnulla∈A, if a =ua1· · ·am =vb1· · ·bn

whereu, v∈A∗ (u, v are units) andai∈A andbj∈A are irreducible, thenm =n, and ifm =n = 0 thenu =v, else ifm≥ 1, then there is a permutationσ of{1, . . . , m}and some unitsu1, . . . , um∈A∗ such thatai =uibσ(i) for alli, 1≤i≤m.

We are now ready to prove that ifA is a UFD, then the polynomial ringA[X] is also a UFD.

The fact that nonnull and nonunit polynomials in A[X] factor as products of irreducible polynomials is rather easy to prove. First, observe that the units ofA[X] are just the units of A. Iff(X) is a polynomial of degree 0 that is not a unit, the fact thatA is a UFD yields the desired factorization off(X). Iff(X) has degreem > 0 andf(X) is reducible, thenf(X) factors as the product of two nonunit polynomialsg(X), h(X). Letfm be the coefficient of degreem inf. We have

f(X) =g(X)h(X),

and if both g(X) andh(X) have degree strictly less thanm, by induction, we get a factorization off(X) as a product of irreducible polynomials. Otherwise, eitherg(X) orh(X) is a constant. Consider the case whereg(X) is a constant, the other case being similar. Then, g(X) =b is not a unit, andb factors as a productb =b1· · ·bn of irreducible elementsbi, wheren only depends onb. Since

fm =bhm,

where hm be the coefficient of degreem inh, we see thathm is a product ofp of thebi’s, up to units, and thus,p < m. Again, we conclude by induction. More formally, we can proceed by induction on (m, n), wherem is the degree off(X) andn is the number of irreducible factors infm.

For the uniqueness of the factorization, by Proposition 21.2, it is enough to prove that condition (2 ) holds. This is a little more tricky. There are several proofs, but they all involve a pretty Lemma due to Gauss.

First, note the following trivial fact. Given a ring A, for anya∈A,a = 0, ifa divides every coefficient of some nonnull polynomialf(X)∈A[X], thena dividesf(X). IfA is an integral domain, we get the following converse.

Proposition 21.4. LetA be an integral domain. For anya∈A,a = 0, ifa divides a nonnull polynomialf(X)∈A[X], thena divides every coefficient off(X). Proof. Assume thatf(X) = ag(X), for someg(X)∈ A[X]. Sincea = 0 andA is an integral ring,f(X) andg(X) have the same degreem, and since for everyi (0≤i≤m) the coefficient ofXi inf(X) is equal to the coefficient ofXi inag(x), we havefi =agi, and wheneverfi = 0, we see thata dividesfi.

Lemma 21.5. (Gauss’s lemma) LetA be a UFD. For anya∈A, ifa is irreducible anda divides the productf(X)g(X) of two polynomialsf(X), g(X)∈A[X], then eithera divides f(X) ora dividesg(X).

Proof. Letf(X) =fmXm +· · · +fiXi +· · · +f_0 andg(X) =gnXn +· · · +gjXj +· · · +g_0. Assume thata divides neitherf(X) norg(X). By the (easy) converse of Proposition 21.4, there is somei (0≤i≤m) such thata does not dividefi, and there is somej (0≤j≤n) such thata does not dividegj. Picki andj minimal such thata does not dividefi anda does not dividegj. The coefficientci+j ofXi+j inf(X)g(X) is

ci+j =f_0gi+j +f_1gi+j−1 +· · · +figj +· · · +fi+jg_0

(letting fh = 0 ifh > m andgk = 0 ifk > n). From the choice ofi andj,a cannot divide figj, sincea being irreducible, by (2 ) of Proposition 21.2,a would dividefi orgj. However, by the choice ofi andj,a divides every other nonnull term in the sum forci+j, and sincea is irreducible and dividesf(X)g(X), by Proposition 21.4,a dividesci+j, which implies that a dividesfigj, a contradiction. Thus, eithera dividesf(X) ora dividesg(X).

As a corollary, we get the following proposition.

Proposition 21.6. Let A be a UFD. For any a∈ A, a = 0, if a divides the product f(X)g(X) of two polynomialsf(X), g(X)∈A[X] andf(X) is irreducible and of degree at least 1, thena dividesg(X).

Proof. The Proposition is trivial isa is a unit. Otherwise,a =a1· · ·am whereai∈ A is irreducible. Using induction and applying Lemma 21.5, we conclude thata dividesg(X).

We now show that Lemma 21.5 also applies to the case wherea is an irreducible polynomial. This requires a little excursion involving the fraction fieldF ofA.

Remark: IfA is a UFD, it is possible to prove the uniqueness condition (2) forA[X] directly without using the fraction field ofA, see Malliavin [72], Chapter 3.

Given an integral domain A, we can construct a fieldF such that every element ofF is of the forma/b, wherea, b∈A,b = 0, using essentially the method for constructing the field Q of rational numbers from the ring Z of integers.

Proposition 21.7. LetA be an integral domain.

(1) There is a fieldF and an injective ring homomorphismi:A→F such that every element ofF is of the formi(a)i(b)−1, wherea, b∈A,b = 0.

(2) For every field K and every injective ring homomorphism h:A→ K, there is a (unique) field homomorphismh:F→K such that

h(i(a)i(b)−1) =h(a)h(b)−1 for alla, b∈A,b = 0. (3) The fieldF in (1) is unique up to isomorphism.

Proof. (1) Consider the binary relation onA× (A− {0}) defined as follows: (a, b) (a , b ) iff ab =a b.

It is easily seen that is an equivalence relation. Note that the fact thatA is an integral domain is used to prove transitivity. The equivalence class of (a, b) is denoted bya/b. Clearly, (0, b) (0,1) for allb∈A, and we denote the class of (0,1) also by 0. The equivalence class a/1 of (a,1) is also denoted bya. We define addition and multiplication onA× (A− {0}) as follows:

(a, b) + (a , b ) = (ab +a b, bb ), (a, b)· (a , b ) = (aa , bb ).

It is easily verified that is congruential w.r.t. + and·, which means that + and· are well-defined on equivalence classes modulo . Whena, b = 0, the inverse ofa/b isb/a, and it is easily verified thatF is a field. The mapi:A→F defined such thati(a) =a/1 is an injection ofA intoF and clearlya =i(a)i(b)−1.b

(2) Given an injective ring homomorphismh:A→K into a fieldK,

a =a iff ab =a b,b b

which implies that

h(a)h(b ) =h(a )h(b),

and since h is injective andb, b = 0, we get

h(a)h(b)−1 =h(a )h(b )−1.

Thus, there is a maph:F→K such that

h(a/b) =h(i(a)i(b)−1) =h(a)h(b)−1

for alla, b∈A,b = 0, and it is easily checked thath is a field homomorphism. The maph is clearly unique.

(3) The uniqueness ofF up to isomorphism follows from (2), and is left as an exercise. The fieldF given by Proposition 21.7 is called the fraction field ofA, and it is denoted by Frac(A).

In particular, given an integral domain A, sinceA[X_1, . . . , Xn] is also an integral domain, we can form the fraction field of the polynomial ring A[X_1, . . . , Xn], denoted by F(X_1, . . . , Xn), whereF = Frac(A) is the fraction field ofA. It is also called the field of rational functions overF, although the terminology is a bit misleading, since elements of F(X_1, . . . , Xn) only define functions when the dominator is nonnull.

We now have the following crucial lemma which shows that if a polynomial f(X) is reducible overF[X] whereF is the fraction field ofA, thenf(X) is already reducible over A[X].

Lemma 21.8. Let A be a UFD and let F be the fraction field of A. For any nonnull polynomialf(X)∈ A[X] of degreem, iff(X) is not the product of two polynomials of degree strictly smaller than m, thenf(X) is irreducible inF[X].

Proof. Assume thatf(X) is reducible inF[X] and thatf(X) is neither null nor a unit. Then,

f(X) =G(X)H(X),

where G(X), H(X)∈F[X] are polynomials of degreep, q≥ 1. Leta be the product of the denominators of the coefficients ofG(X), andb the product of the denominators of the coefficients ofH(X). Then, a, b = 0, g_1(X) = aG(X)∈ A[X] has degreep≥ 1, h_1(X) =bH(X)∈A[X] has degreeq≥ 1, and

abf(X) =g_1(X)h_1(X). Letc =ab. Ifc is a unit, thenf(X) is also reducible inA[X]. Otherwise,c =c1· · ·cn, whereci∈A is irreducible. We now use induction onn to prove that

f(X) =g(X)h(X), for some polynomialsg(X)∈A[X] of degreep≥ 1 andh(X)∈A[X] of degreeq≥ 1.

If n = 1, sincec =c_1 is irreducible, by Lemma 21.5, eitherc dividesg_1(X) orc divides h_1(X). Say thatc dividesg_1(X), the other case being similar. Then,g_1(X) =cg(X) for someg(X)∈A[X] of degreep≥ 1, and sinceA[X] is an integral ring, we get

f(X) =g(X)h_1(X), showing thatf(X) is reducible inA[X]. Ifn > 1, since c1· · ·cnf(X) =g_1(X)h_1(X),

c_1 dividesg_1(X)h_1(X), and as above, eitherc_1 dividesg_1(X) orc dividesh_1(X). In either case, we get

for some polynomialsg_2(X)∈A[X] of degreep≥ 1 andh_2(X)∈A[X] of degreeq≥ 1. By the induction hypothesis, we get

c2· · ·cnf(X) =g_2(X)h_2(X)

f(X) =g(X)h(X), for some polynomialsg(X)∈A[X] of degreep≥ 1 andh(X)∈A[X] of degreeq≥ 1, showing thatf(X) is reducible inA[X].

Finally, we can prove that (2 ) holds.

Lemma 21.9. LetA be a UFD. Given any three nonnull polynomialsf(X), g(X), h(X)∈A[X], if f(X) is irreducible and f(X) divides the product g(X)h(X), then either f(X)

dividesg(X) orf(X) dividesh(X).

Proof. Iff(X) has degree 0, then the result follows from Lemma 21.5. Thus, we may assume that the degree off(X) ism≥ 1. LetF be the fraction field ofA. By Lemma 21.8,f(X) is also irreducible inF[X]. SinceF[X] is a UFD (by Theorem 20.16), eitherf(X) divides g(X) orf(X) dividesh(X), inF[X]. Assume thatf(X) dividesg(X), the other case being similar. Then,

g(X) =f(X)G(X),

for someG(X)∈F[X]. Ifa is the product the denominators of the coefficients ofG, we have

ag(X) =q_1(X)f(X),

whereq_1(X) =aG(X)∈ A[X]. Ifa is a unit, we see thatf(X) dividesg(X). Otherwise, a =a1· · ·an, whereai∈A is irreducible. We prove by induction onn that

g(X) =q(X)f(X) for someq(X)∈A[X].

Ifn = 1, sincef(X) is irreducible and of degreem≥ 1 and a_1g(X) =q_1(X)f(X), by Lemma 21.5,a_1 dividesq_1(X). Thus,q_1(X) =a_1q(X) whereq(X)∈A[X]. SinceA[X] is an integral domain, we get

g(X) =q(X)f(X),

andf(X) dividesg(X). Ifn > 1, from

a1· · ·ang(X) =q_1(X)f(X),

we note thata_1 dividesq_1(X)f(X), and as in the previous case,a_1 dividesq_1(X). Thus, q_1(X) =a_1q_2(X) whereq_2(X)∈A[X], and we get

a2· · ·ang(X) =q_2(X)f(X). By the induction hypothesis, we get

g(X) =q(X)f(X)

for someq(X)∈A[X], andf(X) dividesg(X).

We finally obtain the fact that A[X] is a UFD whenA is.

Theorem 21.10. IfA is a UFD then the polynomial ringA[X] is also a UFD.

Proof. As we said earlier, the factorization property (1) is easy to prove. Assume thatf(X) has degreem and that its coefficientfm of degreem is the product ofn irreducible elements (wheren = 0 iffm is a unit). We proceed by induction on the pair (m, n), using the well-founded ordering on pairs, i.e.,

(m, n)≤ (m , n )

iff either m < m , orm =m andn < n . Iff(X) is a nonnull polynomial of degree 0 which is not a unit, thenf(X)∈A, andf(X) =fm =a1· · ·an for some irreducibleai∈A, since A is a UFD. Iff(X) has degreem > 0 and is reducible, then

f(X) =g(X)h(X),

where g(X) andh(X) have degreep, q≤m and are not units. Ifp, q < m, then (p, n_1)< (m, n) and (q, n_2)< (m, n), wheren_1 is the number of irreducible factors ingp andn_2 is the number of irreducible factors inhq, and by the induction hypothesis, bothg(X) andh(X) can be written as products of irreducible factors. Ifp = 0, theng(X) =g_0 is not a unit, and since

fm =g_0hm,

hm is a product ofn_2 irreducible elements wheren_2< n. Since (m, n_2)< (m, n), by the induction hypothesis,h(X) can be written as products of irreducible polynomials. Since g0∈A is not a unit,g_0 can also be factored as a product of irreducible elements. The case whereq = 0 is similar.

Property (2 ) follows by Lemma 21.9. By Proposition 21.2,A[X] is a UFD.

As a corollary of Theorem 21.10 and using induction, we note that for any fieldK, the polynomial ringK[X_1, . . . , Xn] is a UFD.

For the sake of completeness, we shall prove that every PID is a UFD. First, we review the notion of gcd and the characterization of gcd’s in a PID.

Given an integral domainA, for any two elementsa, b∈A,a, b = 0, we say thatd∈A (d = 0) is a greatest common divisor (gcd) ofa andb if

(1) d divides botha andb. (2) For anyh∈A (h = 0), ifh divides botha andb, thenh dividesd.

We also say thata andb are relatively prime if 1 is a gcd ofa andb.

Note thata andb are relatively prime iff every gcd ofa andb is a unit. IfA is a PID, then gcd’s are characterized as follows.

Proposition 21.11. LetA be a PID.

(1) For anya, b, d∈A (a, b, d = 0),d is a gcd ofa andb iff

(d) = (a, b) = (a) + (b),

i.e.,d generates the principal ideal generated bya andb. (2) (Bezout identity) Two nonnull elementsa, b∈A are relatively prime iff there are some x, y∈A such that

ax +by = 1.

Proof. (1) Recall that the ideal generated bya andb is the set

(a) + (b) =aA +bA ={ax +by|x, y∈A}.

First, assume thatd is a gcd ofa andb. If so,a∈Ad,b∈Ad, and thus, (a)⊆ (d) and (b)⊆ (d), so that

(a) + (b)⊆ (d). SinceA is a PID, there is somet∈A,t = 0, such that

(a) + (b) = (t),

and thus, (a)⊆ (t) and (b)⊆ (t), which means thatt divides botha andb. Sinced is a gcd ofa andb,t must divided. But then,

(d)⊆ (t) = (a) + (b),

and thus, (d) = (a) + (b). Assume now that (d) = (a) + (b) = (a, b).

Since (a)⊆ (d) and (b)⊆ (d),d divides botha andb. Assume thatt divides botha andb, so that (a)⊆ (t) and (b)⊆ (t). Then,

(d) = (a) + (b)⊆ (t),

which means thatt dividesd, andd is indeed a gcd ofa andb. (2) By (1), ifa andb are relatively prime, then

(1) = (a) + (b),

which yields the result. Conversely, if

ax +by = 1,

then (1) = (a) + (b), and 1 is a gcd ofa andb.

Given two nonnull elements a, b∈A, ifa is an irreducible element anda does not divide b, thena andb are relatively prime. Indeed, ifd is not a unit andd divides botha andb, thena =dp andb =dq wherep must be a unit, so that

b =ap−1q,

anda dividesb, a contradiction. Theorem 21.12. LetA be ring. IfA is a PID, thenA is a UFD.

Proof. First, we prove that every any nonnull element that is a not a unit can be factored as a product of irreducible elements. LetS be the set of nontrivial principal ideals (a) such thata = 0 is not a unit and cannot be factored as a product of irreducible elements. Assume thatS is nonempty. We claim that every ascending chain inS is finite. Otherwise, consider an infinite ascending chain

(a1)⊂ (a2)⊂ · · · ⊂ (an)⊂ · · ·. It is immediately verified that (an)

n≥1

is an ideal inA. SinceA is a PID, (an) = (a)

n≥1

for somea∈A. However, there must be somen such thata∈ (an), and thus,

(an)⊆ (a)⊆ (an),

and the chain stabilizes at (an). As a consequence, for any ideal (d) such that

( an)⊂ (d) and (an) = (d),d has the desired factorization. Observe thatan is not irreducible, since (an)∈ S, and thus,

an =bc for someb, c∈A, where neitherb norc is a unit. Then,

(an)⊆ (b) and (an)⊆ (c).

If (an) = (b), thenb =anu for someu∈A, and then

an =anuc,

so that 1 =uc,

since A is an integral domain, and thus,c is a unit, a contradiction. Thus, (an) = (b), and similarly, (an) = (c). But then, bothb andc factor as products of irreducible elements and so doesan =bc, a contradiction. This implies thatS =∅.

To prove the uniqueness of factorizations, we use Proposition 21.2. Assume thata is irreducible and thata dividesbc. Ifa does not divideb, by a previous remark,a andb are relatively prime, and by Proposition 21.11, there are somex, y∈A such that

ax +by = 1.

Thus, acx +bcy =c, and sincea dividesbc, we see thata must dividec, as desired.

Thus, we get another justification of the fact that Z is a UFD and that ifK is a field, thenK[X] is a UFD.

It should also be noted that in a UFD, gcd’s of nonnull elements always exist. Indeed, this is trivial ifa orb is a unit, and otherwise, we can write

a =p1· · ·pm and b =q1· · ·qn

wherepi, qj∈A are irreducible, and the product of the common factors ofa andb is a gcd ofa andb (it is 1 is there are no common factors).

We conclude this section on UFD’s by proving a proposition characterizing when a UFD is a PID. The proof is nontrivial and makes use of Zorn’s lemma (several times).

Proposition 21.13. LetA be a ring that is a UFD, and not a field. Then,A is a PID iff every nonzero prime ideal is maximal.

Proof. Assume thatA is a PID that is not a field. Consider any nonzero prime ideal, (p), and pick any proper ideal A inA such that

(p)⊆ A.

SinceA is a PID, the ideal A is a principal ideal, so A = (q), and since A is a proper nonzero ideal,q = 0 andq is not a unit. Since

(p)⊆ (q),

q dividesp, and we havep =qp_1 for somep1∈A. Now, by Proposition 21.1, sincep = 0

and (p) is a prime ideal,p is irreducible. But then, sincep =qp_1 andp is irreducible,p_1

must be a unit (sinceq is not a unit), which implies that

(p) = (q);

that is, (p) is a maximal ideal.

Conversely, let us assume that every nonzero prime ideal is maximal. First, we prove that every prime ideal is principal. This is obvious for (0). If A is a nonzero prime ideal, then, by hypothesis, it is maximal. Since A = (0), there is some nonzero elementa∈ A. Since A is maximal,a is not a unit, and sinceA is a UFD, there is a factorizationa =a1· · ·an ofa into irreducible elements. Since A is prime, we haveai∈ A for somei. Now, by Proposition 21.3, sinceai is irreducible, the ideal (ai) is prime, and so, by hypothesis, (ai) is maximal. Since (ai)⊆ A and (ai) is maximal, we get A = (ai).

Next, assume that A is not a PID. Define the set,F, by

={A| A⊆A, A is not a principal ideal}.F

Since A is not a PID, the setF is nonempty. Also, the reader will easily check that every chain inF is bounded. Then, by Zorn’s lemma (Lemma 31.1), the setF has some maximal element, A. Clearly, A = (0) is a proper ideal (sinceA = (1)), and A is not prime, since we just showed that prime ideals are principal. Then, by Theorem 31.3, there is some maximal ideal, M, so that A⊂ M. However, a maximal ideal is prime, and we have shown that a prime ideal is principal. Thus,

A⊆ (p),

for somep∈A that is not a unit. Moreover, by Proposition 21.1, the elementp is irreducible. Define

B ={a∈A|pa∈ A}.

Clearly, A =pB, B = (0), A⊆ B, and B is a proper ideal. We claim that A = B. Indeed, if A = B were true, then we would have A =pB = B, but this is impossible sincep is irreducible,A is a UFD, and B = (0) (we get B =pmB for allm, and every element of B would be a multiple ofpm for arbitrarily largem, contradicting the fact thatA is a UFD). Thus, we have A⊂ B, and since A is a maximal element ofF, we must have B /∈ F. However, B /∈ F means that B is a principal ideal, and thus, A =pB is also a principal ideal, a contradiction.

Observe that the above proof shows that Proposition 21.13 also holds under the assumption that every prime ideal is principal.

21.2 The Chinese Remainder Theorem

In this section, which is a bit of an interlude, we prove a basic result about quotients of commutative rings by products of ideals that are pairwise relatively prime. This result has applications in number theory and in the structure theorem for finitely generated modules over a PID, which will be presented later.

Given two ideals a and b of a ringA, we define the ideal ab as the set of all finite sums of the form

a_1b_1 +· · · +akbk, ai∈ a, bi∈ b.

The reader should check that ab is indeed an ideal. Observe that ab⊆ a and ab⊆ b, so that

ab⊆ a∩b.

In general, equality does not hold. However if

a +b =A,

then we have ab = a∩b. This is because there is somea∈ a and someb∈ b such that

a +b = 1,

so for everyx∈ a∩b, we have x =xa +xb,

which shows thatx∈ ab. Ideals a and b ofA that satisfy the condition a + b =A are sometimes said to be comaximal.

We define the homomorphismÕ:A→A/a×A/b by

Õ(x) = (xa, xb),

wherexa is the equivalence class ofx modulo a (resp.xb is the equivalence class ofx modulo b). Recall that the ideal a defines the equivalence relation≡a onA given by

x≡ay iff x−y∈ a,

and thatA/a is the quotient ring of equivalence classesxa, wherex∈A, and similarly for A/b. Sometimes, we also writex≡y (mod a) forx≡ay.

Clearly, the kernel of the homomorphism Õ is a∩ b. If we assume that a + b =A, then Ker (Õ) = a∩b = ab, and becauseÕ has a constant value on the equivalence classes modulo ab, the mapÕ induces a quotient homomorphism

θ:A/ab→A/a×A/b.

Because Ker (Õ) = ab, the homomorphismθ is injective. The Chinese Remainder Theorem says thatθ is an isomorphism.

Theorem 21.14. Given a commutative ringA, let a and b be any two ideals ofA such that a +b =A. Then, the homomorphismθ:A/ab→A/a×A/b is an isomorphism. Proof. We already showed thatθ is injective, so we need to prove thatθ is surjective. We need to prove that for anyy, z∈A, there is somex∈A such that

x≡y (mod a) x≡z (mod b).

Since a +b =A, there exist somea∈ a and someb∈ b such that

a +b = 1.

If we let x =az +by, then we have

x≡aby≡a (1−a)y≡ay−ay≡ay, and similarly

x≡baz≡b (1−b)z≡bz−bz≡bz, which shows thatx =az +by works.

Theorem 21.14 can be generalized to any (finite) number of ideals.

Theorem 21.15. (Chinese Remainder Theorem) Given a commutative ringA, let a_1, . . . ,an be anyn≥ 2 ideals ofA such that ai + aj =A for alli =j. Then, the homomorphism θ:A/a1· · ·an→A/a1× · · · ×A/an is an isomorphism.

Proof. The mapθ:A/a1∩ · · · ∩ an →A/a1× · · · ×A/an is induced by the homomorphism Õ:A→A/a1× · · · ×A/an given by

Õ(x) = (xa1, . . . , xan).

Clearly, Ker (Õ) = a1∩ · · · ∩an, soθ is well-defined and injective. We need to prove that

a 1∩ · · · ∩an = a1· · ·an and thatθ is surjective. We proceed by induction. The casen = 2 is Theorem 21.14. By induction, assume that

a2∩ · · · ∩an = a2· · ·an. We claim that

a_1 +a2· · ·an =A.

Indeed, since a_1 + ai =A fori = 2, . . . , n, there exist someai∈ a_1 and somebi∈ ai such that

ai +bi = 1, i = 2, . . . , n,

and by multiplying these equations, we get

a +b2· · ·bn = 1,

wherea is a sum of terms each containing someaj as a factor, soa∈ a_1 andb2· · ·bn∈a2· · ·an, which shows that

a_1 +a2· · ·an =A, as claimed. It follows that

a 1∩a2∩ · · · ∩an = a1∩ (a2· · ·an) = a1a2· · ·an. Let us now prove thatθ is surjective by induction. The casen = 2 is Theorem 21.14. Let x_1, . . . , xn be anyn≥ 3 elements ofA. First, applying Theorem 21.14 to a_1 and a2· · ·an, we can findy1∈A such that

y1≡ 1 (mod a_1)

y1≡ 0 (mod a2· · ·an). By the induction hypothesis, we can findy_2, . . . , yn∈A such that for alli, j with 2≤i, j≤n, yi≡ 1 (mod ai)

yi≡ 0 (mod aj), j =i. We claim that x =x_1y_1 +x_2y_2 +· · · +xnyn

works. Indeed, using the above congruences, for i = 2, . . . , n, we get x≡x_1y_1 +xi (mod ai), (∗)

but since a2· · ·an⊆ ai fori = 2, . . . , n andy1≡ 0 (mod a2· · ·an), we have x ai), i = 2, . . . , n_1y1≡ 0 (mod

and equation (∗) reduces to x≡xi (mod ai), i = 2, . . . , n.

Fori = 1, we get x≡x_1 (mod a_1), therefore

x≡xi (mod ai), i = 1, . . . , n. proving surjectivity.

The classical version of the Chinese Remainder Theorem is the case where A = Z and where the ideals ai are defined byn pairwise relatively prime integersm_1, . . . , mn. By the Bezout identity, sincemi andmj are relatively prime wheneveri = j, there exist some ui, uj∈ Z such thatuimi +ujmj = 1, and somiZ +mjZ = Z. In this case, we get an isomorphismn

Z/(m1· · ·mn)Z≈ Z/miZ.

i=1

In particular, ifm is an integer greater than 1 and

m

=

p

ri i i

is its factorization into prime factors, then

Z

/m

Z

≈

i

Z/pri Z.i

In the previous situation where the integers m_1, . . . , mn are pairwise relatively prime, if we writem =m1· · ·mn andmi =m/mi fori = 1. . . , n, thenmi andmi are relatively prime, and somi has an inverse modulomi. Ifti is such an inverse, so that

miti≡ 1 (modmi),

then it is not hard to show that for anya_1, . . . , an∈ Z,

x =a_1t_1m_1 +· · · +antnmn

satisfies the congruences x≡ai (modmi), i = 1, . . . , n.

Theorem 21.15 can be used to characterize rings isomorphic to finite products of quotient rings. Such rings play a role in the structure theorem for torsion modules over a PID. Givenn ringsA_1, . . . , An, recall that the product ringA =A1× · · · ×An is the ring in which addition and multiplication are defined componenwise. That is,

( a_1, . . . , an) + (b_1, . . . , bn) = (a_1 +b_1, . . . , an +bn) (a_1, . . . , an)· (b_1, . . . , bn) = (a_1b_1, . . . , anbn). The additive identity is 0A = (0, . . . ,0) and the multiplicative identity is 1A = (1, . . . ,1). Then, fori = 1, . . . , n, we can define the elementei∈A as follows:

ei = (0, . . . ,0,1,0, . . . ,0),

where the 1 occurs in positioni. Observe that the following properties hold for alli, j = 1, . . . , n:

e^2 =ei

eiej = 0, i =j e_1 +· · · +en = 1A.

Also, for any elementa = (a_1, . . . , an)∈A, we have

eia = (0, . . . ,0, ai,0, . . . ,1) =pri(a),

wherepri is the projection ofA ontoAi. As a consequence

Ker (pri) = (1A−ei)A.

Definition 21.3. Given a commutative ringA, a direct decomposition ofA is a sequence (b_1, . . . ,bn) of ideals inA such that there is an isomorphismA≈A/b1× · · · ×A/bn.

The following theorem gives useful conditions characterizing direct decompositions of a ring.

Theorem 21.16. LetA be a commutative ring and let (b_1, . . . ,bn) be a sequence of ideals inA. The following conditions are equivalent:

(a) The sequence (b_1, . . . ,bn) is a direct decomposition ofA.

(b) There exist some elementse_1, . . . , en ofA such that

e^2 =ei

eiej = 0, i =j e_1 +· · · +en = 1A,

and bi = (1A−ei)A, fori, j = 1, . . . , n.

(c) We have bi +bj =A for alli =j, and b1· · ·bn = (0).

(d) We have bi +bj =A for alli =j, and b1∩ · · · ∩bn = (0). Proof. Assume (a). Since we have an isomorphismA≈A/b1× · · · × A/bn, we may identify A withA/b1× · · · ×A/bn, and bi with Ker (pri). Then,e1, . . . , en are the elements defined just before Definition 21.3. As noted, bi = Ker (pri) = (1A−ei)A. This proves (b).

Assume (b). Since bi = (1A−ei)A andA is a ring with unit 1A, we have 1A− ei∈ bi fori = 1, . . . , n. For alli =j, we also haveei(1A−ej) =ei−eiej =ei, so (because bj is an ideal),ei∈ bj, and thus, 1A = 1A−ei +ei∈ bi + bj, which shows that bi + bj =A for all i =j. Furthermore, for anyxi∈A, with 1≤i≤n, we have

n n n

xi(1A−ei) = xi (1A−ei)

i=1 i=1 i=1

n n

= ei)

i=1

xi (1A− i=1

= 0,

which proves that b1· · ·bn = (0). Thus, (c) holds.

The equivalence of (c) and (d) follows from the proof of Theorem 21.15. The fact that (c) implies (a) is an immediate consequence of Theorem 21.15.

21.3 Noetherian Rings and Hilbert’s Basis Theorem

Given a (commutative) ringA (with unit element 1), an ideal A⊆A is said to be finitely generated if there exists a finite set{a_1, . . . , an} of elements from A so that

A = (a_1, . . . , an) ={λ_1a_1 +· · · +λnan|λi∈A, 1≤i≤n}.

If K is a field, it turns out that every polynomial ideal A in K[X_1, . . . , Xm] is finitely generated. This fact due to Hilbert and known as Hilbert’s basis theorem, has very important consequences. For example, in algebraic geometry, one is interested in the zero locus of a set of polyomial equations, i.e., the set,V (P), ofn-tuples (λ_1, . . . , λn)∈Kn so that

Pi(λ_1, . . . , λn) = 0 for all polynomialsPi(X_1, . . . , Xn) in some given family,P = (Pi)i∈I. However, it is clear that

V (P) =V (A),

where A is the ideal generated byP. Then, Hilbert’s basis theorem says thatV (A) is actually defined by a finite number of polynomials (any set of generators of A), even ifP is infinite.

The property that every ideal in a ring is finitely generated is equivalent to other natural properties, one of which is the so-called ascending chain condition, abbreviated a.c.c. Before proving Hilbert’s basis theorem, we explore the equivalence of these conditions. Definition 21.4. LetA be a commutative ring with unit 1. We say thatA satisfies the ascending chain condition, for short, the a.c.c, if for every ascending chain of ideals

A1⊆ A2⊆ · · · ⊆ Ai⊆ · · ·, there is some integern≥ 1 so that

Ai = An for all i≥n + 1.

We say that A satisfies the maximum condition if every nonempty collectionC of ideals in A has a maximal element, i.e., there is some ideal A∈C which is not contained in any other ideal inC.

Proposition 21.17. A ringA satisfies the a.c.c if and only if it satisfies the maximum condition.

Proof. Suppose thatA does not satisfy the a.c.c. Then, there is an infinite strictly ascending sequence of ideals

A1⊂ A2⊂ · · · ⊂ Ai⊂ · · ·,

and the collectionC ={Ai} has no maximal element.

Conversely, assume that A satisfies the a.c.c. LetC be a nonempty collection of ideals SinceC is nonempty, we may pick some ideal A_1 inC. If A_1 is not maximal, then there is some ideal A_2 inC so that

A1⊂ A_2.

Using this process, ifC has no maximal element, we can define by induction an infinite strictly increasing sequence

A 1⊂ A2⊂ · · · ⊂ Ai⊂ · · ·.

However, the a.c.c. implies that such a sequence cannot exist. Therefore,C has a maximal element.

Having shown that the a.c.c. condition is equivalent to the maximal condition, we now prove that the a.c.c. condition is equivalent to the fact that every ideal is finitely generated. Proposition 21.18. A ringA satisfies the a.c.c if and only if every ideal is finitely generated. Proof. Assume that every ideal is finitely generated. Consider an ascending sequence of ideals

A 1⊆ A2⊆ · · · ⊆ Ai⊆ · · ·.

Observe that A =iAi is also an ideal. By hypothesis, A has a finite generating set {a_1, . . . , an}. By definition of A, eachai belongs to some Aji, and since the Ai form an ascending chain, there is somem so thatai∈ Am fori = 1, . . . , n. But then,

Ai = Am for alli≥m + 1, and the a.c.c. holds.

Conversely, assume that the a.c.c. holds. Let A be any ideal inA and consider the family C of subideals of A that are finitely generated. The familyC is nonempty, since (0) is a subideal of A. By Proposition 21.17, the familyC has some maximal element, say B. For anya∈ A, the ideal B + (a) (where B + (a) ={b +λa|b∈ B, λ∈A}) is also finitely generated (since B is finitely generated), and by maximality, we have

B = B + (a).

So, we geta∈ B for alla∈ A, and thus, A = B, and A is finitely generated. Definition 21.5. A commutative ringA (with unit 1) is called noetherian if it satisfies the a.c.c. condition. A noetherian domain is a noetherian ring that is also a domain.

By Proposition 21.17 and Proposition 21.18, a noetherian ring can also be defined as a ring that either satisfies the maximal property or such that every ideal is finitely generated. The proof of Hilbert’s basis theorem will make use the following lemma:

Lemma 21.19. LetA be a (commutative) ring. For every ideal A inA[X], for everyi≥ 0, letLi(A) denote the set of elements ofA consisting of 0 and of the coefficients ofXi in all the polynomialsf(X)∈ A which are of degreei. Then, theLi(A)’s form an ascending chain of ideals inA. Furthermore, if B is any ideal ofA[X] so that A⊆ B and ifLi(A) =Li(B) for alli≥ 0, then A = B.

Proof. ThatLi(A) is an ideal and thatLi(A)⊆Li+1(A) follows from the fact that iff(X)∈ A andg(X)∈ A, thenf(X) +g(X),λf(X), andXf(X) all belong to A. Now, letg(X) be any polynomial in B, and assume thatg(X) has degreen. SinceLn(A) =Ln(B), there is some polynomialfn(X) in A, of degreen, so thatg(X)−fn(X) is of degree at mostn− 1. Now, since A⊆ B, the polynomialg(X)−fn(X) belongs to B. Using this process, we can define by induction a sequence of polynomialsfn+i(X)∈ A, so that eachfn+i(X) is either zero or has degreen−i, and

g(X)− (fn(X) +fn+1(X) +· · · +fn+i(X))

is of degree at mostn−i−1. Note that this last polynomial must be zero wheni =n, and thus,g(X)∈ A.

We now prove Hilbert’s basis theorem. The proof is substantially Hilbert’s original proof. A slightly shorter proof can be given but it is not as transparent as Hilbert’s proof (see the remark just after the proof of Theorem 21.20, and Zariski and Samuel [112], Chapter IV, Section 1, Theorem 1).

Theorem 21.20. (Hilbert’s basis theorem) IfA is a noetherian ring, thenA[X] is also a noetherian ring.

Proof. Let A be any ideal inA[X], and denote byL the set of elements ofA consisting of 0 and of all the coefficients of the highest degree terms of all the polynomials in A. Observe that

L

= Li(A).

i

Thus,L is an ideal inA (this can also be proved directly). SinceA is noetherian,L is finitely generated, and let{a_1, . . . , an} be a set of generators ofL. Letf_1(X), . . . , fn(X) be polynomials in A having respectivelya_1, . . . , an as highest degree term coefficients. These polynomials generate an ideal B. Letq be the maximum of the degrees of thefi(X)’s. Now, pick any polynomialg(X)∈ A of degreed≥q, and letaXd be its term of highest degree. Sincea∈ L, we have

a =λ_1a_1 +· · · +λnan, for someλi∈A. Consider the polynomial

n

g_1(X) = λifi(X)Xd−di,

i=1

where di is the degree offi(X). Now,g(X)−g_1(X) is a polynomial in A of degree at most d− 1. By repeating this procedure, we get a sequence of polynomialsgi(X) in B, having strictly decreasing degrees, and such that the polynomial

g(X)− (g_1(X) +· · · +gi(X))

is of degree at mostd−i. This polynomial must be of degree at mostq− 1 as soon as i =d−q + 1. Thus, we proved that every polynomial in A of degreed≥q belongs to B.

It remains to take care of the polynomials in A of degree at mostq− 1. SinceA is noetherian, each idealLi(A) is finitely generated, and let{ai1, . . . , aini} be a set of generators forLi(A) (fori = 0, . . . , q−1). Letfij(X) be a polynomial in A havingaijXi as its highest degree term. Given any polynomial g(X)∈ A of degreed≤q− 1, if we denote its term of highest degree byaXd, then, as in the previous argument, we can write

a =λ_1ad1 +· · · +λndadnd, and we definend

g_1(X) = λifdi(X)Xd−di,

i=1

where di is the degree offdi(X). Then,g(X)−g_1(X) is a polynomial in A of degree at most d− 1, and by repeating this procedure at mostq times, we get an element of A of degree 0, and the latter is a linear combination of thef0i’s. This proves that every polynomial in A of degree at mostq− 1 is a combination of the polynomialsfij(X), for 0≤i≤q− 1 and 1≤j≤ni. Therefore, A is generated by thefk(X)’s and thefij(X)’s, a finite number of polynomials.

Remark: Only a small part of Lemma 21.19 was used in the above proof, namely, the fact thatLi(A) is an ideal. A shorter proof of Theorem 21.21 making full use of Lemma 21.19 can be given as follows:

Proof. (Second proof) Let (Ai)i≥1 be an ascending sequence of ideals inA[X]. Consider the doubly indexed family (Li(Aj)) of ideals inA. SinceA is noetherian, by the maximal property, this family has a maximal elementLp(Aq). Since theLi(Aj)’s form an ascending sequence when eitheri orj is fixed, we haveLi(Aj) =Lp(Aq) for alli andj withi≥p and j≥q, and thus,Li(Aq) =Li(Aj) for alli andj withi≥p andj≥q. On the other hand, for any fixedi, the a.c.c. shows that there exists some integern(i) so thatLi(Aj) =Li(An(i)) for allj≥n(i). SinceLi(Aq) =Li(Aj) wheni≥p andj≥q, we may taken(i) =q if i≥p. This shows that there is somen_0 so thatn(i)≤n_0 for alli≥ 0, and thus, we have Li(Aj) =Li(An(0)) for everyi and for everyj≥n(0). By Lemma 21.19, we get Aj = An(0) for everyj≥n(0), establishing the fact thatA[X] satisfies the a.c.c.

Using induction, we immediately obtain the following important result. Corollary 21.21. IfA is a noetherian ring, thenA[X_1, . . . , Xn] is also a noetherian ring.

Since a fieldK is obviously noetherian (since it has only two ideals, (0) andK), we also have:

Corollary 21.22. IfK is a field, thenK[X_1, . . . , Xn] is a noetherian ring.

21.4 Futher Readings

The material of this Chapter is thoroughly covered in Lang [65], Artin [3], Mac Lane and Birkhoff [70], Bourbaki [12, 13], Malliavin [72], Zariski and Samuel [112], and Van Der Waerden [108].

Chapter 22 Annihilating Polynomials and the Primary Decomposition

22.1 Annihilating Polynomials and the Minimal Polynomial

In Section 5.7, we explained that if f :E→E is a linear map on aK-vector spaceE, then for any polynomialp(X) =a_0Xd +a_1Xd−1 +· · ·+ad with coefficients in the fieldK, we can define the linear mapp(f):E→E by

p(f) =a_0fd +a_1fd−1 +· · · +adid,

wherefk =fæ · · · æf, thek-fold composition off with itself. Note that

p(f)(u) =a_0fd(u) +a_1fd−1(u) +· · · +adu,

for every vector u∈E. Then, we showed that ifE is finite-dimensional and ifχf(X) = det(XI−f) is the characteristic polynomial off, by the Cayley–Hamilton Theorem, we have

χf(f) = 0.

This fact suggests looking at the set of all polynomials p(X) such that p(f) = 0.

We say that the polynomial p(X) annihilates f. It is easy to check that the set Ann(f) of polynomials that annihilatef is an ideal. Furthermore, whenE is finite-dimensional, the Cayley–Hamilton Theorem implies that Ann(f) is not the zero ideal. Therefore, by Proposition 20.9, there is a unique monic polynomialmf that generates Ann(f). Results from Chapter 20, especially about gcd’s of polynomials, will come handy.

Definition 22.1. Iff :E→E, is linear map on a finite-dimensional vector spaceE, the unique monic polynomialmf that generates the ideal Ann(f) of polynomials which annihilate f (the annihilator off) is called the minimal polynomial off.

589

The minimal polynomial mf off is the monic polynomial of smallest degree that annihilatesf. Thus, the minimal polynomial divides the characteristic polynomialχf, and deg(mf)≥ 1. For simplicity of notation, we often writem instead ofmf.

IfA is anyn×n matrix, the set Ann(A) of polynomials that annihilateA is the set of polynomials

p(X) =a_0Xd +a_1Xd−1 +· · · +ad−1X +ad such that

a_0Ad +a_1Ad−1 +· · · +ad−1A +adI = 0.

It is clear that Ann( A) is a nonzero ideal and its unique monic generator is called the minimal polynomial ofA. We check immediately that ifQ is an invertible matrix, thenA andQ−1AQ have the same minimal polynomial. Also, ifA is the matrix off with respect to some basis, thenf andA have the same minimal polynomial.

The zeros (inK) of the minimal polynomial off and the eigenvalues off (inK) are intimately related.

Proposition 22.1. Letf :E→E be a linear map on some finite-dimensional vector space E. Then,λ∈K is a zero of the minimal polynomialmf off iffλ is an eigenvalue off iff λ is a zero of χf. Therefore, the minimal and the characteristic polynomials have the same zeros (inK), except for multiplicities.

Proof. First, assume thatm(λ) = 0 (withλ∈K, and writingm instead ofmf). If so, using polynomial division,m can be factored as

m = (X−λ)q,

with deg( q)< deg(m). Sincem is the minimal polynomial,q(f) = 0, so there is some nonzero vectorv∈E such thatu =q(f)(v) = 0. But then, becausem is the minimal polynomial,

0 = m(f)(v)

= (f−λid)(q(f)(v)) = (f−λid)(u),

which shows thatλ is an eigenvalue off.

Conversely, assume thatλ∈K is an eigenvalue off. This means that for someu = 0, we havef(u) =λu. Now, it is easy to show that

m(f)(u) =m(λ)u,

and sincem is the minimal polynomial off, we havem(f)(u) = 0, som(λ)u = 0, and since u = 0, we must havem(λ) = 0.

If we assume that f is diagonalizable, then its eigenvalues are all inK, and ifλ_1, . . . , λk are the distinct eigenvalues off, then by Proposition 22.1, the minimal polynomialm off must be a product of powers of the polynomials (X−λi). Actually, we claim that

m = (X−λ_1)· · ·(X−λk). For this this, we just have to show thatm annihilatesf. However, for any eigenvectoru of f, one of the linear mapsf−λiid sendsu to 0, so

m(f)(u) = (f−λ_1id)æ · · · æ (f−λkid)(u) = 0.

Since E is spanned by the eigenvectors off, we conclude that m(f) = 0.

Therefore, if a linear map is diagonalizable, then its minimal polynomial is a product of distinct factors of degree 1. It turns out that the converse is true, but this will take a little work to establish it.

22.2 Minimal Polynomials of Diagonalizable Linear Maps

In this section, we prove that if the minimal polynomial mf of a linear mapf is of the form mf = (X−λ_1)· · ·(X−λk)

for disctinct scalarsλ_1, . . . , λk∈K, thenf is diagonalizable. This is a powerful result that has a number of implications. We need of few properties of invariant subspaces. Given a linear mapf :E→E, recall that a subspaceW ofE is invariant underf if f(u)∈W for allu∈W.

Proposition 22.2. LetW be a subspace ofE invariant under the linear mapf :E→E (whereE is finite-dimensional). Then, the minimal polynomial of the restrictionf|W of f toW divides the minimal polynomial off, and the characteristic polynomial off|W divides the characteristic polynomial off.

Sketch of proof. The key ingredient is that we can pick a basis (e_1, . . . , en) ofE in which (e_1, . . . , ek) is a basis ofW. Then, the matrix off over this basis is a block matrix of the form

A = B C ,0 D

whereB is ak×k matrix,D is a (n−k)× (n−k) matrix, andC is ak× (n−k) matrix. Then

det(XI−A) = det(XI−B) det(XI−D), which implies the statement about the characteristic polynomials. Furthermore, Ai = Bi Ci ,0 Di

for some k× (n−k) matrixCi. It follows that any polynomial which annihilatesA also annihilatesB andD. So, the minimal polynomial ofB divides the minimal polynomial of A.

For the next step, there are at least two ways to proceed. We can use an old-fashion argument using Lagrange interpolants, or use a slight generalization of the notion of annihilator. We pick the second method because it illustrates nicely the power of principal ideals.

What we need is the notion of conductor (also called transporter).

Definition 22.2. Letf :E→E be a linear map on a finite-dimensional vector spaceE, let W be an invariant subspace off, and letu be any vector inE. The setSf(u, W) consisting of all polynomialsq∈K[X] such thatq(f)(u)∈W is called thef-conductor ofu intoW.

Observe that the minimal polynomial mf off always belongs toSf(u, W), so this is a nontrivial set. Also, ifW = (0), thenSf(u,(0)) is just the annihilator off. The crucial property ofSf(u, W) is that it is an ideal.

Proposition 22.3. IfW is an invariant subspace forf, then for eachu∈E, thef-conductor Sf(u, W) is an ideal inK[X].

We leave the proof as a simple exercise, using the fact that ifW invariant underf, then W is invariant under every polynomialq(f) inf.

Since Sf(u, W) is an ideal, it is generated by a unique monic polynomialq of smallest degree, and because the minimal polynomialmf off is inSf(u, W), the polynomialq divides m.

Proposition 22.4. Letf :E→E be a linear map on a finite-dimensional spaceE, and assume that the minimal polynomial m off is of the form

m = (X−λ_1)r1 (X−λk)rk,· · ·

where the eigenvaluesλ_1, . . . , λk off belong toK. IfW is a proper subspace ofE which is invariant underf, then there is a vectoru∈E with the following properties:

(a) u /W;

(b) (f−λid)(u)∈W, for some eigenvalueλ off. Proof. Observe that (a) and (b) together assert that the f-conductor of u into W is a polynomial of the formX−λi. Pick any vectorv∈E not inW, and letg be the conductor ofv intoW. Sinceg dividesm andv /∈W, the polynomialg is not a constant, and thus it is of the form

g = (X−λ_1)s1 (X−λk)sk,· · ·

with at least somesi> 0. Choose some indexj such thatsj> 0. ThenX−λj is a factor ofg, so we can write

g = (X−λj)q.

By definition ofg, the vectoru =q(f)(v) cannot be inW, since otherwiseg would not be of minimal degree. However,

(f−λjid)(u) = (f−λjid)(q(f)(v)) =g(f)(v)

is inW, which concludes the proof.

We can now prove the main result of this section. Theorem 22.5. Letf :E→E be a linear map on a finite-dimensional spaceE. Thenf is diagonalizable iff its minimal polynomialm is of the form

m = (X−λ_1)· · ·(X−λk),

whereλ_1, . . . , λk are distinct elements ofK.

Proof. We already showed in Section 22.2 that iff is diagonalizable, then its minimal polynomial is of the above form (whereλ_1, . . . , λk are the distinct eigenvalues off).

For the converse, let W be the subspace spanned by all the eigenvectors off. IfW =E, sinceW is invariant underf, by Proposition 22.4, there is some vectoru /∈W such that for someλj, we have

(f−λjid)(u)∈W. Letv = (f−λjid)(u)∈W. Sincev∈W, we can write

v =w_1 +· · · +wk

wheref(wi) =λiwi (eitherwi = 0 orwi is an eigenvector forλi), and so, for every polynomial h, we have

h(f)(v) =h(λ_1)w_1 +· · · +h(λk)wk, which shows thath(f)(v)∈W for every polynomialh. We can write

m = (X−λj)q for some polynomialq, and also q−q(λj) =p(X−λj) for some polynomialp. We know thatp(f)(v)∈W, and sincem is the minimal polynomial off, we have

0 =m(f)(u) = (f−λjid)(q(f)(u)),

which implies thatq(f)(u)∈W (eitherq(f)(u) = 0, or it is an eigenvector associated with λj). However,

q(f)(u)−q(λj)u =p(f)((f−λjid)(u)) =p(f)(v),

and since p(f)(v)∈W andq(f)(u)∈W, we conclude thatq(λj)u∈W. But,u /∈W, which implies thatq(λj) = 0, soλj is a double root ofm, a contradiction. Therefore, we must have W =E.

Remark: Proposition 22.4 can be used to give a quick proof of Theorem 12.4.

Using Theorem 22.5, we can give a short proof about commuting diagonalizable linear maps. IfF is a family of linear maps on a vector spaceE, we say thatF is a commuting family ifffæg =gæf for allf, g∈ F.

Proposition 22.6. LetF be a finite commuting family of diagonalizable linear maps on a vector spaceE . There exists a basis ofE such that every linear map inF is represented in that basis by a diagonal matrix.

Proof. We proceed by induction onn = dim(E). Ifn = 1, there is nothing to prove. If n > 1, there are two cases. If all linear maps inF are of the formλid for someλ∈ K, then the proposition holds trivially. In the second case, let f∈ F be some linear map inF which is not a scalar multiple of the identity. In this case,f has at least two distinct eigenvaluesλ_1, . . . , λk, and becausef is diagonalizable,E is the direct sum of the corresponding eigenspacesEλ1, . . . , Eλk. For every indexi, the eigenspaceEλi is invariant underf and under every other linear mapg inF, since for anyg∈ F and anyu∈Eλi, becausef andg commute, we have

f(g(u)) =g(f(u)) =g(λiu) =λig(u)

so g(u)∈ Eλi. LetFi be the family obtained by restricting each f∈ F to Eλi. By proposition 22.2, the minimal polynomial of every linear mapf| Eλi inFi divides the minimal polynomialmf off, and sincef is diagonalizable, mf is a product of distinct linear factors, so the minimal polynomial off| Eλi is also a product of distinct linear factors. By Theorem 22.5, the linear mapf|Eλi is diagonalizable. Sincek > 1, we have dim(Eλi)< dim(E) fori = 1, . . . , k, and by the induction hypothesis, for eachi there is a basis ofEλi over whichf| Eλi is represented by a diagonal matrix. Since the above argument holds for alli, by combining the bases of theEλi, we obtain a basis ofE such that the matrix of every linear mapf∈ F is represented by a diagonal matrix. Remark: Proposition 22.6 also holds for infinite commuting familesF of diagonalizable linear maps, becauseE being finite dimensional, there is a finite subfamily of linearly independent linear maps inF spanningF. There is also an analogous result for commuting families of linear maps represented by upper triangular matrices.

22.3 The Primary Decomposition Theorem

If f :E→E is a linear map andλ∈K is an eigenvalue off, recall that the eigenspaceEλ associated withλ is the kernel of the linear mapλid−f. If all the eigenvaluesλ_1. . . , λk of f are inK, it may happen that

E =Eλ1⊕ · · · ⊕Eλk, but in general there are not enough eigenvectors to spanE. What if we generalize the notion of eigenvector and look for (nonzero) vectorsu such that

(λid−f)r(u) = 0, for somer≥ 1?

Then, it turns out that if the minimal polynomial of f is of the form

m = (X−λ_1)r1 (X−λk)rk,· · ·

thenr =ri does the job forλi; that is, if we let

Wi = Ker (λiid−f)ri,

then

E =W1⊕ · · · ⊕Wk.

This result is very nice but seems to require that the eigenvalues of f all belong toK. Actually, it is a special case of a more general result involving the factorization of the minimal polynomialm into its irreducible monic factors (See Theorem 20.16),

m =pr1 pr,1· · · k

where thepi are distinct irreducible monic polynomials overK.

Theorem 22.7. (Primary Decomposition Theorem) Letf :E→E be a linear map on the finite-dimensional vector spaceE over the fieldK. Write the minimal polynomialm off as

m =pr1 pr,1· · · k

where thepi are distinct irreducible monic polynomials overK, and theri are positive integers. Let

Wi = Ker (pr (f)), i = 1, . . . , k.i

(a) E =W1⊕ · · · ⊕Wk. (b) EachWi is invariant underf. (c) The minimal polynomial of the restrictionf|Wi off toWi ispr .i

Proof. The trick is to construct projectionsπi using the polynomialsprj so that the rangej

ofπi is equal toWi. Let

g

i

=

m/p

ri = prj i j. j=i

Note that

prgi =m.i

Since p_1, . . . , pk are irreducible and distinct, they are relatively prime. Then, using Proposition 20.13, it is easy to show thatg_1, . . . , gk are relatively prime. Otherwise, some irreducible polynomialp would divide all ofg_1, . . . , gk, so by Proposition 20.13 it would be equal to one of the irreducible factorspi. But, thatpi is missing fromgi, a contradiction. Therefore, by Proposition 20.14, there exist some polynomialsh_1, . . . , hk such that

g_1h_1 +· · · +gkhk = 1.

Let qi =gihi and letπi =qi(f) =gi(f)hi(f). We have q_1 +· · · +qk = 1,

and sincem dividesqiqj fori =j, we get

π_1 +· · · +πk = id

πiπj = 0, i =j.

(We implicitly used the fact that if p, q are two polynomials, the linear mapsp(f)æq(f) andq(f)æp(f) are the same sincep(f) andq(f) are polynomials in the powers off, which commute.) Composing the first equation withπi and using the second equation, we get

π^2 =πi.

Therefore, theπi are projections, andE is the direct sum of the images of theπi. Indeed, everyu∈E can be expressed as

u =π_1(u) +· · · +πk(u). Also, if π_1(u) +· · · +πk(u) = 0, then by applyingπi we get

0 =π^2(u) =πi(u), i = 1, . . . k.

To finish proving (a), we need to show that

Wi = Ker (pr (f)) =πi(E).i

Ifv∈πi(E), thenv =πi(u) for someu∈E, so

pr (f)(v) =pr (f)(πi(u))i i

=pr (f)gi(f)hi(f)(u)i

=hi(f)pr (f)gi(f)(u)i

=hi(f)m(f)(u) = 0,

becausem is the minimal polynomial off. Therefore,v∈Wi.

Conversely, assume that v∈Wi = Ker (pr (f)). Ifj =i, thengjhj is divisible bypr , soi i gj(f)hj(f)(v) =πj(v) = 0, j =i.

Then, sinceπ_1 +· · · +πk = id, we havev =πiv, which shows thatv is in the range ofπi. Therefore,Wi = Im(πi), and this finishes the proof of (a).

Ifpr (f)(u) = 0, thenpr (f)(f(u)) =f(pr (f)(u)) = 0, so (b) holds.i i i

If we write

f

i

=

f

|

W

i

, then

p

r (fi) = 0, becausepr

i

the minimal polynomial off

(f) = 0 onWi (its kernel). Therefore,i r . Conversely, letq be any polynomial such thati dividespi

q (fi) = 0 (onWi). Sincem =prgi, the fact thatm(f)(u) = 0 for allu∈E shows thati

pr (f)(gi(f)(u)) = 0, u∈E,i

and thus Im(gi(f))r (f)) =Wi. Consequently, sinceq(f) is zero onWi,⊆ Ker (pi

q(f)gi(f) = 0 for allu∈E.

But then,qgi is divisible by the minimal polynomialm =prgi off, and sincepri andgi are relatively prime, by Euclid’s Proposition,

p

r

i i

i must divideq. This finishes the proof that the minimal polynomial of

f

i

is

p

ri

i , which is (c).

If all the eigenvalues off belong to the fieldK, we obtain the following result.

Theorem 22.8. (Primary Decomposition Theorem, Version 2) Letf :E→E be a linear map on the finite-dimensional vector spaceE over the fieldK. If all the eigenvalues λ_1, . . . , λk off belong toK, write

m

= (

X

−

λ

1

)

r1 (X−λk)rk

· · ·

for the minimal polynomial off,

χ

f

= (

X

−

λ

1

)

n1 (X−λk)nk

· · ·

for the characteristic polynomial off, with 1≤ri≤ni, and let

Wi = Ker (λiid−f)ri, i = 1, . . . , k.

(a) E =W1⊕ · · · ⊕Wk. (b) EachWi is invariant underf. (c) dim(Wi) =ni. (d) The minimal polynomial of the restrictionf|Wi off toWi is (X−λi)ri.

Proof. Parts (a), (b) and (d) have already been proved in Theorem 22.8, so it remains to prove (c). SinceWi is invariant underf, letfi be the restriction off toWi. The characteristic polynomialχfi offi dividesχ(f), and sinceχ(f) has all its roots inK, so doesχi(f). By Theorem 12.4, there is a basis ofWi in whichfi is represented by an upper triangular matrix, and since (λiid−f)ri = 0, the diagonal entries of this matrix are equal toλi. Consequently,

χfi = (X−λi)dim(Wi),

and sinceχfi dividesχ(f), we conclude hat

dim(Wi)≤ni, i = 1, . . . , k.

BecauseE is the direct sum of theWi, we have dim(W_1) +· · · + dim(Wk) =n, and since n_1 +· · · +nk =n, we must have

dim(Wi) =ni, i = 1, . . . , k,

proving (c).

Ifλ∈K is an eigenvalue off, it is customary to define a generalized eigenvector off as a nonzero vectoru∈E such that

(λid−f)r(u) = 0, for somer≥ 1.

It is clear that Ker (λid−f)i Ker (λid−f)i+1 for alli≥ 1, and the index ofλ is defined⊆

as the smallestr≥ 1 such that

Ker (λid−f)r = Ker (λid−f)r+1.

By Theorem 22.8(d), ifλ =λi, the index ofλi is equal tori.

Another important consequence of Theorem 22.8 is thatf can be written as the sum of a diagonalizable and a nilpotent linear map (which commute). If we write

D =λ_1π_1 +· · · +λkπk,

where theπk are the projections defined in the proof of Theorem 22.7, since

πk +· · · +πk = id, we have f =fπk +· · · +fπk, and so we get

f−D = (f−λ_1id)π_1 +· · · + (f−λkid)πk.

Since theπi are polynomials inf, they commute withf, and if we writeN =f−D, using the properties of theπi, we get

Nr = (f−λ_1id)rπ_1 +· · · + (f−λkid)rπk.

Therefore, ifr = maxr = 0 fori = 1, . . . , k, which implies that{ri}, we have (f−λkid)

Nr = 0.

A linear mapg:E nilpotent if there is some positive integerr such thatgr = 0. →E is said to be

Since N is a polynomial inf, it commutes withf, and thus withD. Since theλi are distinct, by Theorem 22.5, the linear mapD is diagonalizable, so we have shown that when all the eigenvalues off belong toK, there exist a diagonalizable linear mapD and a nilpotent linear mapN, such that

f =D +N

DN =ND,

andN andD are polynomials inf.

A decomposition off as above is called a Jordan decomposition. In fact, we can prove more: The mapsD andN are uniquely determined byf.

Theorem 22.9. (Jordan Decomposition) Letf :E→ E be a linear map on the finitedimensional vector spaceE over the fieldK. If all the eigenvaluesλ_1, . . . , λk off belong to K, then there exist a diagonalizable linear mapD and a nilpotent linear mapN such that

f =D +N

DN =ND.

Furthermore,D andN are uniquely determined by the above equations and they are polynomials inf.

Proof. We already proved the existence part. Suppose we also havef =D +N , with D N =N D , whereD is diagonalizable,N is nilpotent, and both are polynomials inf. We need to prove thatD =D andN =N .

Since D andN commute with one another andf =D +N , we see thatD andN commute withf. Then,D andN commute with any polynomial inf; hence they commute withD andN. From

D +N =D +N , we get D−D =N−N,

and D, D , N, N commute with one another. SinceD andD are both diagonalizable and commute, by Proposition 22.6, they are simultaneousy diagonalizable, soD−D is diagonalizable. SinceN andN commute, by the binomial formula, for anyr≥ 1,

r r (N )r−jNj.−N)r = (−1)j(Njj=0

Since both N andN are nilpotent, we haveNr1 = 0 and (N )r2 = 0, for somer_1, r_2> 0, so forr≥r_1+r_2, the right-hand side of the above expression is zero, which shows thatN−N is nilpotent. (In fact, it is easy thatr_1 =r_2 =n works). It follows thatD−D =N−N is both diagonalizable and nilpotent. Clearly, the minimal polynomial of a nilpotent linear map is of the formXr for somer > 0 (andr≤ dim(E)). ButD−D is diagonalizable, so its minimal polynomial has simple roots, which means thatr = 1. Therefore, the minimal polynomial ofD−D isX, which says thatD−D = 0, and thenN =N .

If K is an algebraically closed field, then Theorem 22.9 holds. This is the case when K = C. This theorem reduces the study of linear maps (fromE to itself) to the study of nilpotent operators. There is a special normal form for such operators which is discussed in the next section.

22.4 Nilpotent Linear Maps and Jordan Form

This section is devoted to a normal form for nilpotent maps. We follow Godement’s exposition [45]. Letf :E→E be a nilpotent linear map on a finite-dimensional vector space over a fieldK, and assume thatf is not the zero map. Then, there is a smallest positive integer r≥ 1 suchfr = 0 andfr+1 = 0. Clearly, the polynomialXr+1 annihilatesf, and it is the minimal polynomial off sincefr = 0. It follows thatr + 1≤n = dim(E). Let us define the subspacesNi by

Ni = Ker (fi), i≥ 0.

Note thatN_0 = (0),N_1 = Ker (f), andNr+1 =E. Also, it is obvious that

Ni⊆Ni+1, i≥ 0.

Proposition 22.10. Given a nilpotent linear mapf withfr = 0 andfr+1 = 0 as above, the inclusions in the following sequence are strict:

(0) = N0⊂N1⊂ · · · ⊂Nr⊂Nr+1 =E. Proof. We proceed by contradiction. Assume thatNi =Ni+1 for somei with 0≤i≤r. Sincefr+1 = 0, for everyu∈E, we have

0 =fr+1(u) =fi+1(fr−i(u)),

which shows thatfr−i(u)∈ Ni+1. SinceNi =Ni+1, we getfr−i(u)∈Ni, and thusfr(u) = 0. Since this holds for allu∈E, we see thatfr = 0, a contradiction.

Proposition 22.11. Given a nilpotent linear mapf withfr = 0 andfr+1 = 0, for any integeri with 1≤i≤r, for any subspaceU ofE, ifU∩ Ni = (0), thenf(U)∩Ni−1 = (0), and the restriction off toU is an isomorphism ontof(U).

Proof. Pickv∈f(U)∩Ni−1. We havev =f(u) for someu∈U andfi−1(v) = 0, which means that

f

i

( u) = 0. Then,u∈U∩Ni, sou = 0 sinceU∩Ni = (0), andv =f(u) = 0. Therefore,f(U)∩ Ni−1 = (0). The restriction off toU is obviously surjective onf(U). Suppose thatf(u) = 0 for someu∈U. Thenu∈U∩N1⊆U∩Ni = (0) (sincei≥ 1), so u = 0, which proves thatf is also injective onU.

Proposition 22.12. Given a nilpotent linear mapf withfr = 0 andfr+1 = 0, there exists a sequence of subspaceU_1, . . . , Ur+1 ofE with the following properties:

(1) Ni =Ni−1⊕Ui, fori = 1, . . . , r + 1.

(2) We havef(Ui)⊆Ui−1, and the restriction off toUi is an injection, fori = 2, . . . , r+1. Proof. We proceed inductively, by defining the sequenceUr+1, Ur, . . . , U_1. We pickUr+1 to be any supplement ofNr inNr+1 =E, so that

E =Nr+1 =Nr⊕Ur+1.

Since fr+1 = 0 andNr = Ker (fr), we havef(Ur+1)⊆Nr, and by Proposition 22.11, as Ur+1∩Nr = (0), we havef(Ur+1)∩Nr− 1 = (0). As a consequence, we can pick a supplement Ur ofNr−1 inNr so thatf(Ur+1)⊆Ur. We have

Nr =Nr−1⊕Ur and f(Ur+1)⊆Ur.

By Proposition 22.11,f is an injection fromUr+1 toUr. Assume inductively thatUr+1, . . . , Ui have been defined fori≥ 2 and that they satisfy (1) and (2). Since

Ni =Ni−1⊕Ui,

we have Ui−1(f(Ui)) =fi(Ui) = (0), which implies thatf(Ui)⊆Ni−1. Also,i⊆Ni, sof

sinceUi∩Ni−1 = (0), by Proposition 22.11, we havef(Ui)∩Ni−2 = (0). It follows that there is a supplementUi−1 ofNi−2 inNi−1 that containsf(Ui). We have

Ni−1 =Ni−2⊕Ui−1 and f(Ui)⊆Ui−1.

The fact thatf is an injection fromUi intoUi−1 follows from Proposition 22.11. Therefore, the induction step is proved. The construction stops wheni = 1.

Because N_0 = (0) andNr+1 =E, we see thatE is the direct sum of theUi: E =U1⊕ · · · ⊕Ur+1,

withf(Ui)⊆Ui−1, andf an injection fromUi toUi−1, fori =r + 1, . . . ,2. By a clever choice of bases in theUi, we obtain the following nice theorem.

Theorem 22.13. For any nilpotent linear mapf :E→E on a finite-dimensional vector spaceE of dimensionn over a fieldK, there is a basis of E such that the matrixN off is of the formë0 ν1 0 · · · 0 0 ö

ì0 0 ν2 · · · 0 0÷

ì ÷

N

=

ì. . . . . .÷

ì ÷

÷

í

,

ì0 0 0 · · · 0 νnø0 0 0 · · · 0 0

whereνi = 1 orνi = 0.

Proof. First, apply Proposition 22.12 to obtain a direct sumE =r+1Ui. Then, we definei=1

a basis ofE inductively as follows. First, we choose a basis

e

r+1 , . . . , er+1

1 nr+1

ofUr+1. Next, fori =r + 1, . . . ,2, given the basis e

i

1, . . . , eini ofUi, sincef is injective onUi andf(Ui)⊆Ui−1, the vectorsf(ei), . . . , f(eini) are linearly independent, so we define a basis of

U

i

−

1

by completing

f

(

e

i

1

), . . . , f(eini) to a basis inUi−1:1

e

i 1 , . . . , ei−1 , ei−1 , . . . , ei−1

1 ni ni+1 ni−1

with ei−1 =f(eij), j = 1. . . , ni.j

Since U_1 =N_1 = Ker (f), we have

f(e^1) = 0, j = 1, . . . , n_1.

These basis vectors can be arranged as the rows of the following matrix:

ëer+1 er+1 ö1 · · ·nr+1

ì

÷ ì

.

.

÷ ì ÷ ìer

÷ ì

e

rn

r

+1

e

rn

r

+1

+1

· · ·

e

rn

r

÷

ì^1 · · ·. . . ÷

ì . ÷

ìer−1 er−1 er−1 er−1 er−1 er−1 ÷

ì 1 · · · nr+1 nr +1+1 · · · nr nr+1 · · · nr −1÷ ì ÷ ì . . . . . . ÷ ì ÷ ì. . . . . . ÷

í ø e

1

1 e1nr+1 e1nr+1+1 · · · e1nr e1nr+1 · · · e1nr−1 · · · · · · e1n1 · · ·

Finally, we define the basis ( e_1, . . . , en) by listing each column of the above matrix from the bottom-up, starting with column one, then column two, etc. This means that we list the vectorseij in the following order:

Forj = 1, . . . , nr+1, liste^1, . . . , er+1;j

In general, fori =r, . . . ,1,

forj =ni+1 + 1, . . . , ni, liste^1, . . . , eij.

Then, becausef(e^1) = 0 andei−1 =f(eij) fori≥ 2, eitherj

f(ei) = 0 or f(ei) =ei−1,

which proves the theorem.

As an application of Theorem 22.13, we obtain the Jordan form of a linear map. Definition 22.3. A Jordan block is anr×r matrixJr(λ), of the form

ëλ 1 0 · · ·0ö ì0 λ 1 · · · 0÷

÷ J ì

í0 0 0

ì ... ... .÷,r(λ) =ì . .÷

ì ... ÷

1ø 0 0 0 · · · λ

whereλ∈K, withJ_1(λ) = (λ) ifr = 1. A Jordan matrix,J, is ann×n block diagonal matrix of the formëJr1(λ1) · · · 0 ö

J = ì . ... . ÷,í 0 · · · Jrm(λm)ø

where eachJrk(λk) is a Jordan block associated with someλk∈K, and withr_1+· · ·+rm =n.

To simplify notation, we often writeJ(λ) forJr(λ). Here is an example of a Jordan matrix with four blocks:

ë λ 1 0 0 0 0 0 0ö ì0 λ 1 0 0 0 0 0÷ ì0 0 λ 0 0 0 0 0÷

ì ÷

ì0 0 0 λ 1 0 0 0÷ J

=

ì ÷

ì .ì0 0 0 0 λ 0 0 0÷

÷

ì0 0 0 0 0 λ 0 0÷

ì ÷

ì0 0 0 0 0 0 µ 1÷ í ø

0 0 0 0 0 0 0 µ Theorem 22.14. (Jordan form) LetE be a vector space of dimensionn over a fieldK and letf :E→E be a linear map. The following properties are equivalent:

(1) The eigenvalues off all belong toK (i.e. the roots of the characteristic polynomialχf all belong toK).

(2) There is a basis ofE in which the matrix off is a Jordan matrix.

Proof. Assume (1). First we apply Theorem 22.8, and we get a direct sumE =k Wk,j=1 such that the restriction ofgi =f−λjid toWi is nilpotent. By Theorem 22.13, there is a basis ofWi such that the matrix of the restriction ofgi is of the form

ë0 ν_1 0 · · · 0 0ö

ì0 0 ν2 · · · 0 0÷

ì ÷

G

i

=

ì. . . . . . ÷

ì ÷

÷

í

,

ì0 0 0 · · · 0 νniø0 0 0 · · · 0 0

where νi = 1 orνi = 0. Furthermore, over any basis,λiid is represented by the diagonal matrixDi withλi on the diagonal. Then, it is clear that we can splitDi +Gi into Jordan blocks by forming a Jordan block for every uninterrupted chain of 1s. By Putting the bases of theWi together, we obtain a matrix in Jordan form forf.

Now, assume (2). Iff can be represented by a Jordan matrix, it is obvious that the diagonal entries are the eigenvalues off, so they all belong toK.

Observe that Theorem 22.14 applies if K = C. It turns out that there are uniqueness properties of the Jordan blocks. There are also other fundamental normal forms for linear maps, such as the rational canonical form, but to prove these results, it is better to develop more powerful machinery about finitely generated modules over a PID. To accomplish this most effectively, we need some basic knowledge about tensor products.

Chapter 23 Tensor Algebras, Symmetric Algebras and Exterior Algebras

23.1 Tensors Products

We begin by defining tensor products of vector spaces over a field and then we investigate some basic properties of these tensors, in particular the existence of bases and duality. After this, we investigate special kinds of tensors, namely, symmetric tensors and skew-symmetric tensors. Tensor products of modules over a commutative ring with identity will be discussed in Chapter 24.

Given a linear map, f :E→F, we know that if we have a basis, (ui)i∈I, forE, thenf is completely determined by its values,f(ui), on the basis vectors. For a multilinear map, f :En F, we don’t know if there is such a nice property but it would certainly be very useful.→

In many respects, tensor products allow us to define multilinear maps in terms of their action on a suitable basis. The crucial idea is to linearize, that is, to create a new vector space, E⊗n, such that the multilinear map,f :En F, is turned into a linear map,f⊗:E⊗n F,→ → which is equivalent tof in a strong sense. If in addition,f is symmetric, then we can define a symmetric tensor power, Symn(E), and every symmetric multilinear map,f :En F, is

turned into a

linear map

,

f

: Sym

n

→

( E)→F, which is equivalent tof in a strong sense. Similarly, iff is alternating, then we can define a skew-symmetric tensor power,n(E), and every alternating multilinear map is turned into a linear map,f∧:n(E)→F, which is equivalent tof in a strong sense.

Tensor products can be defined in various ways, some more abstract than others. We tried to stay down to earth, without excess!

LetK be a given field, and letE_1, . . . , En ben≥ 2 given vector spaces. For any vector space,F, recall that a map,f :E1× · · · ×En→F, is multilinear iff it is linear in each of

605 its argument, that is,

f(u_1, . . . ui1, v +w, ui+1, . . . , un) = f(u_1, . . . ui1, v, ui+1, . . . , un) +f(u_1, . . . ui1, w, ui+1, . . . , un) f(u_1, . . . ui1, λv, ui+1, . . . , un) = λf(u_1, . . . ui1, v, ui+1, . . . , un), for alluj∈Ej (j =i), allv, w∈Ei and allλ∈K, fori = 1. . . , n.

The set of multilinear maps as above forms a vector space denoted L( E_1, . . . , En;F) or Hom(E_1, . . . , En;F). Whenn = 1, we have the vector space of linear maps, L(E, F) or Hom(E, F). (To be very precise, we write HomK(E_1, . . . , En;F) and HomK(E, F).) As usual, the dual space,E∗, ofE is defined byE∗ = Hom(E, K).

Before proceeding any further, we recall a basic fact about pairings. We will use this fact to deal with dual spaces of tensors.

Definition 23.1. Given two vector spaces,E andF, a map, (−,−):E×F→ K, is a nondegenerate pairing iff it is bilinear and iff (u, v) = 0 for allv∈F impliesu = 0 and (u, v) = 0 for allu∈E impliesv = 0. A nondegenerate pairing induces two linear maps, Õ:E→F∗ andψ:F→E∗, defined by

Õ(u)(y) = (u, y) ψ(v)(x) = (x, v),

for allu, x∈E and allv, y∈F.

Proposition 23.1. For every nondegenerate pairing, (−,−):E×F→K, the induced maps Õ:E→F∗ andψ:F→E∗ are linear and injective. Furthermore, if E andF are finite dimensional, thenÕ:E→F∗ andψ:F→E∗ are bijective.

Proof. The mapsÕ:E→ F∗ andψ:F→ E∗ are linear becauseu, v→ (u, v) is bilinear. Assume thatÕ(u) = 0. This means thatÕ(u)(y) = (u, y) = 0 for ally ∈F and as our pairing is nondegenerate, we must haveu = 0. Similarly,ψ is injective. IfE andF are finite dimensional, then dim(E) = dim(E∗) and dim(F) = dim(F∗). However, the injectivity ofÕ andψ implies that that dim(E)≤ dim(F∗) and dim(F)≤ dim(E∗). Consequently dim(E)≤ dim(F) and dim(F)≤ dim(E), so dim(E) = dim(F). Therefore, dim(E) = dim(F∗) andÕ is bijective (and similarly dim(F) = dim(E∗) andψ is bijective).

Proposition 23.1 shows that when E andF are finite dimensional, a nondegenerate pairing induces canonical isomorphimsÕ:E→F∗ andψ:F→E∗, that is, isomorphisms that do not depend on the choice of bases. An important special case is the case whereE =F and we have an inner product (a symmetric, positive definite bilinear form) onE.

Remark: When we use the term “canonical isomorphism” we mean that such an isomorphism is defined independently of any choice of bases. For example, ifE is a finite dimensional vector space and (e_1, . . . , en) is any basis ofE, we have the dual basis, (e∗, . . . , e∗n), ofE∗1 (where,e∗(ej) =δi j) and thus, the mapei→e∗ is an isomorphism betweenE andE∗. This isomorphism is not canonical.

On the other hand, if−,− is an inner product onE, then Proposition 23.1 shows that the nondegenerate pairing,−,−, induces a canonical isomorphism betweenE andE∗. This isomorphism is often denoted:E→E∗ and we usually writeu for (u), withu∈E.

Given any basis, (e_1, . . . , en), ofE (not necessarily orthonormal), if we letgij = (ei, ej), then for everyu =n ujej, sinceu (v) = u, v for allv∈V , we havej=1

n n n

u (ei) = (u, ei) = ujej, ei = uj(ej, ei) = gijuj,

j=1 j=1 j=1

so we getn n

u = ωie∗, with ωi = gijuj.

i=1 j=1

If we use the convention that coordinates of vectors are written using superscripts (u =n uiei) and coordinates of one-forms (covectors) are written using subscriptsi=1

(ω =n ωie∗), then the map, , has the effect of lowering (flattening!) indices. Thei=1

inverse of is denoted :E∗ n ωie∗ andω∈E as

ω

=

n (ω )jej, since →E. If we writeω∈E∗ asω =i=1

j=1

n

ωi =ω(ei) = ω , ei = (ω )jgij, 1≤i≤n,

j=1

we getn

(ω )i = gijωj,

j=1

where (gij) is the inverse of the matrix (gij).

The inner product, (−,−), onE induces an inner product onE∗ also denoted (−,−) and given by

(ω_1, ω_2) = (ω_1, ω_2), for allω_1, ω2∈E∗. Then, it is obvious that

(u, v) = (u , v ), for all u, v∈E.

If (e_1, . . . , en) is a basis ofE andgij = (ei, ej), as

n

(e∗) = gikek,

k=1

an easy computation shows that

(e∗, e∗) = ((e∗),(e∗) ) =gij,

that is, in the basis (e∗, . . . , e∗n), the inner product onE∗ is represented by the matrix (gij),

1

the inverse of the matrix (gij).

The inner product on a finite vector space also yields a natural isomorphism between the space, Hom(E, E;K), of bilinear forms onE and the space, Hom(E, E), of linear maps fromE to itself. Using this isomorphism, we can define the trace of a bilinear form in an intrinsic manner. This technique is used in differential geometry, for example, to define the divergence of a differential one-form.

Proposition 23.2. If−,− is an inner product on a finite vector space,E, (over a field, K), then for every bilinear form, f :E×E→K, there is a unique linear map,f :E→E, such that

f (u, v) = f (u), v , for allu, v∈E.

The map,f→f , is a linear isomorphism between Hom(E, E;K) and Hom(E, E). Proof. For everyg∈ Hom(E, E), the map given by

f(u, v) = g(u), v , u, v∈E,

is clearly bilinear. It is also clear that the above defines a linear map from Hom( E, E) to Hom(E, E;K). This map is injective because iff(u, v) = 0 for allu, v∈E, as−,− is an inner product, we getg(u) = 0 for allu∈E. Furthermore, both spaces Hom(E, E) and Hom(E, E;K) have the same dimension, so our linear map is an isomorphism.

If (e_1, . . . , en) is an orthonormal basis ofE, then we check immediately that the trace of a linear map,g, (which is independent of the choice of a basis) is given by

n

tr(g) = g(ei), ei,

i=1

where n = dim(E). We define the trace of the bilinear form,f, by

tr(f) = tr(f ).

From Proposition 23.2, tr(f) is given by

n

tr(f) = f(ei, ei),

i=1

for any orthonormal basis, (e_1, . . . , en), ofE. We can also check directly that the above expression is independent of the choice of an orthonormal basis.

We will also need the following Proposition to show that various families are linearly independent.

Proposition 23.3. LetE andF be two nontrivial vector spaces and let (ui)i∈I be any family of vectorsui∈ E. The family, (ui)i∈I, is linearly independent iff for every family, (vi)i∈I, of vectorsvi∈F, there is some linear map,f :E→F, so thatf(ui) =vi, for alli∈I.

Proof. Left as an exercise.

First, we define tensor products, and then we prove their existence and uniqueness up to isomorphism.

Definition 23.2. A tensor product ofn≥ 2 vector spacesE_1, . . . , En, is a vector spaceT, together with a multilinear mapÕ:E1× · · · ×En →T, such that, for every vector spaceF and for every multilinear mapf :E1×· · ·×En→F, there is a unique linear mapf⊗:T→F, with

f(u_1, . . . , un) =f⊗(Õ(u_1, . . . , un)), for allu1∈E_1, . . . , un∈En, or for short

f =f⊗æÕ.

Equivalently, there is a unique linear mapf⊗ such that the following diagram commutes:

E

1

× · · · ×

N

E

Õ n

T NNNNNNNNNN F

f⊗ f

First, we show that any two tensor products (T_1, Õ_1) and (T_2, Õ_2) forE_1, . . . , En, are isomorphic.

Proposition 23.4. Given any two tensor products (T_1, Õ_1) and (T_2, Õ_2) forE_1, . . . , En, there is an isomorphismh:T1→T_2 such that

Õ_2 =hæÕ_1.

Proof. Focusing on (T_1, Õ_1), we have a multilinear mapÕ_2:E1× · · · ×En→T_2, and thus, there is a unique linear map (Õ_2)⊗:T1→T_2, with

Õ_2 = (Õ_2)⊗æÕ_1.

Similarly, focusing now on on (T_2, Õ_2), we have a multilinear mapÕ_1:E1× · · · ×En→T_1, and thus, there is a unique linear map (Õ_1)⊗:T2→T_1, with

Õ_1 = (Õ_1)⊗æÕ_2. But then, we get Õ_1 = (Õ_1)⊗æ (Õ_2)⊗æÕ_1, and

Õ_2 = (Õ_2)⊗æ (Õ_1)⊗æÕ_2.

On the other hand, focusing on (T_1, Õ_1), we have a multilinear mapÕ_1:E1× · · · ×En→T_1, but the unique linear maph:T1→T_1, with

Õ_1 =hæÕ_1

ish = id, and since (Õ_1)⊗æ (Õ_2)⊗ is linear, as a composition of linear maps, we must have

(Õ_1)⊗æ (Õ_2)⊗ = id.

Similarly, we must have (Õ_2)⊗æ (Õ_1)⊗ = id.

This shows that (Õ_1)⊗ and (Õ_2)⊗ are inverse linear maps, and thus, (Õ_2)⊗:T1→T_2 is an isomorphism betweenT_1 andT_2.

Now that we have shown that tensor products are unique up to isomorphism, we give a construction that produces one.

Theorem 23.5. Givenn≥ 2 vector spacesE_1, . . . , En, a tensor product (E1⊗ · · · ⊗En, Õ) forE_1, . . . , En can be constructed. Furthermore, denotingÕ(u_1, . . . , un) asu1⊗ · · · ⊗ un, the tensor productE1⊗ · · · ⊗En is generated by the vectorsu1⊗ · · · ⊗ un, whereu1∈E1, . . . , un ∈En , and for every multilinear mapf :E1× · · · ×En→F, the unique linear

mapf⊗:E1⊗ · · · ⊗En→F such thatf =f⊗æÕ, is defined by

f⊗(u1⊗ · · · ⊗un) =f(u_1, . . . , un), on the generatorsu1⊗ · · · ⊗un ofE1⊗ · · · ⊗En.

Proof. Given any set,I, viewed as an index set, letK(I) be the set of all functions,f :I→K, such thatf(i) = 0 only for finitely manyi∈I. As usual, denote such a function by (fi)i∈I, it is a family of finite support. We makeK(I) into a vector space by defining addition and scalar multiplication by

(fi) + (gi) = (fi +gi) λ(fi) = (λfi).

The family, ( ei)i∈I, is defined such that (ei)j = 0 ifj =i and (ei)i = 1. It is a basis of the vector spaceK(I), so that everyw∈K(I) can be uniquely written as a finite linear combination of theei. There is also an injection,ι:I→K(I), such thatι(i) =ei for every i∈I. Furthermore, it is easy to show that for any vector space,F, and for any function, f :I→F, there is a unique linear map,f :K(I) F, such thatf =fæι, as in the following diagram:→

I

CCCCCCCCC ff

ι K(I)

F

This shows that K(I) is the free vector space generated byI. Now, apply this construction to the cartesian product,I =E1× · · · ×En, obtaining the free vector spaceM =K(I) on I =E1×· · ·×En. Since every,ei, is uniquely associated with somen-tuplei = (u_1, . . . , un)∈E1× · · · ×En, we will denoteei by (u1, . . . , un).

Next, letN be the subspace ofM generated by the vectors of the following type: (u_1, . . . , ui +vi, . . . , un)− (u_1, . . . , ui, . . . , un)− (u_1, . . . , vi, . . . , un), (u_1, . . . , λui, . . . , un)−λ(u_1, . . . , ui, . . . , un).

We letE1⊗ · · · ⊗En be the quotientM/N of the free vector spaceM byN,π:M→M/N be the quotient map and set

Õ =πæι.

By construction, Õ is multilinear, and sinceπ is surjective and theι(i) =ei generateM, sincei is of the formi = (u_1, . . . , un)∈E1×· · ·×En, theÕ(u_1, . . . , un) generateM/N. Thus, if we denoteÕ(u_1, . . . , un) asu1 ⊗ · · · ⊗un, the tensor productE1⊗ · · · ⊗En is generated by the vectorsu1⊗ · · · ⊗un, whereu1∈E1, . . . , un∈En.

For every multilinear map f :E1× · · · ×En →F, if a linear mapf⊗:E1⊗ · · · ⊗ En→F exists such thatf =f⊗æÕ, since the vectorsu1⊗ · · · ⊗un generateE1⊗ · · · ⊗En, the map f⊗ is uniquely defined by

f⊗(u1⊗ · · · ⊗un) =f(u_1, . . . , un).

On the other hand, becauseM =K(E1×···×En) is free onI =E1× · · · ×En, there is a unique linear mapf :K(E1×···×En) F, such that→

f =fæι, as in the diagram below:

E

1

× · · · ×

E

ι

n K(E1×···×En)

RRRRRRR f

f

RRRRRRRR F

Becausef is multilinear, note that we must havef(w) = 0, for everyw∈N. But then, f :M→F induces a linear maph:M/N→F, such that

f =hæπæι,

by definingh([z]) =f(z), for everyz∈M, where [z] denotes the equivalence class inM/N ofz∈M:

E1× · · · ×En πæι /K(E1×···×En)/N

SSSSSSSS h

f

SSSSSSSSS F

Indeed, the fact that f vanishes onN insures thath is well defined onM/N, and it is clearly linear by definition. However, we showed that such a linear maph is unique, and thus it agrees with the linear mapf⊗ defined by

f⊗(u1⊗ · · · ⊗un) =f(u_1, . . . , un)

on the generators ofE1⊗ · · · ⊗En.

What is important about Theorem 23.5 is not so much the construction itself but the fact that it produces a tensor product with the universal mapping property with respect to multilinear maps. Indeed, Theorem 23.5 yields a canonical isomorphism,

L(E1⊗ · · · ⊗En, F)∼= L(E_1, . . . , En;F),

between the vector space of linear maps, L(E1⊗ · · · ⊗En, F), and the vector space of multilinear maps,L(E_1, . . . , En;F), via the linear map− æÕ defined by

h→hæÕ,

where h∈ L(E1⊗ · · · ⊗En, F). Indeed,hæÕ is clearly multilinear, and since by Theorem 23.5, for every multilinear map,f∈ L(E_1, . . . , En;F), there is a unique linear mapf⊗∈ L(E1⊗ · · · ⊗En, F) such thatf =f⊗æÕ, the map− æÕ is bijective. As a matter of fact, its inverse is the map

f→f⊗.

Using the “Hom” notation, the above canonical isomorphism is written

Hom(E1⊗ · · · ⊗En, F)∼= Hom(E_1, . . . , En;F).

Remarks: (1) To be very precise, since the tensor product depends on the field,K, we should subscript the symbol⊗ withK and write

E1⊗K· · · ⊗KEn. However, we often omit the subscriptK unless confusion may arise. 23.2. BASES OF TENSOR PRODUCTS 613

(2) For F =K, the base field, we obtain a canonical isomorphism between the vector space L(E1 ⊗ · · · ⊗En, K), and the vector space of multilinear formsL(E_1, . . . , En;K). However, L(E1⊗ · · · ⊗En, K) is the dual space, (E1⊗ · · · ⊗En)∗, and thus, the vector space of multilinear formsL(E_1, . . . , En;K) is canonically isomorphic to (E1⊗ · · · ⊗En)∗. We write

L(E_1, . . . , En;K)∼= (E1⊗ · · · ⊗En)∗.

The fact that the mapÕ:E1× · · · ×En→E1⊗ · · · ⊗En is multilinear, can also be expressed as follows:

u1⊗ · · · ⊗ (vi +wi)⊗ · · · ⊗un = (u1⊗ · · · ⊗vi⊗ · · · ⊗un) + (u1⊗ · · · ⊗wi⊗ · · · ⊗un), u1⊗ · · · ⊗ (λui)⊗ · · · ⊗un =λ(u1⊗ · · · ⊗ui⊗ · · · ⊗un).

Of course, this is just what we wanted! Tensors in E1⊗· · ·⊗En are also calledn-tensors, and tensors of the formu1⊗ · · · ⊗un, whereui∈Ei, are called simple (or indecomposable) n-tensors. Thosen-tensors that are not simple are often called compoundn-tensors.

Not only do tensor products act on spaces, but they also act on linear maps (they are functors). Given two linear mapsf :E→E andg:F→F , we can defineh:E×F→E⊗F by

h(u, v) =f(u)⊗g(v).

It is immediately verified thath is bilinear, and thus, it induces a unique linear map

f⊗g:E⊗F→E⊗F ,

such that (f⊗g)(u⊗v) =f(u)⊗g(u).

If we also have linear maps f :E→E andg :F →F , we can easily verify that the linear maps (fæf)⊗ (gæg) and (f⊗ g )æ (f⊗g) agree on all vectors of the form u⊗v∈E⊗F. Since these vectors generateE⊗F, we conclude that

(fæf)⊗ (gæg) = (f⊗g )æ (f⊗g).

The generalization to the tensor productf1⊗ · · · ⊗fn ofn≥ 3 linear mapsfi:Ei→Fi is immediate, and left to the reader.

23.2 Bases of Tensor Products

We showed that E1⊗· · ·⊗En is generated by the vectors of the formu1⊗· · ·⊗un. However, there vectors are not linearly independent. This situation can be fixed when considering bases, which is the object of the next proposition.

Proposition 23.6. Givenn≥ 2 vector spacesE_1, . . . , En, if (uki)i∈Ik is a basis forEk, 1≤k≤n, then the family of vectors

(u^1 unin)(i_1,...,in)∈I1×...×In1⊗ · · · ⊗

is a basis of the tensor productE1⊗ · · · ⊗En.

Proof. For eachk, 1≤k≤n, everyvk Ek can be written uniquely as∈

vk = vkjukj,

j∈Ik

for some family of scalars (vkj)j∈Ik. LetF be any nontrivial vector space. We show that for every family

(wi_1,...,in)(i_1,...,in)∈I1×...×In,

of vectors inF, there is some linear maph:E1⊗ · · · ⊗En→F, such that

h(u^1 unin) =wi1,...,in.1⊗ · · · ⊗

Then, by Proposition 23.3, it follows that

(u^1 unin)(i_1,...,in)∈I1×...×In1⊗ · · · ⊗

is linearly independent. However, since (uki)i∈Ik is a basis forEk, theu^1 unin also1⊗ · · · ⊗generateE1⊗ · · · ⊗En, and thus, they form a basis ofE1⊗ · · · ⊗En.

We define the functionf :E1× · · · ×En→F as follows:

f( v^1 u^1 , . . . , vn unjn) = v^1 vn wj1,...,jn.1 1 jn 1· · · jn

j1∈I_1 jn∈In j1∈I_1,...,jn∈In

It is immediately verified that f is multilinear. By the universal mapping property of the tensor product, the linear mapf⊗:E1⊗ · · · ⊗En→F such thatf =f⊗æÕ, is the desired maph.

In particular, when each Ik is finite and of sizemk = dim(Ek), we see that the dimension of the tensor productE1⊗· · ·⊗En ism1· · ·mn. As a corollary of Proposition 23.6, if (uki)i∈Ik is a basis forEk, 1≤k≤n, then every tensorz∈E1⊗ · · · ⊗En can be written in a unique way as

z = λi1,...,inu1 unin,1⊗ · · · ⊗(i_1,...,in)∈I1×...×In

for some unique family of scalarsλi1,...,in∈K, all zero except for a finite number. 23.3. SOME USEFUL ISOMORPHISMS FOR TENSOR PRODUCTS 615

23.3 Some Useful Isomorphisms for Tensor Products

Proposition 23.7. Given three vector spacesE, F, G, there exists unique canonical isomorphisms

(1) E⊗F F⊗E

(2) (E⊗F)⊗G E⊗ (F⊗G) E⊗F⊗G

(3) (E⊕F)⊗G (E⊗G)⊕ (F⊗G) (4) K⊗E E

such that respectively

(a) u⊗v→v⊗u

(b) (u⊗v)⊗w→u⊗ (v⊗w)→u⊗v⊗w (c) (u, v)⊗w→ (u⊗w, v⊗w)

(d) λ⊗u→λu.

Proof. These isomorphisms are proved using the universal mapping property of tensor products. We illustrate the proof method on (2). Fix somew∈G. The map

(u, v)→u⊗v⊗w

fromE×F toE⊗F⊗G is bilinear, and thus, there is a linear mapfw:E⊗F→E⊗F⊗G, such thatfw(u⊗v) =u⊗v⊗w.

Next, consider the map

(z, w)→fw(z),

from (E⊗F)×G intoE⊗F⊗G. It is easily seen to be bilinear, and thus, it induces a linear map

f : (E⊗F)⊗G→E⊗F⊗G, such thatf((u⊗v)⊗w) =u⊗v⊗w.

Also consider the map

(u, v, w)→ (u⊗v)⊗w

fromE×F×G to (E⊗F)⊗G. It is trilinear, and thus, there is a linear map

g:E⊗F⊗G→ (E⊗F)⊗G,

such thatg(u⊗v⊗w) = (u⊗v)⊗w. Clearly,fæg andgæf are identity maps, and thus, f andg are isomorphisms.

To prove (4), we consider the bilinear map Õ:K×E→ E given byÕ(λ, u) = λu, which induces a linear mapÕ:K⊗E→E such thatÕ(λ⊗u) =λu, and the linear map ψ:E→K⊗E given byψ(u) = 1⊗u. Then, we have

ψ (Õ(λ⊗u)) =ψ(λu) = 1⊗λu =λ⊗u,

and Õ(ψ(u)) =Õ(1⊗u) =u,

soÕ andψ are mutual inverses on generators, which implies that they are isomorphisms. The other cases are similar.

Given any three vector spaces, E, F, G, we have the canonical isomorphism Hom(E, F;G)∼= Hom(E,Hom(F, G)).

Indeed, any bilinear map,f :E×F→G, gives the linear map,Õ(f)∈ Hom(E,Hom(F, G)), whereÕ(f)(u) is the linear map in Hom(F, G) given by

Õ(f)(u)(v) =f(u, v). Conversely, given a linear map,g∈ Hom(E,Hom(F, G)), we get the bilinear map,ψ(g), given by

ψ(g)(u, v) =g(u)(v),

and it is clear thatÕ andψ and mutual inverses. Consequently, we have the important corollary:

Proposition 23.8. For any three vector spaces,E, F, G, we have the canonical isomorphism, Hom(E⊗F, G)∼= Hom(E,Hom(F, G)),

23.4 Duality for Tensor Products

In this section, all vector spaces are assumed to have finite dimension. Let us now see how tensor products behave under duality. For this, we define a pairing betweenE∗ E∗ and1⊗· · ·⊗

E1 ⊗ · · · ⊗En as follows: For any fixed (v∗, . . . , v∗)∈E∗ E∗, we have the multilinear 1 1× · · · ×map,

lv∗,...,v∗: (u_1, . . . , un)→v∗(u_1)· · ·v∗(un),1 n 1

fromE1× · · · ×En toK. The maplv∗,...,v∗ extends uniquely to a linear map,1 n

Lv∗,...,v∗:E1⊗ · · · ⊗En−→K. We also have the multilinear map,1 n

(v∗, . . . , v∗n)→Lv∗,...,v∗,1 1 n 23.4. DUALITY FOR TENSOR PRODUCTS 617

fromE∗ E∗ to Hom(E1⊗ · · · ⊗ En, K), which extends to a linear map,L, from_1

E∗ × · · · × to Hom(E1⊗ · · · ⊗En, K). However, in view of the isomorphism,1⊗ · · · ⊗E∗

Hom(U⊗V, W)∼= Hom(U,Hom(V, W)), we can viewL as a linear map,

L: (E∗⊗ · · · ⊗E∗)⊗ (E1⊗ · · · ⊗En)→K,

1

which corresponds to a bilinear map,

(E∗⊗ · · · ⊗E∗)× (E1⊗ · · · ⊗En)−→K,

1

via the isomorphism (U⊗V )∗∼= L(U, V ;K). It is easy to check that this bilinear map is nondegenerate and thus, by Proposition 23.1, we have a canonical isomorphism,

( E1⊗ · · · ⊗En)∗ =E∗ E∗.∼ 1⊗ · · · ⊗

This, together with the isomorphism, L(E_1, . . . , En;K)∼= (E1⊗· · ·⊗En)∗, yields a canonical isomorphism

L(E_1, . . . , En;K)=E∗⊗ · · · ⊗E∗n.∼ 1

We prove another useful canonical isomorphism that allows us to treat linear maps as tensors.

LetE andF be two vector spaces and letα:E∗×F→ Hom(E, F) be the map defined such that

α(u∗, f)(x) =u∗(x)f,

for allu∗∈E∗,f∈F, andx∈E. This map is clearly bilinear and thus, it induces a linear map,

α⊗:E∗⊗F→ Hom(E, F), such that

α⊗(u∗⊗f)(x) =u∗(x)f. Proposition 23.9. IfE andF are vector spaces withE of finite dimension, then the linear map,α⊗:E∗⊗F→ Hom(E, F), is a canonical isomorphism.

Proof. Let (ej)1≤j≤n be a basis ofE and, as usual, lete∗∈E∗ be the linear form defined by

e∗(ek) =δj,k,

where δj,k = 1 iffj =k and 0 otherwise. We know that (e∗)1≤j≤n is a basis ofE∗ (this is where we use the finite dimension ofE). Now, for any linear map,f∈ Hom(E, F), for every x =x_1e_1 +· · · +xnen∈E, we have

f(x) =f(x_1e_1 +· · · +xnen) =x_1f(e_1) +· · · +xnf(en) =e∗(x)f(e_1) +· · · +e∗n(x)f(en).1

Consequently, every linear map,f∈ Hom(E, F), can be expressed as

f(x) =e∗(x)f_1 +· · · +e∗n(x)fn,1

for somefi∈F. Furthermore, if we applyf toei, we getf(ei) =fi, so thefi are unique. Observe that

n n

(α⊗(e∗⊗f_1 +· · · +e∗n⊗fn))(x) = (α⊗(e∗⊗fi))(x) = e∗(x)fi.1

i=1 i=1

Thus,α⊗ is surjective. As (e∗)1≤j≤n is a basis ofE∗, the tensorse∗⊗f, withf∈ F, span E∗⊗F. Thus, every element ofE∗⊗F is of the formn e∗⊗fi, for somefi∈F. Assumei=1

n n

α⊗( e∗⊗fi) =α⊗( e∗⊗fi) =f,

i=1 i=1

for somefi, fi∈F and somef∈ Hom(E, F). Then for everyx∈E,

n n

e∗(x)fi = e∗(x)fi =f(x).

i=1 i=1

Since thefi andfi are uniquely determined by the linear map,f, we must havefi =fi and α⊗ is injective. Therefore,α⊗ is a bijection.

Note that in Proposition 23.9, the spaceF may have infinite dimension butE has finite dimension. In view of the canonical isomorphism

Hom(E_1, . . . , En;F)∼= Hom(E1⊗ · · · ⊗En, F)

and the canonical isomorphism (E1⊗ · · · ⊗En)∗ =E∗⊗ · · · ⊗E∗, where theEi’s are finite∼ 1

dimensional, Proposition 23.9 yields the canonical isomorphism

Hom(E_1, . . . , En;F)=E∗ E∗⊗F.∼ 1⊗ · · · ⊗

23.5 Tensor Algebras

The tensor product

V⊗ · · · ⊗V

m

is also denoted asm

V or V⊗m and is called them-th tensor power ofV (withV⊗1 =V , andV⊗0 =K). We can pack all the tensor powers ofV into the “big” vector space,

T(V ) = V⊗m,

m≥0

also denoted T•(V ), to avoid confusion with the tangent bundle. This is an interesting object because we can define a multiplication operation on it which makes it into an algebra called the tensor algebra ofV . WhenV is of finite dimensionn, this space corresponds to the algebra of polynomials with coefficients inK inn noncommuting variables.

Let us recall the definition of an algebra over a field. Let K denote any (commutative) field, although for our purposes, we may assume thatK = R (and occasionally,K = C). Since we will only be dealing with associative algebras with a multiplicative unit, we only define algebras of this kind.

Definition 23.3. Given a field,K, aK-algebra is aK-vector space,A, together with a bilinear operation,·:A×A→A, called multiplication, which makesA into a ring with unity, 1 (or 1A, when we want to be very precise). This means that· is associative and that there is a multiplicative identity element, 1, so that 1·a =a· 1 =a, for alla∈A. Given twoK-algebrasA andB, aK-algebra homomorphism, h:A→B, is a linear map that is also a ring homomorphism, withh(1A) = 1B.

For example, the ring,Mn(K), of alln×n matrices over a field,K, is aK-algebra.

There is an obvious notion of ideal of aK-algebra: An ideal, A⊆A, is a linear subspace ofA that is also a two-sided ideal with respect to multiplication inA. If the fieldK is understood, we usually simply say an algebra instead of aK-algebra.

We would like to define a multiplication operation onT(V ) which makes it into aK- algebra. As

T(V ) = V⊗i,

i≥0

for everyi≥ 0, there is a natural injectionιn:V⊗n T(V ), and in particular, an injection ι

0

:

K

→

T

(

V

). The multiplicative unit,

1

→

, ofT(V ) is the image,ι_0(1), inT(V ) of the unit, 1, of the fieldK. Since everyv∈T(V ) can be expressed as a finite sum

v =ιn1(v_1) +· · · +ιnk(vk),

where vi∈ V⊗ni and theni are natural numbers withni =nj ifi =j, to define multiplication inT(V ), using bilinearity, it is enough to define multiplication operations, ·:V⊗m V⊗n V⊗(m+n), which, using the isomorphisms,V⊗n =ιn(V⊗n), yield multi× −→ m)×ιn(V⊗n)−→ιm+n(V⊗(m+n) ∼

plication operations,·:ιm(V⊗ ). More precisely, we use the canonical isomorphism,

V⊗m V⊗n =V⊗(m+n),⊗ ∼

which defines a bilinear operation,

V⊗m V⊗n V⊗(m+n),× −→

which is taken as the multiplication operation. The isomorphismV⊗m V⊗n =V⊗(m+n) can be established by proving the isomorphisms⊗ ∼

V⊗m V⊗n ∼= V⊗m V⊗ · · · ⊗V⊗ ⊗

n

V⊗m V⊗ · · · ⊗V = V⊗(m+n),⊗ ∼

n

which can be shown using methods similar to those used to proved associativity. Of course, the multiplication,V⊗m V⊗n V⊗(m+n), is defined so that× −→

( v1⊗ · · · ⊗vm)· (w1⊗ · · · ⊗wn) =v1⊗ · · · ⊗vm⊗w1⊗ · · · ⊗wn. (This has to be made rigorous by using isomorphisms involving the associativity of tensor products, for details, see see Atiyah and Macdonald [4].)

Remark: It is important to note that multiplication inT(V ) is not commutative. Also, in all rigor, the unit, 1, ofT(V ) is not equal to 1, the unit of the fieldK. However, in view of the injectionι_0:K→T(V ), for the sake of notational simplicity, we will denote 1 by 1. More generally, in view of the injectionsιn:V⊗n T(V ), we identify elements ofV⊗n with their images inT(V ).→

The algebra, T(V ), satisfies a universal mapping property which shows that it is unique up to isomorphism. For simplicity of notation, leti:V→T(V ) be the natural injection of V intoT(V ).

Proposition 23.10. Given anyK-algebra,A, for any linear map,f :V→A, there is a uniqueK-algebra homomorphism,f :T(V )→A, so that

f =fæi, as in the diagram below: V

EE

i / T(V ) EEEEEEE ff

A Proof. Left an an exercise (use Theorem 23.5).

Most algebras of interest arise as well-chosen quotients of the tensor algebra T(V ). This is true for the exterior algebra, (V ) (also called Grassmann algebra), where we take the quotient ofT(V ) modulo the ideal generated by all elements of the formv⊗v, wherev∈V , and for the symmetric algebra, Sym(V ), where we take the quotient ofT(V ) modulo the ideal generated by all elements of the formv⊗w−w⊗v, wherev, w∈V .

Algebras such asT(V ) are graded, in the sense that there is a sequence of subspaces, V⊗n T(V ), such that⊆ T(V ) = V⊗n

k≥0

and the multiplication,⊗, behaves well w.r.t. the grading, i.e.,⊗:V⊗m V⊗n V⊗(m+n). Generally, a

K

-algebra,

E

, is said to be a

graded algebra

× →

iff there is a sequence of subspaces, En E, such that⊆ E = En

k≥0

(E^0 =K) and the multiplication,·, respects the grading, that is,·:Em En Em+n. Elements inEn are called homogeneous elements of rank (or degree)n.× → In differential geometry and in physics it is necessary to consider slightly more general tensors.

Definition 23.4. Given a vector space,V , for any pair of nonnegative integers, (r, s), the tensor space,Tr,s(V ), of type (r, s), is the tensor product

Tr,s(V ) =V⊗r (V∗)⊗s =V⊗ · · · ⊗V⊗V∗⊗ · · · ⊗V∗,⊗

r s

withT0,0(V ) =K. We also define the tensor algebra,T•,•(V ), as the coproduct

T•,•(V ) = Tr,s(V ).

r,s≥0

Tensors inTr,s(V ) are called homogeneous of degree (r, s).

Note that tensors inTr,0(V ) are just our “old tensors” inV⊗r. We makeT•,•(V ) into an algebra by defining multiplication operations,

Tr1,s1(V )×Tr2,s2(V )−→Tr1+r2,s1+s2(V ), in the usual way, namely: Foru =u1 ⊗ · · · ⊗ur1⊗u∗ u∗1 and1⊗ · · · ⊗v =v1⊗ · · · ⊗vr2⊗v∗ v∗2, let1⊗ · · · ⊗

u⊗v =u1⊗ · · · ⊗ur1⊗v1⊗ · · · ⊗vr2⊗u∗ u∗1⊗v∗ v∗2.1⊗ · · · ⊗1⊗ · · · ⊗ Denote by Hom(

V

r,(V∗)s;W) the vector space of all multilinear maps fromVr (V∗)s

×

toW. Then, we have the universal mapping property which asserts that there is a canonical isomorphism

Hom(Tr,s(V ), W)∼= Hom(Vr,(V∗)s;W). In particular, (Tr,s(V ))∗∼= Hom(Vr,(V∗)s;K).

For finite dimensional vector spaces, the duality of Section 23.4 is also easily extended to the tensor spacesTr,s(V ). We define the pairing

Tr,s(V∗)×Tr,s(V )−→K

as follows: If

v∗ =v∗⊗ · · · ⊗v∗⊗ur+1⊗ · · · ⊗ur+s∈Tr,s(V∗) 1

and

u =u1⊗ · · · ⊗ur⊗v∗+1⊗ · · · ⊗v∗+s∈Tr,s(V ), then

(v∗, u) =v∗(u_1)· · ·v∗+s(ur+s).1

This is a nondegenerate pairing and thus, we get a canonical isomorphism, (Tr,s(V ))∗ =Tr,s(V∗).∼

Consequently, we get a canonical isomorphism,

Tr,s(V∗)∼= Hom(Vr,(V∗)s;K).

Remark: The tensor spaces,Tr,s(V ) are also denotedTrs(V ). A tensor,α∈ Tr,s(V ) is said to be contravariant in the firstr arguments and covariant in the lasts arguments. This terminology refers to the way tensors behave under coordinate changes. Given a basis, (e_1, . . . , en), ofV , if (e∗, . . . , e∗n) denotes the dual basis, then every tensorα∈Tr,s(V ) is_1

given by an expression of the form

α = ai1,...,irei1⊗ · · · ⊗eir⊗e∗1⊗ · · · ⊗e∗s.j1,...,js

i_1,...,ir

j_1,...,js

The tradition in classical tensor notation is to use lower indices on vectors and upper indices on linear forms and in accordance to Einstein summation convention (or Einstein notation) the position of the indices on the coefficients is reversed. Einstein summation convention is to assume that a summation is performed for all values of every index that appears simultaneously once as an upper index and once as a lower index. According to this convention, the tensorα above is written

α =ai1,...,irei1⊗ · · · ⊗eir⊗ej1 ejs.j1,...,js ⊗ · · · ⊗

An older view of tensors is that they are multidimensional arrays of coefficients,

ai1,...,ir ,j_1,...,js

subject to the rules for changes of bases.

Another operation on general tensors, contraction, is useful in differential geometry. Definition 23.5. For allr, s≥ 1, the contraction,ci,j:Tr,s(V )→Tr−1,s−1(V ), with 1≤i≤r and 1≤j≤s, is the linear map defined on generators by

ci,j(u1⊗ · · · ⊗ur⊗v∗ v∗)1⊗ · · · ⊗

= v∗(ui)u1⊗ · · · ⊗ui⊗ · · · ⊗ur⊗v∗ v∗⊗ · · · ⊗v∗,1⊗ · · · ⊗ where the hat over an argument means that it should be omitted.

Let us figure our what isc1,1:T1,1(V )→ R, that isc1,1:V⊗ V∗→ R. If (e_1, . . . , en) is a basis ofV and (e∗, . . . , e∗n) is the dual basis, everyh∈V⊗V∗∼= Hom(V, V ) can be

1

expressed asn

h = aijei⊗e∗.

i,j=1

As

c1,1(ei⊗e∗) =δi,j,

we getn

c1,1(h) = aii = tr(h),

i=1

where tr( h) is the trace ofh, whereh is viewed as the linear map given by the matrix, (aij). Actually, sincec1,1 is defined independently of any basis,c1,1 provides an intrinsic definition of the trace of a linear map,h∈ Hom(V, V ).

Remark: Using the Einstein summation convention, if

α =ai1,...,irei1⊗ · · · ⊗eir⊗ej1 ejs,j1,...,js ⊗ · · · ⊗ then

ck,l(α) =ai1,...,ik−1,i,ik+1...,irei1⊗ · · · ⊗eik⊗ · · · ⊗eir⊗ej1 ejl ejs.j1,...,jl−1,i,jl+1,...,js ⊗ · · · ⊗ ⊗ · · · ⊗

If E andF are twoK-algebras, we know that their tensor product,E⊗F, exists as a vector space. We can makeE⊗F into an algebra as well. Indeed, we have the multilinear map

E ×F×E×F−→E⊗F

given by (a, b, c, d)→ (ac)⊗ (bd), whereac is the product ofa andc inE andbd is the product ofb andd inF. By the universal mapping property, we get a linear map,

E⊗F⊗E⊗F−→E⊗F. Using the isomorphism, E⊗F⊗E⊗F = (E⊗F)⊗ (E⊗F),

∼

we get a linear map, (E⊗F)⊗ (E⊗F)−→E⊗F, and thus, a bilinear map,

(E⊗F)× (E⊗F)−→E⊗F,

which is our multiplication operation inE⊗F. This multiplication is determined by

(a⊗b)· (c⊗d) = (ac)⊗ (bd).

One immediately checks thatE⊗F with this multiplication is aK-algebra. We now turn to symmetric tensors.

23.6 Symmetric Tensor Powers

Our goal is to come up with a notion of tensor product that will allow us to treat symmetric multilinear maps as linear maps. First, note that we have to restrict ourselves to a single vector space,E, rather thenn vector spacesE_1, . . . , En, so that symmetry makes sense. Recall that a multilinear map,f :En F, is symmetric iff→

f(uσ(1), . . . , uσ(n)) =f(u_1, . . . , un),

for all ui∈E and all permutations,σ:{1, . . . , n} → {1, . . . , n}. The group of permutations on{1, . . . , n} (the symmetric group) is denoted Sn. The vector space of all symmetric multilinear maps,f :En F, is denoted by Sn(E;F). Note that S^1(E;F) = Hom(E, F).→

We could proceed directly as in Theorem 23.5, and construct symmetric tensor products from scratch. However, since we already have the notion of a tensor product, there is a more economical method. First, we define symmetric tensor powers.

Definition 23.6. Ann-th symmetric tensor power of a vector spaceE, wheren≥ 1, is a vector spaceS, together with a symmetric multilinear mapÕ:En S, such that, for every

vector space

F

and for every symmetric multilinear map

f

:

E

n

→

F, there is a unique linear mapf :S→F, with→

f(u_1, . . . , un) =f (Õ(u_1, . . . , un)),

for allu_1, . . . , un∈E, or for short

f =f æÕ.

Equivalently, there is a unique linear mapf such that the following diagram commutes:

E

n

CC

Õ S CCCCCC ff

F 23.6. SYMMETRIC TENSOR POWERS 625

First, we show that any two symmetricn-th tensor powers (S_1, Õ_1) and (S_2, Õ_2) forE, are isomorphic.

Proposition 23.11. Given any two symmetricn-th tensor powers (S_1, Õ_1) and (S_2, Õ_2) for E, there is an isomorphismh:S1→S_2 such that

Õ_2 =hæÕ_1.

Proof. Replace tensor product byn-th symmetric tensor power in the proof of Proposition 23.4.

We now give a construction that produces a symmetricn-th tensor power of a vector spaceE.

Theorem 23.12. Given a vector spaceE, a symmetricn-th tensor power (Symn(E), Õ) forE can be constructed (n≥ 1). Furthermore, denotingÕ(u_1, . . . , un) asu1 · · · un, the symmetric tensor power Symn(E) is generated by the vectors u1 · · · un, where u_1, . . . , un∈E, and for every symmetric multilinear mapf :En F, the unique linear mapf : Symn(E)→F such thatf =f æÕ, is defined by→

f (u1 · · · un) =f(u_1, . . . , un),

on the generatorsu1 · · · un of Symn(E).

Proof. The tensor powerE⊗n is too big, and thus, we define an appropriate quotient. Let C be the subspace ofE⊗n generated by the vectors of the form

u1⊗ · · · ⊗un−uσ(1)⊗ · · · ⊗uσ(n), for allui∈ E, and all permutationsσ:{1, . . . , n} → {1, . . . , n}. We claim that the quotient space (

E

n

⊗ )/C does the job.

Letp:E⊗n (E⊗n)/C be the quotient map. LetÕ:En (E⊗n)/C be the map→ →

(u_1, . . . , un)→p(u1⊗ · · · ⊗un),

or equivalently,Õ =pæÕ_0, whereÕ_0(u_1, . . . , un) =u1⊗ · · · ⊗un.

Let us denote Õ(u_1, . . . , un) asu1 · · · un. It is clear thatÕ is symmetric. Since the vectorsu1⊗ · · · ⊗un generateE⊗n, andp is surjective, the vectorsu1 · · · un generate (E⊗n)/C.

Given any symmetric multilinear mapf :En F, there is a linear mapf⊗:E⊗n F such thatf =f⊗æÕ_0, as in the diagram below:→ →

Õ0 E⊗n EnFFF

f f⊗FFFFFF

F

However, since f is symmetric, we havef⊗(z) = 0 for everyz∈E⊗n. Thus, we get an induced linear maph: (E⊗n)/C→F, such thath([z]) =f⊗(z), where [z] is the equivalence class in (E⊗n)/C ofz∈E⊗n:

However, if a linear mapf (E⊗n)/C, we must have En pæÕ0 / (E⊗n)/CJJJJ

f hJJJJJJJ F

: (E⊗n)/C→F exists, since the vectorsu1 · · · un generate

which shows thath andf n-th tensor power ofE. f (u1 · · · un) =f(u_1, . . . , un),

agree. Thus, Symn(E) = (E⊗n)/C andÕ constitute a symmetric

Again, the actual construction is not important. What is important is that the symmetric n-th power has the universal mapping property with respect to symmetric multilinear maps.

Remark: The notation for the commutative multiplication of symmetric tensor powers is not standard. Another notation commonly used is·. We often abbreviate “symmetric tensor power” as “symmetric power”. The symmetric power, Symn(E), is also denoted SymnE or S(E). To be consistent with the use of , we could have used the notationnE. Clearly, Sym^1(E)=E and it is convenient to set Sym^0(E) =K.∼

The fact that the mapÕ:En Symn(E) is symmetric and multinear, can also be expressed as follows:→

u1 · · · (vi +wi) · · · un = (u1 · · · vi · · · un) + (u1 · · · wi · · · un),

u1 · · ·(λu

uσ(1) · · · uσ(n) =u1 · · · un, · · · un),i) · · · un =λ(u1 · · · ui

for all permutationsσ∈ Sn.

The last identity shows that the “operation” is commutative. Thus, we can view the symmetric tensoru1 · · · un as a multiset.

Theorem 23.12 yields a canonical isomorphism

Hom(Symn(E), F)= (En;F),∼S

between the vector space of linear maps Hom(Symn(E), F), and the vector space of symmetric multilinear mapsS(En;F), via the linear map− æÕ defined by

h→hæÕ,

where hn(E), F). Indeed,hæÕ is clearly symmetric multilinear, and since by∈ Hom(Sym

Theorem 23.12, for every symmetric multilinear mapf∈ S(En;F), there is a unique linear mapf ∈ Hom(Symn(E), F) such thatf =f æÕ, the map− æÕ is bijective. As a matter of fact, its inverse is the map

f→f . In particular, whenF =K, we get a canonical isomorphism (Symn(E))∗∼= Sn(E;K).

Symmetric tensors in Symn(E) are also called symmetricn-tensors, and tensors of the formu1 · · · un, whereui∈E, are called simple (or decomposable) symmetricn-tensors. Those symmetricn-tensors that are not simple are often called compound symmetricn- tensors.

Given two linear mapsf :E→E andg:E→E , we can defineh:E×E→ Sym^2(E ) by

h(u, v) =f(u) g(v).

It is immediately verified thath is symmetric bilinear, and thus, it induces a unique linear map

f g: Sym^2(E)→ Sym^2(E ), such that

(f g)(u v) =f(u) g(u).

If we also have linear mapsf :E→E andg :E→E , we can easily verify that

(fæf) (gæg) = (f g )æ (f g).

The generalization to the symmetric tensor productf1 · · · fn ofn≥ 3 linear maps fi:E→E is immediate, and left to the reader.

23.7 Bases of Symmetric Powers

The vectors u1 · · · un, whereu_1, . . . , un∈E, generate Symn(E), but they are not linearly independent. We will prove a version of Proposition 23.6 for symmetric tensor powers. For this, recall that a (finite) multiset over a setI is a functionM :I→ N, such thatM(i) = 0 for finitely manyi∈I, and that the set of all multisets overI is denoted as N(I). We let dom(M) ={i∈I|M(i) = 0}, which is a finite set. Then, for any multisetM∈ N(I), note that the sumi∈IM(i) makes sense, sincei∈IM(i) =i∈dom(M)M(i), and dom(M) is finite. For every multisetM∈ N(I), for anyn≥ 2, we define the setJM of functions η:{1, . . . , n} → dom(M), as follows:

JM ={η|η:{1, . . . , n} → dom(M),|η−1(i)| =M(i), i∈ dom(M), M(i) =n}.

i∈I

In other words, ifi∈IM(i) =n and dom(M) ={i_1, . . . , ik},1 any functionη∈JM specifies a sequence of lengthn, consisting ofM(i_1) occurrences ofi_1,M(i_2) occurrences ofi_2, . . ., M(ik) occurrences ofik. Intuitively, anyη defines a “permutation” of the sequence (of length n)

i_1, . . . , i_1, i_2, . . . , i_2, . . . , ik, . . . , ik .

M(i_1) M(i_2) M(ik)

Given anyk≥ 1, and anyu∈E, we denote

u · · · u

k

as u k.

We can now prove the following Proposition.

Proposition 23.13. Given a vector spaceE, if (ui)i∈I is a basis forE, then the family of vectors

u

M(i_1) u M(ik)

i1 · · · ik

M(I),i∈IM(i)=n,{i_1,...,ik}=dom(M)∈N

is a basis of the symmetricn-th tensor power Symn(E).

Proof. The proof is very similar to that of Proposition 23.6. For any nontrivial vector space F, for any family of vectors

(wM)M∈N(I),i∈IM(i)=n,

we show the existence of a symmetric multilinear maph: Symn(E)→F, such that for every M∈ N(I) withi∈IM(i) =n, we have

h(u M(i1) u M(ik)) =wM,i1 · · · ik

where{i_1, . . . , ik} = dom(M). We define the mapf :En F as follows:→

f( v^1 u^1 , . . . , vn unjn) = v^1 vn wM.1 1 jn (1)· · · η(n)

j1∈I jn∈IM∈N(I) η∈JM

i∈IM(i)=n

1Note that must have k≤n.

It is not difficult to verify that f is symmetric and multilinear. By the universal mapping property of the symmetric tensor product, the linear mapf : Symn(E)→ F such that f =f æÕ, is the desired maph. Then, by Proposition 23.3, it follows that the family

u

M(i_1) u M(ik)

i1 · · · ik

M(I),i∈IM(i)=n,{i_1,...,ik}=dom(M)∈N

is linearly independent. Using the commutativity of , we can also show that these vectors generate Symn(E), and thus, they form a basis for Symn(E). The details are left as an exercise.

As a consequence, when I is finite, say of sizep = dim(E), the dimension of Symn(E) is the number of finite multisets (j_1, . . . , jp), such thatj_1 +· · · +jp =n,jk≥ 0. We leave as an exercise to show that this number is p+n−1 . Thus, if dim(E) =p, then the dimension of Symn(E) is p+n−1 n n, which ispn. In particular, whenn . Compare with the dimension ofE⊗

p = 2, the dimension of Symn(E) isn + 1. This can also be seen directly.

Remark: The number p+n−1 is also the number of homogeneous monomialsn

X

j1 Xjp

1· · ·p

of total degree n inp variables (we havej_1 +· · · +jp =n). This is not a coincidence! Symmetric tensor products are closely related to polynomials (for more on this, see the next remark).

Given a vector spaceE and a basis (ui)i∈I forE, Proposition 23.13 shows that every symmetric tensorz∈ Symn(E) can be written in a unique way as

z = λMu M(i1) u M(ik),i1 · · · ik

M

∈N(I)

M(i)=ni∈I

i_1,...,ik}=dom(M){

for some unique family of scalarsλM∈K, all zero except for a finite number. This looks like a homogeneous polynomial of total degreen, where the monomials of total degreen are the symmetric tensors

u M(i_1) u M(ik),i1 · · · ik

in the “indeterminates”ui, wherei∈I (recall thatM(i_1) +· · · +M(ik) =n). Again, this is not a coincidence. Polynomials can be defined in terms of symmetric tensors.

23.8 Some Useful Isomorphisms for Symmetric Powers

We can show the following property of the symmetric tensor product, using the proof technique of Proposition 23.7:

n

Symn(E⊕F)= Symk(E)⊗ Symn−k(F).∼

k=0

23.9 Duality for Symmetric Powers

In this section, all vector spaces are assumed to have finite dimension. We define a nondegenerate pairing, Symn(E∗)× Symn(E)−→K, as follows: Consider the multilinear map,

(E∗)n En K,× −→

given by

(v∗, . . . , v∗, u_1, . . . , un)→ v∗

σ∈Sn · · ·v∗σ(n)(un). 1 (1)(u_1)

Note that the expression on the right-hand side is “almost” the determinant, det( v∗(ui)), except that the sign sgn(σ) is missing (where sgn(σ) is the signature of the permutation σ, that is, the parity of the number of transpositions into whichσ can be factored). Such an expression is called a permanent. It is easily checked that this expression is symmetric w.r.t. theui’s and also w.r.t. thev∗. For any fixed (v∗, . . . , v∗n)∈ (E∗)n, we get a symmetric_1

multinear map,

lv∗,...,v∗: (u_1, . . . , un)→σ∈Sn · · ·v∗(n)(un),1 n v∗σ(1)(u_1)

fromEn toK. The maplv∗,...,v∗ extends uniquely to a linear map,Lv∗,...,v∗: Symn(E)→K.1 n 1 n Now, we also have the symmetric multilinear map,

(v∗, . . . , v∗n)→Lv∗,...,v∗,

1 1 n

from (E∗)n to Hom(Symn(E), K), which extends to a linear map,L, from Symn(E∗) to Hom(Symn(E), K). However, in view of the isomorphism,

Hom(U⊗V, W)∼= Hom(U,Hom(V, W)),

we can viewL as a linear map,

L: Symn(E∗)⊗ Symn(E)−→K,

which corresponds to a bilinear map,

Symn(E∗)× Symn(E)−→K. 23.9. DUALITY FOR SYMMETRIC POWERS 631

Now, this pairing in nondegenerate. This can be done using bases and we leave it as an exercise to the reader (see Knapp [62], Appendix A). Therefore, we get a canonical isomorphism, (Symn(E))∗∼= Symn(E∗).

Since we also have an isomorphism

(Symn(E))∗∼= Sn(E, K), we get a canonical isomorphism

Symn(E∗)∼= Sn(E, K)

which allows us to interpret symmetric tensors overE∗ as symmetric multilinear maps.

Remark: The isomorphism,µ: Symn(E∗)∼= Sn(E, K), discussed above can be described explicity as the linear extension of the map given by

µ(v∗ v∗n)(u_1, . . . , un) = v∗σ(1)(u_1)· · ·v∗(n)(un).1 · · · σ∈Sn

Now, the map from En to Symn(E) given by (u_1, . . . , un)→ u1 · · · un yields a surjection,π:E⊗n Symn(E). Because we are dealing with vector spaces, this map has→

some section, that is, there is some injection,ι: Symn(E)→E⊗n, withπæι = id. If our field, K, has characteristic 0, then there is a special section having a natural definition involving a symmetrization process defined as follows: For every permutation,σ, we have the map, rσ:En E⊗n, given by→

rσ(u_1, . . . , un) =uσ(1)⊗ · · · ⊗uσ(n).

Asrσ is clearly multilinear,rσ extends to a linear map,rσ:E⊗n E⊗n, and we get a map, S

n

×

E

⊗

n E⊗n, namely,→

−→ σ·z =rσ(z).

It is immediately checked that this is a left action of the symmetric group, Sn, onE⊗n and the tensorsz∈E⊗n such that

σ·z =z, for all σ∈ Sn

are called symmetrized tensors. We define the map,ι:En E⊗n, by→

ι(u_1, . . . , un) = 1 σ· (u1⊗ · · · ⊗un) = 1 uσ(1)⊗ · · · ⊗uσ(n).

n!σ∈Sn n!σ∈Sn

As the right hand side is clearly symmetric, we get a linear map, ι: Symn(E)→ E⊗n. Clearly, ι(Symn(E)) is the set of symmetrized tensors inE⊗n. If we consider the map, S =ιæπ:E⊗n E⊗n, it is easy to check thatSæS =S. Therefore,S is a projection and−→

by linear algebra, we know that

E⊗n =S(E⊗n)⊕ KerS =ι(Symn(E))⊕ KerS.

It turns out that Ker S =E⊗n I = Kerπ, where I is the two-sided ideal ofT(E) generated∩

by all tensors of the formu^2 (for example, see Knapp [62], Appendix A). Therefore,ι is injective,⊗v−v⊗u∈E⊗

E⊗n =ι(Symn(E))⊕E⊗n I =ι(Symn(E))⊕ Kerπ,∩

and the symmetric tensor power, Symn(E), is naturally embedded intoEn.⊗

23.10 Symmetric Algebras

As in the case of tensors, we can pack together all the symmetric powers, Symn(V ), into an algebra,

Sym(V ) = Symm(V ),

m≥0

called the symmetric tensor algebra ofV . We could adapt what we did in Section 23.5 for general tensor powers to symmetric tensors but since we already have the algebra,T(V ), we can proceed faster. If I is the two-sided ideal generated by all tensors of the form u⊗v−v⊗u∈V⊗2, we set

Sym •(V ) =T(V )/I.

Then, Sym•(V ) automatically inherits a multiplication operation which is commutative and sinceT(V ) is graded, that is,

T(V ) = V⊗m,

m≥0

we have

Sym•(V ) = V⊗m/(I∩V⊗m).

m≥0

However, it is easy to check that

Symm(V )=V⊗m/(I∩V⊗m),∼

so

Sym•(V )∼= Sym(V ).

WhenV is of finite dimension, n, T(V ) corresponds to the algebra of polynomials with

coefficients inK inn variables (this can be seen from Proposition 23.13). WhenV is of

infinite dimension and (ui)i∈I is a basis ofV , the algebra, Sym(V ), corresponds to the

algebra of polynomials in infinitely many variables inI. What’s nice about the symmetric

tensor algebra, Sym(V ), is that it provides an intrinsic definition of a polynomial algebra in

any set,I, of variables.

It is also easy to see that Sym(V ) satisfies the following universal mapping property: 23.10. SYMMETRIC ALGEBRAS 633

Proposition 23.14. Given any commutativeK-algebra,A, for any linear map,f :V→A, there is a uniqueK-algebra homomorphism,f : Sym(V )→A, so that

f =fæi,

as in the diagram below: V

HHH

i / Sym(V ) HHHHHHH ff

A

Remark: IfE is finite-dimensional, recall the isomorphism,µ: Symn(E∗)−→ Sn(E, K), defined as the linear extension of the map given by

µ(v∗ v∗n)(u_1, . . . , un) = v∗σ(1)(u_1)· · ·v∗(n)(un),1 · · · σ∈Sn

Now, we have also a multiplication operation, Symm(E∗)×Symn(E∗)−→ Symm+n(E∗). The following question then arises:

Can we define a multiplication, Sm(E, K)× Sn(E, K)−→ Sm+n(E, K), directly on symmetric multilinear forms, so that the following diagram commutes:

Symm(E∗)× Symn(E∗) Symm+n(E∗)

µ×µ µ

Sm(E, K)× Sn(E, K) · Sm+n(E, K).

The answer is yes! The solution is to define this multiplication such that, forf∈ Sm(E, K) andg∈ Sn(E, K),

(f·g)(u_1, . . . , um+n) = f(uσ(1), . . . , uσ(m))g(uσ(m+1), . . . , uσ(m+n)),

σ∈shuffle(m,n)

where shuffle( m, n) consists of all (m, n)-“shuffles”, that is, permutations,σ, of{1, . . . m+n}, such thatσ(1)<· · ·< σ(m) andσ(m + 1)<· · ·< σ(m +n). We urge the reader to check this fact.

Another useful canonical isomorphim (ofK-algebras) is Sym(E⊕F)∼= Sym(E)⊗ Sym(F).

23.11 Exterior Tensor Powers

We now consider alternating (also called skew-symmetric) multilinear maps and exterior tensor powers (also called alternating tensor powers), denotedn(E). In many respect, alternating multilinear maps and exterior tensor powers can be treated much like symmetric tensor powers except that the sign, sgn(σ), needs to be inserted in front of the formulae valid for symmetric powers. Roughly speaking, we are now in the world of determinants rather than in the world of permanents. However, there are also some fundamental differences, one of which being that the exterior tensor power,n(E), is the trivial vector space, (0), when E is finite-dimensional and whenn > dim(E). As in the case of symmetric tensor powers, since we already have the tensor algebra,T(V ), we can proceed rather quickly. But first, let us review some basic definitions and facts.

Definition 23.7. Letf :En F be a multilinear map. We say thatf alternating iff→

f(u_1, . . . , un) = 0 wheneverui = ui+1, for somei with 1≤ i≤ n− 1, for allui∈ E, that is,f(u_1, . . . , un) = 0 whenever two adjacent arguments are identical. We say thatf is skew-symmetric (or anti-symmetric) iff

f(uσ(1), . . . , uσ(n)) = sgn(σ)f(u_1, . . . , un),

for every permutation,σ∈ Sn, and allui∈E.

Forn = 1, we agree that every linear map, f :E→ F, is alternating. space of all multilinear alternating maps,

f

:

E

n F, is denoted Altn

→ (E;F). Alt^1(E;F) = Hom(E, F). The following basic proposition shows the relationship between alternation and skew-symmetry.

Proposition 23.15. Letf :En F be a multilinear map. Iff is alternating, then the following properties hold:→

The vector Note that

(1) For alli, with 1≤i≤n− 1,

f(. . . , ui, ui+1, . . .) =−f(. . . , ui+1, ui, . . .).

(2) For every permutation,σ∈ Sn,

f(uσ(1), . . . , uσ(n)) = sgn(σ)f(u_1, . . . , un).

(3) For alli, j, with 1≤i < j≤n,

f(. . . , ui, . . . uj, . . .) = 0 wheneverui =uj.

Moreover, if our field,K, has characteristic different from 2, then every skew-symmetric multilinear map is alternating.

Proof. (i) By multilinearity applied twice, we have f(. . . , ui +ui+1, ui +ui+1, . . .) =f(. . . , ui, ui, . . .) +f(. . . , ui, ui+1, . . .)

+f(. . . , ui+1, ui, . . .) +f(. . . , ui+1, ui+1, . . .).

Since f is alternating, we get

0 =f(. . . , ui, ui+1, . . .) +f(. . . , ui+1, ui, . . .),

that is,f(. . . , ui, ui+1, . . .) =−f(. . . , ui+1, ui, . . .).

(ii) Clearly, the symmetric group, Sn, acts on Altn(E;F) on the left, via σ·f(u_1, . . . , un) =f(uσ(1), . . . , uσ(n)).

Consequently, as Sn is generated by the transpositions (permutations that swap exactly two elements), since for a transposition, (ii) is simply (i), we deduce (ii) by induction on the number of transpositions inσ.

(iii) There is a permutation,σ, that sendsui anduj respectively tou_1 andu_2. Asf is alternating,

f(uσ(1), . . . , uσ(n)) = 0. However, by (ii),

f(u_1, . . . , un) = sgn(σ)f(uσ(1), . . . , uσ(n)) = 0.

Now, whenf is skew-symmetric, ifσ is the transposition swappingui andui+1 =ui, as sgn(σ) =−1, we get

f(. . . , ui, ui, . . .) =−f(. . . , ui, ui, . . .),

so that

2f(. . . , ui, ui, . . .) = 0,

and in every characteristic except 2, we conclude thatf(. . . , ui, ui, . . .) = 0, namely,f is

alternating.

Proposition 23.15 shows that in every characteristic except 2, alternating and skewsymmetric multilinear maps are identical. Using Proposition 23.15 we easily deduce the following crucial fact:

Proposition 23.16. Letf :En F be an alternating multilinear map. For any families of→

vectors, (u_1, . . . , un) and (v_1, . . . , vn), withui, vi∈E, if

n

vj = aijui, 1≤j≤n,

i=1

then

f(v_1, . . . , vn) = sgn(σ)aσ(1),1· · ·aσ(n),n f(u_1, . . . , un) = det(A)f(u_1, . . . , un),

σ∈Sn

whereA is then×n matrix,A = (aij). Proof. Use property (ii) of Proposition 23.15.

We are now ready to define and construct exterior tensor powers.

Definition 23.8. Ann-th exterior tensor power of a vector space,E, wheren≥ 1, is a vector space,A, together with an alternating multilinear map,Õ:En A, such that, for→ every vector space,F, and for every alternating multilinear map,f :En F, there is a unique linear map,f∧:A→F, with→

f(u_1, . . . , un) =f∧(Õ(u_1, . . . , un)),

for allu_1, . . . , un∈E, or for short f =f∧æÕ.

Equivalently, there is a unique linear mapf∧ such that the following diagram commutes:

E

n

CC

Õ A

CCCCCC f∧f

F

First, we show that any twon-th exterior tensor powers (A_1, Õ_1) and (A_2, Õ_2) forE, are isomorphic.

Proposition 23.17. Given any twon-th exterior tensor powers (A_1, Õ_1) and (A_2, Õ_2) for E, there is an isomorphismh:A1→A_2 such that

Õ_2 =hæÕ_1.

Proof. Replace tensor product byn exterior tensor power in the proof of Proposition 23.4.

We now give a construction that produces ann-th exterior tensor power of a vector space E.

Theorem 23.18. Given a vector spaceE, ann-th exterior tensor power ( n(E), Õ) forE can be constructed (n≥ 1). Furthermore, denotingÕ(u_1, . . . , un) asu1∧· · ·∧un, the exterior tensor powern(E) is generated by the vectorsu1∧ · · · ∧un, whereu_1, . . . , un∈E, and for every alternating multilinear mapf :En F, the unique linear mapf∧:n(E)→F such thatf =f∧æÕ, is defined by→

f∧(u1∧ · · · ∧un) =f(u_1, . . . , un),

on the generators u1∧ · · · ∧un ofn(E). Proof sketch. We can give a quick proof using the tensor algebra,T(E). let Ia be the twosided ideal ofT(E) generated by all tensors of the formu⊗u∈E⊗2. Then, let

n

(E) =E⊗n/(Ia∩E⊗n)

and letπ be the projection,π:E⊗n n(E). If we letu1∧ · · · ∧un =π(u1⊗ · · · ⊗un), it is easy to check that (

n

→

(E),∧) satisfies the conditions of Theorem 23.18.

Remark: We can also define

n

(E) =T(E)/Ia = (E),

n≥0

the exterior algebra ofE. This is the skew-symmetric counterpart of Sym(E) and we will study it a little later.

For simplicity of notation, we may write tensor power” as “exterior power”. Clearly, K.

nE forn(E). We also abbreviate “exterior1(E) =E and it is convenient to set 0(E) =∼

The fact that the mapÕ:En n(E) is alternating and multinear, can also be expressed as follows:→

u1∧ · · · ∧ (ui +vi)∧ · · · ∧un = (u1 ∧ · · · ∧ui∧ · · · ∧un) + (u1∧ · · · ∧vi∧ · · · ∧un), u1∧ · · · ∧ (λui)∧ · · · ∧un = λ(u1∧ · · · ∧ui∧ · · · ∧un),

uσ(1)∧ · · · ∧uσ(n) = sgn(σ)u1∧ · · · ∧un,

for allσ∈ Sn.

Theorem 23.18 yields a canonical isomorphism

n

Hom( (E), F)∼= Altn(E;F),

between the vector space of linear maps Hom( n(E), F), and the vector space of alternating multilinear maps Altn(E;F), via the linear map− æÕ defined by

whereh h→hæÕ,

n(E), F). In particular, whenF =K, we get a canonical isomorphism∈ Hom(

n∗

(E)n(E;K).∼= Alt

Tensors

α

n

∈ (E) are called alternatingn-tensors or alternating tensors of degreen and we write deg(α) =n. Tensors of the formu1∧ · · · ∧un, whereui∈E, are called simple (or decomposable) alternatingn-tensors. Those alternatingn-tensors that are not simple are

often called

compound alternating

n

-tensors

. Simple tensors

u

1

∧ · · · ∧

u

n

n∈ (E) are also calledn-vectors and tensors inn(E∗) are often called (alternating)n-forms. Given two linear maps

f

:

E

→

E

and

g

:

E

→

E

, we can define

h

:

E × E

2

→ (E ) by

h(u, v) =f(u)∧g(v).

It is immediately verified thath is alternating bilinear, and thus, it induces a unique linear map

2 2

f∧g: (E)→ (E ), such that (f∧g)(u∧v) =f(u)∧g(u).

If we also have linear mapsf :E→E andg :E→E , we can easily verify that

(fæf)∧ (gæg) = (f∧g )æ (f∧g).

The generalization to the alternating productf1∧· · ·∧fn ofn≥ 3 linear mapsfi:E→E is immediate, and left to the reader.

23.12 Bases of Exterior Powers

Let E be any vector space. For any basis, (ui)i∈Σ, forE, we assume that some total ordering, ≤, on Σ, has been chosen. Call the pair ((ui)i∈Σ,≤) an ordered basis. Then, for any nonempty finite subset,I⊆ Σ, let

uI =ui1∧ · · · ∧uim,

whereI ={i_1, . . . , im}, withi_1<· · ·< im.

Sincen(E) is generated by the tensors of the formv1 ∧ · · · ∧vn, withvi∈E, in view of skew-symmetry, it is clear that the tensorsuI, with|I| =n, generaten(E). Actually, they form a basis.

Proposition 23.19. Given any vector space,E, ifE has finite dimension,d = dim(E), then for alln > d, the exterior powern(E) is trivial, that isn(E) = (0). Otherwise, for every ordered basis, ((ui)i∈Σ,≤), the family, (uI), is basis ofn(E), whereI ranges over finite nonempty subsets of Σ of size|I| =n.

23.12. BASES OF EXTERIOR POWERS 639

Proof. First, assume thatE has finite dimension,d = dim(E) and thatn > d. We know thatn(E) is generated by the tensors of the formv1∧ · · · ∧ vn, withvi∈E. Ifu_1, . . . , ud is a basis ofE, as everyvi is a linear combination of theuj, when we expandv1∧ · · · ∧vn using multilinearity, we get a linear combination of the form

v1∧ · · · ∧vn = λ(j1,...,jn)uj1∧ · · · ∧ujn,

(j_1,...,jn)

where each ( j_1, . . . , jn) is some sequence of integersjk∈ {1, . . . , d}. Asn > d, each sequence (j_1, . . . , jn) must contain two identical elements. By alternation,uj1∧ · · · ∧ujn = 0 and so, v1∧ · · · ∧vn = 0. It follows thatn(E) = (0).

Now, assume that either dim( E) =d and thatn≤d or thatE is infinite dimensional. The argument below shows that theuI are nonzero and linearly independent. As usual, let u∗∈E∗ be the linear form given by

u∗(uj) =δij.

For any nonempty subset,I ={i_1, . . . , in} ⊆ Σ, withi_1<· · ·< in, letlI be the map given by

lI(v_1, . . . , vn) = det(u∗j(vk)),

for allvk∈E. AslI is alternating multilinear, it induces a linear map,LI :n(E)→K. Observe that for any nonempty finite subset,J⊆ Σ, with|J| =n, we have

L

I

(

u

J

) =

1 ifI =J

0 ifI =J.

Note that when dim(E) =d andn≤d, the formsu∗1, . . . , u∗n are all distinct so, the above does hold. SinceLI(uI) = 1, we conclude thatuI = 0. Now, if we have a linear combination,

λIuI = 0,

I

where the above sum is finite and involves nonempty finite subset,I⊆ Σ, with|I| =n, for every suchI, when we applyLI we get

λI = 0,

proving linear independence.

As a corollary, ifE is finite dimensional, say dim(E) =d and if 1≤n≤d, then we have

n ndim( (E)) =d and ifn > d, then dim( n(E)) = 0.

Remark: Whenn = 0, if we setu∅ = 1, then (u∅) = (1) is a basis of^0(V ) =K. It follows from Proposition 23.19 that the family, (uI)I, whereI⊆ Σ ranges over finite subsets of Σ is a basis of

(

V

) =

n

n≥0 (V ).

As a corollary of Proposition 23.19 we obtain the following useful criterion for linear independence:

Proposition 23.20. For any vector space, E, the vectors, u_1, . . . , un∈ E, are linearly independent iffu1∧ · · · ∧un = 0.

Proof. Ifu1∧ · · · ∧un = 0, thenu_1, . . . , un must be linearly independent. Otherwise, some ui would be a linear combination of the otheruj’s (withj =i) and then, as in the proof of Proposition 23.19,u1∧ · · · ∧un would be a linear combination of wedges in which two vectors are identical and thus, zero.

Conversely, assume thatu_1, . . . , un are linearly independent. Then, we have the linear forms,u∗∈E∗, such that

u∗(uj) =δi,j 1≤i, j≤n. As in the proof of Proposition 23.19, we have a linear map,Lu1,...,un:n(E)→K, given by

Lu1,...,un(v1∧ · · · ∧vn) = det(u∗(vi)),

for all

v

1

∧ · · · ∧

v

n

n∈ (E). As,

Lu1,...,un(u1∧ · · · ∧un) = 1,

we conclude thatu1∧ · · · ∧un = 0.

Proposition 23.20 shows that, geometrically, every nonzero wedge,u1∧ · · · ∧un, corresponds to some oriented version of ann-dimensional subspace ofE.

23.13 Some Useful Isomorphisms for Exterior Powers

We can show the following property of the exterior tensor product, using the proof technique of Proposition 23.7:

n n k n−k

(E⊕F)= (E)⊗ (F).∼

k=0

23.14. DUALITY FOR EXTERIOR POWERS 641

23.14 Duality for Exterior Powers

In this section, all vector spaces are assumed to have finite dimension. We define a nondegenerate pairing,

n(E∗)n

× (E)−→K, as follows: Consider the multilinear map, (E∗)n En K,× −→

given by

(v∗, . . . , v∗, u_1, . . . , un)→σ∈Sn · · ·v∗σ(n)(un) = det(v∗(ui)).1 sgn(σ)v∗(1)(u_1)

It is easily checked that this expression is alternating w.r.t. theui’s and also w.r.t. thev∗. For any fixed (v∗, . . . , v∗n)∈ (E∗)n, we get an alternating multinear map,1

lv∗,...,v∗: (u_1, . . . , un)→ det(v∗(ui)),1 n

fromEn toK. By the argument used in the symmetric case, we get a bilinear map,

n n

(E∗)× (E)−→K.

Now, this pairing in nondegenerate. This can be done using bases and we leave it as an exercise to the reader. Therefore, we get a canonical isomorphism,

n n

( (E))∗ = (E∗).∼

Since we also have a canonical isomorphism

n

( (E))∗∼= Altn(E;K),

we get a canonical isomorphism

n

(E∗)∼= Altn(E;K)

which allows us to interpret alternating tensors overE∗ as alternating multilinear maps. The isomorphism,µ:n(E∗)∼= Altn(E;K), discussed above can be described explicity as the linear extension of the map given by

µ(v∗∧ · · · ∧v∗n)(u_1, . . . , un) = det(v∗(ui)).1

Remark: Variants of our isomorphism,µ, are found in the literature. For example, there is a version,µ , where

µ = 1µ,

n!

with the factor^1 added in front of the determinant. Each version has its its own meritsn!

and inconvenients. Morita [80] usesµ because it is more convenient thanµ when dealing with characteristic classes. On the other hand, when usingµ , some extra factor is needed in defining the wedge operation of alternating multilinear forms (see Section 23.15) and for exterior differentiation. The versionµ is the one adopted by Warner [109], Knapp [62], Fulton and Harris [40] and Cartan [18, 19].

Iff :E→F is any linear map, by transposition we get a linear map,f :F∗→E∗, given by

f (v∗) =v∗æf, v∗∈F∗. Consequently, we have

f (v∗)(u) =v∗(f(u)), for allu∈E and allv∗∈F∗.

For anyp≥ 1, the map,

(u_1, . . . , up)→f(u_1)∧ · · · ∧f(up),

from

E

n topF is multilinear alternating, so it induces a linear map,pf :pEp → F, defined on generators by

p

f (u1∧ · · · ∧up) =f(u_1)∧ · · · ∧f(up).

Combining

p and duality, we get a linear map,pf :pF∗ p

→

E∗, defined on generators by

p

f (v∗ v∗) =f (v∗)∧ · · · ∧f (v∗).1∧ · · · ∧ 1

Proposition 23.21. Iff :E→F is any linear map between two finite-dimensional vector spaces,E andF, then

p p

µ f (ω) (u_1, . . . , up) =µ(ω)(f(u_1), . . . , f(up)), ω∈ F∗, u_1, . . . , up∈E.

Proof. It is enough to prove the formula on generators. By definition ofµ, we have

p

µ f (v∗ v∗) (u_1, . . . , up) = µ(f (v∗)∧ · · · ∧f (v∗))(u_1, . . . , up)1∧ · · · ∧ 1

= det(f (v∗)(ui))

= det(v∗(f(ui)))

= µ(v∗ v∗)(f(u_1), . . . , f(up)),1∧ · · · ∧

as claimed.

The mappf is often denotedf∗, although this is an ambiguous notation sincep is dropped. Proposition 23.21 gives us the behavior off∗ under the identification ofpE∗ and Altp(E;K) via the isomorphismµ.

As in the case of symmetric powers, the map from En ton(E) given by (u_1, . . . , un)→ u1∧ · · · ∧un yields a surjection,π:E⊗n n(E). Now, this map has some section so there

is some injection,

ι

:

n

(

E

)

→

E

⊗

n

→

, with πæι = id. If our field,K, has characteristic 0, then there is a special section having a natural definition involving an antisymmetrization process.

Recall that we have a left action of the symmetric group, Sn, onE⊗n. The tensors, z∈E⊗n, such that

σ·z = sgn(σ)z, for all σ∈ Sn

are called

antisymmetrized

tensors. We define the map,

ι

:

E

n E⊗n

→

ι(u

, by

) = 1 sgn(σ)uσ(1)⊗ · · · ⊗uσ(n).1, . . . , un n!σ∈Sn

As the right hand side is clearly an alternating map, we get a linear map, ι:n(E)→E⊗n. Clearly,ι( n(E)) is the set of antisymmetrized tensors inE⊗n. If we consider the map, A =ιæπ:E⊗n E⊗n, it is easy to check thatAæA =A. Therefore,A is a projection−→

and by linear algebra, we know that

n

E⊗n =A(E⊗n)⊕ KerA =ι( (A))⊕ KerA.

It turns out that Ker A = E⊗n Ia = Kerπ, where Ia is the two-sided ideal ofT(E)∩

generated by all tensors of the formu^2 (for example, see Knapp [62], Appendix A). Therefore,ι is injective,⊗u∈E⊗

n n

E⊗n =ι( (E))⊕E⊗n I =ι( (E))⊕ Kerπ,∩

and the exterior tensor power,n(E), is naturally embedded intoE⊗n.

23.15 Exterior Algebras

As in the case of symmetric tensors, we can pack together all the exterior powers,n(V ), into an algebra,m

(V ) = (V ),

m≥0

called the exterior algebra (or Grassmann algebra) ofV . We mimic the procedure used for symmetric powers. If Ia is the two-sided ideal generated by all tensors of the form u⊗u∈V⊗2, we set

• (V ) =T(V )/Ia. Then, •(V ) automatically inherits a multiplication operation, called wedge product, and sinceT(V ) is graded, that is,

T(V ) = V⊗m,

m≥0

we have

• (V ) = V⊗m/(Ia∩V⊗m).

m≥0

However, it is easy to check that

m

(V )=V⊗m/(Ia∩V⊗m),∼

so

• (V )= (V ).∼

WhenV has finite dimension,d, we actually have a finite coproduct

d m

(V ) = (V ),

m=0

and since eachm(V ) has dimension, d , we deduce thatm

dim( (V )) = 2d = 2dim(V ).

The multiplication,

∧

:

m(V ) n m+n

×

(V )→ (V ), is skew-symmetric in the following precise sense:

Proposition 23.22.

For all

α

m n

∈ (V ) and allβ∈ (V ), we have

β∧α = (−1)mnα∧β.

Proof. Sincev∧u =−u∧v for allu, v∈V , Proposition 23.22 follows by induction.

Since α∧α = 0 for every simple tensor,α =u1∧ · · · ∧un, it seems natural to infer that α∧α = 0 for every tensorα∈ (V ). If we consider the case where dim(V )≤ 3, we can indeed prove the above assertion. However, if dim(V )≥ 4, the above fact is generally false! For example, when dim(V ) = 4, ifu_1, u_2, u_3, u_4 are a basis forV , forα =u1∧u_2 +u3∧u_4, we check that

α∧α = 2u1∧u2∧u3∧u_4,

which is nonzero.

The above discussion suggests that it might be useful to know when an alternating tensor is simple, that is, decomposable. It can be shown that for tensors,

α

2

∈ (V ),α∧α = 0 iff α is simple. A general criterion for decomposability can be given in terms of some operations known as left hook and right hook (also called interior products), see Section 23.17.

It is easy to see that (V ) satisfies the following universal mapping property: Proposition 23.23. Given anyK-algebra,A, for any linear map,f :V→A, if (f(v))2 = 0 for allv∈V , then there is a uniqueK-algebra homomorphism,f : (V )→A, so that

f =fæi,

as in the diagram below: V

FFF

i / (V ) FFFFFF ff

A WhenE is finite-dimensional, recall the isomorphism,µ:n(E∗)−→ Altn(E;K), defined as the linear extension of the map given by

µ(v∗ v∗)(u_1, . . . , un) = det(u∗(ui)).1∧ · · · ∧

Now, we have also a multiplication operation,

m(E∗) n m+n

×

(E∗)−→ (E∗). The following question then arises:

Can we define a multiplication, Altm(E;K)×Altn(E;K)−→ Altm+n(E;K), directly on alternating multilinear forms, so that the following diagram commutes:

m(E∗)n

× (E∗) ∧m+n(E∗)

µ×µ µ

Altm(E;K)× Altn(E;K) ∧ Altm+n(E;K).

As in the symmetric case, the answer is yes! The solution is to define this multiplication such that, forf∈ Altm(E;K) andg∈ Altn(E;K),

(f∧g)(u_1, . . . , um+n) = sgn(σ)f(uσ(1), . . . , uσ(m))g(uσ(m+1), . . . , uσ(m+n)),

σ∈shuffle(m,n)

where shuffle( m, n) consists of all (m, n)-“shuffles”, that is, permutations,σ, of{1, . . . m+n}, such thatσ(1)<· · ·< σ(m) andσ(m+1)<· · ·< σ(m+n). For example, whenm =n = 1, we have

(f∧g)(u, v) =f(u)g(v)−g(u)f(v). Whenm = 1 andn≥ 2, check that

m+1

(f∧g)(u_1, . . . , um+1) = (−1)i−1f(ui)g(u_1, . . . , ui, . . . , um+1),

i=1

where the hat over the argumentui means that it should be omitted. As a result of all this, the coproduct

Alt(E) = Altn(E;K)

n≥0

is an algebra under the above multiplication and this algebra is isomorphic to (E∗). For the record, we state

Proposition 23.24. WhenE is finite dimensional, the maps,µ:n(E∗)−→ Altn(E;K), induced by the linear extensions of the maps given by

µ(v∗ v∗n)(u_1, . . . , un) = det(u∗(ui))1∧ · · · ∧

yield a canonical isomorphism of algebras,

µ

:

(

E

∗

)

−→

(E;K)−→ Altm+n(E;K), with∧: Altm(E;K)× Alt

Alt(E), where the multiplication in Alt(E) is defined by the maps,

(f∧g)(u_1, . . . , um+n) = sgn(σ)f(uσ(1), . . . , uσ(m))g(uσ(m+1), . . . , uσ(m+n)),

σ∈shuffle(m,n)

where shuffle(m, n) consists of all (m, n)-“shuffles”, that is, permutations,σ, of{1, . . . m+n}, such thatσ(1)<· · ·< σ(m) andσ(m + 1)<· · ·< σ(m +n).

Remark: The algebra, (E) is a graded algebra. Given two graded algebras,E andF, we can make a new tensor product,E⊗F, whereE⊗F is equal toE⊗F as a vector space, but with a skew-commutative multiplication given by

(a⊗b)∧ (c⊗d) = (−1)deg(b)deg(c)(ac)⊗ (bd),

wherea∈Em, b∈Fp,c∈En, d∈Fq. Then, it can be shown that (E⊕F)= (E)⊗ (F).∼

23.16 The Hodge∗-Operator

In order to define a generalization of the Laplacian that will apply to differential forms on a Riemannian manifold, we need to define isomorphisms,

k n−k

V−→V,

for any Euclidean vector space,V , of dimensionn and anyk, with 0≤k≤n. If−,−denotes the inner product onV , we define an inner product onkV , also denoted−,−, by setting

u1∧ · · · ∧uk, v1∧ · · · ∧vk = det(ui, vj ), 23.16. THE HODGE∗-OPERATOR 647

for allui, vi∈V and extending−,− by bilinearity.

It is easy to show that if ( e_1, . . . , en) is an orthonormal basis ofV , then the basis ofkV consisting of theeI (whereI ={i_1, . . . , ik}, with 1≤i_1<· · ·< ik≤n) is an orthonormal basis ofkV . Since the inner product onV induces an inner product onV∗ (recall that

ω_1, ω_2 = ω_1, ω_2 , for allω_1, ω2∈V∗), we also get an inner product onkV∗.

Recall that an orientation of a vector space,V , of dimensionn is given by the choice of some basis, (e_1, . . . , en). We say that a basis, (u_1, . . . , un), ofV is positively oriented iff det(u_1, . . . , un)> 0 (where det(u_1, . . . , un) denotes the determinant of the matrix whosejth column consists of the coordinates ofuj over the basis (e_1, . . . , en)), otherwise it is negatively oriented. An oriented vector space is a vector space,V , together with an orientation ofV . IfV is oriented by the basis (e_1, . . . , en), thenV∗ is oriented by the dual basis, (e∗, . . . , e∗n).1

IfV is an oriented vector space of dimensionn, then we can define a linear map,

k n−k

: V→V,∗

called the Hodge∗-operator, as follows: For any choice of a positively oriented orthonormal basis, (e_1, . . . , en), ofV , set

∗(e1∧ · · · ∧ek) =ek+1∧ · · · ∧en.

In particular, fork = 0 andk =n, we have

(1) = e1∧ · · · ∧en∗

∗(e1∧ · · · ∧en) = 1.

It is easy to see that the definition of∗ does not depend on the choice of positively oriented orthonormal basis.

The Hodge

∗

-operators,

∗

:

kVn

→ −kV , induces a linear bijection, ∗

:

(

V

)

→

(

V

). We also have Hodge

∗

-operators,

∗

:

kVn

∗→−kV∗. The following proposition is easy to show:

Proposition 23.25. IfV is any oriented vector space of dimensionn, for everyk, with 0≤k≤n, we have

(i)∗∗ = (−id)k(n−k).

(ii)

x, y

=

∗

(

x

∧ ∗

y

) =

∗

(

y

∧ ∗

x

)

, for all

x, y

k

∈V .

If (e_1, . . . , en) is an orthonormal basis ofV and (v_1, . . . , vn) is any other basis ofV , it is easy to see that

v1∧ · · · ∧vn = det(vi, vj )e1∧ · · · ∧en, from which it follows that

∗(1) = 1 v1∧ · · · ∧vndet(vi, vj ) (see Jost [59], Chapter 2, Lemma 2.1.3).

23.17 Testing Decomposability; Left and Right Hooks

In this section, all vector spaces are assumed to have finite dimension. Say dim(E) =n. Using our nonsingular pairing,

p p

,−: E∗×E−→K (1≤p≤n),−

defined on generators by

u∗ u∗, v1∧ · · · ∧up = det(u∗(vj)),1∧ · · · ∧

we define various contraction operations,

p p+q q

: E× E∗−→ E∗ (left hook)

and p+q p q

: E∗× E−→ E∗ (right hook),

as well as the versions obtained by replacingE byE∗ andE∗∗ byE. We begin with the left interior product or left hook, .

Let

u

p

∈

E. For anyq such thatp+q≤n, multiplication on the right byu is a linear map

q p+q

(u): E−→ E,∧R

given by

v→v∧u where

v

q

∈ E. The transpose of∧R(u) yields a linear map,

p+q q

(∧R(u))t: ( E)∗−→ ( E)∗,

which, using the isomorphisms ( p+qE)∗ = p+qE∗ and ( qE)∗ = qE∗ can be viewed as a map∼ ∼

p+q q

(∧R(u))t: E∗−→ E∗,

given by

z∗→z∗æ ∧R(u), where

z

p+q

∗∈ E∗.

We denotez∗æ ∧R(u) by

u z∗. In terms of our pairing, theq-vectoru z∗ is uniquely defined by

u

z

∗

, v

=

z

∗

, v

∧

u

,

for all

u

p q p+q

∈ E,v∈ E andz∗∈ E∗.

It is immediately verified that

(u∧v) z∗ =u (v z∗),

so defines a left actionp p+q q

: E×E∗−→E∗. By interchangingE andE∗ and using the isomorphism,

k k

( F)∗ = F∗,∼

we can also define a left action

p p+q q

: E∗×E−→E. In terms of our pairing,u∗ z is uniquely defined by

v

∗

, u

∗

z

=

v

∗

∧

u

∗

, z

,

for all

u

∗

p q p+q

∈ E∗,v∗∈ E∗ andz∈ E.

p

In order to proceed any further, we need some combinatorial properties of the basis of E constructed from a basis, (e_1, . . . , en), ofE. Recall that for any (nonempty) subset, I⊆ {1, . . . , n}, we let

eI =ei1∧ · · · ∧eip, whereI ={i_1, . . . , ip} withi_1<· · ·< ip. We also lete∅ = 1. Given any two subsetsH, L⊆ {1, . . . , n}, let

ρ

H,L

=

0 ifH∩L =∅, (−1)ν ifH∩L =∅,

where ν =|{(h, l)| (h, l)∈H×L, h > l}|. Proposition 23.26. For any basis, (e_1, . . . , en), ofE the following properties hold:

(1) IfH∩L =∅,|H| =h, and|L| =l, then

ρH,LρL,H = (−1)hl. (2) ForH, L⊆ {1, . . . , m}, we have eH∧eL =ρH,LeH∪L.

(3) For the left hook,

p p+q q

: E×E∗−→E∗, we have

eH e∗L = 0 ifH⊆L

eH e∗L = ρL−H,He∗L−H ifH⊆L.

Similar formulae hold for

:

pE∗ p+q q

×

E−→ E. Using Proposition 23.26, we have the

Proposition 23.27. For the left hook,

p p+q q

: E× E∗−→ E∗,

for everyu∈E, we have

u (x∗∧y∗) = (−1)s(u x∗)∧y∗ +x∗∧ (u y∗),

wheresE∗.y∈

Proof. We can prove the above identity assuming thatx∗ andy∗ are of the forme∗ ande∗J using Proposition 23.26 but this is rather tedious. There is also a proof involving determinants, see Warner [109], Chapter 2.

Thus, is almost an anti-derivation, except that the sign, (−1)s is applied to the wrong factor.

It is also possible to define a right interior product or right hook, , using multiplication on the left rather than multiplication on the right. Then, defines a right action,

p+q p q

: E∗× E−→ E∗,

such that z

∗

, u

∧

v

=

z

∗

u, v

,

for all

u

p q p+q

∈ E,v∈ E, andz∗∈ E∗. Similarly, we have the right action

p+q p q

: E× E∗−→ E, such that

u

∗

∧

v

∗

, z

=

v

∗

, z

u

∗

,

for all

u

∗

p q p+q

∈ E∗,v∗∈ E∗, andz∈ E.

Since the left hook,

:

pE p+q q

× E∗−→ E∗, is defined by

u

z

∗

, v

=

z

∗

, v

∧

u

,

for all

u

p q p+q

∈ E,v∈ E andz∗∈ E∗,

the right hook,

p+q p q

: E∗×E−→E∗, by

z

∗

u, v

=

z

∗

, u

∧

v

,

for all

u

p q p+q

∈ E,v∈ E, andz∗∈ E∗, andv∧u = (−1)pqu∧v, we conclude that

u z∗ = (−1)pqz∗ u,

where

u

p p+q

∈ E andz∈ E∗.

Using the above property and Proposition 23.27 we get the following version of Proposition 23.27 for the right hook:

Proposition 23.28. For the right hook,

p+q p q

: E∗× E−→ E∗,

for everyu∈E, we have

(x∗∧y∗) u = (x∗ u)∧y∗ + (−1)rx∗∧ (y∗ u),

where

x

∗

r

∈ E∗.

Thus, is an anti-derivation.

For u∈E, the right hook,z∗ u, is also denoted,i(u)z∗, and called insertion operator or interior product . This operator plays an important role in differential geometry. If we view z

∗

n+1

∈ (E∗) as an alternating multilinear map in Altn+1(E;K), theni(u)z∗∈ Altn(E;K) is given by

(i(u)z∗)(v_1, . . . , vn) =z∗(u, v_1, . . . , vn).

Note that certain authors, such as Shafarevitch [94], denote our right hookz∗ u (which is also the right hook in Bourbaki [12] and Fulton and Harris [40]) byu z∗. Using the two versions of , we can define linear mapsγ:pE→ −pE∗ and δ:pE∗

n n

→

−pE. For any basis (e_1, . . . , en) ofE, if we letM ={1, . . . , n},e =e1∧· · ·∧en, ande∗ =e1∧ · · · ∧e∗n, then

γ(u) =u e∗ and δ(v) =v∗ e,

for alluppE and allv∗∈ E∗. The following proposition is easily shown.∈

Proposition 23.29.

The linear maps

γ

:

pE n n

→ −pE∗ andδ:pE∗→ −pE are isomorphims. The isomorphismsγ andδ map decomposable vectors to decomposable vectors. Furthermore, ifz∈ E is decomposable, then γ(z), z = 0, and similarly forz∈ n E∗. If (e_1, . . . , en) is any other basis ofE andγ :pE→ =λ−1δ for some nonzero→λ∈ &.γ =λγ and δ

p p

n−pE∗ andδ :pE∗ −pE are the corresponding isomorphisms, then

Proof. Using Proposition 23.26, for any subsetJ⊆ {1, . . . , n} =M such that|J| =p, we have

γ(eJ) =eJ e∗ =ρM−J,Je∗M−J and δ(e∗J) =e∗J e =ρM−J,JeM−J. Thus,

δæγ(eJ) =ρM−J,JρJ,M−JeJ = (−1)p(n−p)eJ. A similar result holds forγæδ. This implies that

δæγ = (−1)p(n−p)id and γæδ = (−1)p(n−p)id.

Thus, γ andδ are isomorphisms. IfzpE is decomposable, thenz =u1 ∧ · · · ∧ up where∈

u_1, . . . , up are linearly independent sincez = 0, and we can pick a basis ofE of the form (u_1, . . . , un). Then, the above formulae show that

γ(z) =±u∗+1∧ · · · ∧u∗n.

Clearly γ(z), z = 0.

If (e_1, . . . , en) is any other basis ofE, becausemE has dimension 1, we have

e1∧ · · · ∧en =λe1∧ · · · ∧en for some nonnullλ∈ &, and the rest is trivial.

We are now ready to tacke the problem of finding criteria for decomposability. We need a few preliminary results.

Proposition 23.30. Givenz∈ E, withz = 0, the smallest vector spaceW⊆E such thatz

p

p

∈ W is generated by the vectors of the form

u

∗

z,

with

u

∗

p

∈ −1E∗. Proof.

First, let

W

be any subspace such that

z

p

∈ (E) and let (e_1, . . . , er, er+1, . . . , en) be a basis ofE such that (e_1, . . . , er) is a basis ofW. Then,u∗ =Ie∗, whereI⊆ {1, . . . , n} and|I| =p−1, andz =JeJ, whereJ⊆ {1, . . . , r} and|J| =p≤r. It follows immediately from the formula of Proposition 23.26 (3) thatu∗ z∈W.

Next, we prove that if W is the smallest subspace ofE such thatz∈ (W), thenW is generated by the vectors of the formu∗ z, whereu∗ ∈ −1E∗. Suppose not, then the vectorsu∗ z withu∗

p p

p

∈ −1E∗ span a proper subspace,U, ofW. We prove that for every subspace,W , ofW, with dim(W ) = dim(W)−1 =r−1, it is not possible thatu∗ z∈W

for all

u

∗

p

∈ −1E∗. But then, asU is a proper subspace ofW, it is contained in some subspace,W , with dim(W ) =r− 1 and we have a contradiction.

Letw∈W−W and pick a basis ofW formed by a basis (e_1, . . . , er−1) ofW andw. We can write

z

=

z

+

w

∧

z

, where

z

p p

∈ W andz∈ −1W , and sinceW is the smallest subspace containingz, we havez = 0. Consequently, if we writez =IeI in terms of the basis (e_1, . . . , er−1) ofW , there is someeI, withI⊆ {1, . . . , r− 1} and|I| =p− 1, so that the coefficientλI is nonzero. Now, using any basis ofE containing (e_1, . . . , er−1, w), by Proposition 23.26 (3), we see that

e∗ (w∧eI) =λw, λ =±1.

It follows that

e∗ z =e∗ (z +w∧z ) =e∗ z +e∗ (w∧z ) =e∗ z +λw,

withe∗ z∈W , which shows thate∗ z /∈W . Therefore,W is indeed generated by the vectors of the form

u

∗

z

, where

u

∗

p

∈ −1E∗.

Proposition 23.31. Any nonzerozpE is decomposable iff∈

(

u

∗

z

)

∧

z

= 0

,

for all

u

∗

p

∈ −1E∗.

Proof.

Clearly,

z

p

∈ E is decomposable iff the smallest vector space,W, such thatz∈pW has dimensionp. If dim(W) =p, we havez =e1

p−1E∗, we haveu∗ z

∧ · · · ∧ep = 0. Now, assume that (u∗ z)

e

∧ · · · ∧ep wheree_1, . . . , ep form a basis ofW. By Proposition 23.30, for everyu∗∈z = (u∗ z)∧e1 ∈W, so eachu∗ z

is a linear combination of the ei’s and (u∗ z)∧ p−1E∗ and that dim(W) =n > p. If∧z = 0 for allu∗∈ IeI, whereI⊆ {1, . . . , n} and|I| =p.1, . . . , en is a basis ofW, then we havez =Iλ

Recall thatz = 0, and so, someλI is nonzero. By Proposition 23.30, eachei can be written

as

u

∗

z

for some

u

p p

∗∈ −1E∗ and since (u∗ z)∧z = 0 for allu∗∈ −1E∗, we get

ej∧z = 0 for j = 1, . . . , n.

By wedgingz =IλIeI with eachej, asn > p, we deduceλI = 0 for allI, soz = 0, a contradiction. Therefore,n =p andz is decomposable.

In Proposition 23.31, we can letu∗ range over a basis of p−1E∗, and then, the conditions are

(e∗H z)∧z = 0

for all

H

⊆ {

1

, . . . , n

}

, with

|

H

|

=

p

−

1. Since (

e

∗

H

z

)

∧

z

p+1

∈ E, this is equivalent to

e∗J ((e∗H z)∧z) = 0

for allH, J⊆ {1, . . . , n}, with|H| =p− 1 and|J| =p + 1. Then, for allI, I⊆ {1, . . . , n}with|I| =|I| =p, we can show that

e∗J ((e∗H eI)∧eI ) = 0,

unless there is somei∈ {1, . . . , n} such that

I−H ={i}, J−I ={i}.

In this case, e∗J (e∗H eH∪{i})∧eJ−{i} =ρ{i},Hρ{i},J−{i}. If we let

i,J,H =ρ{i},Hρ{i},J−{i},

we have i,J,H = +1 if the parity of the number ofj∈J such thatj < i is the same as the parity of the number ofh∈H such thath < i, and i,J,H =−1 otherwise. Finally, we obtain the following criterion in terms of quadratic equations (Pl¨ucker’s equations) for the decomposability of an alternating tensor:

Proposition 23.32.

(Grassmann-Pl¨

ucker’s Equations) For

z

=

I

λ

I

e

p

I∈ E, the conditions forz = 0 to be decomposable are

i,J,HλH∪{i}λJ−{i} = 0,

i∈J−H

for allH, J⊆ {1, . . . , n} such that|H| =p− 1 and|J| =p + 1. Using these criteria, it is a good exercise to prove that if dim(E) =n, then every tensor inn−1(E) is decomposable. This can also be shown directly.

It should be noted that the equations given by Proposition 23.32 are not independent. For example, when dim(E) =n = 4 andp = 2, these equations reduce to the single equation

λ_12λ34−λ_13λ_24 +λ_14λ_23 = 0.

When the field, K, is the field of complex numbers, this is the homogeneous equation of a quadric in CP^5 known as the Klein quadric. The points on this quadric are in one-to-one correspondence with the lines in CP^3.

23.18 Vector-Valued Alternating Forms

In this section, the vector space, E, is assumed to have finite dimension. We know that there is a canonical isomorphism,n(E∗)∼= Altn(E;K), between alternatingn-forms and alternating multilinear maps. As in the case of general tensors, the isomorphisms,

n

Altn(E;F) ∼= Hom( (E), F)

n n

Hom( (E), F) ∼= ( (E))∗⊗F

n n

( (E))∗ ∼= (E∗)

yield a canonical isomorphism

n

Altn(E;F)= (E∗) ⊗F.∼

Note that F may have infinite dimension. This isomorphism allows us to view the tensors in n(E∗)×F as vector valued alternating forms, a point of view that is useful in differential

geometry. If (

f

1

, . . . , f

r

) is a basis of

F

, every tensor,

ω

n

∈

(E∗)×F can be written as some linear combinationr

ω = αi⊗fi,

i=1

withαn(E∗). We also leti∈

n

(E;F) = (E∗) ⊗F = (E) ⊗F.

n=0

Given three vector spaces,F, G, H, if we have some bilinear map, Φ:F⊗G→H, then we can define a multiplication operation,

∧Φ: (E;F)× (E;G)→ (E;H),

as follows: For every pair, (m, n), we define the multiplication,

m n m+n

: (E∗) ⊗F ×(E∗) ⊗G −→(E∗) ⊗H,∧Φ

by (α⊗f)∧Φ (β⊗g) = (α∧β)⊗ Φ(f, g).

As in Section 23.15 (following H. Cartan [19]) we can also define a multiplication, m(E;F)× Altm(E;G)−→ Altm+n(E;H),∧Φ: Alt

directly on alternating multilinear maps as follows: Forf∈ Altm(E;F) andg∈ Altn(E;G), (f∧Φg)(u_1, . . . , um+n) = sgn(σ) Φ(f(uσ(1), . . . , uσ(m)), g(uσ(m+1), . . . , uσ(m+n))),

σ∈shuffle(m,n)

where shuffle(m, n) consists of all (m, n)-“shuffles”, that is, permutations,σ, of{1, . . . m+n}, such thatσ(1)<· · ·< σ(m) andσ(m + 1)<· · ·< σ(m +n).

In general, not much can be said about∧Φ unless Φ has some additional properties. In particular,∧Φ is generally not associative. We also have the map,

n

µ: (E∗) ⊗F−→ Altn(E;F),

defined on generators by

µ((v∗∧ · · · ∧v∗n)⊗a)(u_1, . . . , un) = (det(v∗(ui))a.

1

Proposition 23.33. The map

n

µ: (E∗) ⊗F−→ Altn(E;F),

defined as above is a canonical isomorphism for every n≥ 0. Furthermore, given any three vector spaces,F, G, H, and any bilinear map, Φ:F×G→H, for allω∈ ( n(E∗))⊗F and allη∈ ( n(E∗))⊗G,

µ(α∧Φβ) =µ(α)∧Φµ(β).

Proof. Since we already know that ( n(E∗))⊗F and Altn(E;F) are isomorphic, it is enough to show thatµ maps some basis of ( n(E∗))⊗F to linearly independent elements. Pick some bases, (e_1, . . . , ep) inE and (fj)j∈J inF. Then, we know that the vectors,e∗⊗fj, where I⊆ {1, . . . , p} and|I| =n form a basis of ( n(E∗))⊗F. If we have a linear dependence,

λI,jµ(e∗⊗fj) = 0,

I,j

applying the above combination to each (ei1, . . . , ein) (I ={i_1, . . . , in},i_1<· · ·< in), we get the linear combination

λI,jfj = 0,

j

and by linear independence of the fj’s, we getλI,j = 0, for allI and allj. Therefore, the µ(e∗⊗fj) are linearly independent and we are done. The second part of the proposition is easily checked (a simple computation).

A special case of interest is the case where F =G =H is a Lie algebra and Φ(a, b) = [a, b], is the Lie bracket ofF. In this case, using a base, (f_1, . . . , fr), ofF if we writeω =iαi⊗fi andη =jβj⊗fj, we have

[ω, η] = αi∧βj⊗ [fi, fj].

i,j

Consequently, [η, ω] = (−1)mn+1[ω, η]. The following proposition will be useful in dealing with vector-valued differential forms: Proposition 23.34. If (e_1, . . . , ep) is any basis ofE, then every element,ω∈ ( n(E∗))⊗F, can be written in a unique way as

ω = e∗⊗fI, fI∈F,

I

where thee∗ are defined as in Section 23.12.

Proof. Since, by Proposition 23.19, thee∗ form a basis ofn(E∗), elements of the form e∗ ⊗f span ( n(E∗))⊗F. Now, if we applyµ(ω) to (ei1, . . . , ein), whereI ={i_1, . . . , in} ⊆ {1, . . . , p}, we get

µ(ω)(ei1, . . . , ein) =µ(e∗⊗fI)(ei1, . . . , ein) =fI. Therefore, thefI are uniquely determined byf.

Proposition can also be formulated in terms of alternating multilinear maps, a fact that will be useful to deal with differential forms.

Define the product,·: Altn(E;R)×F→ Altn(E;F), as follows: For allω∈ Altn(E;R) and allf∈F,

(ω·f)(u_1, . . . , un) =ω(u_1, . . . , un)f,

for all u_1, . . . , un∈E. Then, it is immediately verified that for everyω∈ ( n(E∗))⊗F of the form

ω =u∗ u∗n⊗f,1∧ · · · ∧we have

µ(u∗ u∗n⊗f) =µ(u∗ u∗n)·f.1∧ · · · ∧ 1∧ · · · ∧Then, Proposition 23.34 yields

Proposition 23.35. If (e_1, . . . , ep) is any basis ofE, then every element,ω∈ Altn(E;F), can be written in a unique way as

ω = e∗·fI, fI∈F,

I

where thee∗ are defined as in Section 23.12.

23.19 The Pfaffian Polynomial

Let so(2n) denote the vector space (actually, Lie algebra) of 2n× 2n real skew-symmetric matrices. It is well-known that every matrix,A∈ so(2n), can be written as A =P DP ,

whereP is an orthogonal matrix and whereD is a block diagonal matrix ëD ö

1

ì D_2 ÷

D

=

ì... ÷

ì ÷

í ø

Dn

consisting of 2× 2 blocks of the form

Di =0 −

ai

0 ,ai

as shown in Theorem 13.18. (This is also shown in Horn and Johnson [55], Corollary 2.5.14, and Gantmacher [44], Chapter IX.)

Since det( Di) =a^2 and det(A) = det(P DP ) = det(D) = det(D_1)· · ·det(Dn), we get det(A) = (a^2.1· · ·an)

The Pfaffian is a polynomial function, Pf(A), in skew-symmetric 2n× 2n matrices,A, (a polynomial in (2n− 1)n variables) such that

Pf(A)2 = det(A)

and for every arbitrary matrix,B, Pf(BAB ) = Pf(A) det(B).

The Pfaffian shows up in the definition of the Euler class of a vector bundle. There is a simple way to define the Pfaffian using some exterior algebra. Let (e_1, . . . , e2n) be any basis of R2n. For any matrix,A∈ so(2n), let

ω(A) = aijei∧ej,

i<j

whereA = (aij). Then,nω(A) is of the formCe1∧e2∧ · · ·∧e2n for some constant,C∈ R. Definition 23.9. For every skew symmetric matrix,A∈ so(2n), the Pfaffian polynomial or Pfaffian is the degreen polynomial, Pf(A), defined by

n

ω(A) =n! Pf(A)e1∧e2∧ · · · ∧e2n. Clearly, Pf(A) is independent of the basis chosen. IfA is the block diagonal matrixD, a simple calculation shows that

ω(D) =−(a_1e1∧e_2 +a_2e3∧e_4 +· · · +ane2n−1∧e2n)

and thatn ω(D) = (−1)nn!a1· · ·ane1∧e2∧ · · · ∧e2n, and so

Pf(D) = (−1)na1· · ·an.

Since Pf(D)2 = (a1· · ·an)2 = det(A), we seem to be on the right track.

Proposition 23.36. For every skew symmetric matrix, A∈ so(2n) and every arbitrary matrix,B, we have:

(i) Pf(A)2 = det(A)

(ii) Pf(BAB ) = Pf(A) det(B).

Proof. If we assume that (ii) is proved then, since we can write A = P DP for some orthogonal matrix,P, and some block diagonal matrix,D, as above, as det(P) =±1 and Pf(D)2 = det(A), we get

Pf(A)2 = Pf(P DP )2 = Pf(D)2 det(P)2 = det(A), which is (i). Therefore, it remains to prove (ii).

Letfi =Bei, fori = 1, . . . ,2n, where (e_1, . . . , e2n) is any basis ofR2n. Sincefi =kbkiek, we have

τ = aijfi∧fj = bkiaijbljek∧el = (BAB )klek∧el,

i,j i,j k,l k,l

and so, as BAB is skew symmetric andek∧el =−el∧ek, we get τ = 2ω(BAB ).

Consequently,

n n

τ = 2n ω(BAB ) = 2nn! Pf(BAB )e1∧e2∧ · · · ∧e2n.

Now,n τ =C f1∧f2∧ · · · ∧f2n,

for someC∈ R. IfB is singular, then thefi are linearly dependent which implies that f1∧f2∧ · · · ∧f2n = 0, in which case,

Pf(BAB ) = 0, ase1∧e2∧ · · · ∧e2n = 0. Therefore, ifB is singular, det(B) = 0 and Pf(BAB ) = 0 = Pf(A) det(B). IfB is invertible, asτ =i,jaijfi∧fj = 2i<jaijfi∧fj, we have

n

τ = 2nn! Pf(A)f1∧f2∧ · · · ∧f2n.

However, as fi =Bei, we have

f1∧f2∧ · · · ∧f2n = det(B)e1∧e2∧ · · · ∧e2n,

son

τ = 2nn! Pf(A) det(B)e1∧e2∧ · · · ∧e2n

and asn

τ = 2nn! Pf(BAB )e1∧e2∧ · · · ∧e2n,

we get

Pf(BAB ) = Pf(A) det(B), as claimed.

Remark: It can be shown that the polynomial, Pf(A), is the unique polynomial with integer coefficients such that Pf(A)2 = det(A) and Pf(diag(S, . . . , S)) = +1, where

S =0 1 ,

−1 0

see Milnor and Stasheff [79] (Appendix C, Lemma 9). There is also an explicit formula for Pf(A), namely:n

Pf(

A

) =

2n

1 sgn(σ) an! σ∈S2n i=1 −1) σ(2i).σ(2i

Beware, some authors use a different sign convention and require the Pfaffian to have the value +1 on the matrix diag(S , . . . , S ), where

1

S =0 − .1 0

For example, if R2n is equipped with an inner product,−,−, then some authors define ω(A) as

ω(A) = Aei, ej ei∧ej,

i<j

where A = (aij). But then, Aei, ej =aji and notaij, and this Pfaffian takes the value +1 on the matrix diag(S , . . . , S ). This version of the Pfaffian differs from our version by the factor (−1)n. In this respect, Madsen and Tornehave [71] seem to have an incorrect sign in Proposition B6 of Appendix C.

We will also need another property of Pfaffians. Recall that the ring, Mn(C), ofn×n matrices over C is embedded in the ring,M2n(R), of 2n× 2n matrices with real coefficients, using the injective homomorphism that maps every entryz =a+ib∈ C to the 2×2 matrix

a

b

− .b a

If A∈Mn(C), letAR∈M2n(R) denote the real matrix obtained by the above process. Observe that every skew Hermitian matrix,A∈ u(n), (i.e., withA∗ =A =−A) yields a matrixAR∈ so(2n).

Proposition 23.37. For every skew Hermitian matrix,A∈ u(n), we have

Pf(AR) =in det(A).

Proof. It is well-known that a skew Hermitian matrix can be diagonalized with respect to a unitary matrix,U, and that the eigenvalues are pure imaginary or zero, so we can write

A =U diag(ia_1, . . . , ian)U∗,

for some reals,ai∈ R. Consequently, we get

AR =UR diag(D_1, . . . , Dn)UR,

where0 − aiDi = ai 0

and

Pf(AR) = Pf(diag(D_1, . . . , Dn)) = (−1)na1· · ·an, as we saw before. On the other hand,

det(A) = det(diag(ia_1, . . . , ian)) =ina1· · ·an,

and as (−1)n =inin, we get Pf(AR) =in det(A), as claimed.

Madsen and Tornehave [71] state Proposition 23.37 using the factor (−i)n, which is wrong.

Chapter 24 Introduction to Modules; Modules over a PID

24.1 Modules over a Commutative Ring

In this chapter, we introduce modules over a commutative ring (with unity). After a quick overview of fundamental concepts such as free modules, torsion modules, and some basic results about them, we focus on finitely generated modules over a PID and we prove the structure theorems for this class of modules (invariant factors and elementary divisors). Our main goal is not to give a comprehensive exposition of modules, but instead to apply the structure theorem to theK[X]-moduleEf defined by a linear mapf acting on a finitedimensional vector spaceE, and to obtain several normal forms forf, including the rational canonical form.

A module is the generalization of a vector space E over a fieldK obtained replacing the fieldK by a commutative ringA (with unity 1). Although formally, the definition is the same, the fact that some nonzero elements ofA are not invertible has some serious conequences. For example, it is possible thatλ·u = 0 for some nonzeroλ∈A and some nonzerou∈E, and a module may no longer have a basis.

For the sake of completeness, we give the definition of a module, although it is the same as Definition 2.9 with the fieldK replaced by a ringA. In this chapter, all rings under consideration are assumed to be commutative and to have an identity element 1.

Definition 24.1. Given a ringA, a (left) module overA (orA-module) is a setM (of vectors) together with two operations +:M×M→M (called vector addition),1 and·:A×M→M (called scalar multiplication) satisfying the following conditions for allα, β∈ A and all u, v∈M;

(M0) M is an abelian group w.r.t. +, with identity element 0;

1The symbol + is overloaded, since it denotes both addition in the ring A and addition of vectors in M. It is usually clear from the context which + is intended.

663 (M1) α· (u +v) = (α·u) + (α·v); (M2) (α +β)·u = (α·u) + (β·u);

(M3) (α∗β)·u =α· (β·u);

(M4) 1·u =u.

Givenα∈A andv∈M, the elementα·v is also denoted byαv. The ringA is often called the ring of scalars.

Unless specified otherwise or unless we are dealing with several different rings, in the rest of this chapter, we assume that allA-modules are defined with respect to a fixed ringA. Thus, we will refer to aA-module simply as a module.

From (M0), a module always contains the null vector 0, and thus is nonempty. From (M1), we getα· 0 = 0, andα· (−v) =−(α·v). From (M2), we get 0·v = 0, and (−α)·v =−(α·v). The ringA itself can be viewed as a module over itself, addition of vectors being addition in the ring, and multiplication by a scalar being multiplication in the ring.

When the ringA is a field, anA-module is a vector space. WhenA = Z, a Z-module is just an abelian group, with the action given by

0·u = 0,

n·u =u +· · · +u, n > 0

n

n·u =−(−n)·u, n < 0.

All definitions from Section 2.3, linear combinations, linear independence and linear dependence, subspaces renamed as submodules, apply unchanged to modules. Proposition 2.5 also holds for the module spanned by a set of vectors. The definition of a basis (Definition 2.12) also applies to modules, but the only result from Section 2.4 that holds for modules is Proposition 2.11. Unfortunately, it is longer true that every module has a basis. For example, for any nonzero integerm∈ Z, the Z-module Z/mZ has no basis. Similarly, Q, as a Z-module, has no basis. In fact, any two distinct nonzero elementsp_1/q_1 andp_2/q_2 are linearly dependent, since

(p_2q_1) p1 (p_1q_2) p2 = 0.q1 − q2

Definition 2.13 can be generalized to rings and yields free modules.

Definition 24.2. Given a commutative ringA and any (nonempty) setI, letA(I) be the subset of the cartesian productAI consisting of all families (λi)i∈I with finite support of scalars inA.2 We define addition and multiplication by a scalar as follows:

(λi)i∈I + (µi)i∈I = (λi +µi)i∈I,

2Where AI denotes the set of all functions from I to A.

and λ· (µi)i∈I = (λµi)i∈I.

It is immediately verified that addition and multiplication by a scalar are well defined. Thus,A(I) is a module. Furthermore, because families with finite support are considered, the family (ei)i∈I of vectorsei, defined such that (ei)j = 0 ifj =i and (ei)i = 1, is clearly a basis of the moduleA(I). WhenI ={ 1, . . . , n}, we denoteA(I) byAn. The functionι:I→A(I), such thatι(i) =ei for everyi∈I, is clearly an injection.

Definition 24.3. AnA-moduleM is free iff it has a basis.

The moduleA(I) is a free module.

All definitions from Section 2.5 apply to modules, linear maps, kernel, image, except the definition of rank, which has to be defined differently. Propositions 2.12, 2.13, 2.14, and 2.15 hold for modules. However, the other propositions do not generalize to modules. The definition of an isomorphism generalizes to modules. As a consequence, a module is free iff it is isomorphic to a module of the formA(I).

Section 2.6 generalizes to modules. Given a submoduleN of a moduleM, we can define the quotient moduleM/N.

If a is an ideal inA and ifM is anA-module, we define aM as the set of finite sums of the form

a_1m_1 +· · · +akmk, ai∈ a, mi∈M.

It is immediately verified that aM is a submodule ofM.

Interestingly, the part of Theorem 2.10 that asserts that any two bases of a vector space have the same cardinality holds for modules. One way to prove this fact is to “pass” to a vector space by a quotient process.

Theorem 24.1. For any free moduleM, any two bases ofM have the same cardinality.

Proof sketch. We give the argument for finite bases, but it also holds for infinite bases. The trick is to pick any maximal ideal m inA (whose existence is guaranteed by Theorem 31.3). Then,A/m is a field, andM/mM can be made into a vector space overA/m; we leave the details as an exercise. If (u_1, . . . , un) is a basis ofM, then it is easy to see that the image of this basis is a basis of the vector spaceM/mM. By Theorem 2.10, the numbern of elements in any basis ofM/mM is an invariant, so any two bases ofM must have the same number of elements.

The common number of elements in any basis of a free module is called the dimension (or rank) of the free module.

One should realize that the notion of linear independence in a module is a little tricky. According to the definition, the one-element sequence (u) consisting of a single nonzero vector is linearly independent if for allλ∈A, ifλu = 0 thenλ = 0. However, there are free modules that contain nonzero vectors that are not linearly independent! For example, the ringA = Z/6Z viewed as a module over itself has the basis (1), but the zero-divisors, such as 2 or 4, are not linearly independent. Using language introduced in Definition 24.4, a free module may have torsion elements. There are also nonfree modules such that every nonzero vector is linearly independent, such as Q over Z.

All definitions from Section 3.1 about matrices apply to free modules, and so do all the proposition. Similarly, all definitions from Section 4.1 about direct sums and direct products apply to modules. All propositions that do not involve extending bases still hold. The important proposition 4.10 survives in the following form.

Proposition 24.2. Letf :E→F be a surjective linear between twoA-modules withF a free module. Given any basis (v1, . . . , vr) ofF, for anyr vectorsu_1, . . . , ur∈E such that f(ui) =vi fori = 1, . . . , r, the vectors (u_1, . . . , ur) are linearly independent and the module E is the direct sum

E = Ker (f)⊕U,

whereU is the free submodule ofE spanned by the basis (u_1, . . . , ur).

Proof. Pick anyw∈E, writef(w) over the basis (v_1, . . . , vr) asf(w) =a_1v_1 +· · · +arvr, and letu =a_1u_1 +· · · +arur. Observe that

f (w−u) =f(w)−f(u)

=a_1v_1 +· · · +arvr− (a_1f(u_1) +· · · +arf(ur)) =a_1v_1 +· · · +arvr− (a_1v_1 +· · · +arvr) = 0.

Therefore,h =w−u∈ Ker (f), and sincew =h +u withh∈ Ker (f) andu∈U, we have E = Ker (f) +U.

Ifu =a_1u_1 +· · · +arur∈U also belongs to Ker (f), then

0 =f(u) =f(a_1u_1 +· · · +arur) =a_1v_1 +· · · +arvr,

and since (v_1, . . . , vr) is a basis,ai = 0 fori = 1, . . . , r, which shows that Ker (f)∩U = (0). Therefore, we have a direct sum

E = Ker (f)⊕U. Finally, if a_1u_1 +· · · +arur = 0,

the above reasoning shows thatai = 0 fori = 1, . . . , r, so (u_1, . . . , ur) are linearly independent. Therefore, the moduleU is a free module.

One should be aware that if we have a direct sum of modules

U =U1⊕ · · · ⊕Um,

every vectoru∈U can be written is a unique way as

u =u_1 +· · · +um,

withui∈Ui but, unlike the case of vector spaces, this does not imply that anym nonzero vectors (u_1, . . . , um) are linearly independent. For example,

Z = Z/2Z⊕Z/2Z

where Z and Z/2Z are view as Z-modules, but (1,0) and (0,1) are not linearly independent, since

2(1,0) + 2(0,1) = (0,0).

A useful fact is that every module is a quotient of some free module. Indeed, ifM is anA-module, pick any spanning setI forM (such a set exists, for example,I =M), and consider the unique homomorphismÕ:A(I) M extending the identity function fromI to

itself. Then we have an isomorphism

A

(

I

)

→

/Ker (Õ)≈M.

In particular, if M is finitely generated, we can pickI to be a finite set of generators, in which case we get an isomorphismAn/Ker (Õ)≈M, for some natural numbern. A finitely generated module is sometimes called a module of finite type.

The case n = 1 is of particular interest. A moduleM is said to be cyclic if it is generated by a single element. In this caseM = Ax, for somex∈ M. We have the linear map mx:A→M given bya→ax for everya∈A, and it is obviously surjective sinceM =Ax. Since the kernel a = Ker (mx) ofmx is an ideal inA, we get an isomorphismA/a≈Ax. Conversely, for any ideal a ofA, ifM =A/a, we see thatM is generated by the imagex of 1 inM, soM is a cyclic module.

The ideal a = Ker (mx) is the set of alla∈A such thatax = 0. This is called the annihilator ofx, and it is the special case of the following more general situation.

Definition 24.4. IfM is anyA-module, for any subsetS ofM, the set of alla∈A such thatax = 0 for allx∈S is called the annihilator ofS, and it is denoted by Ann(S). If S ={x}, we write Ann(x) instead of Ann({x}). A nonzero elementx∈M is called a torsion element iff Ann(x) = (0). The set consisting of all torsion elements inM and 0 is denoted byMtor.

It is immediately verified that Ann(S) is an ideal ofA, and by definition,

Mtor ={x∈M| (∃a∈A, a = 0)(ax = 0)}. If a ring has zero divisors, then the set of all torsion elements in anA-moduleM may not be a submodule ofM. For example, ifM =A = Z/6Z, thenMtor ={2,3,4}, but 3 + 4 = 1 is not a torsion element. Also, a free module may not be torsion-free because there may be torsion elements, as the example of Z/6Z as a free module over itself shows.

However, ifA is an integral domain, then a free module is torsion-free andMtor is a submodule ofM. (Recall that an integral domain is commutative).

Proposition 24.3. IfA is an integral domain, then for anyA-moduleM, the setMtor of torsion elements inM is a submodule ofM.

Proof. Ifx, y∈M are torsion elements (x, y = 0), then there exist some nonzero elements a, b∈A such thatax = 0 andby = 0. SinceA is an integral domain,ab = 0, and then for allλ, µ∈A, we have

ab(λx +µy) =bλax +aµby = 0. Therefore,Mtor is a submodule ofM.

The moduleMtor is called the torsion submodule ofM. IfMtor = (0), then we say that M is torsion-free, and ifM =Mtor, then we say thatM is a torsion module.

If M is not finitely generated, then it is possible thatMtor = 0, yet the annihilator of Mtor is reduced to 0 (find an example). However, ifM is finitely generated, this cannot happen, since ifx_1, . . . , xn generateM and ifa_1, . . . , an annihilatex_1, . . . , xn, thena1· · ·an annihilates every element ofM.

Proposition 24.4. IfA is an integral domain, then for anyA-moduleM, the quotient moduleM/Mtor is torsion free.

Proof. Letx be an element ofM/Mtor and assume thatax = 0 for somea = 0 inA. This means thatax∈Mtor, so there is someb = 0 inA such thatbax = 0. Sincea, b = 0 andA is an integral domain,ba = 0, sox∈Mtor, which means thatx = 0.

If A is an integral domain and ifF is a freeA-module with basis (u_1, . . . , un), thenF can be embedded in aK-vector spaceFK isomorphic toKn, whereK = Frac(A) is the fraction field ofA. Similarly, any submoduleM ofF is embedded into a subspaceMK of FK. Note that any linearly independent vectors (u_1, . . . , um) in theA-moduleM remain linearly independent in the vector spaceMK, because any linear dependence overK is of

the form a1 u_1 +· · · +am um = 0b1 bm

for some ai, bi∈A, withb1· · ·bm = 0, so if we multiply byb1· · ·bm = 0, we get a linear dependence in theA-moduleM. Then, we see that the maximum number of linearly independent vectors in theA-moduleM is at mostn. The maximum number of linearly independent vectors in a finitely generated submodule of a free module (over an integral domain) is called the rank of the moduleM. If (u_1, . . . , um) are linearly independent where m is the rank ofm, then for every nonzerov∈M, there are somea, a_1, . . . , am∈A, not all zero, such that

av =a_1u_1 +· · · +amum.

We must have a = 0, since otherwise, linear independence of the ui would imply that a_1 =· · · =am = 0, contradicting the fact thata, a_1, . . . , am∈A are not all zero.

Unfortunately, in general, a torsion-free module is not free. For example, Q as aZ-module is torsion-free but not free. If we restrict ourselves to finitely generated modules over PID’s, then such modules split as the direct sum of their torsion module with a free module, and a torsion module has a nice decomposition in terms of cyclic modules.

The following proposition shows that over a PID, submodules of a free module are free. There are various ways of proving this result. We give a proof due to Lang [65] (see Chapter III, Section 7).

Proposition 24.5. IfA is a PID and ifF is a freeA-module of dimensionn, then every submoduleM ofF is a free module of dimension at mostn.

Proof. Let (u_1, . . . , un) be a basis ofF, and letMr =M∩(Au1⊕· · ·⊕ Aur), the intersection ofM with the free module generated by (u_1, . . . , ur), forr = 1, . . . , n. We prove by induction onr that eachMr is free and of dimension at mostr. SinceM =Mr for somer, this will prove our result.

ConsiderM_1 =M∩Au_1. IfM_1 = (0), we are done. Otherwise let

a ={a∈A|au1∈M}.

It is immediately verified that a is an ideal, and sinceA is a PID, a =a_1A, for somea1∈A. Since we are assuming thatM_1 = (0), we havea_1 = 0, anda_1u1 ∈M. Ifx∈M_1, then x =au_1 for somea∈A, soa∈a_1A, and thusa =ba_1 for someb∈A. It follows that M_1 =Aa_1u_1, which is free.

Assume inductively thatMr is free of dimension at mostr < n, and let

a ={a∈A| (∃b1∈A)· · ·(∃br∈A)(b_1u_1 +· · · +brur +aur+1∈M)}.

It is immediately verified that a is an ideal, and sinceA is a PID, a =ar+1A, for some ar+1∈A. Ifar+1 = 0, thenMr+1 =Mr, and we are done.

Ifar+1 = 0, then there is somev1∈Au1⊕ · · · ⊕Aur such that

w =v_1 +ar+1ur+1∈M.

For anyx∈Mr+1, there is somev∈Au1⊕· · ·⊕Aur and somea∈A such thatx =v+aur+1. Then,a∈ar+1A, so there is someb∈A such thata =bar+1. As a consequence

x−bw =v−bv1∈Mr, and sox =x−bw +bw withx−bw∈Mr, which shows that Mr+1 =Mr +Aw.

On the other hand, if u∈Mr∩Aw, then sincew =v_1 +ar+1ur+1 we have u =bv_1 +bar+1ur+1,

for someb∈A, withu, v1∈Au1⊕ · · · ⊕Aur, and ifb = 0, this yields the nontrivial linear combination

bv1−u +bar+1ur+1 = 0,

contradicting the fact that (u_1, . . . , ur+1) are linearly independent. Therefore, Mr+1 =Mr⊕Aw, which shows thatMr+1 is free of dimension at mostr + 1.

Proposition 24.5 implies that ifM is a finitely generated module over a PID, then any submoduleN ofM is also finitely generated.

Indeed, if (u_1, . . . , un) generateM, then we have a surjectionÕ:An M from the free module

A

n

onto

M

. The inverse image

Õ

−

1

→

( N) ofN is a submodule of the free moduleAn, therefore by Proposition 24.5,Õ−1(N) is free and finitely generated. This implies thatN is finitely generated (and that it has a number of generators≤n).

We can also prove that a finitely generated torsion-free module over a PID is actually free. We will give another proof of this fact later, but the following proof is instructive. Proposition 24.6. IfA is a PID and ifM is a finitely generated module which is torsionfree, thenM is free.

Proof. Let (y_1, . . . , yn) be some generators forM, and let (u_1, . . . , um) be a maximal subsequence of (y_1, . . . , yn) which is linearly independent. Ifm =n, we are done. Otherwise, due to the maximality ofm, fori = 1, . . . , n, there is someai = 0 such that such that aiyi can be expressed as a linear combination of (u_1, . . . , um). If we leta =a_1. . . an, then a_1. . . anyi∈Au1⊕ · · · ⊕Aum fori = 1, . . . , n, which shows that

aM⊆Au1⊕ · · · ⊕Aum.

Now, A is an integral domain, and sinceai = 0 fori = 1, . . . , n, we havea =a_1. . . an = 0, and becauseM is torsion-free, the mapx→ax is injective. It follows thatM is isomorphic to a submodule of the free moduleAu1⊕ · · · ⊕Aum. By Proposition 24.5, this submodule if free, and thus,M is free.

Although we will obtain this result as a corollary of the structure theorem for finitely generated modules over a PID, we are in the position to give a quick proof of the following theorem.

Theorem 24.7. LetM be a finitely generated module over a PID. ThenM/Mtor is free, and there exit a free submoduleF ofM such thatM is the direct sum

M =Mtor⊕F. The dimension ofF is uniquely determined.

Proof. By Proposition 24.4M/Mtor is torsion-free, and sinceM is finitely generated, it is also finitely generated. By Proposition 24.6,M/Mtor is free. We have the quotient linear mapπ:M→M/Mtor, which is surjective, andM/Mtor is free, so by Proposition 24.2, there is a free moduleF isomorphic toM/Mtor such that

M = Ker (π)⊕F =Mtor⊕F.

SinceF is isomorphic toM/Mtor, the dimension ofF is uniquely determined.

Theorem 24.7 reduces the study of finitely generated module over a PID to the study of finitely generated torsion modules. This is the path followed by Lang [65] (Chapter III, section 7).

24.2 Finite Presentations of Modules

Since modules are generally not free, it is natural to look for techniques for dealing with nonfree modules. The hint is that ifM is anA-module and if (ui)i∈I is any set of generators forM, then we know that there is a surjective homomorphismÕ:A(I) M from the free

module

A

(

I

)

→

generated by I ontoM. FurthermoreM is isomorphic toA(I)/Ker (Õ). Then, we can pick a set of generators (vj)j∈J for Ker (Õ), and again there is a surjective map ψ:A(J) Ker (Õ) from the free moduleA(J) generated byJ onto Ker (Õ). The mapψ can→

be viewed a linear map fromA(J) toA(I), we have

Im(ψ) = Ker (Õ),

andÕ is surjective. Note thatM is isomorphic toA(I)/Im(ψ). In such a situation we say that we have an exact sequence and this is denoted by the diagram

A(J) ψ A(I) Õ M 0.

Definition 24.5. Given anA-moduleM, a presentation ofM is an exact sequence

A(J) ψ A(I) Õ M 0 which means that

1. Im( ψ) = Ker (Õ).

2. Õ is surjective.

Consequently,M is isomorphic toA(I)/Im(ψ). IfI andJ are both finite, we say that this is a finite presentation ofM.

Observe that in the case of a finite presentation, I andJ are finite, and if|J| =n and |I| =m, thenψ is a linear mapψ:An Am, so it is given by somem×n matrixR with coefficients inA called the presentation matrix→ ofM. Every columnR ofR may thought of as a relation

aj1e_1 +· · · +ajmem = 0

among the generators e_1, . . . , em ofAm, so we haven relations among these generators. Also the images ofe_1, . . . , em inM are generators ofM, so we can think of the above relations as relations among the generators ofM. The submodule ofAm spanned by the columns of R is the set of relations ofM, and the columns ofR are called a complete set of relations forM. The vectorse_1, . . . , em are called a set of generators forM. We may also say that the generatorse_1, . . . , em and the relationsR^1, . . . , Rn (the columns ofR) are a (finite) presentation of the moduleM.

For example, the Z-module presented by the 1×1 matrixR = (5) is the quotient, Z/5Z, of Z by the submodule 5Z corresponding to the single relation

5e_1 = 0. But Z/5Z has other presentations. For example, if we consider the matrix of relations

R

=

2

1

− ,1 2

presenting the moduleM, then we have the relations 2e_1 +e_2 = 0

e_1 + 2e_2 = 0.−

From the first equation, we gete_2 =−2e_1, and substituting into the second equation we get −5e_1 = 0.

It follows that the generatore_2 can be eliminated andM is generated by the single generator e_1 satisfying the relation

5e_1 = 0,

which shows thatM≈ Z/5Z.

The above example shows that many different matrices can present the same module. Here are some useful rules for manipulating a relation matrix without changing the isomorphism class of the moduleM it presents.

Proposition 24.8. IfR is anm×n matrix presenting anA-moduleM, then the matrices S of the form listed below present the same module (a module isomorphic toM):

(1) S =QRP−1, whereQ is am×m invertible matrix andP an×n invertible matrix (both overA).

(2) S is obtained fromR by deleting a column of zeros.

(3) Thejth column ofR isei, andS is obtained fromR by deleting theith row and the jth column.

Proof. (1) By definition, we have an isomorphismM≈Am/RAn, where we denote byRAn the image ofAn by the linear map defined byR. Going fromR toQRP−1 corresponds to making a change of basis inAm and a change of basis inAn, and this yields a quotient module isomorphic toM.

(2) A zero column does not contribute to the span of the columns ofR, so it can be eliminated.

(3) If the jth column ofR isei, then when taking the quotientAm/RAn, the generator ei goes to zero. This means that the generatorei is redundant, and when we delete it, we get a matrix of relations in which theith row ofR and thejth column ofR are deleted.

The matricesP andQ are often products of elementary operations. One should be careful that rows of zeros cannnot be eliminated. For example, the 2× 1 matrix

4R_1 =0

gives the single relation

4e_1 = 0,

but the second generatore_2 cannot be eliminated. This matrix presents the moduleZ/4Z×Z. On the other hand, the 1× 2 matrix

R_2 = 4 0 gives two relations

4e_1 = 0,

0 = 0,

so the second generator can be eliminated andR_2 presents the module Z/4Z. The rules of Proposition 24.8 make it possible to simplify a presentation matrix quite a lot in some cases. For example, consider the relation matrix

ë3 8 7 9ö

R = í2 4 6 6ø.

1 2 2 1

By subtracting 2 times row 3 from row 2 and subtracting 3 times row 3 from row 1, we get

ë 0 2 1 6ö

í0 0 2 4ø.

1 2 2 1

After deleting column 1 and row 3, we get 2 1 6 .0 2 4

By subtracting 2 times row 1 from row 2, we get 2 1 6 . −4 0 −8 After deleting column 2 and row 1, we get

−4 −8 . By subtracting 2 times column 1 from column 2, we get

−4 0 .

Finally, we can drop the second column and we get

(4),

which shows thatR presents the module Z/4Z.

Unfortunately a submodule of a free module of finite dimension is not necessarily finitely generated but, by Proposition 24.5, ifA is a PID, then any submodule of a finitely generated module is finitely generated. This property actually characterizes Noetherian rings. To prove it, we need a slightly different version of Proposition 24.2.

Proposition 24.9. Letf :E→F be a linear map between twoA-modulesE andF.

(1) Given any set of generators (v_1, . . . , vr) of Im(f), for anyr vectorsu_1, . . . , ur∈E such thatf(ui) =vi fori = 1, . . . , r, ifU is the finitely generated submodule ofE generated by (u_1, . . . , ur), then the moduleE is the sum

E = Ker (f) +U. Consequently, if both Ker (f) and Im(f) are finitely generated, thenE is finitely generated.

(2) If E is finitely generated, then so is Im(f). Proof. (1) Pick anyw∈E, writef(w) over the generators (v_1, . . . , vr) of Im(f) asf(w) = a_1v_1 +· · · +arvr, and letu =a_1u_1 +· · · +arur. Observe that

f (w−u) =f(w)−f(u)

=a_1v_1 +· · · +arvr− (a_1f(u_1) +· · · +arf(ur)) =a_1v_1 +· · · +arvr− (a_1v_1 +· · · +arvr) = 0.

Therefore, h =w−u∈ Ker (f), and sincew =h +u withh∈ Ker (f) andu∈U, we have E = Ker (f) +U, as claimed. If Ker (f) is also finitely generated, by taking the union of a finite set of generators for Ker (f) and (v_1, . . . , vr), we obtain a finite set of generators forE.

(2) If (u_1, . . . , un) generateE, it is obvious that (f(u_1), . . . , f(un)) generate Im(f). Theorem 24.10. A ring A is Noetherian iff every submodule N of a finitely generated A-moduleM is itself finitely generated.

Proof. First, assume that every submoduleN of a finitely generatedA-moduleM is itself finitely generated. The ringA is a module over itself and it is generated by the single element 1. Furthermore, every submodule ofA is an ideal, so the hypothesis implies that every ideal inA is finitely generated, which shows thatA is Noetherian.

Now, assume A is Noetherian. First, observe that it is enough to prove the theorem for the finitely generated free modulesAn (withn≥ 1). Indeed, assume that we proved for everyn≥ 1 that every submodule ofAn is finitely generated. IfM is any finitely generated A-module, then there is a surjectionÕ:An M for somen (wheren is the number of→

elements of a finite generating set forM). Given any submoduleN ofM,L =Õ^1(N) is a− submodule ofAn. SinceAn is finitely generated, the submoduleN ofAn is finitely generated, and thenN =Õ(L) is finitely generated.

It remains to prove the theorem for M =An. We proceed by induction onn. Forn = 1, a submoduleN ofA is an ideal, and sinceA is Noetherian,N is finitely generated. For the induction step wheren > 1, consider the projectionπ:An An−1 given by→

π(a_1, . . . , an) = (a_1, . . . , an−1).

The kernel of π is the module Ker (π) ={(0, . . . ,0, an)∈An an∈A} ≈A.|

For any submodule N ofAn, letÕ:N→An−1 be the restriction ofπ toN. SinceÕ(N) is a submodule ofAn−1, by the induction hypothesis, Im(Õ) =Õ(N) is finitely generated. Also, Ker (Õ) =N∩ Ker (π) is a submodule of Ker (π)≈A, and thus Ker (Õ) is isomorphic to an ideal ofA, and thus is finitely generated (sinceA is Noetherian). Since both Im(Õ) and Ker (Õ) are finitely generated, by Proposition 24.9, the submoduleN is also finitely generated.

As a consequence of Theorem 24.10, every finitely generated A-module over a Noetherian ringA is finitely presented, because ifÕ:An M is a surjection onto the finitely generated→

moduleM, then Ker (Õ) is finitely generated. In particular, ifA is a PID, then every finitely generated module is finitely presented.

If the ringA is not Noetherian, then there exist finitely generatedA-modules that are not finitely presented. This is not so easy to prove.

We will prove in Theorem 25.14 that ifA is a Euclidean ring, and more generally in Theorem 25.17 ifA is a PID, then a matrixR can “diagonalized” as

R =QDP−1

whereD is a diagonal matrix. It follows from Proposition 24.8 that every finitely generated moduleM over a PID has a presentation withm generators andr relations of the form

αiei = 0,

whereαi = 0 andα1|α2| · · · |αr, which shows thatM is isomorphic to the direct sum

M≈Am−r A/(α_1A)⊕ · · · ⊕A/(αrA).⊕

This is a version of Theorem 24.32 that will be proved in Section 24.7.

24.3 Tensor Products of Modules over a Commutative Ring

It is possible to define tensor products of modules over a ring, just as in Section 23.1, and the results of this section continue to hold. The results of Section 23.3 also continue to hold since they are based on the universal mapping property. However, the results of Section 23.2 on bases generally fail, except for free modules. Similarly, the results of Section 23.4 on duality generally fail. Tensor algebras can be defined for modules, as in Section 23.5. Symmetric tensor and alternating tensors can be defined for modules but again, results involving bases generally fail.

Tensor products of modules have some unexpected properties. For example, ifp andq are relatively prime integers, then

Z/pZ⊗ZZ/qZ = (0).

This is because, by Bezout’s identity, there area, b∈ Z such that

ap +bq = 1, 24.3. TENSOR PRODUCTS OF MODULES OVER A COMMUTATIVE RING 677

so, for allx∈ Z/pZ and ally∈ Z/qZ, we have

x⊗y =ap(x⊗y) +bq(x⊗y)

=a(px⊗y) +b(x⊗qy)

=a(0⊗y) +b(x⊗ 0)

= 0.

It is possible to salvage certain properties of tensor products holding for vector spaces by restricting the class of modules under consideration. For example, projective modules have a pretty good behavior w.r.t. tensor products.

A free A-moduleF, is a module that has a basis (i.e., there is a family, (ei)i∈I, of linearly independent vectors inF that spanF). Projective modules have many equivalent characterizations. Here is one that is best suited for our needs:

Definition 24.6. AnA-module,P, is projective if it is a summand of a free module, that is, if there is a freeA-module,F, and someA-module,Q, so that

F =P⊕Q.

Given anyA-module,M, we letM∗ = HomA(M, A) be its dual. We have the following proposition:

Proposition 24.11. For any finitely-generated projectiveA-modules,P, and anyA-module, Q, we have the isomorphisms:

P∗∗ ∼= P

HomA(P, Q) ∼= P∗⊗AQ.

Proof sketch. We only consider the second isomorphism. SinceP is projective, we have some A-modules,P_1, F, with

P⊕P_1 =F,

whereF is some free module. Now, we know that for anyA-modules,U, V, W, we have HomA(U⊕V, W)∼= HomA(U, W) HomA(V, W)∼= HomA(U, W)⊕ HomA(V, W), so

P∗⊕P∗ =F∗, HomA(P, Q)⊕ HomA(P_1, Q)∼= HomA(F, Q). 1∼

By tensoring withQ and using the fact that tensor distributes w.r.t. coproducts, we get (P∗⊗AQ)⊕ (P∗⊗Q)∼= (P∗⊕P∗)⊗AQ=F∗⊗AQ.

1 1 ∼

Now, the proof of Proposition 23.9 goes through because F is free and finitely generated, so α⊗: (P∗⊗AQ)⊕ (P∗⊗Q)=F∗⊗AQ−→ HomA(F, Q)∼= HomA(P, Q)⊕ HomA(P_1, Q)1 ∼

is an isomorphism and asα⊗ mapsP∗⊗AQ to HomA(P, Q), it yields an isomorphism between these two spaces.

The isomorphismα⊗:P∗⊗AQ∼= HomA(P, Q) of Proposition 24.11 is still given by

α⊗(u∗⊗f)(x) =u∗(x)f, u∗∈P∗, f∈Q, x∈P.

It is convenient to introduce the evaluation map, Evx:P∗⊗AQ→Q, defined for every x∈P by

Evx(u∗⊗f) =u∗(x)f, u∗∈P∗, f∈Q.

We will need the following generalization of part (4) of Proposition 23.7. Proposition 24.12. Given any two families ofA-modules (Mi)i∈I and (Nj)j∈J (whereI andJ are finite index sets), we have an isomorphism

Mi⊗ M

j

∈

I

≈

(

i,j

)

∈

I

×

J

⊗ Nj).j (Mi

i∈I

Proposition 24.12 also holds for infinite index sets.

Proposition 24.13. LetM andN be twoA-module withN a free module, and pick any basis (v_1, . . . , vn) forN. Then, every element ofM⊗N can expressed in a unique way as a sum of the form

u1⊗v_1 +· · · +un⊗vn, ui∈M, so thatM⊗N is isomorphic toMn (as anA-module).

Proof. SinceN is free with basis (v_1, . . . , vn), we have an isomorphism

N≈Av1⊕ · · · ⊕Avn. By Proposition 24.12, we obtain an isomorphism

M⊗N≈M⊗ (Av1⊕ · · · ⊕Avn)≈ (M⊗Av_1)⊕ · · · ⊕ (M⊗Avn).

Because (v_1, . . . , vn) is a basis of N, each vj is torsion-free so the map a→ avj is an isomorphism ofA ontoAvj, and becauseM⊗A≈M, we have the isomorphism

M⊗N≈ (M⊗A)⊕ · · · ⊕ (M⊗A)≈M⊕ · · · ⊕M =Mn,

as claimed.

Proposition 24.13 also holds for an infinite basis (vj)j∈J ofN. Obviously, a version of Proposition 24.13 also holds ifM is free andN is arbitrary.

The next proposition will be also be needed.

Proposition 24.14. Given anyA-moduleM and any ideal a inA, there is an isomorphism (A/a)⊗AM≈M/aM

given by the map (a⊗u)→au (mod aM), for alla∈A/a and allu∈M. Sketch of proof. Consider the mapÕ: (A/a)×M→M/aM given by Õ(a, u) =au (mod aM)

for all a∈A/a and allu∈M. It is immediately checked thatÕ is well-defined becauseau (mod aM) does not depend on the representativea∈ A chosen in the equivalence classa, andÕ is bilinear. Therefore,Õ induces a linear mapÕ: (A/a)⊗M→ M/aM, such that Õ(a⊗u) =au (mod aM). We also define the mapψ:M→ (A/a)⊗M by

ψ(u) = 1⊗u. Since aM is generated by vectors of the formau witha∈ a andu∈M, and since ψ(au) = 1⊗au =a⊗u = 0⊗u = 0, we see that aM⊆ Ker (ψ), soψ induces a linear mapψ:M/aM→ (A/a)⊗M. We have

ψ (Õ(a⊗u)) =ψ(au) = 1⊗au =a⊗u

and

Õ (ψ(u)) =Õ(1⊗u) = 1u

=u,

which shows thatÕ andψ are mutual inverses.

24.4 Extension of the Ring of Scalars

The need to extend the ring of scalars arises, in particular when dealing with eigenvalues. First, we need to define how to restrict scalar multiplication to a subring. The situation is that we have two ringsA andB, aB-moduleM, and a ring homomorphismρ:A→B. The special case that arises often is thatA is a subring ofB (B could be a field) andρ is the inclusion map. Then, we can makeM into anA-module by defining the scalar multiplication ·:A×M→M as follows:

a·x =ρ(a)x, for alla∈A and allx∈M.

ThisA-module is denoted byρ∗(M). In particular, viewingB asB-module, we obtain the A-moduleρ∗(B).

Now, we can describe the process of scalar extension. Given anyA-moduleM, we make ρ∗(B)⊗AM into a (left)B-module as follows: for everyβ∈B, letµβ:ρ∗(B)×M→ρ∗(B)⊗AM be given by

µβ(β , x) = (ββ )⊗x.

The mapµβ is bilinear so it induces a linear mapµβ:ρ∗(B)⊗AM→ρ∗(B)⊗AM such that µβ(β⊗x) = (ββ )⊗x.

If we define the scalar multiplication·:B× (ρ∗(B)⊗AM)→ρ∗(B)⊗AM by β·z =µβ(z), for allβ∈B and allz∈ρ∗(B)⊗AM,

then it is easy to check that the axioms M1, M2, M3, M4 hold. Let us check M2 and M3. We have

µβ1+β2(β⊗x) = (β_1 +β_2)β⊗x = (β_1β +β_2β )⊗x =β_1β⊗x +β_2β⊗x =µβ1(β⊗x) +µβ2(β⊗x)

and

µβ1β2(β⊗x) =β_1β_2β⊗x

=µβ1(β_2β⊗x)

=µβ1(µβ2(β⊗x)).

With the scalar multiplication by elements of B given by

β· (β⊗x) = (ββ )⊗x,

the tensor product ρ∗(B)⊗AM is aB-module denoted byρ∗(M), orM(B) whenρ is the inclusion ofA intoB. TheB-moduleρ∗(M) is sometimes called the module induced from M by extension toB of the ring of scalars throughρ.

The above process can also be applied to linear maps. We have the following proposition whose proof is given in Bourbaki [12] (Chapter II, Section 5, Proposition 1).

Proposition 24.15. Given a ring homomomorphismρ:A→B and given anyA-module M, the mapÕ:M→ρ∗(ρ∗(M)) given byÕ(x) = 1⊗x isA-linear andÕ(M) spans the B-moduleρ∗(M). For everyB-moduleN, and for everyA-linear mapf :M→ρ∗(N), there is a uniqueB-linear mapf :ρ∗(M)→N such that

fæÕ =f,

or equivalently,

f(1⊗x) =f(x), for allx∈M. As a consequence of Proposition 24.15, we obtain the following result.

Proposition 24.16. Given a ring homomomorphismρ:A→B, for any twoA-modulesM anN, for everyA-linear mapf :M→N, there is a uniqueB-linear mapf :ρ∗(M)→ρ∗(N) (also denoted byρ∗(f)) given by

f = idB⊗f, such that the following diagam commutes:

M ÕM ρ∗(M)

ff

N ÕN ρ∗(N)

Proof. Apply Proposition 24.16 to theA-linear mapÕNæf.

IfS spans the moduleM, it is clear thatÕ(S) spansρ∗(M). In particular, ifM is finitely generated, so ifρ∗(M). Bases ofM also extend to bases ofρ∗(M).

Proposition 24.17. Given a ring homomomorphismρ:A→B, for anyA-modulesM, if (u_1, . . . , un) is a basis ofM, then (Õ(u_1), . . . , Õ(un)) is a basis ofρ∗(M), whereÕ is the A-linear map given byÕ(x) = 1⊗x. Furthermore, ifρ is injective, then so isÕ.

Proof. The first assertion follows immediately from Proposition 24.13, since it asserts that every elementz ofρ∗(M) =ρ∗(B)⊗AM can be written in a unique way as z =b1⊗u_1 +· · · +bn⊗un =b_1(1⊗u_1) +· · · +bn(1⊗un),

andÕ(ui) = 1⊗ui. Next, ifρ is injective, by definition of the scalar multiplication in the A-moduleρ∗(ρ∗(M)), we haveÕ(a_1u_1 +· · · +anun) = 0 iff

ρ(a_1)Õ(u_1) +· · · +ρ(an)Õ(un) = 0, and since (Õ(u_1), . . . , Õ(un)) is a basis ofρ∗(M), we must haveρ(ai) = 0 fori = 1, . . . , n, which (by injectivity ofρ) implies thatai = 0 fori = 1, . . . , n. Therefore,Õ is injective.

In particular, if A is a subring ofB, thenρ is the inclusion map and Proposition 24.17 shows that a basis ofM becomes a basis ofM(B) and thatM is embedded intoM(B). It is also easy to see that ifM andN are two freeA-modules andf :M→N is a linear map represented by the matrixX with respect to some bases (u_1, . . . , un) ofM and (v_1, . . . , vm) ofN, then theB-linear mapf is also represented by the matrixX over the bases (Õ(u_1), . . . , Õ(un)) and (Õ(v_1), . . . , Õ(vm)).

Proposition 24.17 yields another proof of the fact that any two bases of a free A-modules have the same cardinality. Indeed, if m is a maximal ideal in the ringA, then we have the quotient ring homomorphismπ:A→A/m, and we get theA/m-moduleπ∗(M). IfM is free, any basis (u_1, . . . , un) ofM becomes the basis (Õ(u_1), . . . , Õ(un)) ofπ∗(M); butA/m is a field, so the dimensionn is uniquely determined. This argument also applies to an infinite basis (ui)i∈I. Observe that by Proposition 24.14, we have an isomorphism

π∗(M) = (A/m)⊗AM≈M/mM,

soM/mM is a vector space over the fieldA/m, which is the argument used in Theorem 24.1.

Proposition 24.18. Given a ring homomomorphismρ:A→B, for any twoA-modulesM andN, there is a unique isomorphism

ρ∗(M)⊗Bρ∗(N)≈ρ∗(M⊗AN),

such that (1⊗u)⊗ (1⊗v)→ 1⊗ (u⊗v), for allu∈M and allv∈N.

The proof uses identities from Proposition 23.7. It is not hard but it requires a little gymnastic; a good exercise for the reader.

24.5 The Torsion Module Associated With An Endomorphism

We saw in Section 5.7 that given a linear map f :E→E from aK-vector spaceE into itself, we can define a scalar multiplication·:K[X]×E→E that makesE into aK]X]-module. IfE is finite-dimensional, thisK[X]-module denoted byEf is a torsion module, and the main results of this chapter yield important direct sum decompositions ofE into subspaces invariant underf.

Recall that given any polynomialp(X) =a_0Xn +a_1Xn−1 +· · · +an with coefficients in the fieldK, we define the linear mapp(f):E→E by

p(f) =a_0fn +a_1fn−1 +· · · +anid,

wherefk =fæ · · · æf, thek-fold composition off with itself. Note that

p(f)(u) =a_0fn(u) +a_1fn−1(u) +· · · +anu,

for every vectoru∈E. Then, we define the scalar multiplication·:K[X]×E→E by polynomials as follows: for every polynomialp(X)∈K[X], for everyu∈E,

p(X)·u =p(f)(u).3 3If necessary to avoid confusion, we use the notion p(X)·fu instead of p(X)·u.

It is easy to verify that this scalar multiplication satisfies the axioms M1, M2, M3, M4:

p· (u +v) =p·u +p·v (p +q)·u =p·u +q·u (pq)·u =p· (q·u)

1·u =u,

for allp, q∈K[X] and allu, v∈E. Thus, with this new scalar multiplication,E is a K[X]-module denoted byEf.

Ifp =λ is just a scalar inK (a polynomial of degree 0), then

λ·u = (λid)(u) =λu,

which means thatK acts onE by scalar multiplication as before. Ifp(X) =X (the monomial X), then

X·u =f(u).

SinceK is a field, the ringK[X] is a PID.

If E is finite-dimensional, say of dimensionn, sinceK is a subring ofK[X] and sinceE is finitely generated overK, theK[X]-moduleEf is finitely generated overK[X]. Furthermore, Ef is a torsion module. This follows from the Cayley-Hamilton Theorem (Theorem 5.16), but this can also be shown in an elementary fashion as follows. The space Hom(E, E) of linear maps ofE into itself is a vector space of dimensionn^2, therefore then^2+1 linear maps

id, f, f^2, . . . , fn2

are linearly dependent, which yields a nonzero polynomialq such thatq(f) = 0. We can now translate notions defined for modules into notions for endomorphisms of vector spaces.

1. To say thatU is a submodule ofEf means thatU is a subspace ofE invariant under f; that is,f(U)⊆U.

2. To say that V is a cyclic submodule ofEf means that there is some vectoru∈V , such thatV is spanned by (u, f(u), . . . , fk(u), . . .). IfE has finite dimensionn, thenV is spanned by (u, f(u), . . . , fk(u)) for somek≤n−1. We say thatV is a cyclic subspace forf with generatoru. Sometimes,V is denoted byZ(u;f).

3. To say that the ideal a = (p(X)) (withp(X) a monic polynomial) is the annihilator of the submoduleV means thatp(f)(u) = 0 for allu∈V , and we callp the minimal polynomial ofV .

4. SupposeEf is cyclic and let a = (q) be its annihilator, where

q(X) =Xn +an−1Xn−1 +· · · +a_1X +a_0.

Then, there is some vector u such that (u, f(u), . . . , fk(u)) spanEf, and becauseq is the minimal polynomial ofEf, we must havek =n−1. The fact thatq(f) = 0 implies that

fn(u) =−a_0u−a_1f(u)− · · · −an−1fn−1(u),

and sof is represented by the following matrix known as the companion matrix of

q(X): ë0 0 0 · · ·0 −a0 ö

ì

÷

ì

1

0

0

· · ·

0

−

a

1

÷

ì

÷

ì

0 1 0 · · ·0 −a2 ÷

U = ì ... ...... . . ÷.

ì. ÷

ì ... 0 −an−2ø

0 0 0 · · ·1 −an−1

÷

í0 0 0

It is an easy exercise to prove that the characteristic polynomialχU(X) ofU gives backq(X):

χU(X) =q(X).

We will need the following proposition to characterize when two linear maps are similar.

Proposition 24.19. Letf :E→ E andf :E→E be two linear maps over the vector spacesE andE . A linear map g:E→ E can be viewed as a linear map between the K[X]-modulesEf andEf iff

gæf =fæg. Proof. First, supposeg isK[X]-linear. Then, we have

g(p·fu) =p·f g(u)

for allp∈K[X] and allu∈E, so forp =X we get

g(p·fu) =g(X·fu) =g(f(u))

and p·f g(u) =X·f g(u) =f (g(u)), which means thatgæf =fæg.

Conversely, ifgæf =fæg, we prove by induction that

gæfn =fn g, for alln≥ 1.æ

Indeed, we have

gæfn+1 =gæfn f

=fn gæ f

=fn æfæ gæ æ

=fn+1 g,æ

establishing the induction step. It follows that for any polynomialp(X) =n akXk, wek=0 have

n

g(p(X)·fu) =g akfk(u)

k=0

n

= akgæfk(u)

k=0

n

= akfk g(u)

k=0æ

n

= akfk (g(u))

k=0

=p(X)·f g(u),

so,g is indeedK[X]-linear. Definition 24.7. We say that the linear mapsf :E→E andf :E→E are similar iff there is an isomorphismg:E→E such that

f =gæfæg−1,

or equivalently, gæf =fæg. Then, Proposition 24.19 shows the following fact: Proposition 24.20. With notation of Proposition 24.19, two linear maps f andf are similar iffg is an isomorphism betweenEf andEf .

Later on, we will see that the isomorphism of finitely generated torsion modules can be characterized in terms of invariant factors, and this will be translated into a characterization of similarity of linear maps in terms of so-called similarity invariants. Iff andf are represented by matricesA andA over bases ofE andE , thenf andf are similar iff the matricesA andA are similar (there is an invertible matrixP such thatA =P AP−1). Similar matrices (and endomorphisms) have the same characteristic polynomial.

It turns out that there is a useful relationship betweenEf and the moduleK[X]⊗KE. Observe that the map·:K[X]×E→E given by

p·u =p(f)(u)

isK-bilinear, so it yields aK-linear mapσ:K[X]⊗KE→E such that

σ(p⊗u) =p·u =p(f)(u).

We know from Section 24.4 that K[X]⊗ KE is aK[X]-module (obtained from the inclusion K⊆ K[X]), which we will denote byE[X]. SinceE is a vector space, E[X] is a free K[X]-module, and if (u_1, . . . , un) is a basis ofE, then (1⊗u_1, . . . ,1⊗un) is a basis ofE[X].

The freeK[X]-moduleE[X] is not as complicated as it looks. Over the basis (1⊗u_1, . . . ,1⊗un), every elementz∈E[X] can be written uniquely as

z =p_1(1⊗u_1) +· · · +pn(1⊗un) =p1⊗u_1 +· · · +pn⊗un,

wherep_1, . . . , pn are polynomials inK[X]. For notational simplicity, we may write z =p_1u_1 +· · · +pnun,

wherep_1, . . . , pn are viewed as coefficients inK[X]. With this notation, we see thatE[X] is isomorphic to (K[X])n, which is easy to understand.

Observe thatσ isK[X]-linear, because

σ (q(p⊗u)) =σ((qp)⊗u) = (qp)·u

=q(f)(p(f)(u)) =q· (p(f)(u)) =q·σ(p⊗u).

Therefore,σ is a linear map ofK[X]-modules,σ:E[X]→Ef. Using our simplified notation, ifz =p_1u_1 +· · · +pnun∈E[X], then

σ(z) =p_1(f)(u_1) +· · · +pn(f)(un),

which amounts to pluggingf forX and evaluating. Similarly,f is aK[X]-linear map ofEf, because

f (p·u) =f(p(f)(u)) = (fp(f))(u) =p(f)(f(u)) =p·f(u), where we used the fact thatfp(f) =p(f)f becausep(f) is a polynomial inf. By Proposition 24.16, the linear mapf :E→E induces aK[X]-linear mapf :E[X]→E[X] such that

f(p⊗u) =p⊗f(u). Observe that we have f(σ(p⊗u)) =f(p(f)(u)) =p(f)(f(u)) and

σ(f(p⊗u)) =σ(p⊗f(u)) =p(f)(f(u)), so we get

σæf =fæσ. (∗) Using our simplified notation,

f(p_1u_1 +· · · +pnun) =p_1f(u_1) +· · · +pnf(un).

Define the K[X]-linear mapψ:E[X]→E[X] by ψ(p⊗u) = (Xp)⊗u−p⊗f(u). Observe thatψ =X1E[X]−f, which we abbreviate asX1−f. Using our simplified notation ψ(p_1u_1 +· · · +pnun) =Xp_1u_1 +· · · +Xpnun− (p_1f(u_1) +· · · +pnf(un)).

It should be noted that everything we did in Section 24.5 applies to modules over a commutative ringA, except for the statements that assume thatA[X] is a PID. So, ifM is anA-module, we can define theA[X]-modulesMf andM[X] =A[X]⊗AM, except that Mf is generally not a torsion module, and all the results showed above hold. Then, we have the following remarkable result.

Theorem 24.21. (The Characteristic Sequence) LetA be a ring and letE be anA-module. The following sequence ofA[X]-linear maps is exact:

0 E[X] ψ E[X] σ Ef 0.

This means thatψ is injective,σ is surjective, and that Im(ψ) = Ker (σ). As a consequence, Ef is isomorphic to the quotient ofE[X] by Im(X1−f).

Proof. Becauseσ(1⊗u) =u for allu∈E, the mapσ is surjective. We have σ(X(p⊗u)) =X·σ(p⊗u) =f(σ(p⊗u)), which shows that σæX1 =fæσ =σæf, using (∗). This implies that

σæψ =σæ (X1−f) =σæX1−σæf =σæf−σæf = 0,

and thus, Im(ψ)⊆ Ker (σ). It remains to prove that Ker (σ)⊆ Im(ψ).

Since the monomialsXk form a basis ofA[X], by Proposition 24.13 (with the roles ofM andN exchanged), everyz∈E[X] =A[X]⊗AE has a unique expression as

z = Xk uk, k ⊗

for a family (uk) of finite support ofuk∈E. Ifz∈ Ker (σ), then

0 =σ(z) = fk(uk),

k

which allows us to write z = Xk uk− 1⊗ 0 k ⊗

=

k ⊗ fk(uk)Xk uk− 1⊗ k

= (Xk uk− 1⊗fk(uk)) k ⊗

= (Xk(1⊗uk)−fk(1⊗uk))

k

= (Xk1−fk)(1⊗uk).

k

Now,X1 andf commute, since (X1æf)(p⊗u) = (X1)(p⊗f(u)) = (Xp)⊗f(u) and (fæX1)(p⊗u) =f((Xp)⊗u) = (Xp)⊗f(u), so we can writek−1

Xk1k−j−1 ,−fk = (X1−f) (X1)jf

j=0

andk−1

z = (X1−f) (X1)jfk−j−1 (1⊗uk) ,

k j=0

which shows thatz =ψ(y) for somey∈E[X].

Finally, we prove thatψ is injective as follows. We have

ψ(z) =ψ Xk uk k ⊗

= (X1−f) Xk uk k ⊗

= Xk+1 (uk−f(uk+1)), k ⊗

where (uk) is a family of finite support ofuk∈E. Ifψ(z) = 0, then Xk+1 (uk−f(uk+1)) = 0, k ⊗

and because the Xk form a basis ofA[X], we must have uk−f(uk+1) = 0, for allk.

Since (uk) has finite support, there is a largestk, saym+ 1 so thatum+1 = 0, and then from uk =f(uk+1),

we deduce thatuk = 0 for allk. Therefore,z = 0, andψ is injective.

Remark: The exact sequence of Theorem 24.21 yields a presentation ofMf.

Since A[X] is a freeA-module,A[X]⊗AM is a freeA-module, butA[X]⊗AM is generally not a freeA[X]-module. However, ifM is a free module, thenM[X] is a freeA[X]-module, since if (ui)i∈I is a basis forM, then (1⊗ui)i∈I is a basis forM[X]. This allows us to define the characterisctic polynomialχf(X) of an endomorphism of a free moduleM as

χf(X) = det(X1−f).

Note that to have a correct definition, we need to define the determinant of a linear map allowing the indeterminateX as a scalar, and this is what the definition ofM[X] achieves (among other things). Theorem 24.21 can be used to quick a short proof of the CayleyHamilton Theorem, see Bourbaki [12] (Chapter III, Section 8, Proposition 20). Proposition 5.10 is still the crucial ingredient of the proof.

We now develop the theory necessary to understand the structure of finitely generated modules over a PID.

24.6 Torsion Modules over a PID; The Primary Decomposition

We begin by considering modules over a product ring obtained from a direct decomposition, as in Definition 21.3. In this section and the next, we closely follow Bourbaki [13] (Chapter VII). LetA be a commutative ring and let (b_1, . . . ,bn) be ideals inA such that there is an isomorphismA≈A/b1× · · · ×A/bn. From Theorem 21.16 part (b), there exist some elementse_1, . . . , en ofA such that

e^2 =ei

eiej = 0, i =j e_1 +· · · +en = 1A,

and bi = (1A−ei)A, fori, j = 1, . . . , n.

Given anA-moduleM withA≈A/b1×· · ·×A/bn, letMi be the subset ofM annihilated by bi; that is,

Mi ={x∈M|bx = 0, for allb∈ bi}.

Because bi is an ideal, eachMi is a submodule ofM. Observe that ifλ, µ∈A,b∈ bi, and ifλ−µ =b, then for anyx∈Mi, sincebx = 0,

λx = (µ +b)x =µx +bx =µx, soMi can be viewed as aA/bi- module.

Proposition 24.22. Given a ringA≈A/b1× · · · ×A/bn as above, theA-moduleM is the direct sum

M =M1⊕ · · · ⊕Mn, whereMi is the submodule ofM annihilated by bi. Proof. Fori = 1, . . . , n, letpi:M→M be the map given by pi(x) =eix, x∈M.

The mappi is clearly linear, and because of the properties satisfied by theeis, we have p^2 =pi

pipj = 0, i =j

p_1 +· · · +pn = id.

This shows that thepi are projections, and by Proposition 4.6 (which also holds for modules), we have a direct sum

M =p_1(M)⊕ · · · ⊕pn(M) =e_1M⊕ · · · ⊕enM. It remains to show thatMi =eiM. Since (1−ei)ei =ei−e^2 =ei−ei = 0, we see that eiM is annihilated by bi = (1−ei)A. Furthermore, fori =j, for anyx∈M, we have (1−ei)ejx = (ej−eiej)x =ejx, so no nonzero element ofejM is annihilated by 1−ei, and thus not annihilated by bi. It follows thateiM =Mi, as claimed.

Given anA-moduleM, for any nonzeroα∈A, let

M(α) ={x∈M|αx = 0},

the submodule ofM annihilated byα. Ifα dividesβ, thenM(α)⊆M(β), so we can define

Mα = M(αn) ={x∈M| (∃n≥ 1)(αnx = 0)},

n≥1

the submodule ofM consisting of all elements ofM annihilated by some power ofα. IfN is any submodule ofM, it is clear that

Nα =M∩Mα.

Recall that in a PID, an irreducible element is also called a prime element. Definition 24.8. IfA is a PID andp is a prime element inA, we say that a moduleM is p-primary ifM =Mp.

Proposition 24.23. LetM be module over a PIDA. For every nonzeroα∈A, if

α

=

up

n1 pnr

1 · · ·r is a factorization ofα into prime factors (whereu is a unit), then the moduleM(α) annihilated byα is the direct sum

M(α) =M(pn )⊕ · · · ⊕M(pn ).1 r Furthermore, the projection fromM(α) ontoM(pn ) is of the formx→γix, for someγi∈A,i

and

M(pn ) =M(α)∩Mpi.i

Proof. First, observe that sinceM(α) is annihilated byα, we can viewM(α) as aA/(α)module. By the Chinese Remainder Theorem (Theorem 21.15) applied to the ideals (upn ) =

(

p

n

)

,

(

p

n

1

), . . . ,(pn ), we have an isomorphism1 2 r

A/(α)≈A/(pn )× · · · ×A/(pn ).1 r

Since we also have isomorphisms

A/(pn )≈ (A/(α))/((pn )/(α)),i i

we can apply Proposition 24.22, and we get a direct sum

M(α) =N1⊕ · · · ⊕Nr,

where Ni is theA/(α)-submodule ofM(α) annihilated by (pn )/(α), and the projectionsi

onto theNi are of the form stated in the proposition. However,Ni is just theA-module

M

(

p

n ) annihilated bypn , because every nonzero element of (pn

i

modulo (α) of the formap

)/(α) is an equivalence classi i ni for some nonzeroa∈A, and by definition,x∈Ni iffi

0 =apn x =apn x, for alla∈A− {0},i i

in particular fora = 1, which implies thatx∈M(pn ).i

The inclusion M(pn )⊆M(α)∩Mpi is clear. Conversely, pickx∈M(α)∩Mpi, whichi

means thatαx = 0 andpsix = 0 for somes≥ 1. Ifs < ni, we are done, so assumes≥ni. Sincepni is a gcd ofα andpsi, by Bezout, we can writei

pni =λpsi +µαi

for someλ, µ∈A, and thenpn x =λpsix +µαx = 0, which shows thatx∈M(pn ), asi i desired.

Recall that if M is a torsion module over a ringA which is an integral domain, then every finite set of elementsx_1, . . . , xn inM is annihilated bya =a1· · ·an, where eachai annihilatesxi.

Since A is a PID, we can pick a setP of irreducible elements ofA such that every nonzero nonunit ofA has a unique factorization up to a unit. Then, we have the following structure theorem for torsion modules which holds even for modules that are not finitely generated.

Theorem 24.24. (Primary Decomposition Theorem) LetM be a torsion-module over a PID. For every irreducible elementp∈P, letMp be the submodule ofM annihilated by some power ofp. Then,M is the (possibly infinite) direct sum

M = Mp.

p∈P

Proof. SinceM is a torsion-module, for everyx∈ M, there is someα∈ A such that x∈M(α). By Proposition 24.23, ifα =upn1 pnr is a factorization ofα into prime factors1· · · r

(whereu is a unit), then the moduleM(α) is the direct sum

M(α) =M(pn )⊕ · · · ⊕M(pn ).1 r

This means thatx can be written as

x = xp, xp∈Mp,

p∈P

with only finitely manyxp nonzero. If

xp = yp

p∈P p∈P

for all p∈P, with only finitely manyxp andyp nonzero, thenxp andyp are annihilated by some common nonzero elementa∈A, soxp, yp∈M(a). By Proposition 24.23, we must havexp =yp for allp, which proves that we have a direct sum.

It is clear that ifp andp are two irreducible elements such thatp =up for some unitu, thenMp =Mp . Therefore,Mp only depends on the ideal (p).

Definition 24.9. Given a torsion-moduleM over a PID, the modulesMp associated with irreducible elements inP are called thep-primary components ofM.

Thep-primary components of a torsion module uniquely determine the module, as shown by the next proposition.

Proposition 24.25. Two torsion modulesM andN over a PID are isomorphic iff for every every irreducible elementp∈P, thep-primary componentsMp andNp ofM andN are isomorphic.

Proof. Letf :M→N be an isomorphism. For anyp∈P, we havex∈Mp iffpkx = 0 for somek≥ 1, so

0 =f(pkx) =pkf(x),

which shows that f(x)∈Np. Therefore,f restricts to a linear mapf|Mp fromMp to Np. Sincef is an isomorphism, we also have a linear mapf−1:M→N, and our previous reasoning shows thatf−1 restricts to a linear mapf−1 Np fromNp toMp. But,f|Mp and f

−

1

|

Np are mutual inverses, soMp andNp are isomorphic.|

Conversely, ifMp≈ Np for allp∈P, by Theorem 24.24, we get an isomorphism between M =p∈PMp andN =p∈PNp.

In view of Proposition 24.25, the direct sum of Theorem 24.24 in terms of itsp-primary components is called the canonical primary decomposition ofM.

IfM is a finitely generated torsion-module, then Theorem 24.24 takes the following form.

Theorem 24.26. (Primary Decomposition Theorem for finitely generated torsion modules) LetM be a finitely generated torsion-module over a PIDA. If Ann(M) = (a) and ifa = upn1 pnr is a factorization ofa into prime factors, thenM is the finite direct sum1· · · r

Furthermore, the projection ofM overM(p

r

M = M(pn ).i

i=1

n ) is of the formx→γix, for someγi∈A.i

Proof. This is an immediate consequence of Proposition 24.23.

In particular, Theorem 24.26 applies when A = Z. In this case,M is a finitely generated torsion abelian group, and the theorem says that such a group is the direct sum of a finite number of groups whose elements have order some power of a prime numberp.

Theorem 24.24 has several useful corollaries.

Proposition 24.27. IfM is a torsion module over a PID, for every submoduleN ofM, we have a direct sum

N = N∩Mp.

p∈P

Proof. It is easily verified thatN∩Mp is thep-primary component ofN. Proposition 24.28. IfM is a torsion module over a PID, a submoduleN ofM is a direct factor ofM iffNp is a direct factor ofMp for every irreducible elementp∈A.

Proof. This is because ifN andN are two submodules ofM, we haveM =N⊕N iff, by Proposition 24.27,Mp =Np⊕Np for every irreducible elementsp∈A.

AnA-moduleM is said to be semi-simple iff for every submoduleN ofM, there is some submoduleN ofM such thatM =N⊕N .

Proposition 24.29. LetA be a PID which is not a field, and letM be anyA-module. Then, M is semi-simple iff it is a torsion module and ifMp =M(p) for every irreducible element p∈A (in other words, ifx∈M is annihilated by a power ofp, then it is already annihilated byp).

Proof. Assume thatM is semi-simple. Letx∈ M and pick any irreducible elementp∈A. Then, the submodulepAx has a supplementN such that

M =pAx⊕N,

so we can writex =pax +y, for somey∈N and somea∈A. But then,

y = (1−pa)x,

and sincep is irreducible,p is not a unit, so 1−pa = 0. Observe that

p(1−ap)x =py∈pAx∩N = (0).

Sincep(1−ap) = 0,x is a torsion element, and thusM is a torsion module. The above argument shows that

p(1−ap)x = 0, which implies thatpx =ap^2x, and by induction,

px =anpn+1x, for alln≥ 1.

If we pickx inMp, then there is somem≥ 1 such thatpmx = 0, and we conclude that

px = 0.

Therefore,Mp =M(p), as claimed.

Conversely, assume that M is a torsion-module and thatMp =M(p) for every irreducible elementp∈A. By Proposition 24.28, it is sufficient to prove that a module annihilated by a an irreducible element is semi-simple. This is because such a module is a vector space over the fieldA/(p) (recall that in a PID, an ideal (p) is maximal iffp is irreducible), and in a vector space, every subspace has a supplement.

Theorem 24.26 shows that a finitely generated torsion module is a direct sum of p-primary modulesMp. We can do better. In the next section, we show that each primary moduleMp is the direct sum of cyclic modules of the formA/(pn).

24.7 Finitely Generated Modules over a PID; Invariant Factor Decomposition

There are several ways of obtaining the decomposition of a finitely generated module as a direct sum of cyclic modules. One way to proceed is to first use the Primary Decomposition Theorem and then to show how each primary moduleMp is the direct sum of cyclic modules of the formA/(pn). This is the approach followed by Lang [65] (Chapter III, section 7), among others. We prefer to use a proposition that produces a particular basis for a submodule of a finitely generated free module, because it yields more information. This is the approach followed in Dummitt and Foote [30] (Chapter 12) and Bourbaki [13] (Chapter VII). The proof that we present is due to Pierre Samuel.

Proposition 24.30. LetF be a finitely generated free module over a PIDA, and letM be any submodule ofF. Then,M is a free module and there is a basis (e_1, ..., en) ofF, some q≤n, and some nonzero elementsa_1, . . . , aq∈A, such that (a_1e_1, . . . , aqeq) is a basis ofM andai dividesai+1 for alli, with 1≤i≤q− 1.

Proof. The proposition is trivial whenM ={0}, thus assume thatM is nontrivial. Pick some basis (u_1, . . . , un) forF. LetL(F, A) be the set of linear forms onF. For anyf∈L(F, A), it is immediately verified thatf(M) is an ideal inA. Thus,f(M) =ahA, for someah∈A, since every ideal inA is a principal ideal. SinceA is a PID, any nonempty family of ideals inA has a maximal element, so letf be a linear map such thatahA is a maximal ideal inA. Letπi:F→A be thei-th projection, i.e.,πi is defined such thatπi(x_1u_1+· · ·+xnun) =xi. It is clear thatπi is a linear map, and sinceM is nontrivial, one of theπi(M) is nontrivial, andah = 0. There is somee∈M such thatf(e ) =ah.

We claim that, for everyg∈L(F, A), the elementah∈A dividesg(e ). Indeed, ifd is the gcd ofah andg(e ), by the B´ezout identity, we can write d =rah +sg(e ), for somer, s∈A, and thus d =rf(e ) +sg(e ) = (rf +sg)(e ).

However, rf +sg∈L(F, A), and thus, ahA⊆dA⊆ (rf +sg)(M),

since d dividesah, and by maximality ofahA, we must haveahA =dA, which implies that d =ah, and thus,ah dividesg(e ). In particular,ah divides eachπi(e ) and letπi(e ) =ahbi, withbi∈A.

Let e =b_1u_1 +· · · +bnun. Note that

e =π_1(e )u_1 +· · · +πn(e )un =ahb_1u_1 +· · · +ahbnun,

and thus,e =ahe. Sinceah =f(e ) =f(ahe) =ahf(e), and sinceah = 0, we must have f(e) = 1.

Next, we claim that

F =Ae⊕f−1(0) and

M =Ae⊕ (M∩f−1(0)), withe =ahe.

Indeed, everyx∈F can be written as

x =f(x)e + (x−f(x)e), and sincef(e) = 1, we havef(x−f(x)e) =f(x)−f(x)f(e) =f(x)−f(x) = 0. Thus, F =Ae +f−1(0). Similarly, for anyx∈M, we havef(x) =rah, for somer∈A, and thus, x =f(x)e + (x−f(x)e) =rahe + (x−f(x)e) =re + (x−f(x)e), we still havex^1(0), and clearly,x−f(x)e =x−rahe =x−re∈M, since−f(x)e∈f−

e∈M. Thus,M =Ae + (M∩f−1(0)).

To prove that we have a direct sum, it is enough to prove that Ae∩f−1(0) ={0}. For anyx =re∈Ae, iff(x) = 0, thenf(re) =rf(e) =r = 0, sincef(e) = 1 and, thus,x = 0. Therefore, the sums are direct sums.

We can now prove thatM is a free module by induction on the size,q, of a maximal linearly independent family forM.

Ifq = 0, the result is trivial. Otherwise, since

M =Ae⊕ (M∩f−1(0)),

it is clear that M∩ f−1(0) is a submodule ofF and that every maximal linearly independent family inM∩f− (0) has at mostq− 1 elements. By the induction hypothesis,M∩f−1(0) is a free module, and by addinge to a basis ofM∩f−1(0), we obtain a basis forM, since the sum is direct.

The second part is shown by induction on the dimensionn ofF.

The casen = 0 is trivial. Otherwise, since

F =Ae⊕f−1(0),

and since, by the previous argument, f−1(0) is also free,f−1(0) has dimensionn− 1. By the induction hypothesis applied to its submoduleM∩f−1(0), there is a basis (e_2, . . . , en) off−1(0), someq≤n, and some nonzero elementsa_2, . . . , aq∈A, such that, (a_2e_2, . . . , aqeq) is a basis ofM∩f−1(0), andai dividesai+1 for alli, with 2≤i≤q− 1. Lete_1 =e, and a_1 =ah, as above. It is clear that (e_1, . . . , en) is a basis ofF, and that that (a_1e_1, . . . , aqeq) is a basis ofM, since the sums are direct, ande =a_1e_1 =ahe. It remains to show thata_1 dividesa_2. Consider the linear mapg:F→ A such thatg(e_1) =g(e_2) = 1, andg(ei) = 0, for alli, with 3≤i≤n. We haveah =a1 =g(a_1e_1) =g(e )∈g(M), and thusahA⊆ g(M). SinceahA is maximal, we must haveg(M) =ahA =a_1A. Sincea_2 =g(a_2e_2)∈g(M), we havea2∈a_1A, which shows thata_1 dividesa_2.

We need the following basic proposition. Proposition 24.31. For any commutative ringA, ifF is a freeA-module and if (e_1, . . . , en) is a basis ofF, for any elementsa_1, . . . , an∈A, there is an isomorphism

F/(Aa_1e1⊕ · · · ⊕Aanen)≈ (A/a_1A)⊕ · · · ⊕ (A/anA).

Proof. Letσ:F→A/(a_1A)⊕ · · · ⊕A/(anA) be the linear map given by

σ(x_1e_1 +· · · +xnen) = (x_1, . . . , xn),

where xi is the equivalence class ofxi inA/aiA. The mapσ is clearly surjective, and its kernel consists of all vectorsx_1e_1 +· · · +xnen such thatxi∈aiA, fori = 1, . . . , n, which means that

Ker (σ) =Aa_1e1⊕ · · · ⊕Aanen. SinceM/Ker (σ) is isomorphic to Im(σ), we get the desired isomorphism. We can now prove the existence part of the structure theorem for finitely generated modules over a PID.

Theorem 24.32. LetM be a finitely generated nontrivialA-module, whereA a PID. Then, M is isomorphic to a direct sum of cyclic modules

M≈A/a1⊕ · · · ⊕A/am,

where the ai are proper ideals ofA (possibly zero) such that

a1⊆ a2⊆ · · · ⊆ am =A.

More precisely, if a_1 =· · · = ar = (0) and (0) = ar+1⊆ · · · ⊆ am =A, then M≈Ar (A/ar+1⊕ · · · ⊕A/am),⊕

whereA/ar+1⊕ · · · ⊕A/am is the torsion submodule ofM. The moduleM is free iffr =m, and a torsion-module iff r = 0. In the latter case, the annihilator ofM is a_1.

Proof. SinceM is finitely generated and nontrivial, there is a surjective homomorphism Õ:An M for somen≥ 1, andM is isomorphic toAn/Ker (Õ). Since Ker (Õ) is a submod→

ule of the free moduleAn, by Proposition 24.30, Ker (Õ) is a free module and there is a basis (e_1, . . . , en) ofAn and some nonzero elementsa_1, . . . , aq (q≤n) such that (a_1e_1, . . . , aqeq) is a basis of Ker (Õ) anda1|a2| · · · |aq. Letaq+1 =. . . =an = 0.

By Proposition 24.31, we have an isomorphism

An/Ker (Õ)≈A/a_1A⊕ · · · ⊕A/anA.

Whenever ai is unit, the factorA/aiA = (0), so we can weed out the units. Letr =n−q, and lets∈ N be the smallest index such thatas+1 is not a unit. Note thats = 0 means that there are no units. Also, asM = (0),s < n. Then,

M≈An/Ker (Õ)≈A/as+1A⊕ · · · ⊕A/anA.

Letm =r +q−s =n−s. Then, we have the sequence

as+1, . . . , aq, aq+1, . . . , an,

q−s r=n−q

whereas+1|as+2| · · · |aq are nonzero and nonunits andaq+1 =· · · =an = 0, so we define them ideals ai as follows:

a

i

=

(0) if 1≤i≤r

ar+q+1−iA ifr + 1≤i≤m.

With these definitions, the ideals ai are proper ideals and we have

ai⊆ ai+1, i = 1, . . . , m− 1.

Whenr = 0, sinceas+1|as+2| · · · |an, it is clear that a_1 =anA is the annihilator ofM. The other statements of the theorem are clear.

The natural number r is called the free rank or Betti number of the moduleM. The generatorsα_1, . . . , αm of the ideals a_1, . . . ,am (defined up to a unit) are often called the invariant factors of M (in the notation of Theorem 24.32, the generators of the ideals a_1, . . . ,am are denoted byaq, . . . , as+1,s≤q).

As corollaries of Theorem 24.32, we obtain again the following facts established in Section 24.1:

1. A finitely generated module over a PID is the direct sum of its torsion module and a free module.

2. A finitely generated torsion-free module over a PID is free.

It turns out that the ideals a1⊆ a2⊆ · · · ⊆ am =A are uniquely determined by the moduleM. Uniqueness proofs found in most books tend to be intricate and not very intuitive. The shortest proof that we are aware of is from Bourbaki [13] (Chapter VII, Section 4), and uses wedge products.

The following preliminary results are needed.

Proposition 24.33. IfA is a commutative ring and if a_1, . . . ,am are ideals ofA, then there is an isomorphism

A/a1⊗ · · · ⊗A/am≈A/(a_1 +· · · +am). Sketch of proof. We proceed by induction onm. Form = 2, we define the map Õ:A/a1×A/a2→A/(a_1 +a_2) by

Õ(a, b) =ab (mod a_1 +a_2).

It is well-defined because if a =a +a_1 andb =b +a_2 witha1∈ a_1 anda2∈ a_2, then a b = (a +a_1)(b +a_2) =ab +ba_1 +aa_2 +a_1a_2,

and so a b≡ab (mod a_1 +a_2).

It is also clear that this map is bilinear, so it induces a linear mapÕ:A/a1⊗A/a2→A/(a1 +a2) such thatÕ(a⊗b) =ab (mod a1 +a2).

Next, observe that any arbitrary tensor

a1⊗b_1 +· · · +an⊗bn inA/a1⊗A/a_2 can be rewritten as 1⊗ (a_1b_1 +· · · +anbn), which is of the form 1⊗s, withs∈A. We can use this fact to show thatÕ is injective and surjective, and thus an isomorphism.

For example, ifÕ(1⊗s) = 0, becauseÕ(1⊗s) =s (mod a_1 + a_2), we haves∈ a_1 + a_2, so we can writes =a +b witha∈ a_1 andb∈ a2. Then

1⊗s = 1⊗a +b = 1⊗ (a +b) = 1⊗a + 1⊗b =a⊗ 1 + 1⊗b = 0 + 0 = 0,

sincea∈ a_1 andb∈ a_2, which proves injectivity.

Recall that the exterior algebra of anA-moduleM is defined by

k

M = (M).

k≥0

Proposition 24.34. IfA is a commutative ring, then for anyn modulesMi, there is an isomorphism

n n

( Mi.

i=1

Mi)≈ i=1

A proof can be found in Bourbaki [12] (Chapter III, Section 7, No 7, Proposition 10). Proposition 24.35. LetA be a commutative ring and let a_1, . . . ,an ben ideals ofA. If the moduleM is the direct sum ofn cyclic modules

M =A/a1⊕ · · · ⊕A/an,

then for everyp > 0, the exterior powerpM is isomorphic to the direct sum of the modules A/aH, whereH ranges over all subsetsH⊆ {1, . . . , n} withp elements, and with

aH = ah.

h∈H

Proof. Ifui is the image of 1 inA/ai, thenA/ai is equal toAui. By Proposition 24.34, we haven

M≈ (Aui).

i=1

We also havek

(Aui) = (Aui)≈A⊕Aui,

k≥0

sinceaui∧bui = 0, and it follows that

p

M

≈

H ⊆{1,...,n}H={k1,...,kp}

(Auk1)⊗ · · · ⊗ (Aukp).

However, by Proposition 24.33, we have

(Auk1)⊗ · · · ⊗ (Aukp) =A/ak1⊗ · · · ⊗A/akp≈A/(ak1 +· · · +akp) =A/aH.

Therefore,p

M≈H⊆{1,...,n} |H|=p

A/aH,

as claimed.

When the ideals ai form a chain of inclusions a1⊆ · · · ⊆ an, we get the following remarkable result.

Proposition 24.36. LetA be a commutative ring and let a_1, . . . ,an ben ideals ofA such that a1⊆ a2⊆ · · · ⊆ an. If the moduleM is the direct sum ofn cyclic modules

M =A/a1⊕ · · · ⊕A/an,

then for everyp with 1≤p≤n, the ideal ap is the annihilator of the exterior powerpM. If an =A, thenpM = (0) forp = 1, . . . , n, andpM = (0) forp > n.

Proof. With the notation of Proposition 24.35, we have aH = amax(H), where max(H) is the greatest element in the setH. Since max(H)≥p for any subset withp elements and since max(H) =p whenH ={1, . . . , p}, we see that

ap = aH.

H⊆{1,...,n}

|H|=p

By Proposition 24.35, we have

p

M≈H⊆{1,...,n} |H|=p

A/aH

which proves that ap is indeed the annihilator ofpM. The rest is clear.

Propostion 24.36 immediately implies the following crucial fact. Proposition 24.37. LetA be a commutative ring and let a_1, . . . ,am bem ideals ofA and a_1, . . . ,an ben ideals ofA such that a1⊆ a2⊆ · · · ⊆ am =A and a1⊆ a2⊆ · · · ⊆ an =A If we have an isomorphism

A/a1⊕ · · · ⊕A/am≈A/a1⊕ · · · ⊕A/an, thenm =n and ai = ai fori = 1, . . . , n.

Proposition 24.37 yields the uniqueness of the decomposition in Theorem 24.32. Theorem 24.38. (Invariant Factors Decomposition) LetM be a finitely generated nontrivial A-module, whereA a PID. Then,M is isomorphic to a direct sum of cyclic modules M≈A/a1⊕ · · · ⊕A/am, where the ai are proper ideals ofA (possibly zero) such that

a1⊆ a2⊆ · · · ⊆ am =A. More precisely, if a_1 =· · · = ar = (0) and (0) = ar+1⊆ · · · ⊆ am =A, then Mr (A/ar+1⊕ · · · ⊕A/am),≈A⊕

where A/ar+1⊕ · · · ⊕A/am is the torsion submodule ofM. The moduleM is free iffr =m, and a torsion-module iff r = 0. In the latter case, the annihilator ofM is a_1. Furthermore, the integerr and ideals a1⊆ a2⊆ · · · ⊆ am =A are uniquely determined byM.

Proof. By Theorem 24.7, sinceMtor =A/ar+1⊕ · · · ⊕A/am, we know that the dimensionr of the free summand only depends onM. The uniqueness of the sequence of ideals follows from Proposition 24.37.

In view of the uniqueness part of Theorem 24.38, we make the following definition.

Definition 24.10. Given a finitely generated moduleM over a PIDA as in Theorem 24.38, the ideals ai =αiA are called the invariant factors ofM. The generatorsαi of these ideals (uniquely defined up to a unit) are also called the invariant factors ofM.

Proposition 24.30 can be sharpened as follows:

Proposition 24.39. LetF be a finitely generated free module over a PIDA, and letM be any submodule ofF. Then,M is a free module and there is a basis (e_1, ..., en) ofF, some q≤n, and some nonzero elementsa_1, . . . , aq∈A, such that (a_1e_1, . . . , aqeq) is a basis ofM andai dividesai+1 for alli, with 1≤i≤q−1. Furthermore, the free moduleM with basis (e_1, . . . , eq) and the idealsa_1A, . . . , aqA are uniquely determined byM; the quotient module M /M is the torsion module ofF/M, and we have an isomorphism

M /M≈A/a_1A⊕ · · · ⊕A/aq/A. Proof. Sinceai = 0 fori = 1, . . . , q, observe that

M ={x∈F| (∃β∈A, β = 0)(βx∈M)},

which shows thatM /M is the torsion module ofF/M. Therefore,M is uniquely determined. Since

M =Aa_1e1⊕ · · · ⊕Aaqeq, by Proposition 24.31 we have an isomorphism

M /M≈A/a_1A⊕ · · · ⊕A/aqA.

Now, it is possible that the firsts elementsai are units, in which caseA/aiA = (0), so we can eliminate such factors and we get

M /M≈A/as+1A⊕ · · · ⊕A/aqA,

with aqA⊆aq−1A⊆ · · · ⊆ as+1A =A. By Proposition 24.37,q−s and the idealsajA are uniquely determined forj =s + 1, . . . , q, and sincea_1A =· · · =asA =A, theq idealsaiA are uniquely determined.

The idealsa_1A, . . . , aqA of Proposition 24.39 are called the invariant factors ofM with respect toF. They should not be confused with the invariant factors of a moduleM.

It turns out that a_1, . . . , aq can also be computed in terms of gcd’s of minors of a certain matrix. Recall that ifX is anm×n matrix, then ak×k minor ofX is the determinant of anyk×k matrix obtained by pickingk columns ofX, and thenk rows from thesek columns.

Proposition 24.40. LetF be a free module of finite dimension over a PID, (u_1, . . . , un) be a basis ofF,M be a submodule ofF, and (x_1, . . . , xm) be a set of generators ofM. If a_1A, . . . , aqA are the invariant factors ofM with respect toF as in Proposition 24.39, then fork = 1, . . . , q, the producta1· · ·ak is a gcd of thek×k minors of then×m matrixX whose columns are the coordinates of the xj over theui.

Proof. Proposition 24.30 shows thatM⊆a_1F. Consequently, the coordinates of any element ofM are multiples ofa_1. On the other hand, we know that there is a linear formf for which a_1A is a maximal ideal and somee∈M such thatf(e ) =a_1. If we writee as a linear combination of thexi, we see thata_1 belongs to the ideal spanned by the coordinates of the xi over the basis (u_1, . . . , un). Since these coordinates are all multiples ofa_1, it follows that a_1 is their gcd, which proves the casek = 1.

For any k≥ 2, consider the exterior powerkM. Using the notation of the proof of Proposition 24.30, the moduleM has the basis (a_1e_1, . . . , aqeq), sokM has a basis consisting of elements of the form

ai1ei1∧ · · · ∧aikeik =ai1· · ·aikei1∧ · · · ∧eik,

However, the vectors kF induced by the for all sequences (i_1, . . . , ik) such that 1≤i_1< i_2<· · ·< ik ≤q. ei1∧ · · · ∧eik form a basis ofkF. Thus, the map fromkM into inclusionM⊆F defines an isomorphism ofkM onto the submodule ofkF having the elementsai1· · ·aikei1∧ · · · ∧eik as a basis. Sinceaj is a multiple of theai fori < j, the productsai1· · ·aik are all multiples ofδk =a1· · ·ak, and one of these is equal toδk. The reasoning used fork = 1 shows thatδk is a gcd of the set of coordinates of any spanning set of

k M over any basis ofkF. If we pick as basis ofkF the wedge productsui1∧ · · · ∧uik, and as generators ofkM the wedge productsxi1∧ · · · ∧xik, it is easy to see that the coordinates of thexi1∧ · · · ∧xik are indeed determinants which are thek×k minors of the matrixX.

Proposition 24.40 yields a_1, . . . , aq (up to units) as follows: First,a_1 is a gcd of the entries inX. Having computeda_1, . . . , ak, letbk =a1· · ·, ak, computebk+1 =a1· · ·akak+1 as a gcd of all the (k + 1)× (k + 1) minors ofX, and thenak+1 is obtained by dividingbk+1 bybk (recall that a PID is an integral domain).

We also have the following interesting result about linear maps between free modules over a PID.

Proposition 24.41. LetA be a PID, letF be a free module of dimensionn,F be a free module of dimensionm, andf :F→F be a linear map fromF toF . Then, there exist a basis (e_1, . . . , en) ofF, a basis (e_1, . . . , em) ofF , and some nonzero elementsα_1, . . . αr∈A such that

f

(

e

i

) =

αiei if 1≤i≤r

0 ifr + 1≤i≤n,

andα1|α2 | · · · |αr. Furthermore, the idealsα_1A, . . . , αrA are the invariant factors off(F) with respectF .

Proof. LetF_0 be the kernel off. SinceM =f(F) is a submodule of the free moduleF , it is free, and similarlyF_0 is free as a submodule of the free moduleF (by Proposition 24.30). By Proposition 24.2, we have

F =F0⊕F_1,

where F_1 is a free module, and the restriction off toF_1 is an isomorphism ontoM = f(F). Proposition 24.39 applied toF andM yields a basis (e_1, . . . , em) ofF such that (α_1e_1, . . . , αrer) is a basis ofM , whereα_1A, . . . , αrA are the invariant factors forM with respect toF . Since the restriction off toF_1 is and isomorphism, there is a basis (e_1, . . . , er) ofF_1 such that

f(ei) =αiei, i = 1, . . . , r.

We can extend this basis to a basis ofF by picking a basis ofF_0 (a free module), which yields the desired result.

The matrix version of Proposition 24.41 is the following proposition. Proposition 24.42. IfX is anm×n matrix of rankr over a PIDA, then there exist some invertiblen×n matrixP, some invertible m×m matrixQ, and am×n matrixD of the

form ëα1 0 · · ·0 0 · · ·0ö

ì 0 α2 · · ·0 0 · · ·0÷

ì ÷

ì . . ... . . · · ·.÷

ì ÷

D

=

ì 0 0 · · · αr 0 · · ·0÷

ì ÷

ì 0 0 · · ·0 0 · · ·0÷

ì ÷

ì . . · · ·. . ... .÷

í ø

0 0 · · ·0 0 · · ·0

for some nonzeroαi∈A, such that

(1) α1|α2| · · · |αr, (2)1, and X =QDP−

(3) Theαis are uniquely determined up to a unit.

The idealsα_1A, . . . , αrA are called the invariant factors of the matrixX. Recall that twom×n matricesX andY are equivalent iff

Y =QXP−1,

for some invertible matrices,P andQ. Then, Proposition 24.42 implies the following fact. Proposition 24.43. Twom×n matricesX andY are equivalent iff they have the same invariant factors.

If X is the matrix of a linear mapf :F→F with respect to some basis (u_1, . . . , un) ofF and some basis (u_1, . . . , um) ofF , then the columns ofX are the coordinates of the f(uj) over theui, where thef(uj) generatef(F), so Proposition 24.40 applies and yields the following result:

Proposition 24.44. IfX is am×n matrix or rankr over a PIDA, and ifα_1A, . . . , αrA are its invariant factors, thenα_1 is a gcd of the entries inX, and fork = 2, . . . , r, the productα1· · ·αk is a gcd of allk×k minors ofX.

There are algorithms for converting a matrix X over a PID to the formX =QDP−1 as described in Proposition 24.42. For Euclidean domains, this can be achieved by using the elementary row and column operationsP(i, k),Ei,j;β, andEi,λ described in Chapter 6, where we require the scalarλ used inEi,λ to be a unit. For an arbitrary PID, another kind of elementary matrix (containing some 2× 2 submatrix in addition to diagonal entries) is needed. These procedures involve computing gcd’s and use the Bezout identity to mimic division. Such methods are presented in Serre [92], Jacobson [57], and Van Der Waerden [108], and sketched in Artin [3]. We describe and justify several of these methods in Section 25.4.

From Section 24.2, we know that a submodule of a finitely generated module over a PID is finitely presented. Therefore, in Proposition 24.39, the submoduleM of the free module F is finitely presented by some matrixR with a number of rows equal to the dimension ofF. Using Theorem 25.17, the matrixR can be diagonalized asR =QDP−1 whereD is a diagonal matrix. Then, the columns ofQ form a basis (e_1, . . . , en) ofF, and since RP =QD, the nonzero columns ofRP form the basis (a_1e_1, . . . , aqeq) ofM. When the ring A is a Euclidean domain, Theorem 25.14 shows thatP andQ are products of elementary row and column operations. In particular, whenA = Z, in which cases our Z-modules are abelian groups, we can findP andQ using Euclidean division.

In this case, a finitely generated submoduleM of Zn is called a lattice. It is given as the set of integral linear combinations of a finite set of integral vectors.

Here is an example taken from Artin [3] (Chapter 12, Section 4). LetF be the free Z-module Z^2, and letM be the lattice generated by the columns of the matrix

R

=

2

1

− .1 2

The columns ( u_1, u_2) ofR are linearly independent, but they are not a basis of Z^2. For example, in order to obtaine_1 as a linear combination of these columns, we would need to solve the linear system

2x−y = 1 x + 2y = 0.

From the second equation, we getx =−2y, which yields

−5y = 1.

But,y =−1/5 is not an integer. We leave it as an exercise to check that 1 0 2 −1 1 1= 1 0 , −3 1 1_2 1 2 0 5 which means that2 −1 = 1 0 1 0 2 1

− ,1_2 3 1 0 5−1 1 soR =QDP−1 with

Q =1 0 , D =1 0 , P =1 1 .3 1 0 5 1 2 The new basis (u_1, u_2) for Z^2 consists of the columns ofQ and the new basis forM consists of the columns (u_1,5u_2) ofQD, where

QD =1 0 .3 5

A picture of the lattice and its generators (u_1, u_2) and of the same lattice with the new basis (u_1,5u_2) is shown in Figure 24.1, where the lattice points are displayed as stars.

* * ** ** ** ** * * ** ** * * ** ** ** ** * * ** **

Figure 24.1: Diagonalization applied to a lattice

The invariant factor decomposition of a finitely generated moduleM over a PIDA given by Theorem 24.38 says that

M tor≈A/ar+1⊕ · · · ⊕A/am,

a direct sum of cyclic modules, with (0) = ar+1⊆ · · · ⊆ am = A. Using the Chinese Remainder Theorem (Theorem 21.15), we can further decompose each moduleA/αiA into a direct sum of modules of the formA/pnA, wherep is a prime inA.

Theorem 24.45. (Elementary Divisors Decomposition) LetM be a finitely generated nontrivialA-module, whereA a PID. Then,M is isomorphic to the direct sumAr Mtor, where Ar is a free module and where the torsion moduleMtor is a direct sum of cyclic modules of⊕ the formA/pni,j, for some primesp_1, . . . , pt∈A and some positive integersni,j, such thati

for eachi = 1, . . . , t, there is a sequence of integers

1≤ni,1, . . . , ni,1 < ni,2, . . . , ni,2 <· · ·< ni,si, . . . , ni,si,

withsi≥ 1, and whereni,j occursmi,j≥ 1 times, forj = 1, . . . , si. Furthermore, the irreducible elementspi and the integersr, t, ni,j, si,mi,j are uniquely determined.

Proof. By Theorem 24.38, we already know that M≈ Ar Mtor, where r is uniquely determined, and where⊕

Mtor≈A/ar+1⊕ · · · ⊕A/am,

a direct sum of cyclic modules, with (0) = ar+1⊆ · · · ⊆ am =A. Then, each ai is a principal ideal of the formαiA, whereαi = 0 andαi is not a unit. Using the Chinese Remainder Theorem (Theorem 21.15), if we factorαi into prime factors as

k1 pkhαi =up1· · · h,

withkj≥ 1, we get an isomorphism A/αiA≈A/pkA⊕ · · · ⊕A/pkh.1 h This implies thatMtor is the direct sum of modules of the formA/pni,j, for some primesi

pi∈A.

To prove uniqueness, observe that thepi-primary component ofMtor is the direct sum ni,si(A/pni,1 A)mi,1 (A/pi A)mi,si,i ⊕ · · · ⊕

and these are uniquely determined. Sinceni,1<· · ·< ni,si, we have

n

i,

1

p

ni,si pi A =A,i A⊆ · · · ⊆

Proposition 24.37 implies that the irreducible elementspi andni,j,si, andmi,j are unique.

In view of Theorem 24.45, we make the following definition.

Definition 24.11. Given a finitely generated moduleM over a PIDA as in Theorem 24.45, the idealspni,j are called the elementary divisors ofM, and themi,j are their multiplicities.i A

The ideal (0) is also considered to be an elementary divisor andr is its multiplicity.

Remark: Theorem 24.45 shows how the elementary divisors are obtained from the invariant factors: the elementary divisors are the prime power factors of the invariant factors. Conversely, we can get the invariant factors from the elementary divisors. We may assume thatM is a torsion module. Let

m = max{mi,1 +· · · +mi,si},

1≤i≤t

and construct thet×m matrixC = (cij) whoseith row is the sequence ni,si, . . . , ni,si, . . . , ni,2, . . . , ni,2, ni,1, . . . , ni,1,0, . . . ,0, padded with 0’s if necessary to make it of lengthm. Then, thejth invariant factor is αc1j pc2j pct,j A.jA =p1 2 · · · t

Observe that because the last column contains at least one prime, theαi are not units, and αm|αm−1| · · · |α_1, so thatα_1A⊆ · · · ⊆αm−1A⊆αmA =A, as desired.

From a computational point of view, finding the elementary divisors is usually practically impossible, because it requires factoring. For example, ifA =K[X] whereK is a field, such asK = R orK = C, factoring amounts to finding the roots of a polynomial, but by Galois theory, in general, this is not algorithmically doable. On the other hand, the invariant factors can be computed using elementary row and column operations.

It can also be shown that A and the modules of the formA/pnA are indecomposable (withn > 0). A moduleM is said to be indecomposable ifM cannot be written as a direct sum of two nonzero proper submodules ofM. For a proof, see Bourbaki [13] (Chapter VII, Section 4, No. 8, Proposition 8). Theorem 24.45 shows that a finitely generated module over a PID is a direct sum of indecomposable modules.

We will now apply the structure theorems for finitely generated (torsion) modules to the K[X]-moduleEf associated with an endomorphismf on a vector spaceE.

Chapter 25 The Rational Canonical Form and Other Normal Forms

25.1 The Rational Canonical Form

Let E be a finite-dimensional vector space over a fieldK, and letf :E→ E be an endomorphism ofE. We know from Section 24.5 that there is aK[X]-moduleEf associated withf, and thatMf is a finitely generated torsion module over the PIDK[X]. In this chapter, we show how Theorems from Sections 24.6 and 24.7 yield important results about the structure of the linear mapf.

Recall that the annihilator of a subspaceV is an ideal (p) uniquely defined by a monic polynomialp called the minimal polynomial ofV .

Our first result is obtained by translating the primary decomposition theorem, Theorem 24.26. It is not too surprising that we obtain again Theorem 22.7!

Theorem 25.1. (Primary Decomposition Theorem) Letf :E→E be a linear map on the finite-dimensional vector spaceE over the fieldK. Write the minimal polynomialm off as

m =pr1 pr,1· · · k where thepi are distinct irreducible monic polynomials overK, and theri are positive integers. Let

Wi = Ker (pi(f)ri), i = 1, . . . , k. Then

(a) E =W1⊕ · · · ⊕Wk. (b) EachWi is invariant underf and the projection fromW ontoWi is given by a polynomial inf.

(c) The minimal polynomial of the restrictionf|Wi off toWi ispr .i

711 Next, we apply the Invariant Factors Decomposition Theorem, Theorem 24.38, toEf. This theorem says thatEf is isomorphic to a direct sum

Ef≈K[X]/(p_1)⊕ · · · ⊕K[X]/(pm)

ofm≤n cyclic modules, where thepj are uniquely determined monic polynomials of degree at least 1, such that

pm|pm−1| · · · |p_1.

Each cyclic moduleK[X]/(pi) is isomorphic to a cyclic subspace forf, sayVi, whose minimal polynomial ispi.

It is customary to renumber the polynomialspi as follows. Then polynomialsq_1, . . . , qn are defined by:

q

j

(

X

) =

1 if 1≤j≤n−m pn−j+1(X) ifn−m + 1≤j≤n.

Then, we see that q1|q2| · · · |qn, where the firstn−m polynomials are equal to 1, and we have the direct sum

E =E1⊕ · · · ⊕En,

where Ei is a cyclic subspace forf whose minimal polynomial isqi. In particular,Ei = (0) fori = 1, . . . , n−m. Theorem 24.38 also says that the minimal polynomial off isqn =p_1. We sum all this up in the following theorem.

Theorem 25.2. (Cyclic Decomposition Theorem, First Version) Let f :E→ E be an endomorphism on a K-vector space of dimension n. There exist n monic polynomials q_1, . . . , qn∈K[X] such that

q1|q2| · · · |qn, andE is the direct sum

E =E1⊕ · · · ⊕En

of cyclic subspaces Ei =Z(ui;f) forf, such that the minimal polynomial of the restriction off toEi isqi. The polynomialsqi satisfying the above conditions are unique, andqn is the minimal polynomial off.

In view of translation point (4) at the beginning of Section 24.5, we know that over the basis

( ui, f(ui), . . . , fni−1(ui)) of the cyclic subspaceEi =Z(ui;f), withni = deg(qi), the matrix of the restriction off to Ei is the companion matrix ofpi(X), of the form

ë0 0 0 · · ·0 −a0 ö

ì

÷

ì

ì

1

0

0

· · ·

0

−

a

1

÷

ì0 1 0 · · ·0 −a2 ÷

ì ... ...÷

...

.

.

÷ .

ì. ÷

ì ... 0 −ani−2ø

0 0 0 · · ·1 −ani−1

÷

í0 0 0

If we put all these bases together, we obtain a block matrix whose blocks are of the above form. Therefore, we proved the following result.

Theorem 25.3. (Rational Canonical Form, First Version) Letf :E→E be an endomorphism on aK-vector space of dimensionn. There existn monic polynomials q_1, . . . , qn∈K[X] such that

q1|q2| · · · |qn,

withq_1 =· · · =qn−m = 1, and a basis ofE such that the matrixX off is a block matrix of the formëAn−m+1 0 · · · 0 0 ö

ì 0 An−m+2 · · · 0 0÷

ì ÷

X

=

ì. . ... . . ÷

ì ÷,

ì 0 0 · · · An−1 0 ÷

í 0 0 · · ·0 A ø

n

where eachAi is the companion matrix ofqi. The polynomialsqi satisfying the above conditions are unique, andqn is the minimal polynomial off.

Definition 25.1. A matrixX as in Theorem 25.3 is called a matrix in rational form. The polynomialsq_1, . . . , qn arising in Theorems 25.2 and 25.3 are called the similarity invariants (or invariant factors) off.

Theorem 25.3 shows that every matrix is similar to a matrix in rational form. Such a matrix is unique.

By Proposition 24.20, two linear maps f andf are similar iff there is an isomorphism betweenEf andEf , and thus by the uniqueness part of Theorem 24.38, iff they have the same similarity invariantsq_1, . . . , qn.

Proposition 25.4. IfE andE are two finite-dimensional vector spaces and iff :E→E andf :E →E are two linear maps, thenf andf are similar iff they have the same similarity invariants.

The effect of extending the fied K to a fieldL is the object of the next proposition. Proposition 25.5. Letf :E→E be a linear map on aK-vector spaceE, and let (q_1, . . . , qn) be the similarity invariants off. IfL is a field extension ofK (which means thatK⊆L), and ifE(L) =L⊗KE is the vector space obtained by extending the scalars, andf(L) = 1L⊗f the linear map of E(L) induced byf, then the similarity invariants off(L) are (q_1, . . . , qn) viewed as polynomials inL[X].

Proof. We know thatEf is isomorphic to the direct sum

Ef≈K[X]/(q_1K[X])⊕ · · · ⊕K[X]/(qnK[X]),

so by tensoring withL[X] and using Propositions 24.12 and 23.7, we get

L[X]⊗K[X]Ef≈L[X]⊗K[X] (K[X]/(q_1K[X])⊕ · · · ⊕K[X]/(qnK[X]))

L[X]⊗K[X] (K[X]/(q1K[X]))⊕ · · · ⊕L[X]⊗K[X] (K[X]/(qnK[X]))≈ (K[X]/(q_1K[X]))⊗K[X]L[X]⊕ · · · ⊕ (K[X]/(qnK[X]))⊗K[X]L[X].≈

However, by Proposition 24.14, we have isomorphisms

(K[X]/(qiK[X]))⊗K[X]L[X]≈L[X]/(qiL[X]),

so we get L[X]⊗K[X]Ef≈L[X]/(q_1L[X])⊕ · · · ⊕L[X]/(qnL[X]).

Since Ef is aK[X]-module, theL[X] moduleL[X]⊗K[X]Ef is the module obtained from Ef by the ring extensionK[X]⊆L[X], and sincef is aK[X]-linear map ofEf, it becomes f(L[X]) onL[X]⊗K[X]Ef, which is the same asf(L) viewed as anL-linear map of the space E(L) =L⊗KE, soL[X]⊗K[X]Ef is actually isomorphic toE(L)f(L), and we have

E(L)f(L)≈L[X]/(q_1L[X])⊕ · · · ⊕L[X]/(qnL[X]),

which shows that (q_1, . . . , qn) are the similarity invariants off(L).

Proposition justifies the terminology “invariant” in similarity invariants. Indeed, under a field extensionK⊆L, the similarity invariants off(L) remain the same. This is not true of the elementary divisors, which depend on the field; indeed, an irreducible polynomial p∈K[X] may split overL[X]. Sinceqn is the minimal polynomial off, the above reasoning also shows that the minimal polynomial off(L) remains the same under a field extension.

Proposition 25.5 has the following corollary.

Proposition 25.6. LetK be a field and letL⊇K be a field extension ofK. For any two square matricesX andY overK , if there is an invertible matrixQ overL such that Y =QXQ−1, then there is an invertible matrixP overK such thatY =P XP−1. Recall from Theorem 24.21 that the sequence ofK[X]-linear maps

0 E[X] ψ E[X] σ Ef 0

is exact, and as a consequence, Ef is isomorphic to the quotient ofE[X] by Im(X1− f). Furthermore, becauseE is a vector space,E[X] is a free module with basis (1⊗u_1, . . . ,1⊗un), where (u_1, . . . , un) is a basis ofE. By Theorem 24.38, we have an isomorphism

Ef≈K[X]/(q_1K[X])⊕ · · · ⊕K[X]/(qnK[X]),

and by Proposition 24.39,E[X]/Im(X1−f) isomorphic to a direct sum

E[X]/Im(X1−f)≈K[X]/(p_1K[X])⊕ · · · ⊕K[X]/(pmK[X]),

where p_1, . . . , pm are the invariant factors of Im(X1−f) with respect toE[X]. SinceE[X]≈ E[X]/Im(X1−f), by the uniqueness part of Theorem 24.38 and because the polynomials are monic, we must havem =n andpi =qi, fori = 1, . . . , n. Therefore, we proved the following crucial fact:

Proposition 25.7. For any linear mapf :E→E over aK-vector spaceE of dimensionn, the similarity invariants off are equal to the invariant factors of Im(X1−f) with respect toE[X].

Proposition 25.7 is the key to computing the similarity invariants of a linear map. This can be done using a procedure to convertXI−U to its Smith normal form. Propositions 25.7 and 24.44 yield the following result.

Proposition 25.8. For any linear mapf :E→E over aK-vector spaceE of dimensionn, , for any matrixU representingf with respect if (q_1, . . . , qn) are the similarity invariants off to any basis, then fork = 1, . . . , n the product

dk(X) =q_1(X)· · ·qk(X)

is the gcd of thek×k-minors of the matrixXI−U.

Note that the matrixXI−U is nonother than the matrix that yields the characteristic polynomialχf(X) = det(XI−U) off.

Proposition 25.9. For any linear mapf :E→E over aK-vector spaceE of dimension n, if (q_1, . . . , qn) are the similarity invariants off, then the following properties hold: (1) Ifχf(X) is the characteristic polynomial off, then

χf(X) =q_1(X)· · ·qn(X). (2) The minimal polynomialm(X) = qn(X) off divides the characteristic polynomial χfX) off.

(3) The characteristic polynomialχfX) dividesm(X)n.

(4) E is cyclic forf iffm(X) =χ(X).

Proof. Property (1) follows from Proposition 25.8 fork =n. It also follows from Theorem 25.3 and the fact that for the companion matrix associated withqi, the characteristic polynomial of this matrix is alsoqi. Property (2) is obvious from (1). Since eachqi dividesqi+1, eachqi dividesqn, so their productχf(X) dividesm(X)n =qn(X)n. The last condition says thatq_1 =· · · =qn−1 = 1, which means thatEf has a single summand.

Observe that Proposition 25.9 yields another proof of the Cayley–Hamilton Theorem. It also implies that a linear map is nilpotent iff its characteristic polynomial isXn.

25.2 The Rational Canonical Form, Second Version

Let us now translate the Elementary Divisors Decomposition Theorem, Theorem 24.45, in terms ofEf. We obtain the following result.

Theorem 25.10. (Cyclic Decomposition Theorem, Second Version) Letf :E→E be an endomorphism on aK-vector space of dimensionn. Then,E is the direct sum of of cyclic subspacesEj =Z(uj;f) forf, such that the minimal polynomial ofEj is of the formpni,j,i for some irreducible monic polynomialsp_1, . . . , pt∈K[X] and some positive integersni,j, such that for eachi = 1, . . . , t, there is a sequence of integers

1≤ni,1, . . . , ni,1 < ni,2, . . . , ni,2 <· · ·< ni,si, . . . , ni,si,

mi,1 mi,2 mi,si

withsi≥ 1, and whereni,j occursmi,j≥ 1 times, forj = 1, . . . , si. Furthermore, the monic polynomials pi and the integersr, t, ni,j, si,mi,j are uniquely determined.

Note that there areµ = mi,j cyclic subspacesZ(uj;f). Using bases for the cyclic subspacesZ(uj;f) as in Theorem 25.3, we get the following theorem.

Theorem 25.11. (Rational Canonical Form, Second Version) Letf :E→E be an endomorphism on aK-vector space of dimensionn. There existt distinct irreducible monic polynomialsp_1, . . . , pt∈K[X] and some positive integersni,j, such that for eachi = 1, . . . , t, there is a sequence of integers

1≤ni,1, . . . , ni,1 < ni,2, . . . , ni,2 <· · ·< ni,si, . . . , ni,si,

mi,1 mi,2 mi,si

25.3. THE JORDAN FORM REVISITED 717

withsi≥ 1, and whereni,j occursmi,j≥ 1 times, forj = 1, . . . , si, and there is a basis ofE such that the matrixX off is a block matrix of the form

ëA_1 0 · · · 0 0ö

ì 0 A2 · · · 0 0÷

ì ÷

X

=

ì. . ... . . ÷

ì ÷,

ì 0 0 · · · Aµ−1 0÷

í ø

0 0 · · · 0 Aµ

where eachAj is the companion matrix of somepni,j, andµ = mi,j. The monic polynoi

mialsp_1, . . . , pt and the integersr, t, ni,j, si,mi,j are uniquely determined

The polynomialspni,j are called the elementary divisors off (andX). These polynomialsi

are factors of the minimal polynomial.

As we pointed earlier, unlike the similarity invariants, the elementary divisors may change when we pass to a field extension.

We will now consider the special case where all the irreducible polynomials pi are of the formX−λi; that is, when are the eigenvalues off belong toK. In this case, we find again the Jordan form.

25.3 The Jordan Form Revisited

In this section, we assume that all the roots of the minimal polynomial of f belong toK. This will be the case ifK is algebraically closed. The irreducible polynomialspi of Theorem 25.10 are the polynomialsX−λi, for the distinct eigenvaluesλi off. Then, each cyclic subspaceZ(uj;f) has a minimal polynomial of the form (X−λ)m, for some eigenvalueλ of f and somem≥ 1. It turns out that by choosing a suitable basis for the cyclic subspace Z(uj;f), the matrix of the restriction off toZ(uj;f) is a Jordan block.

Proposition 25.12. LetE be a finite-dimensionalK-vector space and letf :E→E be a linear map. IfE is a cyclicK[X]-module and if (X−λ)n is the minimal polynomial off, then there is a basis ofE of the form

((fn−1(u),(f−λid)n−2(u), . . . ,(f−λid)(u), u),−λid)

for someu∈E. With respect to this basis, the matrix off is the Jordan block

ëλ 1 0 · · ·0ö

ì0 λ 1 · · · 0÷

÷

J ì

í0 0 0

ì ... ... .÷.n(λ) =ì. .÷

ì ... ÷

1 ø

0 0 0 · · · λ

Proof. SinceE is a cyclicK[X]-module, there is someu∈E so thatE is generated by u, f(u), f^2(u), . . ., which means that every vector inE is of the formp(f)(u), for some polynomial,p(X). We claim thatu, f(u), . . . , fn−2(u), fn−1(u) generateE, which implies that the dimension ofE is at mostn.

This is because ifp(X) is any polynomial of degree at leastn, then we can dividep(X) by (X−λ)n, obtaining

p = (X−λ)nq +r,

where 0≤ deg(r)< n, and as (X−λ)n annihilatesE, we get

p(f)(u) =r(f)(u),

which means that every vector of the formp(f)(u) withp(X) of degree≥n is actually a linear combination ofu, f(u), . . . , fn−2(u), fn−1(u).

We claim that the vectors

u,(f−λid)(u), . . . ,(f−λid)n−2(u)(f−λid)n−1(u)

are linearly independent. Indeed, if we had a nontrivial linear combination a_0(f−λid)n−1(u) +a_1(f−λid)n−2(u) +· · · +an−2(f−λid)(u) +an−1u = 0, then the polynomial

a_0(X−λ)n−1 +a_1(X−λ)n−2 +· · · +an−2(X−λ) +an−1

of degree at most n− 1 would annihilateE, contradicting the fact that (X−λ)n is the minimal polynomial off (and thus, of smallest degree). Consequently, as the dimension of E is at mostn,

((f−λid)n−1(u),(f−λid)n−2(u), . . . ,(f−λid)(u), u), is a basis ofE and sinceu, f(u), . . . , fn−2(u), fn−1(u) spanE,

(u, f(u), . . . , fn−2(u), fn−1(u))

is also a basis ofE.

Let us see howf acts on the basis

((f−λid)n−1(u),(f−λid)n−2(u), . . . ,(f−λid)(u), u).

If we writef =fn annihilatesE, we get−λid +λid, as (f−λid)

f((f−λid)n−1(u)) = (f−λid)n(u) +λ(f−λid)n−1(u) =λ(f−λid)n−1(u) and

f((f−λid)k(u)) = (f−λid)k+1(u) +λ(f−λid)k(u), 0≤k≤n− 2. But this means precisely that the matrix off in this basis is the Jordan blockJn(λ). Combining Theorem 25.11 and Proposition 25.12, we obtain a strong version of the Jordan form.

Theorem 25.13. (Jordan Canonical Form) LetE be finite-dimensionalK-vector space. The following properties are equivalent:

(1) The eigenvalues of f all belong toK.

(2) There is a basis ofE in which the matrix off is upper (or lower) triangular.

(3) There exist a basis of E in which the matrixA off is Jordan matrix. Furthermore, the number of Jordan blocksJr(λ) appearing inA, for fixedr andλ, is uniquely determined byf.

Proof. The implication (1) =⇒ (3) follows from Theorem 25.11 and Proposition 25.12. The implications (3) =⇒ (2) and (2) =⇒ (1) are trivial.

Compared to Theorem 22.14, the new ingredient is the uniqueness assertion in (3), which is not so easy to prove.

Observe that the minimal polynomial of f is the least common multiple of the polynomials (X−λ)r associated with the Jordan blocksJr(λ) appearing inA, and the characteristic polynomial ofA is the product of these polynomials.

We now return to the problem of computing effectively the similarity invariants of a matrixA. By Proposition 25.7, this is equivalent to computing the invariant factors of XI−A. In principle, this can be done using Proposition 24.42. A procedure to do this effectively for the ringA =K[X] is to convertXI−A to its Smith normal form. This will also yield the rational canonical form forA.

25.4 The Smith Normal Form

The Smith normal form is the special case of Proposition 24.42 applied to the PID K[X] whereK is a field, but it also says that the matricesP andQ are products of elementary matrices. It turns out that such a result holds for any Euclidean ring, and the proof is basically the same.

Recall from Definition 20.9 that a Euclidean ring is an integral domainA such that there exists a functionσ:A→ N with the following property: For alla, b∈A withb = 0, there are someq, r∈A such that

a =bq +r and σ(r)< σ(b). Note that the pair (q, r) is not necessarily unique.

We make use of the elementary row and column operations P(i, k),Ei,j;β, andEi,λ described in Chapter 6, where we require the scalarλ used inEi,λ to be a unit. Theorem 25.14. IfM is anm×n matrix over a Euclidean ringA, then there exist some invertiblen×n matrixP and some invertiblem×m matrixQ, whereP andQ are products of elementary matrices, and am×n matrixD of the form

ëα1 0 · · ·0 0 · · ·0ö

ì 0 α2 · · ·0 0 · · ·0÷

ì ÷

ì . . ... . . · · ·.÷

ì ÷

D

=

ì 0 0 · · · αr 0 · · ·0÷

ì ÷

ì 0 0 · · ·0 0 · · ·0÷

ì ÷

ì ÷

í . . · · · . . ... .ø

0 0 · · ·0 0 · · ·0

for some nonzeroαi∈A, such that

(1) α1|α2| · · · |αr, and

1.−

(2) M =QDP

Proof. We follow Jacobson’s proof [57] (Chapter 3, Theorem 3.8). We proceed by induction onm +n.

Ifm =n = 1, letP = (1) andQ = (1).

For the induction step, ifM = 0, letP =In andQ =Im. IfM = 0, the stategy is to apply a sequence of elementary transformations that convertsM to a matrix of the form

ëα1 0 · · ·0ö

ì 0 ÷

M

=

ì ÷

ì. Y ÷

í_0 ø

whereY is a (m− 1)× (n− 1)-matrix such thatα_1 divides every entry inY . Then, we proceed by induction onY . To findM , we perform the following steps.

Step 1 . Pick some nonzero entryaij inM such thatσ(aij) is minimal. Then permute columnj and column 1, and permute rowi and row 1, to bring this entry in position (1,1). We denote this new matrix again byM.

Step 2a .

Ifm = 1 go to Step 2b.

Ifm > 1, then there are two possibilities:

(i)M is of the formëa11 a12 · · · a1n ö

ì 0 a22 · · · a2n ÷

ì ÷.ì . . ... . ÷

í 0 am2 · · · aø

mn

If n = 1, stop; else go to Step 2b.

(ii) There is some nonzero entryai1 (i > 1) belowa_11 in the first column.

(a) If there is some entry ak1 in the first column such thata_11 does not divideak1, then pick such an entry (say, with the smallest indexi such thatσ(ai1) is minimal), and divide ak1 bya_11; that is, findbk andbk1 such that

ak1 =a_11bk +bk1, with σ(bk1)< σ(a_11).

Subtractbk times row 1 from rowk and permute rowk and row 1, to obtain a matrix of the

form ë bk1 bk2 · · · bknö

ìa_21 a22 · · · a2n÷

M

=

ì ÷

ì .

í

.

.

...

.

÷

ø

am1 am2 · · · amn

Go back to Step 2a.

(b) Ifa_11 divides every (nonzero) entryai1 fori≥ 2, sayai1 =a_11bi, then subtractbi times row 1 from rowi fori = 2, . . . , m; go to Step 2b.

Observe that whenever we return to the beginning of Step 2a, we have σ(bk1)< σ(a_11). Therefore, after a finite number of steps, we must exit Step 2a with a matrix in which all entries in column 1 but the first are zero and go to Step 2b.

Step 2b .

This step is reached only ifn > 1 and if the only nonzero entry in the first column isa_11. (a) IfM is of the formëa11 0 · · · 0 ö

ì 0 a22 · · · a2n÷

ì ÷

ì. . ... . ÷

í ø

0 am2 · · · amn

andm = 1 stop; else go to Step 3.

(b) If there is some entry a1k in the first row such thata_11 does not dividea1k, then pick such an entry (say, with the smallest indexj such thatσ(a1j) is minimal), and dividea1k by a_11; that is, findbk andb1k such that

a1k =a_11bk +b1k, with σ(b1k)< σ(a_11).

Subtractbk times column 1 from columnk and permute columnk and column 1, to obtain a matrix of the formëb1k ak2 · · · aknö

ìb2k a22 · · · a2n÷

M =ì ÷ .ì . . ... . ÷

íbmk am2 · · · aø

mn

Go back to Step 2b.

(c) If a_11 divides every (nonzero) entrya1j forj≥ 2, saya1j =a_11bj, then subtractbj times column 1 from columnj forj = 2, . . . , n; go to Step 3.

As in Step 2a, whenever we return to the beginning of Step 2b, we haveσ(b1k)< σ(a_11). Therefore, after a finite number of steps, we must exit Step 2b with a matrix in which all entries in row 1 but the first are zero.

Step 3. This step is reached only if the only nonzero entry in the first row isa_11. (i) Ifëa11 0 · · · 0ö

ì 0 ÷

M

=

ì ÷

ì. Y ÷

í_0 ø

go to Step 4.

(ii) If Step 2b ruined column 1 which now contains some nonzero entry belowa_11, go back to Step 2a.

We perform a sequence of alternating steps between Step 2a and Step 2b. Because the σ-value of the (1,1)-entry strictly decreases whenever we reenter Step 2a and Step 2b, such a sequence must terminate with a matrix of the form

ëa11 0 · · ·0ö

ì 0 ÷

M

=

ì ÷

ì. Y ÷

í ø

0

Step 4. Ifa_11 divides all entries inY , stop.

Otherwise, there is some column, sayj, such thata_11 does not divide some entryaij, so add thejth column to the first column. This yields a matrix of the form

ëa11 0 · · ·0ö

ìb2j ÷

M

=

ì ÷

ì. Y ÷

í ø

bmj

where theith entry in column 1 is nonzero, so go back to Step 2a,

Again, since theσ-value of the (1,1)-entry strictly decreases whenever we reenter Step 2a and Step 2b, such a sequence must terminate with a matrix of the form

ëα1 0 · · ·0ö

ì 0 ÷

M

=

ì ÷

ì. Y ÷

í_0 ø

whereα_1 divides every entry inY . Then, we apply the induction hypothesis toY .

If the PID A is the polynomial ringK[X] whereK is a field, theαi are nonzero polynomials, so we can apply row operations to normalize their leading coefficients to be 1. We obtain the following theorem.

Theorem 25.15. (Smith Normal Form) IfM is anm× n matrix over the polynomial ring K[X], whereK is a field, then there exist some invertible n×n matrixP and some invertible m×m matrixQ, whereP andQ are products of elementary matrices with entries inK[X], and am×n matrixD of the form

ëq1 0 · · ·0 0 · · ·0ö

ì0 q2 · · ·0 0 · · ·0÷

ì ÷

ì . . ... . . · · ·.÷

ì ÷

D

=

ì0 0 · · · qr 0 · · ·0÷

ì ÷

ì0 0 · · ·0 0 · · ·0÷

ì ÷

ì . . · · ·. . ... .÷

í ø

0 0 · · ·0 0 · · ·0

for some nonzero monic polynomialsqi∈k[X], such that (1) q1|q2| · · · |qr, and (2) M =QDP−1.

In particular, if we apply Theorem 25.15 to a matrix M of the formM =XI−A, where A is a square matrix, then det(XI−A) =χA(X) is never zero, and sinceXI−A =QDP−1 withP, Q invertible, all the entries inD must be nonzero and we obtain the following result showing that the similarity invariants ofA can be computed using elementary operations.

Theorem 25.16. IfA is ann× n matrix over the fieldK, then there exist some invertible n×n matricesP andQ, where P andQ are products of elementary matrices with entries inK[X], and an×n matrixD of the form

ë1 · · ·0 0 0 · · ·0 ö

ì. ... . . . . .÷

ì ÷

ì0 · · ·1 0 0 · · ·0 ÷

ì ÷

D

=

ì0 · · ·0 q1 0 · · ·0 ÷

ì ÷

ì0 · · ·0 0 q2 · · ·0 ÷

ì ÷

ì. . . . . ... .÷

í ø

0 · · ·0 0 0 · · · qm

for some nonzero monic polynomialsqi∈k[X] of degree≥ 1, such that (1) q1|q2| · · · |qm, (2) q_1, . . . qm are the similarity invariants ofA, and

(3) XI−A =QDP−1.

The matrixD in Theorem 25.16 is often called Smith normal form ofA, even though this is confusing terminology sinceD is really the Smith normal form ofXI−A.

Of course, we know from previous work that in Theorem 25.15, the α_1, . . . , αr are unique, and that in Theorem 25.16, theq_1, . . . , qm are unique. This can also be proved using some simple properties of minors, but we leave it as an exercise (for help, look at Jacobson [57], Chapter 3, Theorem 3.9).

The rational canonical form of A can also be obtained fromQ−1 andD, but first, let us consider the generalization of Theorem 25.15 to PID’s that are not necessarily Euclidean rings.

We need to find a “norm” that assigns a natural number σ(a) to any nonzero element of a PIDA, in such a way thatσ(a) decreases whenever we return to Step 2a and Step 2b. Since a PID is a UFD, we use the number

σ(a) =k_1 +· · · +kr

of prime factors in the factorization of a nonunit element

a =upk1 pk,1· · · r

and we set σ(u) = 0

ifu is a unit.

We can’t divide anymore, but we can find gcd’s and use Bezout to mimic division. The key ingredient is this: for any two nonzero elementsa, b∈A, ifa does not divideb then let d = 0 be a gcd ofa andb. By Bezout, there existx, y∈A such that

ax +by =d.

We can also writea =td andb =−sd, for somes, t∈A, so thattdx−sdy =d, which implies that

tx−sy = 1, sinceA is an integral domain. Observe that

t −s x s= 1 0 , −y x y t 0 1 which shows that both matrices on the left of the equation are invertible, and so is the transpose of the second one,

x y s t

(they all have determinant 1). We also have

as +bt =tds−sdt = 0,

so x y a= d s t b 0

and a b x s = d 0 .y t

Becausea does not divideb, their gcdd has strictly fewer prime factors thana, so

σ(d)< σ(a).

Using matrices of the form

ë x y 0 0 · · · 0ö ìs t 0 0 · · · 0÷ ì0 0 1 0 · · · 0÷

ì ÷ ì ÷

ì0 0 0 1 · · · 0÷

ì ÷ ì. . . . ... .÷

í ø 0 0 · · ·0 · · ·1 withxt−ys = 1, we can modify Steps 2a and Step 2b to obtain the following theorem.

Theorem 25.17. IfM is anm×n matrix over a PIDA, then there exist some invertible n×n matrix P and some invertible m×m matrix Q, where P and Q are products of elementary matrices and matrices of the form

ëx y 0 0 · · · 0ö ìs t 0 0 · · · 0÷

ì ÷

ì0 0 1 0 · · · 0÷

ì

ì0 0 0 1÷

ì ÷ ... .÷

withxt−ys = 1, and am×n matrixD of the form

ëα1 0 · · ·0 0 · · ·0ö

ì 0 α2 · · ·0 0 · · ·0÷

ì ÷

ì . . ... . . · · ·.÷

ì ÷

D

=

ì 0 0 · · · αr 0 · · ·0÷

ì ÷

ì 0 0 · · ·0 0 · · ·0÷

ì ÷

ì . . · · ·. . ... .÷

í ø

0 0 · · ·0 0 · · ·0

for some nonzeroαi∈A, such that

(1) α1|α2| · · · |αr, and

(2) M =QDP−1.

Proof sketch. In Step 2a, ifa_11 does not divideak1, then first permute row 2 and rowk (if k = 2). Then, if we writea =a_11 andb =ak1, ifd is a gcd ofa andb and ifx, y, s, t are determined as explained above, multiply on the left by the matrix

ëx y 0 0 · · · 0ö

ìs t 0 0 · · · 0÷

ì ÷

ì0 0 1 0 · · · 0÷

ì0 0 0 1 · · · 0÷

ì ÷

ì ÷

ì. . . . ... .÷

í0 0 · · ·0 · · ·1ø

to obtain a matrix of the form ë d a12 · · · a1nö ì 0 a22 · · · a2n÷

ì ÷

ìa_31 a32 · · · a3n÷

ì ÷ ì. . ... . ÷

í ø am1 am2 . . . amn withσ(d)< σ(a_11). Then, go back to Step 2a.

In Step 2b, if a_11 does not dividea1k, then first permute column 2 and columnk (if k = 2). Then, if we writea =a_11 andb =a1k, ifd is a gcd ofa andb and ifx, y, s, t are determined as explained above, multiply on the right by the matrix

ëx s 0 0 · · · 0ö

ìy t 0 0 · · · 0÷

ì ÷

ì0 0 1 0 · · · 0÷

ì

ì0 0 0 1÷

ì ÷

... .÷

to obtain a matrix of the form ë d 0 a13 · · · a1nö ì a_21 a_22 a23 · · · a2n÷

ì ÷ ì. . . ... . ÷

í ø am1 am2 am3 . . . amn

with σ(d)< σ(a_11). Then, go back to Step 2b. The other steps remain the same. Whenever we return to Step 2a or Step 2b, theσ-value of the (1,1)-entry strictly decreases, so the whole procedure terminates.

We conclude this section by explaining how the rational canonical form of a matrixA can be obtained from the canonical formQDP−1 ofXI−A.

Let f :E→ E be a linear map over aK-vector space of dimensionn. Recall from Theorem 24.21 (see Section 24.5) that as aK[X]-module,Ef is the image of the free module E[X] by the mapσ:E[X]→Ef, whereE[X] consists of all linear combinations of the form

p_1e_1 +· · · +pnen, where (e_1, . . . , en) is a basis ofE andp_1, . . . , pn∈K[X] are polynomials, andσ is given by σ(p_1e_1 +· · · +pnen) =p_1(f)(e_1) +· · · +pn(f)(en).

Furthermore, the kernel ofσ is equal to the image of the mapψ:E[X]→E[X], where ψ(p_1e_1 +· · · +pnen) =Xp_1e_1 +· · · +Xpnen− (p_1f(e_1) +· · · +pn(en)).

The matrix A is the representation of a linear mapf over the canonical basis (e_1, . . . , en) ofE = Kn, and andXI−A is the matrix ofψ with respect to the basis (e_1, . . . , en) (overK[X]). What Theorem 25.16 tells us is that there areK[X]-bases (u_1, . . . , un) and (v_1, . . . , vn) ofEf with respect to which the matrix ofψ isD. Then

ψ(ui) =vi, i = 1, . . . , n−m, ψ(un−m+i) =qivn−m+i, i = 1, . . . , m,

and because Im( ψ) = Ker (σ), this implies that σ(vi) = 0, i = 1, . . . , n−m.

Consequently,w_1 =σ(vn−m+1), . . . , wm =σ(vn) spanEf as aK[X]-module, withwi∈E, and we have

Mf =K[X]w1⊕ · · · ⊕K[X]wm,

whereK[X]wi≈K[X]/(qi) as a cyclicK[X]-module. Since Im(ψ) = Ker (σ), we have

0 = σ(ψ(un−m+i)) =σ(qivn−m+i) =qiσ(vn−m+i) =qiwi, so as aK-vector space, the cyclic subspaceZ(wi;f) =K[X]wi hasqi as annihilator, and by a remark from Section 24.5, it has the basis (overK)

(wi, f(wi), . . . , fni−1(wi)), ni = deg(qi).

Furthermore, over this basis, the restriction of f toZ(wi;f) is represented by the companion matrix ofqi. By putting all these bases together, we obtain a block matrix which is the canonical rational form off (andA).

Now, XI−A =QDP−1 is the matrix ofψ with respect to the canonical basis (e_1, . . . , en) (overK[X]), andD is the matrix ofψ with respect to the bases (u_1, . . . , un) and (v_1, . . . , vn) (overK[X]), which tells us that the columns ofQ consist of the coordinates (inK[X]) of the basis vectors (v_1, . . . , vn) with respect to the basis (e_1, . . . , en). Therefore, the coordinates (in K) of the vectors (w_1, . . . , wm) spanningEf overK[X], wherewi =σ(vn−m+i), are obtained by substituting the matrixA forX in the coordinates of the columns vectors ofQ, and evaluating the resulting expressions.

Since

D =Q−1(XI−A)P,

the matrix D is obtained fromA by a sequence of elementary row operations whose product isQ−1 and a sequence of elementary column operations whose product isP. Therefore, to compute the vectorsw_1, . . . , wm fromA, we simply have to figure out how to constructQ from the sequence of elementary row operations that yieldQ−1. The trick is to use column operations to gather a product of row operations in reverse order.

Indeed, ifQ−1 is the product of elementary row operations

Q−1 =Ek· · ·E_2E_1,

then Q

=

E

−

1 E2−1 E^1

k

.

1 · · ·

Now, row operations operate on the left and column operations operate on the right, so the productE− 1 E−1 Ek−1 can be computed from left to right as a sequence of column1 2 · · ·operations.

Let us review the meaning of the elementary row and column operationsP(i, k),Ei,j;β, andEi,λ.

1. As a row operation, P(i, k) permutes rowi and rowk.

2. As a column operation,P(i, k) permutes columni and columnk.

3. The inverse ofP(i, k) isP(i, k) itself.

4. As a row operation,Ei,j;β addsβ times rowj to rowi.

5. As a column operation,Ei,j;β addsβ times columni to columnj (note the switch in the indices).

6. The inverse ofEi,j;β isEi,j;−β.

7. As a row operation,Ei,λ multiplies rowi byλ.

8. As a column operation,Ei,λ multiplies columni byλ.

9. The inverse ofEi,λ isEi,λ−1.

Given a square matrix A (overK), the row and column operations applied toXI−A in converting it to its Smith normal form may involve coefficients that are polynomials and it is necessary to explain what is the action of an operationEi,j;β in this case. If the coefficient β inEi,j;β is a polynomial overK, as a row operation, the action ofEi,j;β on a matrixX is to multiply thejth row ofM by the matrixβ(A) obtained by substituting the matrixA for X and then to add the resulting vector to rowi. Similarly, as a column operation, the action ofEi,j;β on a matrixX is to multiply theith column ofM by the matrixβ(A) obtained by substituting the matrixA forX and then to add the resulting vector to columnj. An algorithm to compute the rational canonical form of a matrix can now be given. We apply the elementary column operationsEi−1 fori = 1, . . . k, starting with the identity matrix.

Algorithm for Converting ann×n matrix to Rational Canonical Form While applying elementary row and column operations to compute the Smith normal formD ofXI−A, keep track of the row operations and perform the following steps:

1. LetP =In, and for every elementary row operationE do the following:

(a) IfE =P(i, k), permute columni and columnk ofP .

(b) If E =Ei,j;β, multiply theith column ofP by the matrixβ(A) obtained by substituting the matrixA forX, and then subtract the resulting vector from columnj.

(c) IfE =Ei,λ whereλ∈K, then multiply theith column ofP byλ−1.

2. When step (1) terminates, the first n−m columns ofP are zero and the lastm are linearly independent. Fori = 1, . . . , m, multiply the (n−m +i)th columnwi ofP successively byI, A^1, A^2, Ani−1, whereni is the degree of the polynomialqi (appearing inD), and form then×n matrixP consisting of the vectors

w_1, Aw_1, . . . , An1−1w_1, w_2, Aw_2, . . . , An2−1w_2, . . . , wm, Awm, . . . , Anm−1wm.

Then,P−1AP is the canonical rational form ofA. Here is an example taken from Dummit and Foote [30] (Chapter 12, Section 12.2). Let A be the matrixë1 2 −4 4 ö

ì2 −1 4 −8÷.í1 0 1 −2÷A =ì

ø

0 1 −2 3

One should check that the following sequence of row and column operations produces the Smith normal formD ofXI−A:

rowP(1,3) rowE1,−1 rowE2,1;2 rowE3,1;−(X−1) columnE1,3;X−1 columnE1,4;2 rowP(2,4) rowE2,−1 rowE3,2;2 rowE4,2;−(X+1) columnE2,3;2 columnE2,4;X−3, withë1 0 0 ö

ì0 1 0 0 ÷.í0 0 (X− 1)2 0÷D = ì

0 0_0 (X− 1)2ø Then, applying Step 1 of the above algorithm, we get the sequence of column operations:

ë 1 0 0 0ö ë0 0 1 0ö ë 0 0 1 0ö ì0 1 0 0÷ P (1,3) ì0 1 0 0÷ E1,−1 ì 0 1 0 0÷ E2,1,−2ì0 0 1 0÷ ì1 0 0 0÷ ì 1 0 0 0÷ í ø −→

0 0 0 1 0 0 0 1_0

ø −→ í ø −→ í− 0 0 1

ë 0 0 1 0ö ë0 0 1 0ö ë0 0 1 0ö ì 2 1 0 0÷ E3,1,A−I ì0 1 0 0÷ P (2,4) ì0 0 0 1÷ E2,−1ì−1 0 0 0÷ ì0 0 0 0÷ ì0 0 0 0÷ í− ø −→ í ø −→ í ø −→

0 0 0 1 0 0 0 1 0 1 0 0 ë0 0 1 0ö ë0 −2 1 0ö ë0 0 1 0ö

ì0 0 0 1÷ E3,2,−2 ì0_0 0 1÷ E4,2;A+I ì0 0 0 1÷ ì =P .í0 0 0 0÷ ì0 0 0 0÷ ì0 0 0 0÷ ø −→ í ø −→ í ø 0 −1 0 0 0 −1 0 0 0 0 0 0 Step 2 of the algorithm yields the vectors

ë_1ö ë_1ö ë_1ö ë_0ö ë_0ö ë_2 ö

ì0÷, Aì0÷ = ì2÷, ì1÷, Aì1÷ = ì 1÷ ì ,í0÷ ì_0÷ ì_1÷ ì_0÷ ì_0÷ ì− ÷

ø í ø í ø í ø í ø í_0 ø 0 0 0 0 0 1

so we getë1 1 0 2 ö

P

=

ì ÷

ì0 2 1 −1÷.í0 1 0 0 ø

0 0 0 1

We find thatë1 0 −1 −2ö

ì0 0 1 0 ÷,í0 1 −2 1 ÷P−1 = ì

ø 0 0_0 1 and thus, the rational canonical form ofA is

ë0 −1 0 0ö ì1 2 0 0 ÷.í0 0 0 −1÷P−1AP = ì

ø 0 0 1_2

Chapter 26 Topology

26.1 Metric Spaces and Normed Vector Spaces

This chapter contains a review of basic topological concepts. First, metric spaces are defined. Next, normed vector spaces are defined. Closed and open sets are defined, and their basic properties are stated. The general concept of a topological space is defined. The closure and the interior of a subset are defined. The subspace topology and the product topology are defined. Continuous maps and homeomorphisms are defined. Limits of seqences are defined. Continuous linear maps and multilinear maps are defined and studied briefly. The chapter ends with the definition of a normed affine space.

Most spaces considered in this book have a topological structure given by a metric or a norm, and we first review these notions. We begin with metric spaces. Recall that R+ =

{x∈ R|x≥ 0}.

Definition 26.1. A metric space is a setE together with a functiond:E×E→ R+, called a metric, or distance, assigning a nonnegative real numberd(x, y) to any two points x, y∈E, and satisfying the following conditions for allx, y, z∈E:

(D1) d(x, y) =d(y, x). (symmetry) (D2) d(x, y)≥ 0, andd(x, y) = 0 iffx =y. (positivity) (D3) d(x, z)≤d(x, y) +d(y, z). (triangle inequality)

Geometrically, condition (D3) expresses the fact that in a triangle with vertices x, y, z, the length of any side is bounded by the sum of the lengths of the other two sides. From (D3), we immediately get

|d(x, y)−d(y, z)| ≤d(x, z).

Let us give some examples of metric spaces. Recall that the absolute value|x| of a real numberx∈ R is defined such that|x| =x ifx≥ 0,|x| =−x ifx < 0, and for a complex numberx =a +ib, by|x| =√a^2 +b^2.

733 Example 26.1. 1. LetE = R, andd(x, y) =|x−y|, the absolute value ofx−y. This is the so-called natural metric on R.

2. LetE =n (orE = Cn). We have the Euclidean metric R

d

2

(

x, y

) =

|x1−y1 2 2 1

2

| +· · · +|xn−yn| ,

the distance between the points (x_1, . . . , xn) and (y_1, . . . , yn). 3. For every setE, we can define the discrete metric, defined such thatd(x, y) = 1 iff x =y, andd(x, x) = 0.

4. For anya, b∈ R such thata < b, we define the following sets:

[a, b] ={x∈ R|a≤x≤b}, (closed interval)

]a, b[ ={x∈ R|a < x < b}, (open interval) [a, b[ ={x∈ R|a≤x < b}, (interval closed on the left, open on the right)

]a, b] ={x∈ R|a < x≤b}, (interval open on the left, closed on the right)

LetE = [a, b], andd(x, y) =|x−y|. Then, ([a, b], d) is a metric space.

We will need to define the notion of proximity in order to define convergence of limits and continuity of functions. For this, we introduce some standard “small neighborhoods.” Definition 26.2. Given a metric spaceE with metricd, for everya∈E, for everyρ∈ R, withρ > 0, the set

B(a, ρ) ={x∈E|d(a, x)≤ρ} is called the closed ball of centera and radiusρ, the set

B_0(a, ρ) ={x∈E|d(a, x)< ρ}

is called the open ball of centera and radiusρ, and the set

S(a, ρ) ={x∈E|d(a, x) =ρ}

is called the sphere of centera and radiusρ. It should be noted thatρ is finite (i.e., not +∞). A subsetX of a metric spaceE is bounded if there is a closed ballB(a, ρ) such that X⊆B(a, ρ).

Clearly,B(a, ρ) =B_0(a, ρ)∪S(a, ρ). 26.1. METRIC SPACES AND NORMED VECTOR SPACES 735

Example 26.2.

1. InE = R with the distance|x−y|, an open ball of centera and radiusρ is the open interval ]a−ρ, a +ρ[.

2. In E = R^2 with the Euclidean metric, an open ball of centera and radiusρ is the set of points inside the disk of centera and radiusρ, excluding the boundary points on the circle.

3. In E = R^3 with the Euclidean metric, an open ball of centera and radiusρ is the set of points inside the sphere of centera and radiusρ, excluding the boundary points on the sphere.

One should be aware that intuition can be misleading in forming a geometric image of a closed (or open) ball. For example, ifd is the discrete metric, a closed ball of centera and radiusρ < 1 consists only of its centera, and a closed ball of centera and radiusρ≥ 1 consists of the entire space!

IfE = [a, b], andd(x, y) =|x−y|, as in Example 26.1, an open ballB_0(a, ρ), with ρ < b−a, is in fact the interval [a, a +ρ[, which is closed on the left.

We now consider a very important special case of metric spaces, normed vector spaces. Normed vector spaces have already been defined in Chapter 7 (Definition 7.1) but for the reader’s convenience we repeat the definition.

Definition 26.3. LetE be a vector space over a fieldK, whereK is either the field R of reals, or the field C of complex numbers. A norm onE is a function :E→ R+, assigning a nonnegative real number u to any vectoru∈E, and satisfying the following conditions for allx, y, z∈E:

(N1) x ≥ 0, and x = 0 iffx = 0. (positivity)

(N2) λx =|λ| x . (scaling)

(N3) x +y ≤ x + y . (triangle inequality)

A vector spaceE together with a norm is called a normed vector space.

From (N3), we easily get |x− y| ≤ x−y .

Given a normed vector spaceE, if we defined such that

d (x, y) = x−y , it is easily seen thatd is a metric. Thus, every normed vector space is immediately a metric space. Note that the metric associated with a norm is invariant under translation, that is,

d(x +u, y +u) =d(x, y).

For this reason, we can restrict ourselves to open or closed balls of center 0. Examples of normed vector spaces were given in Example 7.1. We repeat the most important examples.

Example 26.3. LetE = Rn (orE = Cn). There are three standard norms. For every (x_1, . . . , xn)∈E, we have the norm x_1, defined such that,

x_1 =|x1| +· · · +|xn|,

we have the Euclidean norm x_2, defined such that,

x

2

=

|x1 2 2 1

2

| +· · · +|xn| ,

and the sup-norm x∞, defined such that,

x∞ = max{|xi| | 1≤i≤n}.

More generally, we define the p-norm (forp≥ 1) by

x

p

= (

|

x

1

p p

| +· · · +|xn| )1/p.

We proved in Proposition 7.1 that the p-norms are indeed norms. One should work out what are the open balls in R^2 for_1 and ∞.

In a normed vector space, we define a closed ball or an open ball of radiusρ as a closed ball or an open ball of center 0. We may use the notationB(ρ) andB_0(ρ). We will now define the crucial notions of open sets and closed sets, and of a topological space.

Definition 26.4. LetE be a metric space with metricd. A subsetU⊆ E is an open set inE if eitherU =∅, or for everya∈U, there is some open ballB0(a, ρ) such that, B_0(a, ρ)⊆U.1 A subsetF⊆E is a closed set inE if its complementE−F is open inE.

The set E itself is open, since for everya∈E, every open ball of centera is contained in E. InE = Rn, givenn intervals [ai, bi], withai< bi, it is easy to show that the openn-cube is an open set. In fact, it is possible to find a metric for which such openn-cubes are open balls! Similarly, we can define the closedn-cube

{(x_1, . . . , xn)∈E|ai< xi< bi, 1≤i≤n} 1Recall that ρ > 0.

(x_1, . . . , xn)∈E|ai≤xi≤bi, 1≤i≤n},{

which is a closed set.

The open sets satisfy some important properties that lead to the definition of a topological space.

Proposition 26.1. Given a metric spaceE with metricd, the familyO of all open sets defined in Definition 26.4 satisfies the following properties:

(O1) For every finite family (Ui)1≤i≤n of setsUi∈ O, we haveU1∩ · · · ∩Un∈ O, i.e.,O is closed under finite intersections.

(O2) For every arbitrary family (Ui)i∈I of setsUi∈ O, we havei∈IUi∈ O, i.e.,O is closed under arbitrary unions.

(O3)∅ ∈ O, andE∈ O, i.e.,∅ andE belong toO.

Furthermore, for any two distinct pointsa =b inE, there exist two open setsUa andUb such that,a∈Ua,b∈Ub, andUa∩Ub =∅.

Proof. It is straightforward. For the last point, lettingρ =d(a, b)/3 (in factρ =d(a, b)/2 works too), we can pickUa =B_0(a, ρ) andUb =B_0(b, ρ). By the triangle inequality, we must haveUa∩Ub =∅.

The above proposition leads to the very general concept of a topological space. One should be careful that, in general, the family of open sets is not closed under infinite intersections. For example, in R under the metric|x−y|, lettingUn =]− 1/n, +1/n[, eachUn is open, butnUn ={0}, which is not open.

26.2 Topological Spaces

Motivated by Proposition 26.1, a topological space is defined in terms of a family of sets satisfing the properties of open sets stated in that proposition.

Definition 26.5. Given a setE, a topology onE (or a topological structure onE), is defined as a familyO of subsets ofE called open sets, and satisfying the following three properties: (1) For every finite family (Ui)1≤i≤n of setsUi∈ O, we haveU1∩ · · · ∩Un∈ O, i.e.,O is closed under finite intersections.

(2) For every arbitrary family (Ui)i∈I of setsUi∈ O, we havei∈IUi∈ O, i.e.,O is closed under arbitrary unions.

(3)∅ ∈ O, andE∈ O, i.e.,∅ andE belong toO.

A set E together with a topologyO onE is called a topological space. Given a topological space (E,O), a subsetF ofE is a closed set ifF =E−U for some open setU∈ O, i.e.,F is the complement of some open set.

It is possible that an open set is also a closed set. For example,∅ andE are both open and closed. When a topological space contains a proper nonempty subsetU which is both open and closed, the spaceE is said to be disconnected.

A topological space ( E,O) is said to satisfy the Hausdorff separation axiom (or T_2- separation axiom) if for any two distinct pointsa =b inE, there exist two open setsUa and Ub such that,a∈Ua,b∈Ub, andUa∩Ub =∅. When theT_2-separation axiom is satisfied, we also say that (E,O) is a Hausdorff space .

As shown by Proposition 26.1, any metric space is a topological Hausdorff space, the family of open sets being in fact the family of arbitrary unions of open balls. Similarly, any normed vector space is a topological Hausdorff space, the family of open sets being the family of arbitrary unions of open balls. The topologyO consisting of all subsets ofE is called the discrete topology.

Remark: Most (if not all) spaces used in analysis are Hausdorff spaces. Intuitively, the Hausdorff separation axiom says that there are enough “small” open sets. Without this axiom, some counter-intuitive behaviors may arise. For example, a sequence may have more than one limit point (or a compact set may not be closed). Nevertheless, non-Hausdorff topological spaces arise naturally in algebraic geometry. But even there, some substitute for separation is used.

One of the reasons why topological spaces are important is that the definition of a topology only involves a certain familyO of sets, and not how such family is generated from a metric or a norm. For example, different metrics or different norms can define the same family of open sets. Many topological properties only depend on the familyO and not on the specific metric or norm. But the fact that a topology is definable from a metric or a norm is important, because it usually implies nice properties of a space. All our examples will be spaces whose topology is defined by a metric or a norm.

By taking complements, we can state properties of the closed sets dual to those of Definition 26.5. Thus,∅ andE are closed sets, and the closed sets are closed under finite unions and arbitrary intersections.

It is also worth noting that the Hausdorff separation axiom implies that for every a∈E, the set{a} is closed. Indeed, ifx∈E− {a}, thenx =a, and so there exist open setsUa andUx such thata∈Ua,x∈Ux, andUa∩Ux =∅. Thus, for everyx∈E− {a}, there is an open setUx containingx and contained inE− {a}, showing by (O3) thatE− {a} is open, and thus that the set{a} is closed.

Given a topological space ( E,O), given any subsetA ofE, sinceE∈ O andE is a closed set, the familyCA ={F|A⊆F, F a closed set} of closed sets containingA is nonempty, and since any arbitrary intersection of closed sets is a closed set, the intersection CA of the sets in the familyCA is the smallest closed set containingA. By a similar reasoning, the union of all the open subsets contained inA is the largest open set contained inA.

Definition 26.6. Given a topological space (E,O), given any subsetA ofE, the smallest closed set containingA is denoted byA, and is called the closure, or adherence ofA. A subsetA ofE is dense inE ifA =E. The largest open set contained inA is denoted by

æA, and is called the interior ofA. The set FrA =A∩E−A is called the boundary (or frontier) ofA. We also denote the boundary ofA by∂A.

Remark: The notationA for the closure of a subsetA ofE is somewhat unfortunate, sinceA is often used to denote the set complement ofA inE. Still, we prefer it to more cumbersome notations such as clo(A), and we denote the complement ofA inE byE−A (or sometimes,Ac).

By definition, it is clear that a subset A ofE is closed iffA =A. The set Q of rationals is dense in R. It is easily shown thatA =æA∪∂A and æA∩∂A =∅. Another useful characterization ofA is given by the following proposition.

Proposition 26.2. Given a topological space (E,O), given any subsetA ofE, the closure A ofA is the set of all pointsx∈E such that for every open setU containingx, then U∩A =∅.

Proof. IfA =∅, since∅ is closed, the proposition holds trivially. Thus, assume thatA =∅. First, assume thatx∈A. LetU be any open set such thatx∈U. IfU∩A =∅, sinceU is open, thenE− U is a closed set containingA, and sinceA is the intersection of all closed sets containingA, we must havex∈E−U, which is impossible. Conversely, assume that x∈E is a point such that for every open setU containingx, thenU∩A =∅. LetF be any closed subset containingA. Ifx /∈F, sinceF is closed, thenU =E−F is an open set such thatx∈ U, andU∩A =∅, a contradiction. Thus, we havex∈F for every closed set containingA, that is,x∈A.

Often, it is necessary to consider a subset A of a topological spaceE, and to view the subsetA as a topological space. The following proposition shows how to define a topology on a subset.

Proposition 26.3. Given a topological space (E,O), given any subsetA ofE, let

U ={U∩A|U∈ O}

be the family of all subsets ofA obtained as the intersection of any open set inO withA. The following properties hold.

(1) The space (A,U) is a topological space.

(2) If E is a metric space with metricd, then the restrictiondA:A×A→ R+ of the metricd toA defines a metric space. Furthermore, the topology induced by the metric

dA agrees with the topology defined byU, as above.

Proof. Left as an exercise.

Proposition 26.3 suggests the following definition.

Definition 26.7. Given a topological space (E,O), given any subsetA ofE, the subspace topology onA induced byO is the familyU of open sets defined such that

U ={U∩A|U∈ O}

is the family of all subsets ofA obtained as the intersection of any open set inO withA. We say that (A,U) has the subspace topology. If (E, d) is a metric space, the restriction dA:A×A→ R+ of the metricd toA is called the subspace metric.

For example, ifE = Rn andd is the Euclidean metric, we obtain the subspace topology on the closedn-cube

{(x_1, . . . , xn)∈E|ai≤xi≤bi, 1≤i≤n}.

One should realize that every open set U∈ O which is entirely contained inA is also in the familyU, butU may contain open sets that are not inO. For example, ifE = R with|x−y|, andA = [a, b], then sets of the form [a, c[, witha < c < b belong toU, but they are not open sets for R under|x−y|. However, there is agreement in the following situation.

Proposition 26.4. Given a topological space (E,O), given any subsetA ofE, ifU is the subspace topology, then the following properties hold.

(1) IfA is an open setA∈ O, then every open setU∈ U is an open setU∈ O. (2) IfA is a closed set inE, then every closed set w.r.t. the subspace topology is a closed set w.r.t.O.

Proof. Left as an exercise.

The concept of product topology is also useful. We have the following proposition. Proposition 26.5. Givenn topological spaces (Ei,Oi), letB be the family of subsets of E1× · · · ×En defined as follows:

B ={U1× · · · ×Un|Ui∈ Oi, 1≤i≤n},

and letP be the family consisting of arbitrary unions of sets inB, including∅. Then,P is a topology onE1× · · · ×En.

Proof. Left as an exercise.

Definition 26.8. Givenn topological spaces (Ei,Oi), the product topology onE1× · · · ×En is the familyP of subsets ofE1× · · · ×En defined as follows: if

B ={U1× · · · ×Un|Ui∈ Oi, 1≤i≤n}, thenP is the family consisting of arbitrary unions of sets inB, including∅.

If each (Ei,i) is a normed vector space, there are three natural norms that can be defined onE1× · · · ×En:

(x_1, . . . , xn)1 = x1 1 +· · · + xn n,

1

(x_1, . . . , xn)2 = x_1 2 +· · · + xn 2 2,1 n (x_1, . . . , xn)∞ = max{x1 1, . . . , xn n}.

It is easy to show that they all define the same topology, which is the product topology. It can also be verified that whenEi = R, with the standard topology induced by|x−y|, the topology product on Rn is the standard topology induced by the Euclidean norm.

Definition 26.9. Two metricsd_1 andd_2 on a spaceE are equivalent if they induce the same topologyO onE (i.e., they define the same familyO of open sets). Similarly, two norms 1 and_2 on a spaceE are equivalent if they induce the same topologyO onE.

Remark: Given a topological space (E,O), it is often useful, as in Proposition 26.5, to define the topologyO in terms of a subfamilyB of subsets ofE. We say that a familyB of subsets ofE is a basis for the topologyO, ifB is a subset ofO, and if every open setU in O can be obtained as some union (possibly infinite) of sets inB (agreeing that the empty union is the empty set).

It is immediately verified that if a familyB = (Ui)i∈I is a basis for the topology of (E,O), thenE = i∈IUi, and the intersection of any two setsUi, Uj∈ B is the union of some sets in the familyB (again, agreeing that the empty union is the empty set). Conversely, a family B with these properties is the basis of the topology obtained by forming arbitrary unions of sets inB.

A subbasis forO is a familyS of subsets ofE, such that the familyB of all finite intersections of sets inS (includingE itself, in case of the empty intersection) is a basis of O

.

The following proposition gives useful criteria for determining whether a family of open subsets is a basis of a topological space.

Proposition 26.6. Given a topological space (E,O) and a familyB of open subsets inOthe following properties hold:

(1) The familyB is a basis for the topologyO iff for every open setU∈ O and every x∈U, there is someB∈ B such thatx∈B andB⊆U.

(2) The familyB is a basis for the topologyO iff (a) For everyx∈E, there is someB∈ B such thatx∈B.

(b) For any two open subsets,B_1, B2∈ B, for everyx∈E, ifx∈B1∩B_2, then there is someB3∈ B such thatx∈B_3 andB3⊆B1∩B_2.

We now consider the fundamental property of continuity.

26.3 Continuous Functions, Limits

Definition 26.10. Let (E,OE) and (F,O F) be topological spaces, and letf :E→F be a function. For everya∈E, we say thatf is continuous ata, if for every open setV∈ OF containingf(a), there is some open setU∈ OE containinga, such that,f(U)⊆V . We say thatf is continuous if it is continuous at everya∈E.

Define a neighborhood ofa∈E as any subsetN ofE containing some open setO∈ O such thata∈O. Now, iff is continuous ata andN is any neighborhood off(a), there is some open set V⊆ N containingf(a), and sincef is continuous ata, there is some open setU containinga, such thatf(U)⊆V . SinceV⊆N, the open setU is a subset off−1(N) containinga, andf−1(N) is a neighborhood ofa. Conversely, iff−1(N) is a neighborhood ofa wheneverN is any neighborhood off(a), it is immediate thatf is continuous ata. It is easy to see that Definition 26.10 is equivalent to the following statements.

Proposition 26.7. Let (E,OE) and (F,OF) be topological spaces, and letf :E→F be a function. For everya∈E, the functionf is continuous ata∈E iff for every neighborhood N off(a)∈F, thenf−1(N) is a neighborhood ofa. The function f is continuous onE iff f−1(V ) is an open set inOE for every open setV∈ OF.

IfE andF are metric spaces defined by metricsd_1 andd_2, we can show easily thatf is continuous ata iff

for every > 0, there is someη > 0, such that, for everyx∈E,

ifd_1(a, x)≤η, thend_2(f(a), f(x))≤.

Similarly, ifE andF are normed vector spaces defined by norms_1 and_2, we can show easily thatf is continuous ata iff

for every > 0, there is someη > 0, such that, for everyx∈E,

if x−a1≤η, then f(x)−f(a)2≤. It is worth noting that continuity is a topological notion, in the sense that equivalent metrics (or equivalent norms) define exactly the same notion of continuity.

If ( E,OE) and (F,O F) are topological spaces, andf :E→F is a function, for every nonempty subsetA⊆E ofE, we say thatf is continuous onA if the restriction off toA is continuous with respect to (A,U) and (F,OF), whereU is the subspace topology induced byOE onA.

Given a product E1×· · ·×En of topological spaces, as usual, we letπi:E1×· · ·×En→Ei be the projection function such that,πi(x_1, . . . , xn) =xi. It is immediately verified that each πi is continuous.

Given a topological space ( E,O), we say that a pointa∈E is isolated if{ a} is an open set inO. Then, if (E,OE) and (F,OF) are topological spaces, any functionf :E→F is continuous at every isolated pointa∈E. In the discrete topology, every point is isolated.

In a nontrivial normed vector space ( E, ) (withE ={0}), no point is isolated. To show this, we show that every open ballB_0(u, ρ,) contains some vectors different fromu. Indeed, sinceE is nontrivial, there is somev∈E such thatv = 0, and thusλ = v > 0 (by (N1)). Let

w =u + ρ v.λ + 1

Sincev = 0 andρ > 0, we havew =u. Then,

w

−

u

=

ρ

λ + 1v = ρλ < ρ,λ + 1 which shows that w−u < ρ, forw =u. The following proposition is easily shown.

Proposition 26.8. Given topological spaces (E,OE), (F,OF), and (G,OG), and two functionsf :E→F andg:F→G, iff is continuous at a∈E andg is continuous atf(a)∈F, thengæf :E→G is continuous ata∈E. Givenn topological spaces (Fi,Oi), for every functionf :E→F1 × · · · ×Fn, thenf is continuous ata∈E iff everyfi:E→Fi is continuous ata , wherefi =πiæf.

One can also show that in a metric space ( E, d), the normd:E×E→ R is continuous, whereE×E has the product topology, and that for a normed vector space (E, ), the norm:E→ R is continuous.

Given a function f :E1× · · · × En→ F, we can fix n− 1 of the arguments, say a_1, . . . , ai−1, ai+1, . . . , an, and viewf as a function of the remaining argument,

xi→f(a_1, . . . , ai−1, xi, ai+1, . . . , an),

where xi∈Ei. Iff is continuous, it is clear that eachfi is continuous. One should be careful that the converse is false! For example, consider the function f : R×R→ R, defined such that,

f(x, y) = xy if (x, y) = (0,0), and f(0,0) = 0.x2 +y2

The function f is continuous on R×R− {(0,0)}, but on the liney =mx, withm = 0, we havef(x, y) =m = 0, and thus, on this line,f(x, y) does not approach 0 when (x, y)1+m2

approaches (0,0).

The following proposition is useful for showing that real-valued functions are continuous.

Proposition 26.9. IfE is a topological space, and (R,|x−y|) the reals under the standard topology, for any two functionsf :E→,λf,f·g → R, for anya∈E, for anyλ∈ R, if R andg:E

f andg are continuous ata, thenf +g , are continuous ata , andf/g is continuous ata ifg(a) = 0.

Proof. Left as an exercise.

Using Proposition 26.9, we can show easily that every real polynomial function is continuous.

The notion of isomorphism of topological spaces is defined as follows.

Definition 26.11. Let (E,OE) and (F,OF) be topological spaces, and letf :E→F be a function. We say thatf is a homeomorphism betweenE andF iff is bijective, and both f :E→F andf−1:F→E are continuous.

One should be careful that a bijective continuous function f :E→F is not necessarily an homeomorphism. For example, ifE = R with the discrete topology, andF = R with the standard topology, the identity is not a homeomorphism. Another interesting example involving a parametric curve is given below. LetL: R→ R^2 be the function, defined such that,

L

1

(

t

) =

t(1 +t^2) 1 +t4 , t

(1

−

t

2)

L_2(t) =1 +t4 .

If we think of ( x(t), y(t)) = (L_1(t), L_2(t)) as a geometric point in R^2, the set of points (x(t), y(t)) obtained by lettingt vary inR from−∞ to +∞, defines a curve having the shape of a “figure eight”, with self-intersection at the origin, called the “lemniscate of Bernoulli”. The mapL is continuous, and in fact bijective, but its inverseL−1 is not continuous. Indeed, when we approach the origin on the branch of the curve in the upper left quadrant (i.e., points such that,x≤ 0,y≥ 0), thent goes to−∞, and when we approach the origin on the branch of the curve in the lower right quadrant (i.e., points such that,x≥ 0,y≤ 0), thent goes to +∞.

We also review the concept of limit of a sequence. Given any setE, a sequence is any functionx: N→E, usually denoted by (xn)n∈N, or (xn)n≥0, or even by (xn).

Definition 26.12. Given a topological space (E,O), we say that a sequence (xn)n∈N converges to somea∈E if for every open setU containinga, there is somen0≥ 0, such that, xn∈U, for alln≥n_0. We also say that a is a limit of (xn)n∈N.

WhenE is a metric space with metricd, it is easy to show that this is equivalent to the fact that,

for every > 0, there is somen0≥ 0, such that,d(xn, a)≤ , for alln≥n_0. WhenE is a normed vector space with norm , it is easy to show that this is equivalent to the fact that,

for every > 0, there is somen0≥ 0, such that, xn−a ≤ , for alln≥n_0. The following proposition shows the importance of the Hausdorff separation axiom. Proposition 26.10. Given a topological space (E,O), if the Hausdorff separation axiom holds, then every sequence has at most one limit.

Proof. Left as an exercise.

It is worth noting that the notion of limit is topological, in the sense that a sequence converge to a limitb iff it converges to the same limitb in any equivalent metric (and similarly for equivalent norms).

We still need one more concept of limit for functions.

Definition 26.13. Let (E,OE) and (F,OF) be topological spaces, letA be some nonempty subset ofE, and letf :A→F be a function. For anya∈A and anyb∈F, we say thatf(x) approachesb asx approaches a with values inA if for every open setV∈ OF containingb, there is some open setU∈ OE containinga, such that,f(U∩A)⊆V . This is denoted by

x

limAf(x) =b.

→a,x∈

First, note that by Proposition 26.2, since a∈A, for every open setU containinga, we haveU∩A =∅, and the definition is nontrivial. Also, even ifa∈A, the valuef(a) off at a plays no role in this definition. WhenE andF are metric space with metricsd_1 andd_2, it can be shown easily that the definition can be stated as follows:

For every > 0, there is someη > 0, such that, for everyx∈A,

ifd_1(x, a)≤η, thend_2(f(x), b)≤. WhenE andF are normed vector spaces with norms_1 and_2, it can be shown easily that the definition can be stated as follows:

For every > 0, there is someη > 0, such that, for everyx∈A,

if x−a1≤η, then f(x)−b2≤.

We have the following result relating continuity at a point and the previous notion.

Proposition 26.11. Let (E,OE) and (F,OF) be two topological spaces, and letf :E→ F be a function. For anya∈E, the function f is continuous ata ifff(x) approachesf(a) whenx approachesa (with values inE).

Proof. Left as a trivial exercise.

Another important proposition relating the notion of convergence of a sequence to continuity, is stated without proof.

Proposition 26.12. Let (E,OE) and (F,OF) be two topological spaces, and letf :E→F be a function.

(1) Iff is continuous, then for every sequence (xn)n∈N inE, if (xn) converges toa, then (f(xn)) converges tof(a).

(2) IfE is a metric space, and (f(xn)) converges tof(a) whenever (xn) converges toa, for every sequence (xn)n∈N inE, thenf is continuous.

A special case of Definition 26.13 will be used when E andF are (nontrivial) normed vector spaces with norms_1 and_2. LetU be any nonempty open subset ofE. We showed earlier thatE has no isoled points and that every set{v} is closed, for everyv∈E. SinceE is nontrivial, for everyv∈U, there is a nontrivial open ball contained inU (an open ball not reduced to its center). Then, for everyv∈U,A =U− {v} is open and nonempty, and clearly,v∈A. For anyv∈U, iff(x) approachesb whenx approachesv with values inA =U− {v}, we say thatf(x) approachesb whenx approachesv with values =v inU. This is denoted by

x

lim

→v,x∈U,x=vf(x) =b.

Remark: Variations of the above case show up in the following case:E = R, andF is some arbitrary topological space. LetA be some nonempty subset of R, and letf :A→F be some function. For anya∈A, we say thatf is continuous on the right ata if

x

→

a,x

lima, +∞[f(x) =f(a).

∈A∩[

We can define continuity on the left ata in a similar fashion. Let us consider another variation. LetA be some nonempty subset ofR, and letf :A→ F be some function. For anya∈A, we say thatf has a discontinuity of the first kind ata if

x

→

a,x

lim,a[f(x) =f(a−)

∈A∩ ]−∞

and

x

→

a,x

lima, +∞[f(x) =f(a+)

∈A∩ ]

both exist, and eitherf(a−) =f(a), orf(a+) =f(a).

Note that it is possible that f(a−) = f(a+), butf is still discontinuous ata if this common value differs fromf(a). Functions defined on a nonempty subset of R, and that are continuous, except for some points of discontinuity of the first kind, play an important role in analysis.

We now turn to connectivity properties of topological spaces.

26.4 Connected Sets

Connectivity properties of topological spaces play a very important role in understanding the topology of surfaces. This section gathers the facts needed to have a good understanding of the classification theorem for compact surfaces (with boundary). The main references are Ahlfors and Sario [1] and Massey [74, 75]. For general backgroud on topology, geometry, and algebraic topology, we also highly recommend Bredon [16] and Fulton [39].

Definition 26.14. A topological space, (E,O), is connected if the only subsets ofE that are both open and closed are the empty set andE itself. Equivalently, (E,O) is connected if E cannot be written as the union,E =U∪V , of two disjoint nonempty open sets,U, V , ifE cannot be written as the union,E =U∪V , of two disjoint nonempty closed sets. A subset, S⊆E, is connected if it is connected in the subspace topology onS induced by (E,O). A connected open set is called a region and a closed set is a closed region if its interior is a connected (open) set.

Intuitively, if a space is not connected, it is possible to define a continuous function which is constant on disjoint “connected components” and which takes possibly distinct values on disjoint components. This can be stated in terms of the concept of a locally constant function. Given two topological spaces,X, Y , a function,f :X→Y , is locally constant if for every x∈X, there is an open set,U⊆X, such thatx∈X andf is constant onU.

We claim that a locally constant function is continuous. In fact, we will prove that f−1(V ) is open for every subset,V⊆Y (not just for an open setV ). It is enough to show thatf−1(y) is open for everyy∈Y , since for every subsetV⊆Y ,

f−1(V ) = f−1(y),

y∈V

and open sets are closed under arbitrary unions. However, either f−1(y) =∅ ify∈Y−f(X) orf is constant onU =f−1(y) ify∈f(X) (with valuey), and sincef is locally constant, for everyx∈U, there is some open set,W⊆X, such thatx∈ W andf is constant onW, which implies thatf(w) =y for allw∈W and thus, thatW⊆U, showing thatU is a union of open sets and thus, is open. The following proposition shows that a space is connected iff every locally constant function is constant:

Proposition 26.13. A topological space is connected iff every locally constant function is constant.

Proof. First, assume thatX is connected. Letf :X→Y be a locally constant function to some spaceY and assume thatf is not constant.

constant,U_1 =f−1(y) =X, and of course,U_1 =∅.

Pick anyy∈f(Y ). Sincef is not

We proved just before Proposition 26.13 thatf−1(V ) is open for every subsetV⊆Y , and thusU_1 =f−1(y) =f−1({y}) and U_2 =f−1(Y− {y}) are both open, nonempty, and clearlyX =U1∪U_2 andU_1 andU_2 are disjoint. This contradicts the fact thatX is connected andf must be constant.

Assume that every locally constant function, f :X→ Y , to a Hausdorff space,Y , is constant. IfX is not connected, we can writeX =U1∪U_2, where bothU_1, U_2 are open, disjoint, and nonempty. We can define the function,f :X→ R, such thatf(x) = 1 onU_1 andf(x) = 0 onU_2. SinceU_1 andU_2 are open, the functionf is locally constant, and yet not constant, a contradiction.

The following standard proposition characterizing the connected subsets of R can be found in most topology texts (for example, Munkres [81], Schwartz [89]). For the sake of completeness, we give a proof.

Proposition 26.14. A subset of the real line, R, is connected iff it is an interval, i.e., of the form [a, b], ]a, b], wherea =−∞ is possible, [a, b[ , whereb = +∞ is possible, or ]a, b[ , wherea =−∞ orb = +∞ is possible.

Proof. Assume thatA is a connected nonempty subset of R. The cases whereA =∅ or A consists of a single point are trivial. We show that whenevera, b∈A,a < b, then the entire interval [a, b] is a subset ofA. Indeed, if this was not the case, there would be some c∈]a, b[ such thatc /∈A, and then we could writeA = ( ]−∞, c[∩A)∪( ]c,+∞[∩A), where ]− ∞, c[∩A and ]c,+∞[∩A are nonempty and disjoint open subsets ofA, contradicting the fact thatA is connected. It follows easily thatA must be an interval.

Conversely, we show that an interval, I, must be connected. LetA be any nonempty subset ofI which is both open and closed inI. We show thatI =A. Fix anyx∈A and consider the set,Rx, of ally such that [x, y]⊆A. If the setRx is unbounded, then Rx = [x,+∞[ . Otherwise, if this set is bounded, letb be its least upper bound. We claim thatb is the right boundary of the intervalI. BecauseA is closed inI, unlessI is open on the right andb is its right boundary, we must haveb∈A. In the first case, A∩[x, b[ =I∩[x, b[ = [x, b[ . In the second case, becauseA is also open inI, unlessb is the right boundary of the intervalI (closed on the right), there is some open set ]b−η, b +η[ contained inA, which implies that [x, b+η/2]⊆A, contradicting the fact thatb is the least upper bound of the setRx. Thus,b must be the right boundary of the intervalI (closed on the right). A similar argument applies to the set,Ly, of allx such that [x, y]⊆A and either Ly is unbounded, or its greatest lower bounda is the left boundary ofI (open or closed on the left). In all cases, we showed thatA =I, and the interval must be connected.

A characterization on the connected subsets of Rn is harder and requires the notion of arcwise connectedness. One of the most important properties of connected sets is that they are preserved by continuous maps.

Proposition 26.15. Given any continuous map,f :E→F, ifA⊆E is connected, then f(A) is connected.

Proof. Iff(A) is not connected, then there exist some nonempty open sets,U, V , inF such thatf(A)∩U andf(A)∩V are nonempty and disjoint, and

Then,f− f(A) = (f(A)∩U)∪ (f(A)∩V ).

1(U) andf−1(V ) are nonempty and open sincef is continuous and 1(U))∪ (A∩f−1(V )),∩f−

A = (A

withA^1(U) andA∩f−1(V ) nonempty, disjoint, and open inA, contradicting the fact∩f−

thatA is connected.

An important corollary of Proposition 26.15 is that for every continuous function, f :E→ R, whereE is a connected space,f(E) is an interval. Indeed, this follows from Proposition 26.14. Thus, iff takes the valuesa andb wherea < b, thenf takes all valuesc∈ [a, b]. This is a very important property.

Even if a topological space is not connected, it turns out that it is the disjoint union of maximal connected subsets and these connected components are closed inE. In order to obtain this result, we need a few lemmas.

Lemma 26.16. Given a topological space,E, for any family, (Ai)i∈I, of (nonempty) connected subsets ofE, ifAi∩Aj =∅ for alli, j∈I, then the union, A =i∈IAi, of the family, (Ai)i∈I, is also connected.

Proof. Assume that i∈IAi is not connected. Then, there exists two nonempty open subsets, U andV , ofE such that A∩U andA∩V are disjoint and nonempty and such that A = (A∩U)∪ (A∩V ). Now, for everyi∈I, we can write

Ai = (Ai∩U)∪ (Ai∩V ), whereAi∩U andAi∩V are disjoint, sinceAi⊆A andA∩U andA∩V are disjoint. Since Ai is connected, eitherAi∩U =∅ orAi∩V =∅. This implies that eitherAi⊆A∩U or Ai⊆A∩V . However, by assumption,Ai∩Aj =∅, for alli, j∈I, and thus, either both Ai⊆A∩U andAj⊆A∩U, or bothAi⊆A∩V andAj⊆A∩V , sinceA∩U andA∩V are disjoint. Thus, we conclude that eitherAi⊆A∩U for alli∈I, orAi⊆A∩V for all i∈I. But this proves that either

A = Ai⊆A∩U,

i∈I

or A = Ai⊆A∩V,

i∈I

contradicting the fact that bothA∩U andA∩V are disjoint and nonempty. Thus,A must be connected.

In particular, the above lemma applies when the connected sets in a family (Ai)i∈I have a point in common.

Lemma 26.17. IfA is a connected subset of a topological space,E, then for every subset, B, such thatA⊆B⊆A, whereA is the closure ofA inE, the setB is connected.

Proof. IfB is not connected, then there are two nonempty open subsets,U, V , ofE such thatB∩U andB∩V are disjoint and nonempty, and

B = (B∩U)∪ (B∩V ).

SinceA⊆B, the above implies that

A = (A∩U)∪ (A∩V ),

and since A is connected, eitherA∩U =∅, orA∩V =∅. Without loss of generality, assume thatA∩V =∅, which implies thatA⊆A∩ U⊆B∩U. However,B∩U is closed in the subspace topology forB and sinceB⊆A andA is closed inE, the closure ofA inB w.r.t. the subspace topology ofB is clearlyB∩A =B, which implies thatB⊆B∩U (since the closure is the smallest closed set containing the given set). Thus,B∩V =∅, a contradiction.

In particular, Lemma 26.17 shows that ifA is a connected subset, then its closure,A, is also connected. We are now ready to introduce the connected components of a space.

Definition 26.15. Given a topological space, (E,O), we say that two points,a, b∈E, are connected if there is some connected subset,A, ofE such thata∈A andb∈A.

It is immediately verified that the relation “ a andb are connected inE” is an equivalence relation. Only transitivity is not obvious, but it follows immediately as a special case of Lemma 26.16. Thus, the above equivalence relation defines a partition ofE into nonempty disjoint connected components. The following proposition is easily proved using Lemma 26.16 and Lemma 26.17:

Proposition 26.18. Given any topological space,E, for anya∈E, the connected component containinga is the largest connected set containinga . The connected components ofE are closed.

The notion of a locally connected space is also useful.

Definition 26.16. A topological space, (E,O), is locally connected if for everya∈E, for every neighborhood,V , ofa, there is a connected neighborhood,U, ofa such thatU⊆V . As we shall see in a moment, it would be equivalent to require thatE has a basis of connected open sets.

There are connected spaces that are not locally connected and there are locally connected spaces that are not connected. The two properties are independent.

Proposition 26.19. A topological space,E, is locally connected iff for every open subset, A, ofE, the connected components ofA are open.

Proof. Assume thatE is locally connected. LetA be any open subset ofE and letC be one of the connected components ofA. For anya∈C⊆A, there is some connected neigborhood, U, ofa such thatU⊆A and sinceC is a connected component ofA containinga, we must haveU⊆C. This shows that for everya∈C, there is some open subset containinga contained inC, soC is open.

Conversely, assume that for every open subset,A, ofE, the connected components ofA are open. Then, for everya∈E and every neighborhood,U, ofa, sinceU contains some

open set A containinga, the interior, æU, ofU is an open set containinga and its connected components are open. In particular, the connected componentC containinga is a connected open set containinga and contained inU.

Proposition 26.19 shows that in a locally connected space, the connected open sets form a basis for the topology. It is easily seen that Rn is locally connected. Another very important property of surfaces and more generally, manifolds, is to be arcwise connected. The intuition is that any two points can be joined by a continuous arc of curve. This is formalized as follows.

Definition 26.17. Given a topological space, (E,O), an arc (or path) is a continuous map, γ: [a, b]→E, where [a, b] is a closed interval of the real line, R. The pointγ(a) is the initial point of the arc and the pointγ(b) is the terminal point of the arc. We say thatγ is an arc joiningγ(a) andγ(b). An arc is a closed curve ifγ(a) =γ(b). The setγ([a, b]) is the trace of the arcγ.

Typically,a = 0 andb = 1. In the sequel, this will be assumed.

One should not confuse an arc,γ: [a, b]→E, with its trace. For example,γ could be constant, and thus, its trace reduced to a single point.

An arc is a Jordan arc ifγ is a homeomorphism onto its trace. An arc,γ: [a, b]→E, is a Jordan curve ifγ(a) =γ(b) andγ is injective on [a, b[ . Since [a, b] is connected, by Proposition 26.15, the traceγ([a, b]) of an arc is a connected subset ofE.

Given two arcsγ: [0,1]→E andδ: [0,1]→E such thatγ(1) =δ(0), we can form a new arc defined as follows:

Definition 26.18. Given two arcs,γ: [0,1]→E andδ: [0,1]→E, such thatγ(1) =δ(0), we can form their composition (or product),γδ,, defined such that

γδ

(

t

) =

γ(2t) if 0≤t≤ 1/2; δ(2t− 1) if 1/2≤t≤ 1.

The inverse,γ−1, of the arc,γ, is the arc defined such thatγ−1(t) =γ(1−t), for allt∈ [0,1].

It is trivially verified that Definition 26.18 yields continuous arcs.

Definition 26.19. A topological space, E, is arcwise connected if for any two points, a, b∈ E, there is an arc, γ: [0,1]→ E, joininga andb, i.e., such thatγ(0) = a and γ(1) =b. A topological space,E, is locally arcwise connected if for everya∈E, for every neighborhood,V , ofa, there is an arcwise connected neighborhood,U, ofa such thatU⊆V .

The space Rn is locally arcwise connected, since for any open ball, any two points in this ball are joined by a line segment. Manifolds and surfaces are also locally arcwise connected. Proposition 26.15 also applies to arcwise connectedness (this is a simple exercise). The following theorem is crucial to the theory of manifolds and surfaces:

Theorem 26.20. If a topological space,E, is arcwise connected, then it is connected. If a topological space,E, is connected and locally arcwise connected, thenE is arcwise connected. Proof. First, assume thatE is arcwise connected. Pick any point,a, inE. SinceE is arcwise connected, for everyb∈E, there is a path,γb: [0,1]→E, froma tob and so,

E = γb([0,1])

b∈E

a union of connected subsets all containinga. By Lemma 26.16,E is connected.

Now assume that E is connected and locally arcwise connected. For any pointa∈E, let Fa be the set of all points,b, such that there is an arc,γb: [0,1]→E, froma tob. Clearly, Fa containsa. We show thatFa is both open and closed. For anyb∈Fa, sinceE is locally arcwise connected, there is an arcwise connected neighborhoodU containingb (becauseE is a neighborhood ofb). Thus,b can be joined to every pointc∈U by an arc, and since by the definition ofFa, there is an arc froma tob, the composition of these two arcs yields an arc froma toc, which shows thatc∈ Fa. But thenU⊆Fa and thus,Fa is open. Now assume thatb is in the complement ofFa. As in the previous case, there is some arcwise connected neighborhoodU containingb. Thus, every pointc∈U can be joined tob by an arc. If there was an arc joininga toc, we would get an arc froma tob, contradicting the fact that b is in the complement ofFa. Thus, every pointc∈U is in the complement ofFa, which shows thatU is contained in the complement ofFa, and thus, that the the complement of Fa is open. Consequently, we have shown thatFa is both open and closed and since it is nonempty, we must haveE =Fa, which shows thatE is arcwise connected.

IfE is locally arcwise connected, the above argument shows that the connected components ofE are arcwise connected.

It is not true that a connected space is arcwise connected. For example, the space consisting of the graph of the function

f(x) = sin(1/x),

wherex > 0, together with the portion of they-axis, for which−1≤y≤ 1, is connected, but not arcwise connected.

A trivial modification of the proof of Theorem 26.20 shows that in a normed vector space,E, a connected open set is arcwise connected by polygonal lines (i.e., arcs consisting of line segments). This is because in every open ball, any two points are connected by a line segment. Furthermore, ifE is finite dimensional, these polygonal lines can be forced to be parallel to basis vectors.

We now consider compactness.

26.5 Compact Sets

The property of compactness is very important in topology and analysis. We provide a quick review geared towards the study of surfaces and for details, we refer the reader to Munkres [81], Schwartz [89]. In this section, we will need to assume that the topological spaces are Hausdorff spaces. This is not a luxury, as many of the results are false otherwise.

There are various equivalent ways of defining compactness. For our purposes, the most convenient way involves the notion of open cover.

Definition 26.20. Given a topological space,E, for any subset,A, ofE, an open cover, (Ui)i∈I, ofA is a family of open subsets ofE such thatA⊆ i∈IUi. An open subcover of an open cover, (Ui)i∈I, ofA is any subfamily, (Uj)j∈J, which is an open cover ofA, withJ⊆I. An open cover, (Ui)i∈I, ofA is finite ifI is finite. The topological space,E, is compact if it is Hausdorff and for every open cover, (Ui)i∈I, ofE, there is a finite open subcover, (Uj)j∈J, ofE. Given any subset,A, ofE, we say thatA is compact if it is compact with respect to the subspace topology. We say thatA is relatively compact if its closureA is compact.

It is immediately verified that a subset, A, ofE is compact in the subspace topology relative toA iff for every open cover, (Ui)i∈I, ofA by open subsets ofE, there is a finite open subcover, (Uj)j∈J, ofA. The property that every open cover contains a finite open subcover is often called the Heine-Borel-Lebesgue property. By considering complements, a Hausdorff space is compact iff for every family, (Fi)i∈I, of closed sets, ifi∈IFi =∅, then

j∈JFj =∅ for some finite subset,J, ofI.

Definition 26.20 requires that a compact space be Hausdorff. There are books in which a compact space is not necessarily required to be Hausdorff. Following Schwartz, we prefer calling such a space quasi-compact.

Another equivalent and useful characterization can be given in terms of families having the finite intersection property. A family, (Fi)i∈I, of sets has the finite intersection property ifj∈JFj =∅ for every finite subset,J, ofI. We have the following proposition:

Proposition 26.21. A topological Hausdorff space, E, is compact iff for every family, (Fi)i∈I, of closed sets having the finite intersection property, theni∈IFi =∅.

Proof. IfE is compact and (Fi)i∈I is a family of closed sets having the finite intersection property, then i∈IFi cannot be empty, since otherwise we would have j∈JFj =∅ for some finite subset,J, of I, a contradiction. The converse is equally obvious.

Another useful consequence of compactness is as follows. For any family, ( Fi)i∈I, of closed sets such thatFi+1⊆Fi for alli∈I, ifi∈IFi =∅, thenFi =∅ for somei∈I. Indeed, there must be some finite subset,J, ofI such that j ∈JFj =∅ and sinceFi+1⊆Fi for all i∈I, we must haveFj =∅ for the smallestFj in (Fj)j∈J. Using this fact, we note that R is not compact. Indeed, the family of closed sets, ([n,+∞[ )n≥0, is decreasing and has an empty intersection.

Given a metric space, if we define a bounded subset to be a subset that can be enclosed in some closed ball (of finite radius), then any nonbounded subset of a metric space is not compact. However, a closed interval [a, b] of the real line is compact.

Proposition 26.22. Every closed interval, [a, b], of the real line is compact.

Proof. We proceed by contradiction. Let (Ui)i∈I be any open cover of [a, b] and assume that there is no finite open subcover. Letc = (a +b)/2. If both [a, c] and [c, b] had some finite open subcover, so would [a, b], and thus, either [a, c] does not have any finite subcover, or [c, b] does not have any finite open subcover. Let [a_1, b_1] be such a bad subinterval. The same argument applies and we split [a_1, b_1] into two equal subintervals, one of which must be bad. Thus, having defined [an, bn] of length (b−a)/2n as an interval having no finite open subcover, splitting [an, bn] into two equal intervals, we know that at least one of the two has no finite open subcover and we denote such a bad interval by [an+1, bn+1]. The sequence (an) is nondecreasing and bounded from above byb, and thus, by a fundamental property of the real line, it converges to its least upper bound,α. Similarly, the sequence (bn) is nonincreasing and bounded from below bya and thus, it converges to its greatest lowest bound,β. Since [an, bn] has length (b−a)/2n, we must haveα =β. However, the common limitα =β of the sequences (an) and (bn) must belong to some open set,Ui, of the open cover and sinceUi is open, it must contain some interval [c, d] containingα. Then, because α is the common limit of the sequences (an) and (bn), there is someN such that the intervals [an, bn] are all contained in the interval [c, d] for alln≥N, which contradicts the fact that none of the intervals [an, bn] has a finite open subcover. Thus, [a, b] is indeed compact.

The argument of Proposition 26.22 can be adapted to show that in Rm, every closed set, [a_1, b_1]× · · · × [am, bm], is compact. At every stage, we need to divide into 2m subpieces instead of 2.

The following two propositions give very important properties of the compact sets, and they only hold for Hausdorff spaces:

Proposition 26.23. Given a topological Hausdorff space,E, for every compact subset,A, and every point,b, not inA, there exist disjoint open sets,U andV , such thatA⊆U and b∈V . As a consequence, every compact subset is closed.

Proof. SinceE is Hausdorff, for everya∈A, there are some disjoint open sets,Ua andVb, containinga andb respectively. Thus, the family, (Ua)a∈A, forms an open cover ofA. Since A is compact there is a finite open subcover, (Uj)j∈J, ofA, whereJ⊆A, and thenj∈JUj is an open set containingA disjoint from the open setj∈JVj containingb. This shows that every point,b, in the complement ofA belongs to some open set in this complement and thus, that the complement is open, i.e., thatA is closed.

Actually, the proof of Proposition 26.23 can be used to show the following useful property:

Proposition 26.24. Given a topological Hausdorff space, E, for every pair of compact disjoint subsets,A andB, there exist disjoint open sets,U andV , such thatA⊆U and B⊆V .

Proof. We repeat the argument of Proposition 26.23 withB playing the role ofb and use Proposition 26.23 to find disjoint open sets,Ua, containinga∈A and,Va, containingB.

The following proposition shows that in a compact topological space, every closed set is compact:

Proposition 26.25. Given a compact topological space,E, every closed set is compact. Proof. SinceA is closed,E−A is open and from any open cover, (Ui)i∈I, ofA, we can form an open cover ofE by addingE−A to (Ui)i∈I and, sinceE is compact, a finite subcover, (Uj)j∈J∪ {E−A}, ofE can be extracted such that (Uj)j∈J is a finite subcover ofA.

Remark: Proposition 26.25 also holds for quasi-compact spaces, i.e., the Hausdorff separation property is not needed.

Putting Proposition 26.24 and Proposition 26.25 together, we note that if X is compact, then for every pair of disjoint closed, setsA andB, there exist disjoint open sets,U andV , such thatA⊆U andB⊆V . We say thatX is a normal space.

Proposition 26.26. Given a compact topological space, E, for every a∈ E, for every neighborhood,V , ofa, there exists a compact neighborhood,U, ofa such thatU⊆V

Proof. SinceV is a neighborhood ofa, there is some open subset,O, ofV containinga. Then the complement,K =E−O, ofO is closed and sinceE is compact, by Proposition 26.25,K is compact. Now, if we consider the family of all closed sets of the form,K∩F, whereF is any closed neighborhood ofa, sincea /∈K, this family has an empty intersection and thus, there is a finite number of closed neighborhood,F_1, . . . , Fn, ofa, such thatK∩F1∩ · · · ∩Fn =∅. Then,U =F1∩ · · · ∩Fn is a compact neigborhood ofa contained inO⊆V .

It can be shown that in a normed vector space of finite dimension, a subset is compact iff it is closed and bounded. For Rn, the proof is simple.

In a normed vector space of infinite dimension, there are closed and bounded sets that are not compact!

More could be said about compactness in metric spaces but we will only need the notion of Lebesgue number, which will be discussed a little later. Another crucial property of compactness is that it is preserved under continuity.

Proposition 26.27. LetE be a topological space and letF be a topological Hausdorff space. For every compact subset,A, ofE, for every continuous map,f :E→F, the subspacef(A) is compact.

Proof. Let (Ui)i∈I be an open cover off(A). We claim that (f−1(Ui))i∈I is an open cover of A, which is easily checked. SinceA is compact, there is a finite open subcover, (f−1(Uj))j∈J, ofA, and thus, (Uj)j∈J is an open subcover off(A).

As a corollary of Proposition 26.27, if E is compact,F is Hausdorff, andf :E→F is continuous and bijective, thenf is a homeomorphism. Indeed, it is enough to show thatf−1 is continuous, which is equivalent to showing thatf maps closed sets to closed sets. However, closed sets are compact and Proposition 26.27 shows that compact sets are mapped to compact sets, which, by Proposition 26.23, are closed.

It can also be shown that if E is a compact nonempty space andf :E→ R is a continuous function, then there are pointsa, b∈E such thatf(a) is the minimum off(E) andf(b) is the maximum off(E). Indeed,f(E) is a compact subset of R and thus, a closed and bounded set which contains its greatest lower bound and its least upper bound.

Another useful notion is that of local compactness. Indeed, manifolds and surfaces are locally compact.

Definition 26.21. A topological space,E, is locally compact if it is Hausdorff and for every a∈E, there is some compact neighborhood,K, ofa.

From Proposition 26.26, every compact space is locally compact but the converse is false. It can be shown that a normed vector space of finite dimension is locally compact. Proposition 26.28. Given a locally compact topological space,E, for everya∈ E, for every neighborhood,N, ofa, there exists a compact neighborhood,U, ofa, such that U⊆N.

Proof. For anya∈E, there is some compact neighborhood,V , ofa. By Proposition 26.26, every neigborhood ofa relative toV contains some compact neighborhoodU ofa relative toV . But every neighborhood ofa relative toV is a neighborhood ofa relative toE and every neighborhoodN ofa inE yields a neighborhood,V∩N, ofa inV and thus, for every neighborhood,N, ofa, there exists a compact neighborhood,U, ofa such thatU⊆N.

It is much harder to deal with noncompact surfaces (or manifolds) than it is to deal with compact surfaces (or manifolds). However, surfaces (and manifolds) are locally compact and it turns out that there are various ways of embedding a locally compact Hausdorff space into a compact Hausdorff space. The most economical construction consists in adding just one point. This construction, known as the Alexandroff compactification, is technically useful, and we now describe it and sketch the proof that it achieves its goal.

To help the reader’s intuition, let us consider the case of the plane, R^2. If we view the plane, R^2, as embedded in 3-space, R^3, say as thexOy plane of equationz = 0, we can consider the sphere, Σ, of radius 1 centered on thez-axis at the point (0,0,1) and tangent to thexOy plane at the origin (sphere of equationx^2 +y^2 + (z− 1)2 = 1). IfN denotes the north pole on the sphere, i.e., the point of coordinates (0,0,2), then any line,D, passing through the north pole and not tangent to the sphere (i.e., not parallel to thexOy plane) intersects thexOy plane in a unique point,M, and the sphere in a unique point,P, other than the north pole,N. This, way, we obtain a bijection between thexOy plane and the punctured sphere Σ, i.e., the sphere with the north poleN deleted. This bijection is called a stereographic projection. The Alexandroff compactification of the plane puts the north pole back on the sphere, which amounts to adding a single point at infinity∞ to the plane. Intuitively, as we travel away from the originO towards infinity (in any direction!), we tend towards an ideal point at infinity∞. Imagine that we “bend” the plane so that it gets wrapped around the sphere, according to stereographic projection. A simpler example takes a line and gets a circle as its compactification. The Alexandroff compactification is a generalization of these simple constructions.

Definition 26.22. Let (E,O) be a locally compact space. Letω be any point not inE, and letEω =E∪ {ω}. Define the family,Oω, as follows:

=O ∪ {(E−K)∪ {ω} |K compact inE}.Oω

The pair, (Eω,Oω), is called the Alexandroff compactification (or one point compactification) of (E,O).

The following theorem shows that (Eω,Oω) is indeed a topological space, and that it is compact.

Theorem 26.29. LetE be a locally compact topological space. The Alexandroff compactification,Eω, ofE is a compact space such thatE is a subspace ofEω and ifE is not compact, thenE =Eω.

Proof. The verification thatOω is a family of open sets is not difficult but a bit tedious. Details can be found in Munkres [81] or Schwartz [89]. Let us show thatEω is compact. For every open cover, (Ui)i∈I, ofEω, sinceω must be covered, there is someUi0 of the form

Ui0 = (E−K_0)∪ {ω}

whereK_0 is compact inE. Consider the family, (Vi)i∈I, defined as follows:

Vi = Ui if Ui∈ O,

Vi = E−K if Ui = (E−K)∪ {ω},

where K is compact inE. Then, because eachK is compact and thus closed inE (sinceE is Hausdorff),E−K is open, and everyVi is an open subset ofE. Furthermore, the family, (Vi)i∈(I−{i0}), is an open cover ofK_0. SinceK_0 is compact, there is a finite open subcover, (Vj)j∈J, ofK_0, and thus, (Uj)j∈J∪{i0} is a finite open cover ofEω.

Let us show that Eω is Hausdorff. Given any two points,a, b∈Eω, if botha, b∈E, since E is Hausdorff and every open set inO is an open set inOω, there exist disjoint open sets, U, V (inO), such thata∈U andb∈V . Ifb =ω, sinceE is locally compact, there is some compact set,K, containing an open set,U, containinga and then,U andV = (E−K)∪{ω}are disjoint open sets (inOω) such thata∈U andb∈V .

The space E is a subspace ofEω because for every open set,U, inOω, eitherU∈ O andE∩U =U is open inE, orU = (E−K)∪ {ω}, whereK is compact inE, and thus, U∩E =E−K, which is open inE, sinceK is compact inE and thus, closed (sinceE is Hausdorff). Finally, ifE is not compact, for every compact subset,K, ofE,E−K is nonempty and thus, for every open set,U = (E−K)∪{ω}, containingω, we haveU∩E =∅, which shows thatω∈E and thus, thatE =Eω.

Finally, in studying surfaces and manifolds, an important property is the existence of a countable basis for the topology. Indeed, this property guarantees the existence of trianguations of surfaces, a crucial property.

Definition 26.23. A topological spaceE is called second-countable if there is a countable basis for its topology, i.e., if there is a countable family, (Ui)i≥0, of open sets such that every open set ofE is a union of open setsUi.

It is easily seen that Rn is second-countable and more generally, that every normed vector space of finite dimension is second-countable. It can also be shown that ifE is a locally compact space that has a countable basis, thenEω also has a countable basis (and in fact, is metrizable). We have the following properties.

Proposition 26.30. Given a second-countable topological spaceE, every open cover (Ui)i∈I, ofE contains some countable subcover.

Proof. Let (On)n≥0 be a countable basis for the topology. Then, all setsOn contained in someUi can be arranged into a countable subsequence, (&m)m≥0, of (On)n≥0 and for every &m, there is someUim such that &m⊆Uim. Furthermore, everyUi is some union of sets &j, and thus, everya∈E belongs to some &j, which shows that (&m)m≥0 is a countable open subcover of (Ui)i∈I.

As an immediate corollary of Proposition 26.30, a locally connected second-countable space has countably many connected components.

In second-countable Hausdorff spaces, compactness can be characterized in terms of accumulation points (this is also true for metric spaces).

Definition 26.24. Given a topological Hausdorff space,E, given any sequence, (xn), of points inE, a point,l∈E, is an accumulation point (or cluster point) of the sequence (xn) if every open set,U, containingl containsxn for infinitely manyn.

Clearly, ifl is a limit of the sequence, (xn), then it is an accumulation point, since every open set,U, containinga contains allxn except for finitely manyn.

Proposition 26.31. A second-countable topological Hausdorff space,E, is compact iff every sequence, (xn), has some accumulation point.

Proof. Assume that every sequence, (xn), has some accumulation point. Let (Ui)i∈I be some open cover ofE. By Proposition 26.30, there is a countable open subcover, (On)n≥0, forE. Now, ifE is not covered by any finite subcover of (On)n≥0, we can define a sequence, (xm), by induction as follows:

Let x_0 be arbitrary and for everym≥ 1, letxm be some point inE not inO1∪ · · · ∪Om, which exists, sinceO1∪ · · · ∪Om is not an open cover ofE. We claim that the sequence, (xm), does not have any accumulation point. Indeed, for everyl∈E, since (On)n≥0 is an open cover ofE, there is someOm such thatl∈Om, and by construction, everyxn with n≥m + 1 does not belong toOm, which means thatxn∈Om for only finitely manyn and l is not an accumulation point.

Conversely, assume that E is compact, and let (xn) be any sequence. Ifl∈E is not an accumulation point of the sequence, then there is some open set,Ul, such thatl∈Ul andxn∈Ul for only finitely manyn. Thus, if (xn) does not have any accumulation point, the family, (Ul)l∈E, is an open cover ofE and sinceE is compact, it has some finite open subcover, (Ul)l∈J, whereJ is a finite subset ofE. But everyUl withl∈J is such that xn∈Ul for only finitely manyn, and sinceJ is finite,xn∈ l∈JUl for only finitely manyn, which contradicts the fact that (Ul)l∈J is an open cover ofE , and thus contains all thexn. Thus, (xn) has some accumulation point.

Remark: It should be noted that the proof showing that ifE is compact, then every sequence has some accumulation point, holds for any arbitrary compact space (the proof does not use a countable basis for the topology). The converse also holds for metric spaces. We will prove this converse since it is a major property of metric spaces.

Given a metric space in which every sequence has some accumulation point, we first prove the existence of a Lebesgue number.

Lemma 26.32. Given a metric space,E, if every sequence, (xn), has an accumulation point, for every open cover, (Ui)i∈I, ofE, there is someδ > 0 (a Lebesgue number for (Ui)i∈I) such that, for every open ball,B_0(a, ), of radius ≤δ, there is some open subset,Ui , such that B_0(a, )⊆Ui.

Proof. If there was noδ with the above property, then, for every natural number,n, there would be some open ball,B_0(an,1/n), which is not contained in any open set,Ui, of the open cover, (Ui)i∈I. However, the sequence, (an), has some accumulation point,a, and since (Ui)i∈I is an open cover ofE, there is someUi such thata∈Ui. SinceUi is open, there is some open ball of centera and radius contained inUi. Now, sincea is an accumulation point of the sequence, (an), every open set containinga containsan for infinitely manyn and thus, there is somen large enough so that

1/n≤/2 and an∈B_0(a, /2),

which implies that B_0(an,1/n)⊆B_0(a, )⊆Ui, a contradiction.

By a previous remark, since the proof of Proposition 26.31 implies that in a compact topological space, every sequence has some accumulation point, by Lemma 26.32, in a compact metric space, every open cover has a Lebesgue number. This fact can be used to prove another important property of compact metric spaces, the uniform continuity theorem. Definition 26.25. Given two metric spaces, (E, dE) and (F, dF), a function,f :E→F, is uniformly continuous if for every > 0, there is someη > 0, such that, for alla, b∈E,

if dE(a, b)≤η then dF(f(a), f(b))≤.

The uniform continuity theorem can be stated as follows: Theorem 26.33. Given two metric spaces, (E, dE) and (F, dF), ifE is compact andf :E→F is a continuous function, then it is uniformly continuous.

Proof. Consider any > 0 and let (B_0(y, /2))y∈F be the open cover ofF consisting of open balls of radius/2. Sincef is continuous, the family,

(f−1(B_0(y, /2)))y∈F,

is an open cover of E. Since,E is compact, by Lemma 26.32, there is a Lebesgue number, δ, such that for every open ball,B_0(a, η), of radiusη≤δ, thenB_0(a, η)⊆f−1(B_0(y, /2)), for somey∈F. In particular, for anya, b∈E such thatdE(a, b)≤η =δ/2, we have a, b∈B_0(a, δ) and thus,a, b∈f−1(B_0(y, /2)), which implies thatf(a), f(b)∈B_0(y, /2). But then,dF(f(a), f(b))≤ , as desired.

We now prove another lemma needed to obtain the characterization of compactness in metric spaces in terms of accumulation points.

Lemma 26.34. Given a metric space,E, if every sequence, (xn), has an accumulation point, then for every > 0, there is a finite open cover,B_0(a_0, )∪ · · · ∪B_0(an, ), ofE by open balls of radius .

Proof. Leta_0 be any point inE. IfB_0(a_0, ) =E, then the lemma is proved. Otherwise, assume that a sequence, (a_0, a_1, . . . , an), has been defined, such thatB_0(a_0, )∪· · ·∪B_0(an, ) does not coverE. Then, there is somean+1 not inB_0(a_0, )∪ · · · ∪B_0(an, ) and either

B_0(a_0, )∪ · · · ∪B_0(an+1, ) =E,

in which case the lemma is proved, or we obtain a sequence, ( a_0, a_1, . . . , an+1), such that B_0(a_0, )∪ · · · ∪B_0(an+1, ) does not coverE. If this process goes on forever, we obtain an infinite sequence, (an), such thatd(am, an)> for allm =n. Since every sequence inE has some accumulation point, the sequence, (an), has some accumulation point,a. Then, for infinitely manyn, we must haved(an, a)≤/3 and thus, for at least two distinct natural numbers,p, q, we must haved(ap, a)≤ /3 andd(aq, a)≤/3, which impliesd(ap, aq)≤ 2/3, contradicting the fact thatd(am, an)> for allm =n. Thus, there must be somen such that

B_0(a_0, )∪ · · · ∪B_0(an, ) =E. A metric space satisfying the condition of Lemma 26.34 is sometimes called precompact (or totally bounded). We now obtain the Weierstrass–Bolzano property.

Theorem 26.35. A metric space,E, is compact iff every sequence, (xn), has an accumulation point.

Proof. We already observed that the proof of Proposition 26.31 shows that for any compact space (not necessarily metric), every sequence, (xn), has an accumulation point. Conversely, letE be a metric space, and assume that every sequence, (xn), has an accumulation point. Given any open cover, (Ui)i∈I, forE, we must find a finite open subcover ofE. By Lemma 26.32, there is someδ > 0 (a Lebesgue number for (Ui)i∈I) such that, for every open ball, B_0(a, ), of radius ≤ δ, there is some open subset,Uj, such thatB_0(a, )⊆Uj. By Lemma 26.34, for everyδ > 0, there is a finite open cover,B_0(a_0, δ)∪ · · · ∪B_0(an, δ), ofE by open balls of radiusδ. But from the previous statement, every open ball,B_0(ai, δ), is contained in some open set,Uji, and thus,{Uj1, . . . , Ujn} is an open cover ofE.

Another very useful characterization of compact metric spaces is obtained in terms of Cauchy sequences. Such a characterization is quite useful in fractal geometry (and elsewhere). First, recall the definition of a Cauchy sequence and of a complete metric space.

Definition 26.26. Given a metric space, (E, d), a sequence, (xn)n∈N, inE is a Cauchy sequence if the following condition holds: for every > 0, there is somep≥ 0, such that, for allm, n≥p, thend(xm, xn)≤ .

If every Cauchy sequence in (E, d) converges we say that (E, d) is a complete metric space.

First, let us show the following proposition:

Proposition 26.36. Given a metric space,E, if a Cauchy sequence, (xn), has some accumulation point,a, thena is the limit of the sequence, (xn).

Proof. Since (xn) is a Cauchy sequence, for every > 0, there is somep≥ 0, such that, for allm, n≥p, thend(xm, xn)≤/2. Sincea is an accumulation point for (xn), for infinitely manyn, we haved(xn, a)≤/2, and thus, for at least somen≥p, we haved(xn, a)≤/2. Then, for allm≥p,

d(xm, a)≤d(xm, xn) +d(xn, a)≤, which shows thata is the limit of the sequence (xn).

Recall that a metric space is precompact (or totally bounded) if for every > 0, there is a finite open cover,B_0(a_0, )∪ · · · ∪B_0(an, ), ofE by open balls of radius . We can now prove the following theorem.

Theorem 26.37. A metric space,E, is compact iff it is precompact and complete. Proof. LetE be compact. For every > 0, the family of all open balls of radius is an open cover forE and sinceE is compact, there is a finite subcover,B_0(a_0, )∪ · · · ∪B_0(an, ), of E by open balls of radius . Thus,E is precompact. SinceE is compact, by Theorem 26.35, every sequence, (xn), has some accumulation point. Thus, every Cauchy sequence, (xn), has some accumulation point,a, and, by Proposition 26.36,a is the limit of (xn). Thus,E is complete.

Now, assume that E is precompact and complete. We prove that every sequence, (xn), has an accumulation point. By the other direction of Theorem 26.35, this shows thatE is compact. Given any sequence, (xn), we construct a Cauchy subsequence, (yn), of (xn) as follows: SinceE is precompact, letting = 1, there exists a finite cover,U1, ofE by open balls of radius 1. Thus, some open ball,B^1, in the cover,U1, contains infinitely many elements from the sequence (xn). Lety_0 be any element of (xn) inB^1. By induction, assume that a sequence of open balls, (Bio)1≤i≤m, has been defined, such that every ball,Bio, has radius^1, contains infinitely many elements from the sequence (xn) and contains someyi2i

from (xn) such that

1 d(yi, yi+1)≤ 2i, for all

i

, 0

≤

i

≤

m

−

1. Then, letting

=

1

2 m+1, becauseE is precompact, there is some finite cover,Um+1, ofE by open balls of radius and thus, of the open ballBm . Thus, some

open ball,

B

o

m

+1

o

, in the cover,Um+1, contains infinitely many elements from the sequence, (xn), and we letym+1 be any element of (xn) inBm+1. Thus, we have defined by inductiono

a sequence, (yn), which is a subsequence of, (xn), and such that

1 d(yi, yi+1)≤ 2i,

for alli. However, for allm, n≥ 1, we have

d(y n 1 1 ,m, yn)≤d(ym, ym+1) +· · · +d(yn−1, yn)≤ i=m 2i≤ 2m−1

and thus, (yn) is a Cauchy sequence SinceE is complete, the sequence, (yn), has a limit, and since it is a subsequence of (xn), the sequence, (xn), has some accumulation point.

If (E, d) is a nonempty complete metric space, every map,f :E→E, for which there is somek such that 0≤k < 1 and

d(f(x), f(y))≤kd(x, y)

for all x, y∈E, has the very important property that it has a unique fixed point, that is, there is a unique,a∈E, such thatf(a) =a. A map as above is called a contraction mapping. Furthermore, the fixed point of a contraction mapping can be computed as the limit of a fast converging sequence.

The fixed point property of contraction mappings is used to show some important theorems of analysis, such as the implicit function theorem and the existence of solutions to certain differential equations. It can also be used to show the existence of fractal sets defined in terms of iterated function systems. Since the proof is quite simple, we prove the fixed point property of contraction mappings. First, observe that a contraction mapping is (uniformly) continuous.

Proposition 26.38. If (E, d) is a nonempty complete metric space, every contraction mapping,f :E→E, has a unique fixed point. Furthermore, for everyx0∈E, defining the sequence, (xn), such thatxn+1 =f(xn), the sequence, (xn), converges to the unique fixed point off.

Proof. First, we prove thatf has at most one fixed point. Indeed, iff(a) =a andf(b) =b, since

d(a, b) =d(f(a), f(b))≤kd(a, b) and 0≤k < 1, we must haved(a, b) = 0, that is,a =b.

Next, we prove that (xn) is a Cauchy sequence. Observe that d(x_2, x_1) ≤ kd(x_1, x_0),

d(x_3, x_2) ≤ kd(x_2, x_1)≤k^2d(x_1, x_0), . .

d(xn+1, xn) ≤ kd(xn, xn−1)≤ · · · ≤knd(x_1, x_0).

Thus, we have

d(xn+p, xn) ≤ d(xn+p, xn+p−1) +d(xn+p−1, xn+p−2) +· · · +d(xn+1, xn) (kp−1 +kp−2 +· · · +k + 1)knd(x_1, x_0)≤ kn

kd(x_1, x_0).≤ 1−

We conclude that d(xn+p, xn) converges to 0 whenn goes to infinity, which shows that (xn) is a Cauchy sequence. SinceE is complete, the sequence (xn) has a limit,a. Sincef is continuous, the sequence (f(xn)) converges tof(a). Butxn+1 =f(xn) converges toa and sof(a) =a, the unique fixed point off.

Note that no matter how the starting pointx_0 of the sequence (xn) is chosen, (xn) converges to the unique fixed point off. Also, the convergence is fast, since k

n

d(xn, a)≤ 1−kd(x_1, x_0).

The Hausdorff distance between compact subsets of a metric space provides a very nice illustration of some of the theorems on complete and compact metric spaces just presented. Definition 26.27. Given a metric space, (X, d), for any subset,A⊆X, for any, ≥ 0, define the -hull ofA as the set

V (A) ={x∈X,∃a∈A|d(a, x)≤ }. Given any two nonempty bounded subsets,A, B ofX, defineD(A, B), the Hausdorff distance betweenA andB, by

D(A, B) = inf{ ≥ 0|A⊆V (B) andB⊆V (A)}.

Note that since we are considering nonempty bounded subsets, D(A, B) is well defined (i.e., not infinite). However,D is not necessarily a distance function. It is a distance function if we restrict our attention to nonempty compact subsets ofX (actually, it is also a metric on closed and bounded subsets). We letK(X) denote the set of all nonempty compact subsets ofX. The remarkable fact is thatD is a distance onK(X) and that ifX is complete or compact, then so isK(X). The following theorem is taken from Edgar [31].

Theorem 26.39. If (X, d) is a metric space, then the Hausdorff distance,D, on the set, (X), of nonempty compact subsets ofX is a distance. If (X, d) is complete, then (K(X), D)Kis complete and if (X, d) is compact, then (K(X), D) is compact.

Proof. Since (nonempty) compact sets are bounded,D(A, B) is well defined. Clearly,D is symmetric. Assume thatD(A, B) = 0. Then, for every > 0,A⊆V (B), which means that for everya∈A, there is someb∈B such thatd(a, b)≤ , and thus, thatA⊆B. Since B is closed,B =B, and we haveA⊆B. Similarly,B⊆A, and thus,A =B. Clearly, if A =B, we haveD(A, B) = 0. It remains to prove the triangle inequality. IfB⊆V 1(A) andC⊆V 2(B), then

V 2(B)⊆V 2(V 1(A)), and since

V 2(V 1(A))⊆V 1+ 2(A), we get

C⊆V 2(B)⊆V 1+ 2(A). Similarly, we can prove that

A⊆V 1+ 2(C),

and thus, the triangle inequality follows.

Next, we need to prove that if ( X, d) is complete, then (K(X), D) is also complete. First, we show that if (An) is a sequence of nonempty compact sets converging to a nonempty compact setA in the Hausdorff metric, then

A ={x∈X| there is a sequence, (xn), withxn∈An converging tox}.

Indeed, if ( xn) is a sequence withxn∈An converging tox and (An) converges toA then, for every > 0, there is somexn such thatd(xn, x)≤/2 and there is somean∈A such that d(an, xn)≤/2 and thus,d(an, x)≤ , which shows thatx∈A. SinceA is compact, it is closed, andx∈A. Conversely, since (An) converges toA, for everyx∈A, for everyn≥ 1, there is somexn∈An such thatd(xn, x)≤ 1/n and the sequence (xn) converges tox.

Now, let (An) be a Cauchy sequence inK(X). It can be proven that (An) converges to the set

A ={x∈X| there is a sequence, (xn), withxn∈An converging tox},

and thatA is nonempty and compact. To prove thatA is compact, one proves that it is totally bounded and complete. Details are given in Edgar [31].

Finally, we need to prove that if ( X, d) is compact, then (K(X), D) is compact. Since we already know that (K(X), D) is complete if (X, d) is, it is enough to prove that (K(X), D) is totally bounded if (X, d) is, which is not hard.

In view of Theorem 26.39 and Theorem 26.38, it is possible to define some nonempty compact subsets ofX in terms of fixed points of contraction maps. This can be done in terms of iterated function systems, yielding a large class of fractals. However, we will omit this topic and instead refer the reader to Edgar [31].

Finally, returning to second-countable spaces, we give another characterization of accumulation points.

Proposition 26.40. Given a second-countable topological Hausdorff space,E, a point,l, is an accumulation point of the sequence, (xn), iffl is the limit of some subsequence, (xnk), of (xn).

Proof. Clearly, ifl is the limit of some subsequence (xnk) of (xn), it is an accumulation point of (xn).

Conversely, let ( Uk)k≥0 be the sequence of open sets containingl, where eachUk belongs to a countable basis ofE, and letVk =U1∩ · · · ∩Uk. For everyk≥ 1, we can find some nk> nk−1 such thatxnk∈Vk, sincel is an accumulation point of (xn). Now, since every open set containingl contains someUk0 and sincexnk∈Uk0 for allk≥ 0, the sequence (xnk) has limitl.

Remark: Proposition 26.40 also holds for metric spaces.

In Chapter 27 we show how certain fractals can be defined by iterated function systems, using Theorem 26.39 and Theorem 26.38.

Before considering differentials, we need to look at the continuity of linear maps.

26.6 Continuous Linear and Multilinear Maps

IfE andF are normed vector spaces, we first characterize when a linear mapf :E→F is continuous.

Proposition 26.41. Given two normed vector spacesE andF, for any linear mapf :E→F, the following conditions are equivalent:

(1) The functionf is continuous at 0. (2) There is a constantk≥ 0 such that,

f(u) ≤k, for everyu∈E such that u ≤ 1.

(3) There is a constantk≥ 0 such that,

f(u) ≤k u , for everyu∈E.

(4) The functionf is continuous at every point ofE.

Proof. Assume (1). Then, for every > 0, there is someη > 0 such that, for everyu∈ E, if u ≤η, then f(u) ≤ . Pick = 1, so that there is someη > 0 such that, if u ≤ η, then f(u) ≤ 1. If u ≤ 1, then ηu ≤η u ≤η, and so, f(ηu) ≤ 1, that is,η f(u) ≤ 1,

which implies f(u) ≤η−1. Thus, (2) holds withk =η−1.

Assume that (2) holds. Ifu = 0, then by linearity,f(0) = 0, and thus f(0) ≤k 0 holds trivially for allk≥ 0. Ifu = 0, then u > 0, and since

u = 1,u

we have

f u k,u ≤ which implies that

f(u) ≤k u . Thus, (3) holds.

If (3) holds, then for allu, v∈E, we have

f(v)−f(u) = f(v−u) ≤k v−u .

If k = 0, thenf is the zero function, and continuity is obvious. Otherwise, ifk > 0, for every > 0, if v−u ≤ k, then f(v−u) ≤ , which shows continuity at everyu∈E. Finally,

it is obvious that (4) implies (1).

Among other things, Proposition 26.41 shows that a linear map is continuous iff the image of the unit (closed) ball is bounded. IfE andF are normed vector spaces, the set of all continuous linear mapsf :E→F is denoted byL(E;F).

Using Proposition 26.41, we can define a norm onL(E;F) which makes it into a normed vector space. This definition has already been given in Chapter 7 (Definition 7.7) but for the reader’s convenience, we repeat it here.

Definition 26.28. Given two normed vector spacesE andF, for every continuous linear mapf :E→F, we define the norm f off as

f = min{k≥ 0| f(x) ≤k x , for allx∈E} = max{f(x) | x ≤ 1}.

From Definition 26.28, for every continuous linear mapf∈ L(E;F), we have

f(x) ≤ f x ,

for every x∈E. It is easy to verify thatL(E;F) is a normed vector space under the norm of Definition 26.28. Furthermore, ifE, F, G, are normed vector spaces, andf :E→F and g:F→G are continuous linear maps, we have

gæf ≤ g f .

We can now show that whenE = Rn orE = Cn, with any of the norms_1,2, or , then every linear mapf :E→F is continuous.∞

Proposition 26.42. IfE = Rn orE = Cn, with any of the norms_1,2, or ∞, and F is any normed vector space, then every linear mapf :E→F is continuous. Proof. Let (e_1, . . . , en) be the standard basis of Rn (a similar proof applies to Cn). In view of Proposition 7.2, it is enough to prove the proposition for the norm

x∞ = max{|xi| | 1≤i≤n}.

We have,

f(v)−f(u) = f(v−u) = f( (vi−ui)ei) = (vi−ui)f(ei) ,

1≤i≤n 1≤i≤n

and so,

f(v)−f(u) ≤ f(e

1

≤

i

≤

n

1

≤

i

≤

n

|

vi−ui| = f(ei) v−u∞.i) max

1≤i≤n

By the argument used in Proposition 26.41 to prove that (3) implies (4), f is continuous. Actually, we proved in Theorem 7.3 that ifE is a vector space of finite dimension, then any two norms are equivalent, so that they define the same topology. This fact together with Proposition 26.42 prove the following:

Theorem 26.43. IfE is a vector space of finite dimension (over R or C), then all norms are equivalent (define the same topology). Furthermore, for any normed vector spaceF, every linear mapf :E→F is continuous.

IfE is a normed vector space of infinite dimension, a linear mapf :E→F may not be continuous. As an example, letE be the infinite vector space of all polynomials over R. Let

P(X) = max|P(x)|.

0≤x≤1

We leave as an exercise to show that this is indeed a norm. Let F = R, and letf :E→F be the map defined such that,f(P(X)) =P(3). It is clear thatf is linear. Consider the sequence of polynomialsn

Pn(X) = X .2

n

It is clear that Pn =1 , and thus, the sequencePn has the null polynomial as a limit.2

However, we haven

f(Pn(X)) =Pn(3) =3 ,2

and the sequencef(Pn(X)) diverges to +∞. Consequently, in view of Proposition 26.12 (1), f is not continuous.

We now consider the continuity of multilinear maps. We treat explicitly bilinear maps, the general case being a straightforward extension.

Proposition 26.44. Given normed vector spacesE,F andG, for any bilinear mapf :E×E→G, the following conditions are equivalent:

(1) The functionf is continuous at 0,0 . 2) There is a constantk≥ 0 such that,

f(u, v) ≤k, for allu, v∈E such that u , v ≤ 1.

3) There is a constantk≥ 0 such that,

f(u, v) ≤k u v , for allu, v∈E.

4) The function f is continuous at every point ofE×F. Proof. It is similar to that of Proposition 26.41, with a small subtlety in proving that (3) implies (4), namely that two differentη’s that are not independent are needed.

If E,F, andG, are normed vector spaces, we denote the set of all continuous bilinear mapsf :E×F→G byL2(E, F;G). Using Proposition 26.44, we can define a norm on L2(E, F;G) which makes it into a normed vector space.

Definition 26.29. Given normed vector spacesE,F, andG, for every continuous bilinear mapf :E×F→G, we define the norm f off as

f = min{k≥ 0| f(x, y) ≤k x y , for allx, y∈E}= max{f(x, y) | x , y ≤ 1}.

From Definition 26.28, for every continuous bilinear mapf∈ L2(E, F;G), we have

f(x, y) ≤ f x y ,

for allx, y∈E. It is easy to verify thatL2(E, F;G) is a normed vector space under the norm of Definition 26.29.

Given a bilinear mapf :E×F→G, for everyu∈E, we obtain a linear map denoted fu:F→G, defined such that,fu(v) =f(u, v). Furthermore, since

f(x, y) ≤ f x y ,

it is clear thatfu is continuous. We can then consider the mapÕ:E→ L(F;G), defined such that,Õ(u) =fu, for anyu∈E, or equivalently, such that,

Õ(u)(v) =f(u, v).

Actually, it is easy to show that Õ is linear and continuous, and that Õ = f . Thus,f→Õ defines a map fromL2(E, F;G) toL(E;L(F;G)). We can also go back fromL(E;L(F;G)) toL2(E, F;G). We summarize all this in the following proposition.

Proposition 26.45. LetE, F, G be three normed vector spaces. The mapf→Õ, from L2(E, F;G) toL(E;L(F;G)), defined such that, for everyf∈ L2(E, F;G),

Õ(u)(v) =f(u, v),

is an isomorphism of vector spaces, and furthermore, Õ = f .

As a corollary of Proposition 26.45, we get the following proposition which will be useful when we define second-order derivatives.

Proposition 26.46. LetE, F be normed vector spaces. The map app fromL(E;F)×E to F, defined such that, for everyf∈ L(E;F), for everyu∈E,

app(f, u) =f(u),

is a continuous bilinear map.

Remark: IfE andF are nontrivial, it can be shown that app = 1. It can also be shown that composition

æ:L(E;F)× L(F;G)→ L(E;G),

is bilinear and continuous.

The above propositions and definition generalize to arbitraryn-multilinear maps, with n≥ 2. Proposition 26.44 extends in the obvious way to anyn-multilinear mapf :E1× · · · ×En→F, but condition (3) becomes:

There is a constantk≥ 0 such that,

f(u_1, . . . , un) ≤k u1· · · un , for allu1∈E_1, . . . , un∈En.

Definition 26.29 also extends easily to

f = min{k≥ 0| f(x_1, . . . , xn) ≤k x1· · · xn , for allxi∈Ei,1≤i≤n}= max{f(x1, . . . , xn) | xn , . . . , xn ≤ 1}.

Proposition 26.45 is also easily extended, and we get an isomorphism between continuous n-multilinear maps inLn(E_1, . . . , En;F), and continuous linear maps in

L(E_1;L(E_2;. . .;L(En;F)))

An obvious extension of Proposition 26.46 also holds.

Definition 26.30. A normed vector space (E, ) over R (or C) which is a complete metric space for the distance v−u , is called a Banach space.

It can be shown that every normed vector space of finite dimension is a Banach space (is complete). It can also be shown that ifE andF are normed vector spaces, andF is a Banach space, thenL(E;F) is a Banach space. IfE, F andG are normed vector spaces, andG is a Banach space, thenL2(E, F;G) is a Banach space.

Finally, we consider normed affine spaces.

26.7 Normed Affine Spaces

For geometric applications, we will need to consider affine spaces (E, E) where the associated space of translationsE is a vector space equipped with a norm.

Definition 26.31. Given an affine space (E, E), where the space of translationsE is a vector space over R or C, we say that (E, E) is a normed affine space ifE is a normed vector space with norm .

Given a normed affine space, there is a natural metric onE itself, defined such that

d(a, b) = −→ .

Observe that this metric is invariant under translation, that is,

d(a +u, b +u) =d(a, b).

Also, for every fixeda∈E andλ > 0, if we consider the maph:E→E, defined such that,

h(x) =a +λax,

−→

thend(h(x), h(y)) =λd(x, y).

Note that the map (a, b)→−→ab fromE×E toE is continuous, and similarly for the map a→a+u fromE×E toE. In fact, the mapu→a+u is a homeomorphism fromE toEa. Of course,Rn is a normed affine space under the Euclidean metric, and it is also complete.

If an affine space E is a finite direct sum (E_1, a_1)⊕ · · · ⊕(Em, am), and eachEi is also a normed affine space with normi, we make (E_1, a1)⊕ · · · ⊕ (Em, am) into a normed affine space, by giving it the norm

(x_1, . . . , xn) = max( x1 1, . . . , xn n).

Similarly, the finite productE1× · · · ×Em is made into a normed affine space, under the same norm.

We are now ready to define the derivative (or differential) of a map between two normed affine spaces. This will lead to tangent spaces to curves and surfaces (in normed affine spaces).

26.8 Futher Readings

A thorough treatment of general topology can be found in Munkres [81, 82], Dixmier [27], Lang [68], Schwartz [89, 88], Bredon [16], and the classic, Seifert and Threlfall [91].

Chapter 27 A Detour On Fractals

27.1 Iterated Function Systems and Fractals

A pleasant application of the Hausdorff distance and of the fixed point theorem for contracting mappings is a method for defining a class of “self-similar” fractals. For this, we can use iterated function systems.

Definition 27.1. Given a metric space, (X, d), an iterated function system, for short, an ifs, is a finite sequence of functions, (f_1, . . . , fn), where eachfi:X→X is a contracting mapping. A nonempty compact subset,K, ofX is an invariant set (or attractor) for the ifs, (f_1, . . . , fn), if

K =f_1(K)∪ · · · ∪fn(K).

The major result about ifs’s is the following:

Theorem 27.1. If (X, d) is a nonempty complete metric space, then every iterated function system, (f_1, . . . , fn), has a unique invariant set,A, which is a nonempty compact subset of X. Furthermore, for every nonempty compact subset,A_0, ofX, this invariant set,A, if the limit of the sequence, (Am), whereAm+1 =f_1(Am)∪ · · · ∪fn(Am).

Proof. SinceX is complete, by Theorem 26.39, the space (K(X), D) is a complete metric space. The theorem will follow from Theorem 26.38 if we can show that the map, F :K(X)→ K(X), defined such that

F(K) =f_1(K)∪ · · · ∪fn(K),

for every nonempty compact set, K, is a contracting mapping. LetA, B be any two nonempty compact subsets ofX and consider anyη≥D(A, B). Since eachfi:X→X is a contracting mapping, there is someλi, with 0≤λi< 1, such that

d(fi(a), fi(b))≤λid(a, b),

773 for alla, b∈X. Letλ = max{λ_1, . . . , λn}. We claim that D(F(A), F(B))≤λD(A, B). For anyx∈F(A) =f_1(A)∪· · ·∪fn(A), there is someai∈Ai such thatx =fi(ai) and since η≥D(A, B), there is somebi∈B such that

d(ai, bi)≤η, and thus, d(x, fi(bi)) =d(fi(ai), fi(bi))≤λid(ai, bi)≤λη. This show that

F(A)⊆Vλη(F(B)). Similarly, we can prove that

F(B)⊆Vλη(F(A)), and since this holds for allη≥D(A, B), we proved that D(F(A), F(B))≤λD(A, B) whereλ = max{λ_1, . . . , λn}. Since 0≤λi < 1, we have 0≤λ < 1 andF is indeed a contracting mapping.

Theorem 27.1 justifies the existence of many familiar “self-similar” fractals. One of the best known fractals is the Sierpinski gasket.

Example 27.1. Consider an equilateral triangle with verticesa, b, c, and letf_1, f_2, f_3 be the dilatations of centersa, b, c and ratio 1/2. The Sierpinski gasket is the invariant set of the ifs (f_1, f_2, f_3). The dilationsf_1, f_2, f_3 can be defined explicitly as follows, assuming that a = (−1/2,0),b = (1/2,0), andc = (0,√3/2). The contractionsf_1, f_2, f_3 are specified by

x =11x− 4,2

y =1y,2

x =1x + 1,2 4

y =1y,2

and

2

2 4 . Figure 27.1: The Sierpinski gasket

We wrote a Mathematica program that iterates any finite number of affine maps on any input figure consisting of combinations of points, line segments, and polygons (with their interior points). Starting with the edges of the trianglea, b, c, after 6 iterations, we get the picture shown in Figure 27.1.

It is amusing that the same fractal is obtained no matter what the initial nonempty compact figure is. It is interesting to see what happens if we start with a solid triangle (with its interior points). The result after 6 iterations is shown in Figure 27.2. The convergence towards the Sierpinski gasket is very fast. Incidently, there are many other ways of defining the Sierpinski gasket.

A nice variation on the theme of the Sierpinski gasket is the Sierpinski dragon. Example 27.2. The Sierpinski dragon is specified by the following three contractions:

x =√31x− 4 y + 3, −4 4

y

=

√3 1y +√3

4 x− 4 4 ,

x

=

1

−4x +√3 3,4 y− 4 y = − 4 x− 4y +√3√3 1 ,4

2

2 2 . Figure 27.2: The Sierpinski gasket, version 2

The result of 7 iterations starting from the line segment (−1,0),(1,0)), is shown in Figure 27.3. This curve converges to the boundary of the Sierpinski gasket.

A different kind of fractal is the Heighway dragon.

Example 27.3. The Heighway dragon is specified by the following two contractions:

x

=

1x 1

− 2y,2

y =1x + 1y,2 2

x = 11x− 2y, −2

y

=

1x 1

− 2y + 1.2

It can be shown that for any number of iterations, the polygon does not cross itself. This means that no edge is traversed twice and that if a point is traversed twice, then this point is the endpoint of some edge. The result of 13 iterations, starting with the line segment ((0,0),(0,1)), is shown in Figure 27.4.

The Heighway dragon turns out to fill a closed and bounded set. It can also be shown that the plane can be tiled with copies of the Heighway dragon.

Another well known example is the Koch curve. Figure 27.3: The Sierpinski dragon

Figure 27.4: The Heighway dragon Figure 27.5: The Koch curve

Example 27.4. The Koch curve is specified by the following four contractions:

x =21x− 3,3

y =1y,3

x

=

1x √3 1

− 6 y− 6,6

y

=

√3 + 1y +√3

6

x_6 ,

6

1

x

+

√

3

x = + 1,6 6 y

6

√

3 + 1y +√3

y = − 6 x_6 ,

6

x =1x + 2,3 3

y =1y.3

The Koch curve is an example of a continuous curve which is nowhere differentiable (because it “wiggles” too much). It is a curve of infinite length. The result of 6 iterations, starting with the line segment ((−1,0),(1,0)), is shown in Figure 27.5.

The curve obtained by putting three Kock curves together on the sides of an equilateral triangle is known as the snowflake curve (for obvious reasons, see below!). Figure 27.6: The snowflake curve

Example 27.5. The snowflake curve obtained after 5 iterations is shown in Figure 27.6. The snowflake curve is an example of a closed curve of infinite length bounding a finite area.

We conclude with another famous example, a variant of the Hilbert curve. Example 27.6. This version of the Hilbert curve is defined by the following four contractions:

x

=

1x 1 − 2,2

y =1y + 1,2

x =1x + 1,2 2

y =1y + 1,2

1 x = −2y + 1,

y =1x + 1,2 2

x =1y− 1,2

y

=

1

−

2

x + 1.

2

Figure 27.7: A Hilbert curve

This continuous curve is a space-filling curve, in the sense that its image is the entire unit square. The result of 6 iterations, starting with the two lines segments ((−1,0),(0,1)) and ((0,1),(1,0)), is shown in Figure 27.7.

For more on iterated function systems and fractals, we recommend Edgar [31].

Chapter 28 Differential Calculus

28.1 Directional Derivatives, Total Derivatives

This chapter contains a review of basic notions of differential calculus. First, we review the definition of the derivative of a functionf : R→ R. Next, we define directional derivatives and the total derivative of a functionf :E→F between normed affine spaces. Basic properties of derivatives are shown, including the chain rule. We show how derivatives are represented by Jacobian matrices. The mean value theorem is stated, as well as the implicit function theorem and the inverse function theorem. Diffeomorphisms and local diffeomorphisms are defined. Tangent spaces are defined. Higher-order derivatives are defined, as well as the Hessian. Schwarz’s lemma (about the commutativity of partials) is stated. Several versions of Taylor’s formula are stated, and a famous formula due to Fa`a di Bruno’s is given.

We first review the notion of the derivative of a real-valued function whose domain is an open subset of R.

Let f :A→ R, whereA is a nonempty open subset of R, and consider anya∈ A. The main idea behind the concept of the derivative off ata, denoted byf (a), is that locally arounda (that is, in some small open setU⊆A containinga), the functionf is approximated linearly by the map

x→f(a) +f (a)(x−a).

Part of the difficulty in extending this idea to more complex spaces is to give an adequate notion of linear approximation. Of course, we will use linear maps! Let us now review the formal definition of the derivative of a real-valued function.

Definition 28.1. LetA be any nonempty open subset of R, and leta∈A. For any function f :A→ R, the derivative off ata∈A is the limit (if it exists)

h

lim

f(a +h)

U

h

−f(a),

→0, h∈

781 whereU ={h∈ R|a +h∈ A, h = 0}. This limit is denoted byf (a), or Df(a), or df(a).

If

f

(

a

) exists for every

a

∈

A

, we say that

f

is differentiable on

dx

A. In this case, the map a→f (a) is denoted byf , or Df, or df.dx

Note that since A is assumed to be open,A− {a} is also open, and since the function h→a +h is continuous andU is the inverse image ofA− {a} under this function,U is indeed open and the definition makes sense.

We can also definef (a) as follows: there is some function , such that,

f(a +h) =f(a) +f (a)·h + (h)h, whenevera +h∈A, where (h) is defined for allh such thata +h∈A, and

h

limU (h) = 0.

→0, h∈

Remark: We can also define the notion of derivative off ata on the left, and derivative off ata on the right. For example, we say that the derivative off ata on the left is the limitf (a−) (if it exists)

h

lim

f(a +h)

U

h

−f(a),

→0, h∈

whereU ={h∈ R|a +h∈A, h < 0}.

If a function f as in Definition 28.1 has a derivativef (a) ata, then it is continuous at a. Iff is differentiable onA, thenf is continuous onA. The composition of differentiable functions is differentiable.

Remark: A functionf has a derivativef (a) ata iff the derivative off on the left ata and the derivative off on the right ata exist, and if they are equal. Also, if the derivative off on the left ata exists, thenf is continuous on the left ata (and similarly on the right).

We would like to extend the notion of derivative to functions f :A→F, whereE andF are normed affine spaces, andA is some nonempty open subset ofE. The first difficulty is to make sense of the quotient

f(a +h)−f(a).h

If E andF are normed affine spaces, it will be notationally convenient to assume that the vector space associated withE is denoted byE, and that the vector space associated withF is denoted asF.

Since F is a normed affine space, making sense off(a+h)−f(a) is easy: we can define this as−−−−−−−−−→), the unique vector translatingf(a) tof(a +h). We should note however, that this quantity is a vector and not a point. Nevertheless, in defining derivatives, it is notationally more pleasant to denote−−−−−−−−−→) byf(a +h)−f(a). Thus, in the rest of this chapter, the vector−→ will be denoted byb−a. But now, how do we define the quotient by a vector? Well, we don’t!

A first possibility is to consider the directional derivative with respect to a vectoru = 0 inE. We can consider the vectorf(a +tu)−f(a), wheret∈ R (ort∈ C). Now,

f(a +tu)−f(a)

t

makes sense. The idea is that in E, the points of the forma+tu fort in some small interval [−, + ] in R (or C) form a line segment [r, s] inA containinga, and that the image of this line segment defines a small curve segment onf(A). This curve segment is defined by the mapt→f(a +tu), from [r, s] toF, and the directional derivative Duf(a) defines the direction of the tangent line ata to this curve. This leads us to the following definition.

Definition 28.2. LetE andF be two normed affine spaces, letA be a nonempty open subset ofE, and letf :A→F be any function. For anya∈A, for anyu = 0 inE, the directional derivative off ata w.r.t. the vectoru, denoted by Duf(a), is the limit (if it exists)

t

lim

f(a +tu)

U

t

−f(a),

→0, t∈

whereU ={t∈ R|a +tu∈A, t = 0} (orU ={t∈ C|a +tu∈A, t = 0}).

Since the map t→a +tu is continuous, and sinceA− {a} is open, the inverse imageU ofA− {a} under the above map is open, and the definition of the limit in Definition 28.2 makes sense.

Remark: Since the notion of limit is purely topological, the existence and value of a directional derivative is independent of the choice of norms inE andF, as long as they are equivalent norms.

The directional derivative is sometimes called the Gˆateaux derivative.

In the special case where E = R andF = R, and we letu = 1 (i.e., the real number 1, viewed as a vector), it is immediately verified that D_1f(a) =f (a), in the sense of Definition 28.1. WhenE = R (orE = C) andF is any normed vector space, the derivative D_1f(a), also denoted byf (a), provides a suitable generalization of the notion of derivative.

However, when E has dimension≥ 2, directional derivatives present a serious problem, which is that their definition is not sufficiently uniform. Indeed, there is no reason to believe that the directional derivatives w.r.t. all nonnull vectorsu share something in common. As a consequence, a function can have all directional derivatives ata, and yet not be continuous ata. Two functions may have all directional derivatives in some open sets, and yet their composition may not. Thus, we introduce a more uniform notion.

Definition 28.3. LetE andF be two normed affine spaces, letA be a nonempty open subset ofE, and letf :A→F be any function. For anya∈A, we say thatf is differentiable at a∈A if there is a linear continuous mapL:E→F and a function , such that

f(a +h) =f(a) +L(h) + (h) h

for everya +h∈A, where (h) is defined for everyh such thata +h∈A and limU (h) = 0,h→0, h∈

where U ={h∈E|a +h∈A, h = 0}. The linear mapL is denoted by Df(a), or Dfa, or df(a), ordfa, orf (a), and it is called the Fr´echet derivative, or derivative, or total derivative, or total differential, or differential, off ata.

Since the maph→a+h fromE toE is continuous, and sinceA is open inE, the inverse imageU ofA− {a} under the above map is open inE, and it makes sense to say that limU (h) = 0.h→0, h∈

Note that for everyh∈U, sinceh = 0, (h) is uniquely determined since (h) =f(a +h)−f(a)−L(h),h

and that the value (0) plays absolutely no role in this definition. The condition forf to be differentiable ata amounts to the fact that

lim f(a +h)−f(a)−L(h)= 0

h→0 h

ash = 0 approaches 0, whena +h∈A. However, it does no harm to assume that (0) = 0, and we will assume this from now on.

Again, we note that the derivative Df(a) off ata provides an affine approximation of f, locally arounda.

Remark: Since the notion of limit is purely topological, the existence and value of a derivative is independent of the choice of norms inE andF, as long as they are equivalent norms.

Note that the continuous linear map L is unique, if it exists. In fact, the next proposition implies this as a corollary. The following proposition shows that our new definition is consistent with the definition of the directional derivative.

Proposition 28.1. LetE andF be two normed affine spaces, letA be a nonempty open subset ofE, and letf :A→F be any function. For anya∈A, if Df(a) is defined, then f is continuous ata andf has a directional derivative Duf(a) for everyu = 0 inE, and furthermore,

Duf(a) = Df(a)(u). Proof. Ifh = 0 approaches 0, sinceL is continuous, (h) h approaches 0, and thus,f is continuous ata. For anyu = 0 inE, for|t| ∈ R small enough (wheret∈ R ort∈ C), we havea +tu∈A, and lettingh =tu, we have

f(a +tu) =f(a) +tL(u) + (tu)|t|u , and fort = 0, f(a +tu)−f(a) =L(u) +|t| (tu) u ,t

and the limit whent = 0 approaches 0 is indeed Duf(a).

The uniqueness ofL follows from Proposition 28.1. Also, whenE is of finite dimension, it is easily shown that every linear map is continuous, and this assumption is then redundant.

It is important to note that the derivative D f(a) off ata is a continuous linear map from the vector spaceE to the vector spaceF, and not a function from the affine spaceE to the affine spaceF.

As an example, consider the map, f : Mn(R)→ Mn(R), given by

f(A) =A A−I,

where Mn(R) is equipped with any matrix norm, since they are all equivalent; for example, pick the Frobenius norm, A F = tr(A A). We claim that

Df(A)(H) =A H +H A, for allA andH in Mn(R). We have

f (A +H)−f(A)− (A H +H A) = (A +H) (A +H)−I− (A A−I)−A H−H A =A A +A H +H A +H H−A A−A H−H A =H H.

It follows that f(A +H)−f(A)− (A H +H A)=H H(H) = H ,

H

and since our norm is the Frobenius norm,

(H) = H H H H = H = H ,H ≤ H

so

lim (H) = 0,

H→0

and we conclude that

Df(A)(H) =A H +H A.

If D f(a) exists for everya∈A, we get a map Df :A→ L(E;F),

called the derivative off onA, and also denoted bydf. Recall thatL(E;F) denotes the vector space of all continuous maps fromE toF.

When E is of finite dimensionn, for any frame (a_0,(u_1, . . . , un)) ofE, where (u_1, . . . , un) is a basis ofE, we can define the directional derivatives with respect to the vectors in the basis (u_1, . . . , un) (actually, we can also do it for an infinite frame). This way, we obtain the definition of partial derivatives, as follows.

Definition 28.4. For any two normed affine spacesE andF, ifE is of finite dimension n, for every frame (a_0,(u_1, . . . , un)) forE, for everya∈E, for every functionf :E→F, the directional derivatives Dujf(a) (if they exist) are called the partial derivatives off with respect to the frame (a_0,(u_1, . . . , un)). The partial derivative Dujf(a) is also denoted by

∂jf(a), or ∂f (a).∂xj

The notation ∂f (a) for a partial derivative, although customary and going back to∂xj

Leibniz, is a “logical obscenity.” Indeed, the variable xj really has nothing to do with the formal definition. This is just another of these situations where tradition is just too hard to overthrow!

We now consider a number of standard results about derivatives.

Proposition 28.2. Given two normed affine spacesE andF, iff :E→F is a constant function, then Df(a) = 0, for everya∈E. Iff :E→F is a continuous affine map, then Df(a) =f, for everya∈E, the linear map associated withf.

Proof. Straightforward.

Proposition 28.3. Given a normed affine spaceE and a normed vector spaceF, for any two functionsf, g:E→F, for everya∈E, if Df(a) and Dg(a) exist, then D(f +g)(a) and D(λf)(a) exist, and

D(f +g)(a) = Df(a) + Dg(a),

D(λf)(a) =λDf(a).

Proof. Straightforward.

Proposition 28.4. Given three normed vector spacesE_1,E_2, andF, for any continuous bilinear map

f :E1×E2→F, for every (a, b)∈E1×E_2, Df(a, b) exists, and for everyu∈E_1 and v∈E2,

Df(a, b)(u, v) =f(u, b) +f(a, v).

Proof. Straightforward.

We now state the very useful chain rule.

Theorem 28.5. Given three normed affine spacesE,F, andG, letA be an open set in E, and letB an open set inF. For any functionsf :A→F andg:B→G, such that f(A)⊆B, for anya∈A, if Df(a) exists and Dg(f(a)) exists, then D(gæf)(a) exists, and

D(gæf)(a) = Dg(f(a))æ Df(a). Proof. It is not difficult, but more involved than the previous two.

Theorem 28.5 has many interesting consequences. We mention two corollaries.

Proposition 28.6. Given three normed affine spacesE,F, andG, for any open subsetA in E, for anya∈A, letf :A→F such that Df(a) exists, and letg:F→G be a continuous affine map. Then, D(gæf)(a) exists, and

D( gæf)(a) =gæ Df(a),

whereg is the linear map associated with the affine mapg.

Proposition 28.7. Given two normed affine spacesE andF, letA be some open subset in E, letB be some open subset inF, letf :A→ B be a bijection fromA toB, and assume that Df exists onA and that Df−1 exists onB. Then, for everya∈A,

Df−1(f(a)) = (Df(a))−1.

Proposition 28.7 has the remarkable consequence that the two vector spaces E andF have the same dimension. In other words, a local property, the existence of a bijectionf between an open setA ofE and an open setB ofF, such thatf is differentiable onA and f−1 is differentiable onB, implies a global property, that the two vector spacesE andF have the same dimension.

We now consider the situation where the normed affine spaceF is a finite direct sum F = (F_1, b_1)⊕ · · · ⊕ (Fm, bm).

Proposition 28.8. Given normed affine spacesE andF = (F_1, b_1)⊕ · · · ⊕ (Fm, bm), given any open subsetA ofE, for anya∈A, for any functionf :A→F, lettingf = (f_1, . . . , fm), Df(a) exists iff every Dfi(a) exists, and

Df(a) =in1æ Df_1(a) +· · · +inmæ Dfm(a).

Proof. Observe thatf(a +h)−f(a) = (f(a +h)−b)− (f(a)−b), whereb = (b_1, . . . , bm), and thus, as far as dealing with derivatives, Df(a) is equal to Dfb(a), wherefb:E→F is defined such thatfb(x) =f(x)− b, for everyx∈E. Thus, we can work with the vector space F instead of the affine spaceF. The proposition is then a simple application of Theorem 28.5.

In the special case where F is a normed affine space of finite dimensionm, for any frame (b_0,(v_1, . . . , vm)) ofF, where (v_1, . . . , vm) is a basis ofF, every pointx∈F can be expressed uniquely as

x =b_0 +x_1v_1 +· · · +xmvm,

where ( x_1, . . . , xm)∈Km, the coordinates ofx in the frame (b_0,(v_1, . . . , vm)) (whereK = R or K = C). Thus, letting Fi be the standard normed affine space K with its natural structure, we note thatF is isomorphic to the direct sumF = (K,0)⊕ · · · ⊕ (K,0). Then, every functionf :E→F is represented bym functions (f_1, . . . , fm), wherefi:E→K (whereK = R orK = C), and

f(x) =b_0 +f_1(x)v_1 +· · · +fm(x)vm,

for everyx∈E. The following proposition is an immediate corollary of Proposition 28.8.

Proposition 28.9. For any two normed affine spacesE andF, ifF is of finite dimension m, for any frame (b_0,(v_1, . . . , vm)) ofF, where (v_1, . . . , vm) is a basis ofF, for everya∈E, a functionf :E→F is differentiable ata iff eachfi is differentiable ata, and

Df(a)(u) = Df_1(a)(u)v_1 +· · · + Dfm(a)(u)vm, for everyu∈E.

We now consider the situation where E is a finite direct sum. Given a normed affine spaceE = (E_1, a_1)⊕ · · · ⊕ (En, an) and a normed affine spaceF, given any open subsetA ofE, for anyc = (c1, . . . , cn)∈A, we define the continuous functionsicj:Ej→E, such that

icj(x) = (c_1, . . . , cj−1, x, cj+1, . . . , cn).

For any function f :A→F, we have functionsfæicj:Ej→F, defined on (icj)−1(A), which containscj. If D(fæicj)(cj) exists, we call it the partial derivative off w.r.t. itsjth argument, atc. We also denote this derivative by Djf(c). Note that Djf(c)∈ L(Ej;F).

This notion is a generalization of the notion defined in Definition 28.4. In fact, when E is of dimensionn, and a frame (a_0,(u_1, . . . , un)) has been chosen, we can writeE = (E_1, a_1)⊕ · · · ⊕(En, an), for some obvious (Ej, aj) (as explained just after Proposition 28.8), and then

Djf(c)(λuj) =λ∂jf(c),

and the two notions are consistent.

The definition oficj and of Djf(c) also makes sense for a finite productE1× · · · ×En of affine spacesEi. We will use freely the notation∂jf(c) instead of Djf(c).

The notion ∂jf(c) introduced in Definition 28.4 is really that of the vector derivative, whereas Djf(c) is the corresponding linear map. Although perhaps confusing, we identify the two notions. The following proposition holds.

Proposition 28.10. Given a normed affine spaceE = (E_1, a_1)⊕ · · · ⊕ (En, an), and a normed affine spaceF, given any open subsetA ofE, for any functionf :A→F, for every c∈A, if Df(c) exists, then each Djf(c) exists, and

Df(c)(u_1, . . . , un) = D_1f(c)(u_1) +· · · + Dnf(c)(un),

for everyui∈Ei, 1≤i≤n. The same result holds for the finite productE1× · · · ×En.

Proof. Since everyc∈ E can be written asc =a +c−a, wherea = (a_1, . . . , an), defining fa:E→F such that,fa(u) =f(a +u), for everyu∈E, clearly, Df(c) = Dfa(c−a), and thus, we can work with the functionfa whose domain is the vector spaceE. The proposition is then a simple application of Theorem 28.5.

28.2 Jacobian Matrices

If both E andF are of finite dimension, for any frame (a_0,(u_1, . . . , un)) ofE and any frame (b_0,(v_1, . . . , vm)) ofF, every functionf :E→F is determined bym functionsfi:E→ R (orfi:E→ C), where

f(x) =b_0 +f_1(x)v_1 +· · · +fm(x)vm, for everyx∈E. From Proposition 28.1, we have

Df(a)(uj) = Dujf(a) =∂jf(a),

and from Proposition 28.9, we have

Df(a)(uj) = Df_1(a)(uj)v_1 +· · · + Dfi(a)(uj)vi +· · · + Dfm(a)(uj)vm,

that is, Df(a)(uj) =∂jf_1(a)v_1 +· · · +∂jfi(a)vi +· · · +∂jfm(a)vm.

Since the j-th column of them×n-matrix representing Df(a) w.r.t. the bases (u_1, . . . , un) and (v_1, . . . , vm) is equal to the components of the vector Df(a)(uj) over the basis (v_1, . . . ,vm), the linear map Df(a) is determined by them×n-matrixJ(f)(a) = (∂jfi(a)), (orJ(f)(a) =

(∂fi (a))):∂xj

ë∂1f_1(a) ∂2f_1(a) . . . ∂nf_1(a)ö

ì

÷

ì. .

∂

1

f

2

(

a

)

∂

2

f

2

(

a

)

. . . ∂

n

f

2

( a )

÷

J(f)(a) =ì .... ÷ í∂1fm(a) ∂2fm(a) . . . ∂nfm(a)ø

or ë ∂f1(a) ∂f1(a) . . . ∂f^1 (a)ö

ì

÷

ì ÷

ì

∂x

1

∂x

2

∂x n ÷

ì ∂f2(a) ∂f2(a) . . . ∂f^2 (a)÷

J

(

f

)(

a

) =

ì ∂x_1 ∂x2 ... n ÷

ì∂x ÷ ì

÷

í

.

.

.

÷ ì ÷

ì∂fm(a) ∂fm(a) . . . ∂fm(a)ø

∂x_1 ∂x2 ∂xn

This matrix is called the Jacobian matrix of Df ata. Whenm =n, the determinant, det(J(f)(a)), ofJ(f)(a) is called the Jacobian of Df(a). From a previous remark, we know that this determinant in fact only depends on Df(a), and not on specific bases. However, partial derivatives give a means for computing it.

WhenE = Rn andF = Rm, for any functionf : Rn Rm, it is easy to compute the partial derivatives∂fi(a). We simply treat the functionf→ n R as a function of itsj-th∂x i: R

j →

argument, leaving the others fixed, and compute the derivative as in Definition 28.1, that is, the usual derivative.

Example 28.1. For example, consider the functionf : R^2 R^2, defined such that→

f(r, θ) = (rcos(θ), rsin(θ)).

Then, we have J

(

f

)(

r, θ

) =

cos(θ) −rsin(θ) sin(θ) rcos(θ)

and the Jacobian (determinant) has value det(J(f)(r, θ)) =r.

In the case where E = R (orE = C), for any functionf : R→F (orf : C→F), the Jacobian matrix of Df(a) is a column vector. In fact, this column vector is just D_1f(a). Then, for everyλ∈ R (orλ∈ C),

Df(a)(λ) =λD_1f(a).

This case is sufficiently important to warrant a definition.

Definition 28.5. Given a functionf : R→F (orf : C→F), whereF is a normed affine space, the vector

Df(a)(1) = D_1f(a)

is called the vector derivative or velocity vector (in the real case) ata. We usually identify Df(a) with its Jacobian matrix D_1f(a), which is the column vector corresponding to D_1f(a). By abuse of notation, we also let Df(a) denote the vector Df(a)(1) = D_1f(a).

When E = R, the physical interpretation is thatf defines a (parametric) curve that is the trajectory of some particle moving in Rm as a function of time, and the vector D_1f(a) is the velocity of the moving particlef(t) att =a.

It is often useful to consider functions f : [a, b]→F from a closed interval [a, b]⊆ R to a normed affine spaceF, and its derivative Df(a) on [a, b], even though [a, b] is not open. In this case, as in the case of a real-valued function, we define the right derivative D_1f(a+) at a, and the left derivative D_1f(b−) atb, and we assume their existence.

Example 28.2.

1. WhenE = [0,1], andF = R^3, a functionf : [0,1]→ R^3 defines a (parametric) curve in R^3. Lettingf = (f_1, f_2, f_3), its Jacobian matrix ata

ö

ì

∈

R is

ë∂f1 (a)÷

ì∂t ÷

J

(

f

)(

a

) =

ì ∂f_2 ÷

ì (a)÷

ì ∂t ÷

ì ÷

í∂f3 (a)ø

∂t

2. WhenE = R^2, andF = R^3, a functionÕ: R^2 R^3 defines a parametric surface.→

LettingÕ = (f, g, h), its Jacobian matrix ata∈ R is

ë ∂f(a) ∂f(a)ö

ì∂u ∂v÷

ì ∂g(a) ∂g÷

(

a

)

÷

J

(

Õ

)(

a

) =

ì ÷

ì

ì∂u ∂v÷

ì ÷

í∂h(a) ∂h(a)ø

∂u ∂v

3. When

E

=

R

3, andF = R, for a functionf : R^3 R, the Jacobian matrix ata∈ R^3 →is

J(f)(a) = ∂f(a) ∂f(a) ∂f(a) .∂x ∂y ∂z

More generally, whenf : Rn R, the Jacobian matrix ata∈ Rn is the row vector→

J

(

f

)(

a

) =

∂f

∂f

(

a

)

.

∂x

1

(a)· · · ∂xn

Its transpose is a column vector called the gradient off ata, denoted by gradf(a) or f(a). Then, given anyv∈ Rn, note that

Df(a)(v) = ∂f (a)v_1 +· · · + ∂f (a)vn = gradf(a)·v,∂x1 ∂xn

the scalar product of gradf(a) andv.

When E,F, andG have finite dimensions, and (a_0,(u_1, . . . , up)) is an affine frame forE, (b_0,(v_1, . . . , vn)) is an affine frame forF, and (c_0,(w_1, . . . , wm)) is an affine frame forG, ifA is an open subset ofE,B is an open subset ofF, for any functionsf :A→F andg:B→G, such thatf(A)⊆B, for anya∈A, lettingb =f(a), andh =gæf, if Df(a) exists and Dg(b) exists, by Theorem 28.5, the Jacobian matrixJ(h)(a) =J(gæf)(a) w.r.t. the bases (u_1, . . . , up) and (w_1, . . . , wm) is the product of the Jacobian matricesJ(g)(b) w.r.t. the bases (v_1, . . . , vn) and (w_1, . . . , wm), andJ(f)(a) w.r.t. the bases (u_1, . . . , up) and (v_1, . . . , vn):

ë∂1g_1(b) ∂2g_1(b) . . . ∂ng_1(b)öë∂1f_1(a) ∂2f_1(a) . . . ∂pf_1(a)ö ì∂1g_2(b) ∂2g_2(b) . . . ∂ng_2(b)÷ì

÷ì ÷ ì

∂

1

f

2

(

a

)

∂

2

f

2

(

a

)

. . . ∂

p

f

2

( a )

÷

J (h)(a) =ì .... ÷ì . . .... ÷ í . . øí ø ∂1gm(b) ∂2gm(b) . . . ∂ngm(b) ∂1fn(a) ∂2fn(a) . . . ∂pfn(a)

or

ë ∂g1(b) ∂g1(b) . . . ∂g^1(b)öë∂f1(a) ∂f1(a) . . . ∂f^1 (a)ö

ì ∂y_1 ∂y2 ∂yn ÷ì

÷ì ÷ ì

∂x

1

∂x

2

∂x p ÷

ì ∂g2(b) ∂g2(b) . . . ∂g^2(b)÷ì∂f2(a) ∂f2(a) . . . ∂f^2 (a)÷

ì ÷ì ÷

J(h)(a) =ì ∂y_1 ∂y2 ... n ÷ì∂x_1 ∂x_2 ∂xp ÷.

ì. . . ÷ì. . .... ÷ ì ÷ì ÷

ì∂gm(b) ∂gm(b) . . . ∂gm (b)øí∂fn(a) ∂fn(a) . . . ∂fn(a)ø ∂y_1 ∂y2 ∂yn ∂x_1 ∂x2 ∂xp

ì∂y ÷ì ÷

÷ì ÷

í

Thus, we have the familiar formula

∂h

k=n

i(a) =∂gi (b)∂fk(a).∂xj k=1 ∂yk ∂xj

Given two normed affine spacesE andF of finite dimension, given an open subsetA of E, if a functionf :A→F is differentiable ata∈A, then its Jacobian matrix is well defined.

One should be warned that the converse is false. There are functions such that all the partial derivatives exist at somea∈A, but yet, the function is not differentiable ata, and not even continuous ata. For example, consider the functionf : R^2 R, defined such thatf(0,0) = 0, and→

x

2

f(x, y) =y if (x, y) = (0,0).x4 +y2

For anyu = 0, lettingu = h , we havek

f(0 +tu)−f(0)= h^2k t^2h^4 +k2,t

so that 2 ifk = 0Duf(0,0) =h

k

0 ifk = 0.

Thus, Duf(0,0) exists for allu = 0. On the other hand, if Df(0,0) existed, it would be a linear map Df(0,0): R^2 R represented by a row matrix (α β), and we would have→

Duf(0,0) = Df(0,0)(u) =αh +βk, but the explicit formula for Duf(0,0) is not linear. As a matter of fact, the functionf is not continuous at (0,0). For example, on the parabola y =x^2,f(x, y) = 1, and when we approach the origin on this parabola, the limit is 1, when2 2in fact,f(0,0) = 0.

However, there are sufficient conditions on the partial derivatives for Df(a) to exist, namely, continuity of the partial derivatives.

If f is differentiable onA, thenf defines a function Df :A→ L(E;F). It turns out that the continuity of the partial derivatives onA is a necessary and sufficient condition for Df to exist and to be continuous onA.

Iff : [a, b]→ R is a function which is continuous on [a, b] and differentiable on ]a, b], then there is somec witha < c < b such that

f(b)−f(a) = (b−a)f (c).

This result is known as the mean value theorem and is a generalization of Rolle’s theorem, which corresponds to the case wheref(a) =f(b).

Unfortunately, the mean value theorem fails for vector-valued functions. For example, the functionf : [0,2π]→ R^2 given by

f(t) = (cost,sint)

is such thatf(2π)−f(0) = (0,0), yet its derivativef (t) = (−sint,cost) does not vanish in ]0,2π[.

A suitable generalization of the mean value theorem to vector-valued functions is possible if we consider an inequality (an upper bound) instead of an equality. This generalized version of the mean value theorem plays an important role in the proof of several major results of differential calculus.

If E is an affine space (over R or C), given any two pointsa, b∈E, the closed segment [a, b] is the set of all pointsa+λ(b−a), where 0≤λ≤ 1,λ∈ R, and the open segment ]a, b[ is the set of all pointsa +λ(b−a), where 0< λ < 1,λ∈ R.

Lemma 28.11. LetE andF be two normed affine spaces, letA be an open subset ofE, and letf :A→F be a continuous function onA. Given anya∈A and anyh = 0 inE, if the closed segment [a, a +h] is contained inA, iff :A→F is differentiable at every point of the open segment ]a, a +h[, and

sup Df(x) ≤M,

x∈]a,a+h[

for someM≥ 0, then f(a +h)−f(a) ≤M h . As a corollary, ifL:E→F is a continuous linear map, then

f(a +h)−f(a)−L(h) ≤M h ,

whereM = supx∈]a,a+h[ Df(x)−L .

The above lemma is sometimes called the “mean value theorem.” Lemma 28.11 can be used to show the following important result.

Theorem 28.12. Given two normed affine spacesE andF, whereE is of finite dimension n, and where (a_0,(u_1, . . . , un)) is a frame ofE, given any open subsetA ofE, given any functionf :A→F, the derivative Df :A→ L(E;F) is defined and continuous onA iff

every partial derivative∂jf (or ∂f ) is defined and continuous onA, for allj, 1≤j≤n.∂xj

As a corollary, ifF is of finite dimensionm, and (b_0,(v_1, . . . , vm)) is a frame ofF, the derivative Df :A→ L(E;F) is defined and continuous onA iff every partial derivative∂jfi (or ∂fi) is defined and continuous onA, for alli, j, 1≤i≤m, 1≤j≤n.∂xj

Theorem 28.12 gives a necessary and sufficient condition for the existence and continuity of the derivative of a function on an open set. It should be noted that a more general version of Theorem 28.12 holds, assuming thatE = (E_1, a_1)⊕ · · · ⊕(En, an), orE =E1× · · · ×En, and using the more general partial derivatives Djf introduced before Proposition 28.10.

Definition 28.6. Given two normed affine spacesE andF, and an open subsetA ofE, we say that a functionf :A→ F is of classC^0 onA or aC^0-function onA iff is continuous onA. We say thatf :A→F is of classC^1 onA or aC^1-function onA if Df exists and is continuous onA.

Since the existence of the derivative on an open set implies continuity, a C^1-function is of course aC^0-function. Theorem 28.12 gives a necessary and sufficient condition for a functionf to be aC^1-function (whenE is of finite dimension). It is easy to show that the composition ofC^1-functions (on appropriate open sets) is aC^1-function.

28.3 The Implicit and The Inverse Function Theorems

Given three normed affine spacesE, F, andG, given a functionf :E×F→G, given any c∈G, it may happen that the equation

f (x, y) =c has the property that, for some open setsA⊆E, andB⊆F, there is a functiong:A→B, such that

f(x, g(x)) =c,

for all x∈A. Such a situation is usually very rare, but if some solution (a, b)∈E×F such thatf(a, b) =c is known, under certain conditions, for some small open setsA⊆E containinga andB⊆F containingb, the existence of a uniqueg:A→B, such that

f(x, g(x)) =c,

for all x∈ A, can be shown. Under certain conditions, it can also be shown thatg is continuous, and differentiable. Such a theorem, known as the implicit function theorem, can be shown. We state a version of this result below, following Schwartz [90]. The proof (see Schwartz [90]) is fairly involved, and uses a fixed-point theorem for contracting mappings in complete metric spaces. Other proofs can be found in Lang [67] and Cartan [18].

Theorem 28.13. LetE, F, andG, be normed affine spaces, let & be an open subset of E×F, letf : &→G be a function defined on &, let (a, b)∈ &, letc∈G, and assume that f(a, b) =c . If the following assumptions hold

(1) The functionf : &→G is continuous on &;

(2) F is a complete normed affine space (and so isG);

(3) ∂f(x, y) exists for every (x, y)∈ &, and∂f : &→ L(F;G) is continuous;∂y ∂y

(4) ∂f(a, b) is a bijection ofL(F;G), and ∂f(a, b) −1 (G;F);∂y ∂y ∈ L

then the following properties hold:

(a) There exist some open subset A⊆ E containing a and some open subset B⊆ F containingb, such thatA×B⊆ &, and for everyx∈A, the equationf(x, y) =c has a single solutiony =g(x), and thus, there is a unique functiong:A→B such that f(x, g(x)) =c, for allx∈A;

(b) The functiong:A→B is continuous. If we also assume that

(5) The derivative Df(a, b) exists; then

(c) The derivative Dg(a) exists, and

D

g

(

a

) =

∂f

− ∂y(a, b) −1 ∂f(a, b);æ ∂x and if in addition

(6) ∂f : &→ L(E;G) is also continuous (and thus, in view of (3),f isC^1 on &);∂x

then

(d) The derivative Dg:A→ L(E;F) is continuous, and

∂f

(

x, g

(

x

))

−

1

Dg(x) =∂f(x, g(x)),− ∂y æ ∂x

for allx∈A.

The implicit function theorem plays an important role in the calculus of variations. We now consider another very important notion, that of a (local) diffeomorphism.

Definition 28.7. Given two topological spacesE andF, and an open subsetA ofE, we say that a functionf :A→ F is a local homeomorphism fromA toF if for everya∈ A, there is an open setU⊆A containinga and an open setV containingf(a) such thatf is a homeomorphism fromU toV =f(U). IfB is an open subset ofF, we say thatf :A→ F is a (global) homeomorphism fromA toB iff is a homeomorphism fromA toB =f(A). IfE andF are normed affine spaces, we say thatf :A→F is a local diffeomorphism from A toF if for everya∈A, there is an open setU⊆A containinga and an open setV containingf(a) such thatf is a bijection fromU toV ,f is aC^1-function onU, andf−1 is aC^1-function onV =f(U). We say thatf :A→F is a (global) diffeomorphism fromA toB iff is a homeomorphism fromA toB =f(A),f is aC^1-function onA, andf−1 is a C^1-function onB.

Note that a local diffeomorphism is a local homeomorphism. Also, as a consequence of Proposition 28.7, iff is a diffeomorphism onA, then Df(a) is a linear isomorphism for every a∈A. The following theorem can be shown. In fact, there is a fairly simple proof using Theorem 28.13; see Schwartz [90], Lang [67] and Cartan [18].

Theorem 28.14. LetE andF be complete normed affine spaces, letA be an open subset ofE, and letf :A→F be aC^1-function onA. The following properties hold: (1) For every A, if Df(a) is a linear isomorphism (which means that both Df(a) and (Df(a))−

a∈1 are linear and continuous),1 then there exist some open subsetU⊆A containinga, and some open subsetV ofF containingf(a), such thatf is a diffeomorphism fromU toV =f(U). Furthermore,

Df−1(f(a)) = (Df(a))−1.

For every neighborhoodN ofa, its imagef(N) is a neighborhood off(a), and for every open ballU⊆A of centera, its imagef(U) contains some open ball of centerf(a). 1Actually, since E and F are Banach spaces, by the Open Mapping Theorem, it is sufficient to assume that Df(a) is continuous and bijective; see Lang [67].

(2) If Df(a) is invertible for everya∈A, thenB =f(A) is an open subset ofF, and f is a local diffeomorphism fromA toB. Furthermore, iff is injective, thenf is a diffeomorphism fromA toB.

Part (1) of Theorem 28.14 is often referred to as the “(local) inverse function theorem.” It plays an important role in the study of manifolds and (ordinary) differential equations.

If E andF are both of finite dimension, and some frames have been chosen, the invertibility of Df(a) is equivalent to the fact that the Jacobian determinant det(J(f)(a)) is nonnull. The case where Df(a) is just injective or just surjective is also important for defining manifolds, using implicit definitions.

Definition 28.8. LetE andF be normed affine spaces, whereE andF are of finite dimension (or bothE andF are complete), and letA be an open subset ofE. For anya∈A, a C^1-functionf :A→F is an immersion ata if Df(a) is injective. AC^1-functionf :A→F is a submersion ata if Df(a) is surjective. AC^1-functionf :A→F is an immersion on A (resp. a submersion onA) if Df(a) is injective (resp. surjective) for everya∈A.

The following results can be shown.

Proposition 28.15. LetA be an open subset of Rn, and letf :A→ Rm be a function. For everya∈A,f :A→ Rm is a submersion ata iff there exists an open subsetU ofA containinga , an open subset W⊆ Rn−m, and a diffeomorphismÕ:U→f(U)×W, such that,

f =π1æÕ,

whereπ_1:f(U)×W→f(U) is the first projection. Equivalently,

(fæÕ−1)(y_1, . . . , ym, . . . , yn) = (y_1, . . . , ym).

U⊆AÕ f(U)×W NNNNN

f π_1NNNNNN

f(U)⊆ Rm Futhermore, the image of every open subset ofA underf is an open subset ofF. (The same result holds for Cn and Cm).

Proposition 28.16. LetA be an open subset of Rn, and letf :A→ Rm be a function. For everya∈A,f :A→ Rm is an immersion ata iff there exists an open subsetU of A containinga, an open subset V containingf(a) such thatf(U)⊆V , an open subsetW containing 0 such thatW⊆ Rm−n, and a diffeomorphismÕ:V→U×W, such that,

Õæf =in_1, wherein_1:U→U×W is the injection map such thatin_1(u) = (u,0), or equivalently,

(Õæf)(x_1, . . . , xn) = (x_1, . . . , xn,0, . . . ,0).

U⊆A f f(U)⊆V MMMMMM

in

1

M M M M

Õ M U×W

(The same result holds for Cn and Cm).

28.4 Tangent Spaces and Differentials

In this section, we discuss briefly a geometric interpretation of the notion of derivative. We consider sets of points defined by a differentiable function. This is a special case of the notion of a (differential) manifold.

Given two normed affine spacesE andF, letA be an open subset ofE, and letf :A→F be a function.

Definition 28.9. Givenf :A→F as above, its graph Γ(f) is the set of all points

Γ(f) ={(x, y)∈E×F|x∈A, y =f(x)}.

If Df is defined onA, we say that Γ(f) is a differential submanifold ofE×F of equation y =f(x).

It should be noted that this is a very particular kind of differential manifold.

Example 28.3. IfE = R andF = R^2, lettingf = (g, h), whereg: R→ R andh: R→ R, Γ(f) is a curve in R^3, of equationsy =g(x),z =h(x). WhenE = R^2 andF = R, Γ(f) is a surface in R^3, of equationz =f(x, y).

We now define the notion of affine tangent space in a very general way. Next, we will see what it means for manifolds Γ(f), as in Definition 28.9.

Definition 28.10. Given a normed affine spaceE, given any nonempty subsetM ofE, given any pointa∈M, we say that a vectoru∈E is tangent ata toM if there exist a sequence (an)n∈N of points inM converging toa, and a sequence (λn)n∈N, withλi∈ R and λn≥ 0, such that the sequence (λn(an−a))n∈N converges tou.

The set of all vectors tangent at a toM is called the family of tangent vectors ata to M and the set of all points ofE of the forma +u whereu belongs to the family of tangent vectors ata toM is called the affine tangent family ata toM.

Clearly, 0 is always tangent, and ifu is tangent, then so is everyλu, forλ∈ R,λ≥ 0. If u = 0, then the sequence (λn)n∈N must tend towards +∞. We have the following proposition.

Proposition 28.17. LetE andF be two normed affine spaces, letA be an open subset of E, leta∈ A, and letf :A→F be a function. If Df(a) exists, then the family of tangent vectors at (a, f(a)) to Γ is a subspaceTa(Γ) ofE×F, defined by the condition (equation)

(u, v)∈Ta(Γ) iff v = Df(a)(u),

and the affine tangent family at (a, f(a)) to Γ is an affine varietyTa(Γ) ofE×F, defined by the condition (equation)

(x, y)∈Ta(Γ) iff y =f(a) + Df(a)(x−a), where Γ is the graph off.

The proof is actually rather simple. We have Ta(Γ) =a +Ta(Γ), and sinceTa(Γ) is a subspace ofE×F, the setTa(Γ) is an affine variety. Thus, the affine tangent space at a point (a, f(a)) is a familar object, a line, a plane, etc.

As an illustration, whenE = R^2 andF = R, the affine tangent plane at the point (a, b, c) to the surface of equationz =f(x, y), is defined by the equation

z =c +∂f(a, b)(x−a) +∂f(a, b)(y−b).∂x ∂y

IfE = R andF = R^2, the tangent line at (a, b, c), to the curve of equationsy =g(x), z =h(x), is defined by the equations

y =b + Dg(a)(x−a), z =c + Dh(a)(x−a).

Thus, derivatives and partial derivatives have the desired intended geometric interpretation as tangent spaces. Of course, in order to deal with this topic properly, we really would have to go deeper into the study of (differential) manifolds.

We now briefly consider second-order and higher-order derivatives.

28.5 Second-Order and Higher-Order Derivatives

Given two normed affine spaces E andF, and some open subsetA ofE, if Df(a) is defined for everya∈A, then we have a mapping Df :A→ L (E;F). SinceL(E;F) is a normed vector space, if Df exists on an open subsetU ofA containinga, we can consider taking the derivative of Df at somea∈A. If D(Df)(a) exists for everya∈A, we get a mapping D^2f :A→ L(E;L (E;F)), where D^2f(a) = D(Df)(a), for everya∈A. If D^2f(a) exists, then for everyu∈E,

D^2f(a)(u) = D(Df)(a)(u) = Du(Df)(a)∈ L(E;F).

Recall from Proposition 26.46, that the map app fromL(E;F)×E toF, defined such that for everyL∈ L(E;F), for everyv∈E,

app(L, v) =L(v),

is a continuous bilinear map. Thus, in particular, given a fixedv∈ E, the linear map appv:L(E;F)→F, defined such that appv(L) =L(v), is a continuous map. Also recall from Proposition 28.6, that ifh:A→G is a function such that Dh(a) exits, andk:G→H is a continuous linear map, then, D(kæh)(a) exists, and

k(Dh(a)(u)) = D(kæh)(a)(u),

that is, k(Duh(a)) = Du(kæh)(a), Applying these two facts toh = Df, and tok = appv, we have

Du(Df)(a)(v) = Du(appvæ Df)(a).

But (appvæ Df)(x) = Df(x)(v) = Dvf(x), for everyx∈A, that is, appvæ Df = Dvf onA. So, we have

Du(Df)(a)(v) = Du(Dvf)(a), and since D^2f(a)(u) = Du(Df)(a), we get

D^2f(a)(u)(v) = Du(Dvf)(a).

Thus, when D^2f(a) exists, Du(Dvf)(a) exists, and

D^2f(a)(u)(v) = Du(Dvf)(a),

for allu, v∈E. We also denote Du(Dvf)(a) by D^2u,vf(a), or DuDvf(a).

Recall from Proposition 26.45, that the map fromL2(E, E;F) toL(E;L(E;F)) defined such thatg→Õ iff for everyg∈ L2(E, E;F),

Õ(u)(v) =g(u, v),

is an isomorphism of vector spaces. Thus, we will consider D^2f(a)∈ L(E;L(E;F)) as a continuous bilinear map inL2(E, E;F), and we will write D^2f(a)(u, v), instead of D^2f(a)(u)(v). Then, the above discussion can be summarized by saying that when D^2f(a) is defined, we have

D^2f(a)(u, v) = DuDvf(a).

WhenE has finite dimension and (a_0,(e_1, . . . , en)) is a frame forE, we denote DejDeif(a) by ∂2f (a), wheni =j, and we denote DeiDeif(a) by∂2f(a).∂xi∂xj ∂x2

The following important lemma attributed to Schwarz can be shown, using Lemma 28.11. Given a bilinear mapf :E×E→F, recall thatf is symmetric, if

f (u, v) =f(v, u),

for allu, v∈E.

Lemma 28.18. (Schwarz’s lemma) Given two normed affine spacesE andF, given any open subsetA ofE, given anyf :A→F, for everya∈A, if D^2f(a) exists, then D^2f(a)∈

L

2(E, E;F) is a continuous symmetric bilinear map. As a corollary, ifE is of finite dimensionn, and (a_0,(e_1, . . . , en)) is a frame forE, we have

∂2f (a) = ∂2f (a).∂xi∂xj ∂xj∂xi

Remark: There is a variation of the above lemma which does not assume the existence of D^2f(a), but instead assumes that DuDvf and DvDuf exist on an open subset containinga and are continuous ata, and concludes that DuDvf(a) = DvDuf(a). This is just a different result which does not imply Lemma 28.18, and is not a consequence of Lemma 28.18.

∂

2f (a) and ∂2

WhenE =f (a) is not sufficient to insure the R^2, the only existence of∂x∂y ∂y∂x

existence of D^2f(a).

When E if of finite dimensionn and (a_0,(e_1, . . . , en)) is a frame forE, if D^2f(a) exists, for everyu =u_1e_1 +· · · +unen andv =v_1e_1 +· · · +vnen inE, since D^2f(a) is a symmetric bilinear form, we haven ∂2

D^2f(a)(u, v) = uf (a),ivj∂xi∂xji=1,j=1

which can be written in matrix form as:

ë ∂2f(a)∂2f (a) . . .∂2f (a)÷ ì ∂x^2 ∂x_1∂x_2 ∂x_1∂xn ÷ ì ∂2f (a)∂2f(a) . . .∂2f (a)÷ D^2f(a)(u, v) =Uì∂x1∂x2 ∂x^2 ∂x2∂xn÷ Vì 2 ... . ÷

ì . . ÷

ì ∂2f (a)∂2f (a) . . .∂2f (a)ø ∂x_1∂xn ∂x_2∂xn ∂x^2n

ö

ì_1

÷ ì

÷ ì

÷ ì

÷

í

ö

ì

whereU is the column matrix representingu, andV is the column matrix representingv, over the frame (a_0,(e_1, . . . , en)).

The above symmetric matrix is called the Hessian of f at a. IfF itself is of finite dimension, and (b_0,(v_1, . . . , vm)) is a frame forF, thenf = (f_1, . . . , fm), and each component D^2f(a)i(u, v) of D^2f(a)(u, v) (1≤i≤m), can be written as

ë ∂2fi(a)∂2fi (a) . . .∂2fi (a)÷

ì

∂x

2

1

ì

ì

ì

∂x_1∂x_2 ∂x_1∂xn ÷ ì ∂2fi (a)∂2fi(a) . . .∂2f ÷ i (a)÷ D

2

f

(

a

)

i

(

u, v

) =

U

ì∂x_1∂x_2 ∂x^2 ∂x_2∂x÷

n ÷Vì^2 ... ÷

.

÷ ì . . ÷

ì ∂2fi (a)∂2fi (a) . . .∂2fi(a)ø ∂x_1∂xn ∂x_2∂xn ∂x^2n

÷

í

Thus, we could describe the vector D^2f(a)(u, v) in terms of anmn×mn-matrix consisting ofm diagonal blocks, which are the above Hessians, and the row matrix (U , . . . , U ) (m times) and the column matrix consisting ofm copies ofV .

We now indicate briefly how higher-order derivatives are defined. Letm≥ 2. Given a functionf :A→F as before, for anya∈A, if the derivatives Dif exist onA for all i, 1≤ i≤ m− 1, by induction, Dm−1f can be considered to be a continuous function Dm−1f :A→ Lm−1(Em−1;F), and we define

Dmf(a) = D(Dm−1f)(a).

Then, Dmf(a) can be identified with a continuousm-multilinear map inLm(Em;F). We can then show (as we did before), that if Dmf(a) is defined, then

Dmf(a)(u_1, . . . , um) = Du1. . .Dumf(a).

WhenE if of finite dimensionn and (a_0,(e_1, . . . , en)) is a frame forE, if Dmf(a) exists, for everyj_1, . . . , jm∈ {1, . . . , n}, we denote Dejm. . .Dej1f(a) by

∂mf (a).∂xj1. . . ∂xjm

Given am-multilinear mapf∈ Lm(Em;F), recall thatf is symmetric if

f(uπ(1), . . . , uπ(m)) =f(u_1, . . . , um),

for all u_1, . . . , um∈E, and all permutationsπ on{1, . . . , m}. Then, the following generalization of Schwarz’s lemma holds.

Lemma 28.19. Given two normed affine spacesE andF, given any open subsetA ofE, given anyf :A→F, for everya∈A, for everym≥ 1, if Dmf(a) exists, then Dmf(a)∈ m(Em;F) is a continuous symmetricm-multilinear map. As a corollary, ifE is of finiteLdimensionn, and (a0,(e1, . . . , en)) is a frame forE, we have

∂

m

f

(

a

) =

∂xπ(j1). . . ∂xπ(jm) ∂mf (a),∂xj1. . . ∂xjm

for everyj_1, . . . , jm∈ {1, . . . , n}, and for every permutationπ on{1, . . . , m}.

IfE is of finite dimensionn, and (a_0,(e_1, . . . , en)) is a frame forE, Dmf(a) is a symmetric m-multilinear map, and we have

D

m

f

(

a

)(

u

1, . . . , um) = u1,j1· · ·um,jm∂xj1. . . ∂xjmj

∂mf (a),

wherej ranges over all functionsj:{1, . . . , m} → {1, . . . , n}, for anym vectors

uj =uj,1e_1 +· · · +uj,nen.

The concept ofC^1-function is generalized to the concept ofCm-function, and Theorem 28.12 can also be generalized.

Definition 28.11. Given two normed affine spacesE andF, and an open subsetA ofE, for anym≥ 1, we say that a functionf :A→F is of classCm onA or aCm-function on A if Dkf exists and is continuous onA for everyk, 1≤k≤m. We say thatf :A→F is of classC∞ onA or aC∞-function onA if Dkf exists and is continuous onA for every k≥ 1. AC∞-function (onA) is also called a smooth function (onA). ACm-diffeomorphism f :A→B betweenA andB (whereA is an open subset ofE andB is an open subset ofB) is a bijection betweenA andB =f(A), such that bothf :A→B and its inverse f−1:B→A areCm-functions.

Equivalently,f is aCm-function onA iff is aC^1-function onA and Df is aCm−1- function onA.

We have the following theorem giving a necessary and sufficient condition for f to a Cm-function onA. A generalization to the case whereE = (E_1, a_1)⊕ · · · ⊕ (En, an) also holds.

Theorem 28.20. Given two normed affine spacesE andF, whereE is of finite dimension n, and where (a_0,(u_1, . . . , un)) is a frame ofE, given any open subsetA ofE, given any functionf :A→F, for anym≥ 1, the derivative Dmf is aCm-function onA iff every

partial derivative

D

ujk. . .Duj1f (or ∂xj1. . . ∂x

∂kf (a)) is defined and continuous onA, for all

jk

k, 1≤k≤m, and allj_1, . . . , jk ∈ {1, . . . , n}. As a corollary, ifF is of finite dimensionp, and (b0,(v1, . . . , vp)) is a frame ofF, the derivative Dmf is defined and continuous onA iff every partial derivative

D

ujk. . .Duj1fi (or ∂xj1. . . ∂xjk .k, 1≤k≤m, for alli, 1≤i≤p, and allj_1, . . . , jk∈ {1, . . . , n}

∂kfi (a)) is defined and continuous onA, for all

When E = R (orE = C), for anya∈E, Dmf(a)(1, . . . ,1) is a vector inF, called themth-order vector derivative. As in the casem = 1, we will usually identify the multilinear map Dmf(a) with the vector Dmf(a)(1, . . . ,1). Some notational conventions can also be introduced to simplify the notation of higher-order derivatives, and we discuss such conventions very briefly.

Recall that whenE is of finite dimensionn, and (a_0,(e_1, . . . , en)) is a frame forE, Dmf(a) is a symmetricm-multilinear map, and we have

D

m

f

(

a

)(

u

1, . . . , um) = u1,j1· · ·um,jm∂xj1. . . ∂xjmj

∂mf (a),

wherej ranges over all functionsj:{1, . . . , m} → {1, . . . , n}, for anym vectors

uj =uj,1e_1 +· · · +uj,nen.

We can then group the various occurrences of∂xjk corresponding to the same variablexjk, and this leads to the notation

∂ α1 ∂ α2 ∂ αnf(a),∂x_1 ∂x_2 · · · ∂xn

whereα_1 +α_2 +· · · +αn =m. If we denote (α_1, . . . , αn) simply byα, then we denote

∂ α1 ∂ α2 ∂ αn ∂x

1 ∂x_2 · · · ∂xn f

by ∂

αf, or∂ α ∂xf. Ifα = (α_1, . . . , αn), we let|α| =α_1 +α_2 +· · ·+αn,α! =α_1!· · ·αn!, and ifh = (h_1, . . . , hn), we denotehα1 hαn byhα.1· · · n

In the next section, we survey various versions of Taylor’s formula.

28.6 Taylor’s formula, Fa`a di Bruno’s formula

We discuss, without proofs, several versions of Taylor’s formula. The hypotheses required in each version become increasingly stronger. The first version can be viewed as a generalization of the notion of derivative. Given anm-linear mapf :Em F, for any vectorh∈E, we abbreviate→

f(h, . . . , h)

m

byf(hm). The version of Taylor’s formula given next is sometimes referred to as the formula of Taylor–Young.

Theorem 28.21. (Taylor–Young) Given two normed affine spacesE andF, for any open subsetA⊆E, for any functionf :A→F, for anya∈A, if Dkf exists inA for allk, 1≤k≤m− 1, and if Dmf(a) exists, then we have:

f(a +h) =f(a) + 11!D^1f(a)(h) +· · · + 1!Dmf(a)(hm) + h m (h),

m

for anyh such thata +h∈A, and where limh→0, h=0 (h) = 0.

The above version of Taylor’s formula has applications to the study of relative maxima (or minima) of real-valued functions. It is also used to study the local properties of curves and surfaces.

The next version of Taylor’s formula can be viewed as a generalization of Lemma 28.11. It is sometimes called the Taylor formula with Lagrange remainder or generalized mean value theorem.

Theorem 28.22. (Generalized mean value theorem) LetE andF be two normed affine spaces, letA be an open subset ofE, and letf :A→F be a function onA. Given any a∈A and anyh = 0 inE, if the closed segment [a, a +h] is contained inA, Dkf exists in A for allk, 1≤k≤m, Dm+1f(x) exists at every pointx of the open segment ]a, a+h[, and

x

maxh[ Dm+1f(x) ≤M,

∈]a,a+

for someM≥ 0, then

1 1f(a)(h) +· · · + 1!Dmf(a)(hm) ≤M h m+1

f(a +h)−f(a)− 1!Dm (m + 1)!.

As a corollary, ifL:Em+1 F is a continuous (m + 1)-linear map, then→

1 1f(a)(h) +· · · + 1!Dmf(a)(hm) +L(hm+1) M h m+1

f(a +h)−f(a)− 1!Dm (m + 1)! ≤ (m + 1)!,

whereM = maxx∈]a,a+h[ Dm+1f(x)−L .

The above theorem is sometimes stated under the slightly stronger assumption that f is aCm-function onA. Iff :A→ R is a real-valued function, Theorem 28.22 can be refined a little bit. This version is often called the formula of Taylor–MacLaurin.

Theorem 28.23. (Taylor–MacLaurin) LetE be a normed affine space, letA be an open subset ofE, and letf :A→ R be a real-valued function onA. Given anya∈ A and any h = 0 inE, if the closed segment [a, a +h] is contained inA, if Dkf exists in A for allk, 1≤k≤m, and Dm+1f(x) exists at every pointx of the open segment ]a, a +h[, then there

is someθ∈ R, with 0< θ < 1, such that

1

f(a +h) =f(a) + 11!D^1f(a)(h) +· · · + 1!Dmf(a)(hm) +(m + 1)!Dm+1f(a +θh)(hm+1).

m

We also mention for “mathematical culture,” a version with integral remainder, in the case of a real-valued function. This is usually called Taylor’s formula with integral remainder.

Theorem 28.24. (Taylor’s formula with integral remainder) LetE be a normed affine space, letA be an open subset ofE, and letf :A→ R be a real-valued function onA. Given any a∈A and anyh = 0 inE, if the closed segment [a, a +h] is contained inA, and iff is a C+1-function onA, then we have

f(a +h) =f(a) + 11!D^1f(a)(h) +· · · + 1!Dmf(a)(hm)

m

+ 1 (1−t)m m+1f(a +th)(hm+1)dt.

0 m!D

The advantage of the above formula is that it gives an explicit remainder. We now examine briefly the situation whereE is of finite dimensionn, and (a_0,(e_1, . . . , en)) is a frame forE. In this case, we get a more explicit expression for the expression

k=m 1 kf(a)(hk)

i=0 k!D

involved in all versions of Taylor’s formula, where by convention, D^0f(a)(h^0) =f(a). If h =h_1e_1 +· · · +hnen, then we have

k=m 1 kf(a)(hk) =hk1 hkn ∂ k1 ∂ knf(a),

k =0k!D k_1+···+kn≤mk

1 n

1!

· · ·kn! ∂x_1 · · · ∂xn· · ·

which, using the abbreviated notation introduced at the end of Section 28.5, can also be

written as k=m 1 kf(a)(hk) =hα

!D

α

|≤

m

α

!

∂αf(a).

k

=0

k

|

The advantange of the above notation is that it is the same as the notation used when n = 1, i.e., whenE = R (orE = C). Indeed, in this case, the Taylor–MacLaurin formula reads as:

h 1f(a) +· · · +hm mf(a) + hm+1

f(a +h) =f(a) +m+1f(a +θh),1!Dm!D(m + 1)!D

for some θkf(a) is the value of thek-th derivative off at∈ R, with 0< θ < 1, where D

a (and thus, as we have already said several times, this is thekth-order vector derivative, which is just a scalar, sinceF = R).

In the above formula, the assumptions are thatf : [a, a +h]→ R is aCm-function on [a, a +h], and that Dm+1f(x) exists for everyx∈]a, a +h[.

Taylor’s formula is useful to study the local properties of curves and surfaces. In the case of a curve, we consider a functionf : [r, s]→F from a closed interval [r, s] of R to some affine spaceF, the derivatives Dkf(a)(hk) correspond to vectorshkDkf(a), where Dkf(a) is thekth vector derivative off ata (which is really Dkf(a)(1, . . . ,1)), and for anya∈]r, s[, Theorem 28.21 yields the following formula:

h 1f(a) +· · · +hm

f(a +h) =f(a) + 1!Dm!Dmf(a) +hm (h),

for anyh such thata +h∈]r, s[, and where limh→0, h=0 (h) = 0.

In the case of functions f : Rn R, it is convenient to have formulae for the Taylor–→

Young formula and the Taylor–MacLaurin formula in terms of the gradient and the Hessian. Recall that the gradient f(a) off ata∈ Rn is the column vector

ë ∂f(a)ö

ì

÷

ì

∂x

1

÷

ì ∂f (a)÷

ì ÷

f

(

a

)

ì ∂x_2 ÷ ì ÷, ì

÷

í

.

÷ ì ÷

ì ∂f (a)ø ∂xn

and that f (a)(u) = Df(a)(u) = f(a)·u,

for anyu∈ Rn (where· means inner product). The Hessian matrix^2f(a) off ata∈ Rn ö

ì

is then×n symmetric matrix

ë ∂2f(a)∂2f (a) . . .∂2f (a)÷

ì

∂x

2

1 ÷ ì

∂x_1∂x_2 ∂x_1∂xn ÷ ì ∂2f (a)∂2f(a) . . .∂2f

÷ ì

ì

( a )

÷

2f(a) =ì∂x_1∂x_2 ∂x^2 ∂x_2∂xn ÷,ì 2 ... ÷ .

÷ ì . . ÷

ì ∂2f (a)∂2f (a) . . .∂2f (a)ø ∂x_1∂xn ∂x_2∂xn ∂x^2n and we have

÷

í

D

2f(a)(u, v) =u^2f(a)v =u^2

· f(a)v =2f(a)u·v, for allu, v∈ Rn. Then, we have the following three formulations of the formula of Taylor– Young of order 2:

f(a +h) =f(a) + Df(a)(h) + 12D^2f(a)(h, h) + h^2 (h)

+ 12(h· f(a)h) + (h·h) (h)·h^2f(a +h) =f(a) + f(a)

f(a +h) =f(a) + ( f(a)) h + 12(h^2f(a)h) + (h h) (h). with limh→0 (h) = 0.

One should keep in mind that only the first formula is intrinsic (i.e., does not depend on the choice of a basis), whereas the other two depend on the basis and the inner product chosen on Rn. As an exercise, the reader should write similar formulae for the Taylor–MacLaurin formula of order 2.

Another application of Taylor’s formula is the derivation of a formula which gives the m- th derivative of the composition of two functions, usually known as “Fa`a di Bruno’s formula.” This formula is useful when dealing with geometric continuity of splines curves and surfaces. Proposition 28.25. Given any normed affine spaceE, for any functionf : R→ R and any functiong: R→E, for anya∈ R, lettingb =f(a),f(i)(a) = Dif(a), andg(i)(b) = Dig(b),

for any

m

≥

1

, if

f

(i)(a) andg( b) exist for alli, 1≤i≤m, then (gæf)(m)(a) = Dm ( (gæf)(a) exists and is given by the following formula:

(gæf)(m)(a) =i1!· · ·im!g(j)(b) f(1)(a) i1 f(m)(a) imm! .

0≤j≤m i_1+i_2+··· +im=j1! · · · m! i_1+2i_2+···+mim=m

i1,i2,···,im≥0

Whenm = 1, the above simplifies to the familiar formula (gæf) (a) =g (b)f (a), and form = 2, we have

(gæf)(2)(a) =g(2)(b)(f(1)(a))2 +g(1)(b)f(2)(a). 28.7. VECTOR FIELDS, COVARIANT DERIVATIVES, LIE BRACKETS 809

28.7 Vector Fields, Covariant Derivatives, Lie Brackets

In this section, we briefly consider vector fields and covariant derivatives of vector fields. Such derivatives play an important role in continuous mechanics. Given a normed affine space (E, E), a vector field over (E, E) is a functionX:E→E. Intuitively, a vector field assigns a vector to every point inE. Such vectors could be forces, velocities, accelerations, etc.

Given two vector fields X, Y defined on some open subset & ofE, for every pointa∈ &, we would like to define the derivative ofX with respect toY ata. This is a type of directional derivative that gives the variation ofX as we move alongY , and we denote it by DYX(a). The derivative DYX(a) is defined as follows.

Definition 28.12. Let (E, E) be a normed affine space. Given any open subset & ofE, given any two vector fieldsX andY defined over &, for anya∈ &, the covariant derivative (or Lie derivative) ofX w.r.t. the vector fieldY ata, denoted by DYX(a), is the limit (if it exists)

t

lim

X(a +tY (a))

U

t−X(a),

→0, t∈

whereU ={t∈ R|a +tY (a)∈ &, t = 0}.

IfY is a constant vector field, it is immediately verified that the map

X→ DYX(a)

is a linear map called the derivative of the vector field X, and denoted by DX(a). If f :E→ R is a function, we define DYf(a) as the limit (if it exists)

t

lim

f(a +tY (a))

U

t

−f(a),

→0, t∈

whereU ={t∈ R|a +tY (a)∈ &, t = 0}. It is the directional derivative off w.r.t. the vector fieldY ata, and it is also often denoted byY (f)(a), orY (f)a.

From now on, we assume that all the vector fields and all the functions under consideration are smooth (C∞). The setC∞(&) of smoothC∞-functionsf : &→ R is a ring. Given a smooth vector fieldX and a smooth functionf (both over &), the vector fieldfX is defined such that (fX)(a) =f(a)X(a), and it is immediately verified that it is smooth. Thus, the setX(&) of smooth vector fields over & is aC∞(&)-module.

The following proposition is left as an exercise. It shows that DYX(a) is a R-bilinear map onX(&), isC∞(&)-linear inY , and satisfies the Leibniz derivation rules with respect toX.

Proposition 28.26. The covariant derivative DYX(a) satisfies the following properties:

D(Y1+Y2)X(a) = DY1X(a) + DY2X(a), DfYX(a) =f(a)DYX(a),

DY(X_1 +X_2)(a) = DYX_1(a) + DYX_2(a),

DYfX(a) = DYf(a)X(a) +f(a)DYX(a), whereX, Y, X_1, X_2, Y_1, Y_2 are smooth vector fields over &, andf :E→ R is a smooth function.

In differential geometry, the above properties are taken as the axioms of affine connections, in order to define covariant derivatives of vector fields over manifolds. In many cases, the vector fieldY is the tangent field of some smooth curveγ: ]−η, η[→E. If so, the following proposition holds.

Proposition 28.27. Given a smooth curveγ: ]−η, η[→E, lettingY be the vector field defined onγ(]−η, η[) such that

Y (γ(u)) =dγ (u),dt for any vector fieldX defined onγ(]−η, η[), we have

DYX(a) = d X(γ(t)) (0),dt

wherea =γ(0).

The derivative DYX(a) is thus the derivative of the vector fieldX along the curveγ, and it is called the covariant derivative ofX alongγ.

Given an affine frame (O,(u_1, . . . , un)) for (E, E), it is easily seen that the covariant derivative DYX(a) is expressed as follows:

n n

D∂Xi (a)ei.YX(a) = Yj ∂xji=1 j=1

Generally, DYX(a) = DXY (a). The quantity [X, Y ] = DXY− DYX

is called the Lie bracket of the vector fieldsX andY . The Lie bracket plays an important role in differential geometry. In terms of coordinates,

[X, Y ] = X∂Xi

n n ∂Yi Yj ∂xji=1 j=1 ∂xj−ei.j

28.8. FUTHER READINGS 811

28.8 Futher Readings

A thorough treatment of differential calculus can be found in Munkres [82], Lang [68], Schwartz [90], Cartan [18], and Avez [5]. The techniques of differential calculus have many applications, especially to the geometry of curves and surfaces and to differential geometry in general. For this, we recommend do Carmo [28, 29] (two beautiful classics on the subject), Kreyszig [63], Stoker [98], Gray [48], Berger and Gostiaux [8], Milnor [78], Lang [66], Warner [109] and Choquet-Bruhat [21].

Chapter 29 Extrema of Real-Valued Functions

29.1 Local Extrema, Constrained Local Extrema, and Lagrange Multipliers

Let J:E→ R be a real-valued function defined on a normed vector spaceE (or more generally, any topological space). Ideally we would like to find where the functionJ reaches a minimum or a maximum value, at least locally. In this chapter, we will usually use the notationsdJ(u) orJ (u) (ordJu orJu) for the derivative ofJ atu, instead of DJ(u). Our presentation follows very closely that of Ciarlet [22] (Chapter 7), which we find to be one of the clearest.

Definition 29.1. IfJ:E→ R is a real-valued function defined on a normed vector space E, we say thatJ has a local minimum (or relative minimum) at the pointu∈E if there is some open subsetW⊆E containingu such that

J(u)≤J(w) for allw∈W .

Similarly, we say thatJ has a local maximum (or relative maximum) at the pointu∈E if there is some open subsetW⊆E containingu such that

J(u)≥J(w) for allw∈W .

In either case, we say that J has a local extremum (or relative extremum) atu. We say that J has a strict local minimum (resp. strict local maximum) at the pointu∈E if there is some open subsetW⊆E containingu such that

J(u)< J(w) for allw∈W− {u}

(resp.

J(u)> J(w) for allw∈W− {u}).

813 By abuse of language, we often say that the pointu itself “is a local minimum” or a

“local maximum,” even though, strictly speaking, this does not make sense. We begin with a well-known necessary condition for a local extremum.

Proposition 29.1. LetE be a normed vector space and letJ: &→ R be a function, with & some open subset ofE. If the functionJ has a local extremum at some point u∈ & and ifJ is differentiable atu, then

dJ(u) =J (u) = 0.

Proof. Pick anyv∈E. Since & is open, fort small enough we haveu +tv∈ &, so there is an open intervalI⊆ R such that the functionÕ given by

Õ(t) =J(u +tv)

for allt∈I is well-defined. By applying the chain rule, we see thatÕ is differentiable at t = 0, and we get

Õ (0) =dJu(v).

Without loss of generality, assume thatu is a local minimum. Then we have

Õ (0) = limÕ(t)−Õ(0) 0

t→0− t ≤

and

ÕÕ(t)−Õ(0) 0, (0) = lim

t ≥→0+ t

which shows thatÕ (0) =dJu(v) = 0. Asv∈E is arbitrary, we conclude thatdJu = 0.

A pointu∈ & such thatJ(u) = 0 is called a critical point ofJ.

It is important to note that the fact that & is open is crucial. For example, ifJ is the identity function on [0,1], thendJ(x) = 1 for allx∈ [0,1], even thoughJ has a minimum at x = 0 and a maximum atx = 1. Also, ifE = Rn, then the conditiondJ(u) = 0 is equivalent to the system

∂J (u_1, . . . , un) = 0∂x1

.

∂J (u_1, . . . , un) = 0.∂xn

In many practical situations, we need to look for local extrema of a function J under additional constraints. This situation can be formalized conveniently as follows: We have a functionJ: &→ R defined on some open subset & of a normed vector space, but we also have some subsetU of & and we are looking for the local extrema ofJ with respect to the setU. Note that in most cases,U is not open. In fact,U is usually closed. Definition 29.2. IfJ: &→ R is a real-valued function defined on some open subset & of a normed vector spaceE and ifU is some subset of &, we say thatJ has a local minimum (or relative minimum) at the pointu∈U with respect toU if there is some open subsetW⊆ & containingu such that

J(u)≤J(w) for allw∈U∩W .

Similarly, we say thatJ has a local maximum (or relative maximum) at the pointu∈U with respect toU if there is some open subsetW⊆ & containingu such that

J(u)≥J(w) for allw∈U∩W .

In either case, we say thatJ has a local extremum atu with respect toU.

We will be particularly interested in the case where &⊆E1×E_2 is an open subset of a product of normed vector spaces and whereU is the zero locus of some continuous function Õ: &→E_2, which means that

U ={(u_1, u_2)∈ &|Õ(u_1, u_2) = 0}.

For the sake of brevity, we say that J has a constrained local extremum atu instead of saying thatJ has a local extremum at the pointu∈U with respect toU. Fortunately, there is a necessary condition for constrained local extrema in terms of Lagrange multipliers.

Theorem 29.2. (Necessary condition for a constrained extremum) Let &⊆E1×E_2 be an open subset of a product of normed vector spaces, withE_1 a Banach space (E1 is complete), letÕ: &→E_2 be aC^1-function (which means thatdÕ(ω) exists and is continuous for all ω∈ &), and let

U ={(u_1, u_2)∈ &|Õ(u_1, u_2) = 0}. Moreover, letu = (u_1, u_2)∈U be a point such that

∂Õ (u_1, u_2)∈ L(E_2;E_2) and∂Õ (u_1, u_2) −1 ∂x_2 ∂x_2 ∈ L(E2;E2), and letJ: &→ R be a function which is differentiable atu. IfJ has a constrained local extremum atu, then there is a continuous linear form Λ(u)∈ L(E_2;R) such that

dJ(u) + Λ(u)ædÕ(u) = 0.

Proof. The plan of attack is to use the implicit function theorem; Theorem 28.13. Observe that the assumptions of Theorem 28.13 are indeed met. Therefore, there exist some open subsetsU1⊆E_1,U2⊆E_2, and a continuous functiong:U1→U_2 with (u_1, u_2)∈U1×U2⊆ & and such that

Õ(v_1, g(v_1)) = 0 for allv1∈U_1. Moreover,g is differentiable atu1∈U_1 and

∂Õ (u) −1 ∂Õ (u). dg(u_1) =− ∂x2 æ ∂x1

It follows that the restriction ofJ to (U1×U_2)∩U yields a functionG of a single variable, with

G(v_1) =J(v_1, g(v_1))

for allv1∈U_1. Now, the functionG is differentiable atu_1 and it has a local extremum at u_1 onU_1, so Proposition 29.1 implies that

dG(u_1) = 0.

By the chain rule,

dG(u_1) = ∂J (u) + ∂J (u)ædg(u_1)∂x1 ∂x2

=

∂x_1

∂J

(

u

)

− ∂x

∂J (u)∂Õ(u) −1 ∂Õ (u). 2 æ ∂x_2 æ ∂x_1

FromdG(u_1) = 0, we deduce

∂Õ

(

u

)

−

1

∂Õ ( u ) , ∂x

∂J (u) = ∂J (u)æ ∂x_2 æ ∂x_11 ∂x_2

and since we also have ∂Õ

(

u

)

−

1

∂Õ ( u ) , ∂x

∂J (u) = ∂J (u)æ ∂x2 æ ∂x22 ∂x2

if we let_1 Λ(

u

) =

−∂x

∂J (u)∂Õ(u) − , 2 æ ∂x_2 then we get

dJ(u) = ∂J (u) + ∂J (u)∂x1 ∂x2

=

∂x_2

∂J

(

u

)

æ ∂x

∂Õ (u) −1 ∂Õ (u) + ∂Õ (u) 2 æ ∂x_1 ∂x_2 =−Λ(u)ædÕ(u),

which yieldsdJ(u) + Λ(u)ædÕ(u) = 0, as claimed.

In most applications, we have E_1 = Rn−m andE_2 = Rm for some integersm, n such that 1≤m < n, & is an open subset of Rn,J: &→ R, and we havem functionsÕi: &→ R defining the subset

U ={v∈ &|Õi(v) = 0, 1≤i≤m}. Theorem 29.2 yields the following necessary condition:

Theorem 29.3. (Necessary condition for a constrained extremum in terms of Lagrange multipliers) Let & be an open subset of Rn, consider m C^1-functions Õi: &→ R (with 1≤m < n), let

U = (v∈ &|Õi(v) = 0, 1≤i≤m},

and let u∈U be a point such that the derivativesdÕi(u)∈ L(Rn;R) are linearly independent; equivalently, assume that them×n matrix (∂Õi/∂xj)(u) has rankm. IfJ: &→ R is a

function which is differentiable at u∈ U and ifJ has a local constrained extremum at u,

then there existm numbersλi(u)∈ R, uniquely defined, such that

dJ(u) +λ_1(u)dÕ_1(u) +· · · +λm(u)dÕm(u) = 0;

equivalently, J(u) +λ_1(u) Õ_1(u) +· · · +λ_1(u) Õm(u) = 0.

Proof. The linear independence of them linear formsdÕi(u) is equivalent to the fact that them×n matrixA = (∂Õi/∂xj)(u) has rankm. By reordering the columns, we may assume that the firstm columns are linearly independent. If we letÕ: &→ Rm be the function defined by

Õ(v) = (Õ_1(v), . . . , Õm(v))

for all v∈ &, then we see that∂Õ/∂x_2(u) is invertible and both∂Õ/∂x_2(u) and its inverse are continuous, so that Theorem 29.3 applies, and there is some (continuous) linear form Λ(u)∈ L(Rm;R) such that

dJ(u) + Λ(u)ædÕ(u) = 0.

However, Λ(u) is defined by somem-tuple (λ_1(u), . . . , λm(u))∈ Rm, and in view of the definition ofÕ, the above equation is equivalent to

dJ(u) +λ_1(u)dÕ_1(u) +· · · +λm(u)dÕm(u) = 0.

The uniqueness of theλi(u) is a consequence of the linear independence of thedÕi(u).

The numbers λi(u) involved in Theorem 29.3 are called the Lagrange multipliers associated with the constrained extremumu (again, with some minor abuse of language). The linear independence of the linear formsdÕi(u) is equivalent to the fact that the Jacobian matrix (∂Õi/∂xj)(u) ofÕ = (Õ_1, . . . , Õm) atu has rankm. Ifm = 1, the linear independence of thedÕi(u) reduces to the condition Õ_1(u) = 0.

A fruitful way to reformulate the use of Lagrange multipliers is to introduce the notion of the Lagrangian associated with our constrained extremum problem. This is the function L: &×Rm R given by→

L(v, λ) =J(v) +λ_1Õ_1(v) +· · · +λmÕm(v),

withλ = (λ_1, . . . , λm). Then, observe that there exists someµ = (µ_1, . . . , µm) and some u∈U such that

dJ(u) +µ_1dÕ_1(u) +· · · +µmdÕm(u) = 0 if and only if

dL(u, µ) = 0, or equivalently

L(u, µ) = 0; that is, iff (u, λ) is a critical point of the LagrangianL. IndeeddL(u, µ) = 0 if equivalent to

∂L (u, µ) = 0∂v

∂L (u, µ) = 0∂λ1

.

∂L (u, µ) = 0,∂λm

and since ∂L (u, µ) =dJ(u) +µ_1dÕ_1(u) +· · · +µmdÕm(u)∂v

and ∂L (u, µ) =Õi(u),∂λi

we get

dJ(u) +µ_1dÕ_1(u) +· · · +µmdÕm(u) = 0 and

Õ_1(u) =· · · =Õm(u) = 0, that is,u∈U.

If we write out explicitly the condition

dJ(u) +µ_1dÕ_1(u) +· · · +µmdÕm(u) = 0, we get then×m system ∂Õ

1

(

u

) +

∂Õ

m

( u ) = 0 ∂x

∂J (u) +λ1∂x1 · · · +λm ∂x11

.

∂Õ

1

(

u

) +

∂Õ

m

( u ) = 0 , ∂xn

∂J (u) +λ1∂xn · · · +λm ∂xn

and it is important to note that the matrix of this system is the transpose of the Jacobian matrix ofÕ atu. If we write Jac(J)(u) = (∂Õi/∂xj)(u) for the Jacobian matrix ofJ (at u), then the above system is written in matrix form as

J(u) + (Jac(J)(u)) λ = 0,

whereλ is viewed as a column vector, and the Lagrangian is equal to

L(u, λ) =J(u) + (Õ_1(u), . . . , Õm(u))λ.

Remark: If the Jacobian matrix Jac(J)(v) = (∂Õi/∂xj)(v) has rankm for allv∈U (which is equivalent to the linear independence of the linear formsdÕi(v)), then we say that 0∈ Rm is a regular value ofÕ. In this case, it is known that

U ={v∈ &|Õ(v) = 0}

is a smooth submanifold of dimensionn−m of Rn. Furthermore, the set

m

TvU ={w∈ Rn dÕi(v)(w) = 0, 1≤i≤m} = KerdÕi(v)| i=1

is the tangent space toU atv (a vector space of dimensionn−m). Then, the condition

dJ(v) +µ_1dÕ_1(v) +· · · +µmdÕm(v) = 0

implies that dJ(v) vanishes on the tangent spaceTvU. Conversely, ifdJ(v)(w) = 0 for allw∈TvU, this means thatdJ(v) is orthogonal (in the sense of Definition 4.7) toTvU. Since (by Theorem 4.17 (b)) the orthogonal ofTvU is the space of linear forms spanned bydÕ_1(v), . . . , dÕm(v), it follows thatdJ(v) must be a linear combination of thedÕi(v). Therefore, when 0 is a regular value ofÕ, Theorem 29.3 asserts that ifu∈U is a local extremum ofJ, thendJ(u) must vanish on the tangent spaceTuU. We can say even more. The subsetZ(J) of & given by

Z (J) ={v∈ &|J(v) =J(u)} (the level set of levelJ(u)) is a hypersurface in &, and ifdJ(u) = 0, the zero locus ofdJ(u) is the tangent spaceTuZ(J) toZ(J) atu (a vector space of dimensionn− 1), where

TuZ(J) ={w∈ Rn dJ(u)(w) = 0}.|

Consequently, Theorem 29.3 asserts that

TuU⊆TuZ(J);

this is a geometric condition.

The beauty of the Lagrangian is that the constraints{Õi(v) = 0} have been incorporated into the functionL(v, λ), and that the necessary condition for the existence of a constrained local extremum ofJ is reduced to the necessary condition for the existence of a local extremum of the unconstrainedL.

However, one should be careful to check that the assumptions of Theorem 29.3 are satisfied (in particular, the linear independence of the linear formsdÕi). For example, let

J: R3 R be given by→ J(x, y, z) =x +y +z2

andg: R3 R by→ g(x, y, z) =x^2 +y^2.

Sinceg(x, y, z) = 0 iffx =y = 0, we haveU ={(0,0, z)|z∈ R} and the restriction ofJ to U is given by

J(0,0, z) =z^2,

which has a minimum forz = 0. However, a “blind” use of Lagrange multipliers would require that there is someλ so that

∂J(0,0, z) =λ∂g(0,0, z), ∂J(0,0, z) =λ∂g(0,0, z), ∂J (0,0, z) =λ∂g(0,0, z),∂x ∂x ∂y ∂y ∂z ∂z

and since ∂g(x, y, z) = 2x, ∂g(x, y, z) = 2y, ∂g(0,0, z) = 0,∂x ∂y ∂z

the partial derivatives above all vanish forx =y = 0, so at a local extremum we should also

have ∂J(0,0, z) = 0, ∂J(0,0, z) = 0, ∂J (0,0, z) = 0,∂x ∂y ∂z

but this is absurd since

∂J(x, y, z) = 1, ∂J(x, y, z) = 1, ∂J (x, y, z) = 2z.∂x ∂y ∂z

The reader should enjoy finding the reason for the flaw in the argument.

One should also keep in mind that Theorem 29.3 gives only a necessary condition. The (u, λ) may not correspond to local extrema! Thus, it is always necessary to analyze the local behavior ofJ near a critical pointu. This is generally difficult, but in the case whereJ is affine or quadratic and the constraints are affine or quadratic, this is possible (although not always easy).

Let us apply the above method to the following example in whichE_1 = R,E_2 = R, & = R^2, and

J(x_1, x_2) =−x_2 Õ(x_1, x_2) =x^2 +x^2 1.1 2− Observe that

U ={(x1, x2)∈ R2 x2 +x2 = 1}is the unit circle, and since| 1 2 Õ(x_1, x_2) =2x1 ,2x2

it is clear that Õ(x_1, x_2) = 0 for every point = (x_1, x_2) on the unit circle. If we form the Lagrangian

L(x_1, x_2, λ) =−x_2 +λ(x^2 +x^2 1),1 2−

Theorem 29.3 says that a necessary condition forJ to have a constrained local extremum is that L(x_1, x_2, λ) = 0, so the following equations must hold:

2λx_1 = 0 1 + 2λx_2 = 0− x2 +x2 = 1.1 2

The second equation implies thatλ = 0, and then the first yieldsx_1 = 0, so the third yields x_2 =±1, and we get two solutions:

λ = 1, (x_1, x_2) = (0,1)

2

λ =1, (x_1, x_2) = (0,−1).−2

We can check immediately that the first solution is a minimum and the second is a maximum. The reader should look for a geometric interpretation of this problem.

Let us now consider the case in whichJ is a quadratic function of the form

J(v) = 1v Av−v b,

2

whereA is ann×n symmetric matrix,b∈ Rn, and the constraints are given by a linear system of the form

Cv =d, whereC is anm×n matrix withm < n andd∈ Rm. We also assume thatC has rankm. In this case, the functionÕ is given by

Õ(v) = (Cv−d) ,

because we viewÕ(v) as a row vector (andv as a column vector), and since

dÕ(v)(w) =C w,

the condition that the Jacobian matrix ofÕ atu have rankm is satisfied. The Lagrangian of this problem is

L(v, λ) = 1v Av−v b + (Cv−d) λ = 1v Av−v b +λ (Cv−d),

2 2

whereλ is viewed as a column vector. Now, becauseA is a symmetric matrix, it is easy to show that

L(v, λ) = Av−b +C λ.Cv−d

Therefore, the necessary condition for contrained local extrema is

Av +C λ =b Cv =d,

which can be expressed in matrix form as

A C v= b ,C 0 λ d

where the matrix of the system is a symmetric matrix. We should not be surprised to find the system of Section 18, except for some renaming of the matrices and vectors involved. As we know from Section 18.2, the functionJ has a minimum iffA is positive definite, so in general, ifA is only a symmetric matrix, the critical points of the Lagrangian do not correspond to extrema ofJ.

We now investigate conditions for the existence of extrema involving the second derivative ofJ.

29.2 Using Second Derivatives to Find Extrema

For the sake of brevity, we consider only the case of local minima; analogous results are obtained for local maxima (replaceJ by−J, since maxuJ(u) =−minu−J(u)). We begin with a necessary condition for an unconstrained local minimum.

29.2. USING SECOND DERIVATIVES TO FIND EXTREMA 823

Proposition 29.4. LetE be a normed vector space and letJ: &→ R be a function, with & some open subset ofE. If the functionJ is differentiable in &, ifJ has a second derivative D^2J(u) at some pointu∈ &, and ifJ has a local minimum atu, then

D^2J(u)(w, w)≥ 0 for allw∈E.

Proof. Pick any nonzero vectorw∈E. Since & is open, fort small enough,u+tw∈ & and J(u +tw)≥J(u), so there is some open intervalI⊆ R such that

u +tw∈ & and J(u +tw)≥J(u)

for allt∈I. Using the Taylor–Young formula and the fact that we must havedJ(u) = 0 sinceJ has a local minimum atu, we get

0 t2 2 (tw),≤J(u +tw)−J(u) = 2 D^2J(u)(w, w) +t^2 w

with limt→0 (tw) = 0, which implies that

D^2J(u)(w, w)≥ 0.

Since the argument holds for allw∈E (trivially ifw = 0), the proposition is proved.

One should be cautioned that there is no converse to the previous proposition. For example, the functionf :x→x^3 has no local minimum at 0, yetdf(0) = 0 and D^2f(0)(u, v) = 0. Similarly, the reader should check that the functionf : R^2 R given by→

f

(

x, y

) =

x

2 3y^3

−

has no local minimum at (0,0); yetdf(0,0) = 0 and D^2f(0,0)(u, v) = 2u^2 0.≥ When E = Rn, Proposition 29.4 says that a necessary condition for having a local minimum is that the Hessian^2J(u) be positive semidefinite (it is always symmetric). We now give sufficient conditions for the existence of a local minimum.

Theorem 29.5. LetE be a normed vector space, letJ: &→ R be a function with & some open subset ofE, and assume thatJ is differentiable in & and thatdJ(u) = 0 at some point u∈ &. The following properties hold:

(1) If D^2J(u) exists and if there is some numberα∈ R such thatα > 0 and

D^2J(u)(w, w)≥α w^2 for allw∈E,

thenJ has a strict local minimum atu. (2) If D^2J(v) exists for allv∈ & and if there is a ballB⊆ & centered atu such that

D^2J(v)(w, w)≥ 0 for allv∈B and allw∈E,

thenJ has a local minimum atu. Proof. (1) Using the formula of Taylor–Young, for every vectorw small enough, we can write

J(u +w) ) = 12D^2J(u)(w, w) + w^2 (w)−J(u

1α + (w) w^2 ≥ 2

with limw→0 (w) = 0. Consequently if we pickr > 0 small enough that| (w)|< α for allw with w < r, thenJ(u+w)> J(u) for allu+w∈B, whereB is the open ball of centeru and radiusr. This proves thatJ has a local strict minimum atu.

(2) The formula of Taylor–Maclaurin shows that for allu +w∈B, we have

J(u +w) =J(u) + 12D^2J(v)(w, w)≥J(u),

for somev∈]u, w +w[.

There are no converses of the two assertions of Theorem 29.5. However, there is a condition on D^2J(u) that implies the condition of part (1). Since this condition is easier to state whenE = Rn, we begin with this case.

Recall that an×n symmetric matrixA is positive definite ifx Ax > 0 for allx∈ Rn 0}. In particular,A must be invertible.−{ Proposition 29.6. For any symmetric matrixA, ifA is positive definite, then there is some α > 0 such that

x Ax≥α x^2 for allx∈ Rn.

Proof. Pick any norm inn (recall that all norms on Rn are equivalent). Since the unit R

sphereSn−1 ={x∈ Rn x = 1} is compact and since the functionf(x) =x Ax is never

zero on

S

n

−

1

|

, the function f has a minimumα > 0 onSn−1. Using the usual trick that x = x (x/ x ) for every nonzero vectorx∈ Rn and the fact that the inequality of the proposition is trivial forx = 0, from

x Ax≥α for allx with x = 1,

we get x Ax≥α x^2 for allx∈ Rn, as claimed.

We can combine Theorem 29.5 and Proposition 29.6 to obtain a useful sufficient condition for the existence of a strict local minimum. First let us introduce some terminology. Given a functionJ: &→ R as before, say that a pointu∈ & is a nondegenerate critical point ifdJ(u) = 0 and if the Hessian matrix^2J(u) is invertible.

Proposition 29.7. LetJ: &→ R be a function defined on some open subset &⊆ Rn. If J is differentiable in & and if some pointu∈ & is a nondegenerate critical point such that 2J(u) is positive definite, thenJ has a strict local minimum atu.

Remark: It is possible to generalize Proposition 29.7 to infinite-dimensional spaces by finding a suitable generalization of the notion of a nondegenerate critical point. Firstly, we assume thatE is a Banach space (a complete normed vector space). Then, we define the dualE ofE as the set of continuous linear forms onE, so thatE =L(E;R). Following Lang, we use the notationE for the space of continuous linear forms to avoid confusion with the spaceE∗ = Hom(E,R) of all linear maps fromE to R. A continuous bilinear map Õ:E×E→ R inL2(E, E;R) yields a map Φ fromE toE given by

Φ(u) =Õu,

whereÕu∈E is the linear form defined by

Õu(v) =Õ(u, v).

It is easy to check that Õu is continuous and that the map Φ is continuous. Then, we say thatÕ is nondegenerate iff Φ:E→E is an isomorphism of Banach spaces, which means that Φ is invertible and that both Φ and Φ−1 are continuous linear maps. Given a function J: &→ R differentiable on & as before (where & is an open subset ofE), if D^2J(u) exists for someu∈ &, we say thatu is a nondegenerate critical point ifdJ(u) = 0 and if D^2J(u) is nondegenerate. Of course, D^2J(u) is positive definite if D^2J(u)(w, w)> 0 for allw∈E−{0}.

Using the above definition, Proposition 29.6 can be generalized to a nondegenerate positive definite bilinear form (on a Banach space) and Theorem 29.7 can also be generalized to the situation whereJ: &→ R is defined on an open subset of a Banach space. For details and proofs, see Cartan [18] (Part I Chapter 8) and Avez [5] (Chapter 8 and Chapter 10).

In the next section, we make use of convexity; both on the domain & and on the function J itself.

29.3 Using Convexity to Find Extrema

We begin by reviewing the definition of a convex set and of a convex function. Definition 29.3. Given any real vector spaceE, we say that a subsetC ofE is convex if eitherC =∅ or if for every pair of pointsu, v∈C,

(1−λ)u +λv∈C for allλ∈ R such that 0≤λ≤ 1.

IfC is a nonempty convex subset ofE, a functionf :C→ R is convex (onC) if for every pair of pointsu, v∈C,

f((1−λ)u +λv)≤ (1−λ)f(u) +λf(v) for allλ∈ R such that 0≤λ≤ 1;

the functionf is strictly convex (onC) if for every pair of distinct pointsu, v∈C (u =v),

f((1−λ)u +λv)< (1−λ)f(u) +λf(v) for allλ∈ R such that 0< λ < 1.

A functionf :C→ R defined on a convex subsetC is concave (resp. strictly concave) if (−f) is convex (resp. strictly convex).

Given any two pointsu v∈E, the line segment [u, v] is the set

[u, v] ={(1−λ)u +λv∈E|λ∈ R, 0≤λ≤ 1}.

Clearly, a nonempty set C is convex iff [u, v]⊆C wheneveru, v∈C. SubspacesV⊆E of a vector spaceE are convex; affine subspaces, that is, sets of the formu +V , whereV is a subspace ofE andu∈E, are convex. Balls (open or closed) are convex. Given any linear formÕ:E→ R, for any scalarc∈ R, the closed half–spaces

H+ ={u∈E|Õ(u)≥c}, H− ={u∈E|Õ(u)≤c},Õ,c Õ,c

are convex. Any intersection of half–spaces is convex. More generally, any intersection of convex sets is convex.

Linear forms are convex functions (but not strictly convex). Any norm :E→ R+ is a convex function. The max function,

max(x_1, . . . , xn) = max{x_1, . . . , xn}

is convex on Rn. The exponential x→ ecx is strictly convex for any c = 0 (c∈ R). The logarithm function is concave on R+− {0}, and the log-determinant function log det is concave on the set of symmetric positive definite matrices. This function plays an important role in convex optimization. An excellent exposition of convexity and its applications to optimization can be found in Boyd [15].

Here is a necessary condition for a function to have a local minimum with respect to a convex subsetU.

Theorem 29.8. (Necessary condition for a local minimum on a convex subset) LetJ: &→ R be a function defined on some open subset & of a normed vector spaceE and letU⊆ & be a nonempty convex subset. Given anyu∈U, ifdJ(u) exists and ifJ has a local minimum inu with respect toU, then

dJ(u)(v−u)≥ 0 for allv∈U.

Proof. Letv =u+w be an arbitrary point inU. SinceU is convex, we haveu+tw∈U for allt such that 0≤t≤ 1. SincedJ(u) exists, we can write

J(u +tw)−J(u) =dJ(u)(tw) + tw (tw)

with limt→0 (tw) = 0. However, because 0≤t≤ 1,

J(u +tw)−J(u) =t(dJ(u)(w) + w (tw))

and sinceu is a local minimum with respect toU, we haveJ(u +tw)−J(u)≥ 0, so we get

t(dJ(u)(w) + w (tw))≥ 0.

The above implies thatdJ(u)(w)≥ 0, because otherwise we could pickt > 0 small enough so that

dJ(u)(w) + w (tw)< 0,

a contradiction. Since the argument holds for allv =u+w∈U, the theorem is proved.

Observe that the convexity ofU is a substitute for the use of Lagrange multipliers, but we now have to deal with an inequality instead of an equality.

Consider the special case where U is a subspace ofE. In this case, sinceu∈U we have 2u∈U, and for anyu +w∈U, we must have 2u− (u +w) =u−w∈U. The previous theorem implies thatdJ(u)(w)≥ 0 anddJ(u)(−w)≥ 0, that is,dJ(w)≤ 0, sodJ(w) = 0. Since the argument holds forw∈U (becauseU is a subspace, ifu, w∈U, thenu+w∈U), we conclude that

dJ(u)(w) = 0 for allw∈U.

We will now characterize convex functions when they have a first derivative or a second derivative.

Proposition 29.9. (Convexity and first derivative) Letf : &→ R be a function differentiable on some open subset & of a normed vector spaceE and letU⊆ & be a nonempty convex subset.

(1) The functionf is convex onU iff

f(v)≥f(u) +df(u)(v−u) for allu, v∈U. (2) The functionf is strictly convex onU iff f(v)> f(u) +df(u)(v−u) for allu, v∈U withu =v.

Proof. Letu, v∈U be any two dictinct points and pickλ∈ R with 0< λ < 1. If the functionf is convex, then

f((1−λ)u +λv)≤ (1−λ)f(u) +λf(v), which yields f((1−λ)u +λv)−f(u)f(v)−f(u).λ ≤

It follows that

df(u)(v−u) = limf((1−λ)u +λv)−f(u) f(v)−f(u). λ→0 λ ≤

If f is strictly convex, the above reasoning does not work, because a strict inequality is not necessarily preserved by “passing to the limit.” We have recourse to the following trick: For anyω such that 0< ω < 1, observe that

(1

−

λ

)

u

+

λv

=

u

+

λ

(

v

−

u

) =

ω

λ

−ω u +λ(u +ω(v−u)).ω

If we assume that 0< λ≤ω, the convexity off yields

λ λ

f(u +λ(vω−ω f(u) + ωf(u +ω(v−u)).−u))≤

If we subtractf(u) to both sides, we get

f(u +λ(v−u))−f(u) f(u +ω(v−u))−f(u).

λ ≤ ω

Now, since 0 < ω < 1 andf is strictly convex,

f(u +ω(v−u)) =f((1−ω)u +ωv)< (1−ω)f(u) +ωf(v),

which implies that f(u +ω(v−u))−f(u)< f(v)−f(u),

ω

and thus we get

f(u +λ(v−u))−f(u) f(u +ω(v−u))−f(u)< f(v)−f(u).

λ ≤ ω

If we letλ go to 0, by passing to the limit we get

df(u)(vf(u +ω(v−u))−f(u) < f(v)−f(u),−u)≤ ω

which yields the desired strict inequality.

Let us now consider the converse of (1); that is, assume that

f(v)≥f(u) +df(u)(v−u) for allu, v∈U. For any two distinct pointsu, v∈U and for anyλ with 0< λ < 1, we get f(v)≥f(v +λ(v−u))−λdf(v +λ(u−v))(u−v) f(u)≥f(v +λ(u−v)) + (1−λ)df(v +λ(u−v))(u−v), and if we multiply the first inequality by 1−λ and the second inequality byλ and them add up the resulting inequalities, we get

(1−λ)f(v) +λf(u)≥f(v +λ(u−v)) =f((1−λ)v +λu), which proves thatf is convex.

The proof of the converse of (2) is similar, except that the inequalities are replaced by strict inequalities.

We now establish a convexity criterion using the second derivative off. This criterion is often easier to check than the previous one.

Proposition 29.10. (Convexity and second derivative) Letf : &→ R be a function twice differentiable on some open subset & of a normed vector spaceE and letU⊆ & be a nonempty convex subset.

(1) The functionf is convex onU iff

D^2f(u)(v−u, v−u)≥ 0 for allu, v∈U.

(2) If D^2f(u)(v−u, v−u)> 0 for allu, v∈U withu =v, thenf is strictly convex.

Proof. First, assume that the inequality in condition (1) is satisfied. For any two distinct pointsu, v∈U, the formula of Taylor–Maclaurin yields

f(v) ) = 12D^2(w)(v−u, v−u)−f(u)−df(u)(v−u

=

ρ

2

2 D^2(w)(v−w, v−w),

for some w = (1−λ)u +λv =u +λ(v−u) with 0< λ < 1, and withρ = 1/(1−λ)> 0, so thatv−u =ρ(v−w). Since D^2f(u)(v−w, v−w)≥ 0 for allu, w∈U, we conclude by applying Theorem 29.9(1).

Similarly, if (2) holds, the above reasoning and Theorem 29.9(2) imply thatf is strictly convex.

To prove the necessary condition in (1), define g: &→ R by

g(v) =f(v)−df(u)(v),

whereu∈U is any point considered fixed. Iff is convex and iff has a local minimum atu with respect toU, since

g(v)−g(u) =f(v)−f(u)−df(u)(v−u),

Theorem 29.9 implies that f(v)−f(u)− df(u)(v−u)≥ 0, which implies thatg has a local minimum atu with respect to allv∈U. Therefore, we havedg(u) = 0. Observe thatg is twice differentiable in & and D^2g(u) = D^2f(u), so the formula of Taylor–Young yields for everyv =u +w∈U and allt with 0≤t≤ 1,

t

2

0^2(u)(tw, tw) + tw^2 (tw)≤g(u +tw)−g(u) = 2 D

=t2 2(u)(w, w) + 2 w^2 (wt)),2 (D

with limt→0 (wt) = 0, and fort small enough, we must have D^2(u)(w, w)≥ 0, as claimed.

The converse of Theorem 29.10 (2) is false as we see by considering the functionf given byf(x) =x^4. On the other hand, iff is a quadratic function of the form

f(u) = 1u Au−u b

2

where A is a symmetric matrix, we know that

df(u)(v) =v (Au−b),

so

f

(

v

)

−

f

(

u

)

−

df

(

u

)(

v

−

u

) = 1

v

Av

−

v

b

1

−

2

u Au +u b− (v−u) (Au−b)

2

= 1

v

Av

1

−

2

u Au− (v−u) Au

2

= 1v Av + 1u Au−v Au

2 2

= 12(v−u) A(v−u).

Therefore, Theorem 29.9 implies that ifA is positive semidefinite, thenf is convex and ifA is positive definite, thenf is strictly convex. The converse follows by Theorem 29.10.

We conclude this section by applying our previous theorems to convex functions defined on convex subsets. In this case, local minima (resp. local maxima) are global minima (resp. global maxima).

Definition 29.4. Letf :E→ R be any function defined on some normed vector space (or more generally, any set). For anyu∈E, we say thatf has a minimum inu (resp. maximum inu) if

f(u)≤f(v) (resp.f(u)≥f(v)) for allv∈E. We say thatf has a strict minimum inu (resp. strict maximum inu) if

f(u)< f(v) (resp.f(u)> f(v)) for allv∈E− {u}.

IfU⊆E is a subset ofE andu∈U, we say thatf has a minimum inu (resp. strict minimum inu) with respect toU if

f(u)≤f(v) for allv∈U (resp.f(u)< f(v) for allv∈U− {u}),

and similarly for a maximum inu (resp. strict maximum inu) with respect toU with≤changed to≥ and< to>.

Sometimes, we say global maximum (or minimum) to stress that a maximum (or a minimum) is not simply a local maximum (or minimum).

Theorem 29.11. Given any normed vector spaceE, letU be any nonempty convex subset ofE.

(1) For any convex functionJ:U→ R, for anyu∈U, ifJ has a local minimum atu in U, thenJ has a (global) minimum atu inU.

(2) Any strict convex functionJ:U→ R has at most one minimum (inU), and if it does, then it is a strict minimum (inU).

(3) Let J: &→ R be any function defined on some open subset & ofE withU⊆ & and assume that J is convex onU. For any pointu∈U, ifdJ(u) exists, thenJ has a minimum inu with respect toU iff

dJ(u)(v−u)≥ 0 for allv∈U.

(4) If the convex subsetU in (3) is open, then the above condition is equivalent to

dJ(u) = 0.

Proof. (1) Letv =u +w be any arbitrary point inU. SinceJ is convex, for allt with 0≤t≤ 1, we have

J(u +tw) =J(u +t(v−u))≤ (1−t)J(u) +tJ(v),

which yields J(u +tw)−J(u)≤t(J(v)−J(u)). BecauseJ has a local minimum inu, there is somet_0 with 0< t_0< 1 such that

0≤J(u +t_0w)−J(u),

which implies thatJ(v)−J(u)≥ 0.

(2) IfJ is strictly convex, the above reasoning withw = 0 shows that there is somet_0 with 0< t_0< 1 such that

0≤J(u +t_0w)−J(u)< t_0(J(v)−J(u)),

which shows thatu is a strict global minimum (inU), and thus that it is unique.

(3) We already know from Theorem 29.9 that the condition dJ(u)(v−u)≥ 0 for allv∈U is necessary (even ifJ is not convex). Conversely, becauseJ is convex, careful inspection of the proof of part (1) of Proposition 29.9 shows that only the fact thatdJ(u) exists in needed to prove that

J(v)−J(u)≥dJ(u)(v−u) for allv∈U, and if

dJ(u)(v−u)≥ 0 for allv∈U, then

J(v)−J(u)≥ 0 for allv∈U,

as claimed.

(4) IfU is open, then for everyu∈U we can find an open ballB centered atu of radius small enough so thatB⊆ U. Then, for anyw = 0 such that w < , we have both v =u +w∈B andv =u−w∈B, so condition (3) implies that

dJ(u)(w)≥ 0 and dJ(u)(−w)≥ 0,

which yields dJ(u)(w) = 0.

Since the above holds for allw = 0 such such that w < and sincedJ(u) is linear, we leave it to the reader to fill in the details of the proof thatdJ(u) = 0.

Theorem 29.11 can be used to rederive the fact that the least squares solutions of a linear systemAx =b (whereA is anm×n matrix) are given by the normal equation

A Ax =A b.

For this, we condider the quadratic function

J(v) = 1 Av−b2 1 b^2, 2^2− 2^2 29.4. SUMMARY 833

and our least squares problem is equivalent to finding the minima ofJ onRn. A computation reveals that

J(v) = 1v A Av−v B b,

2

and so

dJ(u) =A Au−B b.

SinceB B is positive semidefinite, the functionJ is convex, and Theorem 29.11(4) implies that the minima ofJ are the solutions of the equation

A Au−A b = 0.

The considerations in this chapter reveal the need to find methods for finding the zeros of the derivative map

dJ: &→E ,

where & is some open subset of a normed vector space E andE is the space of all continuous linear forms onE (a subspace ofE∗). Generalizations of Newton’s method yield such methods and they are the objet of the next chapter.

29.4 Summary

The main concepts and results of this chapter are listed below:

Local minimum, local maximum, local extremum, strict local minimum, strict local• maximum.

• Necessary condition for a local extremum involving the derivative; critical point.

•

Local minimum with respect to a subsetU, local maximum with respect to a subsetU, local extremum with respect to a subsetU.

• Constrained local extremum.

• Necessary condition for a constrained extremum.

• Necessary condition for a constrained extremum in terms of Lagrange multipliers.

• Lagrangian.

• Critical points of a Lagrangian.

•

Necessary condition of an unconstrained local minimum involving the second-order derivative.

• Sufficient condition for a local minimum involving the second-order derivative.

• A sufficient condition involving nondegenerate critical points.

•

Convex sets, convex functions, concave functions, strictly convex functions, strictly concave functions,

• Necessary condition for a local minimum on a convex set involving the derivative.

• Convexity of a function involving a condition on its first derivative.

• Convexity of a function involving a condition on its second derivative.

• Minima of convex functions on convex sets.

Chapter 30 Newton’s Method and its Generalizations

30.1 Newton’s Method for Real Functions of a Real Argument

In Chapter 29 we investigated the problem of determining when a function J: &→ R defined on some open subset & of a normed vector spaceE has a local extremum. Proposition 29.1 gives a necessary condition whenJ is differentiable: ifJ has a local extremum atu∈ &, then we must have

J (u) = 0.

Thus, we are led to the problem of finding the zeros of the derivative

J : &→E ,

whereE =L(E;R) is the set of linear continuous functions fromE to R; that is, the dual ofE , as defined in the Remark after Proposition 29.7.

This leads us to consider the problem in a more general form, namely: Given a function f : &→Y from an open subset & of a normed vector spaceX to a normed vector spaceY , find

(i) Sufficient conditions which guarantee the existence of a zero of the functionf; that is, an elementa∈ & such thatf(a) = 0.

(ii) An algorithm for approximating such ana, that is, a sequence (xk) of points of & whose limit isa.

WhenX =Y = R, we can use Newton’s method. We pick some initial elementx0∈ R “close enough” to a zeroa off, and we define the sequence (xk) by

xf(xk),k+1 =xk− f (xk)

835 for allk≥ 0, provided thatf (xk) = 0. The idea is to definexk+1 as the intersection of the x-axis with the tangent line to the graph of the functionx→f(x) at the point (xk, f(xk)). Indeed, the equation of this tangent line is

y−f(xk) =f (xk)(x−xk),

and its intersection with thex-axis is obtained fory = 0, which yields

x =x f(xk),k− f (xk)

as claimed.

For example, ifα > 0 andf(x) =x^2 α, Newton’s method yields the sequence−

xk+1 = 1 xk + α

2 xk

to compute the square root√α ofα. It can be shown that the method converges to√α for anyx_0> 0. Actually, the method also converges whenx_0< 0! Find out what is the limit.

The case of a real function suggests the following method for finding the zeros of a functionf : &→Y , with &⊆X: given a starting pointx0∈ &, the sequence (xk) is defined by

xk+1 =xk− (f (xk))−1(f(xk))

for allk≥ 0.

For the above to make sense, it must be ensured that

(1) All the pointsxk remain within &.

(2) The functionf is differentiable within &.

(3) The derivativef (x) is a bijection fromX toY for allx∈ &.

These are rather demanding conditions but there are sufficient conditions that guarantee that they are met. Another practical issue is that irt may be very cotstly to compute (f (xk))−1 at every iteration step. In the next section, we investigate generalizations of Newton’s method which address the issues that we just discussed.

30.2 Generalizations of Newton’s Method

Suppose thatf : &→ Rn is given byn functionsfi: &→ R, where &⊆ Rn. In this case, finding a zeroa off is equivalent to solving the system

f_1(a_1. . . , an) = 0

f_2(a_1. . . , an) = 0

.

fn(a_1. . . , an) = 0. A single iteration of Newton’s method consists in solving the linear system

(J(f)(xk)) k =−f(xk),

and then setting xk+1 =xk + k, whereJ(f)(xk) = (∂fi(xk)) is the Jacobian matrix off atxk.∂xj

In general, it is very costly to compute J(f)(xk) at each iteration and then to solve the corresponding linear system. If the method converges, the consecutive vectorsxk should differ only a little, as also the corresponding matricesJ(f)(xk). Thus, we are led to a variant of Newton’s method which consists in keeping the same matrix forp consecutive steps (where p is some fixed integer≥ 2):

xk+1 =xk− (f (x_0))−1(f(xk)), 0≤k≤p− 1 xk+1 =xk− (f (xp))−1(f(xk)), p≤k≤ 2p− 1 .

xk+1 =xk− (f (xrp))−1(f(xk)), rp≤k≤ (r + 1)p− 1 .

It is also possible to setp =∞, that is, to use the same matrixf (x_0) for all iterations, which leads to iterations of the form

xk+1 =xk− (f (x_0))−1(f(xk)), k≥ 0,

or even to replacef (x_0) by a particular matrixA_0 which is easy to invert:

xk+1 =xk−A− 1 f(xk), k≥ 0.0

In the last two cases, if possible, we use an LU factorization off (x_0) orA_0 to speed up the method. In some cases, it may even possible to setA_0 =I.

The above considerations lead us to the definition of a generalized Newton method, as in Ciarlet [22] (Chapter 7). Recall that a linear mapf∈ L(E;F) is called an isomorphism iff f is continuous, bijective, andf−1 is also continuous.

Definition 30.1. IfX andY are two normed vector spaces and iff : &→Y is a function from some open subset & ofX, a generalized Newton method for finding zeros off consists of

(1) A sequence of families (Ak(x)) of linear isomorphisms fromX toY , for allx∈ & and all integersk≥ 0;

(2) Some starting pointx0∈ &; (3) A sequence (xk) of points of & defined by

xk+1 =xk− (Ak(x ))−1(f(xk)), k≥ 0,

where for every integerk≥ 0, the integer satisfies the condition

0≤ ≤k. The functionAk(x) usually depends onf .

Definition 30.1 gives us enough flexibility to capture all the situations that we have previously discussed:

Ak(x) =f (x), =k

Ak(x) =f (x), = min{rp, k}, ifrp≤k≤ (r + 1)p− 1, r≥ 0 Ak(x) =f (x), = 0

Ak(x) =A_0,

where A_0 is a linear isomorphism fromX toY . The first case corresponds to Newton’s orginal method and the others to the variants that we just discussed. We could also have Ak(x) =Ak, a fixed linear isomorphism independent ofx∈ &.

The following theorem inspired by the Newton–Kantorovich theorem gives sufficient conditions that guarantee that the sequence (xk) constructed by a generalized Newton method converges to a zero off close tox_0. Althoug quite technical, these conditions are not very surprising.

Theorem 30.1. LetX be a Banach space, letf : &→Y be differentiable on the open subset &⊆X, and assume that there are constantsr, M, β > 0 such that if we let

B ={x∈X| x−x0 ≤r} ⊆ &,

then (1) supsup A− 1 x)(Y ;X)≤M,

k

≥

0

x

∈

B

k

(

L

(2) β < 1 and sup

sup

f

(

x

)

−Ak(x ) (X;Y )≤ M≥0 x,x∈B L β

k

(3)

f(xr (1−β).0) ≤ M

Then, the sequence (xk) defined by xk+1 =xk−A− 1 x )(f(xk)), 0≤ ≤kk (

is entirely contained withinB and converges to a zeroa off, which is the only zero off in B. Furthermore, the convergence is geometric, which means that

x

x x1− 0 k.k−a ≤ 1−β β

A proof of Theorem 30.1 can be found in Ciarlet [22] (Section 7.5). It is not really difficult but quite technical.

If we assume that we already know that some element a∈ & is a zero off, the next theorem gives sufficient conditions for a special version of a generalized Newton method to converge. For this special method, the linear isomorphismsAk(x) are independent ofx∈ &.

Theorem 30.2. LetX be a Banach space, and letf : &→Y be differentiable on the open subset &⊆X. Ifa∈ & is a point such thatf(a) = 0, iff (a) is a linear isomorphism, and if there is some λ with 0< λ < 1/2 such that

sup

A

k−f (a) (X;Y )≤ (f (a))−1 (Y ;X)≥0 L

λ,

k

L

then there is a closed ballB of centera such that for everyx0∈B, the sequence (xk) defined by

xk+1 =xk−A−1 f(xk)), k≥ 0,k (

is entirely contained withinB and converges toa, which is the only zero off inB. Furthermore, the convergence is geometric, which means that

xk x0−a ,k−a ≤β

for someβ < 1.

A proof of Theorem 30.2 can be also found in Ciarlet [22] (Section 7.5).

For the sake of completeness, we state a version of the Newton–Kantorovich theorem, which corresponds to the case whereAk(x) =f (x). In this instance, a stronger result can be obtained especially regarding upper bounds, and we state a version due to Gragg and Tapia which appears in Problem 7.5-4 of Ciarlet [22].

Theorem 30.3. (Newton–Kantorovich) LetX be a Banach space, and letf : &→Y be differentiable on the open subset &⊆X. Assume that there exist three positive constants λ, µ, ν and a pointx0∈ & such that

1 0< λµν≤ 2, and if we let

ρ

−

= 1

−√

1− 2λµν µν

+ = 1 +√1− 2λµνρ µν

B ={x∈X| x−x_0 < ρ−}&+ ={x∈ &| x−x0 < ρ+ ,}

thenB⊆ &,f (x_0) is an isomorphism ofL(X;Y ), and

(f (x_0))−1 µ,

(f (x_0))−1f(x_0)≤λ,

sup f (x)−f (y) ≤ν x−y . x,y∈&+ ≤

Then,f (x) is isomorphism ofL(X;Y ) for allx∈B, and the sequence defined by

xk+1 =xk− (f (xk))−1(f(xk)), k≥ 0

is entirely contained within the ballB and converges to a zeroa off which is the only zero off in &+. Finally, if we writeθ =ρ−/ρ+, then we have the following bounds:

x

k

−

a

≤

λµν

1

−

θ

2

k

x

1

−

x

0

if λµν < 1− 2λµν θ2k2√1

2 x

k

−

a

x1−x_0

≤

2

k

−

1

if λµν = 1,

2

and 2 xk+1−xk xk−a ≤θ2k−1 xk−xk−1 .1 + (1 + 4θ2k(1 +θ2k)−2)≤

We can now specialize Theorems 30.1 and 30.2 to the search of zeros of the derivative f : &→E , of a functionf : &→ R, with &⊆E. The second derivativeJ ofJ is a continuous bilinear formJ :E×E→ R, but is is convenient to view it as a linear map inL(E, E ); the continuous linear formJ (u) is given byJ (u)(v) =J (u, v). In our next theorem, we assume that theAk(x) are isomorphisms inL(E, E ).

Theorem 30.4. LetE be a Banach space, letJ: &→ R be twice differentiable on the open subset &⊆E, and assume that there are constantsr, M, β > 0 such that if we let

B ={x∈E| x−x0 ≤r} ⊆ &,

then (1) supsup A− 1 x)(E ;E)≤M,

k

≥

0

x

∈

B

k

(

L

(2) β < 1 and β sup sup J (x)−Ak(x )(E;E )≤ Mk≥0 x,x∈B L

(3)

J (xr (1−β).0) ≤ M

Then, the sequence (xk) defined by

xk+1 =xk−A−1 x )(J (xk)), 0≤ ≤kk (

is entirely contained withinB and converges to a zeroa ofJ , which is the only zero ofJ inB. Furthermore, the convergence is geometric, which means that

x

k

−

a

x_1 x_0

≤

1

−β βk.

−

In the next theorem, we assume that theAk(x) are isomorphisms inL(E, E ) that are independent ofx∈ &.

Theorem 30.5. LetE be a Banach space, and letJ: &→ R be twice differentiable on the open subset &⊆E. Ifa∈ & is a point such thatJ (a) = 0, ifJ (a) is a linear isomorphism, and if there is some λ with 0< λ < 1/2 such that

sup

A

k−J (a) (E;E )≤ (J (a))−1 (E ;E)≥0 Lk

λ,

L

then there is a closed ballB of centera such that for everyx0∈B, the sequence (xk) defined by

x

k

+1

=

x

k

−

A

−

1

(J (xk)), k≥ 0,

k

is entirely contained withinB and converges toa, which is the only zero ofJ inB. Furthermore, the convergence is geometric, which means that

xk−a ≤βk x0−a , for someβ < 1.

WhenE = Rn, the Newton method given by Theorem 30.4 yield an itereation step of the form

x

k

+1

=

x

k

−

A

1

k (x ) J(xk), 0≤ ≤k, where J(xk) is the gradient ofJ atxk (here, we identifyE with Rn). In particular, Newton’s original method picksAk =J , and the iteration step is of the form

xk+1 =xk− (2J(xk))−1 J(xk), k≥ 0,

where^2J(xk) is the Hessian ofJ atxk.

As remarked in [22] (Section 7.5), generalized Newton methods have a very wide range of applicability. For example, various versions of gradient descent methods can be viewed as instances of Newton methods.

Newton’s method also plays an important role in convex optimization, in particular, interior-point methods. A variant of Newton’s method dealing with equality constraints has been developed. We refer the reader to Boyd and Vandenberghe [15], Chapters 10 and 11, for a comprehensive exposition of these topics.

30.3 Summary

The main concepts and results of this chapter are listed below: Newton’s method for functionsf : R→ R.•

• Generalized Newton methods.

The Newton-Kantorovich theorem.•

Chapter 31 Appendix: Zorn’s Lemma; Some Applications

31.1 Statement of Zorn’s Lemma

Zorn’s lemma is a particularly useful form of the axiom of choice, especially for algebraic applications. Readers who want to learn more about Zorn’s lemma and its applications to algebra should consult either Lang [65], Appendix 2,§2 (pp. 878-884) and Chapter III,§5 (pp. 139-140), or Artin [3], Appendix§1 (pp. 588-589). For the logical ramifications of Zorn’s lemma and its equivalence with the axiom of choice, one should consult Schwartz [89], (Vol. 1), Chapter I,§6, or a text on set theory such as Enderton [32], Suppes [103], or Kuratowski and Mostowski [64].

Given a set,S, a partial order,≤, onS is a binary relation onS (i.e.,≤ ⊆S×S) which is

(1) reflexive, i.e.,x≤x, for allx∈S,

(2) transitive, i.e, ifx≤y andy≤z, thenx≤z, for allx, y, z∈S, and (3) antisymmetric, i.e, ifx≤y andy≤x, thenx =y, for allx, y∈S.

A pair ( S,≤), where≤ is a partial order onS, is called a partially ordered set or poset. Given a poset, (S,≤), a subset,C, ofS is totally ordered or a chain if for every pair of elementsx, y∈C, eitherx≤y ory≤x. The empty set is trivially a chain. A subset,P, (empty or not) ofS is bounded if there is someb∈S so thatx≤b for allx∈P. Observe that the empty subset ofS is bounded if and only ifS is nonempty. A maximal element of P is an element,m∈P, so thatm≤x implies thatm =x, for allx∈P. Zorn’s lemma can be stated as follows:

Lemma 31.1. Given a partially ordered set, (S,≤), if every chain is bounded, thenS has a maximal element.

843 Proof. See any of Schwartz [89], Enderton [32], Suppes [103], or Kuratowski and Mostowski [64].

Remark: As we noted, the hypothesis of Zorn’s lemma implies thatS is nonempty (since the empty set must be bounded). A partially ordered set such that every chain is bounded is sometimes called inductive.

We now give some applications of Zorn’s lemma.

31.2 Proof of the Existence of a Basis in a Vector Space

Using Zorn’s lemma, we can prove that Theorem 2.7 holds for arbitrary vector spaces, and not just for finitely generated vector spaces, as promised in Chapter 2.

Theorem 31.2. Given any family,S = (ui)i∈I, generating a vector spaceE and any linearly independent subfamily,L = (uj)j∈J, ofS (whereJ⊆I), there is a basis,B, ofE such that L⊆B⊆S.

Proof. Consider the setL of linearly independent families,B, such thatL⊆B⊆S. Since L∈ L, this set is nonempty. We claim thatL is inductive. Consider any chain, (Bl)l∈Λ, of linearly independent familiesBl inL, and look atB = l∈ΛBl. The familyB is of the form

B = (vh)h∈H, for some index setH, and it must be linearly independent. Indeed, if this was

not true, there would be some family (λh)h∈H of scalars, of finite support, so that

λhvh = 0,

h∈H

where not all λh are zero. SinceB =l∈ΛBl and only finitely manyλh are nonzero, there is a finite subset,F, of Λ, so thatvh∈Bfh iffλh = 0. But (Bl)l∈Λ is a chain, and if we let f = max{fh|fh∈F}, thenvh∈Bf, for allvh for whichλh = 0. Thus,

λhvh = 0

h∈H

would be a nontrivial linear dependency among vectors from Bf, a contradiction. Therefore, B∈ L, and sinceB is obviously an upper bound for theBl’s, we have proved thatL is inductive. By Zorn’s lemma (Lemma 31.1), the setL has some maximal element, say B = (uh)h∈H. The rest of the proof is the same as in the proof of Theorem 2.7, but we repeat it for the reader’s convenience. We claim thatB generatesE. Indeed, ifB does not generateE, then there is someup∈S that is not a linear combination of vectors inB (since S generatesE), withp /H. Then, by Lemma 2.6, the familyB = (uh)h∈H∪{p} is linearly independent, and sinceL⊆ B⊂ B⊆S, this contradicts the maximality ofB. Thus,B is a basis ofE such thatL⊆B⊆S.

Another important application of Zorn’s lemma is the existence of maximal ideals. 31.3. EXISTENCE OF MAXIMAL PROPER IDEALS 845

31.3 Existence of Maximal Ideals Containing a Given Proper Ideal

LetA be a commutative ring with identity element. Recall that an ideal A inA is a proper ideal if A =A. The following theorem holds:

Theorem 31.3. Given any proper ideal, A⊆A, there is a maximal ideal, B, containing A.

Proof. LetI be the set of all proper ideals, B, inA that contain A. The setI is nonempty, since A∈ I. We claim thatI is inductive. Consider any chain (Ai)i∈I of ideals Ai inA. One can easily check that B =i∈I Ai is an ideal. Furthermore, B is a proper ideal, since otherwise, the identity element 1 would belong to B =A, and so, we would have 1∈ Ai for somei, which would imply Ai =A, a contradiction. Also, B is obviously an upper bound for all the Ai’s. By Zorn’s lemma (Lemma 31.1), the setI has a maximal element, say B, and B is a maximal ideal containing A.

Bibliography

[1] Lars V. Ahlfors and Leo Sario. Riemann Surfaces. Princeton Math. Series, No. 2. Princeton University Press, 1960.

[2] Emil Artin. Geometric Algebra. Wiley Interscience, first edition, 1957.

[3] Michael Artin. Algebra. Prentice Hall, first edition, 1991.

[4] M. F. Atiyah and I. G. Macdonald. Introduction to Commutative Algebra. Addison Wesley, third edition, 1969.

[5] A. Avez. Calcul Diff´erentiel. Masson, first edition, 1991.

[6] Marcel Berger. G´eom´etrie 1. Nathan, 1990. English edition: Geometry 1, Universitext, Springer Verlag.

[7] Marcel Berger. G´eom´etrie 2. Nathan, 1990. English edition: Geometry 2, Universitext, Springer Verlag.

[8] Marcel Berger and Bernard Gostiaux. G´eom´etrie diff´erentielle: vari´et´es, courbes et surfaces. Collection Math´ematiques. Puf, second edition, 1992. English edition: Differential geometry, manifolds, curves, and surfaces, GTM No. 115, Springer Verlag.

[9] Rolf Berndt. An Introduction to Symplectic Geometry. Graduate Studies in Mathematics, Vol. 26. AMS, first edition, 2001.

[10] J.E. Bertin. Alg`ebre lin´eaire et g´eom´etrie classique. Masson, first edition, 1981.

[11] Nicolas Bourbaki. Alg`ebre, Chapitre 9. El´ements de Math´ematiques. Hermann, 1968.

[12] Nicolas Bourbaki. Alg`ebre, Chapitres 1-3. El´ements de Math´ematiques. Hermann, 1970.

[13] Nicolas Bourbaki. Alg`ebre, Chapitres 4-7. El´ements de Math´ematiques. Masson, 1981.

[14] Nicolas Bourbaki. Espaces Vectoriels Topologiques. El´ements de Math´ematiques. Masson, 1981.

847 [15] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, first edition, 2004.

[16] Glen E Bredon. Topology and Geometry. GTM No. 139. Springer Verlag, first edition, 1993.

[17] G. Cagnac, E. Ramis, and J. Commeau. Math´ematiques Sp´eciales, Vol. 3, G´eom´etrie. Masson, 1965.

[18] Henri Cartan. Cours de Calcul Diff´erentiel. Collection M´ethodes. Hermann, 1990. [19] Henri Cartan. Differential Forms. Dover, first edition, 2006.

[20] Claude Chevalley. The Algebraic Theory of Spinors and Clifford Algebras. Collected Works, Vol. 2. Springer, first edition, 1997.

[21] Yvonne Choquet-Bruhat, C´ecile DeWitt-Morette, and Margaret Dillard-Bleick. Analysis, Manifolds, and Physics, Part I: Basics. North-Holland, first edition, 1982. [22] P.G. Ciarlet. Introduction to Numerical Matrix Analysis and Optimization. Cambridge University Press, first edition, 1989. French edition: Masson, 1994.

[23] Timoth´ ee Cour and Jianbo Shi. Solving markov random fields with spectral relaxation. In Marita Meila and Xiaotong Shen, editors, Artifical Intelligence and Statistics. Society for Artificial Intelligence and Statistics, 2007.

[24] H.S.M. Coxeter. Introduction to Geometry. Wiley, second edition, 1989. [25] James W. Demmel. Applied Numerical Linear Algebra. SIAM Publications, first edition, 1997.

[26] Jean Dieudonn´e. Alg`ebre Lin´eaire et G´eom´etrie El´ementaire. Hermann, second edition, 1965.

[27] Jacques Dixmier. General Topology. UTM. Springer Verlag, first edition, 1984. [28] Manfredo P. do Carmo. Differential Geometry of Curves and Surfaces. Prentice Hall, 1976.

[29] Manfredo P. do Carmo. Riemannian Geometry. Birkh¨auser, second edition, 1992. [30] David S. Dummit and Richard M. Foote. Abstract Algebra. Wiley, second edition, 1999.

[31] Gerald A. Edgar. Measure, Topology, and Fractal Geometry. Undergraduate Texts in Mathematics. Springer Verlag, first edition, 1992.

[32] Herbert B. Enderton. Elements of Set Theory. Academic Press, 1997. [33] Charles L. Epstein. Introduction to the Mathematics of Medical Imaging. SIAM, second edition, 2007.

[34] Gerald Farin. Curves and Surfaces for CAGD. Academic Press, fourth edition, 1998. [35] Olivier Faugeras. Three-Dimensional Computer Vision, A geometric Viewpoint. the MIT Press, first edition, 1996.

[36] James Foley, Andries van Dam, Steven Feiner, and John Hughes. Computer Graphics. Principles and Practice. Addison-Wesley, second edition, 1993.

[37] David A. Forsyth and Jean Ponce. Computer Vision: A Modern Approach. Prentice Hall, first edition, 2002.

[38] Jean Fresnel. M´ethodes Modernes En G´eom´etrie. Hermann, first edition, 1998. [39] William Fulton. Algebraic Topology, A first course. GTM No. 153. Springer Verlag, first edition, 1995.

[40] William Fulton and Joe Harris. Representation Theory, A first course. GTM No. 129. Springer Verlag, first edition, 1991.

[41] Jean H. Gallier. Curves and Surfaces In Geometric Modeling: Theory And Algorithms. Morgan Kaufmann, 1999.

[42] Jean H. Gallier. Geometric Methods and Applications, For Computer Science and Engineering. TAM, Vol. 38. Springer, second edition, 2011.

[43] Walter Gander, Gene H. Golub, and Urs von Matt. A constrained eigenvalue problem. Linear Algebra and its Applications, 114/115:815–839, 1989.

[44] F.R. Gantmacher. The Theory of Matrices, Vol. I. AMS Chelsea, first edition, 1977. [45] Roger Godement. Cours d’Alg`ebre. Hermann, first edition, 1963.

[46] Gene H. Golub. Some modified eigenvalue problems. SIAM Review, 15(2):318–334, 1973.

[47] H. Golub, Gene and F. Van Loan, Charles. Matrix Computations. The Johns Hopkins University Press, third edition, 1996.

[48] A. Gray. Modern Differential Geometry of Curves and Surfaces. CRC Press, second edition, 1997.

[49] Donald T. Greenwood. Principles of Dynamics. Prentice Hall, second edition, 1988. [50] Larry C. Grove. Classical Groups and Geometric Algebra. Graduate Studies in Mathematics, Vol. 39. AMS, first edition, 2002.

[51] Jacques Hadamard. Le¸cons de G´eom´etrie El´ementaire. I G´eom´etrie Plane. Armand Colin, thirteenth edition, 1947.

[52] Jacques Hadamard. Le¸cons de G´eom´etrie El´ementaire. II G´eom´etrie dans l’Espace. Armand Colin, eighth edition, 1949.

[53] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, second edition, 2009. [54] D. Hilbert and S. Cohn-Vossen. Geometry and the Imagination. Chelsea Publishing Co., 1952.

[55] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, first edition, 1990.

[56] Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, first edition, 1994.

[57] Nathan Jacobson. Basic Algebra I. Freeman, second edition, 1985.

[58] Ramesh Jain, Rangachar Katsuri, and Brian G. Schunck. Machine Vision. McGrawHill, first edition, 1995.

[59] J¨urgen Jost. Riemannian Geometry and Geometric Analysis. Universitext. Springer Verlag, fourth edition, 2005.

[60] Hoffman Kenneth and Kunze Ray. Linear Algebra. Prentice Hall, second edition, 1971. [61] D. Kincaid and W. Cheney. Numerical Analysis. Brooks/Cole Publishing, second edition, 1996.

[62] Anthony W. Knapp. Lie Groups Beyond an Introduction. Progress in Mathematics, Vol. 140. Birkh¨auser, second edition, 2002.

[63] Erwin Kreyszig. Differential Geometry. Dover, first edition, 1991.

[64] K. Kuratowski and A. Mostowski. Set Theory. Studies in Logic, Vol. 86. Elsevier, 1976.

[65] Serge Lang. Algebra. Addison Wesley, third edition, 1993.

[66] Serge Lang. Differential and Riemannian Manifolds. GTM No. 160. Springer Verlag, third edition, 1995.

[67] Serge Lang. Real and Functional Analysis. GTM 142. Springer Verlag, third edition, 1996.

[68] Serge Lang. Undergraduate Analysis. UTM. Springer Verlag, second edition, 1997. [69] Peter Lax. Linear Algebra and Its Applications. Wiley, second edition, 2007.

[70] Saunders Mac Lane and Garrett Birkhoff. Algebra. Macmillan, first edition, 1967. [71] Ib Madsen and Jorgen Tornehave. From Calculus to Cohomology. De Rham Cohomology and Characteristic Classes. Cambridge University Press, first edition, 1998. [72] M.-P. Malliavin. Alg`ebre Commutative. Applications en G´eom´etrie et Th´eorie des Nombres. Masson, first edition, 1985.

[73] Jerrold E. Marsden and J.R. Hughes, Thomas. Mathematical Foundations of Elasticity. Dover, first edition, 1994.

[74] William S. Massey. Algebraic Topology: An Introduction. GTM No. 56. Springer Verlag, second edition, 1987.

[75] William S. Massey. A Basic Course in Algebraic Topology. GTM No. 127. Springer Verlag, first edition, 1991.

[76] Dimitris N. Metaxas. Physics-Based Deformable Models. Kluwer Academic Publishers, first edition, 1997.

[77] Carl D. Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, first edition, 2000. [78] John W. Milnor. Topology from the Differentiable Viewpoint. The University Press of Virginia, second edition, 1969.

[79] John W. Milnor and James D. Stasheff. Characteristic Classes. Annals of Math. Series, No. 76. Princeton University Press, first edition, 1974.

[80] Shigeyuki Morita. Geometry of Differential Forms. Translations of Mathematical Monographs No 201. AMS, first edition, 2001.

[81] James R. Munkres. Topology, a First Course. Prentice Hall, first edition, 1975. [82] James R. Munkres. Analysis on Manifolds. Addison Wesley, 1991.

[83] Ivan Niven, Herbert S. Zuckerman, and Hugh L. Montgomery. An Introduction to the Theory of Numbers. Wiley, fifth edition, 1991.

[84] Joseph O’Rourke. Computational Geometry in C. Cambridge University Press, second edition, 1998.

[85] Dan Pedoe. Geometry, A comprehensive Course. Dover, first edition, 1988. [86] Eug`ene Rouch´e and Charles de Comberousse. Trait´e de G´eom´etrie. Gauthier-Villars, seventh edition, 1900.

[87] Pierre Samuel. Projective Geometry. Undergraduate Texts in Mathematics. Springer Verlag, first edition, 1988.

[88] Laurent Schwartz. Topologie G´en´erale et Analyse Fonctionnelle. Collection Enseignement des Sciences. Hermann, 1980.

[89] Laurent Schwartz. Analyse I. Th´eorie des Ensembles et Topologie. Collection Enseignement des Sciences. Hermann, 1991.

[90] Laurent Schwartz. Analyse II. Calcul Diff´erentiel et Equations Diff´erentielles. Collection Enseignement des Sciences. Hermann, 1992.

[91] H. Seifert and W. Threlfall. A Textbook of Topology. Academic Press, first edition, 1980.

[92] Denis Serre. Matrices, Theory and Applications. GTM No. 216. Springer Verlag, second edition, 2010.

[93] Jean-Pierre Serre. A Course in Arithmetic. Graduate Text in Mathematics, No. 7. Springer, first edition, 1973.

[94] Igor R. Shafarevich. Basic Algebraic Geometry 1. Springer Verlag, second edition, 1994.

[95] Ernst Snapper and Troyer Robert J. Metric Affine Geometry. Dover, first edition, 1989.

[96] Harold M. Stark. An Introduction to Number Theory. MIT Press, first edition, 1994. Eighth Printing.

[97] G.W. Stewart. On the early history of the singular value decomposition. SIAM review, 35(4):551–566, 1993.

[98] J.J. Stoker. Differential Geometry. Wiley Classics. Wiley-Interscience, first edition, 1989.

[99] Eric J. Stollnitz, Tony D. DeRose, and David H. Salesin. Wavelets for Computer Graphics Theory and Applications. Morgan Kaufmann, first edition, 1996. [100] Gilbert Strang. Introduction to Applied Mathematics. Wellesley-Cambridge Press, first edition, 1986.

[101] Gilbert Strang. Linear Algebra and its Applications. Saunders HBJ, third edition, 1988.

[102] Gilbert Strang and Nguyen Truong. Wavelets and Filter Banks. Wellesley-Cambridge Press, second edition, 1997.

[103] Patrick Suppes. Axiomatic Set Theory. Dover, 1972. [104] Donald E. Taylor. The Geometry of the Classical Groups. Sigma Series in Pure Mathematics, Vol. 9. Heldermann Verlag Berlin, 1992.

[105] Claude Tisseron. G´eom´etries affines, projectives, et euclidiennes. Hermann, first edition, 1994.

[106] L.N. Trefethen and D. Bau III. Numerical Linear Algebra. SIAM Publications, first edition, 1997.

[107] Emanuele Trucco and Alessandro Verri. Introductory Techniques for 3D Computer Vision. Prentice-Hall, first edition, 1998.

[108] B.L. Van Der Waerden. Algebra, Vol. 1. Ungar, seventh edition, 1973. [109] Frank Warner. Foundations of Differentiable Manifolds and Lie Groups. GTM No. 94. Springer Verlag, first edition, 1983.

[110] Ernst Witt. Theorie der quadratischen Formen in beliebigen K¨orpern. J. Reine Angew. Math., 176:31–44, 1936.

[111] Stella X. Yu and Jianbo Shi. Grouping with bias. In Thomas G. Dietterich, Sue Becker, and Zoubin Ghahramani, editors, Neural Information Processing Systems, Vancouver, Canada, 3-8 Dec. 2001. MIT Press, 2001.

[112] Oscar Zariski and Pierre Samuel. Commutative Algebra, Vol I. GTM No. 28. Springer Verlag, first edition, 1975.

This book was distributed courtesy of:

For your own Unlimited Reading and FREE eBooks today, visit:

http://www.Free-eBooks.net

Share this eBook with anyone and everyone automatically by selecting any of the options below:

To show your appreciation to the author and help others have wonderful reading experiences and find helpful information too, we'd be very grateful if you'd kindly

post your comments for this book here.

COPYRIGHT INFORMATION

Free-eBooks.net respects the intellectual property of others. When a book's copyright owner submits their work to Free-eBooks.net, they are granting us permission to distribute such material. Unless otherwise stated in this book, this permission is not passed onto others. As such, redistributing this book without the copyright owner's permission can constitute copyright infringement. If you believe that your work has been used in a manner that constitutes copyright infringement, please follow our Notice and Procedure for Making Claims of Copyright Infringement as seen in our Terms of Service here:

http://www.free-ebooks.net/tos.html

Table of Contents

Chapter 1 Introduction

Chapter 2 Vector Spaces, Bases, Linear Maps

Chapter 3 Matrices and Linear Maps

Chapter 4 Direct Sums, The Dual Space, Duality

Chapter 5 Determinants

Chapter 6 Gaussian Elimination, LU-Factorization, Cholesky Factorization, Reduced Row Echelon Form

Chapter 7 Vector Norms and Matrix Norms

Chapter 8 Iterative Methods for Solving Linear Systems

Chapter 9 Euclidean Spaces

Chapter 10 QR-Decomposition for Arbitrary Matrices

Chapter 11 Hermitian Spaces

Chapter 12 Eigenvectors and Eigenvalues

Chapter 13 Spectral Theorems in Euclidean and Hermitian Spaces

Chapter 14 Bilinear Forms and Their Geometries

Chapter 15 Variational Approximation of Boundary-Value Problems; Introduction to the Finite Elements Method

Chapter 16 Singular Value Decomposition and Polar Form

Chapter 17 Applications of SVD and Pseudo-inverses

Chapter 18 Quadratic Optimization Problems

Chapter 19 Basics of Affine Geometry

19.11 Intersection of Affine Spaces

Chapter 20 Polynomials, Ideals and PID’s

Chapter 21 UFD’s, Noetherian Rings, Hilbert’s Basis Theorem

Chapter 22 Annihilating Polynomials and the Primary Decomposition

Chapter 23 Tensor Algebras, Symmetric Algebras and Exterior Algebras

Chapter 24 Introduction to Modules; Modules over a PID

Chapter 25 The Rational Canonical Form and Other Normal Forms

Chapter 26 Topology

Chapter 27 A Detour On Fractals

Chapter 28 Differential Calculus

Chapter 29 Extrema of Real-Valued Functions

Chapter 30 Newton’s Method and its Generalizations

Chapter 31 Appendix: Zorn’s Lemma; Some Applications