CHAPTER 5 LARGE-SAMPLE PROPERTIES OF THE LSE Chapter 5 Large sample properties of the LsE 5.1 Stochastic convergence Suppose that Xn is a sequence of random varia bles with a corresponding sequence of distribution functions ( FnI If Fn()- F(a) at every continuity point a of F, Fn is said to converge weakly to F, written Fn= F. In this case,I Xn is said to converge in distribution to X where X is a random variable with distribution function f. written Xn X If X is a random variable. and for alle>0 lim p Xn is said to converge in proba bility to x, written Xn X. X is known as the proba bility limit of Xn, written X=plimXn Xn is said to converge in mean square to x, written Xn X Some useful results regarding stochastic convergence are 1. Xn X and g() is a continuous function Example 1 Let 0 with probability 1-i =0. Let g(a)=x+1. Then, g (Xn)g(0) 2. Suppose that Yn=Y and Xn c(a const ant).Then (a)Xn+rn-c+y () n-I when c≠0 3. Xn X and g() is cont inuous →9(Xn)+g(X) (This is called cont inuous mapping theorem
CHAPTER 5 LARGE—SAMPLE PROPERTIES OF THE LSE 1 Chapter 5 Large—sample properties of the LSE 5.1 Stochastic convergence Suppose that {Xn} is a sequence of random variables with a corresponding sequence of distribution functions {Fn} . If Fn (x) → F (x) at every continuity point x of F, Fn is said to converge weakly to F, written Fn ⇒ F. In this case, {Xn} is said to converge in distribution to X where X is a random variable with distribution function F, written Xn d→ X. If X is a random variable, and for all ε > 0 limn→∞ P (|Xn − X| < ε) = 1, Xn is said to converge in probability to X, written Xn P→ X. X is known as the probability limit of Xn, written X =plimXn. If lim E (Xn − X) 2 = 0, Xn is said to converge in mean square to X, written Xn m.s. → X. Some useful results regarding stochastic convergence are: 1. Xn P→ X and g (·) is a continuous function ⇒ g (Xn) P→ g (X). Example 1 Let Xn = 1 with probability 1 n 0 with probability 1 − 1 n . Obviously, Xn P→ 0. Let g (x) = x + 1. Then, g (Xn) P→ g (0) = 1. 2. Suppose that Yn d→ Y and Xn P→ c (a constant). Then (a) Xn + Yn d→ c + Y (b) XnYn d→ cY (c) Yn Xn d→ Y c when c = 0 . 3. Xn d→ X and g (·) is continuous ⇒ g (Xn) d→ g (X). (This is called continuous mapping theorem)
CHAPTER 5 LARGE SAMPLE PROPERTIES OF THE LSE Example 2 If Xn=N(0, 1),X2x2(1) →Yn→X. 5.X= X (The converse is not necessarily true. 9c(a constant →X, →X7 If for any E>0 there exists B. o such that Xn B for all n> 1, write Xn=Op(n").(u is stochastically bounded If plimAp=0, write Xn=Op(nr) The weak law of large numbers 1. Let iXi,i>l be a sequence of i i d. r vs with EX1<∞ Then 元∑X一EX1a5n→∞ 2. Let(Xi, i> 1) be sequence of independent r vs with EX;=m If EXI+<B< ∞(6>0) for all.Then ∑ i-m as n 2=1 Example 3 Let Ei N iid(0, a). Then →E=1=0
CHAPTER 5 LARGE—SAMPLE PROPERTIES OF THE LSE 2 Example 2 If Xn d→ N (0, 1), X2 n d→ χ 2 (1). 4. Xn − Yn P→ 0 and Xn d→ X. ⇒ Yn d→ X. 5. Xn P→ X implies Xn d→ X. (The converse is not necessarily true.) 6. Xn d→ c (a constant) ⇒ Xn P→ c. 7. Xn m.s. → X ⇒ Xn P→ X. If for any ε > 0, there exists Bε < ∞ such that P |Xn| n r > Bε < ε for all n ≥ 1, write Xn = Op (n r ). Xn nr is stochastically bounded If plimXn nr = 0, write Xn = op (n r ). The weak law of large numbers 1. Let {Xi , i ≥ 1} be a sequence of i.i.d. r.v.s with EX1 < ∞. Then 1 n n i=1 Xi P→ EX1 as n → ∞. 2. Let {Xi , i ≥ 1} be sequence of independent r.v.s with EXi = m. If E |Xi | 1+δ ≤ B < ∞ (δ > 0) for all i. Then 1 n n i=1 Xi P→ m as n → ∞. Example 3 Let εi ∼ iid (0, σ2 ). Then 1 n n i=1 εi P→ Eε1 = 0.
CHAPTER 5 LARGE-SAMPLE PROPERTIES OF THE LSE The central limit theorem 1. Let Xi, i>l be a sequence of i i d I vs with E(X1)=u and Var(X1)=0+0 nen (0,1) asm→o 2. Let [Xi, i>l be a sequence of independent r v s wit h mean u; and variance a-, and let a2=1∑ max1<i<n Bx=≤B<a(6>0) N(0,1) Example 4 Let X; n iidB (1, p). Then EXI=p and Var (X1)=p(1-p). Thus, p(1-p) N(0,1) For vector sequences, we use the following result known as the Cramer-Wold device If [Xn) is a sequence of random vectors, Xn X iff X'Xn X'X for any vector A Example 5 Let XiN iid (0, 2). Then, m×1 ∑X N 5.2 Consistency of b assume 1.(Xi, Ei) is a sequence of independent observations 2.∑1XX(=是xxX)Q=limn-∑m1E(XX)(>0) 3. For any∈ R and s>0,E|Xei|≤B< oo for all i The least squares estimator b may be written as 8+(∑xx)(∑x Consider for入∈Rk XX
CHAPTER 5 LARGE—SAMPLE PROPERTIES OF THE LSE 3 The central limit theorem 1. Let {Xi , i ≥ 1} be a sequence of i.i.d. r.v.s with E (X1) = µ and V ar (X1) = σ 2 = 0. Then n i=1 (Xi − µ) σ √ n d→ N (0, 1) as n → ∞. 2. Let {Xi , i ≥ 1} be a sequence of independent r.v.s with mean µi and variance σ 2 i , and let σ¯ 2 n = 1 n n i=1 σ 2 i . If max1≤i≤n E|Xi − µi | 2+δ 1 2+δ σ¯n ≤ B < ∞ (δ > 0) for all n, n i=1 (Xi − µi ) σ¯n √ n d→ N (0, 1). Example 4 Let Xi ∼ iidB (1, p). Then EX1 = p and V ar (X1) = p (1 − p). Thus, n i=1 (Xi − p) p (1 − p) √ n d→ N (0, 1). For vector sequences, we use the following result known as the Cramer—Wold device. If {Xn} is a sequence of random vectors, Xn d→ X iff λ ′Xn d→ λ ′X for any vector λ. Example 5 Let Xi m×1 ∼ iid (0, Σ). Then, Xi √ n d→ N (0, Σ). 5.2 Consistency of b Assume 1. (Xi , εi) is a sequence of independent observations. 2. 1 n n i=1 XiX′ i = 1 nX′X P→ Q = limn→∞ 1 n n i=1 E(XiX′ i ) (> 0). 3. For any λ ∈ Rk and δ > 0, E |λ ′Xiεi | 2+δ ≤ B < ∞ for all i. The least squares estimator b may be written as b = β + 1 n XiX ′ i −1 1 n Xiεi Consider for λ ∈ Rk 1 n λ ′Xiεi = 1 n wi .
XF.,a>P< >EE 9Y.C<E >Oxe>, IE YoF, FE < Thn,;i- an ind→ndn/-q→ nc. h E()=EE(XX=|X)=0 In addi n, (u2)=E(XX2=)2 which im=i-Eul|1+≤D<o∞frll apounov's inequality F0<a≤B(Ex≤(E|xr) Th*byh: WLLN f r an ind→ndn/-q→ne, Po finc-hi-h ld-f r any X n之10 5.3 Asymptotic normality of the least squares estimator XX Xi s /n(b-B) m X }imc∑ XiXi by a-m=n, wnd X-h w ha/∑x=i-叫 rmally di fib-/ d in h limi 入∈R >XXiI W. wi-h A ch- ck h-c ndiA- f h- CLT fr a -qnc. f ind-=ndn/rv E(wi=0 a-b.fr 2.(E2+4)m≤ B2+6 all i and o2=∑E2≤B
CHAPTER 5 LARGE—SAMPLE PROPERTIES OF THE LSE 4 Then, wi is an independent sequence with E (wi) = EE (λ ′Xiεi |X) = 0. In addition, E w 2 i = E (λ ′Xiεi) 2 ≤ C < ∞ which implies E|wi | 1+δ ≤ D < ∞ for all i. Lyapounov’s inequality For 0 < α ≤ β, (E |X| α ) 1/α ≤ E |X| β 1/β . Thus, by the WLLN for an independent sequence, 1 n wi P→ 0. Since this holds for any λ, 1 n n i=1 Xiεi P→ 0 and we have b P→ β + Q −1 · 0 = β. 5.3 Asymptotic normality of the least squares estimator Write b − β = XiX ′ i −1Xiεi or √ n (b − β) = 1 n XiX ′ i −1 1 √ n Xiεi . Since 1 n XiX′ i P→ Q by assumption, we need to show that √ 1 n Xiεi is normally distributed in the limit. Consider for λ ∈ R k , 1 √ n λ ′Xiεi = 1 √ n wi . We wish to check the conditions of the CLT for a sequence of independent r.v.’s. 1. E (wi) = 0 as before. 2. E|wi | 2+δ 1 2+δ ≤ B 1 2+δ for all i and σ¯ 2 n = 1 n Ew2 i ≤ B.
}-P-R/∞R2-P∞PReP-I∈F|} Thal R N(0,1) 1EP。T E(XX2=;E1入) E P EEXX2E2X7入 P TT n LXE EX XX: AQ入, thll relalt dn be wrltten m R 0,a2Q入 w hlah m mlel R X T UImc thH Tnd the clven Tlan ntlpn5we hIve T h(beB).“N0,a2Q (Redm thI XnYn. cY H Xn. c(TopnltTnt)Td Yn. y xm e Tathprl wrlte thH relalt T P ThT 5bH Tmrpxhm Teli npm T wlth n eTn B Td vTYThce-opvTYToen Ttrkx1(02Q-1 5.4 Consistency of s2 EEE EX(XXXa E K EE EX XX X
CHAPTER 5 LARGE—SAMPLE PROPERTIES OF THE LSE 5 Thus wi σ¯n √ n d→ N (0, 1). Since σ¯ 2 n = 1 n E w 2 i = 1 n E (λ ′Xiεiε ′ iX ′ iλ) = 1 n EE λ ′Xiε 2 i X ′ iλ|X = 1 n E λ ′XiE ε 2 i |X X ′ iλ = σ 2 n λ ′n i=1 XiX ′ iλ → σ 2λ ′Qλ, this result can be written as wi √ n d→ N 0, σ2λ ′Qλ , which implies Xiεi √ n d→ N 0, σ2Q . Using this and the given assumption, we have √ n (b − β) d→ N 0, σ2Q −1 . (Recall that XnYn d→ cY if Xn P→ c (a constant) and Yn d→ Y ) Some authors write this result as b ≃ N β, 1 n σ 2Q −1 . That is, b is approximately normal with mean β and variance—covariance matrix 1 n (σ 2Q−1 ). 5.4 Consistency of s 2 Write s 2 = 1 n − K ε ′Mε = 1 n − K ε ′ ε − ε ′X (X ′X) −1 X ′ ε = n n − K ε ′ ε n − ε ′X n X′X n −1 X′ ε n .