Maximum Likelihood Estimation

AppliedEconometricsWilliamGreeneDepartmentofEconomicsSternSchoolofBusinessAppliedEconometrics19.MaximumLikelihoodEstimationMaximumLikelihoodEstimationThisdefinesaclassofestimatorsbasedontheparticulardistributionassumedtohavegeneratedtheobservedrandomvariable.ThemainadvantageofMLestimatorsisthatamongallConsistentAsymptoticallyNormalEstimators,MLEshaveoptimalasymptoticproperties.Themaindisadvantageisthattheyarenotnecessarilyrobusttofailuresofthedistributionalassumptions.Theyareverydependentontheparticularassumptions.Theoftciteddisadvantageoftheirmediocresmallsamplepropertiesisprobablyoverstatedinviewoftheusualpaucityofviablealternatives.SettinguptheMLEThedistributionoftheobservedrandomvariableiswrittenasafunctionoftheparameterstobeestimatedP(yi|data,β)=Probabilitydensity|parameters.ThelikelihoodfunctionisconstructedfromthedensityConstruction:Jointprobabilitydensityfunctionoftheobservedsampleofdata–generallytheproductwhenthedataarearandomsample.RegularityConditionsWhattheyare1.logf(.)hasthreecontinuousderivativeswrtparameters2.Conditionsneededtoobtainexpectationsofderivativesaremet.(E.g.,rangeofthevariableisnotafunctionoftheparameters.)3.Thirdderivativehasfiniteexpectation.WhattheymeanMomentconditionsandconvergence.Weneedtoobtainexpectationsofderivatives.WeneedtobeabletotruncateTaylorseries.WewillusecentrallimittheoremsPropertiesoftheMaximumLikelihoodEstimatorWewillsketchformalproofsoftheseresults:Thelog-likelihoodfunction,againThelikelihoodequationandtheinformationmatrix.AlinearTaylorseriesapproximationtothefirstorderconditions:g(ML)=0g()+H()(ML-)(underregularity,higherordertermswillvanishinlargesamples.)Ourusualapproach.Largesamplebehavioroftheleftandrighthandsidesisthesame.AProofofconsistency.(Property1)Thelimitingvarianceofn(ML-).Weareusingthecentrallimittheoremhere.Leadstoasymptoticnormality(Property2).WewillderivetheasymptoticvarianceoftheMLE.Efficiency(wehavenotdevelopedthetoolstoprovethis.)TheCramer-Raolowerboundforefficientestimation(anasymptoticversionofGauss-Markov).Estimatingthevarianceofthemaximumlikelihoodestimator.Invariance.(AVERYhandyresult.)CoupledwiththeSlutskytheoremandthedeltamethod,theinvariancepropertymakesestimationofnonlinearfunctionsofparametersveryeasy.TestingHypotheses–ATrinityofTestsThelikelihoodratiotest:Basedontheproposition(Greene’s)thatrestrictionsalways“makelifeworse”Isthereductioninthecriterion(log-likelihood)large?LeadstotheLRtest.TheLagrangemultipliertest:Underlyingbasis:Reexaminethefirstorderconditions.Formatestofwhetherthegradientissignificantly“nonzero”attherestrictedestimator.TheWaldtest:Theusual.TheLinear(Normal)ModelDefinitionofthelikelihoodfunction-jointdensityoftheobserveddata,writtenasafunctionoftheparameterswewishtoestimate.Definitionofthemaximumlikelihoodestimatorasthatfunctionoftheobserveddatathatmaximizesthelikelihoodfunction,oritslogarithm.Forthemodel:yi=xi+i,wherei~N[0,2],themaximumlikelihoodestimatorsofand2areb=(XX)-1Xyands2=ee/n.Thatis,leastsquaresisMLfortheslopes,butthevarianceestimatormakesnodegreesoffreedomcorrection,sotheMLEisbiased.NormalLinearModelThelog-likelihoodfunction=ilogf(yi|)=sumoflogsofdensities.Forthelinearregressionmodelwithnormallydistributeddisturbanceslog-L=i[-½log2-½log2-½(yi–xi)2/2].LikelihoodEquationsTheestimatorisdefinedbythefunctionofthedatathatequateslog-L/to0.(Likelihoodequation)Thederivativevectorofthelog-likelihoodfunctionisthescorefunction.Fortheregressionmodel,g=[log-L/,log-L/2]’=log-L/=i[(1/2)xi(yi-xi)]log-L/2=i[-1/(22)+(yi-xi)2/(24)]Forthelinearregressionmodel,thefirstderivativevectoroflog-Lis(1/2)X(y-X)and(1/22)i[(yi-xi)2/2-1](K1)(11)Notethatwecouldcomputethesefunctionsatanyand2.Ifwecomputethematbandee/n,thefunctionswillbeidenticallyzero.MomentEquationsNotethatg=igiisarandomvectorandthateachterminthesumhasexpectationzero.ItfollowsthatE[(1/n)g]=0.Ourestimatorisfoundbyfindingthethatsetsthesamplemeanofthegsto0.Thatis,theoretically,E[gi(,2)]=0.Wefindtheestimatorasthatfunctionwhichproduces(1/n)igi(b,s2)=0.Notethesimilaritytothewaywewouldestimateanymean.IfE[xi]=,thenE[xi-]=0.Weestimatebyfindingthefunctionofthedatathatproduces(1/n)i(xi-m)=0,whichis,ofcoursethesamplemean.Therearetwomaincomponentstothe“regularityconditionsformaximumlikelihoodestimation.Thefirstisthatthefirstderivativehasexpectedvalue0.That‘momentequation’motivatestheMLEInformationMatrixThenegativeofthesecondderivativesmatrixofthelog-likelihood,-H=iscalledtheinformationmatrix.Itisusuallyarandommatrix,also.Forthelinearregressionmodel,HessianfortheLinearModelNotethattheoffdiagonalelementshaveexpectationzero.EstimatedInformationMatrix(whichshouldlookfamiliar).Theoffdiagonaltermsgotozero(oneoftheassumptionsofthemodel).Thiscanbecomputedatanyvectorandscalar2.YoucantakeexpectedvaluesofthepartsofthematrixtogetDerivingthePropertiesoftheMaximumLikelihoodEstimatorTheMLEConsistency:ConsistencyProofAsymptoticVarianceAsymptoticVarianceAsymptoticDistributionOtherResults1–VarianceBoundInvarianceThemaximumlikelihoodestimatorofafunctionof,sayh()ish(MLE).Thisisnotalwaystrueofotherkindsofestimators.Togetthevarianceofthisfunction,wewouldusethedeltamethod.E.g.,theMLEofθ=(β/σ)isb/(e’e/n)ComputingtheAsymptoticVarianceWewanttoestimate{-E[H]}-1Threeways:(1)Justcomputethenegativeoftheactualsecondderivativesmatrixandinvertit.(2)Insertthemaximumlikelihoodestimatesintotheknownexpectedvaluesofthesecondderivativesmatrix.Sometimes(1)and(2)givethesameanswer(forexample,inthelinearregressionmodel).(3)SinceE[H]isthevarianceofthefirstderivatives,estimatethiswiththesamplevariance(i.e.,meansquare)ofthefirstderivatives.Thiswillalmostalwaysbedifferentfrom(1)and(2).Sincetheyareestimatingthesamething,inlargesamples,allthreewillgivethesameanswer.Currentpracticeineconometricsoftenfavors(3).Statararelyuses(3).Othersdo.LinearRegressionModelExample:DifferentEstimatorsoftheVarianceoftheMLEConsider,again,thegasolinedata.Weuseasimpleequation:Gt=1+2Yt+3Pgt+t.LinearModelBHHHEstimatorNewton’sMethodPoissonRegressionAsymptoticVarianceoftheMLEEstimatorsoftheAsymptoticCovarianceMatrixROBUSTESTIMATIONSandwichEstimatorH-1(G’G)H-1Isthisappropriate?Whydowedothis?Application:DoctorVisitsGermanIndividualHealthCaredata:N=27,236Modelfornumberofvisitstothedoctor:Poissonregression(fitbymaximumlikelihood)Income,Education,GenderPoissonRegressionIterationspoisson;lhs=docvis;rhs=one,female,hhninc,educ;mar;output=3$Method=Newton;Maximumiterations=100Convergencecriteria:gtHg.1000D-05chg.F.0000D+00max|db|.0000D+00Startvalues:.00000D+00.00000D+00.00000D+00.00000D+001stderivs.-.13214D+06-.61899D+05-.43338D+05-.14596D+07Parameters:.28002D+01.72374D-01-.65451D+00-.47608D-01Itr2F=-.1587D+06gtHg=.2832D+03chg.F=.1587D+06max|db|=.1346D+011stderivs.-.33055D+05-.14401D+05-.10804D+05-.36592D+06Parameters:.21404D+01.16980D+00-.60181D+00-.48527D-01Itr3F=-.1115D+06gtHg=.9725D+02chg.F=.4716D+05max|db|=.6348D+001stderivs.-.42953D+04-.15074D+04-.13927D+04-.47823D+05Parameters:.17997D+01.27758D+00-.54519D+00-.49513D-01Itr4F=-.1063D+06gtHg=.1545D+02chg.F=.5162D+04max|db|=.1437D+001stderivs.-.11692D+03-.22248D+02-.37525D+02-.13159D+04Parameters:.17276D+01.31746D+00-.52565D+00-.49852D-01Itr5F=-.1062D+06gtHg=.5006D+00chg.F=.1218D+03max|db|=.6542D-021stderivs.-.12522D+00-.54690D-02-.40254D-01-.14232D+01Parameters:.17249D+01.31954D+00-.52476D+00-.49867D-01Itr6F=-.1062D+06gtHg=.6215D-03chg.F=.1254D+00max|db|=.9678D-051stderivs.-.19317D-06-.94936D-09-.62872D-07-.22029D-05Parameters:.17249D+01.31954D+00-.52476D+00-.49867D-01Itr7F=-.1062D+06gtHg=.9957D-09chg.F=.1941D-06max|db|=.1602D-10*ConvergedRegressionandPartialEffects+--------+--------------+----------------+--------+--------+----------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|MeanofX|+--------+--------------+----------------+--------+--------+----------+Constant|1.72492985.0200056886.222.0000FEMALE|.31954440.0069687045.854.0000.47877479HHNINC|-.52475878.02197021-23.885.0000.35208362EDUC|-.04986696.00172872-28.846.000011.3206310+-------------------------------------------+|Partialderivativesofexpectedval.with||respecttothevectorofcharacteristics.||Effectsareaveragedoverindividuals.||ObservationsusedformeansareAllObs.||ConditionalMeanatSamplePoint3.1835||ScaleFactorforMarginalEffects3.1835|+-------------------------------------------++--------+--------------+----------------+--------+--------+----------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|MeanofX|+--------+--------------+----------------+--------+--------+----------+Constant|5.49135704.0789008369.598.0000FEMALE|1.01727755.0242760741.905.0000.47877479HHNINC|-1.67058263.07312900-22.844.0000.35208362EDUC|-.15875271.00579668-27.387.000011.3206310ComparisonofStandardErrors+--------+--------------+----------------+--------+--------+----------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|MeanofX|+--------+--------------+----------------+--------+--------+----------+Constant|1.72492985.0200056886.222.0000FEMALE|.31954440.0069687045.854.0000.47877479HHNINC|-.52475878.02197021-23.885.0000.35208362EDUC|-.04986696.00172872-28.846.000011.3206310BHHH+--------+--------------+----------------+--------+--------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|+--------+--------------+----------------+--------+--------+Constant|1.72492985.00677787254.495.0000FEMALE|.31954440.00217499146.918.0000HHNINC|-.52475878.00733328-71.559.0000EDUC|-.04986696.00062283-80.065.0000Whyaretheysodifferent?Modelfailure.Thisisapanel.Thereisautocorrelation.NLSvs.MLENONLINEARLEASTSQUARES+--------+--------------+----------------+--------+--------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|+--------+--------------+----------------+--------+--------+|C0|1.70205***.0670697425.377.0000||C1|.31261***.0222878414.026.0000||C2|-.57513***.07336253-7.840.0000||C3|-.04586***.00588216-7.797.0000|+--------+-------------------------------------------------+MAXIMUMLIKELIHOOD+--------+--------------+----------------+--------+--------+----------+|Variable|Coefficient|StandardError|b/St.Er.|P[|Z|>z]|MeanofX|+--------+--------------+----------------+--------+--------+----------+Constant|1.72492985.0200056886.222.0000FEMALE|.31954440.0069687045.854.0000.47877479HHNINC|-.52475878.02197021-23.885.0000.35208362EDUC|-.04986696.00172872-28.846.000011.3206310TestingHypothesesWaldtests,usingthefamiliardistancemeasureLikelihoodratiotests:LogLU=loglikelihoodwithoutrestrictionsLogLR=loglikelihoodwithrestrictionsLogLU>logLRforanynestedrestrictions2(LogLU–logLR)chi-squared[J]TheLagrangemultipliertest.Waldtestofthehypothesisthatthescoreoftheunrestrictedloglikelihoodiszerowhenevaluatedattherestrictedestimator.TestingtheModel+---------------------------------------------+|PoissonRegression||MaximumLikelihoodEstimates||DependentvariableDOCVIS||Numberofobservations27326||Iterationscompleted7||Loglikelihoodfunction-106215.1|Loglikelihood|Numberofparameters4||Restrictedloglikelihood-108662.1|LogLikelihoodwithonlya|McFaddenPseudoR-squared.0225193|constantterm.|Chisquared4893.983|2*[logL–logL(0)]|Degreesoffreedom3||Prob[ChiSqd>value]=.0000000|+---------------------------------------------+Likelihoodratiotestthatallthreeslopesarezero.WaldTest-->MATRIX;List;b1=b(2:4);v11=varb(2:4,2:4);B1'B1$MatrixB1MatrixV11has3rowsand1columns.has3rowsand3columns1123+--------------+------------------------------------------1|.319541|.4856275D-04-.4556076D-06.2169925D-052|-.524762|-.4556076D-06.00048-.9160558D-053|-.049873|.2169925D-05-.9160558D-05.2988465D-05MatrixResulthas1rowsand1columns.1+--------------1|4682.38779LRstatisticwas4893.983LikelihoodRatioTestpoisson;lhs=docvis;rhs=one,female,hhninc,educ$calc;logL1=logL$poisson;lhs=docvis;rhs=one,hhninc,educ$calc;logL0=logL$calc;logLpool=logL$poisson;for[female=0];lhs=docvis;rhs=one,hhninc,educ$calc;logLM=logL$poisson;for[female=1];lhs=docvis;rhs=one,hhninc,educ$calc;logLF=logL$?ChisquaredtestcoefficientonFEMALE.1Deg.Fr.calc;list;logL1;logL0;LRM_F=2*(logL1-logL0)$?Chisquaredtestforpooling.3Deg.Fr.calc;list;logLM;loglF;logLpool;LRGender=2*(logLM+logLF-logLpool)$LRTestResults-->calc;list;logL1;logL0;LRM_F=2*(logL1-logL0)$+------------------------------------+|ListedCalculatorResults|+------------------------------------+LOGL1=-106215.144165LOGL0=-107277.287979LRM_F=2124.287628Calculator:Computed3scalarresultscalc;list;loglM;loglF;logLpool;LRGender=2*(logLM+logLF-logLpool)$+------------------------------------+|ListedCalculatorResults|+------------------------------------+LOGLM=-51501.135803LOGLF=-54656.153984LOGLPOOL=-107277.287979LRGENDER=2239.996383LMTestHypothesis:3slopes=0.MLEwithall3slopes=0,λ=y-bar=exp(β1),soMLEofβ1islog(y-bar).ConstrainedMLEsofother3slopesarezero.LMStatistic-->calc;beta1=log(xbr(docvis))$-->matrix;bmle0=[beta1/0/0/0]$-->create;lambda0=exp(x'bmle0);res0=docvis-lambda0$-->matrix;list;g0=x'res0;h0=x'[lambda0]x;lm=g0'**g0$MatrixG0has4rowsand1columns.+--------------1|.2664385D-082|7944.944413|-1781.122194|-.3062440D+05MatrixH0has4rowsand4columns.+--------------------------------------------------------1|.8699300D+05.4165006D+05.3062881D+05.9848157D+062|.4165006D+05.4165006D+05.1434824D+05.4530019D+063|.3062881D+05.1434824D+05.1350638D+05.3561238D+064|.9848157D+06.4530019D+06.3561238D+06.1161892D+08MatrixLMhas1rowsand1columns.+--------------1|4715.41008Waldwas4682.38779LRstatisticwas4893.983AppliedEconometricsWilliamGreeneDepartmentofEconomicsSternSchoolofBusinessAppliedEconometrics20.AspectsofMaximumLikelihoodEstimationInvarianceReparameterizingtheLogLikelihoodEstimatingtheTobitModel