Advanced Data Analysis

AdvancedDataAnalysisfromanElementaryPointofViewCosmaRohillaShaliziSpring2013LastLATEX’dThursday25thApril,201314:41Thursday25thApril,2013ContentsIntroduction13TotheReader.........................................130.0.0.0.1Updates.......................14ConceptsYouShouldKnow...............................14IRegressionandItsGeneralizations161RegressionBasics171.1Statistics,DataAnalysis,Regression.......................171.2GuessingtheValueofaRandomVariable...................181.2.1EstimatingtheExpectedValue.....................191.3TheRegressionFunction..............................191.3.1SomeDisclaimers.............................201.4EstimatingtheRegressionFunction.......................231.4.1TheBias-VarianceTradeoff.......................231.4.2TheBias-VarianceTrade-OffinAction................251.4.3OrdinaryLeastSquaresLinearRegressionasSmoothing....251.5LinearSmoothers...................................301.5.1k-Nearest-NeighborRegression....................301.5.2KernelSmoothers.............................321.6Exercises........................................352TheTruthaboutLinearRegression362.1OptimalLinearPrediction:MultipleVariables................362.1.1Collinearity.................................382.1.2ThePredictionandItsError......................382.1.3EstimatingtheOptimalLinearPredictor..............392.1.3.1UnbiasednessandVarianceofOrdinaryLeastSquaresEstimates............................402.2ShiftingDistributions,OmittedVariables,andTransformations....412.2.1ChangingSlopes..............................412.2.1.1R2:DistractionorNuisance?...............412.2.2OmittedVariablesandShiftingDistributions...........4123CONTENTS2.2.3ErrorsinVariables.............................432.2.4Transformation...............................472.3AddingProbabilisticAssumptions........................492.3.1ExaminetheResiduals..........................512.3.2OnSignificantCoefficients.......................522.4LinearRegressionIsNotthePhilosopher’sStone..............532.5Exercises........................................553ModelEvaluation563.1WhatAreStatisticalModelsFor?........................563.2Errors,InandOutofSample...........................573.3Over-FittingandModelSelection........................613.4Cross-Validation...................................663.4.1Data-setSplitting..............................673.4.2k-FoldCross-Validation(CV)......................673.4.3Leave-one-outCross-Validation.....................703.5Warnings........................................703.5.1ParameterInterpretation.........................713.6Exercises........................................724SmoothinginRegression734.1HowMuchShouldWeSmooth?.........................734.2AdaptingtoUnknownRoughness........................744.2.1BandwidthSelectionbyCross-Validation..............844.2.2ConvergenceofKernelSmoothingandBandwidthScaling...854.2.3SummaryonKernelSmoothing....................904.3KernelRegressionwithMultipleInputs....................904.4InterpretingSmoothers:Plots...........................914.5AveragePredictiveComparisons.........................954.6Exercises........................................985Simulation995.1WhatDoWeMeanby“Simulation”?......................995.2HowDoWeSimulateStochasticModels?...................1005.2.1ChainingTogetherRandomVariables................1005.2.2RandomVariableGeneration......................1005.2.2.1Built-inRandomNumberGenerators..........1005.2.2.2Transformations........................1015.2.2.3QuantileMethod.......................1015.2.2.4RejectionMethod.......................1025.2.2.5TheMetropolisAlgorithmandMarkovChainMonteCarlo...............................1055.2.2.6GeneratingUniformRandomNumbers........1065.2.3Sampling...................................1075.2.3.1SamplingRowsfromDataFrames............1105.2.3.2MultinomialsandMultinoullis..............11014:41Thursday25thApril,2013CONTENTS45.2.3.3ProbabilitiesofObservation................1105.2.4RepeatingSimulations..........................1115.3WhySimulate?....................................1115.3.1UnderstandingtheModel;MonteCarlo...............1115.3.2CheckingtheModel............................1125.3.3SensitivityAnalysis............................1155.4TheMethodofSimulatedMoments.......................1175.4.1TheMethodofMoments........................1175.4.2AddingintheSimulation........................1185.4.3AnExample:MovingAverageModelsandtheStockMarket.1185.5Exercises........................................1255.6Appendix:SomeDesignNotesontheMethodofMomentsCode...1276TheBootstrap1296.1StochasticModels,Uncertainty,SamplingDistributions.........1296.2TheBootstrapPrinciple..............................1316.2.1VariancesandStandardErrors.....................1336.2.2BiasCorrection...............................1336.2.3ConfidenceIntervals...........................1346.2.3.1OtherBootstrapConfidenceIntervals.........1356.2.4HypothesisTesting............................1366.2.4.1Doublebootstraphypothesistesting..........1376.2.5