稀疏和低秩理论ppt

JohnWrightElectricalEngineeringColumbiaUniversityLectureIII:Algorithms????Twoconvexoptimizationproblemsminimizationseeksasparsesolutiontoanunderdeterminedlinearsystemofequations:RobustPCAexpressesaninputdatamatrixasasumofalow-rankmatrixandasparsematrix.????Twonoise-awarevariantsBasispursuitdenoisingseeksasparsenear-solutiontoanunderdeterminedlinearsystem:Noise-awareRobustPCAapproximatesaninputdatamatrixasasumofalow-rankmatrixandasparsematrix.CHRYSLERSETSSTOCKSPLIT,HIGHERDIVIDENDChryslerCorpsaiditsboarddeclaredathree-for-twostocksplitintheformofa50pctstockdividendandraisedthequarterlydividendbysevenpct.Thecompanysaidthedividendwasraisedto37.5ctsasharefrom35ctsonapre-splitbasis,equaltoa25ctdividendonapost-splitbasis.ChryslersaidthestockdividendispayableApril13toholdersofrecordMarch23whilethecashdividendispayableApril15toholdersofrecordMarch23.Itsaidcashwillbepaidinlieuoffractionalshares.Withthesplit,Chryslersaid13.2mlnsharesremaintobepurchasedinitsstockrepurchaseprogramthatbeganinlate1984.Thatprogramnowhasatargetof56.3mlnshareswiththelateststocksplit.Chryslersaidinastatementtheactionsre°ectnotonlyourout-standingperformanceoverthepastfewyearsbutalsoouroptimismaboutthecompany'sfuture.Manypossibleapplications……ifwecansolvethesecoreoptimizationproblemsaccurately,efficiently,andscalably.Keychallenges:nonsmoothnessandscaleNonsmoothness:structure-inducingregularizerssuchasarenotdifferentiable:Greatforstructurerecovery……challengingforoptimization.Scale…typicalproblemsinvolveunknowns,ormore.Classicalinteriorpointmethods(e.g.,SeDuMi,SDPT3):greatconvergencerate(linearorbetter),butcostperiteration.Highaccuracyforsmallproblems.First-order(gradient-like)algorithms:slower(sublinear)convergencerate,butverycheapiterations.Moderateaccuracyevenforlargeproblems.Keychallenges:nonsmoothnessandscaleNonsmoothness:structure-inducingregularizerssuchasarenotdifferentiable:Greatforstructurerecovery……challengingforoptimization.Scale…typicalproblemsinvolveunknowns,ormore.Classicalinteriorpointmethods(e.g.,SeDuMi,SDPT3):greatconvergencerate(linearorbetter),butcostperiteration.Highaccuracyforsmallproblems.First-order(gradient-like)algorithms:slower(sublinear)convergencerate,butverycheapiterations.Moderateaccuracyevenforlargeproblems.Whycare?PracticalimpactofalgorithmchoiceTimerequiredtosolvea1,000x1,000matrixrecoveryproblem:AlgorithmAccuracyRank#iterationstime(sec)IT5.99e-00650101,2688,550119,370.3DUAL8.65e-00650100,0248221,855.4APG5.85e-00650100,3471341,468.9APGP5.91e-00650100,34713482.7EALMP2.07e-00750100,0143437.5IALMP3.83e-0075099,9962311.8Fourordersofmagnitudeimprovement,justbychoosingtherightalgorithmtosolvetheconvexprogram.Thisisthedifferencebetweentheorythatwillhaveimpact“someday”andpracticalcomputationaltechniquesthatcanbeappliedrightnow…Thislecture:ThreekeytechniquesProximalgradientmethods:copingwithnonsmoothnessOptimalfirst-ordermethods:acceleratingconvergenceAugmentedLagrangianmethods:handlingconstraintsFormoredepth/breadth,pleaseseethereferencesattheendoftheseslidesorLievenVandenberghe’slecturesthisafternoon!Inthishourlecture,wewillfocusonthreerecurringideasthatallowustoaddressthechallengesofnonsmoothnessandscale:Whyworryaboutnonsmoothness?Thebestuniformrateofconvergenceforfirst-ordermethods*forminimizingdependsverystronglyonsmoothness:FunctionclassMinimaxsuboptimalitysmoothconvex,differentiablenonsmoothconvex*Suchasgradientdescent.Seee.g.,Nesterov,“IntroductoryLecturesonConvexOptimization”Canweexploitspecialstructureoftogetaccuracycomparabletogradientdescent(forsmoothfunctions)?Whyworryaboutnonsmoothness?Thebestuniformrateofconvergenceforfirst-ordermethods*forminimizingdependsverystronglyonsmoothness:FunctionclassMinimaxsuboptimalitysmoothconvex,differentiablenonsmoothconvexFor,neediter.forworstnonsmoothWhatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Whatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Quadraticapproximationtoaround:Whatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Quadraticapproximationtoaround:Whatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Quadraticapproximationtoaround:Doesn’tdependonWhatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Quadraticapproximationtoaround:Keyobservation:Ateachiteration,thegradientdescentminimizesa(separable)quadraticapproximationtotheobjectivefunction,formedat.Whatdoesgradientdescentdo,anyway?Consider,withconvex,differentiable,and-Lipschitz.Gradientdescent:Quadraticapproximationtoaround:Keyobservation:Ateachiteration,thegradientdescentminimizesa(separable)quadraticapproximationtotheobjectivefunction,formedat.Rateforgradientdescent:Borrowingtheapproximationidea…Borrowingtheapproximationidea…nonsmoothsmoothBorrowingtheapproximationidea…nonsmoothsmoothBorrowingtheapproximationidea…nonsmoothsmoothJustapproximatethesmoothpart:Borrowingtheapproximatio