10、Spark on YARN

●DatascientistatCloudera●RecentlyleadApacheSparkdevelopmentatCloudera●Beforethat,committingonApacheYARNandMapReduce●HadoopPMCmemberTraditionalOperatingSystemStorage:FileSystemExecution/Scheduling:Processes/KernelSchedulerHadoopStorage:HadoopDistributedFileSystem(HDFS)Execution/Scheduling:YARN!HDFSImpalaMapReduceSparkEngineering-50%Finance-30%Marketing-20%SparkMRImpalaSparkMRImpalaSparkMRImpalaHDFSHDFSImpalaMapReduceSparkYARN●RunSparkalongsideotherHadoopworkloads○Leverageexistingclusters○Datalocality●Manageworkloadsusingadvancedpolicies○Allocatesharestodifferentteamsandusers○Hierarchicalqueues○Queueplacementpolicies●TakeadvantageofHadoop’ssecurity○RunonKerberizedclusters●Late2012/Spark0.6-experimentalprojectatYahoo●Late2013/Spark0.8-pulledintoSpark,Hadoop-0.23only●Early2014/Spark0.9-Hadoop2.2lineaswell,supportforspark-shell●Early2014/Spark0.9.1/CDH5.0-Stable!●Mid2014/Spark1.0.0/CDH5.1-Easierappsubmissionwithspark-submitResourceManagerNodeManagerNodeManagerResourceManagerNodeManagerNodeManagerContainerContainerContainerResourceManagerNodeManagerNodeManagerClientResourceManagerNodeManagerNodeManagerContainerApplicationMasterClientResourceManagerNodeManagerNodeManagerContainerMapTaskContainerApplicationMasterContainerReduceTaskClientNodeManagerSparkExecutorAppMaster/ExecutorLauncherTaskTaskNodeManagerSparkExecutorTaskTaskResourceManagerClient/SparkDriverNodeManagerSparkExecutorAppMaster/SparkDriverTaskTaskNodeManagerSparkExecutorTaskTaskResourceManagerClient●Whenrunningajob,SparktriestoplacetasksalongsideHDFSblocks●Problem:SparkneedstoaskYARNforexecutorsbeforeitrunsitsjobs●Solution:TellSparkwhatfilesyou’regoingtotouchwhencreatingSparkContextvallocData=InputFormatInfo.computePreferredLocations(Seq(newInputFormatInfo(conf,classOf[TextInputFormat],newPath(“myfile.txt”)))valsc=newSparkContext(conf,locData)●Sparkholdsontofullresourcesevenwhenappisidle●Givebacktocluster●Requirescontainer-resizing(YARN-1197)●SparkHistoryServer●GenericYARNhistory(YARN-321)●Lookingatlogsshouldbeeasier●Betterdocumentationondatalocality●Removeunnecessarysleeps●Long-runningappsonsecureclusters(YARN-941)