飞天开放平台-大数据技术年会-4x324

开放平台ApsaraCloudPlatformAboutAliyunChina’slargestcloudserviceprovider100softhousandsofcustomersBillionsofaccesseseverydayAlibaba-operatedIDCPartner-operatedIDCApsaraCloudPlatformACE3rd-partyPlatformServicesMap,Mail,Search3rd-partyApplicationServicesCustomersISVandSIDevelopersProvidingFoundationServicesoftheCloudEco-systemPaybyusageElasticitySafety(like“tapwater”)TheNatureofCloudComputingScale大规模Economy低成本PublicUtility服务运营Internet-scalecomputing2.5EBgeneratedperday,doublingevery40monthsBillionsoftxnsonTaobaoeveryday,mustbeprocessedin6hoursEconomymeansmorethanlowpricesLeadingtobehaviorchanges(like“telephone”)Keyisscheduling(like“powergrid”)TwoDesignPrinciplesLarge-scalegeneralcomputingplatformasthebaseOnesystemsupportingbothofflineandonlineservicesMulti-tenancy,resourcesharing,loadshiftingWeb-basedAPIasthedeliverymechanismOnlineactivation,pay-by-usageLocation-transparencyLinuxClusterIDCResourceManagement(伏羲)Security(钟馗)RPC(夸父)Naming/Coordination(女娲)ClusterDeployment(大禹)ClusterMonitor(神农)DistributedFileSystem(盘古)JobScheduling(伏羲)ACEOSSOTSODPSECS/SLBRDSMap,Mail,Search,etcCloudMartOtherCloudServicesOSPSCloudComputingServicesElasticComputing弹性计算ECS:virtualizedinstancesofserversthatcanbecreatedandtailoredtomeetapplicationrequirementsSLB:softwareloadbalancingtechnologythatcanelasticallyexpandservicecapacityondemandACE:ConvenientandefficientexecutionenvironmentforWebservices,supportingJava,PHP,Node.jsStorageandDatabases海量存储和数据库Large-scaleComputing大规模计算CloudComputingServicesElasticComputing弹性计算StorageandDatabases海量存储和数据库OSS:large-scaleobjectstorageserviceforunstructureddatasuchasphotos,music,orvideoOTS:largescalestorageserviceforstructuredorsemi-structureddatastorageandreal-timequeryRDS:managedinstancesforrelationaldatabaseswithautomaticbackupandfailoverLarge-scaleDataComputing大规模计算AComparisonofStorageandDatabaseServicesOSSOTSRDSDataModelUnstructuredSemi-structuredFully-structuredTargetDataVolume~10PB~100TB/table~TB/dbTxnSupportNoneLimitedsupportFullSupportProgrammingInterfaceRESTfulAPIRESTfulAPISQLCloudComputingServicesElasticComputing弹性计算StorageandDatabase海量存储和数据库Large-scaleComputing大规模计算ODPS:large-scaledatabatchprocessingandcomputation,supportingSQLandMapReducestyleprogramminglanguagesOSPS:streamdataprocessingservice,supportingSQL-likequerylanguageandautomaticfailurerecoveryApsaraTechnicalHighlightsAcommonplatformsupportingbothofflineandonlineservicesSearch:24Bpagesprocessed,13BonlineindexMail:100Mmailsreceived,10Mmailssent,10mslatencyCapability-basedsecuritymanagementframework,enforcingthePrincipleofLeastPrivilegeDistributeddeployment,monitoringanddiagnosticsZeroSPOF(single-point-of-failure):availability99.9%Alldatahas3replicas:datareliability99.99999999%5K2013/08/15:First-ever5000-nodeApsaracluster(ODPS)wentintoproduction100KCPUcores,100PBrawstorageProcessingpetabytesperday2013/09/24:OpenedaccesstoODPSfor4universities&researchinstitutionsSorting100TBin30minutesCurrentknownrecord:72minutes(Yahoo!,2013/07/03)Pangu:Large-scaleDistributedFileSystemMaster-SlaveArchitectureMasterformetadatamgmt,Slave(ChunkServer)forIOmgmtPaxos-basedmulti-masterarchitecture,failurerecoverytime1minuteEnd-to-endinlinechecksumScalesto1billionfilesPaxosSeparatedIOPipelineandStorageMgmtAdaptiveIOPipelineReplicationmaster:chunkservervsclientReplicationpolicy:chainingvsstar-replicationChunkingpolicy:fixed,variable,orRAIDDurabilityguarantee:txnloggingvssequentialwriteCommonStorageManagementPhysicalIOmanagementPriorityandQoSBackgroundre-replicationChunkplacementStagedEvent-drivenPhysicalIOMgmtChunkServerwouldrearrangeIOrequeststosupportpriority,QoS,andreduceIOseekoverheadDistributedRe-replicationTypical:Mirroring(10hours)Pangu:Distributedre-replication(20min,50-nodes)RAIDBuiltintothecoresysteminsteadofanadd-onlayer(asinHDFSRAID)Bettermanagementofdataintegrity,recovery,andchunkplacementSynchronousredundancyblockgenerationLow-latencyfailurerecoverySmallfilesupport...............JobcontrolResourcerequestsNodecontrolJobsubmissionFuxiResourceSchedulingMulti-dimensionresourcesElasticquotaCGroup-basedisolationFuxiMasterHAAppMasterfailoverIncrementalschedulingFuxiJobProgrammingModelJob:ADAGVertex:TaskEachtaskmayhavemultipleinstancesbasedoninputdatachunksEdge:dataflow,eachtaskmayhavemultipleinput/outputflowsAdataflowconnectingtwotasksrepresentsdatashufflinginput1input2output2output1outputinputMapReduceisadegeneratedcaseExample:FindBest-SellersSELECTprod_id,Sum(count)ASquantityFROMordersGROUPBYprod_idORDERBYquantityDESC;order_idprod_idunit_pricecount0001042151003000203343101000003012345010000404215805…………prod_idquantity0251831790075632845109641229430421520043……ordersFuxiMapReduceComparisonofJob-executionPlanInputprod_idcountprod_idquantityprod_idquantityInputprod_idcountprod_idquantityprod_idquantityprod_idquantity020406080100120020406080100120FuxiMapReduceComparisonofJobExecutionR1MR1MM2R2R2ABriefHistory02/04/2009FirstlineofApsaracode08/27/2010Apsarabecamethecommonplatformforsear