SAP HANA-大数据时代的内存计算技术

BIGDATA:GainValuewithSAPHANA邬学宁首席专家，SAPAPJJune7,2012世界快乐指数“Topossessfactsisknowledge,touseiswisdom,tochooseiseducation.Knowledgeisnotpowerbutriches,onlyhavevaluewhenspending”-ThomasJefferson2.5PB7.9ZB19901992199419961998200020022004200620082010臀部大数据：360个传感器汽车防盗，防止疲劳驾驶！SOCIALBIGDATAREALTIMEPREDICTIVE13我们如何发展到今天?个人电脑和与客户机/服务器架构社交大数据REALTIME199020152000200520101,000,000+销售3,000,000人能互联网访问移动手机比电力和安全饮水更普及Facebook:10亿用户;6亿移动用户;4.2亿页和9百万应用Youtube:40亿访问/天Google+:4亿注册用户Skype:2.5亿每月连接用户数据库(CIRCA1980)ANALYTICS(CIRCA1980)预测分析(CIRCA1980)语义分析(CIRCA1980)CRMDataGPSDemandSpeedVelocityTransactionsOpportunitiesServiceCallsCustomerSalesOrdersInventoryEmailsTweetsPlanningThingsMobileInstantMessagesWorldwidedigitalcontentwilldoublein18months,andevery18monthsthereafter.VELOCITYIn2005,humankindcreated150exabytesofinformation.In2011,1,200exabyteswillbecreated.VOLUMEVARIETY80%ofenterprisedatawillbeunstructured,spanningtraditionalandnontraditionalsources.GartnerIDCTheEconomistSLOAN(SDSS)CRMDataGPSDemandSpeedVelocityTransactionsOpportunitiesServiceCallsCustomerSalesOrdersInventoryEmailsTweetsPlanningThingsMobileInstantMessagesWorldwidedigitalcontentwilldoublein18months,andevery18monthsthereafter.In2005,humankindcreated150exabytesofinformation.In2011,1,200exabyteswillbecreated.VOLUMEVARIETY80%ofenterprisedatawillbeunstructured,spanningtraditionalandnontraditionalsources.GartnerIDCTheEconomistMOBILEMOBILELBSSOCIALMobileCRMDataPlanningOpportunitiesTransactionsCustomerSalesOrderThingsInstantMessagesDemandInventoryBigData数据的体量和类型在急剧增加SalesOrderThingsMobileDemandBigDataCRMDataCustomerPlanningTransactions智能价值除非时间数据整合&分析Ready交付信息时间采取行动消逝价值数据延迟分析延迟决策延迟行动时间增加价值行动时间减少的行动时间来源:Dr.RichardHackathorn.BolderTechnologiesInc.减少延迟/增加价值实时:巨量数据集CLOUD就是现在！处理交易的内存数据库比屋顶还高摩尔定律:IT性能价格每18个月翻倍每十年翻倍6+次~每十年大约100倍将OLAP与OLTP分离是在处理器速度太慢的条件下的临时解决方案，我们不再需要！202019902000198019702010100nm10μm1mm10cm10mCodd’s定义关系型数据库原文发表Codd’s定义OLAP的论文SAPHANA早期OLAP系统创立SAP的内存数据管理创新提供企业应用实时平台一个使用内存计算的列式数据库整合了OLTP和OLAPHassoPlattner通过创新和专注与客户进行转型事务分析加速事务+分析在内存中直接进行VS整合事务与分析内存技术驱动的低延迟计算磁盘存储日志与备份分区差异处理压缩列式存储内存CPUIN-MEMORYFASTERSMARTERSOCIALANALYTICSMOBILEBIGDATACLOUDHANAREAL-TIMEPLATFORM摄入KafkaFlumeScribe处理AzkabanOoziePigHiveHadoopMapReduceS4Storm存储VoldemortCassandraHbase呈现BigDataApplications?33小心！再次落入陷阱!复杂技术架构.令人困惑的供应商选择呈现处理存储摄入需要全面“端到端”的方法跨越所有的数据类型客户信息汽车机器数据智能仪表PoC移动结构化数据点击流社交网络基于位置的数据文本数据IMHO,it’sgreat!RFIDSAPHANA不断创新OLAP（2010-2011）HANA内容(报表及分析)HANA加速器HANA平台(数据集市)HANA应用CloudonHANABusinessOneAnalyticsonHANAAnyDBHANA数据库客户端SAP商务套件AnyDB客户端SAP商务套件HANA数据库(视图)AnyDBHANA数据库SAP商务套件客户市场应用AnyDBHANA数据库应用SAP商务套件MSQL客户端SAPBusi-nessOneHANA数据库AnyDBHANA数据库客户端OD/SF解决方案转变场景HANA新应用HANA数据库应用OLTP+OLAP（2012-Now）BusinessSuiteonHANABusinessOneonHANABWonHANAHANA数据库BWHANA数据库CRMSCMSRMPLMERP视图SAP商务套件BW应用HANA数据库SAPBusinessOne从OLAP到OLTP从Side-by-Side到主数据库从BI到预测从结构化到非结构化HANA–简化BI与分析三十辐共一毂，当其无，有车之用也。埏埴以为器，当其无，有器之用也。凿户牖以为室，当其无，有室之用也。故有之以为利，无之以为用。--老子十一巨量数据分析移动交易处理DB引擎内存技术DB引擎分析网格DB引擎MapReduceBatchComputeFramework摄入存储处理呈现SybaseReplicationServer,SAPBusinessObjectsDataServices(跨越不同部署选项整合/同步数据)SybaseESP事件与流处理SAP巨量数据处理框架SAPHANASybaseIQSybaseESP监控/过滤事件流半结构化数据结构化数据非结构化数据HadoopSybaseASEHive/HDFS巨量数据应用HANA–软件和硬件的共同创新行与列存储压缩分区无汇总表无含数据的视图实时复制只插入差异数据5x压缩1TB数据,~200GB内存列式存储=快速查询分析巨量数据集复杂计算灵活建模无需数据复制快速数据Loading多核CPU大内存计算能力：比硬盘访问快1Mx1TB服务器,64-80核在应用层处理数据PushDown业务和代码内存计算–就是现在!HWTechnologyInnovations64bitaddressspace–2TBincurrentservers100GB/sdatathroughputDramaticdeclineinprice/performanceMulti-CoreArchitecture(8x8coreCPUperblade)MassiveparallelscalingwithmanybladesOneblade~$50.000=1EnterpriseClassServerRowandColumnStoreCompressionSAPSWTechnologyInnovations内存计算–就是现在！HWTechnologyInnovations64bitaddressspace–2TBincurrentservers100GB/sdatathroughputDramaticdeclineinprice/performanceMulti-CoreArchitecture(8x8coreCPUperblade)MassiveparallelscalingwithmanybladesOneblade~$50.000=1EnterpriseClassServerRowandColumnStoreCompressionPartitioningSAPSWTechnologyInnovations内存计算–就是现在！HWTechnologyInnovations64bitaddressspace–2TBincurrentservers100GB/sdatathroughputDramaticdeclineinprice/performanceMulti-CoreArchitecture(8x8coreCPUperblade)MassiveparallelscalingwithmanybladesOneblade~$50.000=1EnterpriseClassServerRowandColumnStoreCompressionPartitioningNoAggregateTablesSAPSWTechnologyInnovations快速！为内存计算进行软件优化传统数据库按行存储按列存储使快速内存操作（例如汇总）成为可能列布局支持顺序内存访问简单汇总行扫描一次即可A10€B35$C2€D40€E12$ABCDE103524012€$€€$memoryaddress按行组织按列组织A10€B35$C2€D40€E12$概念视图映射至内存SAPHANA™■In-Memorysoftware+hardware(HP,IBM,Fujitsu,Cisco,Dell)■DataModelingandDataManagement■Real-timeDataReplication■SAPBusinessObjectsDataServicesforETLcapabilitiesfromSAPBusinessSuite,SAPNetWeaverBusinessWarehouse(SAPNetWeaverBW),and3rdPartySystemsCapabilitiesEnabled■Analyzeinformationinreal-timeatunprecedentedspeedsonlargevolumesofnon-aggregateddata■Createflexibleanalyticmodelsbasedonreal-timeandhistoricbusinessdata■Foundationfornewcategoryofapplications(e.g.,planning,simulation)tosignificantlyoutperformcurrentapplicationsincategoryMiiidtdlitiSAPIn-MemoryAppliance(SAPHANA™)SAPHANAStudioReal-TimeDat