构建可扩展可演进大数据架构-毕洪宇

构建可扩展可演进⼤大数据架构GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC毕洪宇Agenda•背景介绍及⺫⽬目标•数据平台•数据应⽤用•团队•总结GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC关于饿了么16亿元+08年10年12年14年16年交易2年份800600400200城+数年份700+.均交易210城+008年10年12年14年16年日均交易116亿元+日0单+500万+覆.城市700+GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCbackground•••越来越多的场景需要数据驱动决策•⼈人难招业务快速发展GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC数据需求越来越强motivation••可扩展，为持续演进提供基础GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC可演进，形成闭环数据平台•TechArchitecture:•数据服务•数据引擎•数据基础架构Persistenttfm*DataInfrastructureadhoct!m*DataServiceGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCDataEngine查询平台••运⾏行状态⼤大屏•保护机制•笛卡尔积，分区检查•资源检查•splitsizetuning多数据源，查询引擎切换GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC数据视图引擎••SQL即接⼝口SQL即报GITC表GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC报表类型•••细粒度权限•细粒度开关多控件图表⽀支持GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC可拖拽式开发SQL⽣生成图表GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCdrag&drop⽅方式GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC查询引擎选型•MMR与MPP的区别•分析型•交互型GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCOLAPengine:kylin•商业常品对接⽅方便•基于查询热度构建CubeGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC⼤大数据量存储引擎选型•MySQL＋DAL：核⼼心调⽤用路径•Cassandra：⾮非核⼼心调⽤用路径•Redis-Cluster：⾼高速k-vstore•Elasticsearch：多维度聚合查询GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC调度引擎•并⾏行抽取，⽀支持分库分表•⽀支持多种数据库推送：MySQL/HBase/Cassandra/Redis•将guideline融⼊入到⼯工具中：⾃自动依赖•关键路径性能分析•所有任务情况可回溯GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC数据应⽤用••算法应⽤用实时业务GITC看GITCG板ITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC实时业务看板GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC计算链路•解析复⽤用•HAMySQLNginxBinlogParserFlumeKa8aStormMySQLRedisClusterDataPipelineGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC•扩展存储：•⼿手动分⽚片－三层架构－⾃自动分⽚片••本地化聚合•多流异步合并实时计算Storm可扩展优化实时存储Redis-luster实时监控集群性能监控容量-析水位系统扩展计算：GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC计算领域CAP•成本•⾼高吞吐低时延•精准度GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC算法应⽤用•个性化美⾷食推荐•红包营销•物流调度GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITC实验平台•驱动算法快速迭代算法验平台GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCabkey-point•流量分配•••流量shuffle•样本量•异常监控multi-layer请求复⽤用GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCmulti-layer使⽤用正交的hash函数团队⼯工具••jira•workflow：bug管理，敏捷看板，变更管理，任务管理•report⼯工具：资源图，to-dolist，变更⽇日报等；wiki：知识共享，SOP，故障报告管理；GITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCsoftskill•沟通：会议必须可落地&&结果确认需落地&&信息分类••failforward:DERPSceneProtectionEscala%onRemedia%onPreven%onGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITDCeGItTeCcGI%TCoGnITCGITClet’sdatatalk:e.g.alertreport总结•nomeasure，noimprovement•••automateeverything•everythingmayfail•conway’slawbalancedesign，thenoptimisationGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCGITCsyncattherighttime–MarkZhang“ThewaytodoreallybigthingsistodoreallysmallthingsGITCGITCGITCGITCaGITnCGdITCGgITCrGoITCwGITCthGITeCGmITCGIbTCiGgITCgGITeCGrI.T”CGITCGITCGITCGITC