eBay_Cloud_CMS-XXXX_QCon(1)

eBayInc.Proprietary&ConfidentialeBayCloudConfigurationManagementSystem蒋旭平台技术部架构师eBay中国技术研发中心eBayInc.confidential2Agenda• eBayCloudOverview– WhyeBayNeedCloud?– eBayCloudTechOverview• CMS-ConfigurationManagementSystem– Architecture– TryMePage– Functionality&Demo• NoSQLinCMS– WhyCMSchooseNoSQL?– OvercomeNoSQLDesignChallenge– ResolveOpenSourceNoSQLIssueeBayInc.confidential3WhyeBayneedcloud?eBayInc.confidential4eBayScale2Bpageviews/day96Mactiveusers500Mlivelistings5Bqueries/day75Bdatabasecalls/day9PBofdata14,000applicationservers44MlineofcodeData!Analytics!Search!Infrastructure!Front!End!10Mitemsadded/dayeBayInc.confidential5eBayUtilizationNumberofserversrequiredbasedonutilizationfor8poolseBayInc.confidential6eBayGlobalBrandseBayInc.confidential7eBayCloudTechOvervieweBayInc.confidential8eBayCloudTechnologyStackServiceCatalogTicketdrivenrunbookautomationChargebackRESTAPIsModelDrivenCloseLoopAutomationPayAsYouGoConfigurationManagementDatabase(CMDB)DistributedStateManagementMonitoringComplexEventProcessingeBayInc.confidential9eBayCloudArchitectureOverviewCloudManagerConfigurationManagementServiceMonitoringInfrastructure&PlatformMgtServicesRESTAPIQueueAPIRESTAPIQueueAPIRESTAPICloudInfrastructureAgentmetrics!Control!Current/expected!state!Discovery!Control!Events&!alerts!Thresholds/topology!RESTAPIQueueAPIeBayInc.confidential10ModelDrivenAutomationLBPoolServerServerServerCurrentStateSiteDiscoveryComparisonExpectedStateReconciliationOrchestrationLBPoolServerServerServer• DesiredconfigurationisspecifiedintheexpectedstateandpersistedinCMS• Uponapproval,theorchestrationwillconfigurethesitetoreflectthedesiredconfiguration.• Updatedsiteconfigurationisdiscoveredbasedondetectionofconfigurationevents• Reconciliationbetweentheexpectedandcurrentstateallowstoverifytheproperconfiguration.• Ongoingvalidationallowsthedetectionofoutofbandchanges.eBayInc.confidential11ConfigurationManagementSystem(CMS)eBayInc.confidential12CMS-Overview• CMS(ConfigurationManagementSystem)isahigh-performancemetadata-drivenpersistenceandqueryserviceforconfigurationdatawithsupportingofRESTfulAPIandclientlib(Java,Python).• CMSisagenericsystemthatbeusedforcloudconfiguration,aswellothersoftwareneedsforconfiguration.• Asaby-product,CMScanbeapersistencesolutionforreal-timestatedataaswell.• CMSsupportsmultipledatarepositoriesfordesireddataisolation.eBayInc.confidential13CMS-ArchitectureRESTAPIMetadataServiceQueryEngineEntityManagerDataAccessLayerMongoDBPersistenceServiceSearchServiceParserTranslator&OptimizerExecutorEntityMapperRESTRequestEntityServiceBranchServiceHistoryServiceeBayInc.confidential14CMS-TryMePageeBayInc.confidential15CMSFunctionality&DemoeBayInc.confidential16MetadataModel–BasicFeature• Themetadatamodelisbasedonobject-orientedparadigmthatcansupportgraph/treedatamodel– MetaClassdefinethemetatypeofruntimedata(i.e.entity)– Entityrepresentonenodeingraph– Relationshipbetweenentityrepresenttheedgeingraph• Themetadatacancontaintwotypesoffield:– Attributefielddefinepayloadofentity• String,Boolean,Double,Integer,Long,Date• Json– Relationshipfielddefinerelationshipbetweenentity.• Reference• EmbeddedeBayInc.confidential17MetadataModel–AdvancedFeature• MetadataInheritance(parent&child)• ReferenceIntegrity(strong&weak)• IndexSupportonMetadata(uniquecontraints&queryoptimizer)• MongodbCollectionSplitbyMetadata(break64indexlimitation)eBayInc.confidential18PersistenceService–BasicFeature• ThepersistenceserviceprovidesCRUDAPIfortheruntimedata(i.e.entity)ofmetadata.– Create– Retrieval– Update– Delete• Theentitycanbeflat-structureorembedded-structurethatconformedtothemetadatadefinition– Forreferencerelationship,entityisflat-structure– Forembeddedrelationship,entityisembedded-structureeBayInc.confidential19PersistenceService–AdvancedFeature• Branching(main&sub&merge)• AuditTracing(entityhistory)• ReferenceIntegrity(strong&weak)• ConditionalUpdate(versionbasedoptimisticlocking)• SecurityAccessControleBayInc.confidential20QueryService–BasicFeature• Thequeryserviceprovidesanimperativestylequerylanguagethatdefinesthetraversalpathofgraph/treedatamodel.• ThequerylanguagesupportsBooleanfilter,attributeselectionandimplicitjointhatwillextractasub-treeresultfromgraphdataset.• Forexample,*ApplicationService[@name=comp-iq].groups[@name=columns].groups[@name=col1].serviceInstances*willreturnserviceinstancesundercolumn1ofcomp-iqapplication.eBayInc.confidential21QueryService–AdvancedFeature• QueryOptimizer(cost&hint)• ResultPagination(sort/limit/skip)• FullTableScanCheck(queryfilter&indexinfo)• QueryExplanation(executionplan)eBayInc.confidential22SystemManagement• Monitoring(approximate&accurateslidingwindowmetrics)• StateManagement(normal/maintain/overload)• HealthyModel(formulabasedonqps&latency-overloadstate)• APIThrottling(overloadstate-prioritythrottling)eBayInc.confidential23OpenSourceStrategy• PlantoopensourcethecorefunctionalityofCMS• Separatetheebay-relatedcode(e.g.security)fromopensourcecode• Welcometocontributecode!eBayInc.confidential24NoSQLinCMSeBayInc.confidentia