1.Bidding Infrastructure

GoogleConfidentialandProprietaryBiddingInfrastructureWhereAremyCallouts?IgorFooxOriginallybyEricBurnettGoogleConfidentialandProprietaryContents1.GoogleNetworkoCalloutSourcesoDNS2.AdExchangeoQuotaEnforcementoErrorThrottling3.BidderBestPracticesoConnectionLifetimesoOverloadsoPingRequestsGoogleConfidentialandProprietary1.GoogleNetworkoCalloutSourcesoDNS2.AdExchangeoQuotaEnforcementoErrorThrottling3.BidderBestPracticesoConnectionLifetimesoOverloadsoPingRequestsContentsGoogleConfidentialandProprietaryAutonomousSystemsThreebroadclasses(mydefinitions):•ISPsoLarge,wellconnected,regionspecific,predominantlyconsume•TransitProvidersoGeographicallydistributed,verywellconnected,neitherproducenorconsume•Producers/ConsumersoSmallnumberofdisconnectedlocations,fewfatconnectionsVeryroughtaxonomy.WhichisGoogle?GoogleConfidentialandProprietaryRouting-DecisionalgorithmsHotpotato,coldpotato•Hotpotato:Hand-offpacketasquicklyaspossible•Coldpotato:CarrypacketasfaraspossibleNeitherisoptimal!Googleaimsforlowestlatency,erringtowardscoldpotato.GoogleConfidentialandProprietaryGoogle'sNetworkMultipledistributeddatacentersOurown'backbone'network(orinsomerespects,two)•Optimizedforedge-to-coretoserveusertrafficquickly,•Optimizedforcore-to-coretoconnectourdatacentersPerpeeringdb.comtoday•presencein132publicexchanges,72private•Wellover1.5TbpspubliccapacityPer2009*AtlasTop10:•3rdlargestISPglobally*Iknowofnomorerecentstudies,sorryGoogleConfidentialandProprietaryGoogle'sNetwork-PointsofPresenceGoogleConfidentialandProprietaryBidRequestSourcesBeforeTradingLocations:•BidRequestssentdirectlyfromourdatacenters•120mstimeoutcalculatedatdatacenter•Undeclaredlocations!oAllyougotwereURLstoping:(•ChangedregularlyGoogleConfidentialandProprietaryBidRequestSourcesFullTradingLocations:•BidRequestssentviaproxiesattradinglocations•100mstimeoutcalculatedatproxies•Fourdeclared,staticlocationsoNorthernVirginia,SanFrancisco,Amsterdam,andHongKongoAndURLstopingforeachGoogleConfidentialandProprietaryReducingLatencyfromHongKongEarlybiddersetupshadlatencyissuesRecentlysomebiddershaveimprovedtheirinfrastructure•PossibletobidsuccessfullyfromChina•Somebiddersaregettingaslowas15msaveragelatencies•LocatingserversinGuandongGoogleConfidentialandProprietaryDNS•Weusesharedrecursiveresolvers-IPloadbalancing/roundrobindoesn'twork•Ourresolversarenotattheproxylocations-geo-basedsolutionslikeGSLBwon'tworkcorrectly•MultipledatacenterssharingoneURL-SinglePointofFailure,throttledtogetherProtip:don'tdoanythingfancyGoogleConfidentialandProprietaryContents1.GoogleNetworkoCalloutSourcesoDNS2.AdExchangeoQuotaEnforcementoErrorThrottling3.BidderBestPracticesoConnectionLifetimesoOverloadsoPingRequestsGoogleConfidentialandProprietaryQuotaEnforcementWehaveanewsystem!Oldversion:•EachGoogledatacenter(DC)getsoneURLandafixedquotaoSooneDCcouldn'tsplittraffictotwoURLs•Quotasupdatedhourly•URLsforaDCselectedautomatically,butnottotallyreliablyoreliesonbidderssupporting'ping'requests,whichnotalldooguessesbestchoicebasedonnetworklatencyOREGON1:500QPSto://bid2.bidder.com...GoogleConfidentialandProprietaryQuotaEnforcementNewversion:•URLsaredefinedwithaquotaandaregion•MultipleURLscanhappilyoverlap•OverlappingURLsarethrottledindependently,biasedtowardslowerrorratesASIA:1000QPSto://bid2.bidder.com...Also:•Fullydynamic(seconds)•FullyautomatedGoogleConfidentialandProprietaryErrorThrottlingErrorthrottlingaimstocutbackonbidrequestswhenerrorratesclimb,tomaximizeusefulbidrequestsHowdoesitwork?•Enforcedbyindividualworkertasks,perURL•WorkerstasksmonitorthesuccessrateofrequestsoverawindowoIf85%succeeded,throttledown,elsethrottleup•Eachworkerisveryinexact,butinaggregatetheyhavetheeffectof'aiming'forthequotathatresultsin15%errorsGoogleConfidentialandProprietaryErrorThrottlingNotethat•Persistenterrorrateswillcauseworkerstocontinuetothrottledownuntilquotashitthefloor•Eventhrottled,abidderwillbepushedashardasitcango•Withnewquotasystem,ifoneURListhrottledcalloutswillgotoanotherBesttorelyonthrottlingforrarecases,andsetquotasjustunderthelimitotherwiseGoogleConfidentialandProprietaryContents1.GoogleNetworkoCalloutSourcesoDNS2.AdExchangeoCalloutQuotaEnforcementoErrorThrottling3.BidderBestPracticesoConnectionLifetimesoOverloadsoPingRequestsGoogleConfidentialandProprietaryBestPractices:ManagingConnections•KeepConnectionsAlive•HandleOverloadGracefully•RespondtoPings•ConsiderPeeringMorewillfollow!GoogleConfidentialandProprietaryBestPractices-ConnectionLifetimesKeepconnectionsaliveaslongaspossible:•Were-useconnectionsheavily•Newconnectionsareexpensive•Serverdefaultsaren'ttunedforouruse-caseoThekeep-alivetimeoutismostimportant,butthereareafewotherparametersyoumightneedtotuneaswell.MoredetailsontheBestPracticespageGoogleConfidentialandProprietaryBestPractices-OverloadsPrepareforoverloadsituations:•Overloadshappen•Don'ttrytohandleallrequests!•Considerdroppingrequestsearlyaslatenciesr