MIDDLEWARESYSTEMS
RESEARCHGROUP
MSRG.ORG
Big Data Challenges in Application Performance Management
TilmannRablHans-Arno JacobsenSergeMankovskiiXLDB Conference 2011
Abstract
2
Modern Web Data Platforms (WDPs) handle large amounts of data and activity through massively distributed infrastructures. To achieve performance and availability at Internet scale, WDPs restrict querying capability, and provide weaker consistency guarantees than traditional ACID transactions. The reduced functionality is sufficient for many web applications.High data and query rates also appear in application performance management (APM). APM has similar requirements like current web based information systems such as weaker consistency needs, geographical distribution and asynchronous processing. At the same time, APM has some unique features and requirements that make previously published research and existing architectures inapplicable.
Application Performance Management
Enterprise system architecturesVery complex distributed systemsNeed of detailed monitoringService level agreementsApplication performance managementHow many transactions fail?Where is the root cause of failure?What is the end to end response time?Which component is the bottleneck?Which and how many transactions are there?
3
Enterprise System Architecture
4
JSR – 163JVM is augmented with agentAgent can run additional codeNo change of code baseTrace transactionsMeasure response timesOther types of measurementsHuge number of eventsPotentially for every method invocation
Java Byte Code Instrumentation
5
APM Performance Requirements
High insert ratesMillions inserts / secHigh query ratesThousands queries / secWrite ratio: >99 %Agents send data in bulksDifferent periods (seconds to minutes)Big data250 Bytes per record~ 250 MB / sec~ 600 TB / month
6
MADRID Project
Current system’s performanceYCSB results < 15K ops / secTPC-C results ~ 500K transactions / secNeed for a new architectureMassive Asynchronous DistRIbuted DataHighly scalableHigh write throughputApart from measurements data mostly staticStatic queriesHybrid key-value store
7
EntryLog
In-MemoryStorage
Disk Storage
MADRID Architecture
Materialized ViewsStatic queriesFiltersNotificationsHybrid data storeAll nodes are equalDHT style insertsReplication for static dataAsynchronous processing
8
ViewManager
MessageBroker
Schema Excerpt
Transaction typesNo instancesGraph structureMetricpertransactionType of measurementMeasurementsPer transaction typePer metric typeCan be aggregations
9
Materialized Views I
What is the average runtime of transaction XY?
10
SELECTtransaction_name, AVG(end_time-start_time)FROMMeasurement ms, Metricmt, Transaction tWHEREms.metric_id=mt.metric_idANDmt.transaction_id=t.transaction_idANDmt.metric_type= “runtime_metric”ANDms.start_timeBETWEEN“18/10/2011”AND“19/10/2011”ANDt.transaction_name= “XY”
Materialized Views II
What is the average runtime of transaction XY?
11
Contact
TilmannRablUniversity of Torontotilmann@msrg.utoronto.caHans-Arno JacobsenUniversity of Torontowww.msrg.orgSergeMankovskiiCA Labsmankovskii@ca.com
12
0
Embed
Upload