Skip to end of metadata
Go to start of metadata

The Social Intelligence Benchmark (SIB) is designed for evaluating a broad range of technologies for tackling graph database workloads.

Scenario

The scenario of the benchmark, a social network, is chosen with the following goals in mind:

  • it should be understandable to a large audience, and this audience should also understand the relevance of managing such data
  • the scenario in the benchmark should cover the complete range of interesting challenges, according to the benchmark scope
  • the query challenges in it should be realistic in the sense that, though synthetic, similar data and workloads are encountered in practice

SIB is intended to be a plausible look-alike of all aspects of operating a social network site, from the online transactions to business intellligence and graph analytics.  SIB aims at capturing the essential features of these usage scenarios while abstracting away details of specific business deployments.   All the SIB scenarios share a common scalable synthetic data set.  The data generation is specially designed to reproduce features of real data, including realistic data distributions and correlations.

Audience

SIB is intended  to provide the following value to different stakeholders:

  • For end users facing graph processing tasks, SIB provides a recognizable scenario against which it is possible to compare merits of different products and technologies.  By covering a wide variety of scales and price points, SIB can serve as an aid to technlogy selection.
  • For vendors of graph database technology, SIB provides a checklist of features and performance characteristics that helps in product positioning and can serve to guide new development.
  • For researchers, both industrial and academic, the SIB dataset and workload provide interesting challenges in multiple choke-point areas, such as query optimization, (distributed) graph analysis, transactional throughput, and provides a way to objectively compare the effectiveness and effiency of new and existing technology in these areas.

Workloads

The SIB is in fact three distinct benchmarks with a common dataset, since there are three different workloads. Each workload produces a single metric for performance at the given scale and a price/performance metric at the scale.  The full disclosure further breaks down the composition of the metric into its constituent parts, e.g. single query execution times.

  • On-Line Workload.  The Online SIB workload tests a system's throughput with relatively simple queries with concurrent updates.  The workload test ACID features and scalability in an online operational setting. The system under test (SUT) is expected to run in a steady state, providing durable storage with smooth response times.  Updates are typically small, affecting a few nodes at a time, e.g. uploading of a post and its tags.  Transactions may require serializability, e.g. verifying that something does not exist before committing the transaction.   Updates will conflict a small percentage of the time.  Reads do not typically require more than read committed isolation.  Queries typically touch a small fraction of the database, up to hundreds of thousands of values.  The count of values accessed by the queries does not significantly increase as the database is scaled up.
  • Business Intelligence Workload. The workload consists of complex structured queries for analyzing online behavior of users for marketing purposes.  The workload stresses query execution and optimization. Queries typically touch a large fraction of the data and do not require repeatable read.  The queries are concurrent with trickle load.  The queries touch more data as the database grows.
  • Graph Analytics Workload. This workload tests the functionality and scalability  of the SUT for graph analytics that typically cannot be expressed in a query language. The analytics is done on most of the data in the graph as a single operation.  The analysis itself produces large intermediate results.  The analysis is not expected to be transactional or to have isolation from possible concurrent updates.

Systems

The technological scope of the SIB comprises all systems that one might conceivably use to perform social network data management tasks:

  • Graph database systems (e.g. neo4j, InfiniteGraph, DEX, Titan)
    • these systems store directed, labeled graphs; and support traversals via APIs
    • they often also support a query language (e.g. Gremlin, Cypher), but this may not be the primary interface
    • queries/programs/tasks programmed against the graph data often involve updating a state speficif to the task attached to potentially all nodes/edges
    • these systems often support value-based indexes to quickly locate nodes/edges
    • these systems often support transactional queries, with some degree of consistency
    • typically single-machine architecture (non-cluster)
    • workloads: Online + Business Intelligence + Graph Analysis (though maybe with difficulty)
  • Graph programming frameworks (e.g. Giraph, Signal/Collect, Graphlab, Green Marl)
    • the core of this system is a language/api that allow to create graph manipulations focused on successive actions on sets of nodes, executing in parallel or lockstep
    • these systems often interface (or take the form aof a library) inside a programming language, such that graph manipulation steps and custom logic are intertwined
    • these frameworks typically target global graph computations  
      • with long latency 
      • involving many graph nodes/edges
      • which often compute approximation answers of problems that are often NP-complete
    • both single-machine and cluster systems exist
    • workloads: Business Intelligence + Graph analytics
  • RDF database systems (e.g. OWLIM, Virtuoso, BigData, Jena TDB, Stardog, Allegrograph)
    • these systems implement the SPARQL1.1 query language similar in complexity to SQL1992, which allows for structured queries, and simple traversals
    • RDF database system often come with additional support for simple reasoning (sameAs,subClass) , text search and geospatial predicates
    • RDF database systems generally support transactions, but not always with full concurrency and serializability 
    • RDF database systems suppsed strength is integrating multiple data sources (e.g. DBpedia)
    • both single-machine and cluster systems exist
    • workloads: Online + Business Intelligence
  • Relational database systems (e.g. Postgres, MySQL, Oracle, DB2, SQLserver, Virtusoso, MonetDB, Vectorwise, Vertica)
    • data is relational, and queries are formulated in SQL and/or PL/SQL
    • both single-machine and cluster systems exist
    • relational systems do not nornammly support recursion, or stateful recursive algorithms, which makes them not at home in the graph analytics workloads
    • workloads: Online + Business Intelligence 
      • however, the Graph Analytics workload could potentially be expressed by keeping state in temporary tables, and using the functional features of PL-SQL 

Design Objectives

In order to serve these functions, SIB is designed around the following objectives:

  • Rich coverage. SIB is intended to cover most demands encountered in operating a social network.  
  • Modularity. Not all products and technologies will fit all workloads.  For this reason SIB is broken into parts that can be individually addressed.  In this manner SIB stimulates innovation without imposing an overly high threshold for participation.
  • Reasonable implementation cost. For a product offering relevant functionality, the effort for obtaining initial results with SIB should be small, on the order of days.   
  • Relevant selection of challenges.  Benchmarks are known to direct product development in certain directions.  SIB is informed by the state of the art in database research so as to offer optimization challenges for years to come while not having a prohibitively high threshold for entry.
  • Reproducibility and documentation of results. SIB specifies rules for full disclosure of benchmark execution and for auditing of benchmark runs. The benchmarks may be run on any equipment but the exact configuration and price of the hardware and software must be disclosed.
  • No labels