New Website (still old content)

Hi all…

As some of you might have noticed, I moved to WordPress (after the Joomla installation I was running bit-rot due to a php upgrade of my hosting).

The content of the site is however still very old… If you care to see what I am up to more recently you are better of looking me up on:

LinkedIn: https://www.linkedin.com/in/curino

Twitter: https://twitter.com/krlis1337

Google Scholar : https://scholar.google.ca/citations?user=wb9DYOEAAAAJ&hl=en&oi=ao

In Hadoop: https://issues.apache.org/jira/browse/YARN-1051?jql=text%20~%20%22curino%22

… you get the gist …

 

 

OLTP-Bench at VLDB 2014

The work behind this paper started long ago (2009?), working with Evan Jones and Yang Zhang on some infrastructure to test our DBaaS project (http://relationalcloud.com) and continued with Andy Pavlo and I cursing about how painful it is to build a reasonable testing infrastructure (is like a tax on DB/system PhD students).

We decided to make it easier for future generations, and started to combine Andy’s workloads and mine, and the strength of both infrastructures (at the time a beautiful mess of hacky codes and half-fast scripts). Djellel Difallah and Phil Cudre-Maroux join the effort (and arguably Djellel put in more hours than anyone else on this since then). We polished the infrastructure and added several more workloads, with help and input from many people including Rusty Sears, Ippokratis Pandis, Barzan Mozafari, Dimitri Vorona, Sam Madden, and Mark Callaghan.

The goal was to produce enough critical mass of features and workloads. So that other researchers would prefer to pick up this infrastructure and contribute to it, rather than building from scratch. This seems to be working as we received many requests and contributions from companies and academics all around the world.  Andy Pavlo is now heading a revamp of the website, including much needed graphing and comparisons interfaces.

Hopefully our community can rally behind this effort, and drive it in whichever direction seems appropriate (we are open to extensions and changes, even drastic), reducing the repeated work, and fostering some better repeatability and ease of comparison among “scientific” results in papers.

Checkout the paper here:

http://www.vldb.org/pvldb/vol7/p277-difallah.pdf

Our website at:

http://oltpbenchmark.com

And get the code from github:

https://github.com/oltpbenchmark/oltpbench?source=cc

 

2012 VLDBJournal: Automating the Database Schema Evolution Process

Abstract Supporting database schema evolution represents a long-standing challenge of practical and theoretical importance for modern information systems. In this paper we describe techniques and systems for automating the critical tasks of migrating the database and rewriting the legacy applications. In addition to labor saving, the benefits delivered by these advances are many, and include reliable prediction of outcome, minimization of down-time, system-produced documentation, and support for archiving, historical queries and provenance. The PRISM/PRISM++ system delivers these benefits, by solving the difficult problem of automating the migration of databases and the rewriting of queries and updates. In this paper we present the PRISM/PRISM++ system, and the novel technology that made it possible. In particular we focus on the difficult, and previously unsolved problem of supporting legacy queries and updates under schema and integrity constraints evolution.

The PRISM/PRISM++ approach consists in providing the users with a set of SQL-based Schema Modification Operators (SMOs) which describe how the tables in the old schema are modified into those in the new schema. In order to support updates, SMOs are extended with Constraints Modification Operators (ICMOs). By using recent results on schema mapping, the paper (i) characterizes the impact on integrity constraints of structural schema changes, (ii) devises representations that enable the rewriting of updates, and (iii) develop a unified approach for query and update rewriting under constraints.

We complement the system with two novel tools: the first automatically collects and provides statistics on schema evolution histories, whereas the second derives equivalent sequences of SMOs from the migration scripts that were used for schema upgrades. These tools were used to produce an extensive testbed containing 15 evolution histories of scientific databases and web information systems, providing over 100 years of aggregate evolution histories and almost 2000 schema evolution steps.

 

To appear in VLDB Journal “Best of VLDB 2011”


2012: Keynote at CloudDB “Benchmarking OLTP/Web Databases in the Cloud: the OLTP-Bench Framework”

Benchmarking is a key activity in building and tuning data manage- ment systems, but the lack of reference workloads and a common platform makes it a time consuming and painful task. The need for such a tool is heightened with the advent of cloud computing— with its pay-per-use cost models, shared multi-tenant infrastruc- tures, and lack of control on system configuration. Benchmarking is the only avenue for users to validate the quality of service they receive and to optimize their deployments for performance and re- source utilization.

In this talk, we present our experience in building several ad- hoc benchmarking infrastructures for various research projects tar- geting several OLTP DBMSs, ranging from traditional relational databases, main-memory distributed systems, and cloud-based scal- able architectures. We also discuss our struggle to build mean- ingful micro-benchmarks and gather workloads representative of real-world applications to stress-test our systems. This experience motivates the OLTP-Bench project, a “batteries-included” bench- marking infrastructure designed for and tested on several relational DBMSs and cloud-based database-as-a-service (DBaaS) offerings. OLTP-Bench is capable of controlling transaction rate, mixture, and workload skew dynamically during the execution of an ex- periment, thus allowing the user to simulate a multitude of prac- tical scenarios that are typically hard to test (e.g., time-evolving access skew). Moreover, the infrastructure provides an easy way to monitor performance and resource consumption of the database under test. We also introduce the ten included workloads, derived from either synthetic micro benchmarks, popular benchmarks, and real world applications, and how they can be used to investigate various performance and resource-consumption characteristics of a data management system. We showcase the effectiveness of our benchmarking infrastructure and the usefulness of the workloads we selected by reporting sample results from hundreds of side-by- side comparisons on popular DBMSs and DBaaS offerings.

 

More details at: http://oltpbenchmark.com

Compile Hadoop Trunk on Mac Java 7

(UPDATED)

I saw my Java 6 disappear… and I don’t even want to know how to blame or rant about, but I need to compile Hadoop (trunk as of early november) on my macbook running 10.7.5 (latest updates at the time of this post) with the Oracle Java 1.7.0.8.

This is briefly what worked for me… but as a disclaimer this is a hackish quick-fix to get the thing to compile, not a clean mvn marvel.

There are two parts of this:

1) fix a pom.xml in hadoop-common-project/hadoop-annotations by adding the following

 

This fixes the compile errors coming from the hadoop-common-projects. (apologies for the image, but it is joomla is kicking and screaming on the tags)

 

2) (ugly) Fix the compile errors for hadoop-hdfs

sudo mkdir /Library/Java/JavaVirtualMachines/jdk1.7.0_08.jdk/Contents/Home/Classes  cd !$ sudo ln -s ../lib/tools.jar classes.jar

This solves the problem of ant/mvn/make/someone that looks for tools.jar into the wrong place by linking the wrong place to the right place. Now this is a disgusting trick, but after fussing around with mvn (which I don’t know well at all) for a bit, I decided to be pragmatical.

Potential (cleaner) solution that Markus Weimer suggested is the following:

I haven’t tested it yet, but here it is.

 

The final version of this consists in using:


<profile>
   <id>macdep</id>
   <activation>
      <os>
        <name>macprofile</name>
        <family>mac</family>
      </os>
    </activation>
    <dependencies>
    <dependency>
     <groupId>jdk.tools</groupId>
     <artifactId>jdk.tools</artifactId>
     <scope>system</scope>
     <systemPath>/Library/Java/JavaVirtualMachines/jdk1.7.0_08.jdk/Contents/Home/lib/tools.jar</systemPath>
     <version>1.7</version>
    </dependency>
   </dependencies>
  </profile>

In the <profiles> section of the main pom.xml, and in the hadoop-annotations.

 

CFP: Data Management in the Cloud 2013

I am co-organizer with Ashraf Aboulnaga for the Second Workshop on Data Management in the Cloud, co-located with ICDE (Brisbane, Australia). We expect it to be a highly interactive forum for both practitioners and academics interested in the space of data management and cloud computing, and we welcome both novel research and industry experience papers.

CFP and details at: http://db.uwaterloo.ca/dmc2013/ and below the high level idea of the workshop:

Cloud computing has emerged as a promising computing and business model. By decoupling the management of the infrastructure (cloud providers) from its use (cloud tenants), and by allowing the sharing of massive infrastructures, cloud computing delivers unprecedented economical and scalability benefits for existing applications and enables many new scenarios. This comes at the cost of increased complexity in managing a highly multi-tenant infrastructure and limited visibility/access posing new questions on attribution, pricing, isolation, scalability, fault-tolerance, load balancing, etc. This is particularly challenging for stateful, data-intensive applications.

This unique combination of opportunities and challenges attracted much attention from both academia and industry. The DMC workshop aims at bringing researchers and practitioners in cloud computing and data management systems together to discuss the research issues at the intersection of these two areas, and also to draw more attention from the larger data management and systems research communities to this new and highly promising field.

 

See you in Australia!!!

2012: Benchmarking OLTP/Web Databases in the Cloud: The OLTP-Bench Framework

I will be giving the keynote at CloudDB 2012 on relational database benchmarking. Below you can find the abstract of the invited paper.

Benchmarking is a key activity in building and tuning data manage- ment systems, but the lack of reference workloads and a common platform makes it a time consuming and painful task. The need for such a tool is heightened with the advent of cloud computing— with its pay-per-use cost models, shared multi-tenant infrastruc- tures, and lack of control on system configuration. Benchmarking is the only avenue for users to validate the quality of service they receive and to optimize their deployments for performance and re- source utilization.

In this talk, we present our experience in building several ad- hoc benchmarking infrastructures for various research projects tar- geting several OLTP DBMSs, ranging from traditional relational databases, main-memory distributed systems, and cloud-based scal- able architectures. We also discuss our struggle to build mean- ingful micro-benchmarks and gather workloads representative of real-world applications to stress-test our systems. This experience motivates the OLTP-Bench project, a “batteries-included” bench- marking infrastructure designed for and tested on several relational DBMSs and cloud-based database-as-a-service (DBaaS) offerings. OLTP-Bench is capable of controlling transaction rate, mixture, and workload skew dynamically during the execution of an ex- periment, thus allowing the user to simulate a multitude of prac- tical scenarios that are typically hard to test (e.g., time-evolving access skew). Moreover, the infrastructure provides an easy way to monitor performance and resource consumption of the database under test. We also introduce the ten included workloads, derived from either synthetic micro benchmarks, popular benchmarks, and real world applications, and how they can be used to investigate various performance and resource-consumption characteristics of a data management system. We showcase the effectiveness of our benchmarking infrastructure and the usefulness of the workloads we selected by reporting sample results from hundreds of side-by- side comparisons on popular DBMSs and DBaaS offerings.

See http://oltpbenchmark.com for the source code of our benchmark.

 

 


2012: Lookup Tables: Fine-Grained Partitioning for Distributed Databases

The standard way to get linear scaling in a distributed OLTP DBMS is to horizontally partition data across several nodes. Ideally, this partitioning will result in each query being executed at just one node, to avoid the overheads of distributed transactions and allow nodes to be added without increasing the amount of required coordination. For some applications, simple strategies, such as hashing on primary key, provide this property. Unfortunately, for many applications, including social networking and order-fulfillment, many-to-many relationships cause simple strategies to result in a large fraction of distributed queries. Instead, what is needed is a fine-grained partitioning, where related individual tuples (e.g., cliques of friends) are co-located together in the same partition. Maintaining such a fine-grained partitioning requires the database to store a large amount of metadata about which partition each tuple resides in. We call such metadata a lookup table, and present the design of a data distribution layer that efficiently stores these tables and maintains them in the presence of inserts, deletes, and updates. We show that such tables can provide scalability for several difficult to partition database workloads, including Wikipedia, Twitter, and TPC-E. Our implementation provides 40% to 300% better performance on these workloads than either simple range or hash partitioning and shows greater potential for further scale-out.

 

@inproceedings{tatarowicz2012lookup,   author    = {Aubrey Tatarowicz and                Carlo Curino and                Evan P. C. Jones and                Sam Madden},   title     = {Lookup Tables: Fine-Grained Partitioning for Distributed                Databases},   booktitle = {ICDE},   year      = {2012},   pages     = {102-113},   ee        = {http://doi.ieeecomputersociety.org/10.1109/ICDE.2012.26}, }

2012: How Clean Is Your Sandbox? – Towards a Unified Theoretical Framework for Incremental Bidirecti

 

Abstract. Bidirectional transformations (bx) constitute an emerging mechanism for maintaining the consistency of interdependent sources of information in soft- ware systems. Researchers from many different communities have recently in- vestigated the use of bxto solve a large variety of problems, including relational view update, schema evolution, data exchange, database migration, and model co-evolution, just to name a few. Each community leveraged and extended dif- ferent theoretical frameworks and tailored their use for specific sub-problems. Unfortunately, the question of how these approaches actually relate to and differ from each other remains unanswered. This question should be addressed to re- duce replicated efforts among and even within communities, enabling more effec- tive collaboration and fostering cross-fertilization. To effectively move forward, a systematization of these many theories and systems is now required. This paper constitutes a first, humble yet concrete step towards a unified theoretical frame- work for a tractable and relevant subset of bx approaches and tools. It identifies, characterizes, and compares tools that allow the incremental definition of bidi- rectional mappings between software artifacts. Identifying similarities between such tools yields the possibility of developing practical tools with wide-ranging applicability; identifying differences allows for potential new research directions, applying the strengths of one tool to another whose strengths lie elsewhere.

@inproceedings{terwilliger2012howclean,   author    = {James F. Terwilliger and                Anthony Cleve and                Carlo Curino},   title     = {How Clean Is Your Sandbox? - Towards a Unified Theoretical                Framework for Incremental Bidirectional Transformations},   booktitle = {ICMT},   year      = {2012},   pages     = {1-23},   ee        = {http://dx.doi.org/10.1007/978-3-642-30476-7_1},
}

2012: Mobius: unified messaging and data serving for mobile apps

Mobile application development is challenging for several reasons: intermittent and limited network connectivity, tight power constraints, server-side scalability concerns, and a number of fault-tolerance issues. Developers handcraft complex solutions that include client-side caching, conflict resolution, disconnection tolerance, and backend database sharding. To simplify mobile app development, we present Mobius, a system that addresses the messaging and data management challenges of mobile application development. Mobius introduces MUD (Messaging Unified with Data). MUD presents the programming abstraction of a logical table of data that spans devices and clouds. Applications using Mobius can asynchronously read from/write to MUD tables, and also receive notifications when tables change via continuous queries on the tables. The system combines dynamic client-side caching (with intelligent policies chosen on the server-side, based on usage patterns across multiple applications), notification services, flexible query processing, and a scalable and highly available cloud storage system. We present an initial prototype to demonstrate the feasibility of our design. Even in our initial prototype, remote read and write latency overhead is less than 52% when compared to a hand-tuned solution. Our dynamic caching reduces the number of messages by a factor of 4 to 8.5 when compared to fixed strategies, thus reducing latency, bandwidth, power, and server load costs, while also reducing data staleness.

 

@inproceedings{chun2012mobius,   author    = {Byung-Gon Chun and                Carlo Curino and                Russell Sears and                Alexander Shraer and                Samuel Madden and                Raghu Ramakrishnan},   title     = {Mobius: unified messaging and data serving for mobile apps},   booktitle = {MobiSys},   year      = {2012},   pages     = {141-154},   ee        = {http://doi.acm.org/10.1145/2307636.2307650},   }