Facebook, Google, LinkedIn, Twitter Collaborate On Open-Source Database Project

Facebook, Google, LinkedIn and Twitter engineering teams have collaborated on an open-source project called WebScaleSQL, a custom open-source project of the MySQL database management system designed for large companies. I'm highlighting the word "collaboration" because the partnership involves several large Web companies that historically have been viewed as rivals.

Stephan Greene, software engineer at Facebook, announced the WebScaleSQL project in a blog post Thursday. The project supports companies that face similar challenges in running large MySQL databases, and want better performance from a database technology tailored to their needs.

"We aim to create a more integrated system of knowledge-sharing to help companies leverage the great features already found in MySQL 5.6, while building and adding more features that are specific to deployments in large scale environments," he wrote." In the last few months, engineers from all four companies have contributed code and provided feedback to each other to develop a new, more unified, and more collaborative branch of MySQL."

Facebook uses MySQL as the database platform, part of the back-end infrastructure helping to bring Facebook to users daily. The project will help companies improve the performance of large-scale databases running MySQL. 

For those who are into the geeky details, Greene explains how the group set up a system to collaborate, review code, and report bugs. The Facebook team is working on contributing an asynchronous MySQL client, which he explains means that while querying MySQL, users don’t need to wait to connect, send, or retrieve information. Facebook's team also is testing versions of table, user, and compression statistics into WebScaleSQL, as well as preparing to push the remaining components of Facebook's current production-tested version of compression that were not already included in MySQL 5.6 into WebScaleSQL.

Some of the developments involve automating the framework that runs and publishes the results of MySQL's built-in test system, a suite of stress tests and prototype automated performance testing system. The group also created a buffer pool, support for NUMA interleave policy, and new features that make operating WebScaleSQL easier.

"People in a circle holding hands" photo from Shutterstock.

Recommend (3) Print RSS
1 comment about "Facebook, Google, LinkedIn, Twitter Collaborate On Open-Source Database Project".
  1. Clint Dixon from Sem Advance , March 27, 2014 at 5:27 p.m.
    One of the great things about data is they give businesses a way to mine their data and from that, learn new ways to be more productive. One of the best websites to find products to clean the data before mining is http://www.dataladder.com