Thursday, September 26, 2013

Forking MySQL/ for how long can forks keep up?

  • Fact: MySQL 5.6 was released as GA in February 2013
  • Fact: MySQL 5.6 has been available with its complete set of features since September 2012
  • Fact: On September 21st, Oracle has released MySQL 5.7.2, which is the de facto major release after MySQL 5.6 (5.7.1 was just a token “we’re-still-in-business” release).
  • Fact: As of today, there is no GA-ready fork of MySQL 5.6.

Percona Server is still in RC state, while MariaDB, with its runaway version 10, is still in alpha state. Of these releases, Percona Server seems the one in the better shape. Their problem was to adapt Percona Server to the enhanced codebase for 5.6, and the merging problems were bigger than the ones encountered in 5.1 and 5.5. Percona Server is a business oriented fork. It includes specific features for performance and manageability, things that users are asking for, and things that make Percona professional services easier to carry out.

Much different is the case of MariaDB. Their feature set is wider and hard to categorize globally. It includes some features that enhance performance (block commit, subquery optimization) but also plenty of features that are included only because nobody else wanted to touch them. In my experience, adding features to a project does not make it more stable, as the MySQL team knows well, with its disastrous experience with version 6.0, which was aborted after two years of alpha. MariaDB is in a state where they are adding lots of stuff to their code base, while merging selected pieces from MySQL 5.6. It could be a recipe for success or disaster. User experience will tell.

Up to MySQL 5.5, MariaDB has kept itself completely compatible with MySQL, with the goal of being a drop-in replacement. With the capriciously numbered version 10, that promise of compatibility has been broken. Most of the features introduced in MySQL 5.6 have not been included in MariaDB 10. Some of them have been rewritten with a completely different approach. As a result, MariaDb 10 lacks some of MySQL 5.6 features (parallel replication, for example) while showing some that are not in the upstream project (multi-source replication). The design of GTID has been started from scratch, and surely will make interoperability really hard (if not downright impossible) between the two projects.

Looking at the status of the projects, it seems that the MySQL team at Oracle has produced so many changes, that keeping up with it, even in terms of simple patch merging, is a heavy task. Where does it leave us, the community users? We have the choice between using the latest and greatest from Oracle, or waiting for the next latest and greatest from one of the forks. Percona Server seems to be on the final path of a less-than-easy catch-up, but MariaDB is farther away. And even if it releases a GA soon, it will find itself in a situation where merging changes from Oracle is going to be tougher and tougher. It may mean that performance enhancements from mainstream MySQL will take months (or years) to reach MariaDB. Users willing to benefit from the latest improvements will choose between Oracle MySQL and (with some delay) Percona Server. MariaDB will need to either do some sort of cherry-picking from Oracle releases or implement its own improvements. Can they compete with the 200 strong engineering team at Oracle? It’s going to be a tough proposition.

Given the above facts, how is it that we hear of big players like Red Hat and Google announcing that they will adopt MariaDB instead of Oracle’s MySQL? Since I could not find a sensible technical reason, I am forced to conclude that the choice has been political. Both Red Hat and Google have been entangled in commercial competition, and sometimes in court. Yet, choosing software for non-technical reason looks foolish to me.

Summing up: Oracle is releasing a steady stream of improvement for MySQL. This shows a high level of commitment that does not play well with what the doomsayers have been announcing for the past 4 years. So far, facts are contradicting that FUD. Forks are lagging behind. Some more than others.

Disclaimer: The above thoughts are my own, and don’t reflect the opinions of my employer or other companies.

34 comments:

Anonymous said...

Hi,

I don't think 5.6 was ready for GA, and a number of bugs in 5.6 (like the I_S.partitions problem, among others) were fixed in Percona Server before Oracle MySQL.

I don't think Percona should have to support a "GA" release that isn't GA. Keep in mind I'm part of the support department, but this is MY OPINION and may not reflect Percona's.

Stewart Smith said...

One of the interesting things with Percona Server 5.6 is we've spent a *LOT* of time on QA - much more than any previous Percona Server release.

Valerii Kravchuk said...

Real improvements that cause no doubts in MySQL 5.6 and 5.7 come happens in InnoDB (performance and scalability) and PERFORMANCE_SCHEMA (more instrumentation with a hope for less overhead). In other areas, even replication, there are too many questions and problems to solve, and, surely, bugs to fix. It seems Oracle concentrate on new features notably more than on QA and this requires a lot of work from any fork that tries to improve quality...

wlad said...
This comment has been removed by the author.
Mark Callaghan said...

I think that MySQL 5.6 is still early in its life cycle. Given that and the huge number of changes from 5.5 I think it is in very good condition.

I can't judge the forks because I don't have enough experience with them.

I have a question for Stewart. Given that you have spent a lot of time on QA for 5.6, do you have a list of the bugs/problems that you found?

Laurynas said...

Mark -

https://bugs.launchpad.net/percona-server/+bugs?field.tag=qa is a raw list of our QA-found bugs. This includes both 5.5 and 5.6, and I don't see how to filter it for 5.6 only.

We haven't attempted to create a specific list of Oracle MySQL bugs found by QA. An approximation would be bugs tagged with both "qa" and "upstream", or both "qa" and "innodb" on our Launchpad.

But we have a list of MySQL 5.6 bugs fixed in Percona Server 5.6 at http://www.percona.com/doc/percona-server/5.6/upstream-bug-fixes.html. At the moment it does not take into the account the last .14 point release nor the Percona Server fixes since the last RC.

Anonymous said...

Hi,

MariaDB 10 rock and is moving fast, i can report multiple usage of the Alpha release in production on Masters and Slaves without having a single issue for month. The Alpha state is because as you said features are still changing and be backported. Moving from 10.0.3 to 10.0.4 was a pain especially using new feature like muti source or spider.The MP team is a lot more conservative about quality, again the release note of 5.5 s just scary , 5.6 is not production ready in my opinion on the new features. I mean the one that are not closed: ) So the question is more how many release of MySQL can be produced in RC but having Apha quality without hurting the community and their users.

Anonymous said...

It is not very good that Percona using MySQL as upstream and not MariaDB , It's a huge waste of energy. Percona ability in Q&A end testing is something the MariaDB project would value a lot. And that can consolidate alternatives faster with the joined contribution of SkySQL an Percona and others .

It's the MySQL owner that see Percona and SkySQL as competitor for the product. From MySQL we do not participate in any decisions that is taken regarding MySQL. It's possible Percona and SkySQL are competitor on a sale view, technically we can do collaborations as demonstrate with features that get exchanged in and out between the two forks. I really think that we can be more clever and can come with more collaboration on what we have lost , bug database, Q&A. SQL standard. I was very upset that features done by us are copied in MySQL and rename in Syntax just to make things a little bit more complicate for us. All AS directors are playing their own game here . But this is really against the community and the futur.

Giuseppe Maxia said...

@Stephane,

From the release notes
MariaDB 10.0.4 is an Alpha release. This is the fifth 10.0-based release, and we are releasing it now to get it into the hands of any who might want to test it. Not all features planned for the MariaDB 10.0 series are included in this release. Additional features will be pushed in future releases. Do not use alpha releases on production systems.


Using this release in production seems hazardous at best.

Anonymous said...

Very well written! I suggest Percona wherever I work, but it's always a battle making the case over MariaDB which gets more mainstream press.

Mark Callaghan said...

Steaphane,
Are we moving to a world where everyone craps on the releases of others? Too much marketing and not enough tech for me. Was the work of my team the best because it was used by Google, Facebook and Wikimedia? And lots of Wikimedia still runs on the FB patch!

What is the basis for saying 5.6 isn't production ready? Did you do test deployments or is this more an opinion not based on experience? Note that some people on the MariaDB side have made very negative statements about the quality of various 5.X releases in the past and from my experience in using those releases the predictions were extremely wrong.

With respect to Percona and MariaDB efforts sharing one code base I think that would be great. But would Percona be sharing with MariaDB or SkySQL/MontyProgram? The difference between SkySQL/MontyProgram and MariaDB isn't clear to me today.

Mark Callaghan said...

Laurynas - this list is nice. Hopefully Oracle thanks you for helping to make the product better. If not, I thank Percona on their behalf - http://www.percona.com/doc/percona-server/5.6/upstream-bug-fixes.html

Mark Grennan said...

I agree with you assessment of the state of Oracle, Percona and MeriaDB and that choosing an application based on politics is a bad idea. Required fetchers and stability are the only criteria that should be used for production / enterprise systems.

I'm running Percona (5.5) with an eye to Oracle 5.7.

Great post.

Anonymous said...

Mark,

Too much marketing and not enough tech for me as well.

>What is the basis for saying 5.6 isn't production ready?

I was the all time most used consultant at MySQL followed by M. Rubin@percona. In that quality i'm principal consultant for 5 financial institutions that used to have MySQL support when the GA issue started.Under support advice moved to MySQL 5.5 GA and get all floating point computation broken. The Answer form support has been a very long silence until we downgrade and fixed in MariaDB. I switch an other on 5.5.28 release in GA for very long time . because they used filter in replication they get all session variables set to null on some level 2 slaves . Superb when this is a payment transaction Id . Enouth for 5.5 and do agree that since few releases 5.5 looks to be stabilized. Good point MariaDB 5.5 is downstream of 5.5 with backport of 6.0. Now regarding 5.6. SkySQL have a vey experimented support team on MySQL and that's not marketing (Percona Tom team is just awesome as well), i think we have a good idea of the quality of the 5.6 new features . When we get 26 different issues in a single new feature on a GA product . This is just a miss conception of what a GA release his. And it's not like no warnings been raised before from MariaDB developers.

Sabotage mille sabords !

So MariaDB 10 is more mature than 5.6, the backport of the optimizer 6.0 features was done on MariaDB 5.1 to 5.3 forked from MySQL 5.1, having 5 years of community feadback on such features. Tobao multi source and spider are in production to big web shop as well . So the quality of the code can be declare trustable.

Now i thanks Percona but thanks more Jeremy and Arjen who have been first to leave MySQL for some openness issues regarding exactly your patches and licence that would let good contributions at the door of the MySQL home.

> SkySQL/MontyProgram and MariaDB isn't clear to me today.

MontyProgram is Monty and now working as a key member of the foundation, you have now only SkySQL and the MariaDB foundation, MontyProgram was Monty paying the best MySQL talend from his own money the team moved to SkySQL payrole. They are working full time for the MariaDB foundation just like Percona have developers working on Drizzle . Most old time developers are now away from MySQL owner, so YOU GET THE QUALITY OF FRESH MEAT. The all point is how long can fork keep up if it is based on product that does not include
work and architecture decision from the most experimented developers.

Mark Callaghan said...

Stephane - I am not questioning your skills, I am questioning whether you have sufficient experience with 5.6 to have an opinion that others should consider. With respect to the opinions that others in the MariaDB camp have proclaimed about MySQL, there have been a few cases where extremely negative opinions were expressed. Alas they were also extremely wrong. Yet they continue to get re-proclaimed with every major release. Credibility matters and those proclamations are costing people some of it.

MariaDB should be making a name for itself based on being good, not on trashing MySQL.

I will be more clear about the confusion between .com (SkySQL/MontyProgram) and .org (foundation/mariadb.org). What is the foundation doing? I contributed money. I doubt I will again because nothing is happening in the open. Most of the MariaDB content is on askmonty.org or Mariadb.com. That doesn't appear to be owned by the community. Apparently we aren't even worth of the MariaDB downloads. Those are private to SkySQL/MontyProgram.

I haven't been happy with the community side of MySQL, but I am also unhappy with what is going on on the MariaDB side.

Baron Schwartz said...

I believe people almost always choose technologies based on how it makes them feel, regardless of whether they want to admit that or not. In that regard, what Stephane says makes perfect sense to me, as do the comments from Stewart and others at Percona.

I admire the Oracle engineering and product management team, and although I understand some of the decisions they've taken with regards to the community, I don't understand some of the others of which I disapprove. Maybe if I were on the inside I'd understand and be more tolerant.

Kolbe Kegel said...

The problems I've seen with MySQL 5.6 GTID in production and in my own testing have sealed for me that MySQL 5.6 was not ready for GA when it was first given that label. There are still big problems with the GTID implementation, even in MySQL 5.6.14. It's possible that this will stabilize over time, but in my opinion this flagship feature was not ready, and still is not ready, for a GA label.

http://bugs.mysql.com/bug.php?id=69758
http://bugs.mysql.com/bug.php?id=68589
http://bugs.mysql.com/bug.php?id=70167

Kolbe

Anonymous said...

Mark,

I always try to be happy in life:)

Baron tx for those words;

Regarding InnoDB performance i must say that Oracle is doing the job. Compression , read only transaction , yes love it ! And GA before GA if at the end it bring stable feature, why not ? Is that more valuable than transportable tablespace, independent histogram statistics to PIN execution plan or past table index statistic , profilig and atomic writes on Fusion IO. MySQL and MariaDB are SQL RDBMS they must be much more than a single dam fast transactional storage engine ! Why PBXT that was faster and much more open did not make it? Yes it is political. Percona and SkySQL can promise new features for less price and better support. Why would this failed . Like most open source project slowly but surely. You have the ability and the money to pick both of them and force us to work together like in the past . Make this happen for the glory of open source and people that wan't there own destiny in their hands.

Anonymous said...

haproxy 1.5 is not GA but outch every one using it . MySQL proxy is not GA well it is used in MySQL MON. GA is not about marketing it's about perception of the past and usage.

Anonymous said...
This comment has been removed by the author.
Anonymous said...
This comment has been removed by the author.
Anonymous said...

Mark, Guieseppe , at the end Oracle alway's win look at the American cup :) but without chalenger the race is not fun

Jonathan Levin said...

I agree with Baron that people will use the technology that makes them feel good. In that respect, I am of the opinion that people who already use percona server will wait for Percona to release their version of 5.6.. even if it takes them longer than a year.

I also agree with Mark that I really do not think highly of how MariaDB is trashing MySQL and would prefer MariaDB to compete on performance/features/stability.

On the point of which I would prefer and recommend using, my answer is simple. A company I am working with used MySQL 5.5 - I tested Percona 5.5 and everything worked 10-20% faster.
I then tried Mariadb 5.5 + tokudb and MySQL 5.6 (early GA) on a test replicated slave and the lagg went to >30k seconds.
(Admittedly, some of our queries are quite bad)
I am now trying Percona 5.6 + tokudb on the same server, 5 days and no incidents.

That is how I decide on things to use in production..

Anonymous said...

Jonathan -- are sure you have synchonized your time or have not different server id: also done the same test as and my conclusion was the other way around.

MariaDB started 2 years ago integrated some feature to make it a true plugin and not patched release . And there is still some to task to achieve tout complete a full integration.

Ronald Bradford said...

Its amazing which posts generate the most community feedback. It's not about a *new* feature, great new approach or bug for example.

I would like to say that the title or intro text should include "How long do forks want to keep up AND REMAIN COMPATIBLE WITH MYSQL". This is the hidden thing you need to consider when dealing with and answer client inquiries.

I really feel for the community, and especially the storage engine providers, because they ALL loose out. Some of the comments in this thread also helps absolutely nobody in the MySQL ecosystem, so I wish people would stop the black and white bickering.

I use MySQL 5.6 in production with multiple clients. Many core features are extremely stable, and there are a number of immediate benefits, features and instrumentation. I do not for example use GTID (I consider this new feature, still needing work for a number of reasons), and for those at the MySQL Connect opening keynote I posed this exact question. I look forward to 5.7 improvements here.

I have been waiting for any fork to really catchup to MySQL 5.6 for one single reason. With MySQL 5.6 installed on a master, I am unable to effectively test and evaluate other forks (and more specifically storage engines) due to the inability of replication.

As a former MySQL consultant, and now full-time independent consultant in the MySQL ecosystem I see a lot more in the real world then the majority of commenters that work only for their respective companies. Everybody is entitled to their own world view of the state of play, but what is important overall? Perhaps you should all consider this.

It was great to see many friends, alumni and colleagues at MySQL Connect. It was so very sad to see the vibrant community that was MySQL, and that existed in the Golden days at the grass roots conferences all gone. All parties need to accept blame and responsibility. I only wish it was repairable.

Jonathan Levin said...

@stephane, yes, I did check for those things. The issue was a morning data load and processing which could have used better queries.
The replication when initially set was fine, but the next morning it lagged and did not catch up.

Anonymous said...

Jonathan ,

Intresting i observe the exact same issue . Fixed it by reloading the slave from dump, having many InnoDB slaves with the exact same MariaDB release, and we did not observe the lag, all queries have become slow on the TokuDB side. The fist MariaDB release with close integration of TokuDB is 5.5.33a.

The MariaDB team is working hard for the external storage engine to get the cleaner integration as possible but this is never finish job. On Galera Cluster the amount of work to make it pluggable one day is just huge. This is the level of hidden work expected, cross storage engine foreign key, who is working on this because integrating external storage engine like TokuDB is fine but letting them work in a friendly framework is a must do. So the spirit matter as well when choosing a fork.

Anonymous said...

Ronald Bradford

Sorry for non constructive remarks. Yes rephrase like you proposed look's a lot more better.

Now it's probably me that does not accept the short release cycle development model that can make stability of a single feature independent from the general release level.

I congrat the MySQL team for still improving it . 3 of the new features in the 5.7 are directly driven by MariaDB so it make me ask the question who is leading the race ? As i repeat MySQL branch as given a lot regarding InnoDB and performance. Yes i welcome those new release at the end .

rpbouman said...

Giuseppe,

interesting post.

Personally I think Oracle has proven to be a good steward for MySQL. From that perspective, the forks may seem superfluous.

However, with that in mind, one may miss the perspective that there was a very long period where this was not so obvious and where the forks of both Percona and MariaDB made sense on their own, because they had tangible merit over plain MySQL.

From there, it's just one step to pose the question:

Would Oracle have bothered at all if it weren't for those forks keeping them on their toes?

I'm not saying I know the answer, I'm just saying I rather like the situation as it is now, with a choice of forks and multiple teams competing for the best product.

Giuseppe Maxia said...

@Roland,
There has been an interesting loop for some forks. The first "juicy" fork (the one that many wanted but it wasn't achievable) was the Google patch, which was based on 4.0 when the MySQL team was busy trying to ship 5.1.
Some elements of that patch were implemented by Percona, and others found their way to MySQL 5.5, which was then adopted by Percona as basis for its server, and so on.
So, IMO, the answer is yes: for some forks there is a positive influence on the upstream team.

Nerd Progre said...

It's all public relations stunt by the forkers to try to steal mindshare from the orgiginal project, which continues to be GPL, and free as in freedom as well as free as in beer (community edition).

They even resort to blatant lies, all that matter for them is getting the point across that "Oracle is evil" and that "MySQL is not what it used to be" (less free, bugs database closed, whatever, it' doesn't matter if it's a lie, the objective is putting FEAR into the minds of MySQL users, and make it easier to be convinced of "switching" to one of the forks).

Case in point:
"The bugs database is not open anymore!"

http://tinyurl.com/muktware-liars
vs reality
http://tinyurl.com/lsvya6f

FC

Mark Callaghan said...

Is this a troll? You had a reasonable post until the end.

The bugs database is not open refers to bugs in support.mysql.com. That isn't open and most bugs in release notes reference it rather than bugs.mysql.com. So people without a support contract have a very hard time getting details on open bugs and bugs fixed in a release.

Giuseppe Maxia said...

@Nerd Progre
The bugs database issue is more than it shows in the surface. It s not a conspiracy, it's a real fact.
What the community is complaining about is the disappearance of most of the feedback on bugs submitted. The bugs database (bugs.mysql.com) is still available, but three things happen:

1) bugs submitted by non customers get little feedback;
2) many bugs filed by customers or Oracle employees are filed to an internal database, not visible outside. We know that a given issue was addressed by some mention on the release notes, but we don't get the details of it.
3) when you submit a bug, it may be flagged as a duplicate of a bug in the internal database (it happened to me). This means that you may take time and effort to file a bug that was not necessary, and you don't have a clue about the progress on such bug.

Summing up, the bugs database issue is still an issue. Some people in the community are keeping Oracle engaged on this topic, in hope that they will change their stand. But as of today, it is still an open wound for the community.

Justin Swanhart said...

Also, @nerd, "the forkers" is a broad generalization. You won't find FUD from Percona. We stepped in to fill a vacuum when O'Reilly stopped the old conference. While opinions vary on this, most people here would agree Percona Live has been good for the ecosystem. Percona Server is arguably more stable, and this is backed up with facts, as in, bugs reported and fixed.