Skip to main content

Datamung won Netflix Cloud Prize award for best datastore integration


On Nov 14th, in AWS re:invent conference day 2 keynote, Netflix announced 2013 Cloud Prize award winners. I was very lucky and my project, datamung, is one of the ten winners. Datamung received the award for best datastore integration.


Datamung is a Java open source web application that backs up RDS MySQL database into S3 using AWS Simple Workflow and EC2. It's a RESTful service and website on top of it, a single installation allows multiple AWS users to backup their database living in their own AWS accounts. The fact that it uses mysqldump command indicates that the backup result in S3 is a SQL file ready to use across AWS accounts, regions, VPC, or outside of AWS network. If you are interested please jump into the Datamung project wiki to find out more.



Ironically, in the same keynote session where winner list was announced, 40 minutes later Werner Vogels announced a cross-region RDS replication feature which targets the same problem that Datamung attempts to solve.


Netflix flew my family to Las Vegas to attend 2013 AWS re:invent conference and accept award. I was horned to meet the judges, some of which are basically my career changers. For example Martin Fowler, when I talk to people about things he wrote, I got job offers so I could pay mortgage and send daughter to school. I wouldn't restart on open source development after 3 quiet years without Adrian Cockcroft launching the Cloud Prize program, not mentioning Netflix system had me watch hundreds of movies via internet. Anyway, I had good time in sin city and so did my wife and daughter.



Lastly, Chinese food in Vegas is surprisingly good as it turns out. I even found an authentic, in fact the only authentic Xi'an restaurant after 10 years of searching in United States. Having grown up in Xi'an myself, I was pleased to taste childhood food, and suffer from stomach pain of overeating once again.


Labels:

Comments

Post a Comment

Popular posts from this blog

Publish Maven site with Amazon S3 and CloudFront

Amazon S3 now supports static website hosting . As a 10 years Maven user, I wonder how easy it is to deploy Maven generated site to Amazon S3 and let the rock-solid storage provider to host my project websites. There are several existing s3 wagon providers , which all seem to have the same problem, not supporting directory copy. This is understandable since before S3 new website hosting feature, I guess people mostly expect to deploy artifacts rather than website to S3. So my first task is to write an AWS S3 wagon that supports directory copy. With AWS Java SDK , task becomes as simple as one single class . I made my S3 wagon available in Maven central repository at org.cyclopsgroup:awss3-maven-wagon:0.1 . The source code is hosted in github:jiaqi/cym2/awss3 . The next thing is to create an S3 bucket in console . To avoid trouble, bucket name is set to the future website domain name according to this discussion . Website feature needs to be explicitly enabled. I also created an...
5 comments
Read more

Customize IdGenerator in JPA, gap between Hibernate and JPA annotations

JPA annotation is like a subset of Hibernate annotation, this means people will find something available in Hibernate missing in JPA. One of the important missing features in JPA is customized ID generator. JPA doesn't provide an approach for developer to plug in their own IdGenerator. For example, if you want the primary key of a table to be BigInteger coming from sequence, JPA will be out of solution. Assume you don't mind the mixture of Hibernate and JPA Annotation and your JPA provider is Hibernate, which is mostly the case, a solution before JPA starts introducing new Annotation is, to replace JPA @SequenceGenerator with Hibernate @GenericGenerator. Now, let the code talk. /** * Ordinary JPA sequence. * If the Long is changed into BigInteger, * there will be runtime error complaining about the type of primary key */ @Id @Column(name = "id", precision = 12) @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "XyzIdGenerator") @SequenceGe...
4 comments
Read more

1300ms to 160ms, tune Spring/Hibernate on slow MySQL

I write this article to remember the different behaviour various JDBC connection pool displays when they work with slow JDBC connection(to MySQL database, in this case). It starts with a typical Java application on Spring, Hibernate, Jetty, ApacheCXF and MySQL like following code. Version 1: without correct pooling //... service code @Transactional(isolation=Isolation.READ_COMMITTED) public void foo() { //... do something with database } //... connection pool configuration ... class = "com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource"; url = "jdbc:mysql://mysql.far-far-away.com/mysystem"; user = ... //... transaction management configuration in spring ... <tx:annotation-driven transaction-manager="transactionManager" order="100" /> <bean id="transactionManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager"> <property name="sessionFactory" ref="mySessionFact...
2 comments
Read more