Datamung won Netflix Cloud Prize award for best datastore integration

On Nov 14th, in AWS re:invent conference day 2 keynote, Netflix announced 2013 Cloud Prize award winners. I was very lucky and my project, datamung, is one of the ten winners. Datamung received the award for best datastore integration.

Datamung is a Java open source web application that backs up RDS MySQL database into S3 using AWS Simple Workflow and EC2. It's a RESTful service and website on top of it, a single installation allows multiple AWS users to backup their database living in their own AWS accounts. The fact that it uses mysqldump command indicates that the backup result in S3 is a SQL file ready to use across AWS accounts, regions, VPC, or outside of AWS network. If you are interested please jump into the Datamung project wiki to find out more.

Ironically, in the same keynote session where winner list was announced, 40 minutes later Werner Vogels announced a cross-region RDS replication feature which targets the same problem that Datamung attempts to solve.

Netflix flew my family to Las Vegas to attend 2013 AWS re:invent conference and accept award. I was horned to meet the judges, some of which are basically my career changers. For example Martin Fowler, when I talk to people about things he wrote, I got job offers so I could pay mortgage and send daughter to school. I wouldn't restart on open source development after 3 quiet years without Adrian Cockcroft launching the Cloud Prize program, not mentioning Netflix system had me watch hundreds of movies via internet. Anyway, I had good time in sin city and so did my wife and daughter.

Lastly, Chinese food in Vegas is surprisingly good as it turns out. I even found an authentic, in fact the only authentic Xi'an restaurant after 10 years of searching in United States. Having grown up in Xi'an myself, I was pleased to taste childhood food, and suffer from stomach pain of overeating once again.

Comments

Spring, Angular and other reasons I like and hate Bazel at the same time

For several weeks I've been trying to put together an Angular application served Java Spring MVC web server in Bazel. I've seen the Java, Angular combination works well in Google, and given the popularity of Java, I want get it to work with open source. How hard can it be to run arguably the best JS framework on a server in probably the most popular server-side language with the mono-repo of planet-scale ? The rest of this post walks through the headaches and nightmares I had to get things to work but if you are just here to look for a working example, github/jiaqi/angular-on-java is all you need. https://github.com/jiaqi/angular-on-java Java web application with Appengine rule Surprisingly there isn't an official way of building Java web application in Bazel, the closest thing is the Appengine rule and Spring MVC seems to work well with it. 3 Java classes, a JSP and an appengine.xml was all I need. At this point, the server starts well but I got "No ...

Customize IdGenerator in JPA, gap between Hibernate and JPA annotations

JPA annotation is like a subset of Hibernate annotation, this means people will find something available in Hibernate missing in JPA. One of the important missing features in JPA is customized ID generator. JPA doesn't provide an approach for developer to plug in their own IdGenerator. For example, if you want the primary key of a table to be BigInteger coming from sequence, JPA will be out of solution. Assume you don't mind the mixture of Hibernate and JPA Annotation and your JPA provider is Hibernate, which is mostly the case, a solution before JPA starts introducing new Annotation is, to replace JPA @SequenceGenerator with Hibernate @GenericGenerator. Now, let the code talk. /** * Ordinary JPA sequence. * If the Long is changed into BigInteger, * there will be runtime error complaining about the type of primary key */ @Id @Column(name = "id", precision = 12) @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "XyzIdGenerator") @SequenceGe...

Project Euler problem 220 - Heighway Dragon

This document goes through a Java solution for Project Euler problem 220 . If you want to achieve the pleasure of solving the unfamiliarity and you don't have a solution yet, PLEASE STOP READING UNTIL YOU FIND A SOLUTION. Problem 220 is to tell the coordinate after a given large number of steps in a Dragon Curve . The first thing came to my mind, is to DFS traverse a 50 level tree by 10^12 steps, during which it keeps track of a direction and a coordinate. Roughly estimate, this solution takes a 50 level recursion, which isn't horrible, and 10^12 switch/case calls. Written by a lazy and irresponsible Java engineer, this solution vaguely looks like: Traveler traveler = new Traveler(new Coordinate(0, 0), Direction.UP); void main() { try { traverse("Fa", 0); } catch (TerminationSignal signal) { print signal; } } void traverse(String plan, int level) { foreach(char c:plan) { switch(c) { case 'F': traveler.stepForward(); break; ca...

Jiaqi's Blog

Search This Blog