Friday, January 23, 2015

Private Maven repository in Amazon S3

To put together a private Maven repository, Amazon S3 might be the only solution that requires no installation of software. Comparing to hosting a nexus, S3 backend solution is incredibly cheap. In fact most cases don't exceed the free tier. Besides it comes with all benefits of S3 as storage: 11-9s durability, highly available, IAM based authentication, easy to integrate with CDN, etc.

Unfortunately Maven doesn't come with a native S3 wagon. This blog talks about how to setup private Maven repository on S3 using CyclopsGroup open source utilities.

Define repositories in base pom

This following example defines a SNAPSHOT repository for a project to pull dependencies from, in bucket mycompany-bucket-name with prefix /maven/snapshot-repository. A release repository can be setup in a similar way.

    <repositories>
        <repository>
            <id>mycompany.repository</id>
            <name>Snapshot repository of my organization</name>
            <releases>
                <enabled>false</enabled>
                <checksumPolicy>warn</checksumPolicy>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <checksumPolicy>warn</checksumPolicy>
            </snapshots>
            <url>s3://${dist.bucketName}/maven/snapshot-repository</url>
        </repository>
    </repositories>
    <build>
        <extensions>
            <extension>
                <groupId>org.cyclopsgroup</groupId>
                <artifactId>awss3-maven-wagon</artifactId>
                <version>1.4.1</version>
            </extension>
        </extensions>
    </build>
    ...
    <properties>
        <dist.bucketName>mycompany-bucket-name</dist.bucketName>
    </properties> 
To provide AWS credentials for Maven to access S3, add corresponding server definition in developers local settings.xml.

<settings>
...
    <server>
        <id>mycompany.repository</id>
        <username>AWS_ACCESS_KEY_ID</username>
        <password>AWS_SECRET_KEY</password>
    </server>

Define distribution management to upload artifact


Assume engineers are allowed to upload their local-built artifacts to SNAPSHOT repository defined earlier. If we decide to use the same AWS credentials for both download and upload of artifacts, a similar repository definition should be added to distribution management section of base pom.

    <distributionManagement>
        <snapshotRepository>
            <id>mycompany.repository</id>
            <name>My company's private snapshot repository</name>
            <url>s3://${dist.bucketName}/maven/snapshot-repository</url>
        </snapshotRepository>
    </distributionManagement>
Since the repository ID is the same as the repository in repositories section, the same credential applies.

Note that S3 access is implemented by awss3-maven-wagon. In very near future, the next version of awss3-maven-wagon will support username with reserved keyword "INSTANCE_PROFILE", which tells the wagon to get AWS credentials form instance profile on an EC2 instance.  This feature can be used in the case where we disallow engineers from uploading locally built artifacts and only want a few builder EC2 instances to write to repository.

Site distribution


Maven site distribution can be easily supported using similar approach

    <distributionManagement>
        <site>
            <id>mycompany.repository</id>
            <url>
                s3://${dist.bucketName}/sites/myproject
            </url>
        </site>
    </distributionManagement>
Here the site will be uploaded to S3 bucket, which may not be accessible by browser. S3 allows to define access control rule based on object path prefix so the S3 sites can be easily opened up while artifacts are still kept secret.

Best practices

I'm sure there can be many ways of using awss3-maven-wagon. I can't say the ways I use or suggest are the best practices, but during years I found them important, helpful and convenient across many projects.

Externalize bucket name to a property

In all examples above, the bucket name is not directly written in each section of pom, but defined via a property. This is not only to keep value consistent across several places, more importantly, it allows engineer to temporary change bucket name locally without having to release new version of base pom. Engineer can do so by adding property to settings.xml and overwrite the value in base pom.

Use different locations for release and snapshot repositories

This is not merely to keep files clean and pretty. When there's a big hierarchy of artifacts, it happens often where people want to make sure an artifact depends on only released dependencies at certain time.

Disallow engineer from writing to repository

It sure is convenient if engineer can build an artifact and upload it to repository immediately. However, it's only a matter of time before someone builds something from his special local environment, or without fully checking in his change, or checking out latest code from server.  The best way to deal with it is to have one(or more) standalone build server, a Jenkins server perhaps, with sanitized environment, and only give that server write access to repository. Engineer only gets read-only IAM for development.

Publish base pom to Maven central repository

I'm not sure if everyone will agree with me but I find it very useful. In the end of day, I want engineer to be able to checkout a project and be able to build it immediately as long as IAM is in settings.xml. When base pom is publicly available, user experience become very smooth. What people often do instead is to add repository to settings.xml instead of pom.xml and avoid publishing private artifact to public repository. This achieve the same result except when engineer works for more than one private organization and have to switch repository based on project he works on. When repository is defined in base pom, switching repository automatically happens. If not, engineer needs to modify settings.xml frequently as he moves between projects.

I hope this article somewhat helps your work. If you have doubt, please share your comments below. Thank you.

Monday, October 20, 2014

Another weekend project

It takes quite a lot of energy to finish watching the entire five seasons of Breaking Bad. To finish it strong, this weekend I decided to try something that I haven't done for almost 18 years.




Monday, September 01, 2014

Gitcon, runtime configuration via a Git repo

This weekend, aside from taking kid to football game and various classes, changing diapers and watching "Breaking bad", I've been working on an new small open source library, Gitcon, which allows Java application to decouple some configuration into a standalone source repository so that software owner can modify configuration without deploying application for every modification. It is a cheap approach to dynamic configuration.

By the way, I was very surprise to find out that the term "Gitcon" has not been reserved for "Git conference" so far.

Requirements


What led me to this project was a set of requirements:
  1. As a start, the configuration means properties file
  2. As a start, only Git source repository needs to be supported
  3. As a start, it only needs to gets configuration for once when application starts. The configuration is used to populate Spring application context where beans are mostly singletons. Software owner needs to restart application to pick up configuration change.
  4. The library must be very friendly to Spring
  5. It gets file from Git repo authenticated via SSH key.
  6. The SSH private key must be configurable
  7.  This is outside the scope of the library, but the SSH key comes from S3, which is authenticated by instance profile
  8. When SSH key is not specified, default one in user's home directory is applied. This is for development purpose.
  9. One Spring context may includes multiple properties coming from multiple repositories authenticated with different SSH keys.
  10. One properties file can include another under the same repository
 These requirements are the minimal goals that the first version of Gitcon must achieve in order to replace configurations of some existing applications.

Implementation


The design is fairly straightforward. All it needs is to cope with the limitations from Jsch that JGit relies on.


1. S3FileFactoryBean


S3FileFactoryBean is something that takes an IAM, a bucket and a key, downloads a file from S3 into local temporary directory, returns the file and deletes it when Spring context closes. Since the logic is generic to the purpose of Gitcon, it lives in a neutral package, kaufman-aws as artifact org.cyclopsgroup:kaufan-aws:0.0.2.

<bean id="gitSshKeyFile" class="org.cyclopsgroup.kaufman.aws.S3FileFactoryBean">
    <constructor-arg value="my-s3-bucket" />
    <constructor-arg value="my-app/git-ssh-private.key" />
</bean>

2. Clone, checkout Git repo with JGit

The easiest way to clone a Git repo is to use JGitLocalResourceRepository. With support from Eclipse JGit, this class defines Git repository URI, the access to Git repository and optionally the path to SSH private key. You may find out more details in Javadoc. An example:

<bean id="gitResourceRepo"
    class="org.cyclopsgroup.gitcon.spring.JGitLocalResourceRepository">
    <constructor-arg
        value="git@bitbucket.org:me/my-runtime-config.git" />
    <property name="sshIdentityFile" ref="gitSshKeyFile" />
</bean>

3. Create properties


The class GitconPropertiesFactoryBean simply creates Properties instance out of given ResourceRepository and a path to properties file. Since it creates Properties with ExtendedProperties, variable replacement and file inclusion is supported.

<bean id="appProperties"
    class="org.cyclopsgroup.gitcon.spring.GitconPropertiesBeanFactory">
    <constructor-arg ref="gitResourceRepo" />
    <constructor-arg value="myapp/myapp-prod.properties" />
</bean>

If you haven't tried, ExtendedProperties interprets properties file in a different way than Properties and exposes the content with a much more user friendly API. It is an ancient class from commons-collections, if a human is as old as this class, he would probably get a driver license by now.


4. Expand properties in Spring


This is done by a simple setup in Spring. It doesn't belong to Gitcon but I put it here so that the story is complete. The following config does the work.

<context:property-placeholder properties-ref="appProperties" />


To learn more


Like other CyclopsGroup projects, Gitcon Maven site is published to dist.cyclopsgroup.org/projects/gitcon. Very soon I will come up with a few wiki pages with official guides and examples.

Javadoc can be found here. Source code is in Github. The first version, org.cyclopsgroup:gitcon:0.0.1 has been published into Maven central repository.

If you have question, please feel free to contact me in Github, Facebook or Google+. Thank you for reading.

Sunday, March 09, 2014

The longest week

A week ago I came out from a surgery room in Memorial hospital in South Bend and had one of my most painful moments in life. In the following week, pain pill and IV were my friends. Now after a week of recovery, I finally got back to my sanity and sit in front of my computer.

Feb 25th noon, I fell down on an icy drive way on my way to the car. It wasn't a light fall. When it happened I heard a sharp pop and saw my right foot bent up. This is something I have only seen on YouTube. Fortunately my wife was with me, it could be a whole lot worse without her. At the very second I fall on the ground many things passed my mind. My pregnant wife, my daughter whom I read story to, my travel plan in next month, my job in Chicago, and the fact that my son will grow up with a cripple daddy who probably can't hold him. Then came the pain, which confirmed what happened is real.

For the second time in life, an ambulance took me to hospital. A nurse looked at CT scan in hospital and said, "how could you fall to break leg like this". It is a good question, I ask myself this question all the time.


The bed in hospital is a really good bed. Softness is controlled electrically, and buttons are everywhere to adjust position in all imaginable ways. If I were 10 years younger I would be curious, playing with the bed and discovering all hidden features. But in reality, all I worried is how does mommy explain to Yilin daddy's absence.

Finally Yilin came and visited me at night. My girl was a little scared. She saw my leg and covered eyes with hands. However it didn't take long before she warmed up, climbed up and down the room and made funny face to nurses. Before leaving she asked me, "Daddy why can you still talk?", then she skipped away.

The surgery occurred on the next day. Tibia was broken into 3 pieces and Fibula into 2. The orthopedic doctor decided to insert a titanium rod into the broken bone and fix it with nails on both ends. I didn't see how it was done on surgery table, but it sounded to me that the surgery is a walk in the park for the doctor. It went well and then it was just the matter of long, painful recovery process -- swelling, bone recovery, and celebration party.

There's so much that I wanted to do,  articles to read, work to catch up, movies to watch. I'm always lack of time but in hospital, finally I had enough time, enough resource but I couldn't do any of them. Trapped in a narrow space hour after hour, with pain and insomnia day after day, my mindset changed into a state where brain is frozen and eyes are wide open. Not being able to do anything meaningful for long time becomes the biggest torture for me, which is way much worse than the pain itself.


However don't worry my friends. In the beginning of this blog I mentioned I got back to sanity finally. Right now, things are getting better everyday. I moved from hospital to a sofa in home, then started walking with a pair a crutches once a while. The bruise looks different every morning. I had hard time sleeping, but day time becomes enjoyable. Now I'm sitting in a wheelchair, which Yilin is super excited about.

As I started reading news I realized I just missed a very turbulent week. The Ukraine crisis, terrorism in Kunming and missing Malaysia airplaine, all happened in the same week. Lives are lost and many need help much more than I do. I hope they get better every day as well.

Sunday, November 17, 2013

Datamung won Netflix Cloud Prize award for best datastore integration


On Nov 14th, in AWS re:invent conference day 2 keynote, Netflix announced 2013 Cloud Prize award winners. I was very lucky and my project, datamung, is one of the ten winners. Datamung received the award for best datastore integration.


Datamung is a Java open source web application that backs up RDS MySQL database into S3 using AWS Simple Workflow and EC2. It's a RESTful service and website on top of it, a single installation allows multiple AWS users to backup their database living in their own AWS accounts. The fact that it uses mysqldump command indicates that the backup result in S3 is a SQL file ready to use across AWS accounts, regions, VPC, or outside of AWS network. If you are interested please jump into the Datamung project wiki to find out more.



Ironically, in the same keynote session where winner list was announced, 40 minutes later Werner Vogels announced a cross-region RDS replication feature which targets the same problem that Datamung attempts to solve.


Netflix flew my family to Las Vegas to attend 2013 AWS re:invent conference and accept award. I was horned to meet the judges, some of which are basically my career changers. For example Martin Fowler, when I talk to people about things he wrote, I got job offers so I could pay mortgage and send daughter to school. I wouldn't restart on open source development after 3 quiet years without Adrian Cockcroft launching the Cloud Prize program, not mentioning Netflix system had me watch hundreds of movies via internet. Anyway, I had good time in sin city and so did my wife and daughter.



Lastly, Chinese food in Vegas is surprisingly good as it turns out. I even found an authentic, in fact the only authentic Xi'an restaurant after 10 years of searching in United States. Having grown up in Xi'an myself, I was pleased to taste childhood food, and suffer from stomach pain of overeating once again.