Project Euler 359 - Hilbert's New Hotel

Problem 359, Hilbert's new Hotel

Had no idea where to start at all, the only thing I could was to write a small program to print out first hundred numbers in naive way.

[1, 3, 6, 10, 15, 21, 28, 36, 45, 55, 66, 78, 91]
[2, 7, 9, 16, 20, 29, 35, 46, 54, 67, 77, 92]
[4, 5, 11, 14, 22, 27, 37, 44, 56, 65, 79, 90]
[8, 17, 19, 30, 34, 47, 53, 68, 76, 93]
[12, 13, 23, 26, 38, 43, 57, 64, 80, 89]
[18, 31, 33, 48, 52, 69, 75, 94]
[24, 25, 39, 42, 58, 63, 81, 88]
[32, 49, 51, 70, 74, 95]
[40, 41, 59, 62, 82, 87]
[50, 71, 73, 96, 100]
[60, 61, 83, 86]
[72, 97, 99]
[84, 85]
[98]

As you may see, the result is very interesting. The numbers on the first column seems to be ( row + 1 ) / 2 * ( row / 2 ) * 2. Note that row/2*2 != row since all operations are integer operations.

If we calculate the delta values between consecutive numbers in each row, we got

 2, 3, 4, 5, 6, 7, 8, 9,10, 11, 12, 13, 

 5, 2, 7, 4, 9, 6,11, 8,13, 10, 15, 

 1, 6, 3, 8, 5,10, 7,12, 9, 14, 11, 

 9, 2,11, 4,13, 6,15, 8,17, 

 1,10, 3,12, 5,14, 7,16, 9, 

13, 2,15, 4,17, 6,19, 

 1,14, 3,16, 5,18, 7, 

17, 2,19, 4,21, 

 1,18, 3,20, 5, 

21, 2,23, 4, 

 1,22, 3, 

25, 2, 

1, 

Spot the pattern? There are several of them.

These pattern does apply to six known values donated by the original problem. Assume such pattern sustains all the way, it becomes trivial to implement a function returning P(f,r) with arbitrary f and r.  I had to use BigInteger instead of long in Java code as P can easily overflow long primitive.

Combinations of f and r is obvious since 71328803586048 = 2^27*3^12. Just for performance gain, I reused a class, BoundInteger, that I used for other problems earlier to calculate last 8 digits of integers without doing entire calculation. The whole program only takes 15ms, probably a solution that takes the least execution time comparing to other project euler problems.

I still spent a bit more than a day to get it right, mostly because overflow happens everywhere. Even the first number in a row or a number in the first row can easily overflow a Java long primitive. I had to use BigInteger everywhere.



Source Code

Joan Baez at Foley Square 11/11/11 Veterans Day

Pumpkin pie can be mass destructive

From Jiaqi's Blog

Check it out, my family is on TV! But I'm not

http://abclocal.go.com/wls/story?section=news/local/mathie&id=8443324

Dawn of Black Friday 2011

I don't have desire to join Black Friday shopping in early morning anymore, not in last 7 years, not a little bit. Things I bought from 2004 Black Friday are still in sealed boxes. This Thanksgiving, my biggest hope is to stay at home, completely forget about work, and take a long 4 days break with satisfying sleeps. Really want to do it this time.

Reality emerges in unfortunate way. At 3AM, my IPhone jumped up indicating a text message was received. There can be so many reasons for a text message on 3AM, sales promotion,wrong number, payment due, or I'm paged. I kept fingers crossed in dream and hoped it was not the last one. A few seconds later pager beeped. Instead of any of the other reasons, it really was because someone paged me. Finger crossing continue to prove in vain.

Walking to computer in pajama,  I opened two red eyes and read, "blah blah blah... to draw someone's attention, I escalated this ticket". No English words I learned could describe my mood at this point. The author of ticket decided to wake me up on 3AM Black Friday morning. not because system is in trouble, not because release is blocked, but to "draw attention". In a different fine day I would reply with a bunch of links to condition of ticket escalation and cc every manager. However now everything was pointless, I was already awake.

Ticket was nothing but a question of a well known issue, which should never, never be escalated, except that my foreign colleagues believe paging people is an effective way of communication as they live in a different timezone.

Done with work, I ran into sleep trouble again. The moment eyes are closed, all worries come to my mind. What if people recognize technical debt more seriously and decided to deal with problem earlier, what if I work for a different project, or a different company without on call, what if I'm a movie director instead of programmer, what if, what if... Brain spins so fast like Turbo key is pressed.

4:30 AM, just about the time I went over most of "what if" questions, wife called. "Honey I got this sweet deal we always wanted and you should go to Sam's Club in Seattle and get one as well". Indeed it is the deal I always want. My wife spent Thank Giving with her family this year and I'm left alone in Seattle. She brought the entire platoon to shop, which is total reasonable. With family, Black Friday shopping can be a lot of fun. The only gotcha is, she is in Indiana under Eastern timezone. "That's sweet." I murmured, "But it's really early and I prefer to sleep, so I'll skip".

5:00 AM, wife called again. "Great news, I can get one for you here in Indiana. Just tell me the SimCard number. You can take out SimCard with a needle...". So a project started. A project involving sweep of entire 3 bedroom condo to look for a needle, some handcraft, picture taking and emailing started at 5AM. Once I'm done with that, sky started showing light.

It's about 5:20 AM. I've never seen Seattle at 5AM, I don't even know the existence of 5AM in Seattle. It is stunning. After a series of crazy events which completely destroyed my hope for sleep, why not another crazy event, like going out and take photos.



Yet again, after 7 years I got up early in Black Friday morning. Plan is kaput, eyes are sore. Family is thousands mines away, and refrigerator is empty. However bad things pass and good things stay. Looking at these pictures, I am happy in the morning of Black Friday.

Cyclic numbers - Project Euler problem 358

Cyclic number, 1/7 specifically, is something amazed me while I was a child. However I never really thought about cyclic numbers other than 1/7 until this problem. I decided to spend an evening researching and resolving the latest Project Euler problem, #358 Cyclic Number.

To solve this problem, we need an integer number X satisfying following criteria based on my understanding of cyclic number:
1. 1/X starts with 0.00000000137
2. X is dividable by 10^X-1
3. The result of (10^X-1/X) ends with 56789
4. X is a prime number
5. ?

I put a question mark for the fifth criterion because I haven't figured out what it is, while only with four others, the problem can almost be solved.

The first hint limits candidates between 724637681 and 729927007. Given the second and third hint, the last 5 digits of 56789 * X should be 99999, which means 56789 * (X-1) ends with 43210. X must ends with 09891 to satisfy such requirement. There are 53 numbers within range and ends with 09891, in which only 3 are prime numbers, 725509891, 726509891 and 729809891. The final answer hides in these three numbers.

All these three numbers meets requirement 2. I tried answering them to projecteuler and found the last one is correct answer. However I haven't figure out how to filter out two others programmatically. This still bothers me.

The Java code is in GitHub. It's quite efficient as far as I can tell. Most of the time is spent on prime number checking.

>>>>>> Runnining solution of problem 358
Last five digits result is 9891
Searching numbers between 724637681 and 729927007
Checking candidate: 725509891 with sum of digits 3264794505
Checking candidate: 726509891 with sum of digits 3269294505
Checking candidate: 729809891 with sum of digits 3284144505
53 numbers are verified
<<<<<< Solution 358 took 878.149399 ms

Night of the living programmer - Facebook login in Java web application

It is a sunny Saturday, not often in autumn of Seattle. For quite a while, I've been waited for such a chance to walk peacefully in Olympic Sculpture Park. On the other side, I also always wanted to add a "Login with Facebook" button to my new website to attract users as the popularity of the website just about to reach all time low. So the plan is, finishing the login button before afternoon, have peaceful walk during sunset, which would be nice, and watch a relaxing DVD before sleep.

With this guide, adding a button with popup login window wasn't difficult. Out of box, div element with class fb-login-button is decorated as long as Facebook Javascript SDK is included.  The script snippet in tutorial is mostly to enable Javascript SDK in an asynchronous fashion for performance gain. The only gotcha is that the URL of page with button must be under a registered domain of application. I did local testing by modifying my /etc/hosts and mapping local.mydomain.com to 127.0.0.1.

However from this tutorial it isn't clear how the data that application asks user the access for, is accessed or passed over to application? In my first attempt, the pop up window asked for login information, permission to access a few fields, then closed as if it has nothing to do with the Java code running on the server side of application. My guess is that the access token used to make "/me" call should be passed as a parameter to a given URL under mydomain.com, but the tutorial doesn't say anything about it, it doesn't even say how the redirect URL is configured.

Reading further in authentication concept, I realized I could do a three-step authentication all by myself without Javascript SDK or fb-login-button class. It'd give me decent chance to smoothly embed my Java page into the process and get access token, which of course, requires a little bit of work. But then I wonder what's the purpose of the nice fb-login-button class button.

After playing for a while, it turns out in the simple fb-login-button approach, after authentication is done and window is closed, the useful access token that I expected is left in a cookie named fbs_<app-id>. I can totally redirect to an empty page that picks up this cookie after login window is closed. Follow this direction, my application ends up with:

<div  class="fb-login-button" data-perms="email"
        onlogin="location.href='/do_land_facebook.html"
        title="You may sign in right away if you have a facebook account">
    Login with Facebook</div> 

The do_land_facebook.html is back'ed with following Java code that picks up cookie and call "/me" api.

String cookieName = "fbs_" + config.getFacebookAppId();
Cookie fbCookie = null;
for ( Cookie cookie : request.getCookies() )
{
    if ( cookie.getName().equals( cookieName ) )
    {
        fbCookie = cookie;
        break;
    }
}
if ( fbCookie == null )
{    // This could happen when user clicks cancel button
    // The onlogin event is triggered no matter user clicks Login or Cancel
    return "redirect:/login_failed.html"; // It happens to be a SpringMVC based application
}
    // The cookie value is the exact query string required to call facebook API
InputStream in = new URL( "https://graph.facebook.com/me?" + fbCookie.getValue() ).openStream();
try
{
    String content = IOUtils.toString( in );
    LOG.info( "Received facebook response: " + content );
    JSONObject me = new JSONObject( content );
    loginWithFacebook(request, me); // Do my own login code logic with given JSON object graph
    return "redirect:/index.html";
}
finally 
{
    IOUtils.closeQuitely( in );
}

Finally everything starts working. I'm still debating with myself whether to do the three step authentication without Javascript SDK, or leave it as is. Maybe I will make this decision tomorrow with a dice.

Another thing I just realized, sunset is gone now and it's pitch black outside. Time went so fast and it's already the time to watch relaxing horror movie. What a Saturday.

Weekend projects

What would I do in a weekend when there's tons of work to finish before next Monday, there's tons of things I wanted to do but never had chance during week days, the weather finally turns sunny after numerous raining days, wife and daughter are both out of town so I'm left at home and have some time for myself.

I did what I always would do under such situation, spent hours cooking a satisfying meal. And I did that for twice this weekend.

From Jiaqi's Blog

From Jiaqi's Blog

Birthday again

I just realized it was my birthday recently, a wonderful thing. Some time ago I stopped celebrating birhday. It was fun when I was young, but once passed 30, birthday starts feeling much like project approaching deadline, or payment passing due date. I remember once I asked my wife on her birthday, "did I tell you you are ## years old now", and she said "yeah, I try not to think about it". Now it's my turn.

One thing came to my mind as soon as I realized it was birthday is, inevitably, movie Adaptation. In fact it kind of cheers me up. "Today is the first day of the rest of my life", what an inspiration from an old, fat, bald and repulsive Nicholas Cage.

Family organization chart

Publish Maven site with Amazon S3 and CloudFront

Amazon S3 now supports static website hosting. As a 10 years Maven user, I wonder how easy it is to deploy Maven generated site to Amazon S3 and let the rock-solid storage provider to host my project websites.

There are several existing s3 wagon providers, which all seem to have the same problem, not supporting directory copy. This is understandable since before S3 new website hosting feature, I guess people mostly expect to deploy artifacts rather than website to S3. So my first task is to write an AWS S3 wagon that supports directory copy.

With AWS Java SDK, task becomes as simple as one single class. I made my S3 wagon available in Maven central repository at org.cyclopsgroup:awss3-maven-wagon:0.1. The source code is hosted in github:jiaqi/cym2/awss3.

The next thing is to create an S3 bucket in console. To avoid trouble, bucket name is set to the future website domain name according to this discussion. Website feature needs to be explicitly enabled. I also created an IAM account with limited permission just for website management.

Comparing to other S3 wagon configuration, it's pretty much the way to configure a project to use S3 wagon. Add awss3-maven-wagon extension:

<extensions>
  <extension>
    <groupId>org.cyclopsgroup</groupId>
    <artifactId>awss3-maven-wagon</artifactId>
  </extension>
</extensions>

Set distribution management with s3 as communication protocol

<distributionManagement>
  <site>
    <server>
      <id>my-server-id</id>
      <url>s3://my-s3-bucket/project/path</url>
    </server>
  </site>
</distributionManagement>

And configure AWS credentials in settings.xml

<settings>
  <servers>
    <server>
      <id>my-server-id</id>
      <username>AWS_ACCESS_KEY_ID</username>
      <password>AWS_SECRET_KEY</password>
    </server>
    ......

Now after a site:deploy target, entire site is uploaded to S3 bucket. The website is available now under default domain name http://<bucket name>.s3-website-us-east-1.amazonaws.com. S3 bucket doesn't allow me to configure CNAME to match. This is why bucket name needs to match domain name as I plan to create friendly CNAME under my own domain. Obviously my bucket will share the same IP in Amazon cloud with others. Without explicit configuration, they only way to figure out which bucket to serve request is to match bucket name.

The last thing is CloudFront. No reason why not to take advantage of CloudFront. It's easy to setup, and it accepts CNAME configuration(unlike S3 website).

One problem about CloudFront is that since it's not designed to work as a website, request with path like http://mysite.com/a/dir is not mapped to content a/dir/index.html. S3 does not have directory concept, a/dir/index.html and a/dir can be two different objects coexist in the same bucket. Request like http://mysite.com/a/dir is mapped to object a/dir instead of a/dir/index.html. Such problem does not exist in S3 static website, while it exists when I hook up CloudFront and S3 bucket, since the hook has nothing to do with website feature anymore. In the end, I had to create object a/dir with html redirection page to redirect to a/dir/index.html in order to work around this problem.

Yilin's toddler life

Time goes fast. It only take 18 months to become an 18-month girl. Life has been busy and I haven't had chance to post photos for quite a while. Here are some for my toddler life.

1300ms to 160ms, tune Spring/Hibernate on slow MySQL

I write this article to remember the different behaviour various JDBC connection pool displays when they work with slow JDBC connection(to MySQL database, in this case). It starts with a typical Java application on Spring, Hibernate, Jetty, ApacheCXF and MySQL like following code.

Version 1: without correct pooling

//... service code
@Transactional(isolation=Isolation.READ_COMMITTED)
public void foo() {
//... do something with database
}

//... connection pool configuration
...
class = "com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource";
url = "jdbc:mysql://mysql.far-far-away.com/mysystem";
user = ...

//... transaction management configuration in spring
...
<tx:annotation-driven transaction-manager="transactionManager"
order="100" />
<bean id="transactionManager"
class="org.springframework.orm.hibernate3.HibernateTransactionManager">
<property name="sessionFactory" ref="mySessionFactory" />
</bean>
<bean id="mySessionFactory"
class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="hibernateProperties">
   <props>
       <prop key="hibernate.connection.autocommit">false</prop>
       <prop key="hibernate.connection.isolation">2</prop>
   </props>
</property>
...
</bean>
...

The reason I believe the TCP connection to mysql.farfaraway.com is slow, is that most database query from select 1 to super complicated query with X joins all returned in 70ms. While 70ms is pretty slow, comparing to my local mysql which is less than 5ms.

mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.07 sec)

With some AOP setup I was able to manage to print out the delay around foo() call with and without transaction management layer. Surprisingly, the version above took 1300ms to execute with transaction management, and 200ms without it no matter if it's the first call or repeated call. Taking a close look at each step, the longest step is that it took 800ms to establish new JDBC connection. Obviously, connection pooling isn't working. JDBC connection is supposed to be already allocated in most cases.

2011-03-29 16:13:44,682 [DEBUG] servlet.CallTracingFilter/22406408@qtp-31601333-20 - /v1/version/repository
...
2011-03-29 16:13:44,760 [DEBUG] mortbay.log/22406408@qtp-31601333-20 - /v1/version/repository - loaded interface java.sql.Savepoint
2011-03-29 16:13:44,760 [DEBUG] mortbay.log/22406408@qtp-31601333-20 - /v1/version/repository - loaded interface java.sql.Savepoint from null
2011-03-29 16:13:44,762 [DEBUG] jdbc.ConnectionManager/22406408@qtp-31601333-20 - /v1/version/repository - opening JDBC connection
2011-03-29 16:13:45,537 [DEBUG] datasource.DataSourceUtils/22406408@qtp-31601333-20 - /v1/version/repository - Setting JDBC Connection [com.mysql.jdbc.JDBC4Connection@11e1813] rea
d-only
2011-03-29 16:13:45,537 [DEBUG] impl.SessionImpl/22406408@qtp-31601333-20 - /v1/version/repository - setting flush mode to: MANUAL
2011-03-29 16:13:45,540 [DEBUG] transaction.JDBCTransaction/22406408@qtp-31601333-20 - /v1/version/repository - begin
2011-03-29 16:13:45,540 [DEBUG] transaction.JDBCTransaction/22406408@qtp-31601333-20 - /v1/version/repository - current autocommit status: true
2011-03-29 16:13:45,540 [DEBUG] transaction.JDBCTransaction/22406408@qtp-31601333-20 - /v1/version/repository - disabling autocommit
2011-03-29 16:13:45,609 [DEBUG] jdbc.JDBCContext/22406408@qtp-31601333-20 - /v1/version/repository - after transaction begin
...
2011-03-29 16:19:05,634 [INFO] MyService/32623642@qtp-31601333-21 - /v1/version/set/nkX1vZDH8rsA3NPeAkcmrA - [RUPcufrQ] Invocation of foo succeeds and returns null after 246ms
2011-03-29 16:19:05,634 [DEBUG] interceptor.TransactionInterceptor/32623642@qtp-31601333-21 - /v1/version/set/nkX1vZDH8rsA3NPeAkcmrA - Completing transaction for [MyService.foo]
...
2011-03-29 16:19:05,804 [INFO] servlet.CallTracingFilter/32623642@qtp-31601333-21 - /v1/version/set/nkX1vZDH8rsA3NPeAkcmrA - [3plLQVoH] Http call to /v1/version/set/nkX1vZDH8rsA3NPeAkcmrA succeeded after 1313ms

Version 2: commons-dbcp

MysqlConnectionPoolDataSource is not very well documented. There isn't official manual, source code isn't fully documented, and the class literally has hundreds of properties with zero Javadoc. Instead of digging into that, I went with something I'm familiar with, commons-dbcp. With configuration below, service should start up with a pre-allocated JDBC pool and each call should not require to establish new connection in most cases.

//... connection pool configuration
...
class = "org.apache.commons.dbcp.BasicDataSource";
url = "jdbc:mysql://mysql.far-far-away.com/mysystem";
driverClassName = "com.mysql.jdbc.Driver";
initialSize = 2;
minIdle = 0;
maxActive = 10;
minEvictableIdleTimeMillis = 120000;
username = ...

Now it takes 800ms with transaction management and 70ms without it. 70ms without transaction management is pretty much minimal since "select 1" takes 70ms. What an improvement!

However it still takes 70ms to pull out JDBC connection, 140ms to set isolation level and 70ms to disable auto commit. Obviously, blocking call on TCP connection is make once to get existing connection out from pool, twice when setting isolation level and once to disable auto commit.

2011-03-29 17:36:48,487 [INFO] servlet.CallTracingFilter/21266875@qtp-31601333-20 - /v1/version/repository - [HTLIDcFl] Http call to /v1/version/repository started with request: /v1/version/repository
...
2011-03-29 17:36:48,489 [DEBUG] hibernate3.HibernateTransactionManager/21266875@qtp-31601333-20 - /v1/version/repository - Preparing JDBC Connection of Hibernate Session [org.hibernate.impl.SessionImpl@15d533d]
2011-03-29 17:36:48,489 [DEBUG] jdbc.ConnectionManager/21266875@qtp-31601333-20 - /v1/version/repository - opening JDBC connection
2011-03-29 17:36:48,558 [DEBUG] datasource.DataSourceUtils/21266875@qtp-31601333-20 - /v1/version/repository - Setting JDBC Connection [jdbc:mysql://mysql.farfaraway.com/mysystem, UserName=..., MySQL-AB JDBC Driver] read-only
2011-03-29 17:36:48,626 [DEBUG] datasource.DataSourceUtils/21266875@qtp-31601333-20 - /v1/version/repository - Changing isolation level of JDBC Connection [jdbc:mysql://mysql.farfaraway.com/mysystem, UserName=..., MySQL-AB JDBC Driver] to 2
2011-03-29 17:36:48,764 [DEBUG] impl.SessionImpl/21266875@qtp-31601333-20 - /v1/version/repository - setting flush mode to: MANUAL
2011-03-29 17:36:48,764 [DEBUG] transaction.JDBCTransaction/21266875@qtp-31601333-20 - /v1/version/repository - begin
2011-03-29 17:36:48,764 [DEBUG] transaction.JDBCTransaction/21266875@qtp-31601333-20 - /v1/version/repository - current autocommit status: true
2011-03-29 17:36:48,764 [DEBUG] transaction.JDBCTransaction/21266875@qtp-31601333-20 - /v1/version/repository - disabling autocommit
2011-03-29 17:36:48,832 [DEBUG] jdbc.JDBCContext/21266875@qtp-31601333-20 - /v1/version/repository - after transaction begin
...
2011-03-29 17:36:48,975 [INFO] MyService/21266875@qtp-31601333-20 - /v1/version/repository - [Z9f815OF] Invocation of foo succeeds and returns null after 73ms
...
2011-03-29 17:36:49,323 [INFO] servlet.CallTracingFilter/21266875@qtp-31601333-20 - /v1/version/repository - [HTLIDcFl] Http call to /v1/version/repository succeeded after 836ms

Version 3: Set default autocommit and isolation

Default auto commit and transaction isolation can be defined in commons-dbcp, however Spring still always sets isolation for each transaction. This actually makes sense since @Transaction annotation allows operation to override isolation. To be sure isolation is set correctly each time, Spring has to set it each time.

As the writer of service code, I personally know that my code always use READ_COMMITTED isolation level. Based on such fact, I could remove isolation from all @Transactional annotation, rely on default isolation and set default isolation in commons-dbcp. This works fine unless there is one single operation that requires isolation level different from default one.

//... service code
@Transactional(isolation=Isolation.READ_COMMITTED)
public void foo() {
//... do something with database
}
//... connection pool configuration
...
class = "org.apache.commons.dbcp.BasicDataSource";
defaultAutoCommit = false;
defaultTransactionIsolation = 2;

With changes above, time consumption is down to 600ms. Auto commit and isolation setting is gone as expected. The last piece is the 70ms connection pulling time. There also is a 70ms delay in Hibernate JDBCContext/after transaction begin, which I couldn't explain through out this article.

2011-03-29 18:07:35,887 [DEBUG] hibernate3.HibernateTransactionManager/4540490@qtp-14707008-20 -/v1/version/repository - Preparing JDBC Connection of Hibernate Session [org.hiber
nate.impl.SessionImpl@ee5a06]
2011-03-29 18:07:35,887 [DEBUG] jdbc.ConnectionManager/4540490@qtp-14707008-20 - /v1/version/repository - opening JDBC connection
2011-03-29 18:07:36,096 [DEBUG] datasource.DataSourceUtils/4540490@qtp-14707008-20 - /v1/version/repository - Setting JDBC Connection [jdbc:mysql://mysql.farfaraway.com/mysystem, UserName=..., MySQL-AB JDBC Driver] read-only
2011-03-29 18:07:36,096 [DEBUG] impl.SessionImpl/4540490@qtp-14707008-20 - /v1/version/repository - setting flush mode to: MANUAL
2011-03-29 18:07:36,096 [DEBUG] transaction.JDBCTransaction/4540490@qtp-14707008-20 - /v1/version/repository - begin
2011-03-29 18:07:36,096 [DEBUG] transaction.JDBCTransaction/4540490@qtp-14707008-20 - /v1/version/repository - current autocommit status: false
2011-03-29 18:07:36,096 [DEBUG] jdbc.JDBCContext/4540490@qtp-14707008-20 - /v1/version/repository - after transaction begin2011-03-29 18:07:36,164 [DEBUG] hibernate3.HibernateTransactionManager/4540490@qtp-14707008-20 - /v1/version/repository - Exposing Hibernate transaction as JDBC transaction ...

Version 4: C3P0

To further explore, I switched to C3P0 just to see what can happen. Unfortunately C3P0 can't set default auto commit from configuration, I had to write a customizer to set it.

//... connection pool configuration
...
class = "org.apache.commons.dbcp.BasicDataSource";
connectionCustomizerClassName = "myapp.JdbcConnectionInitializer";
minPoolSize = 2;
maxPoolSize = 10;
maxIdleTime = 120;
...
//... Customizer
public class JdbcConnectionInitializer
extends AbstractConnectionCustomizer
{
@Override
public void onAcquire( Connection c, String id )
    throws SQLException
{
    c.setAutoCommit( false );
}
}

The total time is reduced to 300ms. The two delays in dbcp, connection pulling and strange delay in hibernate JDBCContext are both gone.

This is still not finished yet. Since my MYSQL server didn't completely turn auto commit off, the auto commit is actually set on and off during transaction commit phase. Once server is configured correctly, the total execution time reached to 160ms, which is an ideal state where each operation call only involves one single blocking TCP communication.

Summary

Communication delay	Duration	How to avoid	Conclusion
Open JDBC connection	Very long	Use connection pool like commons-dbcp or C3P0 to pre-allocate connection	For long running service, it's almost always true that connection pool should be considered
Borrow connection from pool	1 tcp call in commons-dbcp	Use C3P0 instead	Don't know how to avoid in commons-dbcp
Setting transaction isolation	2 tcp calls	If possible, always use default isolation level for application in Spring	If application does require more than 1 isolation levels in system, this delay can not be avoided
Setting auto commit before transaction starts	1 tcp call	Set default auto commit, which requires customizer class in C3P0	Not a problem is server turns it off
Strange delay in Hibernate JDBCContext	2 tcp call	Only found in commons-dbcp. Use C3P0 instead	Can't explain yet
Turning auto commit on and off to commit transaction	1 or 2 tcp calls	Configure database server to disable auto commit	May not be always feasible if you don't have control of database server

Migrate source code from SourceForge subversion repository to GitHub

Since GitHub.com, a successful commercial git hosting provider that hosts over a million repositories, offers reliable free open source git hosting service now, I decided to go ahead and move the source code that I had in last 8 years from sourceforge subversion repository to GitHub. This page documents the steps that I did, which was pretty simple and smooth anyway.

1. Obviously I need to be a user in GitHub.com. It's easy to sign up, no confirmation is required.
2. Create new repository from here, and import from a public subversion URL. The URL is something like http://<projectname>.svn.sourceforge.net/svnroot/<projectname>.
3. Before import starts, you are required to tell more about each user discovered from subversion repository. The user name and email address each sourceforge user maps to doesn't have to be existing user in GitHub.com.
   
4. It takes very very long time for import to finish. Be patient.
5. I already have git on my computer. If you don't, find out how it's installed on your operating system. If you don't know what git is, stop here and start spending some time on http://git-scm.com.
6. Tell GitHub.com the public SSH keys of each computer that wants to work with the git repository. Sourceforge does it too. Goto Account Settings/SSH Public Keys and update the keys.
7. Assume the import is done. Before pulling code to local, a little configuration is required. This document describes several properties to set. The properties are to tell GitHub.com who the user is. They end up stored in <USERHOME>/.gitconfig file.
8. Find a working directory, run the git clone command to get local copies.

git clone git@github.com:<user>/<repository name>.git