Sunday, May 10, 2020

Strange and difficult time, put on a mask


##covid19 ##coronavirus ##asian ##superhero ##american ##mask ##china

♬ I Like Him - Princess Nokia

Saturday, March 07, 2020

Spring, Angular and other reasons I like and hate Bazel at the same time

For several weeks I've been trying to put together an Angular application served Java Spring MVC web server in Bazel. I've seen the Java, Angular combination works well in Google, and given the popularity of Java, I want get it to work with open source.

How hard can it be to run arguably the best JS framework on a server in probably the most popular server-side language with the mono-repo of planet-scale? The rest of this post walks through the headaches and nightmares I had to get things to work but if you are just here to look for a working example, github/jiaqi/angular-on-java is all you need.

Java web application with Appengine rule

Surprisingly there isn't an official way of building Java web application in Bazel, the closest thing is the Appengine rule and Spring MVC seems to work well with it. 3 Java classes, a JSP and an appengine.xml was all I need. At this point, the server starts well but I got "No permissions needed for this runtime" when I open the home page.

The almighty Internet told me I should set system property use_jetty9_runtime when running the Java process, but how? Neither java_war or appengine_war rule allows to specify JVM arguments so I've been running them with bazel run java/... -- --jvm_flags="-Duse_jetty9_runtime=true" until I started reading appengine Bazel rules and realized it's not too hard to ask the rules to support a new attribute for JVM arguments. With a small code change, Appengine rule now supports a new attribute for additional arguments.

    name = "server",
    srcs = glob(["*.java"]),
    local_jvm_flags = ["-Duse_jetty9_runtime=true"],
    deps = [

Angular in Bazel

For Angular I started by pulling what I need from the latest Angular example into my repository little by little. It did not go well. Dependencies were missing, conflicting with obscure errors. I learned it hard way to start with the full example and remove the unused code. In the end

  • Pretty much every thing in WORKSPACE file needs to be copied over, with the exact version to avoid conflicts.
  • .bazelrc file needs to set incompatible_strict_action_env and angular_ivy_enabled or the package doesn't build.
  • Somehow my projects depend on @npm//@angular/animations indirectly while the example doesn't. I had to add it to the rollup_bundle target.

Get them work together

Now the hard part, connect the 2 things into a single build target. The idea is to bundle all Angular files necessary at runtime into a filegroup and make it a data resource of Appengine's war target.

To be specific, replace the pkg_web target in example with a filegroup like this.

    name = "aoj",
    srcs = [
And add it to the java_war target of Java server.

    name = "server",
    srcs = glob(["*.java"]),
    data = glob(["WEB-INF/**"]) + [
Spring MVC is configured to serve static resources.

public void addResourceHandlers(ResourceHandlerRegistry registry) {
  registry.addResourceHandler("*.png", "*.css", "*.js", "*.map").addResourceLocations("/")

Of course it didn't work out of box. The files in the filegroup target only partially added to the war file. And bazel run only cares about application directory in runfiles where most of the Angular resources are missing. Multiple issues are happening here.

Unexpected resource structure in War file

In my project I added 3 labels to the data attribute of the war file, WEB-INF/**, favicon.png and the Angular bundle. The first 2 are in the War file, something strange happened to the 3rd part.

  • The files of styles.css and external dependencies are in the War file, but at the top directory despite the original structure while files in WEB-INF are placed correctly.
  • The files of js_bundle and js_bundle_es5 are missing while empty directories exist.

$ jar -tvf bazel-bin/java/org/cyclopsgroup/aoj/server/server.war
    94 Fri Jan 01 00:00:00 PST 2010 ./styles.css
 92051 Fri Jan 01 00:00:00 PST 2010 ./core.min.js
 27476 Fri Jan 01 00:00:00 PST 2010 ./system.js
 64650 Fri Jan 01 00:00:00 PST 2010 ./deeppurple-amber.css
 49589 Fri Jan 01 00:00:00 PST 2010 ./zone.min.js
 11413 Fri Jan 01 00:00:00 PST 2010 ./favicon.png
     0 Fri Jan 01 00:00:00 PST 2010 ./js_bundle_es5/
   295 Fri Jan 01 00:00:00 PST 2010 ./WEB-INF/appengine-web.xml
101840 Fri Jan 01 00:00:00 PST 2010 ./WEB-INF/lib/stamped_flogger-0.4.jar

The issue isn't very hard to explain. According to the code, when the code of resource is not in or under the same directory of web application, the output_path in code is the original output so the files are symlink'ed to the top directory. Being symlinks, means that their children file may not be included by the zipper command depend on the command line options.

With the 2 issues, the produced war file is incomplete. However if I serve static JS/CSS resource separately, namely via CDN, instead of being part of the same web application, I could live with these problems.

Missing resources at runtime

Even though some of the files like styles.css and core.min.js are in the war file, when I ran the application with bazel

bazel run java/org/cyclopsgroup/aoj/server

these files are still not accessible at http://localhost:8080/styles.css or http://localhost:8080/core.min.js. It seems what's in the war file really has nothing to do with what's available to the running web server. When I ran bazel run, this directory is the root directory of resources.


And what's in it is surprised me.

$ ls bazel-out/darwin-fastbuild/bin/java/org/cyclopsgroup/aoj/server/server.runfiles/angular_on_java/java/org/cyclopsgroup/aoj/server/
WEB-INF liblibserver.jar server.war
favicon.png server

None of the Angular resources is here. The file structure is almost identical to the source code structure. What I specified in data attribute is almost ignored entirely. But how does it work? If nothing puts resources into this directory, what puts the jar files here under WEB-INF/lib?

The answer, of course, is in the code. Here, Appengine rules symlink jar files into WEB-INF/lib not at the build time, but at runtime. However there's nothing that does the same for other data resources. With the way how code works today, this template should also symlink data sources for the target to run.

The solution

I went ahead and fixed the majority of the issues in my fork of rules_appengine and filed another PR to Google Appengine Bazel rules. Before the PR is accepted and a new version is released, I build things with my fork.

    name = "io_bazel_rules_appengine",
    remote = "",
    tag = "",

With the fix the application works end-to-end now. I can run a single build target to build both Angular and Java server, and run ibazel to sync Angular changes to the server.

bazel run java/org/cyclopsgroup/aoj/server
ibazel build webapp/aoj


  • Bazel is a game-changing build tool and it's proven to work pervasively well in Google. It works well for some cases outside Google too. However the ecosystem is far from mature, to even support some vanilla use cases like running Angular on Java.
  • Appengine support in Bazel is incomplete and lack of development. General Java web application support in Bazel is missing.
  • Bazel and Angular are both from Google, and I'm also working on getting gRPC into the picture, it seems they don't necessarily work well together even though they are all from Google.

Sunday, February 16, 2020

Favorite quote

I spoke frankly, and I expected those around me to speak frankly. I fought for what I thought was best, and I wanted them to do so as well. When I thought someone did something stupid, I said so and I expected them to tell me when I did something stupid. Each of us would be better for it. To me, that was what strong and productive relationships looked like. Operating any other way would be unproductive and unethical.
- Ray Dalio, Principles

These words are so true and yet conflicting. People keep telling me otherwise, books and studies teach the secret of tricking emotion. I can't say the principle works everywhere, in fact it probably doesn't work in most places, but it certainly helps me stay true about myself.

One of the 14 Amazon leadership principles is to "disagree and commit". It's uncomfortable but like Ray Dalio said, each of us are better for it. I've seen it in Amazon and I'm sure BridgeWater is the same. Like it or not, it works.

Tuesday, December 17, 2019

History repeats itself because nobody was listening the first time

I like how Rick Houlihan starts with the statement, "History repeats itself because nobody was listening the first time". However I disagree because nobody ever listens or learns the lesson.

Monday, February 11, 2019

Wednesday, January 30, 2019

Introducing Flixport, a tool that exports photos from Flickr


If you are like me with tens of thousands photos in Flickr which is about to stop 1TB free storage service, moving these photos out would be a very challenging task.

Flixport is an open source command line tool that bulk-copies photos from Flickr to Amazon S3, Google Storage or your local computer.

Flickr is terminating the free 1TB storage service in February 2019. Unless you convert to a premium user, Flickr will only keep 1000 photos and purge others at some point in February. To backup my own photos I evaluated all existing solution and none worked for me. Luckily, being a software engineer, I could write code to work around difficulties.

Please visit the Flixport page to find out more.

Why S3 and Google storage?

AWS S3, as of today, is the de facto online storage. People may hear Dropbox or Box more often but many of these consuming storage product are backed by another Cloud storage service, of which S3 is the clear leader. Since coding is not a problem for me, I got rid of the middle man and went straight to S3.

Because I work for Google, I also felt comfortable implementing a connection to Google Cloud storage. Sorry Azure, no Microsoft support for now.

Why not Dropbox, Google Drive or Google Photos?

These popular consumer products are obviously good next steps. In general for a command line to work with them, some OAuth-based authentication needs to happen and users will have to copy some long, obscure token from browser and paste it in command line. To offer the best user experience, it's better to run the tool from a website instead of as a command line.

Therefore, if I ever had bandwidth to work on the support of them, it'd not be part of the command line tool, but likely a web-based service that integrates with user's Google or Dropbox account.