Thursday, July 16, 2020

Build Java webapp and run it with Tomcat in Bazel

The official support of Java web application in Bazel is bazelbuild/rules_appengine, which comes with a number of problems as discussed in a previous post. After some effort I realized I can not manage to have appengine rule accurately serve my needs. Since there doesn't seem to be another option, I decided to go ahead and create one.

With bazville I was able to change the demo project angular-on-java to build web application correctly without assuming it runs in appengine, and run it with Tomcat in bazel. Please take a look at the project github docs if you are interested.

Saturday, March 07, 2020

Spring, Angular and other reasons I like and hate Bazel at the same time

For several weeks I've been trying to put together an Angular application served Java Spring MVC web server in Bazel. I've seen the Java, Angular combination works well in Google, and given the popularity of Java, I want get it to work with open source.

How hard can it be to run arguably the best JS framework on a server in probably the most popular server-side language with the mono-repo of planet-scale? The rest of this post walks through the headaches and nightmares I had to get things to work but if you are just here to look for a working example, github/jiaqi/angular-on-java is all you need.

Java web application with Appengine rule

Surprisingly there isn't an official way of building Java web application in Bazel, the closest thing is the Appengine rule and Spring MVC seems to work well with it. 3 Java classes, a JSP and an appengine.xml was all I need. At this point, the server starts well but I got "No permissions needed for this runtime" when I open the home page.

The almighty Internet told me I should set system property use_jetty9_runtime when running the Java process, but how? Neither java_war or appengine_war rule allows to specify JVM arguments so I've been running them with bazel run java/... -- --jvm_flags="-Duse_jetty9_runtime=true" until I started reading appengine Bazel rules and realized it's not too hard to ask the rules to support a new attribute for JVM arguments. With a small code change, Appengine rule now supports a new attribute for additional arguments.

    name = "server",
    srcs = glob(["*.java"]),
    local_jvm_flags = ["-Duse_jetty9_runtime=true"],
    deps = [

Angular in Bazel

For Angular I started by pulling what I need from the latest Angular example into my repository little by little. It did not go well. Dependencies were missing, conflicting with obscure errors. I learned it hard way to start with the full example and remove the unused code. In the end

  • Pretty much every thing in WORKSPACE file needs to be copied over, with the exact version to avoid conflicts.
  • .bazelrc file needs to set incompatible_strict_action_env and angular_ivy_enabled or the package doesn't build.
  • Somehow my projects depend on @npm//@angular/animations indirectly while the example doesn't. I had to add it to the rollup_bundle target.

Get them work together

Now the hard part, connect the 2 things into a single build target. The idea is to bundle all Angular files necessary at runtime into a filegroup and make it a data resource of Appengine's war target.

To be specific, replace the pkg_web target in example with a filegroup like this.

    name = "aoj",
    srcs = [
And add it to the java_war target of Java server.

    name = "server",
    srcs = glob(["*.java"]),
    data = glob(["WEB-INF/**"]) + [
Spring MVC is configured to serve static resources.

public void addResourceHandlers(ResourceHandlerRegistry registry) {
  registry.addResourceHandler("*.png", "*.css", "*.js", "*.map").addResourceLocations("/")

Of course it didn't work out of box. The files in the filegroup target only partially added to the war file. And bazel run only cares about application directory in runfiles where most of the Angular resources are missing. Multiple issues are happening here.

Unexpected resource structure in War file

In my project I added 3 labels to the data attribute of the war file, WEB-INF/**, favicon.png and the Angular bundle. The first 2 are in the War file, something strange happened to the 3rd part.

  • The files of styles.css and external dependencies are in the War file, but at the top directory despite the original structure while files in WEB-INF are placed correctly.
  • The files of js_bundle and js_bundle_es5 are missing while empty directories exist.

$ jar -tvf bazel-bin/java/org/cyclopsgroup/aoj/server/server.war
    94 Fri Jan 01 00:00:00 PST 2010 ./styles.css
 92051 Fri Jan 01 00:00:00 PST 2010 ./core.min.js
 27476 Fri Jan 01 00:00:00 PST 2010 ./system.js
 64650 Fri Jan 01 00:00:00 PST 2010 ./deeppurple-amber.css
 49589 Fri Jan 01 00:00:00 PST 2010 ./zone.min.js
 11413 Fri Jan 01 00:00:00 PST 2010 ./favicon.png
     0 Fri Jan 01 00:00:00 PST 2010 ./js_bundle_es5/
   295 Fri Jan 01 00:00:00 PST 2010 ./WEB-INF/appengine-web.xml
101840 Fri Jan 01 00:00:00 PST 2010 ./WEB-INF/lib/stamped_flogger-0.4.jar

The issue isn't very hard to explain. According to the code, when the code of resource is not in or under the same directory of web application, the output_path in code is the original output so the files are symlink'ed to the top directory. Being symlinks, means that their children file may not be included by the zipper command depend on the command line options.

With the 2 issues, the produced war file is incomplete. However if I serve static JS/CSS resource separately, namely via CDN, instead of being part of the same web application, I could live with these problems.

Missing resources at runtime

Even though some of the files like styles.css and core.min.js are in the war file, when I ran the application with bazel

bazel run java/org/cyclopsgroup/aoj/server

these files are still not accessible at http://localhost:8080/styles.css or http://localhost:8080/core.min.js. It seems what's in the war file really has nothing to do with what's available to the running web server. When I ran bazel run, this directory is the root directory of resources.


And what's in it is surprised me.

$ ls bazel-out/darwin-fastbuild/bin/java/org/cyclopsgroup/aoj/server/server.runfiles/angular_on_java/java/org/cyclopsgroup/aoj/server/
WEB-INF liblibserver.jar server.war
favicon.png server

None of the Angular resources is here. The file structure is almost identical to the source code structure. What I specified in data attribute is almost ignored entirely. But how does it work? If nothing puts resources into this directory, what puts the jar files here under WEB-INF/lib?

The answer, of course, is in the code. Here, Appengine rules symlink jar files into WEB-INF/lib not at the build time, but at runtime. However there's nothing that does the same for other data resources. With the way how code works today, this template should also symlink data sources for the target to run.

The solution

I went ahead and fixed the majority of the issues in my fork of rules_appengine and filed another PR to Google Appengine Bazel rules. Before the PR is accepted and a new version is released, I build things with my fork.

    name = "io_bazel_rules_appengine",
    remote = "",
    tag = "",

With the fix the application works end-to-end now. I can run a single build target to build both Angular and Java server, and run ibazel to sync Angular changes to the server.

bazel run java/org/cyclopsgroup/aoj/server
ibazel build webapp/aoj


  • Bazel is a game-changing build tool and it's proven to work pervasively well in Google. It works well for some cases outside Google too. However the ecosystem is far from mature, to even support some vanilla use cases like running Angular on Java.
  • Appengine support in Bazel is incomplete and lack of development. General Java web application support in Bazel is missing.
  • Bazel and Angular are both from Google, and I'm also working on getting gRPC into the picture, it seems they don't necessarily work well together even though they are all from Google.

Sunday, February 16, 2020

Favorite quote

I spoke frankly, and I expected those around me to speak frankly. I fought for what I thought was best, and I wanted them to do so as well. When I thought someone did something stupid, I said so and I expected them to tell me when I did something stupid. Each of us would be better for it. To me, that was what strong and productive relationships looked like. Operating any other way would be unproductive and unethical.
- Ray Dalio, Principles

These words are so true and yet conflicting. People keep telling me otherwise, books and studies teach the secret of tricking emotion. I can't say the principle works everywhere, in fact it probably doesn't work in most places, but it certainly helps me stay true about myself.

One of the 14 Amazon leadership principles is to "disagree and commit". It's uncomfortable but like Ray Dalio said, each of us are better for it. I've seen it in Amazon and I'm sure BridgeWater is the same. Like it or not, it works.