Is Java really faster than Go?

Researching earlier Java/Go benchmarks

Earlier I blogged that Go is better than Java, but it lacked evidence and datapoints. I wanted to start from some earlier work, to prove to my colleagues that Go is as fast as Java and quick to start up. Speed is critical for serverless since the process can be stopped when not in use and started again from cold to serve a request!

So after searching https://dzone.com/ for “Java Go” comparisons I was surprised to find a benchmark article by Ivan Nikitsenka (original author) at the top of results:

Which concluded that Java can serve twice as many simultaneous users as the Go application.

WHAT? 🤯

Is that true?! Can Java’s lazily just-in-time compiler be faster than a Go static binary?! Can I reproduce his results!?

Reproducing the results

To the original author’s credit, he publishes the source of his Java & Golang application as well as how he tested with Jmeter, his results AND how to run it all on a neutral playing ground of AWS via Cloudformation!

I ran it locally on my T14s Thinkpad and indeed the Go application was erroring compared to the “springboot” Java app. Despite the bank-go image weighing in at 10.2MB compared to the 991MB image size of bank-java and consuming a lot less resources whilst running:

Why is Go slower?

Instinctively I thought the database connection must be the bottle neck. The original author’s bank-go database functions use https://github.com/lib/pq#status which recommends using pgx which is actively maintained. Great! All I need is to do, is switch the database driver from pq to pgx. Despite making the change to the “sql.DB” type compatible github.com/jackc/pgx/v4/stdlib … the same type of errors occurred when load testing…

 read tcp 127.0.0.1:XXXXX->127.0.0.1:5432: read: connection reset by peer #6

After some soul searching and a cup of tea, perhaps it has something to do with how the connections are pooled to the database. Unfortunately it meant a refactor from *sql.DB to *pgxpool.Pool where context needs to be added.

Yes! The errors have gone away. It also appears much faster. Whatever pgxpool (limiting connections to the database?) is doing, it seems to be working!

Load testing locally on my T14s

Java does take a few seconds to get going to generate the machine code under the hood…

Using hey load tester instead of Jmeter:

Locally I was seeing Go at ~7457 requests/sec and Java at ~5758 requests/sec once it warmed up. Pretty much the same. However we should run the original author’s jmeter test with a controlled / reproducible environment… enter the Cloud.

Load testing Go/Java on AWS

There are three potential bottlenecks:

1. The client benchmarking tool (as well as the network)
2. The app
3. The database

And lets not forget that AWS’s T type instances (virtual machines) are Burstable Performance Instances and might be too variable for benchmarking.

I decided to use m4.large for both bank-{app,db} and run original jmeter benchmark upon the app server and update the Cloudformation to use AWS Linux ECS. Note that I hard coded the IP address of the database, so you need to change that when you are reproducing results yourself.

I setup my ssh public key like so:

aws --profile dev ec2 import-key-pair --key-name "hendry" --public-key-material fileb://~/.ssh/id_rsa.pub

So my AWS benchmarking workflow was something like:

1. make delete - tear down Cloud resources
2. make deploy - bring up the App and Database on the static IP
3. Wait especially long for the Java version - running mvn seemed to be where it was mostly
4. benchmark.sh “test-name”

Go Benchmark 1 Benchmark 2 Benchmark 3 Java Benchmark 1 Benchmark 2 Benchmark 3

Conclusions

• Java takes a lot longer to stand up than Go - not a good candidate for serverless! 2022 UPDATE may have changed with Lambda SnapStart
• Orchestration like an ALB health check could be incorporated into the stack though I ran out of time. The instances build the Docker image and it’s not clear when they are ready…
• Repeated testing on bank-{go,java} resulted in No file descriptors available exhaustion, this appears to be an AWS ECS issue
• Detailed monitoring via Cloudwatch of the instance was too course grained to tell if the database was the bottle neck… quite a disappointing DX. Further instrumentation is probably needed to work out where the bottle necks lie
• Go appears a little faster, however more stable from a cold start, with the 99p being far lower ~100ms than Java’s >2000ms .. However over some runtime I suspect Java will be more stable.

Not clear what the errors that the original author initially observed, since I think this is how the original author mistakenly concluded that Java could serve twice as many users. In my testing, I could run the tests without errors in either Go/Java stack when I waited patiently for the Java service to be ready, and not run the tests repeatedly as to cause too many open files.

The original authors’s Go code appears to have had a database connection pool limit issue which goes away when using pgxpool.

As the Reddit /r/java and YouTube comments suggest, For most tasks Java and Go are completely fine performance wise. However I did find Java quite unwieldy, with proponents mandating warm up time for Java, which would certainly result in a problematic Serverless cold start. In comparison Go has out of the box developer productivity and serverless friendly execution times.

Nonetheless Java frameworks to their credit are targeting slow startup times with Ahead of Time Compilation (AoT) with GraalVM & Quarkus. It is great to see healthy competition, though currently Java has some catching up to do!