Simpler Management of Modular App Development

Published in

Prezi Engineering

9 min readOct 17, 2014

Pride is a tool we built internally at Prezi to help our developers work on modular applications in their local development environments. We think you might find it useful (provided you use Gradle), so we’re now open-sourcing it. Give it a spin, and tell us what you think!

This is the first post in a series on modular application development at Prezi.

Building a large application means trouble. It’s unavoidable, a fact of life. What you get to choose is the kind of trouble you want to deal with. Some suggest keeping everything in one large, ever-growing repository, while others opt for SOA and similar micro-component architectures. In the past, with our Flash-based editor, we suffered enough from the pains of a monolithic beast to give aggressive modularization a try with our upcoming projects. Working on tooling was a necessary side effect and Pride is one of the outputs. It allows us to split our applications into components without our developers losing the benefits of working on a single codebase.

The trouble with modularization

When you split a large codebase into pieces, you face new problems like versioning and dependency management. You have to ask yourself the question: “What do I do with the diamond problem?” There are pretty good solutions for these problems in the industry. In the JVM space, Ivy, Maven, and Gradle all successfully tackle the issue by building components separately, deploying the built artifacts to a repository, and downloading them when they are needed to build other components.

What these tools don’t have an answer for is how to tie these components together in the local development environment. According to them, components are independent entities, and only the binary artifacts they build are connected via dependencies. As long as you are working with a single component at a time, this model works fine.

However, if you want to set up a local environment to modify module A as well as module C, things will get messy. Before you can build module C using the artifacts of a changed module A, you first have to install those artifacts to a local repository. And you have to remember to do this every time you make a change to module A:

Adding more components to the mix quickly makes the process complicated. And for your own piece of mind, you’ll end up religiously repeating an invented ritual after every change:

$ cd module-a
$ gradle install
# Visit Facebook for two minutes while the build
# finishes, and then forget about it, only to
# come back ten minutes later
$ cd ../module-b
$ gradle install
# Read email for fifteen minutes and forget
# if you installed module-a
$ cd ../module-a
$ gradle install
# Check Twitter and get lost in an Ars Technica
# article for half an hour
$ cd ../module-c
$ gradle runTests

This is time consuming and easily leads to mistakes. Setting up the local environment is a complicated business, so developers tend to maintain a single copy, and soon learn to fear changing it. Ultimately, inconvenience and wasted time will encourage them not to split the codebase into more modules. This is not what you want.

Wouldn’t it be nice if your build system knew how your modules fit together, what’s available locally, what needs to be downloaded from the artifact repository, and in general just did the right thing? Something like this:

In such an environment, all you would need to do after changing any of the locally available modules is this:

$ gradle runTests

As you might have guessed, this is where Pride comes in.

Pride: a walkthrough

Installing Pride is simple:

$ \curl -sSL http://href.prezi.com/install-pride | bash

On Linux and Mac, this will install Pride in ~/.pride/versions:

Download http://repo1.maven.org/maven2/com/prezi/gradle/pride/pride/0.9.8/pride-0.9.8.pom
Download http://repo1.maven.org/maven2/com/prezi/gradle/pride/pride/0.9.8/pride-0.9.8-dist.zip
:copy
:installUnix
Pride version: 0.9.8
Created /usr/local/bin/pride
:setup
:install
Successfully installed Pride version 0.9.8

You might need to use sudo if your /usr/local is not writable by the current user.

If you are on Windows, for now you can grab the ZIP from Sonatype’s OSS repository.

Create an empty pride

As a first step, you need to create a directory to store the code for the modules you want to work on. We’ll call this directory your pride (with a lowercase “p” – we’ll be referring to the tool itself as “Pride”, with a capital “P”). Think of a pride of lions, or a group of dangerous cats you need to herd.

$ mkdir pride-test
$ cd pride-test
$ pride init
Initializing /Users/lptr/pride-test

Add the first module

We’ve prepared an example application for you that you can clone and add to your pride in one step:

$ pride add https://github.com/lptr/pride-example-application
Adding pride-example-application from https://github.com/lptr/pride-example-application
Caching repository https://github.com/lptr/pride-example-application as https-github-com-lptr-pride-example-application-3488cef
Cloning into bare repository ‘/Users/lptr/.pride/cache/https-github-com-lptr-pride-example-application-3488cef’…
Cloning into ‘/Users/lptr/pride-test/pride-example-application’

If everything went well, your pride directory should look like this:

/Users/lptr/pride-test/
|-- pride-example-application/
|   |-- src/
|   |-- LICENSE
|   |-- README.md
|   `-- build.gradle
|-- build.gradle
`-- settings.gradle

It should contain a single module:

$ pride list
m pride-example-application (git)

If you ask Gradle, it will tell you that it can see a root project, and underneath it is your application module project:

$ gradle projects
:projects
------------------------------------------------------------
Root project
------------------------------------------------------------
Root project ‘pride-test’
\--- Project ‘:pride-example-application’

Let’s check the dependencies of the application module:

$ gradle pride-example-application:dependencies —configuration compile
:pride-example-application:dependencies
------------------------------------------------------------
Project :pride-example-application
------------------------------------------------------------
compile — Compile classpath for source set ‘main’.
\--- com.example:pride-example-transformer:1.0-SNAPSHOT
     \--- com.example:pride-example-producer:1.0-SNAPSHOT

The application depends on pride-example-transformer. It’s an external dependency that Gradle downloads from the artifact repository. The transformer module further depends on another external artifact: pride-example-producer.

Enough looking around. Let’s run the application:

$ gradle run
:pride-example-application:compileJava
:pride-example-application:processResources UP-TO-DATE
:pride-example-application:classes
:pride-example-application:run
Hello World!

That “Hello World!” on the last line is the result of the code running. Yay!

Add another module

Let’s change the greeting to “Hello Gradle!” The string “World” comes from the producer module, so we’ll need to modify that module. Let’s add it to your pride:

$ pride add https://github.com/lptr/pride-example-producer
Adding pride-example-producer from https://github.com/lptr/pride-example-producer

At this point, it shouldn’t be a surprise that we now have two modules:

$ pride list
m pride-example-application (git)
m pride-example-producer (git)

The interesting part is how adding the producer module to the pride has changed the application module’s dependencies:

$ gradle pride-example-application:dependencies —configuration compile
:pride-example-application:dependencies
------------------------------------------------------------
Project :pride-example-application
------------------------------------------------------------
compile — Compile classpath for source set ‘main’.
+--- com.example:pride-example-transformer:1.0-SNAPSHOT
|    \--- com.example:pride-example-producer:1.0-SNAPSHOT -> project :pride-example-producer
          \--- project :pride-example-producer

As you see, pride-example-producer became a project dependency. Notice how familiar this picture looks?

You can now change the code of the producer module locally. Let’s just do that and modify the produce() method in com.example.producer.Producer to return “Gradle” instead of “World”:

$ vi pride-example-producer/src/main/java/com/example/producer/Producer.java

Now run the application again:

$ gradle run
:pride-example-producer:compileJava
:pride-example-producer:processResources UP-TO-DATE
:pride-example-producer:classes
:pride-example-producer:jar
:pride-example-application:compileJava
:pride-example-application:processResources UP-TO-DATE
:pride-example-application:classes
:pride-example-application:run
Hello Gradle!

That’s it!

Why is this cool?

Imagine doing something similar, but in an application that consisted of dozens of modules, and with a change that involved several of them. According to our experience with the Prezi codebase, Pride makes a huge difference. There are fewer mistakes, and we all spend less time reading Ars Technica.

Furthermore, because Pride’s über-project allows Gradle to see all the local modules, it can narrow down the set of tasks it needs to rerun after a change to the bare minimum. This saves a lot of time compared to gradle install-ing everything again and again.

How does this work?

There are two tricks employed by Pride. The first one is to tell Gradle that these modules belong to a single über-project. Pride does this by generating a settings.gradle file in the root of your pride directory, listing all your modules. This file is regenerated whenever you add or remove a module.

/Users/lptr/pride-test/
|-- pride-example-application/
| |-- src/
| |-- LICENSE
| |-- README.md
| `-- build.gradle
|-- pride-example-producer/
| |-- src/
| |-- LICENSE
| |-- README.md
| `-- build.gradle
|-- build.gradle      <------- these are
`-- settings.gradle   <------- generated

The second trick is to rewire Gradle’s dependency resolution mechanism. There are two distinct ways to tell Gradle where a dependent module comes from. You can refer to an external dependency loaded from an external artifact repository:

dependencies {
  compile group: "com.google.inject", name: "guice", version: "3.0"
}

…or you can refer to another subproject in a multi-project build:

dependencies {
  compile project(path: ":some-other-subproject")
}

When a module is available locally, we want to use it as a project dependency. When it isn’t, we want to use it as an external dependency, and let Gradle download the binaries. Unfortunately, Gradle only allows you to choose one or the other, so we have to trick it.

In a Pride-compatible module, you define your dynamic dependencies much like how you normally would, but in a block called dynamicDependencies:

dynamicDependencies {
  compile group: "com.mycompany", name: "my-module", version: "4.3.7"
}

The Pride plugin converts this declaration into either an external dependency or a project dependency depending on whether or not the referenced module is present in your pride.

You will have to apply the plugin on all your Gradle projects to make this trick work:

buildscript {
  repositories {
    mavenCentral()
  }
  dependencies {
    classpath "com.prezi.gradle.pride:gradle-pride-plugin:0.9.8"
  }
}apply plugin: "pride"

What else can you do with a pride?

Think of a pride as a work session: you start it when you want to achieve something involving a number of modules. You add those modules to your pride, and when you’re done, you can discard the whole thing. You can also have several prides set up simultaneously, even having the same module in many of them, perhaps working on different branches. Pride caches your repositories, so when you add the same module to different prides, it will only fetch changes instead of re-cloning the entire repository.

There’s also a handy shortcut if your repos happen to be under the same base URL:

$ pride config repo.base.url https://github.com/lptr

Once it’s set, you can add modules to your pride by specifying only the name of the repository:

$ pride add pride-example-transformer

You’ve already seen pride add at work. If you don’t need a module in your pride anymore, it’s easy to remove it with pride remove. Need to repeat the same command in every module? pride do is just for you. To pull from all your modules, you can use pride update.

If you want to explore further, check out the other commands Pride offers:

$ pride help
usage: pride [(-q | —quiet)] [(-v | —verbose)] <command> [<args>]The most commonly used pride commands are:
    add      Add modules to a pride
    config   Set configuration parameters
    do       Execute a command on a set of the modules
    help     Display help information
    init     Initialize pride
    list     Lists modules in a pride
    remove   Remove modules from a pride
    update   Updates a pride
    version  Display program versionSee 'pride help <command>' for more information on a specific command.

IDE support

Because a pride is also a Gradle multi-project, you can load it into any IDE that understands Gradle. For example, in IntelliJ IDEA you can simply go to File -> Open… and select the build.gradle in the root of the pride directory:

What’s next for Pride?

We are working with Gradleware to transfer parts of Pride into Gradle itself. If we are successful, you won’t need a separate Pride plugin, and you will probably be able to use the standard dependencies block instead of the custom dynamicDependencies. Once these matters are settled, we’ll release 1.0.

In the meantime, go ahead and try Pride on your own projects, and give us some feedback. If you find it useful, remember: Pride is an open-source project, and your contributions are more than welcome.

Cover image from Wikipedia