Clojure on Google App Engine

I’ve written an ebook about Programming Language Concepts      Show me!

There are plenty of posts around on how to run Clojure on Google App Engine. However, most of them are quite dated and as times (and Google libraries) change, I’d like to share my effort of trying to run Clojure on App Engine without losing the major benefit of developing interactively with a REPL.

So, let’s go.

The Tools for running Clojure on Google App Engine

What you’ll need to follow along what I’ll be covering in this post is:

By the way, I am doing this on a Linux machine, but it is probably not too difficult to adjust the commands to work on a Mac or Windows just as well.

The Goal

We’ll try to create a tiny Hello World compojure app, have it run on App Engine and also be able to use the REPL to work interactively on our local machine.

Do it

We’ll work ourselves through several steps:

  1. Setup our Compojure app via Leiningen, so that it runs fine locally.
  2. Try to run that app in the Google App Engine dev server locally via the script dev_appserver.sh provided by Google.
  3. Try to deploy the app on the production system via appcfg.sh script provided by Google.
  4. Try to plug ourselves into the local server, so we can have our oh-so-valuable REPL (this is the most difficult part).
  5. Try to read & write to the datastore, to make sure everything we need is working.

Step 1: Bootstrapping our webapp locally

As I am somewhat reluctant to start with third party Leiningen templates (maybe a feeling that I need to get rid of), we’ll instead bootstrap our app using a blank Leiningen app template and add the necessary dependencies (Ring & Compojure) right away:

bash$ lein new default clojure-gae
...
bash$ cd clojure-gae
bash$ vim project.clj

and then add dependencies so that the file project.clj finally looks like this:

(defproject clojure-gae "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring "1.4.0-RC2"]
                 [compojure "1.3.4"]]
  :plugins [[lein-ring "0.9.6" :exclusions [org.clojure/clojure]]]
  :ring {:handler clojure-gae.core/app})

We also need to define some routes. Adjust the file core.clj to look like the this:

(ns clojure-gae.core
  (:require [compojure.core :refer :all]
            [compojure.route :as route]))

(defroutes app
  (GET "/" [] "<h1>Hello World</h1>")
  (route/not-found "<h1>Page not found</h1>"))

Now run the app together with the included webserver (Ring):

bash$ lein ring server

It should take a few seconds and then respond with something like

2015-07-08 16:09:57.548:INFO::main: Logging initialized @961ms
2015-07-08 16:09:59.669:INFO:oejs.Server:main: jetty-9.2.10.v20150310
2015-07-08 16:09:59.692:INFO:oejs.ServerConnector:main: Started ServerConnector@1fe2008{HTTP/1.1}{0.0.0.0:3000}
2015-07-08 16:09:59.692:INFO:oejs.Server:main: Started @3106ms
Started server on port 3000

…telling you that you can now browse to http://localhost:3000 and see the nice Hello World heading on a otherwise blank web page. Most probably, the web page in question will be opened up for you in your browser implicitly.

So far, so good. But up to this point it was nothing more than just basic default stuff.

Step 2: Running our webapp in the App Engine dev server

Let’s take this a step further: The Google App Engine SDK provides us with a local development server that resembles the App Engine environment you’ll find in the cloud. This dev server instance can be run via a script named dev_appserver.sh . Well, but this is an application server and expects an exploded WAR archive.

Let’s create one then, our plugin lein-ring comes in handy as it seems:

bash$ lein ring uberwar

This will build a WAR archive, including not only our app but also all dependencies for running it. It will be placed into the target/ directory. All we have to do is extract it, as the GAE dev server wants it that way:

bash$ mkdir -p target/war
bash$ cd target/war
bash$ jar xf ../clojure-gae-0.1.0-SNAPSHOT-standalone.war
bash$ ls -la
.  ..  META-INF  WEB-INF

Fine, let’s have a try:

bash$ dev_appserver.sh .

Oops… it responds with an exception:

... com.google.apphosting.utils.config.AppEngineConfigException: Could not locate clojure-gae/target/war/WEB-INF/appengine-web.xml ...

Oh yeah, we have to provide a GAE deployment descriptor, namely appengine-web.xml . Let’s create one:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application>clojure-gae-123</application>
    <version>1</version>
    <threadsafe>true</threadsafe>
</appengine-web-app>

I have simply copied that one from an existing GAE Java application that I had created during a tutorial. All you need to adjust in that file is the application name – insert your ProjectID here (the one you got when you created the app in the Google web developer console in the beginning).

Okay, now try again:

bash$ dev_appserver.sh .

Now, this looks better. The output should not throw any exceptions and the last lines should be something similar to

INFO: Started SelectChannelConnector@127.0.0.1:8080
Jul 08, 2015 4:34:18 PM com.google.appengine.tools.development.AbstractModule startup
INFO: Module instance default is running at http://localhost:8080/
Jul 08, 2015 4:34:18 PM com.google.appengine.tools.development.AbstractModule startup
INFO: The admin console is running at http://localhost:8080/_ah/admin
Jul 08, 2015 4:34:18 PM com.google.appengine.tools.development.DevAppServerImpl doStart
INFO: Dev App Server is now running

…telling you that the app is now running on http://localhost:8080 . Fine.

Step 3: Deploy it in production

Just to verify that we are not off the track: Does this also work in production? Let’s have a look:

bash$ appcfg.sh update .
...
Update for module default completed successfully.
Success.
Cleaning up temporary files for module default...

This step can take some time when run the first time, as all the dependencies (Clojure itself, Ring, Compojure etc.) will be uploaded. Luckily, this process will only upload changed files, so subsequent runs should take much less time.

Well, it reports success – what does the browser say?

Yeah!

Again, the first access after a deployment can really take some time, as all stuff needs to be set up on Google’s side. Subsequent accesses respond really fast.

Step 4: Oh noes, gimme my REPL!

Okay, things are looking good. However, we are as of yet missing one important feature that makes developing Clojure so much fun: Interactive development in the REPL, so we don’t have to restart the JVM for every tiny change we make.

Embedding an nREPL in our app?

Well, there are two solution paths I can think of. The first one that initially seems to be simpler and safer is to include an nREPL into our app and start it somewhere in our application code. This way we could start our app like above in the dev_appserver.sh environment and then connect a REPL. Well yeah, but no.

Unfortunately, apps running in an App Engine environment are not allowed to do anything they want. They are for example forbidden to spawn Threads, read/write files on the local filesystem or open server sockets. Well and as you might already have guessed, listening on a port and spawning a Thread is exactly what our nREPL needs to do.

So, to cut a long story short: Embedding an nREPL in our GAE app is not an option.

Boot up our own App Engine environment!

That leave us with the second approach: Forget dev_appserver.sh, start our REPL like we are used to and from there boot up the Google App Engine environment. Sounds difficult? Well yeah, fortunately I found this very helpful post which led me into the right direction but didn’t work out of the box due to being a bit outdated.

As a first note, the GAE SDK is open source and that’s why nothing keeps you from using it in your code which we are about to do.

(Note: There may be a less custom way to achieve this – I will have to analyze the dev_appserver.sh script in detail to find out if the same can be achieved more generally)

As a preparation, we need to add three GAE jars to our local maven repo, as for some reason at least one of these (appengine-local-runtime-shared.jar) can not be found in any remote maven repo (if anyone has a better solution, please leave it in the comments). So what I did was add the lein-localrepo plugin to my project.clj:

(defproject clojure-gae "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring "1.4.0-RC2"]
                 [compojure "1.3.4"]]
  :plugins [[lein-ring "0.9.6" :exclusions [org.clojure/clojure]]
            [lein-localrepo "0.5.3"]]
  :ring {:handler clojure-gae.core/app})

As a side note, you can instead add lein-localrepo as a plugin to your $HOME/.lein/profiles.clj, which will make it available globally, not just for this project.

After that, install the following three jars in your local maven repo via

bash$ export GAE_SDK=$HOME/google-cloud-sdk/platform/appengine-java-sdk
bash$ export GAE_VERSION=1.9.23
bash$ lein localrepo install $GAE_SDK/lib/impl/appengine-api-stubs.jar com.google.appengine/appengine-api-stubs $GAE_VERSION
bash$ lein localrepo install $GAE_SDK/lib/impl/appengine-local-runtime.jar com.google.appengine/appengine-local-runtime $GAE_VERSION
bash$ lein localrepo install $GAE_SDK/lib/shared/appengine-local-runtime-shared.jar com.google.appengine/appengine-local-runtime-shared $GAE_VERSION

Adjust the path and version to what is appropriate for you.

Next, add those jars as dev dependencies to your project.clj; again, don’t forget to adjust the version number:

(defproject clojure-gae "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring "1.4.0-RC2"]
                 [compojure "1.3.4"]]
  :plugins [[lein-ring "0.9.6" :exclusions [org.clojure/clojure]]
            [lein-localrepo "0.5.3"]]
  :ring {:handler clojure-gae.core/app}
  :profiles {:dev
             {:dependencies
              [[com.google.appengine/appengine-api-stubs "1.9.23"]
               [com.google.appengine/appengine-local-runtime "1.9.23"]
               [com.google.appengine/appengine-local-runtime-shared "1.9.23"]]}})

And we need to create the layer that boots up the App Engine environment. The following code takes care of that:

(ns clojure-gae.appengine
  (:require [clojure-gae.core :as handler])
  (:use ring.adapter.jetty))

(def myport 8080)
(def myserver (atom nil))

(defmacro with-app-engine
  "testing macro to create an environment for a thread"
  ([body]
   `(with-app-engine env-proxy ~body))
  ([proxy body]
   `(last (doall [(com.google.apphosting.api.ApiProxy/setEnvironmentForCurrentThread ~proxy)
                  ~body]))))

(defn login-aware-proxy
  "returns a proxy for the google apps environment that works locally"
  [request]
  (let [email (:email (:session request))]
    (proxy [com.google.apphosting.api.ApiProxy$Environment] []
      (isLoggedIn [] (boolean email))
      (getAuthDomain [] "")
      (getRequestNamespace [] "")
      (getDefaultNamespace [] "")
      (getAttributes [] (java.util.HashMap.))
      (getEmail [] (or email ""))
      (isAdmin [] true)
      (getAppId [] "local"))))

(defn environment-decorator
  "decorates the given application with a local version of the app engine environment"
  [application]
  (fn [request]
    (with-app-engine (login-aware-proxy request)
      (application request))))

(defn init-app-engine
  "Initialize the app engine services."
  ([]
   (init-app-engine "."))
  ([dir]
   (let [factory (com.google.appengine.tools.development.ApiProxyLocalFactory.)
         env (reify com.google.appengine.tools.development.LocalServerEnvironment
               (getAppDir [this] (java.io.File. dir))
               (getAddress [this] "localhost")
               (getHostName [this] "localhost")
               (getPort [this] myport)
               (waitForServerToStart [this])
               (enforceApiDeadlines [this] true)
               (simulateProductionLatencies [this] true))
         delegate (.create factory env)]
     (com.google.apphosting.api.ApiProxy/setDelegate delegate))))

(defn start-it []
  (if (not @myserver)
    (init-app-engine))
  (reset! myserver
          (run-jetty (environment-decorator handler/app)
                     {:port myport :join? false}))
  (.start @myserver))

(defn stop-it []
  (.stop @myserver)
  (reset! myserver nil))

Note: this has been taken from http://www.hackers-with-attitude.com/2009/08/intertactive-programming-with-clojure.html and adjusted a bit to reflect the current version of Google App Engine API.

That should be it. Now we can try to boot it all up:

bash$ lein repl

nREPL server started on port 44844 on host 127.0.0.1 - nrepl://127.0.0.1:44844
REPL-y 0.3.5, nREPL 0.2.10
Clojure 1.6.0
Java HotSpot(TM) Server VM 1.8.0_45-b14
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e


user=> (require '[clojure-gae.appengine :as ae])
2015-07-08 18:14:29.507:INFO::nREPL-worker-0: Logging initialized @34280ms
nil
user=> (ae/start-it)
2015-07-08 18:14:38.992:INFO:oejs.Server:nREPL-worker-0: jetty-9.2.10.v20150310
2015-07-08 18:14:39.022:INFO:oejs.ServerConnector:nREPL-worker-0: Started ServerConnector@cc75ee{HTTP/1.1}{0.0.0.0:8080}
2015-07-08 18:14:39.023:INFO:oejs.Server:nREPL-worker-0: Started @43796ms
#'clojure-gae.appengine/myserver
user=> 

And the browser on http://localhost:8080 says: Yeah!

Step 5: Use the Datastore, so we can be sure it’s all working fine

Adjust project.clj once more to include the appengine api jar as dependency:

(defproject clojure-gae "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [ring "1.4.0-RC2"]
                 [compojure "1.3.4"]
                 [com.google.appengine/appengine-api-1.0-sdk "1.9.23"]]
  :plugins [[lein-ring "0.9.6" :exclusions [org.clojure/clojure]]
            [lein-localrepo "0.5.3"]]
  :ring {:handler clojure-gae.core/app}
  :profiles {:dev
             {:dependencies
              [[com.google.appengine/appengine-api-stubs "1.9.23"]
               [com.google.appengine/appengine-local-runtime "1.9.23"]
               [com.google.appengine/appengine-local-runtime-shared "1.9.23"]]}})

Finally, we put some code into our core.clj that makes use of the datastore:

(ns clojure-gae.core
  (:require [compojure.core :refer :all]
            [compojure.route :as route])
  (:import [com.google.appengine.api.datastore DatastoreServiceFactory]
           [com.google.appengine.api.datastore Query]
           [com.google.appengine.api.datastore FetchOptions$Builder]
           [com.google.appengine.api.datastore Entity]))

(defn read-db
  []
  (let [datastore (DatastoreServiceFactory/getDatastoreService)
        query (Query. "item")
        prepared-query (.prepare datastore query)
        result (.asList prepared-query (FetchOptions$Builder/withDefaults))]
    (pr-str (map #(.getProperty %1 "name")
                 result))))

(defn write-db
  [name]
  (let [datastore (DatastoreServiceFactory/getDatastoreService)
        entity (Entity. "item")]
    (.setProperty entity "name" name)
    (let [key (.put datastore entity)]
      (str key))))

(defroutes app
  (GET "/" [] (str "<h1>Hello World</h1><h2>db contains:</h2><p>" (read-db) "</p>"))
  (GET "/new/:name" [name] (write-db name))
  (route/not-found "<h1>Page not found.</h1>"))

You actually have to stop the server once, as we touched the defroutes statement here. If anybody knows how to make the running Jetty instance aware of the change, please let me know.

So, after stopping the server and firing everything up again via

user=> (ae/stop-it)
user=> (ae/start-it)

you can add new entries via the route http://localhost:8080/new/something :

…and have a look at all entries in the db at http://localhost:8080/ :

Great! I hope this little tutorial help others to get Clojure on App Engine running.

By the way, I am aware there is appengine-magic. Sadly however, it does not seem to be under active maintenance.

References

Credits go to the following posts that helped me a lot:

I’ve written an ebook about Programming Language Concepts      Show me!
Comments(4)
  1. Alojamiento June 7, 2016
  2. Tim April 29, 2016
    • oliver April 29, 2016
  3. Andy Dent December 20, 2015

Leave a Reply

Your email address will not be published. Required fields are marked *

*