Anatomy of a Web App: How I Built RedditLater in Clojure

Share this article

I made RedditLater last year to allow people to post to Reddit at some pre-scheduled time. It has a modest usage; a few hundred visitors per day, with some fraction of those scheduling posts. In this article, I’ll write about how RedditLater works, and why I made some of the decisions I did.

Architecture

I wrote the project about a year ago. It’s hosted on a single Heroku dyno. I chose Heroku because, hey, free hosting. So technical decisions were made with these limitations in mind – even though usage is modest, the hosting environment is quite limited. The app had to be able to deliver its scheduled posts (approximately) on time, and survive the occasional traffic surge from itself being posted on reddit.

RedditLater works by spinning up a separate worker thread to monitor the queue of posts destined for Reddit, which runs alongside the web server. The worker thread just goes through the list of queued posts and sends them to reddit whenever it finds one where schedule < now.

RedditLater was written using Clojure. I chose Clojure mostly because I was way into Clojure at the time. I still am, but I was then too. In retrospect, though, I can say it was a fine decision. Clojure is a simple, functional language with top-notch support for concurrency. It’s not exactly mainstream, but it’s popular enough to have first-class support on Heroku. RedditLater relies on the concurrency that Clojure makes so accessible to run parallel tasks on that one Heroku instance, especially the Lamina library and its excellent queue structures.

For persistence, posts and user login data are stored in a Mongo database hosted on MongoHQ, which plays nicely with Heroku. I used Mongo because the app isn’t database-intensive, and because Mongo is easy to use, especially from a language with a hash-map literal as Clojure has.

(I’ve come to think of Mongo as the datastore you use until you need to make a decision about which datastore to use.)

Overview

The application requires a few bits of functionality that can be separated into agnostic modules. The main functionality can be divided between the request handler and the worker. The request handler is the UI and frontend of the app, accepting user input. The worker takes care of actually posting things to Reddit.

Request Handler

Here’s an overview of the tasks the web worker performs in a typical workflow, where a user logs in and schedules a post:

  • Serve a rendered template routed by URL
  • Authenticate user with Reddit via Reddit’s OAuth support
  • Store user’s authentication credentials for future Reddit API calls in mongo
  • Store the desired post (including scheduled time) in mongo
  • Put the post into the worker’s queue for posting

The request handler is written using the Ring library, the de-facto standard for Clojure web apps, with Compojure handling routing, and Enlive taking care of template rendering. I also used Middleman to mock out the UI and generate HTML templates to be used with enlive.

There’s nothing too interesting here, just the Clojure equivalents of some really mundane tasks that any developer could relate to. The more interesting part is the post queue. One bonus aspect of using Clojure is that it’s both easy and performant to start your request-serving machinery from your application. This is because Clojure’s concurrency is thread-based (in-process), while Python, Ruby, PHP etc. use multi-process concurrency.

Worker

On the worker side, the situation is much simpler:

  • Take a post from the queue.
  • Fetch the latest version of that post from Mongo.
  • Check if it’s time to submit this post.
    • If not, put the post at the end of the queue.
    • If so, attempt to post. If the attempt fails, add the post back to the queue.
  • Repeat.

Here’s an annotated example of how this all looks:

(Syntax primer: myfunction(x, y) in Algol-derived languages is (myfunction x y) in Clojure.)

;; Define a post queue
(def upcoming-post-queue (lamina/queue))

(defn enqueue-post
  "Enqueue a post in the post queue."
  [post]
  (lamina/enqueue upcoming-post-queue post))

(defn time-to-post?
  "Is schedule < now?"
  [post]
  (>= (get post :schedule) (helpers/now)))

(defn process-post
  "Grab a post from the queue. If it's time to post it, post it. If not, requeue."
  []
  (let [post @upcoming-post-queue] ; Blocks until a post is in the queue
    (if (time-to-post? post)
      (reddit-api/submit post) ; If so, submit the post with the reddit-api module
      (enqueue-post post)))    ; Otherwise, add the post to the queue
  (Thread/sleep 1000.)) ; Sleep for a second


;; Called by main on startup
(defn start-worker []
  (doall (repeatedly process-post)))

If you can get over the parentheses, you can see how this process is simplified when compared to solutions in languages like Python (Celery) or Ruby (Resque). Both of these require you to run and manage another process (for another $30 per month, on Heroku), and neither has quite as simple of an API.

Of course, there is a downside – scaling this architecture across many servers would require that the post queue implement some sort of sharding. But this method would scale vertically on one server pretty far before it became necessary to distribute processing. After all, since the queue handles locking, there’s no reason but server specs that you couldn’t start as many worker threads as you like.

Conclusion

And that’s all there is to it! Using only these simple tools, RedditLater has been running happily and continuously for over a year (with the occasional bugfix). Of course, there are many other ways to design such an application, but I hope you’ve learned a bit today from the design and the tools I chose. For more on how Redditlater itself works, here’s some more detail.

Frequently Asked Questions (FAQs) about Clojure and Web App Development

Why is Clojure not as widely adopted as other mainstream languages?

Clojure, despite its powerful features, is not as widely adopted as other mainstream languages due to a few reasons. Firstly, it has a steep learning curve, especially for developers who are not familiar with functional programming. Secondly, Clojure’s syntax is quite different from other popular languages, which can be off-putting for some developers. Lastly, Clojure is a JVM language, which means it requires the Java runtime environment to run. This can be a barrier for some developers who prefer languages that can run natively.

Why would one choose Clojure over other languages?

Clojure offers several advantages over other languages. It is a functional programming language, which promotes immutability and high-level abstractions, making it easier to write clean, maintainable code. Clojure also has excellent support for concurrent programming, making it a great choice for developing multi-threaded applications. Additionally, being a JVM language, Clojure can leverage the vast ecosystem of Java libraries and frameworks.

What are some of the challenges faced when leaving Clojure?

Leaving Clojure can be challenging due to its unique features and paradigms. Developers might find it difficult to adapt to other languages that do not support functional programming or concurrency as well as Clojure does. Additionally, they might miss the power and flexibility of Clojure’s macro system, which allows for powerful code transformations at compile time.

Can software professionals explain why they prefer Clojure?

Many software professionals prefer Clojure for its simplicity, power, and flexibility. They appreciate its functional programming paradigm, which encourages immutability and high-level abstractions. They also value its support for concurrent programming, which is crucial for developing modern, multi-threaded applications. Lastly, they enjoy the ability to leverage the vast ecosystem of Java libraries and frameworks, thanks to Clojure being a JVM language.

What is the anatomy of a web app in Clojure?

A web app in Clojure typically consists of several components. The core of the app is written in Clojure, using libraries like Ring for handling HTTP requests and responses, Compojure for routing, and Hiccup for generating HTML. The app might also use a database, with libraries like Korma for database access. Additionally, the app might use ClojureScript, a variant of Clojure that compiles to JavaScript, for client-side code.

How does Clojure handle concurrency?

Clojure has excellent support for concurrent programming. It provides several concurrency primitives, like atoms, refs, and agents, which make it easy to write safe, concurrent code. Additionally, Clojure’s immutable data structures and functional programming paradigm further simplify concurrent programming by eliminating many common concurrency issues, like race conditions.

How does Clojure’s macro system work?

Clojure’s macro system allows for powerful code transformations at compile time. Macros in Clojure are functions that take code as input and produce code as output. This allows developers to extend the language with new constructs, eliminate boilerplate code, and perform optimizations that would be difficult or impossible in other languages.

What is the role of the Java Virtual Machine (JVM) in Clojure?

Clojure is a JVM language, which means it runs on the Java Virtual Machine. This allows Clojure to leverage the vast ecosystem of Java libraries and frameworks. It also means that Clojure programs can run on any platform that supports the JVM, providing excellent cross-platform compatibility.

How does Clojure support functional programming?

Clojure is a functional programming language, which means it encourages programming with functions and immutable data. Clojure provides a rich set of functional programming constructs, like higher-order functions, lazy sequences, and persistent data structures. These features make it easy to write clean, maintainable code that is easy to reason about.

What are some resources for learning Clojure?

There are many resources available for learning Clojure. The official Clojure website provides a wealth of information, including a getting started guide, API documentation, and a list of recommended books and tutorials. Additionally, there are many online tutorials, video courses, and books available that cover Clojure in depth.

Adam BardAdam Bard
View Author

Adam makes a lot of websites. He presently develops full-time for Tapstream, freelances occasionally and posts unsaleable articles on his blog.

clojureCloudHerokuredditredditlater
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week
Loading form