Comparing Rails: Exploring WebSockets in Phoenix

One of the biggest features in Rails 5 is WebSockets. Seems like this feature has been inspired in part by Phoenix, the Elixir web framework. In this article, we take a look at creating a real-time Twitter stream heatmap using Elixir and Phoenix.

The idea is to hook into the Twitter sample stream and plot tweets on a world map. I will not cover installation instructions, because the official Phoenix guides do a much better job.

This is what we are shooting for:

Setting Up the Phoenix Project

Let’s get started! First create a new Phoenix project:

% mix phoenix.new heetweet && cd heetweet

To make sure that everything was set up correctly, open http://localhost:4000 in your browser and you should be greeted by the default Phoenix page:

In order to communicate with the Twitter API, we will have to install ExTwitter, a Twitter client library.

We will specify this dependency in mix.exs, the equivalent of Rails’ Gemfile. Locate the deps function and add ExTwitter and OAuth, which the former relies on:

defmodule Heetweet.Mixfile do
  use Mix.Project

  # ...

  defp deps do
    [{:phoenix, "~> 1.1.4"},
     {:postgrex, ">= 0.0.0"},
     {:phoenix_ecto, "~> 2.0"},
     {:phoenix_html, "~> 2.4"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.9"},
     {:cowboy, "~> 1.0"},
     {:oauth, github: "tim/erlang-oauth"},
     {:extwitter, "~> 0.7.1"}]
  end

end

Erlang and Elixir can happily interop!

You might have notice that OAuth is an Erlang library, yet ExTwitter depends on it. Turns out, Elixir and Erlang can interop happily with each other. This means that libraries from both languages can be used in either language.

Next, you will need to tell Phoenix to start the ExTwitter application. An application in Elixir is essentially a self-contained library that can be used by other applications. In fact, many of the dependencies specified in the deps function are applications.

You might have heard of supervision trees in Elixir/Erlang. Applications can come packaged with their own supervisor trees that can self-heal when something goes wrong.

Take a look at the applications function specified in the same mix.exs file:

defmodule Heetweet.Mixfile do
  use Mix.Project

  # ...

  def application do
    [mod: {Heetweet, []},
     applications:[:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
                    :phoenix_ecto, :postgrex]]
  end

end

We need to add ExTwitter to the applications list:

def application do
  [mod: {Heetweet, []},
   applications:[:phoenix, :phoenix_html, :cowboy, :logger, :gettext,
                  :phoenix_ecto, :postgrex, :extwitter]]
end

Setting Up Twitter

Before we can use ExTwitter, we will need to head to https://apps.twitter.com/, sign in, and click “Create New App”.

Here’s what I filled in:

• Name: Heetweet
• Description: Heetweet displays tweets on a heat map
• Website: http://www.example.com

Note: You would have to supply your mobile phone number in the profile section under “Mobile”.

Once you have completed these steps, you would have access to this page that contains the consumer key and secret and access token and access token secret:

We will need to add these to config/config.exs and add these to the bottom of the file:

config :extwitter, :oauth, [
  consumer_key: System.get_env("TWITTER_CONSUMER_KEY"),
  consumer_secret: System.get_env("TWITTER_CONSUMER_SECRET"),
  access_token: System.get_env("TWITTER_ACCESS_TOKEN"),
  access_token_secret: System.get_env("TWITTER_ACCESS_SECRET")
]

Of course, you never want to store sensitive credentials in plain text and commit them to source control. Therefore, you will need to specify these in ~/.bashrc or something equivalent (depending on your shell):

export TWITTER_CONSUMER_KEY="xxxxxxxxxxxxxxxxxxxxxxxxx"
export TWITTER_CONSUMER_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export TWITTER_ACCESS_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export TWITTER_ACCESS_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Once done, make sure you either open a new terminal window or manually run source ~/.bashrc (or equivalent depending on your shell).

Taking Phoenix for a Spin

Let’s make sure we got everything set up correctly. We will open the Elixir REPL (read-eval-print-loop) and have it load our Phoenix application like so:

% iex -S mix phoenix.server

Let’s check that we configured ExTwitter correctly by running a query that is guaranteed to get results:

iex> ExTwitter.search("bieber", [count: 5]) \
|> Enum.map(fn(tweet) -> tweet.text end) \
|> Enum.join("\n---\n") \
|> IO.puts

Note: If you are copy pasting the above snippet, do it line by line instead of the entire chunk.

Let’s walk through what this code does. The first line searches for “bieber” with a limit of five results. The results come back in a list of ExTwitter.Model.Tweets. Since we are only interested in the content of the tweet, we run the Enum.map function to extract the text of each tweet. Each tweet is represented internally like so:

%ExTwitter.Model.Tweet{id: 724195286688645120,
  lang: "en",
  user: %ExTwitter.Model.User{contributors_enabled: false,
   id: 714513423896342530,
   screen_name: "asloveme4", lang: "pl", name: "ohai",
   url: "https://t.co/A8YqMw5YPh",
   description: "@justinbieber So take my hand and walk with me,\n Show me what to be,\n I need you to set me free",
   profile_image_url: "http://pbs.twimg.com/profile_images/722497035136413696/4xAxtZvN_normal.jpg",
   statuses_count: 1186, friends_count: 177, favourites_count: 80,
   show_all_inline_media: nil, utc_offset: nil, followers_count: 93,
   profile_banner_url: "https://pbs.twimg.com/profile_banners/714513423896342530/1460233025",
   default_profile_image: false, following: false, location: "Poland",
   protected: false,
   profile_background_image_url_https: "https://abs.twimg.com/images/themes/theme1/bg.png",
  quoted_status: nil,
  text: "RT @hot_or_not_pll: #43 Justin Bieber\nRT- hot 🔥🔥\nFav- not 👎👎 https://t.co/3IpoxYL6qZ",
  entities: %{hashtags: [],
    media: [%{display_url: "pic.twitter.com/3IpoxYL6qZ",
       expanded_url: "http://twitter.com/hot_or_not_pll/status/723558996532187137/photo/1",
    user_mentions: [%{id: 723238093705367552, id_str: "723238093705367552",
, favorite_count: 28, favorited: false, geo: nil}

Notice that the Tweet model has a corresponding %ExTwitter.Model.User model. That will be useful for us when trying to determine the user’s country.

Setting Up the Map

We will be using the Leaflet, a Javascript library that let’s us have interactive maps. Open web/templates/layout/app.html.eex. We are going to clear out most of the HTML and add the CSS and Javascript needed for Leaflet. Here’s the result:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta name="description" content="">
    <meta name="author" content="">

    <title>Hello Heetweet!</title>
    <link rel="stylesheet" href="<%= static_path(@conn, "/css/app.css") %>">
    <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet/v0.7.7/leaflet.css" />
  </head>

  <body>
    <%= render @view_module, @view_template, assigns %>
    <script src="<%= static_path(@conn, "/js/app.js") %>"></script>
  </body>
</html>

Setting Up the Heatmap Library

Thankfully, Leaflet comes with a handy heatmap library appropriately called Leaflet.heat.

I simply downloaded the library and placed the file into web/static/vendor.

WebSockets

Let’s get to the WebSocket goodness! First, we need to enable the client-side part of WebSockets in Phoenix. Open app.js, and uncomment the following line:

import socket from "./socket"

Now head to web/static/js/socket.js. Navigate to the bottom of the file:

let channel = socket.channel("tweets:lobby", {})
channel.join()
  .receive("ok", resp => { console.log("Joined successfully", resp) })
  .receive("error", resp => { console.log("Unable to join", resp) })

We basically specify that we want the socket to connect to the “tweets:lobby” topic via a channel. For now, we will leave the console.logs intact.

Setting Up the Channels Backend

Let’s create a Phoenix channel. Phoenix comes with a handy generator just for that:

% mix phoenix.gen.channel Room tweets
* creating web/channels/room_channel.ex
* creating test/channels/room_channel_test.exs

Add the channel to your `web/channels/user_socket.ex` handler, for example:

    channel "tweets:lobby", Heetweet.RoomChannel

Let’s follow the advice of the generator and add the channel to user_socket.ex:

defmodule Heetweet.UserSocket do
  use Phoenix.Socket

  channel "tweets:lobby", Heetweet.RoomChannel

  # ...
end

When you head to your Phoenix app, you should be able to see

Joined successfully Object {}

in the browser console. If you have reached this stage, you have successfully wired up WebSockets in Phoenix. Now on to the fun stuff!

Wiring Everything Together

Now that the browser can successfully connect to the backend, what next? Well, we need to signal to our backend that it can start sending us data. To do that, we push start_stream into the channel:

channel.join()
    .receive("ok", resp => {
        console.log("Joined successfully", resp)
        channel.push("start_stream", {})   // <---
    })
    .receive("error", resp => { console.log("Unable to join", resp) })

We then need to handle the start_stream message in the backend. Head to web/channels/room_channel.ex. Recall that we have previously created this using the mix phoenix.channel generator. Here is the function where all the magic happens:

defmodule Heetweet.RoomChannel do
  use Heetweet.Web, :channel

  # ...

 def handle_in("start_stream", payload, socket) do
    stream = ExTwitter.stream_filter(locations: ['-180,-90,180,90'])
    |> Stream.map(fn x -> x.coordinates end)

    # Twitter gives us coordinates in reverse order (long first, then lat)
    for %{coordinates: [lng, lat]} <- stream do
      push socket, "new_tweet", %{lat: lat, lng: lng}
    end

    {:reply, {:ok, payload}, socket}
  end

  # ...
end

We are doing quite a bit here, so let’s unpack things a bit.

First, we have to get a stream of tweets that contain coordinates. We want tweets that span the entire planet, and the ExTwitter library has a function for that:

ExTwitter.stream_filter(locations: ['-180,-90,180,90'])

Here’s an Elixir tidbit: The above function returns a stream. In Ruby, that’s similar to a lazy enumerable. It doesn’t emit any values until we explicitly request for it. In fact, we can chain other stream operations to manipulate each emitted value, and it still will not return any value. Therefore, this is how we produce a stream of coordinates from the tweets:

stream = ExTwitter.stream_filter(locations: ['-180,-90,180,90'])
         |> Stream.map(fn x -> x.coordinates end)

Finally, we can iterate through the (potentially infinite) stream of coordinates with a for comprehension. While the for comprehension here looks like a normal for loop, there’s more than meets the eye. The Elixir for comprehension helpfully discards values that are null. This is extremely useful since some tweets do not have coordinates attached. Within the body of the for comprehension we can push out the coordinates to the connected socket:

for %{coordinates: [lng, lat]} <- stream do
  push socket, "new_tweet", %{lat: lat, lng: lng}
end

Now back to the client side. Open web/static/js/socket.js. We first set up the map and add an empty heat map layer for now:

var tiles = L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', {
    attribution: '&copy; <a href="http://osm.org/copyright">OpenStreetMap</a> contributors',
}).addTo(map);

var heat = L.heatLayer([]).addTo(map);

Next, we need to handle new_tweet and its payload (the coordinates):

channel.on("new_tweet", payload => {
    // artificially add more points to get the heatmap effect
    for (var i = 0; i < 100; i++) {
        heat.addLatLng([payload.lat, payload.lng])
    }
})

Notice that we have a loop that adds coordinates to the heatmap layer a hundred times for a given coordinate. The reason for that is that I’m too lazy for the nice heatmap effect to appear and in the interests of instant gratification I decided to speed things up a little.

That’s it! Here’s Heetweet in its full glory:

Summary

I hope you enjoyed this article and more importantly, learned something new! I encourage you to give Phoenix a try, especially if you have been itching to experiment with WebSockets.