How to Build a Basic Twitter Analytics App with RestDB

This article was sponsored by RestDB. Thank you for supporting the partners who make SitePoint possible.

Twitter Analytics is a useful tool — it allows you to keep an eye on what works and what doesn’t, and experiment your way into an enviable follower count.

But what if you want to use data from your own account to perform some analysis that’s more specific to your particular needs? It’s not particularly easy — but it is possible. We’ll be using RestDB to help us fetch and store data, and take a look at what’s going on with our followers.

Do you often wonder why some accounts seem to follow you only to unfollow you moments (or days) later? It’s probably not something you said — they’re just follower farming.

Follower farming is a known social media hack taking advantage of people who “#followback” as soon as someone follows them. The big brands, celebs, and wannabe celebs take advantage of this, as it keeps their followers count high but following count low, in turn making them look popular.

In this post, we’ll build an app which lets you log in via Twitter, grabs your followers, and compares the last fetched follower list with a refreshed list in order to identify the new unfollowers and calculate the duration of their follow, potentially auto-identifying the farmers.

Bootstrapping

As usual, we’ll be using Homestead Improved for a high quality local environment setup. Feel free to use your own setup instead if you’ve got one you feel comfortable in.

git clone https://github.com/swader/homestead_improved hi_followfarmers
cd hi_followfarmers
bin/folderfix.sh
vagrant up; vagrant ssh

Once the VM has been provisioned and we find ourselves inside it, let’s bootstrap a Laravel app.

composer create-project --prefer-dist laravel/laravel Code/Project
cd Code/Project

Logging in with Twitter

To make logging in with Twitter possible, we’ll use the Socialite package.

composer require laravel/socialite

As per instructions, we should also register it in config/app.php:

'providers' => [
    // Other service providers...

    Laravel\Socialite\SocialiteServiceProvider::class,
],

'Socialite' => Laravel\Socialite\Facades\Socialite::class,

Finally, we need to register a new Twitter app at http://apps.twitter.com/app/new…

… and add the secret credentials into config/services.php:

    'twitter' => [
        'client_id' => env('TWITTER_CLIENT_ID'),
        'client_secret' => env('TWITTER_CLIENT_SECRET'),
        'redirect' => env('TWITTER_CALLBACK_URL'),
    ],

Naturally, we need to add these environment variables into the .env file in the root of the project:

TWITTER_CLIENT_ID=keykeykeykeykeykeykeykeykey
TWITTER_CLIENT_SECRET=secretsecretsecret
TWITTER_CALLBACK_URL=http://homestead.app/auth/twitter/callback

We need to add some Login routes into routes/web.php next:

Route::get('auth/twitter', 'Auth\LoginController@redirectToProvider');
Route::get('auth/twitter/callback', 'Auth\LoginController@handleProviderCallback');

Finally, let’s add the methods these routes refer to into the LoginController class inside app/Http/Controllers/Auth:

    /**
     * Redirect the user to the GitHub authentication page.
     *
     * @return Response
     */
    public function redirectToProvider()
    {
        return Socialite::driver('twitter')->redirect();
    }

    /**
     * Obtain the user information from GitHub.
     *
     * @return Response
     */
    public function handleProviderCallback()
    {
        $user = Socialite::driver('twitter')->user();

        dd($user);
    }

The dd($user); is there to easily test if the authentication went well, and sure enough, if you visit /auth/twitter, you should be able to authorize the app and see the basic information about your account on screen:

Follower Lists

There are many ways of getting an account’s follower list, and none of them pleasant.

Twitter Still Hates Developers

Ever since Twitter’s Great War on Developers (spoiler: very little has changed since that article came out), it’s been an outright nightmare to fetch full lists of people’s followers. In fact, the API rate limits are so low that people have resorted to third party data aggregators for actually buying that data, or even scraping the followers page. We’ll go the “white hat” route and suffer through their API, but if you have other means of getting followers, feel free to use that instead of the method outlined below.

The Twitter API offers the /followers/list endpoint, but as that one only returns 20 followers per call at most, and only allows 15 requests per 15 minutes, we would be able to, at most, extract 1200 followers per hour – unacceptable. Instead, we’ll use the followers/ids endpoint to fetch 5000 IDs at a time. This is subject to the same limit of 15 calls per 15 minutes, but gives us much more breathing room.

It’s important to keep in mind that ID != Twitter handle. IDs are numeric values representing a unique account across time, even across different handles. So for each unfollower’s ID, we’ll have to make an additional API call to find out who they were (the Users Lookup Bulk API will come in handy).

Basic API Communication

Socialite is only useful for logging in. Actually communicating with the API is less straightforward. Given that Laravel comes with Guzzle pre-installed, installing Guzzle’s Oauth Subscriber (which lets us use Guzzle with the Oauth1 protocol) is the simplest solution:

composer require guzzlehttp/oauth-subscriber

Once that’s in there, we can update our LoginController::handleProviderCallback method to test things out:

    public function handleProviderCallback()
    {
        $user = Socialite::driver('twitter')->user();

        $stack = HandlerStack::create();

        $middleware = new Oauth1([
            'consumer_key' => getenv('TWITTER_CLIENT_ID'),
            'consumer_secret' => getenv('TWITTER_CLIENT_SECRET'),
            'token' => $user->token,
            'token_secret' => $user->tokenSecret
        ]);

        $stack->push($middleware);

        $client = new Client([
            'base_uri' => 'https://api.twitter.com/1.1/',
            'handler' => $stack,
            'auth' => 'oauth'
        ]);

        $response = $client->get('followers/ids.json', [
            'query' => [
                'cursor' => '-1',
                'screen_name' => $user->nickname,
                'count' => 5000
            ]
        ]);

        dd($response->getBody()->getContents());
    }

In the above code, we first create a middleware stack which will chew through the request, pull it through all the middlewares, and output the final version. We can push other middlewares into this stack, but for now, we only need the Oauth1 one.

Next, we create the Oauth1 middleware and pass in the required parameters. The first two we’ve already got – they’re the keys we defined in .env previously. The last two we got from the authenticated Twitter user instance.

We then push the middleware into the stack, and attach the stack onto the Guzzle client. In layman’s terms, this means “when this client does requests, pull the requests through all the middlewares in the stack before sending them to their final destination”. We also tell the client to always authenticate with oauth.

Finally, we make the GET call to the API endpoint with the required query params: the page to start on (-1 is the first page), the user for whom to pull followers, and how many followers to pull. In the end, we die this output onto the screen to see if we’re getting what we need. Sure enough, here’s 5000 of the most recent followers for my account:

Now that we know our API calls are passing and we can talk to Twitter, it’s time for some loops to get the full list for the current user.

The PHP Side – Getting All Followers

Since there are 15 calls per 15 minutes allowed via the API, let’s limit the account size to 70k followers for now for simplicity.

        $user = Socialite::driver('twitter')->user();

        if ($user->user['followers_count'] > 70000) {
            return view(
                'home.index',
                ['message' => 'Sorry, we currently only support accounts with up to 70k followers']
            );
        }

Note: home.index is an arbitrary view file I made just for this example, containing a single directive: {{ $message }}.

Then, let’s iterate through the next_cursor_string value returned by the API, and paginate through other IDs.

Much numbers, very follow, wow.

With some luck, this should execute very quickly – depending on Twitter’s API responsiveness.

Everyone with up to 70k followers can now get a full list of followers generated upon authorization.

If we needed to support bigger accounts, it would be relatively simple to make it repeat the process every 15 minutes (after the API limit resets) for every 75k followers, and stitch the results together. Of course, someone is almost guaranteed to follow/unfollow in that window given the number of followers, so it would be very hard to stay accurate. In those cases, it’s easier to focus on the last 75k followers and only analyze those (the API auto-orders by last-followed), or to find another method of reliably fetching followers, bypassing the API.

Cleaning Up

It’s a bit awkward to have this logic in the LoginController, so let’s move this into a separate service. I created app/Services/Followers/Followers.php for this example, with the following contents:

<?php


namespace App\Services\Followers;

use GuzzleHttp\Client;
use GuzzleHttp\HandlerStack;
use GuzzleHttp\Subscriber\Oauth\Oauth1;

class Followers
{

    /** @var string */
    protected $token;

    /** @var string */
    protected $tokenSecret;

    /** @var string */
    protected $nickname;

    /** @var Client */
    protected $client;

    public function __construct(string $token, string $tokenSecret, string $nickname)
    {
        $this->token = $token;
        $this->tokenSecret = $tokenSecret;
        $this->nickname = $nickname;

        $stack = HandlerStack::create();

        $middleware = new Oauth1(
            [
                'consumer_key' => getenv('TWITTER_CLIENT_ID'),
                'consumer_secret' => getenv('TWITTER_CLIENT_SECRET'),
                'token' => $this->token,
                'token_secret' => $this->tokenSecret,
            ]
        );

        $stack->push($middleware);

        $this->client = new Client(
            [
                'base_uri' => 'https://api.twitter.com/1.1/',
                'handler' => $stack,
                'auth' => 'oauth',
            ]
        );
    }

    public function getClient()
    {
        return $this->client;
    }

    /**
     * Returns an array of follower IDs for a given optional nickname.
     *
     * If no custom nickname is provided, the one used during the construction
     * of this service is used, usually defaulting to the same user authing
     * the application.
     *
     * @param string|null $nickname
     * @return array
     */
    public function getFollowerIds(string $nickname = null)
    {
        $nickname = $nickname ?? $this->nickname;

        $response = $this->client->get(
            'followers/ids.json', [
                'query' => [
                    'cursor' => '-1',
                    'screen_name' => $nickname,
                    'count' => 5000,
                ],
            ]
        );

        $data = json_decode($response->getBody()->getContents());
        $ids = $data->ids;

        while ($data->next_cursor_str !== "0") {

            $response = $this->client->get(
                'followers/ids.json', [
                    'query' => [
                        'cursor' => $data->next_cursor_str,
                        'screen_name' => $nickname,
                        'count' => 5000,
                    ],
                ]
            );
            $data = json_decode($response->getBody()->getContents());
            $ids = array_merge($ids, $data->ids);
        }

        return $ids;
    }

}

We can then clean up the LoginController’s handleProviderCallback method:

    public function handleProviderCallback()
    {
        $user = Socialite::driver('twitter')->user();

        if ($user->user['followers_count'] > 70000) {
            return view(
                'home.index',
                ['message' => 'Sorry, we currently only support accounts with up to 70k followers']
            );
        }

        $flwrs = new Followers(
            $user->token, $user->tokenSecret, $user->nickname
        );
        dd($flwrs->getFollowerIds());
    }

It’s still the wrong method to be doing this, so let’s further improve things. To keep a user logged in, let’s save the token, secret, and nickname into the session.

    /**
     * Get and store token data for authorized user.
     *
     * @param Request $request
     * @return Response
     */
    public function handleProviderCallback(Request $request)
    {
        $user = Socialite::driver('twitter')->user();

        if ($user->user['followers_count'] > 70000) {
            return view(
                'home.index',
                ['message' => 'Sorry, we currently only support accounts with up to 70k followers']
            );
        }

        $request->session()->put('twitter_token', $user->token);
        $request->session()->put('twitter_secret', $user->tokenSecret);
        $request->session()->put('twitter_nickname', $user->nickname);
        $request->session()->put('twitter_id', $user->id);

        return redirect('/');
    }

We save all the information into the session, making the user effectively logged in to our application, and then we redirect to the home page.

Let’s create a new controller now, and give it a simple method to use:

artisan make:controller HomeController

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;

class HomeController extends Controller
{
    public function index(Request $request)
    {
        $nick = $request->session()->get('twitter_nickname');
        if (!$nick) {
            return view('home.loggedout');
        }

        return view('home.index', $request->session()->all());
    }
}

Simple, right? The views are simple, too:

{{--index.blade.php--}}
<h1>FollowerFarmers</h1>

<h2>Hello, {{ $twitter_nickname }}! Not you? <a href="/logout">Log out!</a></h2>

<p>I bet you'd like to see your follower stats, wouldn't you?</p>

{{--loggedout.blade.php--}}
<h1>FollowerFarmers</h1>

<h2>Hello, stranger!</h2>

<p>You're currently logged out. How about you <a href="/auth/twitter">log in with Twitter </a> to get started?</p>

We’ll need to add some routes to routes/web.php, too:

Route::get('/', 'HomeController@index');
Route::get('/logout', 'Auth\LoginController@logout');

With this, we can check if we’re logged in, and we can easily log out.

Note that for security, the logout route should only accept POST requests with CSRF tokens – for simplicity during development, we’re taking the GET approach and revamping it later.

Admittedly, it’s not the prettiest thing to look at, but we’re building a demo here – the real thing can get visually polished once the logic is done.

Registering a Service Provider

It’s common practice to register a service provider for easier access later on, so let’s do that. Our service can’t be instantiated without the token and secret (i.e. before the user logs in with Twitter) so we’ll need to make it deferred – in other words, it’ll only get created when needed, and we’ll make sure we don’t need it until we have those values.

artisan make:provider FollowerServiceProvider

<?php

namespace App\Providers;

use App\Services\Followers\Followers;
use Illuminate\Support\ServiceProvider;

class FollowerServiceProvider extends ServiceProvider
{

    protected $defer = true;

    public function register()
    {
        $this->app->singleton(
            Followers::class, function ($app) {
            return new Followers(
                session('twitter_token'), session('twitter_secret'),
                session('twitter_nickname')
            );
        }
        );
    }

    public function provides()
    {
        return [Followers::class];
    }
}

If we put a simple count echo into our logged in view:

{{ count($ids) }}

… and modify the HomeController to now use this ServiceProvider:

...

        return view(
            'home.index', array_merge(
                $request->session()->all(),
                ['ids'=> resolve(Followers::class)->getFollowerIds()]
            )
        );

… and then we test, sure enough, it works.

Database

Now that we have a neat service to extract follower lists with, we should probably save them somewhere. We could save this into a local MySQL database, or even a flat file, but for performance and portability, I went with something different this time: RestDB.

RestDB is a plug and play hosted database service that’s easy to configure and use, freeing up your choices of hosting platform. By not needing a database that writes to a local filesystem, you can easily push an app like the one we’re building to Google Cloud Engine or Heroku. With the help of its templates, you can instantly set up a blog, a landing page, a web form, a log analyzer, even a mailing system – heck, the service even supports MarkDown for inline field editing, letting you practically have a MarkDown-based blog right there on their service.

RestDB has a free tier, and the first month is virtually limitless so you can thoroughly test it. The database I’m developing this on is on a Basic plan (courtesy of the RestDB team).

Setting up RestDB

Unlike with other database services, with RestDB it’s important to consider record number limits. The Basic plan offers 10000 records, which would be quickly exhausted if we saved the follower of each logged in user as a separate entry, or even a list of followers for each user as a separate entry per 15 minute timeframe. That’s why I chose the following plan:

• each new user will be a record in the accounts collection.
• each new follower list will be a record in the follower-lists collection and will be a child record of accounts.
• at a maximum rate of every 15 minutes (or more if user takes longer to come back and log into the app), a new list will be generated, compared to the last one, and a new list along with a diff towards the last one will be saved.
• every user will be able to keep at most 100 histories

That said, let’s create the new follower-lists collection as per the quick-start docs. Once the collection has been created, let’s add some fields:

• a required followers text field. The text field supports regular expression validations, and since we’re going to use a comma separated list to store the follower IDs, we can apply a regex like this one to make sure the data is always valid: ^(\d+,\s?)*(\d+)$. This will match only lines with comma separated digits, but without a trailing comma. You can see it in action here.
• a diff_new field of text type, which will contain a list of new followers since the last entry. The same regex restriction as for followers will apply, only updated to be optional, becausge sometimes there will be no difference compared to the last entry: (^(\d+,\s?)*(\d+)$)?.
• a diff_gone field of text type, which will contain a list of unfollowers since the last entry. The same regex restriction as for diff_new will apply.

Our collection should look like this:

Now let’s create the parent collection: accounts.

Note: you may be wondering why we don’t just use the built-in users collection. This is because that collection is only meant for authenticating Auth0 users. The fields that are in there would be useful for us, but as per the docs, we have no write access to that database, and we need it. So why not just go with Auth0 for logins and RestDB for data? Feel free take that approach – I personally feel like depending on one third party service for a crucial part of my app is enough, two would be too much for me.

The fields we need are:

• twitter_id, the Twitter account ID of the user. Required number.
• settings, a required JSON field. This will hold all the user’s account-specific settings, like refresh interval, emailing frequency, etc.

After adding these, let’s add a new follower_lists field, and define it as a child relation to our follower-lists collection. Under Properties, we should pick “child of…”. The naming is a little confusing – despite the option saying “child of follower-lists”, it is follower-lists who is the child.

You may have noticed we haven’t used timestamp fields anywhere, like created_at. That’s because RestDB automatically creates them for every collection, along with some other fields. To inspect those System fields, click the “Show System Fields” option in the top right corner of each collection’s Settings table:

Getting these fields in a payload when querying the database requires us to use the ?metafields=true param in the API URLs.

We are now ready to start combining the PHP and RestDB side.

Saving to and Reading from RestDB

To be able to interact with RestDB, we need an API key. We can get it by following instructions here. All options should be left at the default value, with all REST methods enabled. The key should then be saved into .env:

RESTDB_KEY=keykeykey

The idea for accounts is as follows:

• when the user first authorizes Twitter, the app will read the accounts collection for the Twitter ID provided, and if it doesn’t exist, it will write a new entry.
• the user is then redirected to the welcome screen, which will contain a message confirming account creation if one was created, and offer to redirect to the /dashboard.

Let’s first make a RestDB service for talking to the database.

<?php
// Services/Followers/RestDB.php

namespace App\Services\Followers;

use GuzzleHttp\Client;
use GuzzleHttp\ClientInterface;
use Psr\Http\Message\ResponseInterface;

class RestDB
{
    /** @var ClientInterface */
    protected $client;

    /**
     * Sets the Guzzle client to be used
     *
     * @param ClientInterface $client
     * @return $this
     */
    public function setClient(ClientInterface $client)
    {
        $this->client = $client;
        return $this;
    }

    /**
     * @return ClientInterface
     */
    public function getClient()
    {
        return $this->client;
    }

    /**
     * Configures a default Guzzle client so it doesn't need to be injected
     * @return $this
     */
    public function setDefaultClient()
    {
        $client = new Client([
            'base_uri' => 'https://followerfarmers-00df.restdb.io/rest/',
            'headers' => [
                'x-apikey' => getenv('RESTDB_KEY'),
                'content-type' => 'application/json'
            ]
        ]);
        $this->client = $client;
        return $this;
    }

    /**
     * Returns user's account entry if it exists. Caches result for 5 minutes
     * unless told to be `$fresh`.
     *
     * @param int $twitter_id
     * @param bool $fresh
     * @return bool|\stdClass
     */
    public function userAccount(int $twitter_id, bool $fresh = false)
    {
        /** @var ResponseInterface $request */
        $response = $this->client->get(
            'accounts', [
                'body' => '{"twitter_id": ' . $twitter_id . ', "max": 1}',
                'query' => ['metafields' => true],
                'headers' => ['cache-control' => $fresh ? 'no-cache' : 'max-age:300'],
            ]
        );

        $bodyString = json_decode($response->getBody()->getContents());

        if (empty($bodyString)) {
            return false;
        }

        return $bodyString[0];
    }


    /**
     * Creates a new account in RestDB.
     *
     * @param array $user
     * @return bool
     */
    public function createUserAccount(array $user)
    {
        /** @var ResponseInterface $request */
        $response = $this->client->post('accounts', [
            'body' => json_encode([
                'twitter_id' => $user['id'],
                'settings' => array_except($user, 'id')
            ]),
            'headers' => ['cache-control' => 'no-cache']
        ]);

        return $response->getStatusCode() === 201;

    }
}

In this service, we define ways to set the Guzzle client to be used, along with a shortcut method to define a default one. This default one also includes the default authorization header, and sets content type as JSON which is what we’re communicating with. We also demonstrate basic reading and writing from and to RestDB.

The userAccount method directly searches for a Twitter ID in the accountsrecords, and returns a record if found, or false if not. Note the use of the metafields query param – this lets us fetch the _created and other system fields. Notice also that we cache the result for 5 minutes unless the $fresh param is passed in, because the user info will rarely change and we might need it multiple times during a session. The createUserAccount method takes an array of user data (the most important of which is the id key) and creates the account. Note that we’re looking for status 201 which means CREATED.

Let’s also make a ServiceProvider and register the service as a singleton.

artisan make:provider RestdbServiceProvider

<?php

namespace App\Providers;

use App\Services\Followers\RestDB;
use Illuminate\Support\ServiceProvider;

class RestdbServiceProvider extends ServiceProvider
{
    /**
     * Register the application services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->singleton(
            'restdb', function ($app) {
            $r = new RestDB();
            $r->setDefaultClient();

            return $r;
        }
        );
    }
}

Finally, let’s update our LoginController.

      // ...
        $request->session()->put('twitter_id', $user->id);

        $rest = resolve('restdb');
        if (!$rest->userAccount($user->id)) {
            if ($rest->createUserAccount(
                [
                    'token' => $user->token,
                    'secret' => $user->tokenSecret,
                    'nickname' => $user->nickname,
                    'id' => $user->id,
                ]
            )) {
                $request->session()->flash(
                    'info', 'Your account has been created! Welcome!'
                );
            } else {
                $request->session()->flash(
                    'error', 'Failed to create your account :('
                );
            }

        }

        // ...

        return redirect('/');

In the LoginController‘s handleProviderCallback method, we first grab (resolve) the service, use it to check if the user has an account, create it if not, and flash the message to session if either successful or not.

Let’s put these flash messages into the view:

{{--index.blade.php--}}
@isset($info)
    <p>{{ $info }}</p>
@endisset

@isset($error)
    <p>{{ $error }}</p>
@endisset
...

If we test this out, sure enough, our new record is created:

Now let’s offer a /dashboard. The idea is:

• when a user logs in, they’ll be presented with a “Dashboard” link.
• clicking this link will, in order:
  • grab their latest follower-lists entry from RestDB
  • if more than 15 minutes have elapsed since the last entry was created, or the user doesn’t have an entry at all, a new list of followers will be fetched. The new list will be saved. If it wasn’t the first entry, a diff is generated for new followers and unfollowers.
  • if the user has refreshed in the last 15 minutes, they will simply be redirected to the dashboard
• when the user accesses this dashboard, all their follower-lists RestDB entries are fetched
• the applications goes through all the diff entries in the records, and generates reports for unfollowers, displaying information on how long they had been following the user before leaving.
• once these IDs for the report have been fetched, their information is fetched via the /users/lookup endpoint to grab their avatars and Twitter handles.
• if an account had been following for a day or less, it is flagged with a red color, meaning a high certainty of follower farming. 1 – 5 days is orange, 5 – 10 days is yellow, and others are neutral.

Let’s update the index view first, and add a new route.

// routes/web.php
Route::get('/dashboard', 'HomeController@dashboard');

{{--index.blade.php--}}

...

<p>I bet you'd like to see your follower stats, wouldn't you?</p>

Go to <a href="/dashboard">dashboard</a>.

We need a way to fetch the last follower_lists entry of a user. Thus, in the RestDB service, we can add the following method:

    /**
     * Get the last follower_lists entry of the user in question, or false if
     * none exists.
     *
     * @param int $twitter_id
     * @return bool|\stdClass
     */
    public function getUsersLastEntry(int $twitter_id)
    {
        $id = $this->userAccount($twitter_id)->_id;
        /** @var ResponseInterface $request */
        $response = $this->client->get(
            'accounts/' . $id . '/follower_lists', [
                'query' => [
                    'metafields' => true,
                    'sort' => '_id',
                    'dir' => -1,
                    'max' => 1,
                ],
                'headers' => ['cache-control' => 'no-cache'],
            ]
        );

        $bodyString = json_decode($response->getBody()->getContents());

        return !empty($bodyString) ? $bodyString[0] : false;

    }

We either return false, or the last entry. Notice that we’re sorting by the _id metafield, from newest to oldest (dir=-1), and fetching a maximum of 1 entry. These params are all explained here.

Now let’s turn our attention to the dashboardmethod in HomeController:

    public function dashboard(Request $request)
    {
        $twitter_id = $request->session()->get('twitter_id', 0);
        if (!$twitter_id) {
            return redirect('/');
        }

        /** @var RestDB $rest */
        $rest = resolve('restdb');
        $lastEntry = $rest->getUsersLastEntry($twitter_id);

        if ($lastEntry) {
            $created = Carbon::createFromTimestamp(
                strtotime($lastEntry->_created)
            );
            $diff = $created->diffInMinutes(Carbon::now());
        }

        if ((isset($diff) && $diff > 14) || !$lastEntry) {
            $followerIds = resolve(Followers::class)->getFollowerIds();         $rest->addFollowerList($followerIds, $lastEntry, $twitter_id);
        }

        dd("Let's show all previous lists");

    }

Ok, so what’s going on here? First, we do a primitive check if the user is still logged in – the twitter_id has to be in the session. If not, we redirect to homepage. Then, we fetch the Rest service, get the account’s last follower-lists entry (which is either an object or false) and then if it exists, we calculate how old it is. If it’s more than 14 minutes, or if the entry doesn’t exist at all (meaning it’s the very first one for that account), we fetch a new list of followers and save it. How do we save it? By adding a new addFollowerList method to the Rest service.

    /**
     * Adds a new follower_lists entry to an account entry
     *
     * @param array $followerIds
     * @param \stdClass|bool $lastEntry
     * @param int $twitter_id
     * @return bool
     * @internal param array $newEntry
     */
    public function addFollowerList(
        array $followerIds,
        $lastEntry,
        int $twitter_id
    ) {
        $account = $this->userAccount($twitter_id);

        $newEntry = ['followers' => implode(', ', $followerIds)];

        if ($lastEntry !== false) {
            $lastFollowers = array_map(
                function ($el) {
                    return (int)trim($el);
                }, explode(',', $lastEntry->followers)
            );

            sort($lastFollowers);
            sort($followerIds);

            $newEntry['diff_gone'] = implode(
                ', ', array_diff($lastFollowers, $followerIds)
            );
            $newEntry['diff_new'] = implode(
                ', ', array_diff($followerIds, $lastFollowers)
            );

        }

        try {
            /** @var ResponseInterface $request */
            $response = $this->client->post(
                'accounts/' . $account->_id . '/follower_lists', [
                    'body' => json_encode($newEntry),
                    'headers' => ['cache-control' => 'no-cache'],
                ]
            );
        } catch (ClientException $e) {
            // Log the exception message or something
        }

        return $response->getStatusCode() === 201;
    }

This one first grabs the user account to find the ID of the account record in RestDB. Then, it initiates the $newEntry variable with a properly formatted (imploded) string of current follower IDs. Next, if there was a last entry, we:

• get those IDs into a proper array by exploding the string and cleaning whitespace.
• sort both current and past follower arrays for more effective diffing.
• get the differences and add them to $newEntry.

We then save the entry, by targeting the specific account entry with the previously fetched ID, and continuing on into the sub-collection of follower_lists.

To test this, we can fake some data. Let’s alter the $followerIds part of HomeController::dashboard to this:

            $count = rand(50, 75);
            $followerIds = [];
            while ($count--) {
                $flw = rand(1, 100);
                if (in_array($flw, $followerIds)) $count++; else
                $followerIds[] = $flw;
            }

This will generate 50-75 random numbers ranging from 1 to 100. Good enough for us to get some diffs. If we hit the url /dashboard while logged in now, we should get our initial entry.

If we remove the 15 minute limit from the if block and refresh two more times, we’ve generated 3 entries total, with good looking diffs:

It’s time for the final feature. Let’s analyze the entries, and identify some follower farmers.

Final Stretch

Because it contextually makes sense, we’ll put this logic into the Followers service. Let’s create an analyzeUnfollowers method. It will accept an arbitrary number of entries, and do its logic in a loop on all of them. Then, if we later want to provide a quicker way of just checking the last bit of information since the last login session, we can simply pass the two last entries instead of all of them, and the logic remains the same.

    public function analyzeUnfollowers(array $entries)
    {
        ...
    }

To identify unfollowers, we look at the most recent diff_gone, for all who are gone since the last time we checked our follower list, and then find them in the diff_new arrays of previous entries. This then lets us find out how long they had been following us before leaving. While using the entries, we also need to turn the diff_gone and diff_new entries into arrays, for easy seeking.

   /**
     * Accepts an array of entries (stdObjects) ordered from newest to oldest.
     * The objects must have the properties: diff_gone, diff_new, and followers,
     * all of which are comma delimited strings of integers, or arrays of integers.
     * The property `_created` is also essential.
     *
     * @param array $entries
     * @return array
     */
    public function analyzeUnfollowers(array $entries)
    {
        $periods = [];
        $entries = array_map(
            function ($entry) {
                if (is_string($entry->diff_gone)) {
                    $entry->diff_gone = $this->intArray($entry->diff_gone);
                }
                if (is_string($entry->diff_new)) {
                    $entry->diff_new = $this->intArray($entry->diff_new);
                }

                return $entry;
            }, $entries
        );

        $latest = array_shift($entries);

        for ($i = 0; $i < count($entries); $i++) {
            $cur = $entries[$i];
            $curlast = array_last($entries) === $cur;
            if ($curlast) {
                $matches = $latest->diff_gone;
            } else {
                $matches = array_intersect(
                    $cur->diff_new, $latest->diff_gone
                );
            }
            if ($matches) {
                $periods[] = [
                    'matches' => array_values($matches),
                    'from' => (!$curlast) ? Carbon::createFromTimestamp(strtotime($cur->_created)) : 'forever',
                    'to' => Carbon::createFromTimestamp(strtotime($latest->_created))
                ];
            }
        }

        return $periods;
    }

    /**
     * Turns a string of comma separated values, spaces or no, into an array of integers
     *
     * @param string $string
     * @return array
     */
    protected function intArray(string $string): array
    {
        return array_map(
            function ($el) {
                return (int)trim($el);
            }, explode(',', $string)
        );
    }

Of course, we need a way to fetch all the follower list entries. We put the getUserEntries method into the Rest service:

    /**
     * Gets a twitter ID's full list of follower list entries
     *
     * @param int $twitter_id
     * @return array
     */
    public function getUserEntries(int $twitter_id): array
    {
        $id = $this->userAccount($twitter_id)->_id;
        /** @var ResponseInterface $request */
        $response = $this->client->get(
            'accounts/' . $id . '/follower_lists', [
                'query' => [
                    'metafields' => true,
                    'sort' => '_id',
                    'dir' => -1,
                    'max' => 100,
                ],
                'headers' => ['cache-control' => 'no-cache'],
            ]
        );

        $bodyString = json_decode($response->getBody()->getContents());

        return !empty($bodyString) ? $bodyString : [];

    }

It’s possible that the number of followers on some accounts will create big downloads, thus slowing the app down. Since we only really need the diff fields, we can target only those with the h param, as described at the bottom of this page.

Then, if we, for debugging purposes, modify the dashboard method…

        $entries = $rest->getUserEntries($twitter_id);
        dd($followers->analyzeUnfollowers($entries));

The output looks something like this. It’s obvious that 5 of our fake followers have only been following us for 5 seconds, while the rest of them have been following us since before we signed up for this service (i.e. forever).

Finally, we can analyze the periods we got back – it’s easy to identify short ones, and color-code them as described at the beginning of this post. As this is already a post of considerable length, I’ll leave that part, and the part about using Twitter’s Users Lookup API to turn the IDs into user handles as homework. Protip: if you run out of query calls for that part, you can crawl their mini profile with the user_id param!

Conclusion

We went through the process of building a simple application for tracking the amount of time a given person has followed you, and flagging them down as a follower farmer, all without using a local database – RestDB provided us with extreme performance, scalability, and independence from local services.

There are many upgrades we could apply to this app:

• a cronjob to auto-refresh the follower lists behind the scenes
• heavy caching to conserve API calls and increase speed
• a premium account subscription which would let users keep more entries
• a dashboard matching tweets with unfollows, showing you what may have prompted someone to leave your twittersphere
• multi-account support
• Instagram support

What other upgrades to this system can you think of? Feel free to contribute to the app on Github!

Frequently Asked Questions (FAQs) about Twitter Analytics with RestDB