How To Process Tweets in Real-Time with Laravel

Introduction

This tutorial will show how to use the Twitter Streaming APIs to process tweets in real-time from a Laravel application. There are a variety of use cases for this: perhaps you want to auto-respond to mentions of your company, run a contest via Twitter, or create support tickets when users complain about your product. For this tutorial, we’ll build a “featured tweets” widget to display approved tweets on our app’s home page.

You might first look at the Twitter REST APIs, which many developers are familiar with. These APIs allow you to do a lot of cool things: read and write tweets on behalf of app users, manage followers and lists, and more. However, if you want to process tweets as they are sent, the REST APIs are very difficult to manage. Pagination and rate limiting become a logistical nightmare quickly.

Whereas the REST APIs force you to request information, the Streaming APIs feed information to you. You can read more about how it works in the documentation, but it’s basically a pipe from Twitter to your application which delivers a continual flow of tweets.

By the end of the tutorial, you’ll have a constantly updating collection of tweets in your app that you can approve or disapprove to show up in your home page. Here’s a preview of what you’ll end up with.

Overview

The Tools

• Public Streaming API: There are three Streaming APIs, but the one we care about is the Public API - and more specifically, the ‘filter’ endpoint. This endpoint will deliver a stream of all public tweets, filtered by the keywords that you define.
• Phirehose: The Phirehose library takes care of authentication and other implementation details for the Streaming API.
• Homestead: Although you can build your app with your environment of choice, this tutorial will assume you are using Homestead.

Notes

• The Streaming APIs are treated specially by Twitter. You can only have one connection open from a single Twitter app.
• Because of the special nature of the connection to the Streaming API, it’s important that it is separate from the web-facing part of your app. In this tutorial, we’ll be creating the connection with an Artisan console command.
• Because the Streaming APIs are so finicky, it’s important not to do anything intensive with the tweets in the same process that is collecting them. In this tutorial you’ll simply be placing the tweets onto a queue.
• Be careful with the terms you are tracking. If you try to track all mentions of a popular celebrity, your app will probably melt.

App Structure

Here are the major pieces of the app that we’ll be creating:

• A class called TwitterStream which will extend Phirehose and place tweets onto the queue as they arrive
• An Artisan command, ConnectToStreamingAPI, which will open a connection to the Streaming API with an instance of TwitterStream
• A job class, ProcessTweet, which will contain the handler to process tweets when they are pulled off the queue
• A Tweet Eloquent model
• A few blade templates to display the tweet widget

While reading through the tutorial, you can follow along with this GitHub repo. It has separate commits for each step:

https://github.com/dabernathy89/Laravel-Twitter-Streaming-API-Demo

Step 1 — Create a New Laravel App

Using the Laravel installer, create a new Laravel instance:

laravel new twitter-stream-test

Once that’s done, add the app to your Homestead configuration file. Don’t forget to edit your hosts file with whatever custom domain you added to the configuration. Boot up Homestead:

homestead up --provision

SSH into Homestead:

homestead ssh

Finally, cd into your app’s folder.

Step 2 — Install Phirehose

Next, you need to pull in the Phirehose library. Run composer require fennb/phirehose to grab the latest version. Phirehose doesn’t support PSR-0 or PSR-4 autoloading, so you’ll need to use Composer’s classmap autoloading option. Add a line just after the database entry:

        "classmap": [
            "database",
            "vendor/fennb/phirehose/lib"
        ],

Step 3 — Create the ProcessTweet Job

I mentioned in the overview that we will be putting all of the tweets from the Streaming API onto a queue. In Laravel 5.2, there are special Job classes that are meant for processing items in a queue. Your app will have a job called ProcessTweet which will be responsible for pulling tweets off the queue and doing something with them. You can create the job with a simple Artisan command:

php artisan make:job ProcessTweet

This will generate a Job class in your app/Jobs folder. You only need to make two adjustments to it for now.

1. You know that this class will be processing a tweet (even though we haven’t set that up yet). Pass a $tweet variable into the constructor and set it as a property.
2. Do something with the tweet in the handle() method. For now you can just var_dump() some basic information from the tweet.

<?php

namespace App\Jobs;

use App\Jobs\Job;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;

class ProcessTweet extends Job implements ShouldQueue
{
    use InteractsWithQueue, SerializesModels;

    protected $tweet;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct($tweet)
    {
        $this->tweet = $tweet;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $tweet = json_decode($this->tweet,true);
        var_dump($tweet['text']) . PHP_EOL;
        var_dump($tweet['id_str']) . PHP_EOL;
    }
}

Step 4 — Create The TwitterStream Class

To use Phirehose, you need to create a simple class that extends the base Phirehose class. This can go wherever you like, but I’ve just placed it directly in the app folder. Here’s the full class:

<?php

namespace App;

use OauthPhirehose;
use App\Jobs\ProcessTweet;
use Illuminate\Foundation\Bus\DispatchesJobs;

class TwitterStream extends OauthPhirehose
{
    use DispatchesJobs;

    /**
    * Enqueue each status
    *
    * @param string $status
    */
    public function enqueueStatus($status)
    {
        $this->dispatch(new ProcessTweet($status));
    }
}

A couple of things to note:

• The class pulls in the DispatchesJobs trait to make it easy to push the tweet onto the queue.
• There is only a single method, enqueueStatus, which will be called by Phirehose for each tweet.

Step 5 — Register the TwitterStream Class

You need to register the TwitterStream class with the Laravel container so that it can pull in its dependencies properly. In the register method of your AppServiceProvider class, add the following:

        $this->app->bind('App\TwitterStream', function ($app) {
            $twitter_access_token = env('TWITTER_ACCESS_TOKEN', null);
            $twitter_access_token_secret = env('TWITTER_ACCESS_TOKEN_SECRET', null);
            return new TwitterStream($twitter_access_token, $twitter_access_token_secret, Phirehose::METHOD_FILTER);
        });

You will also need to add use Phirehose; and use App\TwitterStream; at the top of your service provider. Don’t worry about the environment variables for now - you’ll create them shortly.

Step 6 — Create the Artisan Command

Generate an Artisan command so that you can initiate the Streaming API connection over the command line:

php artisan make:console ConnectToStreamingAPI

This will generate the boilerplate console command class. Then you need to:

• Update the command’s signature and description
• Pull in an instance of the TwitterStream class through the constructor
• In the command’s handle() method, finish configuring the Phirehose object, including your search terms, and open the connection

Here’s what the completed command looks like. It will pull in tweets that contain the keyword scotch_io:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use App\TwitterStream;

class ConnectToStreamingAPI extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'connect_to_streaming_api';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Connect to the Twitter Streaming API';

    protected $twitterStream;

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct(TwitterStream $twitterStream)
    {
        $this->twitterStream = $twitterStream;

        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        $twitter_consumer_key = env('TWITTER_CONSUMER_KEY', '');
        $twitter_consumer_secret = env('TWITTER_CONSUMER_SECRET', '');

        $this->twitterStream->consumerKey = $twitter_consumer_key;
        $this->twitterStream->consumerSecret = $twitter_consumer_secret;
        $this->twitterStream->setTrack(array('scotch_io'));
        $this->twitterStream->consume();
    }
}

You also need to register the command. Simply update the $commands property in the App\Console\Kernel class:

    protected $commands = [
        Commands\ConnectToStreamingAPI::class
    ];

Step 7 — Create a Twitter App

Generate an app with Twitter, and grab the keys and access tokens. Add them to your .env file:

TWITTER_CONSUMER_KEY=KEY
TWITTER_CONSUMER_SECRET=SECRET
TWITTER_ACCESS_TOKEN=TOKEN
TWITTER_ACCESS_TOKEN_SECRET=SECRET

Step 8 — Configure Your Queue Driver

You can set up the queue however you want, but for demonstration purposes, the database driver will work fine. To set that up, update your .env file:

QUEUE_DRIVER=database

By default, your app will look for a database with the name homestead. Don’t forget to update this in your .env file if you want a separate database for your app:

DB_DATABASE=twitterstreamtest

You’ll also need to run the following commands to prepare the database queue:

php artisan queue:table

php artisan migrate

Step 9 — Read some tweets!

Now you should be ready to start processing some tweets. Run your Artisan command:

php artisan connect_to_streaming_api

You’ll see some information about the connection in your console from Phirehose. It won’t tell you every time a tweet is processed, but it will give you an update every once in a while on the status of the connection.

Before too long, you should have some tweets about Scotch.io sitting in a queue. To check if tweets are coming in, head over to your database and look in the jobs table:

Now let’s try processing the queue. Open up a new shell window, navigate to your app in Homestead, and pull the first item off the queue:

php artisan queue:work

If everything went right, you should see the text and id of the tweet in the console, from the handle() method in the ProcessTweet job.

Step 10 — Set Up the Tweet Model

Now that you have successfully connected to the Streaming API, it’s time to start building your featured tweets widget. Start by generating a Tweet model and a corresponding database migration:

php artisan make:model Tweet --migration

We’re going to add some properties to our model, so go ahead and set those now. Below is what your completed model and migration should look like - note that I’m using a string for the model ID, because Twitter IDs are massive.

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Tweet extends Model
{
    protected $fillable = ['id','json','tweet_text','user_id','user_screen_name','user_avatar_url','public','approved'];
}

<?php

use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;

class CreateTweetsTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('tweets', function (Blueprint $table) {
            $table->string('id');
            $table->text('json');
            $table->string('tweet_text')->nullable();
            $table->string('user_id')->nullable();
            $table->string('user_screen_name')->nullable();
            $table->string('user_avatar_url')->nullable();
            $table->boolean('approved');
            $table->timestamps();
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::drop('tweets');
    }
}

Finally, migrate your database:

php artisan migrate

Step 11 — Make the ProcessTweet Job Useful

Right now the ProcessTweet job is just displaying some information about the tweet in the console. Update the handle method so that it saves the tweets to the database:

    public function handle()
    {
        $tweet = json_decode($this->tweet,true);
        $tweet_text = isset($tweet['text']) ? $tweet['text'] : null;
        $user_id = isset($tweet['user']['id_str']) ? $tweet['user']['id_str'] : null;
        $user_screen_name = isset($tweet['user']['screen_name']) ? $tweet['user']['screen_name'] : null;
        $user_avatar_url = isset($tweet['user']['profile_image_url_https']) ? $tweet['user']['profile_image_url_https'] : null;

        if (isset($tweet['id'])) {
            Tweet::create([
                'id' => $tweet['id_str'],
                'json' => $this->tweet,
                'tweet_text' => $tweet_text,
                'user_id' => $user_id,
                'user_screen_name' => $user_screen_name,
                'user_avatar_url' => $user_avatar_url,
                'approved' => 0
            ]);
        }
    }

When that’s done, you can run the queue listener to empty out the queue and import the tweets into your database:

php artisan queue:listen

You should now be able to see some tweets in your database:

Step 12 — Set Up Authentication

Thanks to Laravel’s auth scaffolding, this is probably the easiest step. Just run:

php artisan make:auth

Step 13 — Pass Tweets Into the Welcome View

For now your app will only display the featured tweets widget on the main landing page. Authenticated users are going to be allowed to approve or disapprove tweets, so they will receive all tweets (paginated). Visitors will only be able to see the five most recent approved tweets.

Pass the tweets into the existing welcome view by updating the main route in your routes.php file:

Route::get('/', function () {
    if (Auth::check()) {
        $tweets = App\Tweet::orderBy('created_at','desc')->paginate(5);
    } else {
        $tweets = App\Tweet::where('approved',1)->orderBy('created_at','desc')->take(5)->get();
    }

    return view('welcome', ['tweets' => $tweets]);
});

Step 14 — Create The Blade Templates

You’ll need to create three blade templates for the featured tweets widget, all stored in the resources/views/tweets directory:

• list.blade.php will be the public-facing list of recent tweets
• list-admin.blade.php will be the admin-facing list of all tweets
• tweet.blade.php will be a small partial common to both of the tweet lists

The list-admin view is the most complex. The list is wrapped in a form and includes some radio inputs so that registered users can easily approve tweets.

Here are the three templates, in order:

// list.blade.php
@foreach($tweets as $tweet)
    <div class="tweet">
        @include('tweets.tweet')
    </div>
@endforeach

// list-admin.blade.php
<form action="/approve-tweets" method="post">
{{ csrf_field() }}

@foreach($tweets as $tweet)
    <div class="tweet row">
        <div class="col-xs-8">
            @include('tweets.tweet')
        </div>
        <div class="col-xs-4 approval">
            <label class="radio-inline">
                <input
                    type="radio"
                    name="approval-status-{{ $tweet->id }}"
                    value="1"
                    @if($tweet->approved)
                    checked="checked"
                    @endif
                    >
                Approved
            </label>
            <label class="radio-inline">
                <input
                    type="radio"
                    name="approval-status-{{ $tweet->id }}"
                    value="0"
                    @unless($tweet->approved)
                    checked="checked"
                    @endif
                    >
                Unapproved
            </label>
        </div>
    </div>
@endforeach

<div class="row">
    <div class="col-sm-12">
        <input type="submit" class="btn btn-primary" value="Approve Tweets">
    </div>
</div>

</form>

{!! $tweets->links() !!}

// tweet.blade.php
<div class="media">
    <div class="media-left">
        <img class="img-thumbnail media-object" src="{{ $tweet->user_avatar_url }}" alt="Avatar">
    </div>
    <div class="media-body">
        <h4 class="media-heading">{{ '@' . $tweet->user_screen_name }}</h4>
        <p>{{ $tweet->tweet_text }}</p>
        <p><a target="_blank" href="https://twitter.com/{{ $tweet->user_screen_name }}/status/{{ $tweet->id }}">
            View on Twitter
        </a></p>
    </div>
</div>

Step 15 - Show the Widget in Your Welcome View

With your blade templates ready, you can now bring them into your welcome view:

@extends('layouts.app')

@section('content')
<div class="container">
    <div class="row">
        <div class="col-md-8 col-md-offset-2">

            <div class="tweet-list">
                @if(Auth::check())
                    @include('tweets.list-admin')
                @else
                    @include('tweets.list')
                @endif
            </div>

        </div>
    </div>
</div>
@endsection

To get some very basic styling in your list, add the following CSS to your app.blade.php file:

        .tweet {
            padding: 10px;
            border: 1px solid #ccc;
            border-radius: 5px;
            margin-bottom: 20px;
        }

With that done, you can now view the widget! Head to the browser and visit your app’s home page. If you’re not seeing any tweets, make sure to run the queue listener (php artisan queue:listen) to process anything that might still be in it. An authorized user should see something like this (with a few more tweets… and less blurry):

Step 16 — Add the Route for Approving Tweets

The final step is to make the admin list from Step 14 functional. The form in the list-admin template currently points to a nonexistent route. You need to add it to your routes.php file. Inside that route we’ll do some basic logic to approve or disapprove tweets. Here’s how it should look:

Route::post('approve-tweets', ['middleware' => 'auth', function (Illuminate\Http\Request $request) {
    foreach ($request->all() as $input_key => $input_val) {
        if ( strpos($input_key, 'approval-status-') === 0 ) {
            $tweet_id = substr_replace($input_key, '', 0, strlen('approval-status-'));
            $tweet = App\Tweet::where('id',$tweet_id)->first();
            if ($tweet) {
                $tweet->approved = (int)$input_val;
                $tweet->save();
            }
        }
    }
    return redirect()->back();
}]);

With that route in place, you should now be able to mark tweets as approved or disapproved. Try approving some, and then visit the page as an unauthenticated user. It should look something like this:

Conclusion

That’s it! You’ve connected your Laravel app to the Twitter Streaming API. However, it’s not quite production ready. There are a few other things you should consider:

• You will likely want to configure Supervisor, which Laravel uses to monitor queue listeners, to monitor your Artisan command and restart it if it fails.
• You’ll want to handle the call to Supervisor into your deployment process.
• Your search terms might change, and Phirehose provides a method for updating them without interrupting your script.