Tutorial

How To Build a Real-time Search Engine With Node, Vue, and ElasticSearch

Node.jsElasticsearchVue.js

While this tutorial has content that we believe is of great benefit to our community, we have not yet tested or edited it to ensure you have an error-free learning experience. It's on our list, and we're working on it! You can help us out by using the "report an issue" button at the bottom of the tutorial.

Introduction

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. Elasticsearch is built on top of Apache Lucene, which is a high-performance text search engine library.

In this tutorial, you will build a real-time search engine using Node.js, Elasticsearch, and Vue.js. As a result, a basic comprehension of Vue.js and Node.js (Express) is needed to follow this tutorial.

Demo search site

Step 1 — Preparing the Environment

Let’s get started with setting up the environment for this lesson. Since you will use Node.js, the easiest way to get started is to create a new folder and run npm init. Create a new folder called elastic-node, change directory into the new folder and then run npm init:

Create a new directory called elastic-node:

  • mkdir elastic-node

Move into the new folder:

  • cd elastic-node

Run npm init to create a package.json file:

  • npm init

The above commands take you through the process of creating a package.json file, which is required to run any Node.js library.

Next, you need to install libraries that will be needed for the real-time search engine. Install the libraries with the following command:

  • npm install express body-parser elasticsearch

The express library will run the server, while the body-parser library works with express to parse body requests. elasticsearch is the official Node.js library for Elasticsearch, which is the engine on which the real-time search will be built.

Now the first part of your environment is set up. However, Elasticsearch itself is missing from your setup. You will need to install Elasticsearch which can be done in a few different ways. If you are using a Debian Linux operating system, you can just download the .deb file and install using dpkg.

Run the following command to download the .deb file:

  • curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.deb

Now install the deb package using dpkg:

  • sudo dpkg -i elasticsearch-5.6.4.deb

For other distributions/operating systems, you can find a guide on how to install Elasticsearch here.

Elasticsearch is not started automatically after installation. Elasticsearch can be started and stopped using the service command.

Run the following to start the elasticsearch service:

  • sudo -i service elasticsearch start

You can stop the elasticsearch service with this command:

  • sudo -i service elasticsearch stop

To configure Elasticsearch to start automatically when the system boots up, run this to reload the systemctl daemon:

  • sudo /bin/systemctl daemon-reload

Then enable elasticsearch so it can be called as a service:

  • sudo /bin/systemctl enable elasticsearch.service

After running the command above, you can start Elasticsearch with this command:

  • sudo systemctl start elasticsearch.service

Stop it with the following:

  • sudo systemctl stop elasticsearch.service

You can also check the status of Elasticsearch:

  • sudo service elasticsearch status

Step 2 — Indexing Data in Elasticsearch

Create a data.js file in your root folder and add the following code:

data.js

//require the Elasticsearch librray
const elasticsearch = require('elasticsearch');
// instantiate an Elasticsearch client
const client = new elasticsearch.Client({
   hosts: [ 'http://localhost:9200']
});
// ping the client to be sure Elasticsearch is up
client.ping({
     requestTimeout: 30000,
 }, function(error) {
 // at this point, eastic search is down, please check your Elasticsearch service
     if (error) {
         console.error('Elasticsearch cluster is down!');
     } else {
         console.log('Everything is ok');
     }
 });

This code first requires the Elasticsearch library and then sets up a new Elasticsearch client, passing in an array of a host, http://localhost:9200. This is because, by default, Elasticsearch listens on :9200. Next, you ping the Elasticsearch client to be sure the server is up. If you run node data.js, you will get a message that says Everything is ok.

Step 3 — Understanding Indexes

Unlike normal databases, an Elasticsearch index is a place to store related documents. For example, you will create an index called scotch.io-tutorial to store data of type cities_list. This is how it’s done in Elasticsearch: 

data.js
// create a new index called scotch.io-tutorial. If the index has already been created, this function fails safely
client.indices.create({
      index: 'scotch.io-tutorial'
  }, function(error, response, status) {
      if (error) {
          console.log(error);
      } else {
          console.log("created a new index", response);
      }
});

Add this piece of code after the ping function you had written before. Now, run node data.js again. You will get two messages:

  • Everything is okay
  • Created a new index (with the response from Elasticsearch)

Step 4 — Adding Documents To Indexes

You can add documents to preexisting indexes with the Elasticsearch API. To do this, use the following code:

// add a data to the index that has already been created
client.index({
     index: 'scotch.io-tutorial',
     id: '1',
     type: 'cities_list',
     body: {
         "Key1": "Content for key one",
         "Key2": "Content for key two",
         "key3": "Content for key three",
     }
 }, function(err, resp, status) {
     console.log(resp);
 });

The body refers to the document you want to add to the scotch.io-tutorial index, while the type is more of a category. However, note that if the id key is omitted, Elasticsearch will auto-generate one.

In this tutorial, your document will be a list of all the cities in the world. If you are to add each city one by one, it can take days to index them all. Luckily, Elasticsearch has a bulk function that can process bulk data.

First, grab the JSON file containing all cities in the world here and save that into your root folder as cities.json.

It’s time to use the bulk API to import the large dataset:

data.js
// require the array of cities that was downloaded
const cities = require('./cities.json');
// declare an empty array called bulk
var bulk = [];
//loop through each city and create and push two objects into the array in each loop
//first object sends the index and type you will be saving the data as
//second object is the data you want to index
cities.forEach(city =>{
   bulk.push({index:{ 
                 _index:"scotch.io-tutorial", 
                 _type:"cities_list",
             }          
         })
  bulk.push(city)
})
//perform bulk indexing of the data passed
client.bulk({body:bulk}, function( err, response  ){ 
         if( err ){ 
             console.log("Failed Bulk operation".red, err) 
         } else { 
             console.log("Successfully imported %s".green, cities.length); 
         } 
}); 

Here, you have looped through all the cities in your JSON file, and at each loop, you append an object with the index and type of the document you will be indexing. There are two pushes to the array in the loop because the bulk API expects an object containing the index definition first, and then the document you want to index. For more information on that, you can check out this article.

Next, you called the client.bulk function, passing in the new bulk array as the body. This indexes all your data into Elasticsearch with the index of scotch.io-tutorial and type cities_list.

Step 5 — Using Express to Serve the Landing Page

Your Elasticsearch instance is up and running, and you can connect with it using Node.js. It’s time to use Express to serve a landing page and use the setup you have running so far.

Create a file called index.js and add the following code:

index.js
//require the Elasticsearch librray
const elasticsearch = require('elasticsearch');
// instantiate an elasticsearch client
const client = new elasticsearch.Client({
   hosts: [ 'http://localhost:9200']
});
//require Express
const express = require( 'express' );
// instanciate an instance of express and hold the value in a constant called app
const app     = express();
//require the body-parser library. will be used for parsing body requests
const bodyParser = require('body-parser')
//require the path library
const path    = require( 'path' );

// ping the client to be sure Elasticsearch is up
client.ping({
     requestTimeout: 30000,
 }, function(error) {
 // at this point, eastic search is down, please check your Elasticsearch service
     if (error) {
         console.error('elasticsearch cluster is down!');
     } else {
         console.log('Everything is ok');
     }
 });


// use the bodyparser as a middleware  
app.use(bodyParser.json())
// set port for the app to listen on
app.set( 'port', process.env.PORT || 3001 );
// set path to serve static files
app.use( express.static( path.join( __dirname, 'public' )));
// enable CORS 
app.use(function(req, res, next) {
  res.header("Access-Control-Allow-Origin", "*");
  res.header('Access-Control-Allow-Methods', 'PUT, GET, POST, DELETE, OPTIONS');
  res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
  next();
});

// defined the base route and return with an HTML file called tempate.html
app.get('/', function(req, res){
  res.sendFile('template.html', {
     root: path.join( __dirname, 'views' )
   });
})

// define the /search route that should return elastic search results 
app.get('/search', function (req, res){
  // declare the query object to search elastic search and return only 200 results from the first result found. 
  // also match any data where the name is like the query string sent in
  let body = {
    size: 200,
    from: 0, 
    query: {
      match: {
          name: req.query['q']
      }
    }
  }
  // perform the actual search passing in the index, the search query and the type
  client.search({index:'scotch.io-tutorial',  body:body, type:'cities_list'})
  .then(results => {
    res.send(results.hits.hits);
  })
  .catch(err=>{
    console.log(err)
    res.send([]);
  });

})
// listen on the specified port
app .listen( app.get( 'port' ), function(){
  console.log( 'Express server listening on port ' + app.get( 'port' ));
} );

Looking at the code above, you will see that the code has done the following:

  • Required the Express, body-parser and path libraries.
  • Set a new instance of Express to the constant called app.
  • Set the app to use the bodyParser middleware.
  • Set the static part of the app to a folder called public. This folder has not been created yet.
  • Defined a middleware which adds CORS header to the app.
  • Defined a GET route for the root URL of the app, represented by /. In this route, the code returns a file called template.html which is in the views folder.
  • Defined a GET route for the /search URL of the app which uses a query object to search for the match of the data passed to it via the query string. The main search query is included within the query object. You can add different search queries to this object. For this query, you add a key with the query and return an object telling it that the name of the document you are looking for should match req.query['q'].

Besides the query object, the search body can contain other optional properties, including size and from. The size property determines the number of documents to be included in the response. If this value is not present, by default ten documents are returned. The from property determines the starting index of the returned documents. This is useful for pagination.

Step 6 — Understanding the Search API Response

If you were to log the response from the search API, it would include a lot of information. Below is an example:

Output
{ took: 88, timed_out: false, _shards: { total: 5, successful: 5, failed: 0 }, hits: { total: 59, max_score: 5.9437823, hits: [ {"_index":"scotch.io-tutorial", "_type":"cities_list", "_id":"AV-xjywQx9urn0C4pSPv", "_score":5.9437823," _source":{"country":"ES","name":"A Coruña","lat":"43.37135","lng":"-8.396"}}, [Object], ... [Object] ] } }

The response includes a took property for the number of milliseconds it took to find the results, timed_out, which is only true if no results were found in the maximum allowed time, _shards for information about the status of the different nodes (if deployed as a cluster of nodes), and hits, which includes the search results.

Within the hits property, you have an object with the following properties:

  • total shows the total number of matched items.
  • max_score is the maximum score of the found items.
  • hits is an array that includes the found items.

This is why you returned response.hits.hits in the search route, which houses the documents found.

Step 7 — Creating The HTML Template

First, create two new folders in your root folder named views and public which were referenced in the previous step. Next, create a file called template.html in the views folder and paste the following code:

template.html
<!-- template.html -->
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/vue"></script>
<div class="container" id="app">
    <div class="row">
        <div class="col-md-6 col-md-offset-3">
            <h1>Search Cities around the world</h1>
        </div>
    </div>
    <div class="row">
        <div class="col-md-4 col-md-offset-3">
            <form action="" class="search-form">
                <div class="form-group has-feedback">
                    <label for="search" class="sr-only">Search</label>
                    <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" >
                    <span class="glyphicon glyphicon-search form-control-feedback"></span>
                </div>
            </form>
        </div>
    </div>
    <div class="row">
        <div class="col-md-3" v-for="result in results">
            <div class="panel panel-default">
                <div class="panel-heading">
                <!-- display the city name and country  -->
                    {{ result._source.name }}, {{ result._source.country }} 
                </div>
                <div class="panel-body">
                <!-- display the latitude and longitude of the city  -->
                    <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
                </div>
            </div>
        </div>
    </div>
</div>
<!--- some styling for the page -->
<style>
    .search-form .form-group {
        float: right !important;
        transition: all 0.35s, border-radius 0s;
        width: 32px;
        height: 32px;
        background-color: #fff;
        box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
        border-radius: 25px;
        border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
        padding-right: 20px;
        border: 0 none;
        background: transparent;
        box-shadow: none;
        display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
        display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
        /* Firefox 18- */
        display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
        /* Firefox 19+ */
        display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
        display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
        width: 100%;
        border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
        position: absolute;
        top: -1px;
        right: -2px;
        z-index: 2;
        display: block;
        width: 34px;
        height: 34px;
        line-height: 34px;
        text-align: center;
        color: #3596e0;
        left: initial;
        font-size: 14px;
    }
</style>

In the code snippet above, there are two main sections: HTML and CSS code.

In the HTML section, you required three different libraries:

  1. Bootstrap CSS for styling the page.
  2. Axios js for making HTTP requests to our server.
  3. Vue.js which is a minimalistic framework that you will use for the view.

In the CSS section you have applied styling to make the search input hide and reveal itself once you hover over the search icon.

Next, there is an input for the search box that you assigned its v-model to query (this will be used by Vue.js). After this, you looped through all our results.

Run the node index.js command and then visit http://localhost:3001/ in a browser. You will see the app’s landing page.

Search app landing page

Next, add a script tag in your template.html file:

template.html
// create a new Vue instance
var app = new Vue({
    el: '#app',
    // declare the data for the component (An array that houses the results and a query that holds the current search string)
    data: {
        results: [],
        query: ''
    },
    // declare methods in this Vue component. here only one method which performs the search is defined
    methods: {
        // make an axios request to the server with the current search query
        search: function() {
            axios.get("http://127.0.0.1:3001/search?q=" + this.query)
                .then(response => {
                    this.results = response.data;

                })
        }
    },
    // declare Vue watchers
    watch: {
        // watch for change in the query string and recall the search method
        query: function() {
            this.search();
        }
    }

})

In this section, you declared a new instance of Vue, mounting it on the element with the id of app. You declared data properties which include query, which you had attached to the search input, and results, which is an array of all results found.

In the methods quota, you have just one function called search, which triggers a GET request to the search route. This passes along the current input in the search box. This in turn returns a response that is then looped in the HTML code block.

Finally, you use watchers in Vue.js, which performs an action anytime data being watched changes. Here, you are watching for a change in the query data, and once it changes, the search method is fired.

If you re-run the node index.js command now and navigate to http://localhost:3001/ again, it should work as seen:
Completed search app that displays results

Step 8 — Searching From The Client Side

If you do not want to send requests to the server every time a search occurs, you can search the Elasticsearch engine from the client side. Some developers might not be comfortable with hitting their servers for every search term, while some feel it’s more secure to search from the server side.

Elasticsearch offers a browser build which can make searches. This step will guide you through searching from the browser client.

First, add a new route to your Express file and restart your server:

index.js
// decare a new route. This route serves a static HTML template called template2.html
app.get('/v2', function(req, res){
  res.sendFile('template2.html', {
     root: path.join( __dirname, 'views' )
   });
})

In this code block, you created a new route for the URL at /v2 to return a static HTML file called template2.html. You will create this file soon.

Next, you need to download the client library for Elasticsearch here. After downloading, extract and copy elasticsearch.min.js to the public folder in your application root.

Note: If you encounter CORS issues after connecting to the Elasticsearch engine from the client side, and the following snippet to the end of the Elasticsearch configuration file. You can locate this file by reviewing Elasticsearch’s reference material.

/etc/elasticsearch/elasticsearch.yml

http.cors.enabled : true
http.cors.allow-origin : "*"

After that is done, restart your Elasticsearch instance:

  • sudo service elasticsearch restart

Next, create a file called template2.html in your views folder and this code:

template2.html
<!-- template2.html -->
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/vue"></script>
<div class="container" id="app">
    <div class="row">
        <div class="col-md-6 col-md-offset-3">
            <h1>Search Cities around the world</h1>
        </div>
    </div>
    <div class="row">
        <div class="col-md-4 col-md-offset-3">
            <form action="" class="search-form">
                <div class="form-group has-feedback">
                    <label for="search" class="sr-only">Search</label>
                    <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" >
                    <span class="glyphicon glyphicon-search form-control-feedback"></span>
                </div>
            </form>
        </div>
    </div>
    <div class="row">
        <div class="col-md-3" v-for="result in results">
            <div class="panel panel-default">
                <div class="panel-heading">
                <!-- display the city name and country  -->
                    {{ result._source.name }}, {{ result._source.country }} 
                </div>
                <div class="panel-body">
                <!-- display the latitude and longitude of the city  -->
                    <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
                </div>
            </div>
        </div>
    </div>
</div>
<script src="/elasticsearch.min.js"></script>
<style>
    .search-form .form-group {
        float: right !important;
        transition: all 0.35s, border-radius 0s;
        width: 32px;
        height: 32px;
        background-color: #fff;
        box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
        border-radius: 25px;
        border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
        padding-right: 20px;
        border: 0 none;
        background: transparent;
        box-shadow: none;
        display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
        display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
        /* Firefox 18- */
        display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
        /* Firefox 19+ */
        display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
        display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
        width: 100%;
        border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
        position: absolute;
        top: -1px;
        right: -2px;
        z-index: 2;
        display: block;
        width: 34px;
        height: 34px;
        line-height: 34px;
        text-align: center;
        color: #3596e0;
        left: initial;
        font-size: 14px;
    }
</style>

Next, add a script tag in your template2.html file and add this code:

template2.html
// instantiate a new Elasticsearch client like you did on the client
var client = new elasticsearch.Client({
    hosts: ['http://127.0.0.1:9200']
});
// create a new Vue instance
var app = new Vue({
    el: '#app',
    // declare the data for the component (An array that houses the results and a query that holds the current search string)
    data: {
        results: [],
        query: ''
    },
    // declare methods in this Vue component. here only one method which performs the search is defined
    methods: {
        // function that calls the elastic search. here the query object is set just as that of the server.
        //Here the query string is passed directly from Vue
        search: function() {
            var body = {
                    size: 200,
                    from: 0,
                    query: {
                        match: {
                            name: this.query
                        }
                    }
                }
                // search the Elasticsearch passing in the index, query object and type
            client.search({ index: 'scotch.io-tutorial', body: body, type: 'cities_list' })
                .then(results => {
                    console.log(`found ${results.hits.total} items in ${results.took}ms`);
                    // set the results to the result array we have 
                    this.results = results.hits.hits;
                })
                .catch(err => {
                    console.log(err)

                });


        }
    },
    // declare Vue watchers
    watch: {
        // watch for change in the query string and recall the search method
        query: function() {
            this.search();
        }
    }

})

The HTML and JavaScript snippet above is similar to the one in the previous step, with the main differences being as follows:

  • You did not require Axios, you required elasticsearch.js instead.
  • At the top of the script tag, you initiated the Elasticsearch client as it was done on the server-side.
  • The search method does not perform an HTTP request, but rather searches the Elasticsearch engine as done in the search route on the server side.

If you browse to http://localhost:3001/v2, It should work as shown:
Client side searching from the search app

Conclusion

In this tutorial, you used Elasticsearch to index data. You have also implemented a real-time search using the client library for Elasticsearch.

Creative Commons License