Tutorial

How To Build a Real-time Search Engine With Node, Vue, and ElasticSearch

Published on December 12, 2019
Default avatar

By Ogundipe Samuel Ayo

How To Build a Real-time Search Engine With Node, Vue, and ElasticSearch

This tutorial is out of date and no longer maintained.

Warning: At the time of verification, the elasticsearch package is a legacy library. The contents herein are presented for educational purposes and not advisable for a production setting.

The modern package is @elastic/elasticsearch. It does not provide official support for the browser environment and recommends the usage of a lightweight proxy to avoid security issues.

Introduction

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. Elasticsearch is built on top of Apache Lucene, which is a high-performance text search engine library.

In this tutorial, you will build a real-time search engine using Node.js, Elasticsearch, and Vue.js. As a result, a basic comprehension of Vue.js and Node.js (Express) is needed to follow this tutorial.

Animated gif of using the demo app to search for cities around the world and displaying search results

Prerequisites

To complete this tutorial, you will need:

Note: For macOS environments you can use Homebrew.

  1. brew tap elastic/tap
  2. brew install elastic/tap/elasticsearch-oss
  3. brew services start elastic/tap/elasticsearch-oss

This tutorial was verified with Node v14.3.0, npm v6.14.5, elasticsearch-oss v7.7.0, and elasticsearch v16.17.1.

Step 1 — Preparing the Environment

Let’s get started with setting up the environment for this lesson. Since you will use Node.js, the easiest way to get started is to create a new folder and run npm init.

Create a new directory called elastic-node:

  1. mkdir elastic-node

Move into the new folder:

  1. cd elastic-node

Run npm init to create a package.json file:

  1. npm init -y

The commands take you through the process of creating a package.json file, which is required to run any Node.js library.

Next, you need to install libraries that will be needed for the real-time search engine. Install the libraries with the following command:

  1. npm install express body-parser elasticsearch

The express library will run the server, while the body-parser library works with express to parse body requests. elasticsearch is the official Node.js library for Elasticsearch, which is the engine on which the real-time search will be built.

Step 2 — Indexing Data in Elasticsearch

Create a data.js file in your root folder and add the following code:

data.js
// Require the Elasticsearch library.
const elasticsearch = require('elasticsearch');

// Instantiate an Elasticsearch client.
const client = new elasticsearch.Client({
  hosts: ['http://localhost:9200'],
});

// Ping the client to be sure Elasticsearch is up.
client.ping(
  {
    requestTimeout: 30000,
  },
  function (error) {
    // At this point, Elasticsearch is down, please check your Elasticsearch service.
    if (error) {
      console.error('Elasticsearch cluster is down!');
    } else {
      console.log('Everything is okay.');
    }
  }
);

This code first requires the Elasticsearch library and then sets up a new Elasticsearch client, passing in an array of a host, http://localhost:9200. This is because, by default, Elasticsearch listens on :9200.

Next, you ping the Elasticsearch client to be sure the server is up.

If you run node data.js, you will get a message that says Everything is okay.

Step 3 — Understanding Indexes

Unlike normal databases, an Elasticsearch index is a place to store related documents. For example, you will create an index called example_index to store data of type cities_list. This is how it’s done in Elasticsearch:

data.js
// ...

// Create a new index called example_index. If the index has already been created, this function fails safely.
client.indices.create(
  {
    index: 'example_index',
  },
  function (error, response, status) {
    if (error) {
      console.log(error);
    } else {
      console.log('Created a new index.', response);
    }
  }
);

Add this piece of code after the ping function you had written before.

Now, run node data.js again. You will get two messages:

  • Everything is okay
  • Created a new index (with the response from Elasticsearch)

Step 4 — Adding Documents to Indexes

You can add documents to preexisting indexes with the Elasticsearch API. Here is an example to accomplish this:

// Test adding data to the index that has already been created.
client.index(
  {
    index: 'example_index',
    id: '1',
    type: 'cities_list',
    body: {
      Key1: 'Content for key one',
      Key2: 'Content for key two',
      key3: 'Content for key three',
    },
  },
  function (err, resp, status) {
    console.log(resp);
  }
);

The body refers to the document you want to add to the example_index index, while the type is more of a category. However, note that if the id key is omitted, Elasticsearch will auto-generate one.

In this tutorial, your document will be a list of all the cities in the world. If you are to add each city one by one, it can take days to index them all. Luckily, Elasticsearch has a bulk function that can process bulk data.

First, grab this JSON file containing all cities in the world and save that into your root folder as cities.json.

It’s time to use the bulk API to import the large dataset:

data.js
// ...

// Require the array of cities that was downloaded.
const cities = require('./cities.json');

// Declare an empty array called bulk.
var bulk = [];

// Loop through each city and create and push two objects into the array in each loop.
// First object sends the index and type you will be saving the data as.
// Second object is the data you want to index.
cities.forEach((city) => {
  bulk.push({
    index: {
      _index: 'example_index',
      _type: 'cities_list',
    },
  });
  bulk.push(city);
});

// Perform bulk indexing of the data passed.
client.bulk({ body: bulk }, function (err, response) {
  if (err) {
    console.log('Failed bulk operation.', err));
  } else {
    console.log('Successfully imported %s'), cities.length);
  }
});

Here, you have looped through all the cities in your JSON file, and at each loop, you append an object with the index and type of the document you will be indexing. There are two pushes to the array in the loop because the bulk API expects an object containing the index definition first, and then the document you want to index. For more information on that, you can check out the documentation for bulk.

Next, you called the client.bulk function, passing in the new bulk array as the body. This indexes all your data into Elasticsearch with the index of example_index and type cities_list.

Step 5 — Using Express to Serve the Landing Page

Your Elasticsearch instance is up and running, and you can connect with it using Node.js. It’s time to use Express to serve a landing page and use the setup you have running so far.

Create a file called index.js and add the following code:

index.js
// Require the Elasticsearch library.
const elasticsearch = require('elasticsearch');

// Instantiate an Elasticsearch client.
const client = new elasticsearch.Client({
  hosts: ['http://localhost:9200'],
});

// require Express.
const express = require('express');

// Instantiate an instance of express and hold the value in a constant called app.
const app = express();

// Require the body-parser library. Will be used for parsing body requests.
const bodyParser = require('body-parser');

// Require the path library.
const path = require('path');

// Ping the client to be sure Elasticsearch is up.
client.ping(
  {
    requestTimeout: 30000,
  },
  function (error) {
    // At this point, Elasticsearch is down, please check your Elasticsearch service.
    if (error) {
      console.error('Elasticsearch cluster is down!');
    } else {
      console.log('Everything is okay.');
    }
  }
);

// Use the bodyparser as a middleware.
app.use(bodyParser.json());

// Set port for the app to listen on.
app.set('port', process.env.PORT || 3001);

// Set path to serve static files.
app.use(express.static(path.join(__dirname, 'public')));

// Enable CORS.
app.use(function (req, res, next) {
  res.header('Access-Control-Allow-Origin', '*');
  res.header('Access-Control-Allow-Methods', 'PUT, GET, POST, DELETE, OPTIONS');
  res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept');
  next();
});

// Define the base route and return with an HTML file called tempate.html.
app.get('/', function (req, res) {
  res.sendFile('template.html', {
    root: path.join(__dirname, 'views'),
  });
});

// Define the /search route that should return Elasticsearch results.
app.get('/search', function (req, res) {
  // Declare the query object to search Elasticsearch.
  // Return only 200 results from the first result found.
  // Also match any data where the name is like the query string sent in.
  let body = {
    size: 200,
    from: 0,
    query: {
      match: {
        name: req.query['q'],
      },
    },
  };
  // Perform the actual search passing in the index, the search query, and the type.
  client
    .search({ index: 'example_index', body: body, type: 'cities_list' })
    .then((results) => {
      res.send(results.hits.hits);
    })
    .catch((err) => {
      console.log(err);
      res.send([]);
    });
});

// Listen on the specified port.
app.listen(app.get('port'), function () {
  console.log('Express server listening on port ' + app.get('port'));
});

Looking at the code, you will see that the code has done the following:

  • Required the Express, body-parser and path libraries.
  • Set a new instance of Express to the constant called app.
  • Set the app to use the bodyParser middleware.
  • Set the static part of the app to a folder called public. This folder has not been created yet.
  • Defined a middleware which adds CORS header to the app.
  • Defined a GET route for the root URL of the app, represented by /. In this route, the code returns a file called template.html which is in the views folder.
  • Defined a GET route for the /search URL of the app which uses a query object to search for the match of the data passed to it via the query string. The main search query is included within the query object. You can add different search queries to this object. For this query, you add a key with the query and return an object telling it that the name of the document you are looking for will match req.query['q'].

Besides the query object, the search body can contain other optional properties, including size and from. The size property determines the number of documents to be included in the response. If this value is not present, by default ten documents are returned. The from property determines the starting index of the returned documents. This is useful for pagination.

Step 6 — Understanding the Search API Response

If you were to log the response from the search API, it would include a lot of information:

Output
{ took: 88, timed_out: false, _shards: { total: 5, successful: 5, failed: 0 }, hits: { total: 59, max_score: 5.9437823, hits: [ {"_index":"example_index", "_type":"cities_list", "_id":"AV-xjywQx9urn0C4pSPv", "_score":5.9437823," _source":{"country":"ES","name":"A Coruña","lat":"43.37135","lng":"-8.396"}}, [Object], ... [Object] ] } }

The response includes a took property for the number of milliseconds it took to find the results, timed_out, which is only true if no results were found in the maximum allowed time, _shards for information about the status of the different nodes (if deployed as a cluster of nodes), and hits, which includes the search results.

Within the hits property, you have an object with the following properties:

  • total shows the total number of matched items.
  • max_score is the maximum score of the found items.
  • hits is an array that includes the found items.

This is why you returned response.hits.hits in the search route, which houses the documents found.

Step 7 — Creating the HTML Template

First, create two new folders in your root folder named views and public which were referenced in the previous step. Next, create a file called template.html in the views folder and paste the following code:

views/template.html
<html>
<head>
  <!-- Bootstrap -->
  <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
  <!-- Axios -->
  <script src="https://unpkg.com/axios/dist/axios.min.js"></script>
  <!-- Vue -->
  <script src="https://cdn.jsdelivr.net/npm/vue"></script>
  <!--- Some styling for the page. -->
  <style>
    .search-form .form-group {
      float: right !important;
      transition: all 0.35s, border-radius 0s;
      width: 32px;
      height: 32px;
      background-color: #fff;
      box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
      border-radius: 25px;
      border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
      padding-right: 20px;
      border: 0 none;
      background: transparent;
      box-shadow: none;
      display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
      display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
      /* Firefox < 18 */
      display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
      /* Firefox 19+ */
      display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
      display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
      width: 100%;
      border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
      position: absolute;
      top: -1px;
      right: -2px;
      z-index: 2;
      display: block;
      width: 34px;
      height: 34px;
      line-height: 34px;
      text-align: center;
      color: #3596e0;
      left: initial;
      font-size: 14px;
    }
  </style>
</head>
<body>
  <div class="container" id="app">
    <div class="row">
      <div class="col-md-6 col-md-offset-3">
        <h1>Search Cities around the world</h1>
      </div>
    </div>
    <div class="row">
      <div class="col-md-4 col-md-offset-3">
        <form action="" class="search-form">
          <div class="form-group has-feedback">
            <label for="search" class="sr-only">Search</label>
            <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" />
            <span class="glyphicon glyphicon-search form-control-feedback"></span>
          </div>
        </form>
      </div>
    </div>
    <div class="row">
      <div class="col-md-3" v-for="result in results">
        <div class="panel panel-default">
          <div class="panel-heading">
            <!-- Display the City Name and Country. -->
            {{ result._source.name }}, {{ result._source.country }}
          </div>
          <div class="panel-body">
            <!-- Display the Latitude and Longitude of the City. -->
            <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
          </div>
        </div>
      </div>
    </div>
  </div>
</body>
</html>

In the code snippet, there are two main sections: HTML and CSS code.

In the HTML section, you required three different libraries:

  1. Bootstrap CSS for styling the page.
  2. Axios for making HTTP requests to our server.
  3. Vue.js is a minimalistic framework that you will use for the view.

In the CSS section, you have applied styling to make the search input hide and reveal itself once you hover over the search icon.

Next, there is an input for the search box that you assigned its v-model to query (this will be used by Vue.js). After this, you looped through all our results.

Run the node index.js command and then visit http://localhost:3001/ in a browser. You will see the app’s landing page.

Search app landing page

Next, add a script tag in your template.html file:

views/template.html
// Create a new Vue instance.
var app = new Vue({
  el: '#app',
  // Declare the data for the component. An array that houses the results and a query that holds the current search string.
  data: {
    results: [],
    query: '',
  },
  // Declare methods in this Vue component. Here only one method which performs the search is defined.
  methods: {
    // Make an axios request to the server with the current search query.
    search: function () {
      axios
        .get('http://127.0.0.1:3001/search?q=' + this.query)
        .then((response) => {
          this.results = response.data;
        });
    },
  },
  // Declare Vue watchers.
  watch: {
    // Watch for change in the query string and recall the search method.
    query: function () {
      this.search();
    },
  },
});

In this section, you declared a new instance of Vue, mounting it on the element with the id of app. You declared data properties which include query, which you had attached to the search input, and results, which is an array of all results found.

In the methods quota, you have just one function called search, which triggers a GET request to the search route. This passes along the current input in the search box. This in turn returns a response that is then looped in the HTML code block.

Finally, you use watchers in Vue.js, which performs an action anytime data being watched changes. Here, you are watching for a change in the query data, and once it changes, the search method is fired.

If you re-run the node index.js command now and navigate to http://localhost:3001/ again:

Completed search app displaying cities as text is entered in the input field

Step 8 — Searching from the Client Side

If you do not want to send requests to the server every time a search occurs, you can search the Elasticsearch engine from the client-side. Some developers might not be comfortable with hitting their servers for every search term, while some feel it’s more secure to search from the server-side.

Elasticsearch offers a browser build that can make searches. This step will guide you through searching from the browser client.

First, add a new route to your Express file and restart your server:

index.js
// Declare a new route. This route serves a static HTML template called template2.html.
app.get('/v2', function (req, res) {
  res.sendFile('template2.html', {
    root: path.join(__dirname, 'views'),
  });
});

In this code block, you created a new route for the URL at /v2 to return a static HTML file called template2.html. You will create this file soon.

Next, you need to download the client library for Elasticsearch.

Note: This tutorial uses version 13.3.1.

After downloading, extract and copy elasticsearch.min.js to the public folder in your application root.

Note: If you encounter CORS issues after connecting to the Elasticsearch engine from the client-side, and the following snippet to the end of the Elasticsearch configuration file. You can locate this file by reviewing Elasticsearch’s reference material.

/etc/elasticsearch/elasticsearch.yml
# file location may differ according to your environment
http.cors.enabled : true
http.cors.allow-origin : "*"

After that is done, restart your Elasticsearch instance:

  1. # command may differ according to your environment
  2. sudo service elasticsearch restart

Next, create a file called template2.html in your views folder and this code:

views/template2.html
<!doctype html>
<html>
<head>
  <!-- Bootstrap -->
  <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
  <!-- Vue -->
  <script src="https://cdn.jsdelivr.net/npm/vue"></script>
  <!--- Some styling for the page. -->
  <style>
    .search-form .form-group {
      float: right !important;
      transition: all 0.35s, border-radius 0s;
      width: 32px;
      height: 32px;
      background-color: #fff;
      box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
      border-radius: 25px;
      border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
      padding-right: 20px;
      border: 0 none;
      background: transparent;
      box-shadow: none;
      display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
      display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
      /* Firefox < 18 */
      display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
      /* Firefox 19+ */
      display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
      display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
      width: 100%;
      border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
      position: absolute;
      top: -1px;
      right: -2px;
      z-index: 2;
      display: block;
      width: 34px;
      height: 34px;
      line-height: 34px;
      text-align: center;
      color: #3596e0;
      left: initial;
      font-size: 14px;
    }
  </style>
</head>
<body>
  <div class="container" id="app">
    <div class="row">
      <div class="col-md-6 col-md-offset-3">
        <h1>Search Cities around the world</h1>
      </div>
    </div>
    <div class="row">
      <div class="col-md-4 col-md-offset-3">
        <form action="" class="search-form">
          <div class="form-group has-feedback">
            <label for="search" class="sr-only">Search</label>
            <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" />
            <span class="glyphicon glyphicon-search form-control-feedback"></span>
          </div>
        </form>
      </div>
    </div>
    <div class="row">
      <div class="col-md-3" v-for="result in results">
        <div class="panel panel-default">
          <div class="panel-heading">
            <!-- Display the City Name and Country. -->
            {{ result._source.name }}, {{ result._source.country }}
          </div>
          <div class="panel-body">
            <!-- Display the Latitude and Longitude of the City. -->
            <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
          </div>
        </div>
      </div>
    </div>
  </div>
  <!-- Elasticsearch -->
  <script src="/elasticsearch.min.js"></script>
</body>
</html>

Next, add a script tag in your template2.html file and add this code:

views/template2.html
// Instantiate a new Elasticsearch client like you did on the client.
var client = new elasticsearch.Client({
  hosts: ['http://127.0.0.1:9200'],
});

// Create a new Vue instance.
var app = new Vue({
  el: '#app',
  // Declare the data for the component. An array that houses the results and a query that holds the current search string.
  data: {
    results: [],
    query: '',
  },
  // Declare methods in this Vue component. Here only one method which performs the search is defined.
  methods: {
    // Function that calls the Elasticsearch. Here the query object is set just as that of the server.
    // Here the query string is passed directly from Vue.
    search: function () {
      var body = {
        size: 200,
        from: 0,
        query: {
          match: {
            name: this.query,
          },
        },
      };
      // Search the Elasticsearch passing in the index, query object, and type.
      client
        .search({
          index: 'example_index',
          body: body,
          type: 'cities_list',
        })
        .then((results) => {
          console.log(`Found ${results.hits.total} items in ${results.took}ms`);
          // Set the results to the result array we have.
          this.results = results.hits.hits;
        })
        .catch((err) => {
          console.log(err);
        });
    },
  },
  // Declare Vue watchers.
  watch: {
    // Watch for change in the query string and recall the search method.
    query: function () {
      this.search();
    },
  },
});

The HTML and JavaScript snippet is similar to the one in the previous step, with the main differences being as follows:

  • You did not require Axios, you required elasticsearch.min.js instead.
  • At the top of the script tag, you initiated the Elasticsearch client as it was done on the server-side.
  • The search method does not perform an HTTP request, but rather searches the Elasticsearch engine as done in the search route on the server-side.

Browse to http://localhost:3001/v2:

Client side search app displaying cities as text is entered in the input field

Conclusion

In this tutorial, you used Elasticsearch to index data. You have also implemented a real-time search using the client library for Elasticsearch.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors
Default avatar
Ogundipe Samuel Ayo

author

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
4 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

thank u

I’ve been looking for an easy Node.js/ElasticSearch sample - this worked for me. I created a GitHub repo for it so I can easily return to the code.

What should I do next, after I’ve commanded “npm init”, and got these sentences?

‘’'This utility will walk you through creating a package.json file. It only covers the most common items, and tries to guess sensible defaults.

See npm help init for definitive documentation on these fields and exactly what they do.

Use npm install <pkg> afterwards to install a package and save it as a dependency in the package.json file.

Press ^C at any time to quit. package name: (elastic-node) ‘’’

Hey there folks :) I have this app setup on a digital ocean droplet, but I cannot update the json file. does anyone know how this is done? I cannot just hit node data.js cause nothing happens. I have it setup via codeanywhere and the server on digitalocean is running it would be really great if you guys at digital ocean could review this tutorial in order to make it work properly on the droplets

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel