In this tutorial, we will scrape the front page of Hacker News to get all the top ranking links as well as their metadata - such as the title, URL and the number of points/comments it received. This is one of many techniques to extract data from web pages using node.js and mainly uses a module called cheerio by Matthew Mueller which implements a subset of jQuery specifically designed for server use.
Cheerio is lightweight, fast, flexible and easy to use, if you're already accustomed to working with jQuery. We will also make use of Mikael Rogers' excellent request module as a simplified HTTP client.
I will assume that you're already familiar with node.js, jQuery and basic Linux administrative tasks like connecting to your VPS using SSH.
If you're unfamiliar with node.js or if you haven't installed it yet, please refer to the Articles & Tutorials section above to find installation instructions for your operating system.
To install the required modules using NPM, simply type the following command:
npm install request cheerio
This will install the modules in your current working directory only.
To install the modules globally run:
npm install -g request cheerio
Create a file called scrape.js
and add the following lines:
var request = require('request'); var cheerio = require('cheerio');
This will load all of our module dependencies.
Now we'll load the front page of Hacker News with a simple request and display the HTML code of the page, if no error occurs and the HTTP status code equals 200.
Append these lines to the file:
request('https://news.ycombinator.com', function (error, response, html) { if (!error && response.statusCode == 200) { console.log(html); } });
Try running the script using node scrape.js
and you
should see the HTML code being logged in your terminal
window.
In order to know how to extract our desired meta-data, we need to know how the elements are structured within the HTML code. A preferred way is to use the web-developer tools built into Google Chrome to inspect the desired target element on the web page simply by right clicking on it and selecting "Inspect element".
In this example, I've opened Hacker News in Chrome, right-clicked and inspected the title of the top ranking story:
After a quick look at the web-developer console, I came to the conclusion that we could simply select each "span" element which has a class added to it named "comhead" and select the "a" element above it by using the jQuery API function prev() to easily parse all 30 top ranking links and their titles.
We can test my assumption by changing our request code to this:
request('https://news.ycombinator.com', function (error, response, html) { if (!error && response.statusCode == 200) { var $ = cheerio.load(html); $('span.comhead').each(function(i, element){ var a = $(this).prev(); console.log(a.text()); }); } });
As expected, by running the code we get a list of 30 titles. Let's modify it a bit more to parse the remaining metadata:
request('https://news.ycombinator.com', function (error, response, html) { if (!error && response.statusCode == 200) { var $ = cheerio.load(html); $('span.comhead').each(function(i, element){ var a = $(this).prev(); var rank = a.parent().parent().text(); var title = a.text(); var url = a.attr('href'); var subtext = a.parent().parent().next().children('.subtext').children(); var points = $(subtext).eq(0).text(); var username = $(subtext).eq(1).text(); var comments = $(subtext).eq(2).text(); // Our parsed meta data object var metadata = { rank: parseInt(rank), title: title, url: url, points: parseInt(points), username: username, comments: parseInt(comments) }; console.log(metadata); }); } });
Select the previous element:
var a = $(this).prev();
Get the rank by parsing the element two levels above the "a" element:
var rank = a.parent().parent().text();
Parse the link title:
var title = a.text();
Parse the href attribute from the "a" element:
var url = a.attr('href');
Get the subtext children from the next row in the HTML table:
var subtext = a.parent().parent().next().children('.subtext').children();
Extract the relevant data from the children:
var points = $(subtext).eq(0).text(); var username = $(subtext).eq(1).text(); var comments = $(subtext).eq(2).text();
Running the modified script should output an array of objects like this:
[ { rank: 1, title: 'The Meteoric Rise of DigitalOcean ', url: 'http://news.netcraft.com/archives/2013/06/13/the-meteoric-rise-of-digitalocean.html', points: 240, username: 'beigeotter', comments: 163 }, { rank: 2, title: 'Introducing Private Networking', url: 'https://www.digitalocean.com/blog_posts/introducing-private-networking', points: 172, username: 'Goranek', comments: 75 }, ...
That's it! You can now store the the extracted data in a database like MongoDB or Redis to further process it. Here is the full source code of our scrape.js file:
var request = require('request'); var cheerio = require('cheerio'); request('https://news.ycombinator.com', function (error, response, html) { if (!error && response.statusCode == 200) { var $ = cheerio.load(html); var parsedResults = []; $('span.comhead').each(function(i, element){ // Select the previous element var a = $(this).prev(); // Get the rank by parsing the element two levels above the "a" element var rank = a.parent().parent().text(); // Parse the link title var title = a.text(); // Parse the href attribute from the "a" element var url = a.attr('href'); // Get the subtext children from the next row in the HTML table. var subtext = a.parent().parent().next().children('.subtext').children(); // Extract the relevant data from the children var points = $(subtext).eq(0).text(); var username = $(subtext).eq(1).text(); var comments = $(subtext).eq(2).text(); // Our parsed meta data object var metadata = { rank: parseInt(rank), title: title, url: url, points: parseInt(points), username: username, comments: parseInt(comments) }; // Push meta-data into parsedResults array parsedResults.push(metadata); }); // Log our finished parse results in the terminal console.log(parsedResults); } });
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Sign up for Infrastructure as a Newsletter.
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
For the part
have a much easier approach,
What I did here is get the feedback of the
elements
and remove all white spaces, as sometimes websites like fedex tracking website contains a lot of white spaces.Newer versions of Cheerio are producing single quotes around attributes. When I use the script to generate JSON this renders the file invalid.
Any way to fix this?
Is it possible to submit html form that got from cheerio?
Okay, I’m currently using this template to scrape data on pending bills on https://www.whitehouse.gov/briefing-room/pending-legislation. I scraped the title, but the url from the tittle is not scraping at all with the title? What am doing wrong to retrieve the url href?
That helped me a lot ! :) Thanks !
This comment has been deleted
How to deal with content added in by Javascript?
Yea, got me confused as well. So will it violate the TOS to use scraping script?
In your TOS you’ve stated that:
“You shall not: (v) use manual or automated software, devices, or other processes to “crawl” or “spider” any page of the Website; (vi) harvest or scrape any Content from the Services;”, but on the other hand - you’re supplying the users with a complete guide to web scraping.
I’m confused, DO.
It seems that in fact Cheerio works, sorry for this comment!