Hi,
Thanks for the very good tutorial. I am trying to return the data instead of saving it in a file. This, so I can call this script from another js file and use the data there.
I tried in index.js to add return scraperController(browserInstance)
and in the pageController.js use a return statement in the try section. This doesn’t work however.
Can you give me any hints?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Accepted Answer
Hi there,
I believe that you should be able to return the scraped data instead of writing it to a file by just returning the scrapedData
object from the scrapeAll
function, eg:
const pageScraper = require('./pageScraper');
async function scrapeAll(browserInstance) {
let browser;
try {
browser = await browserInstance;
let scrapedData = {};
scrapedData['Travel'] = await pageScraper.scraper(browser, 'Travel');
scrapedData['HistoricalFiction'] = await pageScraper.scraper(browser, 'Historical Fiction');
scrapedData['Mystery'] = await pageScraper.scraper(browser, 'Mystery');
await browser.close();
console.log("The data has been scraped successfully!");
return scrapedData;
} catch (err) {
console.log("Could not resolve the browser instance => ", err);
}
}
module.exports = (browserInstance) => scrapeAll(browserInstance);
After that, you can import the function to other files and call it to return the scrapedData
object, so that you can use in other methods, eg:
const scrapeAll = require('./path/to/your/scrapeAllFile');
const browserInstance = ...; // Set up your browser instance here (e.g., using Puppeteer)
(async () => {
const scrapedData = await scrapeAll(browserInstance);
console.log('Scraped data:', scrapedData);
// Use the scraped data in other methods
})();
Let me know if you hit any problems!
Best,
Bobby
It is about [this](https://www.digitalocean.com/community/tutorials/how-to-scrape-a-website-using-node-js-and-puppeteer](https://www.digitalocean.com/community/tutorials/how-to-scrape-a-website-using-node-js-and-puppeteer) tutorial by the way (page scraping with nodejs)
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.