Share

Question

Have a very strange problem. Since 1-2 weeks I am unable to create screenshots of a Craigslist pages with PhantomJS on Digital Ocean. Does anybody have any idea what the issue could be and why it does not work anymore?

It always worked fine before and still works totally fine when I run it locally on my notebook or on linode. It creates the screenshot within 2-4 seconds. However running the same (no matter if in a special Docker-Container like the one used locally and linode or installed directly on the host) on Digital Ocean keeps on loading forever and if I am lucky and wait long enough I get the screenshot after around 7-10+ minutes.

Tried it on different Droplets (existing & totally new) in different zones (SF & Frankfurt) but always have the same issue. Already contacted Digital Ocean about that. They were able to reproduce the issue but according to them nothing changed on their side and have so also no idea what could cause that. They blame PhantomJS or Craigslist.

It can be reproduced very easily. On a new Droplet (Ubuntu 14.04) the following code will install PhantomJS:

# Install dependencies
sudo apt-get install -y libicu52 libjpeg8 libfontconfig libwebp5 

# Install PhantomJS
cd /usr/local/share && \
curl -L -O https://github.com/bprodoehl/phantomjs/releases/download/v2.0.0-20150528/phantomjs-2.0.0-20150528-u1404-x86_64.zip && \
unzip phantomjs-2.0.0-20150528-u1404-x86_64.zip && \
ln -s /usr/local/share/phantomjs-2.0.0-20150528/bin/phantomjs /usr/local/bin/phantomjs

A very basic example script to create a screenshot of a product on craigslist. File called "test-screenshot.js" with this content:

var page = require('webpage').create();
var url = 'http://vancouver.craigslist.ca/van/ctd/5162270100.html';
page.open(url, function() {
  page.render('craigslist.png');
  phantom.exit();
});

To run the script: "phantomjs test-screenshot.js".

Does anybody have any idea what is going on?

Thanks!

Accepted Answer

chrisallick October 4, 2015

Hi, I'm about to go through the process of getting PhantomJS/CasperJS running on an Ubuntu 14 droplet.

Questions:

1) Have you gotten this running locally? I'm on OSX and it looks like you're not initializing thing totally correct maybe?
2) I haven't gotten this far yet, but I'm wondering if there are missing dependencies for rendering the page. Like, for example on os x maybe quartz or something is used.
3) Maybe you need to specify the size of the image?

Reply

janober October 4, 2015

Hi!

1) Yes was always running perfectly locally and on servers of other providers
2) No can not be because the same container works everywhere else and with any other page (also on DO). Btw. I advice you to also pack everything in Docker containers. Everything normally then simply works no matter where you run it. You then have to worry just once about the OSX issues (with Docker) and never ever again for anything else. Also can you use an already set up PhantomJS/CasperJS Container and you save so the whole setup process and everything will just work.
3) No. The problem is and never was the code. It always worked perfectly with any other website and on servers anywhere else.

It turned out that really Digital Ocean is the problem. No matter on which of their servers I did run it I always had the same issue. Other people also confirmed that. One for example did set up a VPN on one of their servers and had also issues with visiting Craigslist. I moved in the end to Vultr and it is working totally fine now without any changes (as it did locally, at Linode and Hetzner). Also save some money now because its cheaper so kind of won in the end... Still sad because lost a lot of time because of them.
- chrisallick October 4, 2015
  
  No, I got it working :)
- rushtonmd October 24, 2015
  
  I believe I have a similar problem with DigialOcean and Craigslist. I had a small app hitting craigslist for months working fine, then in August 2015 (I believe) is just stopped working. I did some curl timings for sfbay.craigslist.com and here are the results.
  
  time_namelookup: 0.004 time_connect: 0.010 time_appconnect: 0.000 time_pretransfer: 0.010 time_redirect: 0.000 time_starttransfer: 1.724 ---------- time_total: 210.691
  
  It turned out my app was still technically working, but it takes an eternity to get results back. Wish I knew what happened, it's just annoying and pretty much renders my app worthless.
  
  janober October 24, 2015
  
  I realized that there was a problem end of July/beginning of August. Cost me a lot of time and nerves. Guess you have to do the same what I did, change to another provider because it did not look like that DigitalOcean is interested into investigating and fixing that issue.
  
  chrisallick October 24, 2015
  
  There is no problem. It works fine.
  
  njuljsong November 27, 2015
  
  Hi Chrisallick, also check out this post http://stackoverflow.com/questions/31776118/unable-create-screenshot-of-craiglist-with-phantomjs-on-digital-ocean, it is easy to reproduce the issue by just loading craigslist homepage in DigitalOcean droplet, or just do curl -X GET http://newyork.craigslist.org, and see how long it takes to load.
  
  njuljsong November 27, 2015
  
  Hi guys I have the same issue, trying to fetch ad data from craigslist in my DigitalOcean droplet, but it is extremely slow. The same code is able to fetch some other website's data very fast without issue.
  
  Hope they can fix this in a timely manner

Answer 2

janober October 4, 2015

Hi!

1) Yes was always running perfectly locally and on servers of other providers
2) No can not be because the same container works everywhere else and with any other page (also on DO). Btw. I advice you to also pack everything in Docker containers. Everything normally then simply works no matter where you run it. You then have to worry just once about the OSX issues (with Docker) and never ever again for anything else. Also can you use an already set up PhantomJS/CasperJS Container and you save so the whole setup process and everything will just work.
3) No. The problem is and never was the code. It always worked perfectly with any other website and on servers anywhere else.

It turned out that really Digital Ocean is the problem. No matter on which of their servers I did run it I always had the same issue. Other people also confirmed that. One for example did set up a VPN on one of their servers and had also issues with visiting Craigslist. I moved in the end to Vultr and it is working totally fine now without any changes (as it did locally, at Linode and Hetzner). Also save some money now because its cheaper so kind of won in the end... Still sad because lost a lot of time because of them.
- chrisallick October 4, 2015
  
  No, I got it working :)
- rushtonmd October 24, 2015
  
  I believe I have a similar problem with DigialOcean and Craigslist. I had a small app hitting craigslist for months working fine, then in August 2015 (I believe) is just stopped working. I did some curl timings for sfbay.craigslist.com and here are the results.
  
  time_namelookup: 0.004 time_connect: 0.010 time_appconnect: 0.000 time_pretransfer: 0.010 time_redirect: 0.000 time_starttransfer: 1.724 ---------- time_total: 210.691
  
  It turned out my app was still technically working, but it takes an eternity to get results back. Wish I knew what happened, it's just annoying and pretty much renders my app worthless.
  
  janober October 24, 2015
  
  I realized that there was a problem end of July/beginning of August. Cost me a lot of time and nerves. Guess you have to do the same what I did, change to another provider because it did not look like that DigitalOcean is interested into investigating and fixing that issue.
  
  chrisallick October 24, 2015
  
  There is no problem. It works fine.
  
  njuljsong November 27, 2015
  
  Hi Chrisallick, also check out this post http://stackoverflow.com/questions/31776118/unable-create-screenshot-of-craiglist-with-phantomjs-on-digital-ocean, it is easy to reproduce the issue by just loading craigslist homepage in DigitalOcean droplet, or just do curl -X GET http://newyork.craigslist.org, and see how long it takes to load.
  
  njuljsong November 27, 2015
  
  Hi guys I have the same issue, trying to fetch ad data from craigslist in my DigitalOcean droplet, but it is extremely slow. The same code is able to fetch some other website's data very fast without issue.
  
  Hope they can fix this in a timely manner

Accepted Answer

chrisallick October 4, 2015

Follow these instructions for installation and setup. This will get you:

node 4.0.0
npm 3.3.5
phantomjs 1.9.8
casperjs 1.1.0-beta3

Can confirm that the following code works:
casper.then(function() {
casper.capture("confirm_purchase.png",{
top: 0,
left: 0,
width: 768,
height: 768
});
});

ubuntu 14.04:
sudo apt-get update
sudo apt-get install git

nodejs:
https://github.com/nodejs/node-v0.x-archive/wiki/Installing-Node.js-via-package-manager?utm_source=%5Bdeliciuos%5D&utm_medium=twitter#debian-and-ubuntu-based-linux-distributions
https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-an-ubuntu-14-04-server

npm:
http://stackoverflow.com/questions/23393707/how-to-update-npm
https://docs.npmjs.com/getting-started/installing-node

phantomjs:
https://gist.github.com/julionc/7476620

casperjs:
http://docs.casperjs.org/en/latest/installation.html

How To Install Node.js on an Ubuntu 14.04 server

by Justin Ellingwood

Node.js is a popular Javascript platform that can be used to build web applications quickly and easily. Node.js uses Javascript on both the front-end and back-end to create a more consistent experience for developers. In this guide, we'll get you up and running with Node.js on an Ubuntu 14.04 server.

Reply

Unable create Screenshot of Craiglist with PhantomJS on Digital Ocean

Leave a Comment

Share

Share your Question

Unable create Screenshot of Craiglist with PhantomJS on Digital Ocean

Leave a Comment

Related Questions

Almost there!

Report a Bug