Unable create Screenshot of Craiglist with PhantomJS on Digital Ocean

August 8, 2015 5.3k views
Miscellaneous DigitalOcean Docker Networking
janober
By:
janober

Have a very strange problem. Since 1-2 weeks I am unable to create screenshots of a Craigslist pages with PhantomJS on Digital Ocean. Does anybody have any idea what the issue could be and why it does not work anymore?

It always worked fine before and still works totally fine when I run it locally on my notebook or on linode. It creates the screenshot within 2-4 seconds. However running the same (no matter if in a special Docker-Container like the one used locally and linode or installed directly on the host) on Digital Ocean keeps on loading forever and if I am lucky and wait long enough I get the screenshot after around 7-10+ minutes.

Tried it on different Droplets (existing & totally new) in different zones (SF & Frankfurt) but always have the same issue. Already contacted Digital Ocean about that. They were able to reproduce the issue but according to them nothing changed on their side and have so also no idea what could cause that. They blame PhantomJS or Craigslist.

It can be reproduced very easily. On a new Droplet (Ubuntu 14.04) the following code will install PhantomJS:

# Install dependencies
sudo apt-get install -y libicu52 libjpeg8 libfontconfig libwebp5 

# Install PhantomJS
cd /usr/local/share && \
curl -L -O https://github.com/bprodoehl/phantomjs/releases/download/v2.0.0-20150528/phantomjs-2.0.0-20150528-u1404-x86_64.zip && \
unzip phantomjs-2.0.0-20150528-u1404-x86_64.zip && \
ln -s /usr/local/share/phantomjs-2.0.0-20150528/bin/phantomjs /usr/local/bin/phantomjs

A very basic example script to create a screenshot of a product on craigslist. File called "test-screenshot.js" with this content:

var page = require('webpage').create();
var url = 'http://vancouver.craigslist.ca/van/ctd/5162270100.html';
page.open(url, function() {
  page.render('craigslist.png');
  phantom.exit();
});

To run the script: "phantomjs test-screenshot.js".

Does anybody have any idea what is going on?

Thanks!

1 comment
  • scott599459 December 19, 2015

    I believe Craigslist is blocking DigitalOcean IPs from accessing the site. I've heard they do it for AWS as well.

Log In to Comment
2 Answers
chrisallick October 4, 2015

Hi, I'm about to go through the process of getting PhantomJS/CasperJS running on an Ubuntu 14 droplet.

Questions:

1) Have you gotten this running locally? I'm on OSX and it looks like you're not initializing thing totally correct maybe?
2) I haven't gotten this far yet, but I'm wondering if there are missing dependencies for rendering the page. Like, for example on os x maybe quartz or something is used.
3) Maybe you need to specify the size of the image?

Reply
  • janober October 4, 2015

    Hi!

    1) Yes was always running perfectly locally and on servers of other providers
    2) No can not be because the same container works everywhere else and with any other page (also on DO). Btw. I advice you to also pack everything in Docker containers. Everything normally then simply works no matter where you run it. You then have to worry just once about the OSX issues (with Docker) and never ever again for anything else. Also can you use an already set up PhantomJS/CasperJS Container and you save so the whole setup process and everything will just work.
    3) No. The problem is and never was the code. It always worked perfectly with any other website and on servers anywhere else.

    It turned out that really Digital Ocean is the problem. No matter on which of their servers I did run it I always had the same issue. Other people also confirmed that. One for example did set up a VPN on one of their servers and had also issues with visiting Craigslist. I moved in the end to Vultr and it is working totally fine now without any changes (as it did locally, at Linode and Hetzner). Also save some money now because its cheaper so kind of won in the end... Still sad because lost a lot of time because of them.

    • chrisallick October 4, 2015

      No, I got it working :)

    • rushtonmd October 24, 2015

      I believe I have a similar problem with DigialOcean and Craigslist. I had a small app hitting craigslist for months working fine, then in August 2015 (I believe) is just stopped working. I did some curl timings for sfbay.craigslist.com and here are the results.

          time_namelookup:  0.004
    time_connect:  0.010
    time_appconnect:  0.000
    time_pretransfer:  0.010
    time_redirect:  0.000
    time_starttransfer:  1.724
                            ----------
    time_total:  210.691

      It turned out my app was still technically working, but it takes an eternity to get results back. Wish I knew what happened, it's just annoying and pretty much renders my app worthless.

chrisallick October 4, 2015

Follow these instructions for installation and setup. This will get you:

node 4.0.0
npm 3.3.5
phantomjs 1.9.8
casperjs 1.1.0-beta3

Can confirm that the following code works:
casper.then(function() {
casper.capture("confirm_purchase.png",{
top: 0,
left: 0,
width: 768,
height: 768
});
});

ubuntu 14.04:
sudo apt-get update
sudo apt-get install git

nodejs:
https://github.com/nodejs/node-v0.x-archive/wiki/Installing-Node.js-via-package-manager?utm_source=%5Bdeliciuos%5D&utm_medium=twitter#debian-and-ubuntu-based-linux-distributions
https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-an-ubuntu-14-04-server

npm:
http://stackoverflow.com/questions/23393707/how-to-update-npm
https://docs.npmjs.com/getting-started/installing-node

phantomjs:
https://gist.github.com/julionc/7476620

casperjs:
http://docs.casperjs.org/en/latest/installation.html

How To Install Node.js on an Ubuntu 14.04 server
by Justin Ellingwood
Node.js is a popular Javascript platform that can be used to build web applications quickly and easily. Node.js uses Javascript on both the front-end and back-end to create a more consistent experience for developers. In this guide, we'll get you up and running with Node.js on an Ubuntu 14.04 server.
Reply
Have another answer? Share your knowledge.

Related Questions