Install Scrapy on Mint/Ubuntu/Debian behind a Proxy

This article shows you how to install Scrapy - a python web crawling toolbox, under debian/ubuntu or linux mint.

use apt-get with proxy:

http_proxy=http://domain\user:pass@hostname:port apt-get install packagename

or set the environment variable upfront

export http_proxy=http://domain\user:pass@hostname:port

Use pip with proxy:

pip install --proxy="http://domain\user:pass@hostname:port" packagename

Now step by step:

Set proxy:
export http_proxy=http://domain\user:pass@hostname:port

Install pip:
apt-get install python-pip

Install service_identity - specify proxy if needed:
pip install service_identity

Install pyOpenSSL:
pip install pyOpenSSL

Install python developer files - needed for the twisted framework:
apt-get install python2.7-dev

Install Scrapy:
pip install Scrapy

Run Scrapy to test if it works:
scrapy

If any errors are displayed - you may have some missing packages to install.