Web Scraping

What is it and how it works

Web scraping has been around for as long as the Web itself. Although it is often associated with web content extraction, it has not always served this purpose. This technique was first developed as a mean to automate complicated or painful tasks.

One of the first use of web scraping is linked with testing frameworks. By using tools such as Selenium, companies such as Ip-Label have built products that enable web developers and web masters to monitor a website’s performance on daily a basis.

Today, scraping websites is best known by digital marketing services inside (tech) startups, thanks to the rise of Growth Hacking. Indeed, it is the perfect mean to automate tasks such as collecting prospect data or automate marketing actions like posting a tweet or following someone on a social network.

Our ambition and mission is to make business data accessible.

Web Scraping

How to start a data extraction project?

In 5 simple steps

1. Define what you need

Start with the basic task of defining exactly what you are trying to achieve : are you looking to drive KPIs, enrich a business database to strengthen your product, etc.

2. Identify target websites

Once you know what kind of data you need, you can identify web sources.

It is important to do this BEFORE creating a structured data schema.

Indeed, once you have selected all the sources you wish to extract data from, you will be able to create a nice JSON document (your template / schema).

It looks like the following :

JSON Document

3. Architect your workflow

Decide how you launch the bot(s): is it manually, from a defined scheduled, triggered by an event from your application?

Also, take a look at how you will integrate data later on: sometimes crawling websites require to wait a very long time, especially if the crawl is setup for multiple websites at once.

If you use a cloud platform, you won’t have to maintain servers or dependencies, which can be a huge pain.

4. Build the bot(s)

Almost done – now you have to code the program that will make your bot alive!

You can use pretty much any programming language, although we do recommend either Python with the great Scrapy library or JavaScript with the Puppeteer library maintained by Google.

5. Integrate your data

Make sure the quality is top notched and that you are not left with tabulations or useless characters.

MongoDB is a great database to dump JSON documents, but you’re free to use anything!

Building a scraper

How to build a web scraper?

Extracting and automating can be fairly easy

1. Analyze the website

The first step consists of analyzing the website’s structure.

Open your web browser and use the “Inspect Tab”.

A website is like a tree made of nodes. Nodes are XPath : they define the path you need to use to get the data you want.

You should also check out incoming requests by opening the “Network” tab.

The website could use an API or load AJAX, which could simplify (or not) the extraction process.

2. Build the template and your schema

Once you spotted the nodes you wish to extract data from (and remember, all the other target websites you need to extract data from), you can build the schema (object) you will save in your database.

Having a unified schema across different sites allows you to simplify the integration process later on.

3. Code the bot

The “fun” part begin. Although it can be very fun to scrape websites, it happens that many challenges arise while coding.

Websites use more and more protection techniques (Cloudflare, Datadome. etc.) so you might not succeed.

4. Avoid to be detected

Always remember to use a different IP than your server’s.

Getting banned from a website happens way faster than you could imagine 🙂