Web Scraping for Data

Learn some automation and automate your life ;)

Introduction

So excited to see you have joined the birds nest 🪶 and become 1 step closer to learning how to code. Let’s get right into it!

Note that this is a little bit of an intermediate level post so we will fly through some of the concepts pretty quick.

My goal is to show you the beauty of coding and how it can help you in your every day life.

Let’s begin!

There was an interesting question in the Left Curve Capital discord this morning.

Let’s try to pull data from coinmarketcap for our own use. To begin we will be using a NodeJS library called Puppeteer. It gives you the ability to control everything on Chrome. You can do things like: opening up a webpage, moving the mouse, take screenshots, generate PDF’s… and the list goes on.

Here’s the finished product

Setup

First you’ll need to have NodeJS installed and have a working development environment and code editor. *Thursday’s post is going to walk you through this step by step so I’d recommend subbing to the paid substack now ;)*

But don’t worry you can still follow along below

Create and initialize the project using

$ npm init

Installing dependencies

Now we can install our dependencies.

$ npm install puppeteer --save

Puppeteer is a node package that makes web scraping extremely easy.

Let’s create a simple web scraper to scrape data from coinmarketcap.

Here’s what we need our scraper to do:

1. Open up a Chrome browser and go to https://coinmarketcap.com/

2. Go to the specific token we want (in this case FTM)

3. extract FTM price and market cap

Alright let’s get to it!

Lets break it down;

Lines 6 and 7 tells puppeteer to open up chrome.

Lines 9 and 10 are two variables. You can change this to the token of your choice.

Line 14 tells puppeteer to go to the website on line 9.

Lines 17 through 27 use’s css selectors to select a certain element on the webpage. In this case we’re using the class name of priceValue and statsValue to get the price and marketcap of FTM.

Lines 30 and 31 output the data to our console and close the browser.

---

Finally use this command to run the file:

node crypto-scraper.js

After running this file you should see this in your terminal

---

All Done

Now it took me about 20 minutes to make this scraper. Do you have a data entry job? Have anything that you can automate in your life? Need to make a trading bot @BowTiedArcticFox ;) ? Puppeteer is the perfect tool for you.

I want to emphasize that the code may be confusing to some of you but that’s what my paid substack is for!