It's the week of Black Friday and I want to share with you how I came up with it. My girlfriend and I wanted an airfryer, so I decided to check how the price will evolve until Friday. This was the only thing I wanted to track, but trying it with another products I like, I found out that it wasn't that simple 🧐.
In this first part, I'll be going through the code and how I came up with it. To begin, this is the airfryer I wanted to track (I still am).
I started to look for something already built and found this website. This is the adaptation I made for it:
import fetch from "node-fetch";
import * as cheerio from "cheerio";
import fs from "fs";
import moment from "moment";
const formattedPrice = []; // 👈 We'll need this later on for the result
// 👇 Page fetching
const cecofry6000 = await fetch(
"https://cecotec.es/es/freidoras-sin-aceite/cecofry-experience-6000",
);
const body = await cecofry6000.text(); // 👈 Transform code to text
const web = cheerio.load(body); // 👈 Rebuild website
const price = web("h3.css-bxi7sc").text(); // 👈 Look for the price element
const webPrice = Array.from(price); // 👈 Transform price to array
// 👇 Now we iterate through the array and delete everything except numbers and commas
for (let i = 0; i < webPrice.length; i++) {
if (Number(webPrice[i]) || webPrice[i] === "," || webPrice[i] === "0") {
// 👇 We push what we need to the array mentioned before
formattedPrice.push(webPrice[i]);
} else {
webPrice.splice(i, 1);
}
}
// 👇 We add the current date to the beginning array and add a new line
formattedPrice.unshift(moment().format("DD-MM-YYYY - "));
formattedPrice.push("\n");
const result = formattedPrice.join(""); // 👈 Join the array
fs.appendFile("trackings-test.txt", result, function (err, data) {}); // 👈 Add result to text file
Results were coming as expected after repeated tests 💪.
Until I tried with another product. The lite sling from Bellroy. After trying with this, I knew it won't work with most of the pages 😭.
Why?
Because the content was built dynamically while the page loads and with node fetch, I was getting the first content shown in the webpage. The problem with that came with the price not loaded when the fetching happened, and the result of that is an empty array.
I decided to look for another way of doing it. I thought 🤔,
Instead of getting the price and all that stuff, why don't I make a screenshot of the whole page?
That's what I did. I started looking for solutions and found this website with many options available. After trying the first, second and third option, I decided to take a closer look to this third one. I saw this line:
let image = await driver.takeScreenshot();
Then I thought 🤔,
If there's a function to make a screenshot, there should be a function to get the source code right?
Turns out I was right, the function is called getPageSource().
Right after this, I decided to make a combination of the first option and this second one, so I could get any price (with selenium-webdriver page loads completely, so price will show up). This is the result:
const moment = require("moment");
const { Builder } = require("selenium-webdriver");
const cheerio = require("cheerio");
const fs = require("fs");
require("chromedriver");
// 👇 Exact url to the product
const url =
"https://bellroy.com/products/lite-sling?color=shadow&material=diamond_ripstop";
async function getPrice(url) {
// 👇 Wait for browser to build and launch properly
let driver = await new Builder().forBrowser("chrome").build();
await driver.get(url); // 👈 Navigate to the url passed in
// 👇 Get the source code completed
let source = await driver.getPageSource();
const web = cheerio.load(source); // 👈 Rebuild website
const price = web("span.price_value").text(); // 👈 Look for the price element
const webPrice = Array.from(price); // 👈 Transform price to array
// 👇 If price is filled
if (webPrice.length) {
// 👇 Delete the currency symbol
webPrice.shift(); // Bellroy uses the currency symbol before the price number
// 👇 We add the current date to the beginning array
webPrice.unshift(moment().format("DD-MM-YYYY - "));
const result = webPrice.join(""); // 👈 Join array
console.log(result); // 👈 Why only console.log will be explained in the second part
} else {
console.log("Something went wrong while getting the price.");
}
await driver.close(); // 👈 Close chrome instance
}
getPrice(url); // 👈 Run function
After execution, this is the result:
For the moment, I have code that works for these websites:
Check it here.
End of part 1.
To take into consideration, if you are using imports, you have to specify "type":"module" in your package.json. If you use the traditional way with require type will be "commonjs".