For my BrickCompare project, I decided to a microservice to do the task for scraping websites for the pricing data I needed. Amazon Web Services with their AWS Lambda service was the perfect service for the task.
Scraping Websites with X-ray
I had already decide to use the node.js platform to run my microservice as I was familiar with it. So then I had to select a module that could get me started on website scraping. Initially I selected noodlejs as it looked to be easily to use and had decent documentation. But after writing about 10 or so scrapers for different websites, I found that it was rather buggy and did not return consistent results.
Continue reading “Scraping Websites with X-ray, AWS Lambda and Serverless”