LEGO Sets are Expensive
As an adult fan of LEGO (AFOL), I know very well that the LEGO cult hobby is not a cheap one. I found myself constantly looking for good deals for LEGO sets online. And there didn’t seem like there was an easy way, I had to go to all the different online stores, check for the set I wanted and then compare the prices.
There was no single place where I could search for the set I wanted and showed me which online store had stock and was the cheapest. Especially in Australia, where LEGO sets are more expensive compared to US or Europe.
In the end, I found that the best way to find deals was to participate in LEGO forums, read LEGO blogs or Facebook groups where other AFOLs will share information on new deals. But this was still not ideal, as you had to constantly sieve through all the pages and pages of discussion for information on the set I wanted. And there was no guarantee that the information was still up to date by the time I got to it.
BrickCompare – The LEGO Set Price Comparison Site
So I decided to take matters into my own hands. As a weekend project, I started creating a website which would allow me to search for the set I was interested and show me the stock availability and prices for all the Australian online store. Thus BrickCompare was born.
The goal of BrickCompare was simple. It was to scrape online store fronts and gather pricing and stock availability for listed LEGO sets. And it had to be up to date, so it would have to do this every few hours. The data had to be easily searchable and presented in a way that the prices could be easily compared.
Also read: Scraping Websites with X-ray, AWS Lambda and Serverless, Review: Using Fiverr for BrickCompare’s Logo
After the first weekend, I managed to set up the first version of BrickCompare which scraped data from about ten online stores. After a few more spare weekends this doubled. Then I also added eBay stores and the number of stores scraped doubled again. Now BrickCompare scrapes over 60 online stores and contains pricing data for more than 1,800 unique LEGO sets.
You should give the site a try here: https://brickcompare.com. There is now also a mobile optimised version if you use your mobile device.
But it’s still far from Perfect
Although I am continually adding features and improving BrickCompare, it is still has limitations.
For one, not all sites can be scraped. Some sites load their data using AJAX which is harder to scrape. So I have skipped these sites for now, but I am hoping to be able to handle these sites in the future. So stay tuned.
Sites provide their data in all kinds of styles and format, there is no standardized format. So the data is prone to error or partially missing for some sites. Some sites also don’t easily provide easily available stock levels.
Shipping fees also vary from site to site and varies depending on where you are ordering from. So it is not trivial to provide the right shipping fee for each listing. So I try to provide the best estimate. The easiest sites are the ones with flat shipping fees or no shipping at all.
Finally store often provide voucher codes (eBay very often) or special promotions like “Buy 1 get one 50% off” which is hard to capture on a site like BrickCompare. Still trying to come up with a solution for this one.