It is a human behavioral pattern — to keep reinventing the wheel or to fall into the same series of events. E-commerce retailers who attempt to create their own web scrapers, or price monitoring systems, to collect data from competitors’ websites will suffer the same fate. Sadly, inevitably leads to disappointment.
If you stop reading, the story ends there; you wake up in a world of millions upon millions of google answers to your query “Over the past few years, I’ve scraped dozens of websites…”, “Top software for the web scraping…”, “How to create scraper” and so on and so forth... All these words equate to the same result.
But if you read until the end, you will become aware of all the difficulties involved with creating your own web scraper, and then you will clearly see how deep the rabbit hole goes.
Our story begins a few years ago, when one of our first clients told us that he would rather hire a programmer for $200 and create his own parser than buy a solution from our company. Yes, he was right — you could feasibly snatch grab data from any website. Yet still, he was wrong in that it’s not such an easy task. This is especially true when we speak about e-commerce websites. This was the main reason for our clients: Before their attempt, they never considered the issues which they could encounter along the scraping path. We’ve described some below.
The majority of modern web stores have built their defense from the different kinds of DDOS attacks. If your scrawling bot does too many requests per a set time (the number of requests and time period are dependent upon the settings of every website you are trying to scrape), it will be blocked in a moment by a rate-limiting tool. (If you know what DDOS is but never saw its visualization, check this video https://www.youtube.com/watch?v=hNjdBSoIa8k).
The DDOS defense is not the only way web retailers keep their prices safe from competitors’ bots. Let’s name just some of them.
The very first kind of “defense” is how data is structured on the website. The term consist of URL qualities (URL parameters and attributes, different pagination styles, etc...) and unstructured data (you need dig really deep into a page’s markup to find where to catch the data). It is kind of an easy task to scrape one website, but if you need to grab data from dozens?
They are not really so common now, but still can be found on different websites that are trying to protect themselves.
There are much more but we will not stop on it. Let’s continue to even more important stages.
OK, you handled all the previous issues. Your price monitoring system is working with different competitors’ and gave you some solid data. But, wait. Do you know exactly when this data is no longer valid? There could be a variety of reasons for this:
Oh sure, you can find a programmer who will create a few scrapers or crawlers. This could be the perfect solution for a SMB retailer who needs to monitor around a thousand of SKUs and just one competitor or your industry does not require often price changes. With that said, to make them work smoothly and correctly, you need to spend a whole lot more resources and build not only a parser but a fully functioning Price Monitoring System. At this point, we need to warn you: If you actually build a price monitoring system, that covers all the aforemented issues, then you probably will have to stop to selling goods online — You simply won’t have the time for it.
By the way, do you remember that story about $200 and a programmer? After a few months, that client came back and bought a Competera solution. Then the same story was, like clockwork, repeated a few more times with a variety of clients.
If your company is ready for a permanent and sophisticated pricing, instead of “reinventing the wheel” (run through a checklist to find out), your best choice is to find a functional, powerful, yet data-based SaaS solution, because it will give you a much greater chance to win the competition with a rules based reprising (and’ll add no extra complexity to your existing business processes).
You can easily find a solution that not only provides you with the correct, yet visualized, competition’s data, but will also help you to do a smart assortment segmentation (ABC-, KVI-analysis), suggesting the best prices for your items according to pre-installed rules, allowing you to work together with all your team, etc...
All you need is some time to google it, and then compare all the options you’ll find to choose which ones will fit your business needs. But please be aware: all solutions have a setup fee which is usually even higher than a monthly fee, so we suggest you ask for a free trial before you'll finally make a decision.