Price Monitoring System:
Stop Reinventing the Wheel!

It is a human behavioral pattern — to keep reinventing the wheel or to fall into the same series of events. E-commerce retailers who attempt to create their own web scrapers, or price monitoring systems, to collect data from competitors’ websites will suffer the same fate. Sadly, inevitably leads to disappointment.

If you stop reading, the story ends there; you wake up in a world of millions upon millions of google answers to your query “Over the past few years, I’ve scraped dozens of websites…”, “Top software for the web scraping…”, “How to create scraper” and so on and so forth... All these words equate to the same result.

But if you read until the end, you will become aware of all the difficulties involved with creating your own web scraper, and then you will clearly see how deep the rabbit hole goes.



The Story

Our story begins a few years ago, when one of our first clients told us that he would rather hire a programmer for $200 and create his own parser than buy a solution from our company. Yes, he was right — you could feasibly snatch grab data from any website. Yet still, he was wrong in that it’s not such an easy task. This is especially true when we speak about e-commerce websites. This was the main reason for our clients: Before their attempt, they never considered the issues which they could encounter along the scraping path. We’ve described some below.

Your Price Monitoring System Will Act Like DDOS

The majority of modern web stores have built their defense from the different kinds of DDOS attacks. If your scrawling bot does too many requests per a set time (the number of requests and time period are dependent upon the settings of every website you are trying to scrape), it will be blocked in a moment by a rate-limiting tool. (If you know what DDOS is but never saw its visualization, check this video https://www.youtube.com/watch?v=hNjdBSoIa8k).

Your Price Monitoring System Will Be Blocked in Many Different Ways

The DDOS defense is not the only way web retailers keep their prices safe from competitors’ bots. Let’s name just some of them.

There are much more but we will not stop on it. Let’s continue to even more important stages.

Your Price Monitoring System Can Be a Real Mess

OK, you handled all the previous issues. Your price monitoring system is working with different competitors’ and gave you some solid data. But, wait. Do you know exactly when this data is no longer valid? There could be a variety of reasons for this:

In the End the Price for Your System Will Be Huge

Oh sure, you can find a programmer who will create a few scrapers or crawlers. This could be the perfect solution for a SMB retailer who needs to monitor around a thousand of SKUs and just one competitor or your industry does not require often price changes. With that said, to make them work smoothly and correctly, you need to spend a whole lot more resources and build not only a parser but a fully functioning Price Monitoring System. At this point, we need to warn you: If you actually build a price monitoring system, that covers all the aforemented issues, then you probably will have to stop to selling goods online — You simply won’t have the time for it.



By the way, do you remember that story about $200 and a programmer? After a few months, that client came back and bought a Competera solution. Then the same story was, like clockwork, repeated a few more times with a variety of clients.

If your company is ready for a permanent and sophisticated pricing, instead of “reinventing the wheel” (run through a checklist to find out), your best choice is to find a functional, powerful, yet data-based SaaS solution, because it will give you a much greater chance to win the competition with a rules based reprising (and’ll add no extra complexity to your existing business processes).

You can easily find a solution that not only provides you with the correct, yet visualized, competition’s data, but will also help you to do a smart assortment segmentation (ABC-, KVI-analysis), suggesting the best prices for your items according to pre-installed rules, allowing you to work together with all your team, etc...

All you need is some time to google it, and then compare all the options you’ll find to choose which ones will fit your business needs. But please be aware: all solutions have a setup fee which is usually it even higher than a monthly fee, so we suggest you ask for a free trial before you'll finally make a decision.

Author: Eugene Kharybin, Pricing Analyst at Competera




Other articles