[Free e-book] Agile Pricing: The Must Have Framework for Modern E-tailers Download nowGet free e-book

Build Or Buy: Price Monitoring System For Online Retail

Every e-commerce retailer needs to keep an eye on its competitors as long as their price changes affect its sales directly. Today’s best way to work with data from competitors’ websites is a price monitoring system. The question is how the retailer gets it: by building it in-house, or buying an existing solution.
In this article we’ll examine both scenarios.

Years ago, one of our first clients told us that he would rather hire a programmer and create his own monitoring system than buy any existing solution. In the end, all of them were facing the same: snatch grabbing data from websites is not as easy as it sounds. This is especially true when we speak about e-commerce websites. We’ve described some issues you need to be prepared for, ahead of building your own price monitoring system below.

Technical Questions

The majority of modern web stores have built their defense from the different bot types. If your crawler does too many requests per a set time (the number of requests and time period are dependent upon the settings of every website you are trying to scrape), it will be blocked in a moment by a rate-limiting tool.

The bot defense is not the only way web retailers keep their prices safe from competitors’ bots. Here is what Dmitriy, Head of Data Delivery Team @Competera shares about his experience of creating, adjusting, and supporting of 2700+ web scrapers:

The are many ways you can be blocked and not all of them are obvious.

The first one is a captcha. They are not really so common now, but still can be found on different websites that are trying to protect themselves.

Instead of captchas, Javascript algorithms are kind of new and a common way to protect prices for the retailer on the web. They use bot activities monitoring and in the case they found them, the following algorithm… changes prices on the website. So your crawler is doing its job, but sending you back worthless data.

Avoiding of all these types of defence is not the easiest task even for advanced programmers: The requirements for resources for creation of human behavioral imitation are extremely high.

And even when your IT-dept will handle all the issues above, there still will be a troubles associated with the validity of the data you scrape:

  • Your price monitoring system has no abnormality alerts (in your system, or on the competitors’ side).
  • It does not provide any error control (even if there are some errors, you’ll never know what was broken and where).
  • Your bots do not compare the scraped data to the previous period (so you do not know if there are more or fewer, items on your competitor’s site today).
  • More

In the end it’s not a question of how complicated the task of data scrapers creation really is. The main issue is that problem solving is toggling the programmers priorities and they act not so effectively in other areas of retail business, because of lack of time and focus.

Financial Questions

Creating a few scrapers or crawlers could be a good-enough solution for a SMB retailer who needs to monitor around a thousand of SKUs from just one competitor. Alternatively, if the industry does not require regular price changes.

With that said, to make the scrapers work smoothly and correctly, you need to spend a whole lot more resources and build not only a parser but a fully functioning Price Monitoring System. This was the main reason for which our clients return to our solution: Before their attempt, they never considered the issues which they could encounter along the scraping path. Afterwards, they switched to Competera Solutions and they get the power of their IT-depts back and applied it to boost other processes in the store.

Therefore if your company is ready for a permanent and sophisticated pricing solution, instead of “reinventing the wheel” (run through a checklist to find out), your best choice is to find a functional, powerful, yet data-based SaaS solution, because it will give you a much greater chance to defeat the competition with a rules based repricing (and’ll add no extra complexity to your existing business processes).

You can easily find a solution that not only provides you with the correct, yet visualized, competition’s data, but will also help you to do a smart assortment segmentation (ABC-, KVI-analysis), suggesting the best prices for your items according to pre-installed rules, allowing you to work together with all your team, etc...

All you need, is some time to google it, and then compare all the options you’ll find to choose which ones will fit your business needs. Please note: all solutions have a setup fee which is usually a bit higher than a monthly fee, so we suggest you ask for a free trial before you'll finally make a decision.