Over 82 per cent of ecommerce organisations are now using web scraping to gather external data to help guide their decision-making, overtaking internal data collection (81 per cent), which stands as the number one method in nearly every other sector. This is one of the key findings from Oxylabs which demonstrates how web scraping is shaping the industry.
In cooperation with Censuswide, Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US eCommerce companies, gathering insights on attitudes towards data collection in the sector. Not only has web scraping entrenched itself as the primary vehicle for decision-making, but it has also become the most popular way to collect data in the eCommerce industry. When asked what methods the respondents use to acquire data, 75 per cent of eCommerce organisations selected web scraping while 74 per cent are using the runner-up, internal data collection. Other methods, such as the procurement of data from third-party aggregators, fall further behind at 65 per cent for traditional, and 59 per cent for alternative data. Such a trend isn’t surprising as having control of the entire pipeline allows for greater flexibility and adaptability to business goals.
When asked which type of data respondents use to make business decisions, third-party aggregators came in third behind web scraping and internal data, with only 66per cent viewing it as the most important source of data-driven decision-making. Such a trend might be caused by eCommerce companies and retailers mostly dealing with regular consumers. Third-party aggregators are often oriented towards B2B data, making their offerings more useful to internal decision-making rather than direct revenue generation.
Gediminas Rickevičius, VP global partnerships, Oxylabs, said: “Web scraping overtaking internal data collection in any industry is a milestone moment. External data collection is an area where businesses across many sectors will compete fiercely with one another in the future, with eCommerce an important trendsetter in this respect. As external data offers exciting, unique, and profitable insights, getting to them first should be a priority for every business, whether involved in eCommerce or not. The reliance on data is clearly important, therefore it is no surprise that the budgets are also expected to increase over the coming year.”
As the data access needs increased (cited by 85 per cent, with 30 per cent citing significant increases), so did the budgets for data departments (cited by 73 per cent, with 27 per cent citing significant increases). Additionally, 70 per cent of respondents expect to continue expanding their data department budgets in the coming 12 months.
Rickevičius continued: “In eCommerce, where margins are thin and the competition is fierce, companies will take absolutely any advantage they can get. Web scraping has now proven itself as the way to do so. Through external data, businesses can monitor the competition, adapt to changing environments, and even predict consumer behaviour, all of which bring it closer to optimal performance.”
Share