Web scraping
We extract and organize data from websites responsibly: prices, catalogs and directories turned into structured information.
REQUEST A DIAGNOSTICWhat scraping is
Scraping is the automatic collection of information published on websites. A program visits the pages, locates the data of interest and stores it in a structured way, so it can be queried, compared or loaded into another system without copying it by hand.
How it differs
From an API
An API is the route a site itself offers to access its data: stable and authorized. If it exists and covers what you need, it is preferable to scraping.
From crawling
Crawling goes through a site following links to discover pages. Scraping extracts specific fields from those pages. A project usually needs both.
Responsible scraping
It respects the site's indications, keeps a pace that does not overload the server, does not bypass blocks or security measures and does not collect personal data without a legal basis.
What data we collect
Prices
Prices published in online stores and marketplaces.
Catalogs
Product sheets, references and features.
Directories
Lists of companies, centers or resources published in public sources.
Examples of what we have already done
Prices
Collecting prices from public catalogs to compare them with your own on a regular basis.
Directories
Turning a public directory into an ordered table, without collecting personal data without a legal basis.
Updates
Following what is published on a website, such as calls or tenders, and receiving it in order.
Supplier availability
Following the stock and lead times of several suppliers in a single view.
Product sheets
Collecting technical data from public product sheets to complete a catalog.
Public opinions
Gathering public reviews and mentions of a product for analysis.
When it helps
When someone on the team checks websites by hand on a regular basis to follow prices, catalogs or listings, and that information ends up copied into a spreadsheet.
What is worth knowing
- Not all data can be collected. It depends on the site's terms of use and on the regulations, especially when personal data is involved.
- When the source site changes its structure, the extractor has to be adjusted.
- The data reflects the moment it was collected. How current it is depends on how often it is collected.
- That personal data is published does not mean it can be collected and used freely: the GDPR applies all the same.
- A catalog or a directory may be protected as a database. Extracting a substantial part without permission may infringe the rights of whoever compiled it.
- Many sites use bot protection. Bypassing blocks or security measures increases the legal risk.
FREQUENTLY ASKED QUESTIONS
What is scraping?
It is the automatic collection of information published on websites: a program visits the pages, locates the data of interest and stores it in a structured way.
What data can be extracted with scraping?
Published data, such as prices, catalogs and directories. Not all of it can be collected: it depends on the site's terms of use and on the regulations, especially when personal data is involved.
What happens if the source website changes?
When the source site changes its structure, the extractor has to be adjusted.
Is scraping legal in Spain?
It depends on what is collected, how and for what purpose. It is not illegal in itself. Personal data, protected content, databases and sites that require registration change the analysis. Doubtful cases should be reviewed with a lawyer.
Can personal data published on a website be collected?
Not freely. Even if it is accessible, it is still personal data, and the GDPR requires a legal basis to process it and, as a general rule, to inform the people affected.
How does Devontic approach scraping?
We aim to respect the site's indications, not overload it and not collect personal data without a legal basis. If a client needs something different, we study the case before deciding.
What is the status of the robots.txt file?
It is not a legal rule, but the way a site indicates what it does not want a program to crawl. Respecting it is part of responsible scraping.
How much maintenance does an extractor need?
It is continuous. Sites change their structure and their protections, and each change can leave the extractor not working until it is adjusted. It is the most underestimated cost.
What is needed to get started?
The list of sites and fields, how often you need the data and the format you want to receive it in. The first thing checked is whether the site offers an API and what its terms of use say.
From the web to the data
We start with a proof of concept on real data and only scale if the value is demonstrated.