Simple Fractal productionized hundreds of scrapers, expanded our alternative data sourcing capabilities, professionalized our infrastructure, and provided line of sight into scraper health and data integrity in real time.
-
Company Size: 1,200–1,500
-
Industry: Finance
-
Revenue: $100M–$150M
Challenge:
A top-tier NYC hedge fund with over $14B in AUM was scrambling to meet the data needs of its investment teams in order to make well-informed investment decisions.
Solution:
We architected and implemented a serverless systems infrastructure to host our web-scraping bots and alternative data in a virtual private cloud. To facilitate analysis and monitoring of these scraper bots, we built a realtime dashboard displaying the scrapers’ performance metrics, acting as a centralized command center for alternative data sourcing.
Built 300+ enterprise-grade bots:
- Collected granular data from hundreds of targets, each with its own set of compliance and rate-limiting constraints.
- Regained trust in alternative data across the organization by ensuring consistent data integrity.
- Extended scraping capabilities to PDFs and foreign languages.
Reduced web scraping FTEs by 50%:
- Freed up half the web scraping staff to be redeployed in the market intelligence department.
- Setup proprietary tools for in-house team to write and deploy custom scrapers with minimal effort.
- Facilitated the workflow for future employees with necessary documentation.
Cut scraper execution time by 500X:
- Outperformed the existing serial web scraping system by a factor of 500 by leveraging parallel execution.
