Client workHundreds of thousands of records / day
Social Media Data Pipeline
Data Engineer
The pipeline behind the social-media & news scrapers — powering two-sided sentiment analysis and NER for a public-sector communications team.
Cut processing time ~40% and improved reliability ~25%, monitored 24/7.
How it works
Sources
Scrapers
Queue
RabbitMQ
Process
Docker workers
Store
Elasticsearch
Monitor
Grafana + Prometheus
Problem
Data volume is enormous and must be processed continuously.
Solution
HPC pipeline with monitoring and a queue system.
Key decisions
- RabbitMQ for flexible routing — speed from scrape to display wasn't the bottleneck, so simple routing beat Kafka's complexity.
- Elasticsearch for fast full-text & context search across large volumes of news and social text.
- An alert bot pings the dev team the moment a scraper returns no data — kept uptime ~99% with fast recovery.
- On-prem HPC: the ~40% efficiency gain freed capacity for co-located AI and queue workloads.
Tech stack
- RabbitMQ
- Docker
- Elasticsearch
- Grafana
- Prometheus
- Airflow