Filter, reshape, and redact logs before export with Transformers
Cloudflare Logpush now integrates Apache DataFusion and Apache Arrow to enable real-time SQL-based log transformation. Users can now filter, redact, and reshape log data directly within the pipeline before it reaches the destination storage.
Verified State Diff
Impact & Verification Analysis
DevOps engineers, security analysts, and compliance officers managing log pipelines and data privacy.
This feature reduces storage costs and improves security posture by ensuring sensitive data is redacted at the edge, while simultaneously simplifying data engineering workflows by eliminating the need for intermediate processing layers.
Full Fact Overview
The introduction of Transformers into the Logpush pipeline shifts data processing from post-ingestion to in-transit. By leveraging Apache DataFusion for query execution and Apache Arrow for memory-efficient data handling, Cloudflare allows users to execute SQL commands to manipulate log schemas and content. This reduces the need for downstream ETL (Extract, Transform, Load) processes, as sensitive data can be redacted and unnecessary fields stripped before the logs are delivered to third-party storage or SIEM providers.