Open to consulting & new opportunities

Mansukh
Sonawala

Data Engineer  ·  Bengaluru, India

I build the data systems companies depend on — real-time streaming pipelines, financial data platforms, cloud lakehouses, and AI-assisted engineering workflows — at production scale, with the reliability that holds up under pressure.

7+Years Experience
50M+Records / Day
35+Pipelines in Prod
Pipeline Architecture
INGEST PROCESS STORE SERVE Kafka Event Streams REST APIs BlackLine · SaaS AWS S3 Raw / Batch SQS Queue Ingestion Databricks Spark · Scala · PySpark Delta Lake · Streaming Airflow Orchestration · Workflows dbt Transformations · Models AWS Glue Batch ETL · Airbyte Redshift Analytics DWH Delta Lake S3 · ACID · Versioning PostgreSQL Operational · RDS Tableau Dashboards Looker BI Reports QuickSight Self-service INFRASTRUCTURE & OBSERVABILITY Terraform GitHub CI/CD Kubernetes CloudWatch Grafana Datadog Claude Code

Engineering data systems
that are built to last

I'm a Data Engineer based in Bengaluru with a track record of building production-grade data infrastructure across four companies — from early-stage startups to global e-commerce platforms. My work spans real-time streaming, financial data systems, cloud architecture, and increasingly AI-assisted engineering.

My technical range covers the full pipeline lifecycle: real-time ingestion with Kafka and SQS, large-scale processing with Databricks, Spark, and PySpark, transformation with dbt and SQL, cloud infrastructure with Terraform and AWS, and analytics delivery with Tableau and Looker. I've operated these systems in production — on-call, at scale, with real accountability.

What makes my profile stand out is deep financial domain expertise: at StockX I design subledger-to-GL pipelines, reconciliation frameworks, and audit-ready data flows that Accounting and Tax teams rely on for global month-end close. I speak the language of both engineers and finance stakeholders, which is rare.

I also integrate AI tooling — Claude Code, prompt engineering, LLM-driven workflows — into daily engineering practice. Not as a novelty, but as a genuine productivity multiplier that compresses delivery cycles without sacrificing quality.

Mansukh Sonawala

StockX — Global Marketplace

Data Engineer  ·  Sep 2024 – Present  ·  Financial Data

Zuma — PropTech

Senior Data Engineer  ·  May 2023 – Sep 2024  ·  Analytics Platform

UNA Brands — E-commerce Roll-up

Senior Data Engineer  ·  Dec 2021 – May 2023  ·  Streaming + Platform

Sopra Steria — IT Services (Airbus)

Software Engineer – Data  ·  Aug 2019 – Nov 2021  ·  Telemetry Pipelines

AI in my workflow

I use Claude Code daily — for SQL optimization, root-cause analysis, code generation, and technical documentation. It's become a genuine force multiplier in how fast I can ship without cutting corners on quality.

Tools I build with

Every tool here has been used in production. No tutorial projects.

Languages

Python Scala SQL Bash

Big Data & Data Platforms

Databricks Apache Spark PySpark Apache Kafka Apache Airflow Databricks Workflows dbt Airbyte Hadoop Apache Hive

Cloud & Infrastructure

Terraform AWS S3 AWS Glue AWS EMR AWS ECS / Fargate AWS Lambda AWS Redshift GCP Kubernetes Docker

Databases & Storage

PostgreSQL MySQL Amazon Redshift DynamoDB MongoDB Amazon RDS Delta Lake

Analytics & BI

Tableau Looker QuickSight

DevOps, CI/CD & Observability

GitHub Actions Jenkins SonarQube CloudWatch Grafana Datadog PagerDuty

AI-Assisted Engineering

Claude Code Prompt Engineering AI-assisted Development Developer Productivity Automation

Where I've shipped

Four companies, four distinct data challenges — all production, all at scale.

Data Engineer

StockX  ·  Bengaluru

Sep 2024 — Present

Building the financial data nervous system for one of the world's largest sneaker and collectibles marketplaces — every order, payment, fee, and tax event flows through pipelines I own into audit-ready ledger entries used for global financial close.

  • Own the end-to-end subledger-to-GL pipeline: ingesting raw transactional events from Kafka and the BlackLine API, processing them in Databricks using Scala and Spark, and mapping outputs to General Ledger accounts across multiple regions and accounting systems.
  • Designed scalable batch and real-time data pipelines on Databricks processing high-volume order, payment, fee, and tax transactions — built with strong consistency guarantees and full audit traceability.
  • Built automated data quality and reconciliation frameworks that catch ledger discrepancies before they reach Finance, significantly reducing downstream reporting incidents.
  • Partner with Accounting, Finance, and Tax teams to support period-end close, trial balance validation, and SOX-compliant reporting across multiple global regions.
  • Delivered Tableau dashboards giving Finance real-time visibility into revenue, taxes, account balances, and reconciliation status — used in board-level reporting.
  • Owned infrastructure automation via Terraform and AWS; managed complex orchestration with Airflow and Databricks Workflows; acted as technical owner driving architecture decisions across the financial data domain.
  • Integrated Claude Code and AI-assisted workflows into daily engineering — using LLMs for SQL optimization, root-cause analysis, code generation, and documentation, measurably accelerating delivery cycles.
DatabricksKafkaSpark ScaladbtAirflow TerraformAWS GlueTableau BlackLine APISOXClaude Code

Senior Data Engineer

Zuma  ·  Bengaluru

May 2023 — Sep 2024

Took ownership of the analytics platform at a fast-moving PropTech startup — stabilised unreliable pipelines, migrated the entire BI layer, and built a data service the engineering team still depends on.

  • Designed and orchestrated 15+ production ETL pipelines using Airflow, Airbyte, AWS Glue, and dbt — improving reliability and giving analytics teams clean, timely data to work with.
  • Built a FastAPI-based data service with structured logging and PagerDuty alerting, dramatically cutting incident response time and reducing outages that previously interrupted business reporting.
  • Led the migration from Tableau to Looker — a full BI platform switch covering data model re-architecture, report recreation, and stakeholder rollout. Improved query performance and reduced platform costs.
  • Owned deployment pipelines via AWS Fargate ECS and GitHub Actions, and migrated Postgres databases to Multi-AZ clusters for better scalability and zero-downtime operations.
  • Implemented centralised observability with CloudWatch and Grafana — the team went from discovering data issues at stand-up to getting alerted automatically.
AirflowdbtAirbyte FastAPIAWS GlueFargate LookerPySparkCloudWatch

Senior Data Engineer

UNA Brands  ·  Bengaluru

Dec 2021 — May 2023

High-throughput pipelines for an e-commerce roll-up managing dozens of brands — real-time Kafka streaming at 50M+ records/day, a Kubernetes-native data platform, and engineering team leadership across a 5-person group.

  • Designed and automated 20+ production ETL pipelines, eliminating manual data processing and improving availability of business-critical datasets across multiple brands.
  • Led batch and streaming pipelines processing 50M+ records per day using PySpark, Kafka, EMR, and dbt — maintaining 99.9% data availability on the streaming layer.
  • Owned Airflow and Airbyte deployments on Kubernetes end-to-end — configuration, scaling, and on-call — while driving cost reduction through right-sized cluster management.
  • Built SQS-based ingestion pipelines for marketing and third-party data, adding reliable delivery guarantees that batch jobs previously lacked.
  • Established CI/CD pipelines with Kubernetes and Terraform, and led DevOps initiatives on AWS Fargate ECS that significantly cut deployment time.
  • Led a 5-person cross-functional team — setting code standards, running reviews, and mentoring junior engineers while aligning delivery with business stakeholders across multiple brands.
PySparkKafkaEMR KubernetesTerraformAirflow dbtSQSAirbyte

Software Engineer — Data Platforms

Sopra Steria  ·  Noida

Aug 2019 — Nov 2021

Started my data engineering career building real-time Spark pipelines for Airbus — aircraft engine telemetry with hard latency requirements and zero tolerance for data loss.

  • Developed and operated real-time Spark pipelines for Airbus, reducing latency in aircraft engine telemetry processing where data freshness had direct operational impact.
  • Implemented AWS infrastructure using Terraform and CloudFormation — API Gateway, CloudFront, DynamoDB — building repeatable, auditable deployments from the start of my career.
  • Built back-end services in Python on AWS, leveraging PySpark for large-scale data processing and throughput optimisation across high-volume telemetry datasets.
  • Contributed to internal open-source tooling, establishing reusable components and engineering standards adopted across the team.
PySparkTerraformCloudFormation DynamoDBPythonAWS

What I build best

The six areas where I can walk into a codebase on day one and make a real difference.

⚡

Real-time Streaming Pipelines

Kafka to Databricks Spark Structured Streaming — high-throughput, low-latency event processing with 99.9% availability. Built and operated at 50M+ records/day in production, not just in theory.

KafkaSpark StreamingDatabricksScalaSQS
🏦

Financial Data Engineering

Deep experience building subledger-to-GL pipelines, account reconciliation frameworks, and SOX-compliant datasets. I understand accounting concepts and the engineering required to make financial data reliable and audit-ready at global scale.

LedgeringGL MappingReconciliationSOXdbt
☁️

Cloud Data Infrastructure

Full IaC ownership on AWS using Terraform — Glue, EMR, ECS/Fargate, Lambda, S3, Redshift. I own the architecture decisions, cost tradeoffs, and deployment governance — not just the resources themselves.

TerraformAWS GlueEMRECS/FargateKubernetes
📐

Data Modeling & Transformation

dbt-based transformation layers, dimensional models, CDC pipelines, and SQL that performs under pressure. I write data models that other engineers can maintain and business users can trust — not just ones that pass tests on the first run.

dbtSQLPySparkCDCDelta Lake
🔍

Data Quality & Observability

Automated DQ frameworks, reconciliation workflows, and monitoring stacks with CloudWatch, Grafana, Datadog, and PagerDuty. My pipelines don't just run — they surface issues before stakeholders notice something is wrong.

CloudWatchGrafanaDatadogPagerDutydbt Tests
🤖

AI-Assisted Data Engineering

I embed Claude Code and LLM-based workflows into my engineering practice — SQL optimisation, intelligent root-cause analysis, automated documentation, and code generation. AI amplifies the engineering judgement; it doesn't replace it.

Claude CodePrompt EngineeringLLM WorkflowsAI Automation

Let's work together

Open to data engineering consulting, freelance pipeline projects, and full-time roles at companies building serious data infrastructure. I respond within 24 hours.

mayur.sonawala13@gmail.com