Yadukrishnan JData Engineer · Bengaluru

Author
Yadukrishnan J
Role
Data Engineer, Shell
Based in
Bengaluru, India
Updated

Data engineer who writes down why.

Four years building lakehouse pipelines and retrieval systems on Azure and Databricks at Shell, including an ingestion framework that processed 50+ TB and 4M+ objects at ~850 GB/day with automated reconciliation. Below are two systems I am building in public, with the decisions behind them and the alternatives I rejected.

Read the work Resume (PDF) Email me
Portrait of Yadukrishnan J

Bangalore Tech Corridor Traffic

A data platform that measured travel time across Bangalore's tech corridors for 60 days, joined it with weather, and answered three questions. Collection has stopped; the full retrospective explains why.

Status
Stopped, all 5 phases built
Collected
17 Aug – 15 Oct 2026
Running cost
$0 / month

Questions it answers

  1. How bad is this corridor right now compared to its own normal?
  2. When should I leave in the next few hours?
  3. Is today unusual, and does rain explain it?

Architecture

Pipeline, as built
Mapboxdriving-trafficOpen-Meteocurrent weatherRawJSONL · 7 daysBronzeParquet · sealedSilverSpark streamGoldstar · SCD2Local appbest time to leaveMapboxdriving-trafficOpen-Meteocurrent weatherRawJSONL · 7 daysBronzeParquet · sealedSilverSpark streamGoldstar · SCD2Local appbest time to leave

Select a step to see what it does.

Decisions

Monitor fixed corridors, not arbitrary routes

Chose
A fixed set of tech corridors monitored continuously; user routes are mapped onto them at query time.
Rejected
Answering any origin and destination on demand.
Why
"Best time to leave" needs weeks of history on a specific route. An arbitrary route has none.

Run on free infrastructure

Chose
GitHub Actions cron for scheduling, Parquet committed to the repo for storage.
Rejected
A hosted scheduler and a paid object store.
Why
Free GitHub Actions minutes on a public repo and no cloud account. The trade-off, a scheduler that delivered about a quarter of hourly runs, is incident 1 in the case study.

Keep only what is derived

Chose
Delete raw provider responses after 7 days; retain and publish only derived measures.
Rejected
Keeping raw responses indefinitely.
Why
The provider licence. Weather data is from Open-Meteo under CC BY 4.0.

Roadmap

  1. Phase 1Ingestion, bronze Parquet, retention, CIDone
  2. Phase 2Streaming silver layer with checkpoints and watermarkingDone
  3. Phase 3Gold star schema, SCD2 corridor dimension, PySpark backfillDone
  4. Phase 4Quality gates and pipeline health fileDone
  5. Phase 5Local web app: route input, now vs normal, best time to leaveDone
  6. CollectionStopped after 60 days; data kept private for licensing reasonsStopped 15 Oct

Read the case study View on GitHub

sift

A document ingestion and retrieval system that validates what it actually extracted, defends itself against malicious uploads, and tunes its own retrieval strategy for a corpus it has never seen.

Repository
Yadu9238/sift
Status
Milestone 1 of 3, Phase 0 done
Runs
Locally, Docker + Ollama
Evaluation
RAGAS + abstention metrics

RAG is a search engine with a text generator stapled to the end. Most RAG failures are search failures, so the measurement harness is built before the optimisations.

Design thesis

Three things it treats as engineering problems

Extraction quality gate

Every page is scored on text density, dictionary-word ratio, column interleave, tables and repeated headers before indexing. Low-text pages go to OCR, complex layouts to a vision-model parser, and unrecoverable documents are quarantined instead of silently indexed.

Self-tuning retrieval

Generates a grounded evaluation set from your own documents, including a deliberately unanswerable slice, runs an ablation across chunk size, dense / lexical / hybrid, reranking and top_k, then keeps the winning configuration.

Prompt-injection defence

Uploaded files are untrusted input. Invisible and off-canvas text is detected from font and colour data captured at extraction, retrieved content is isolated from instructions, and an adversarial PDF corpus runs in CI.

Architecture

Ingest plane
cleanUploadPDFJob queueSKIP LOCKEDExtractPyMuPDFQuality gateper pageQuarantinenot indexedOCR / VLMrecover textChunk+embedbge-m3PostgrespgvectorcleanUploadPDFJob queueSKIP LOCKEDExtractPyMuPDFQuality gateper pageQuarantinenot indexedOCR / VLMrecover textChunk+embedbge-m3Postgrespgvector

Select a step to see what it does.

Query plane
QuestionuserPostgrestsvectorHybriddense + lexicalRerankcross-encoderEvidence?enough to answerAnswerwith citationsAbstainnot foundQuestionuserPostgrestsvectorHybriddense + lexicalRerankcross-encoderEvidence?enough to answerAnswerwith citationsAbstainnot found

Select a step to see what it does.

Decisions

One datastore

Chose
Postgres 16 with pgvector for vectors and tsvector for lexical search.
Rejected
A dedicated vector database.
Why
A second datastore makes deletes non-transactional with the metadata. One store makes deletion provable.

Queue in the database

Chose
Postgres FOR UPDATE SKIP LOCKED workers for per-upload work; Airflow only for corpus-wide batch jobs.
Rejected
Celery and Redis.
Why
Three services for a single-node app. Per-upload work stays event-driven and out of the scheduler.

Measure before optimising

Chose
Ship a deliberately naive baseline first, then add hybrid search, reranking and tuning against it.
Rejected
End-to-end LangChain RAG chains; LangChain is confined to the generation layer.
Why
You cannot claim a retrieval improvement without a recorded baseline, and chains hide exactly the logic this project exists to show.

Roadmap

  1. Milestone 1Core system: data model, extraction and quality gate, baseline RAG, evaluation harness, hybrid retrieval and auto-tune, injection defencePhase 0 done
  2. Milestone 2Data platform: async ingestion worker, Airflow re-embedding / eval / retention DAGs, dbt marts and SCD2 snapshotsPlanned
  3. Milestone 3Production polish: provable deletion, tenant isolation, Langfuse tracing, CI eval gate, hard-PDF benchmarkPlanned

Benchmark tables stay empty until the evaluation harness lands. Numbers will be added only when they are reproducible. Nothing here is an estimate.

View on GitHub

Experience

  1. Data Engineer · Shell

    • Architected a retrieval-augmented generation assistant over three years of incident records and 100+ technical manuals, replacing a full-context-window approach with vector retrieval on Azure AI Search and Azure OpenAI to ground answers in structured and unstructured enterprise data.
    • Engineered the retrieval layer end to end (parsing, chunking, embeddings, semantic search and prompt grounding), tuning chunk size and retrieval depth against a curated question set to lift answer relevance and cut tokens per query; shipped to pilot with Azure AD-gated access and GitHub Actions CI/CD.
    • Designed a metadata-driven ingestion framework in Python, PySpark and Azure Data Factory pulling from REST and Graph APIs into a Delta lakehouse, processing 50+ TB and 4M+ objects via schema-driven configuration instead of per-source code, unlocking up to $1.5M in annual cost avoidance.
    • Tuned Spark and ADF pipelines for parallelism, partitioning and API throttling to sustain ~850 GB/day across concurrent pipelines each processing 200K–300K+ records per run, with checkpointing and idempotent retries for safe reprocessing.
    • Built bronze/silver/gold pipelines on ADLS Gen2 and Databricks with SCD Type 1/2 dimensional models, plus an automated reconciliation layer comparing source and target at record, volume and checksum level that made acceptance criteria measurable and enabled zero-rollback releases.
    • Technical lead for a five-member team; gathered requirements from on-site Upstream asset focals, architects and product owners, enforced quality and security gates via GitHub Actions with SonarQube and Apiiro, and defined data management standards adopted across global business units.

  2. Associate Data Engineer · Shell

    • Designed star-schema models and the semantic layer for Upstream asset performance and operations analytics, defining grain, hierarchies and business rules with on-site asset focals.
    • Tuned SQL queries, indexing and warehouse compute on high-volume analytical workloads, cutting average dashboard load time by 35% and lowering compute consumption.
    • Built Python and SQL pipelines onboarding enterprise datasets to Azure across cleansing, business-rule application and validation, and resolved 200+ production incidents within SLA, eliminating recurring failure classes with durable fixes and automated alerting.

  3. Machine Learning Engineer Intern · ABB

    • Built Python ETL pipelines and trained a material-strength prediction model, deploying it on an Azure pipeline with automated ingestion and scheduled retraining.

Skills

Languages & processing
Python, SQL (advanced), PySpark, Apache Spark, Spark SQL
Data platform
Azure Databricks, Delta Lake, Databricks Workflows, ADLS Gen2, Azure Data Factory, Azure SQL Database, Key Vault, Azure AD
Data engineering
Distributed batch pipeline design, ETL/ELT, orchestration, Medallion Architecture, dimensional modelling, star schema, SCD Type 1/2, partitioning and performance tuning, data quality and reconciliation frameworks
AI & retrieval
RAG, Azure OpenAI, Azure AI Foundry, Azure AI Search, embeddings, chunking strategies, vector and semantic retrieval
Practices & tools
Git, GitHub Actions, Azure DevOps, CI/CD, SonarQube, Apiiro, REST and Graph APIs, Agile Scrum, Power BI; working knowledge of Kafka streaming design and Airflow DAGs

Education and certification

Aug 2018 – Jul 2022
B.E., Information Science and EngineeringRamaiah Institute of Technology · CGPA 9.28 / 10
Certification
Databricks Certified Data Engineer AssociateDatabricks

Contact

Email is the fastest way to reach me.

krishnanyadu60@gmail.com