← All discussionsSpark Structured Streaming

What does late data mean in your streaming use case?

Question

Describe a streaming scenario where events can arrive late or out of order. How long would you wait, what result must remain correct, and what trade-off would you accept? Let’s compare watermark and state strategies.

1 answer

✓ Accepted answer

Measure lateness from event time using real observations rather than guessing. Choose a watermark that covers normal delay while bounding state growth, then define what happens to events beyond it. Test on-time, late-but-accepted, and too-late events. Monitor input rate, processing rate, batch duration, state size, and discarded late records together.

Join the discussion

Sign in to share an answer with the learner community.

Answer