1.

What are the output Modes in Structured Streaming?

Answer»

There are three modes supported by Structured STREAMING. Let’s look at each of them:

  1. Append mode.
  2. Complete mode.
  3. Update mode.

Append mode

Append mode is the default behavior and the simplest to UNDERSTAND. When new ROWS are added to the result table, they will be output to the sink BASED on the trigger (explained next) that you specify. This mode ensures that each row is output once (and only once), assuming that you have a fault-tolerant sink. When you use append mode with event-time and watermarks, only the final results will output to the sink.

Complete mode

The complete model will output the entire state of the result table to your output sink. This is useful when we are working with some stateful data for which all rows are expected to CHANGE over time or the sink you are writing does not support row-level updates. Think of it as the state of a stream at the time the previous batch had run.

Update mode

Update mode is complete mode except that only the rows that are different from the previous write are written out to the sink. Naturally, your sink must support row-level updates to support this mode. If the query doesn’t contain aggregations, this is equivalent to append mode.



Discussion

No Comment Found