I am running a flink job, which takes two data sources and joins them.
The output of the join is a dataStream. But I also emit a sideoutput if the join fails. Now I pass the dataStream from the sideoutput through a ProcessFunction that looks up the missed joined data from the source of truth and generates an enriched record
I take the union of the original joined dataStream with the stream coming out of the processed sideoutput and pass it through a sink
As you can see, the sideoutput after being processed is alreadyemitting 95 records while the sink input 37,280 only matches with what came out of join operator
I just do
unionedDataStream = joinedDataStream.union(processedSideOutputStream);unionedDataStream.addSink(sink)Now if I log the records going to the sink, I could see the ones coming from the sideoutput as well. However, it's only the numbers that are not matching. Is it an issue with the UI or am I doing something wrong?