Filenode pulling all the file in one shot for “n” number of files is only illusion. – Since the flow’s processing time is in milli/nano seconds ,we will not be able to see the files getting placed intermediately in mqsitransitin folder .
Polling happens this way:
1) It starts polling from the time of Deploying the flow
2) It processes each file, one at a time from the folder.
3) First it picks the first file.Until the file is successfully processed in the flow,it is in mqsitransitin folder
4) While the file is there in mqsitransitin folder under process,the FileInput node is still active.It does not go to sleep
5) During this time of processing first file,if it sees next files in the Input folder,it will pick the next files once the first file process is completed
6) Now,after the files are successfully processed(files come out of mqsitransitin and moved to mqsiarchive as configured),if it does not see any file,the FileInput will go to SLEEP mode for the duration of Polling interval
7) When it is in SLEEP mode,if any file placed in the folder,it will not be picked up during this time.The file will be picked up only at next Polling
So the behavior is as below:
If any file is in progress,it means that FileInput instance is still active.
Hence if next file is placed during this time, will be picked up after previous file is completely processed.
If next file is placed after the previous file is completely processed,then those files will be picked up only in the next Polling.
I have tried a scenario where in esql I waited for 20 seconds( boolean b=SLEEP(20000) ) and did multiple test with Polling frequency as 100 seconds.Filename pattern as “*.txt”. The files are getting picked up as specified in the process steps given above.
Output as below:
