Hi Sherry, thanks for the response. We have a suspicion that we know what is happening. Presume we have a process that has 5 steps / activities and the threading value is set to 20. We think the BPEL engine is using the 20 threads to move 20 instances (and thus 20 files) to step 1 of the process and then another 20 to step 1 and then another.... I think you get the idea. When he has brought all processes to step 1 then he starts from scratch again and moves 20 to step 2 and another 20 to step 2 and so on. This would explain why all processes are terminating more or less together. This could conceivably make sense when the processes are long lived. If a process takes 3 hours to complete and the engine dedicates it 20 threads to moving 20 instances through the whole process then it would be more or less blocked for 3 hours. David Lane Senior Entwickler afb Application Services AG Meglingerstrasse 20 81477 München Germany Phone +49 (89) 78 000-314 Fax +49 (89) 78 000-590 E-Mail Lane ... @afb.de Internet www.afb.de Vorstand: Christian Aechter, Gerolf Dienhold, Jan Ph. Wieners Vorsitzender des Aufsichtsrats: Ralph G. Werner Sitz der Gesellschaft: München - Registergericht München, HRB 129 294 Allgemeine Information zur steigenden Zahl von Viren-E-Mails: Virenbehaftete E-Mails, die bei uns eingehen, werden von der afb-Firewall automatisch erkannt und ausselektiert. Bitte beachten Sie, dass keine Benachrichtigung an Absender und Empfänger über die Nicht-Zustellung dieser E-Mails erfolgt. General information regarding the growing number of virus emails: Virus-infected emails received by us are automatically detected by afb-Firewall and singled out. Please note that no messages are sent either to the sender or recipient regarding the failed delivery of such emails. -----Ursprüngliche Nachricht----- Von: Sher ... @Sun.COM [mailto: Sher ... @Sun.COM ] Gesendet: Dienstag, 18. November 2008 19:24 An: de ... @open-esb.dev.java.net Betreff: Re: AW: Re: Behaviour of FIleBC under load conditions Hi David, The File BC thread setting (the one that defaults to 5) applies to "outbound" processing (i.e. file writing) only. On the "inbound" (file polling) side, unless the throttling configuration (max concurrency setting) is applied, the BC will continue to poll for matching input files. It doesn't sound right to me either if no output files are generated until all input files are processed, but I'm not sure if the problem is in File BC or BPEL SE here. I've filed a bug to keep track of this: https://open-esb.dev.java.net/issues/show_bug.cgi?id=1044 . We will look into this problem. Regards Sherry Weng Open ESB Community http://open-esb.org Lane David wrote: Hi Sujit, the files are not huge: around 50Kb. Input pattern is %u.xml. Output is something like CON-%u.xml. They are also separate directories. In which direction should I change the setting? I'm not sure how this setting could lead to the behavior observed though. If this is set to 5 (as it is) and the BPEL thread limit to 10 then the system should behave just like 10 file are in the input directory even when there are a 1000 in there. The CPU and memory load are a big concern here. David Lane Senior Entwickler afb Application Services AG Meglingerstrasse 20 81477 München Germany Phone +49 (89) 78 000-314 Fax +49 (89) 78 000-590 E-Mail Lane ... @afb.de Internet www.afb.de < http://www.afb.de > Vorstand: Christian Aechter, Gerolf Dienhold, Jan Ph. Wieners Vorsitzender des Aufsichtsrats: Ralph G. Werner Sitz der Gesellschaft: München - Registergericht München, HRB 129 294 Allgemeine Information zur steigenden Zahl von Viren-E-Mails: Virenbehaftete E-Mails, die bei uns eingehen, werden von der afb-Firewall automatisch erkannt und ausselektiert. Bitte beachten Sie, dass keine Benachrichtigung an Absender und Empfänger über die Nicht-Zustellung dieser E-Mails erfolgt. General information regarding the growing number of virus emails: Virus-infected emails received by us are automatically detected by afb-Firewall and singled out. Please note that no messages are sent either to the sender or recipient regarding the failed delivery of such emails. *Von:* Suji ... @Sun.COM [mailto: Suji ... @Sun.COM ] *Gesendet:* Mittwoch, 5. November 2008 18:46 *An:* de ... @open-esb.dev.java.net *Betreff:* Re: Behaviour of FIleBC under load conditions Hi David, How large are this input files? note given the default configuration of filebc, at a given time filebc can write to only 5 files ( i.e 5 processors) . Please change this configuration and restart , see if you get any improvement. Also what is file name pattern being used in the input wsdl and output wsdl? Thanks, Sujit Lane David wrote: Hi there, we have a BPEL Application that is file based and we are noticing some strange behavior under load conditions. The receive of the BPEL process is bound to an inbound and in-only port with a file binding that polls a directory for xml files. At the end of the BPEL Process we do an invoke of an operation that is bound to a port with a file binding in outbound mode which writes another xml file to an output directory. So far so good, everything works fine. However, we did a load test yesterday and noticed some strange behavior. When we dump X files into the input directory then processing begins as expected: we see from the logs that the files are being processed over time. What we are observing though is that no files are appearing in the output directory until processing of all is completed. The same behavior was observed for 100, 500 and 1000 files. This seems very inefficient - we are noticing for example that the resource-usage (memory, cpu etc.) is climbing with the increased number of files although we have set the number of BPEL Threads to 10. We need to get the files immediately after they are finished processing. I mean under certain conditions it could theoretically be possible that we receive the output files only at the end of the day when the load goes down. David Lane Senior Entwickler afb Application Services AG Meglingerstrasse 20 81477 München Germany Phone +49 (89) 78 000-314 Fax +49 (89) 78 000-590 E-Mail Lane ... @afb.de Internet www.afb.de < http://www.afb.de > Vorstand: Christian Aechter, Gerolf Dienhold, Jan Ph. Wieners Vorsitzender des Aufsichtsrats: Ralph G. Werner Sitz der Gesellschaft: München - Registergericht München, HRB 129 294 Allgemeine Information zur steigenden Zahl von Viren-E-Mails: Virenbehaftete E-Mails, die bei uns eingehen, werden von der afb-Firewall automatisch erkannt und ausselektiert. Bitte beachten Sie, dass keine Benachrichtigung an Absender und Empfänger über die Nicht-Zustellung dieser E-Mails erfolgt. General information regarding the growing number of virus emails: Virus-infected emails received by us are automatically detected by afb-Firewall and singled out. Please note that no messages are sent either to the sender or recipient regarding the failed delivery of such emails. --------------------------------------------------------------------- To unsubscribe, e-mail: dev- ... @open-esb.dev.java.net For additional commands, e-mail: dev- ... @open-esb.dev.java.net --------------------------------------------------------------------- To unsubscribe, e-mail: dev- ... @open-esb.dev.java.net For additional commands, e-mail: dev- ... @open-esb.dev.java.net
