Programming in Windows - detecting whether an application has finished running & writing file?

Factual Questions
1

How do I write a program to test whether another program has executed and finished writing its output file?

I face this problem in, for example, conducting parametric studies in which a specialized program can solve a numerical problem like building a computational fluid dynamics model or a finite element model. These specialized programs are usually designed to solve a problem once. Maybe there are, say, 6 different parameters that are important that I might want to study, such as 6 different dimensions in a valve I am designing that I know will influence the flow. I want to be able to analyze 20,000 different designs that are spread throughout this 6 dimensional design space. It might seem that I would have to open the specialized program, enter a specific design, run it, record how nicely the design worked, and then do it all over again 19,999 more times.

But I can use some other program to run loops in which it generates a point in that 6 dimensional space, calls out to the operating system to open a session of the numerical analysis program, feed it this design point, instruct it to write its result into a file, wait for it to finish, collect the written result file, add the design point and the result as an entry in an overall table of designs and results, and then repeat.

I do this frequently, and it works. There are a few different kinds of software I use for the outer program, and a few other ones for the inner program. Generally the way they interact is by writing little text files the other will find and read, and by the outer program passing command line type commands to the OS.

But parts of this process are messy and waste time, and that is what I am asking about.

Usually I can have the outer, looping program run a session of the inner, numerical program such that the outer program waits for the inner program to stop executing. But it is a source of messiness that the inner program might hang or crash, so I have to have the outer program decide what is a reasonable processing time, and if the OS doesn’t return control within that time, the outer program should instead kill any processes associated with the name of the inner program. I wish there was a way to detect, for example, that the inner program has not ended its session but the CPU has been mostly idle for a few seconds.

Also, if the outer program tests for the existence of the numerical result file, the test goes positive as soon as the inner program has started writing it. If the outer program waits for the file to appear and then reads it, it will get only the beginning of the file, as it will do the read before the inner program has finished writing it. I think this is especially a problem because Windows will end the inner program session before another program can reliably read the file, as if Windows has not yet synced the filesystem.

I feel uneducated about how to test reliably for these things. Instead I am doing multiple tests, adding waits to be more certain the file write is finished, and so forth.

What do I look for or where do I go to learn more about these kinds of details?

2

What language are you using to program in? There is more than one way to do this, but I would use the ShellExecute command and the WaitForSingleObject.

3

If the inner program grabs a filehandle and hangs on to it (the same file handle) the entire time it’s writing to it, your outer program can “test” to see if it’s done by attempting to get a new exclusive file handle. If the outer program gets an error code, it means the inner program still has its file handle open.

This method doesn’t work if the inner program opens-closes-open-closes the file.

Another method… if your outer program is the one launching the program, you can make note of the process handle and test to see if that process handle has terminated.

4

If that process tries to (1) write a file and then (2) immediately quit, might the handle terminate before the file is finished? In other words, is it the process that actually writes the file, or has the process passed that task off to the OS already?

5

The Win32_Process class has these two properties you might be able to use to determine whether the process is still writing to the file:

WriteOperationCount

Data type: uint64
Access type: Read-only

Number of write operations performed.

WriteTransferCount

Data type: uint64
Access type: Read-only
Qualifiers: Units (Bytes)

Amount of data written.

I’m thinking you might be able to check in a loop and when the transfer size or count stop increasing, assume the write is completed.

6

I wrote a distributed processing framework that does something similar (doing hyper-spectral image analysis). Before going into detail about how I do it (a level of indirection), is your code multi-threaded? (Not that multi-threading is necessary, mind you, but it would make my explanation and subsequent implementation easier.)

I got around this issue by writing to a temporary file then renaming it once it was fully written (I can look up the C++ function if you’d like). This was after a variety of attempts at things like counting bytes, locking files, etc.

ETA: I should add that my code is mostly for *nix systems; while Windows functionality is desirable and something that I do consider, it’s not my main purpose.

7

Ruminator enumerated my first two suggestions so I’ll just append a few more.

Is the finished output file always the same size? You can check that under object properties.
Did you make the inner program? You could have it rename the result file when it’s done as a signal to the outter program.

8

>I’m thinking you might be able to check in a loop and when the transfer size or count stop increasing, assume the write is completed.
Hey, I didn’t think of that. Might work very well, and I think it is easy.

> got around this issue by writing to a temporary file then renaming it once it was fully written
I like this one too. I am not sure that my outer program can cause the renaming, but if it can, this is easiest.

>Did you make the inner program?
No. Well, maybe, depending how you mean it. The inner program is somebody else’s product that one uses by writing a program in their special language. For that matter, so is the outer program, though it’s a different somebody else. I make both of them work by writing programs in their languages, but mostly they aren’t general purpose languages and don’t compile to executables (though one of the outer programs I’ve used is LabVIEW, and it can be fairly general and compile to .exe, if I can bring myself to touch that stupid package again). So, each of these is a bought application running a program written in their special language.

>is your code multi-threaded?
If I use LabVIEW for the outer, it could be. As I understand it.

>Is the finished output file always the same size? You can check that under object properties.
Yes. Or at most it would change by a tiny bit, because a numeric result might truncate a trailing zero or two. I could also write one more (unwanted) multidigit value into the file, which would mean that the last few bytes didn’t matter, so I’m guaranteed to get what I care about once it’s above a certain size. I like this line of thinking.
I gotta say that I’m not much of a Windows developer, and while I could try writing the outer program in C# to have access to a full range of functions, that environment is by far not the easiest for me for doing the populate the design space part or the collect a dataset of results part. I might get there faster with cheezy methods. There are already some very attractive ideas here I hadn’t thought of - thanks!!

9

That was my thought, check for the process and simply proceed only when the list of active processes doesn’t contain that process name.

10

The OP mentioned the inner process tends to hang, so you can’t just check to see if the process is running, you need to check to see if it’s actually doing anything.

11

Missed that paragraph, sorry.

If the inner program can’t rename a file can it create multiple files? If so, just create a file named “done” after completion and clean it up in the outer program once it’s detected and proceed.

12

I thought about this one, but for this to work it’d have to be true that the OS doesn’t start writing a file for the inner program until it has finished writing any earlier files for it. Is that true? Don’t see why it would have to be.

A related idea is to have the inner program write its file and then try to read it. Maybe the OS would enforce that any file whose writes have not been flushed must be sync’d before it can be read (even by the same process that wrote it). This is a way to guarantee that if the inner program has quit, its file write must already have been sync’d

It’s another issue whether the inner program could do this. The inner program I have been using the most lately will generate errors if it reads a file that does not make sense as an INCLUDE() in the user program it is running. I can’t picture any file it can write that it would accept as part of its own user program. Of course, if it generates an error after it has done the important part, that might be fine.

13

Program A kicks off program B. When it starts program B, it passes the name of the file it wants it to write. It waits for program B to end, then checks whether the indicated file exists, whether it’s size is greater than 0, and whether B exited with a successful return code. If B is properly written, you don’t even need to do anything but check the return code (0 for success, anything else for failure). B knows whether it did or didn’t finish, so you don’t really need to check anything beyond whether B was happy when it finalized.

fork() & exec() are all you need, if I’m recalling the names correctly. They should be part of Windows, or have comparable alternatives in the Windows API.

14

FYI – that’s not quite robust enough (I speak from experience). First, he mentioned that program B may hang – so, no exit code. Second, while I may simply be ignorant (I’m mostly a Unix guy), obtaining an exit code is not always easy in Windows (or possible, IIRC). Third, even if a file exists, it may exist in an only partially written state. Finally, a program can exit while not having fully written a file (i.e., getting an exit code does not imply all writes have finished).

15

This is one of the things that has gone wrong. The program exits and returns control to the OS but the file is still only partly written.

16

If the program knows how much information it is supposed to write before it starts, the first thing it writes to the file could be the length, packet-count, or some other sort of check. You can also have a “flag” byte which gets toggled upon completion. For instance, the first byte is 0, but then changed to 0xff at completion.

17

Can’t you just write a simple wrapper program that does nothing but execute your inner program? And then renames the file or adds a “done” line or just interacts with the control program in a more direct way? So you’d only need to find a realistic timeout value (depends on how long the thing usually runs), after that the control kills wrapper and inner program and starts it at again with the same parameters (and maybe a number for maximum retries - you don’t want to have the same endless loop just one level higher ^^). You’d put all the organization into the control and start as many inner programs as you Computer has cores, if they are not multi-threaded.

18

This was (sort of) one of the things I tried. The first thing I did when writing a file was to write a dozen or so spaces. After the inner program finished its data writing, it would go back and write a line “# bytes written: xxxxxx” (where “#” is to indicate a comment to the outer program, and “xxxxxx” is replaced by the numerical string representation of the file length – I made sure that there were at least enough spaces (10? I don’t remember now) to hold a maximum integer).

It worked…sort of. One issue was that I needed to keep the outer program looping to check the file length if it was too small (opening read only in the outer program shouldn’t interfere with the inner program’s write). There were other issues that I don’t recall right now. Writing a temporary file then renaming was just so much easier – bordering on trivial.

This is (essentially) the indirection that I referred to above. While it has a lot of nice characteristics (for instance, one program works for arbitrarily-long running inner programs), there are a couple ugly issues that are still difficult to handle (if not insurmountable). Most notably, if the inner program really does hang rather than fail, you still need some custom watchdog built into the outer program (e.g., a timer and/or a CPU check as OP requested, which I haven’t implemented). If it were Unix and/or a specific language, I might be able to provide info; on Windows and/or with some arbitrary language, not so much. :frowning:

One reason I mentioned multi-threading above was that I think one way to deal with the watchdog issue is to put a listener thread into the inner program, then have the outer program periodically ping it to check for continued execution. If the process froze, then the thread shouldn’t respond. At least, AFAIK – though I’d appreciate correction if someone knows otherwise.

But all that works for me because I have total access to both outer (IDL/ENVI), middle (C++), and inner (Java) programs. Besides which, I rely on Cygwin on Windows computers (required by an unrelated aspect of the framework), thus giving me access to Unix commands/tools. Based on the relatively scant details that Napier supplied, it wasn’t clear to me that the technique would actually be either helpful or useful. So I didn’t post any more about it.

Though I’m still happy to go into more detail about my implementation if its desirable.

19

I missed the above until just now. It’s not clear to me that it’s even possible to have the outer program rename the file while avoiding the issue. That is, renaming a partially written file is just as likely as reading a partially written file (in fact, I’d expect it to happen just as often in just the same circumstances).

In my case, the rename happens in the inner program, so that it is impossible that the outer program read it unless and until it is fully written (since the inner program will not rename it until it is fully written and flushed).

20

To me it sounds like, essentially, you’re trying to get around having to debug your programs. That they hang and are unpredictable is the issue, not how to counteract that, unless you’re working with some sort of complex cloud computing sort of deal.

Ultimately, halfbaked code is halfbaked. If you’re the one who made it, adding further layers of complexity just means that there’s further layers of halfbaked code.