unflake program assignment requirements
you are on the testing team at work. unfortunately, your companies programmers do not have good testing practices. there aren’t many tests written, and the ones that are out there don’t work reliably: they pass some of the time, but they often fail for unrelated reasons. while your developers get their testing act together, your boss wants you to write a tool to at least make the tests relatively usable.
you have to write an unflake program that will rerun a test if it fails the first time (or the 2nd time or 3rd time …). you also need to keep track of stdout and stderr in a single file from the tests, so you need to log them to a file. once the program completes, you need to print how many times the program ran, the exit code or signal that killed the program, and print the program output. the exit code of unflake must match the exit code of the last run. if the last run will killed with a signal, unflake should have an exit code of 255.
the flaky tests sometimes hang, so if a timeout is exceeded, you need to force the running test to stop.
Specification
unflake max_tries max_timeout test_command args…
max_tries – is the maximum amount of times you will try rerunning the test before giving up.
max_timeout – the maximum allowed time for a test to run
test_command – the command you will be running for the test
args – any arguments the test may take
when unflake finishes the current directory will contain files of the form: test_output.X where X is the try number (starting at 1). this file will contain both the stdout and stderr of the test run.
when unflake finishes, it will dump the information about the run and contents of the last run to the screen and exit with a return code of the last run.
make sure you match the format exactly. extra spaces or incorrect capitalization will cause you to lose points. make sure you don’t have any extra print statements and that each line ends in a linefeed (‘\n’). make sure your files are also named correctly with lower case letters.
Example runs
$ ./unflake
USAGE: ./unflake max_tries max_timeout test_command args…
max_tries – must be greater than or equal to 1
max_timeout – number of seconds greater than or equal to 1
$ echo $?
1
$ ./unflake not_num not_num cmd asdf
USAGE: ./unflake max_tries max_timeout test_command args…
max_tries – must be greater than or equal to 1
max_timeout – number of seconds greater than or equal to 1
$ echo $?ec
1
$ ./unflake 2 2 cmd asdf
2 runs
could not exec cmd
exit code 2
$ echo $?
2
$ ls test_output.*
test_output.1 test_output.2
$ cat test_output.1
could not exec cmd
exit code 2
$ cat test_output.2
could not exec cmd
exit code 2
$ rm test_output.*
$ ./unflake 2 2 ./always_succeed.sh
1 runs
i always succeed
exit code 0
$ echo $?
0
$ ./unflake 5 3 ./succeed_after.sh 3
3 runs
i will succeed after 1 tries
0 more to go
exit code 0
$ ls test_output.*
test_output.1 test_output.2 test_output.3
$ cat test_output.1
i will succeed after 3 tries
2 more to go
exit code 2
$ ./unflake 2 3 ./succeed_after.sh 5
2 runs
i will succeed after 4 tries
3 more to go
exit code 3
$ ./unflake 5 3 sleep 5
5 runs
killed with signal 9
$ echo $?
255
test files
always_fail.sh always_succeed.sh succeed_after.sh
Calls to use
when you implement unflake, you will need to use the following system calls: fork, execvp, wait, open, dup2, signal, kill, and alarm.
to get the exit codes from that status returned from wait please look at the macros (they act like functions) in the man page for wait (man 2 wait). when you open a file with create, you will need to set the mode, so you will probably invoke it like:
open(name, O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR)
to format the filename, you might want to look at the snprintf function. finally, when it comes time to kill a child process due to running too long, use the SIGKILL signal.
grading (rubric)
| points | criteria |
| 5 | program compiles without warnings with the -Wall flag |
| 10 | exit code of unflake matches exit code of last run |
| 10 | retry count is processed correctly |
| 10 | the correct output is displayed |
| 10 | the output files for each run are correct |
| 10 | tests that run too long are handled correctly |
| 10 | execvp was used correctly |
| 10 | wait was used correctly |
| 10 | dup2 was used correctly |
| 15 | signal, alarm, and kill were used correctly |
