prithvi and I reproduced this with flow (interactively)
we were using two machines, mr-0xa1 and mr-0xa2 with 1G java specified
prithvi did a bunch of his flow tests (with larger heaps and larger datasets etc) and everything looked fine
I was saying I thought s3 parsing was bad, so we went to a cloud of 2 with 1G heaps.
He ran a test that uses small datasets (it had worked with cloud of 1 in this setup)...The test passed. While we were sitting there, h2o water meter hung
we looked at the h2o log and it had lots of out of memory errors. (they have a different message than the h2o heap problem messages I've seen in the pass..talk about 'out of memory' and 'swap')
We redid the test from scratch with the flow cloud status page open to watch memory consumption.
You could see that while water meter was polling, the GC free memory was decreasing with each poll. (by like 20MB or so each time? didn't note exactly)
This means that if you have water meter open on flow for a long enough time, you will eventually get a out of memory error (apparently)
Cliff mentioned that people had seen this before (Amy), but there wasn't a jira yet. So here it is.