water meter in flow has a memory leak which eventually makes h2o crash with out of memory error

Description

prithvi and I reproduced this with flow (interactively)

we were using two machines, mr-0xa1 and mr-0xa2 with 1G java specified

prithvi did a bunch of his flow tests (with larger heaps and larger datasets etc) and everything looked fine

I was saying I thought s3 parsing was bad, so we went to a cloud of 2 with 1G heaps.

He ran a test that uses small datasets (it had worked with cloud of 1 in this setup)...The test passed. While we were sitting there, h2o water meter hung

we looked at the h2o log and it had lots of out of memory errors. (they have a different message than the h2o heap problem messages I've seen in the pass..talk about 'out of memory' and 'swap')

We redid the test from scratch with the flow cloud status page open to watch memory consumption.

You could see that while water meter was polling, the GC free memory was decreasing with each poll. (by like 20MB or so each time? didn't note exactly)

This means that if you have water meter open on flow for a long enough time, you will eventually get a out of memory error (apparently)

Cliff mentioned that people had seen this before (Amy), but there wasn't a jira yet. So here it is.

Assignee

New H2O Bugs

Reporter

Kevin Normoyle

Labels

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

None

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

Components

Priority

Major
Configure