GLM throws AIIOB

Description

30-Dec 18:50:19.292 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /2/CreateFrame.html binary_ones_fraction=0.02 binary_fraction=0.8 randomize=1 missing_fraction=0.01 integer_fraction=0.1 real_range=100 seed=-2620970089224153941 response_factors=2 factors=100 cols=1000 integer_range=100 value=0 categorical_fraction=0.1 key=testdata rows=100000
30-Dec 18:50:28.035 10.1.10.82:54321 3815 # Session INFO WATER: Created frame 'testdata'.
30-Dec 18:50:31.389 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /2/Inspect2.html src_key=testdata
30-Dec 18:50:44.751 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /2/GLM2.query source=testdata
30-Dec 18:50:48.490 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /2/GLM2.query response=response non_negative=0 tweedie_variance_power=0.0 standardize=1 strong_rules=1 alpha=0.5 max_iter=100 lambda_min_ratio=-1.0 higher_accuracy=0 link=family_default use_all_factor_levels=0 variable_importances=0 lambda=1.0E-5 prior=-1.0 nlambdas=100 source=testdata lambda_search=0 n_folds=0 family=gaussian beta_epsilon=5.0E-5 intercept=1 max_predictors=-1
30-Dec 18:53:43.425 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /2/GLM2.html response=response non_negative=0 tweedie_variance_power=0.0 standardize=1 strong_rules=1 alpha=0.5 max_iter=100 lambda_min_ratio=-1.0 higher_accuracy=0 link=family_default use_all_factor_levels=0 variable_importances=1 lambda=1.0E-5 prior=-1.0 nlambdas=100 source=testdata lambda_search=1 n_folds=0 family=binomial beta_epsilon=5.0E-5 intercept=1 max_predictors=-1
30-Dec 18:53:43.440 10.1.10.82:54321 3815 # Session INFO WATER:
30-Dec 18:53:43.470 10.1.10.82:54321 3815 # Session INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=0, lambda = Infinity]: created progress hex.glm.GLM2$GLM2_Progress@346d984c
30-Dec 18:53:43.479 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: Building H2O GLM2 model with these parameters:
30-Dec 18:53:43.481 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: {
30-Dec 18:53:43.481 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "job_key": "GLM2Job__8a4c8776b3bc4c281abddb242f24e29",
30-Dec 18:53:43.481 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "destination_key": "GLMModel__829d38e39cb5ae2af5f34998204e4e6a",
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "description": "GLM2",
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "start_time": 0,
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "end_time": 0,
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "state": "CREATED",
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "source": {
30-Dec 18:53:43.482 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "_key": "testdata",
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "num_cols": 1001,
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "num_rows": 100000,
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "num_used_cols": 1000
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: },
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "response": {
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "name": "response"
30-Dec 18:53:43.483 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: },
30-Dec 18:53:43.488 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "family": "binomial",
30-Dec 18:53:43.488 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "link": "logit",
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "tweedie_variance_power": 0.0,
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "n_folds": 0,
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "alpha": [
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: 0.5
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: ],
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "lambda": [
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: 1.0E-5
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: ],
30-Dec 18:53:43.489 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "standardize": "true",
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "intercept": "true",
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "non_negative": "false",
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "variable_importances": "true",
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "beta_epsilon": 5.0E-5,
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "max_iter": 100,
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "_proximalPenalty": 0.0,
30-Dec 18:53:43.490 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "_runAllLambdas": "true",
30-Dec 18:53:43.491 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "tweedie_link_power": 1.0,
30-Dec 18:53:43.492 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: "lambda_max": "NaN"
30-Dec 18:53:43.493 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: }
30-Dec 18:53:44.164 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: setting training_start_time to: 1419994423493 for Model: GLMModel__829d38e39cb5ae2af5f34998204e4e6a (GLMModel@1031870935)
30-Dec 18:53:44.186 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=0, lambda = 0.43312753371317353]: starting computation of lambda = 0.43312753371317353, previous lambda = 0.43312753371317353
30-Dec 18:53:44.189 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=0, lambda = 0.43312753371317353]: strong rule at lambda_value=0.43312753371317353, got 1 active cols out of 11000 total.
30-Dec 18:53:44.358 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=1, lambda = 0.43312753371317353]: Gram computed in 162ms, , step = 1, ADMM: 43 iterations, 6ms (2), subgrad_err=0.00778299699431953
30-Dec 18:53:44.375 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=2, lambda = 0.43312753371317353]: Gram computed in 8ms, , step = 1, ADMM: 41 iterations, 0ms (0), subgrad_err=0.008081244830882267
30-Dec 18:53:44.388 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=3, lambda = 0.43312753371317353]: Gram computed in 8ms, , step = 1, ADMM: 41 iterations, 0ms (0), subgrad_err=0.008085318000534497
30-Dec 18:53:44.389 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=3, lambda = 0.43312753371317353]: converged (reached a fixed point with ~ 1e-6 precision), got 2 nzs
30-Dec 18:53:44.498 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=3, lambda = 0.43312753371317353]: computation of current lambda done in 1005ms
30-Dec 18:53:44.498 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=3, lambda = 0.43312753371317353]: starting computation of lambda = 0.4134412188436563, previous lambda = 0.43312753371317353
30-Dec 18:53:44.501 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=3, lambda = 0.4134412188436563]: strong rule at lambda_value=0.4134412188436563, got 1 active cols out of 11000 total.
30-Dec 18:53:44.513 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=4, lambda = 0.4134412188436563]: Gram computed in 12ms, , step = 1, ADMM: 42 iterations, 0ms (0), subgrad_err=0.0029345031204257854
30-Dec 18:53:44.513 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=4, lambda = 0.4134412188436563]: converged (reached a fixed point with ~ 1e-4 precision), got 2 nzs
30-Dec 18:53:44.610 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=4, lambda = 0.4134412188436563]: computation of current lambda done in 1117ms
30-Dec 18:53:44.610 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=4, lambda = 0.4134412188436563]: starting computation of lambda = 0.3946496773675997, previous lambda = 0.4134412188436563
30-Dec 18:53:44.611 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=4, lambda = 0.3946496773675997]: strong rule at lambda_value=0.3946496773675997, got 1 active cols out of 11000 total.
30-Dec 18:53:44.621 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=5, lambda = 0.3946496773675997]: Gram computed in 10ms, , step = 1, ADMM: 42 iterations, 0ms (0), subgrad_err=0.0024924659361325596
30-Dec 18:53:44.621 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=5, lambda = 0.3946496773675997]: converged (reached a fixed point with ~ 1e-4 precision), got 2 nzs
30-Dec 18:53:44.712 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=5, lambda = 0.3946496773675997]: computation of current lambda done in 1219ms
30-Dec 18:53:44.713 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=5, lambda = 0.3946496773675997]: starting computation of lambda = 0.37671224045333307, previous lambda = 0.3946496773675997
30-Dec 18:53:44.713 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=5, lambda = 0.37671224045333307]: strong rule at lambda_value=0.37671224045333307, got 1 active cols out of 11000 total.
30-Dec 18:53:44.724 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=6, lambda = 0.37671224045333307]: Gram computed in 10ms, , step = 1, ADMM: 42 iterations, 0ms (0), subgrad_err=0.002069478774387734
30-Dec 18:53:44.724 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=6, lambda = 0.37671224045333307]: converged (reached a fixed point with ~ 1e-4 precision), got 2 nzs
30-Dec 18:53:44.808 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=6, lambda = 0.37671224045333307]: computation of current lambda done in 1315ms
30-Dec 18:53:44.808 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=6, lambda = 0.37671224045333307]: starting computation of lambda = 0.35959008773035084, previous lambda = 0.37671224045333307
30-Dec 18:53:44.809 10.1.10.82:54321 3815 FJ-0-3 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=6, lambda = 0.35959008773035084]: strong rule at lambda_value=0.35959008773035084, got 5 active cols out of 11000 total.
30-Dec 18:53:44.829 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=7, lambda = 0.35959008773035084]: Gram computed in 16ms, , step = 1, ADMM: 558 iterations, 3ms (1), subgrad_err=0.004263599306105537
30-Dec 18:53:44.851 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=8, lambda = 0.35959008773035084]: Gram computed in 15ms, , step = 1, ADMM: 558 iterations, 1ms (0), subgrad_err=0.004263727758529345
30-Dec 18:53:44.851 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=8, lambda = 0.35959008773035084]: converged (reached a fixed point with ~ 1e-4 precision), got 6 nzs
30-Dec 18:53:44.930 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=8, lambda = 0.35959008773035084]: computation of current lambda done in 1437ms
30-Dec 18:53:44.931 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=8, lambda = 0.35959008773035084]: starting computation of lambda = 0.3432461632738999, previous lambda = 0.35959008773035084
30-Dec 18:53:44.931 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=8, lambda = 0.3432461632738999]: strong rule at lambda_value=0.3432461632738999, got 5 active cols out of 11000 total.
30-Dec 18:53:44.948 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=9, lambda = 0.3432461632738999]: Gram computed in 16ms, , step = 1, ADMM: 551 iterations, 1ms (0), subgrad_err=0.004074483313297961
30-Dec 18:53:44.948 10.1.10.82:54321 3815 FJ-0-1 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=9, lambda = 0.3432461632738999]: converged (reached a fixed point with ~ 1e-4 precision), got 6 nzs
30-Dec 18:53:45.053 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=9, lambda = 0.3432461632738999]: computation of current lambda done in 1560ms
30-Dec 18:53:45.054 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=9, lambda = 0.3432461632738999]: starting computation of lambda = 0.3276450954081971, previous lambda = 0.3432461632738999
30-Dec 18:53:45.054 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=9, lambda = 0.3276450954081971]: strong rule at lambda_value=0.3276450954081971, got 16 active cols out of 11000 total.
30-Dec 18:53:45.098 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=10, lambda = 0.3276450954081971]: Gram computed in 38ms, , step = 1, ADMM: 7591 iterations, 6ms (1), subgrad_err=0.0045335304397972065
30-Dec 18:53:45.186 10.1.10.82:54321 3815 FJ-0-5 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=11, lambda = 0.3276450954081971]: invoking line search
30-Dec 18:53:45.621 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=11, lambda = 0.3276450954081971]: line search: did not find admissible step, smallest step = 0.030903154382632643, objval = 1.3860060703685075, old objval = 1.3809970057387904
30-Dec 18:53:45.621 10.1.10.82:54321 3815 FJ-0-7 INFO WATER: GLM2[dest=GLMModel__829d38e39cb5ae2af5f34998204e4e6a, iteration=11, lambda = 0.3276450954081971]: Line search did not find feasible step, converged.
30-Dec 18:53:45.628 10.1.10.82:54321 3815 FJ-0-7 ERRR WATER: Job GLM2Job__8a4c8776b3bc4c281abddb242f24e29(GLM2) failed.
30-Dec 18:53:45.628 10.1.10.82:54321 3815 FJ-0-7 ERRR WATER: Got exception 'class java.lang.ArrayIndexOutOfBoundsException', with msg '50'
+ java.lang.ArrayIndexOutOfBoundsException: 50
+ at hex.glm.GLMTask.computeEta(GLMTask.java:31)
+ at hex.glm.GLMTask$GLMIterationTask.processRow(GLMTask.java:255)
+ at hex.FrameTask.map(FrameTask.java:743)
+ at water.MRTask2.compute2(MRTask2.java:417)
+ at water.MRTask2.compute2(MRTask2.java:365)
+ at water.MRTask2.compute2(MRTask2.java:365)
+ at water.MRTask2.compute2(MRTask2.java:365)
+ at water.MRTask2.compute2(MRTask2.java:365)
+ at water.H2O$H2OCountedCompleter.compute(H2O.java:653)
+ at jsr166y.CountedCompleter.exec(CountedCompleter.java:429)
+ at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
+ at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
+ at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
+ at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
30-Dec 19:25:47.547 10.1.10.82:54321 3815 # Session INFO HTTPD: GET /LogView.html

Assignee

New H2O Bugs

Reporter

SriSatish Ambati

Labels

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

None

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

Priority

Major
Configure