DRF can't fit in 320GB cluster memory for 1.33GB Frame

Description

1) Launched a 10-node cluster (2a174063c) on d1-d10, with -Xmx=32GB each, total 320GB, with java -jar (NO HADOOP)

2) Created a Frame that has 1M rows and 2.2k columns, 80% are sparse columns:

http://mr-0xd1:53322/2/CreateFrame.query?key=testframe&rows=1000000&cols=2200&seed=7595850248774471522&randomize=1&value=0&real_range=100&categorical_fraction=0.05&binary_fraction=0.8&binary_ones_fraction=0.001&factors=5&integer_fraction=0.1&integer_range=100&missing_fraction=0&response_factors=2

This frame uses 1.33GB memory after compression (about 6.7GB if saved to disk). Note: there’s about 1k chunks (4*32*10), so the total number of keys is around 2.2M, so no need to rebalance to fewer keys for now.

NOTE: You can also parse hdfs://mr-0xd6/datasets/1Mx2.2k.csv instead.

3) Launched DRF on the frame with all default options except for only using 5 trees:

http://mr-0xd1:53322/2/DRF.query?destination_key=&source=testframe&response=response&ignored_cols=&classification=1&validation=&n_folds=0&holdout_fraction=0.0&keep_cross_validation_splits=0&ntrees=5&max_depth=20&min_rows=1&nbins=20&score_each_iteration=0&importance=1&balance_classes=0&max_after_balance_size=Infinity&checkpoint=&overwrite_checkpoint=1&mtries=-1&sample_rate=0.6666666865348816&seed=-1&build_tree_one_node=0

Logs:

30-Dec 22:32:36.658 172.16.2.181:53322 14015 main INFO WATER: ----- H2O started -----
30-Dec 22:32:36.659 172.16.2.181:53322 14015 main INFO WATER: Build git branch: master
30-Dec 22:32:36.659 172.16.2.181:53322 14015 main INFO WATER: Build git hash: 2a174063ca5ffdc4bd117a7206e45d89fa56317b
30-Dec 22:32:36.660 172.16.2.181:53322 14015 main INFO WATER: Build git describe: jenkins-master-1643-9-g2a17406
30-Dec 22:32:36.660 172.16.2.181:53322 14015 main INFO WATER: Build project version: 2.9.0.99999
30-Dec 22:32:36.660 172.16.2.181:53322 14015 main INFO WATER: Built by: 'arno'
30-Dec 22:32:36.660 172.16.2.181:53322 14015 main INFO WATER: Built on: 'Tue Dec 30 22:26:44 PST 2014'
30-Dec 22:32:36.660 172.16.2.181:53322 14015 main INFO WATER: Java availableProcessors: 32
30-Dec 22:32:36.661 172.16.2.181:53322 14015 main INFO WATER: Java heap totalMemory: 1.92 gb
30-Dec 22:32:36.662 172.16.2.181:53322 14015 main INFO WATER: Java heap maxMemory: 28.44 gb
30-Dec 22:32:36.662 172.16.2.181:53322 14015 main INFO WATER: Java version: Java 1.7.0_72 (from Oracle Corporation)
30-Dec 22:32:36.662 172.16.2.181:53322 14015 main INFO WATER: OS version: Linux 3.13.0-35-generic (amd64)
30-Dec 22:32:36.716 172.16.2.181:53322 14015 main INFO WATER: Machine physical memory: 251.89 gb
30-Dec 22:32:36.914 172.16.2.181:53322 14015 main INFO WATER: Max. number of factor levels per column: 1000000
30-Dec 22:32:36.915 172.16.2.181:53322 14015 main INFO WATER: ICE root: '/tmp/h2o-arno'
30-Dec 22:32:36.918 172.16.2.181:53322 14015 main INFO WATER: Possible IP Address: vethJQ8838 (vethJQ8838), fe80:0:0:0:fc0f:adff:fe93:af9b%104
30-Dec 22:32:36.918 172.16.2.181:53322 14015 main INFO WATER: Possible IP Address: br2 (br2), fe80:0:0:0:a236:9fff:fe35:4262%9
30-Dec 22:32:36.918 172.16.2.181:53322 14015 main INFO WATER: Possible IP Address: br2 (br2), 172.16.2.181
30-Dec 22:32:36.918 172.16.2.181:53322 14015 main INFO WATER: Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1
30-Dec 22:32:36.918 172.16.2.181:53322 14015 main INFO WATER: Possible IP Address: lo (lo), 127.0.0.1
30-Dec 22:32:36.952 172.16.2.181:53322 14015 main INFO WATER: Internal communication uses port: 53323
+ Listening for HTTP and REST traffic on http://172.16.2.181:53322/
30-Dec 22:32:37.003 172.16.2.181:53322 14015 main INFO WATER: H2O cloud name: 'ArnoDemo'
30-Dec 22:32:37.004 172.16.2.181:53322 14015 main INFO WATER: (v2.9.0.99999) 'ArnoDemo' on /172.16.2.181:53322, discovery address /228.164.126.213:58532
30-Dec 22:32:37.004 172.16.2.181:53322 14015 main INFO WATER: If you have trouble connecting, try SSH tunneling from your local machine (e.g., via port 55555):
+ 1. Open a terminal and run 'ssh -L 55555:localhost:53322 arno@172.16.2.181'
+ 2. Point your browser to http://localhost:55555
30-Dec 22:32:37.008 172.16.2.181:53322 14015 main DEBG WATER: Announcing new Cloud Membership: /172.16.2.181:53322
30-Dec 22:32:37.009 172.16.2.181:53322 14015 main INFO WATER: Cloud of size 1 formed /172.16.2.181:53322 (00:00:00.000)
30-Dec 22:32:37.010 172.16.2.181:53322 14015 main INFO WATER: Log dir: '/tmp/h2o-arno/h2ologs'
30-Dec 22:32:37.011 172.16.2.181:53322 14015 main INFO WATER: HDFS version specified on the command line: hdp2.1
30-Dec 22:32:38.200 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Cloud voting in progress
30-Dec 22:32:38.203 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0x5821a606 /172.16.2.187:53322=0xac10d306 /172.16.2.181:53322, /172.16.2.187:53322
30-Dec 22:32:38.204 172.16.2.181:53322 14015 FJ-126-15 DEBG WATER: Heartbeat hashes differ, self=0x4327907 /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.187:53322
30-Dec 22:32:38.204 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb0434c0b /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.185:53322, /172.16.2.187:53322
30-Dec 22:32:38.212 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0x5c541f0d /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.185:53322, /172.16.2.187:53322
30-Dec 22:32:38.271 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0x864f212 /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.187:53322
30-Dec 22:32:38.324 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb475c519 /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322
30-Dec 22:32:38.659 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0x60869821 /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.189:53322
30-Dec 22:32:38.811 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0x60869821 /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.189:53322
30-Dec 22:32:38.849 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.182:53322=0x5821a607 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:38.910 172.16.2.181:53322 14015 ApiWatch DEBG WATER: ApiPortWatchdog: Thread run() started
30-Dec 22:32:38.915 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Tutorials.html
30-Dec 22:32:39.024 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.183:53322=0xb0434c0d /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.215 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.183:53322=0xb0434c0d /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.222 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.186:53322=0x864f212 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.280 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.188:53322=0xb475c519 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.337 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.187:53322=0x60869821 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.670 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xc976b2a /172.16.2.187:53322=0x60869821 /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.707 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.182:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.821 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.182:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:39.859 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.182:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:40.035 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.183:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:40.229 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.183:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:40.234 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.186:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:40.292 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.188:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:40.353 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Heartbeat hashes differ, self=0xb8a83e2d /172.16.2.189:53322=0xc976b2a /172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:41.364 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:41.463 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Cloud.html
30-Dec 22:32:41.469 172.16.2.181:53322 14015 # Session INFO WATER: H2O Cloud Status:
30-Dec 22:32:41.476 172.16.2.181:53322 14015 # Session INFO WATER: "version":"2.9.0.99999","cloud_name":"ArnoDemo","node_name":"/172.16.2.181:53322","cloud_size":10,"cloud_uptime_millis":4815,"cloud_healthy":true,"nodes":[
30-Dec 22:32:41.476 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.181:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1541729280,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47421849600,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.08,"elapsed_time":199,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"14015","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":65,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561268
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.182:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1692361728,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47452258304,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.16,"elapsed_time":420,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21095","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561047
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.183:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1703162880,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47515172864,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.25,"elapsed_time":219,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"1752","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561248
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.184:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1767775232,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47348449280,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":4.07,"elapsed_time":752,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"705","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007560716
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.185:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1703164928,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47473229824,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.81,"elapsed_time":230,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"27587","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561238
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.186:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1713986560,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":40319844352,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":2.42,"elapsed_time":162,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"22122","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561306
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.187:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1692427264,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47467986944,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":13.09,"elapsed_time":639,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"29270","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007560829
30-Dec 22:32:41.477 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.188:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1714013184,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47509929984,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.41,"elapsed_time":105,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"7110","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007561363
30-Dec 22:32:41.478 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.189:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1649051648,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47470084096,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.1,"elapsed_time":785,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"24944","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007560683
30-Dec 22:32:41.478 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.190:53322","num_keys":0,"value_size_bytes":0,"free_mem_bytes":1649051648,"tot_mem_bytes":2058354688,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":0.0,"free_disk_bytes":47680847872,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":0.0,"system_load":0.61,"elapsed_time":596,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21689","fj_threads_lo":[],"fj_queue_lo":[],"fj_threads_hi":[0,0,0,0,0,0,0],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":0,"open_fds":64,"my_cpu%":"N/A","sys_cpu_%":"N/A","last_contact":1420007560873
30-Dec 22:32:41.478 172.16.2.181:53322 14015 # Session INFO WATER: ],"consensus":false,"locked":false
30-Dec 22:32:41.700 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:41.748 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:41.837 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:41.890 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:42.058 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:42.249 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:42.258 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Missing common knowledge from all nodes!/172.16.2.182:53322, /172.16.2.181:53322, /172.16.2.184:53322, /172.16.2.183:53322, /172.16.2.186:53322, /172.16.2.185:53322, /172.16.2.188:53322, /172.16.2.187:53322, /172.16.2.190:53322, /172.16.2.189:53322
30-Dec 22:32:42.316 172.16.2.181:53322 14015 FJ-126-29 DEBG WATER: Announcing new Cloud Membership: /172.16.2.181:53322, /172.16.2.182:53322, /172.16.2.183:53322, /172.16.2.184:53322, /172.16.2.185:53322, /172.16.2.186:53322, /172.16.2.187:53322, /172.16.2.188:53322, /172.16.2.189:53322, /172.16.2.190:53322
30-Dec 22:32:42.317 172.16.2.181:53322 14015 FJ-126-29 INFO WATER: Cloud of size 10 formed /172.16.2.181:53322 (00:00:05.164), /172.16.2.182:53322 (00:00:05.383), /172.16.2.183:53322 (00:00:05.131), /172.16.2.184:53322 (00:00:03.690), /172.16.2.185:53322 (00:00:05.178), /172.16.2.186:53322 (00:00:05.142), /172.16.2.187:53322 (00:00:05.569), /172.16.2.188:53322 (00:00:05.052), /172.16.2.189:53322 (00:00:04.737), /172.16.2.190:53322 (00:00:04.555)
30-Dec 22:32:45.114 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/CreateFrame.query binary_ones_fraction=0.001 binary_fraction=0.8 randomize=1 missing_fraction=0 integer_fraction=0.1 real_range=100 seed=7595850248774471522 response_factors=2 factors=5 cols=2200 integer_range=100 value=0 categorical_fraction=0.05 key=testframe rows=1000000
30-Dec 22:32:49.796 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/CreateFrame.html binary_ones_fraction=0.001 binary_fraction=0.8 randomize=1 missing_fraction=0 integer_fraction=0.1 real_range=100 seed=7595850248774471522 response_factors=2 factors=5 cols=2200 integer_range=100 value=0 categorical_fraction=0.05 key=testframe rows=1000000
30-Dec 22:33:02.972 172.16.2.181:53322 14015 # Session INFO WATER: Created frame 'testframe'.
30-Dec 22:33:09.498 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/Inspect2.html src_key=testframe
30-Dec 22:33:28.200 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/DRF.query source=testframe
30-Dec 22:33:32.327 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/DRF.query response=response seed=-1 balance_classes=0 score_each_iteration=0 max_after_balance_size=Infinity sample_rate=0.6666666865348816 nbins=20 build_tree_one_node=0 importance=1 keep_cross_validation_splits=0 holdout_fraction=0.0 source=testframe min_rows=1 max_depth=20 n_folds=0 ntrees=50 mtries=-1 overwrite_checkpoint=1
30-Dec 22:39:47.326 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/DRF.query classification=1 response=response seed=-1 balance_classes=0 score_each_iteration=0 max_after_balance_size=Infinity sample_rate=0.6666666865348816 nbins=20 build_tree_one_node=0 importance=1 keep_cross_validation_splits=0 holdout_fraction=0.0 source=testframe min_rows=1 max_depth=20 n_folds=0 ntrees=5 mtries=-1 overwrite_checkpoint=1
30-Dec 22:40:07.496 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /2/DRF.html classification=1 response=response seed=-1 balance_classes=0 score_each_iteration=0 max_after_balance_size=Infinity sample_rate=0.6666666865348816 nbins=20 build_tree_one_node=0 importance=1 keep_cross_validation_splits=0 holdout_fraction=0.0 source=testframe min_rows=1 max_depth=20 n_folds=0 ntrees=5 mtries=-1 overwrite_checkpoint=1
30-Dec 22:40:07.542 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: Building H2O DRF model with these parameters:
30-Dec 22:40:07.552 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: {
30-Dec 22:40:07.552 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "job_key": "$0301ac1002b54bd0ffffffff$_b7e3479087f6fbd42aaeb3b9c1044059",
30-Dec 22:40:07.553 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "destination_key": "DRF_aebc83401c431197c435d774a3b142ff",
30-Dec 22:40:07.553 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "description": "Distributed RF",
30-Dec 22:40:07.553 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "start_time": 1420008007519,
30-Dec 22:40:07.553 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "end_time": 0,
30-Dec 22:40:07.554 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "state": "RUNNING",
30-Dec 22:40:07.554 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "source": {
30-Dec 22:40:07.554 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "_key": "testframe",
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "num_cols": 2201,
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "num_rows": 1000000,
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "num_used_cols": 2200
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: },
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "response": {
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "name": "response"
30-Dec 22:40:07.555 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: },
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "classification": "true",
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "n_folds": 0,
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "holdout_fraction": 0.0,
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "keep_cross_validation_splits": "false",
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "xval_models": [],
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "ntrees": 5,
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "max_depth": 20,
30-Dec 22:40:07.556 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "min_rows": 1,
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "nbins": 20,
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "score_each_iteration": "false",
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "balance_classes": "false",
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "max_after_balance_size": "Infinity",
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "overwrite_checkpoint": "true",
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "_mtry": 46,
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: "_seed": 3170528642942386000
30-Dec 22:40:07.557 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: }
30-Dec 22:40:07.697 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Prior class distribution: [0.500124, 0.499876]
30-Dec 22:40:07.698 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Model class distribution: null
30-Dec 22:40:08.836 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 22:40:08.837 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is NaN, with 0x2 trees (average of 0.0 nodes)
30-Dec 22:40:08.837 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 0 errors on 0 rows, CM= [[0, 0], [0, 0]]
30-Dec 22:51:11.545 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: 1. tree was built 00:11:02.689 (Wall: 30-Dec 22:51:11.545)
30-Dec 22:51:11.903 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 22:51:11.904 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is 0.3640858144006929, with 1x2 trees (average of 179446.5 nodes)
30-Dec 22:51:11.904 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 164116 errors on 333313 rows, CM= [[84693, 81967], [82149, 84504]]
30-Dec 22:51:11.948 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Computation of variable importance with 1th-tree took: 00:00:00.044 (Wall: 30-Dec 22:51:11.948)
30-Dec 22:54:30.967 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Cloud.html
30-Dec 22:54:30.974 172.16.2.181:53322 14015 # Session INFO WATER: H2O Cloud Status:
30-Dec 22:54:30.980 172.16.2.181:53322 14015 # Session INFO WATER: "version":"2.9.0.99999","cloud_name":"ArnoDemo","node_name":"/172.16.2.181:53322","cloud_size":10,"cloud_uptime_millis":1314318,"cloud_healthy":true,"nodes":[
30-Dec 22:54:30.980 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.181:53322","num_keys":304710,"value_size_bytes":149069824,"free_mem_bytes":9254578176,"tot_mem_bytes":28555870208,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":8.16466542748299E10,"free_disk_bytes":47420801024,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.18000000000001,"system_load":0.62,"elapsed_time":397,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"14015","fj_threads_lo":[3,1],"fj_queue_lo":[0,0],"fj_threads_hi":[17,1,0,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":102,"my_cpu%":5,"sys_cpu_%":5,"last_contact":1420008870572
30-Dec 22:54:30.980 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.182:53322","num_keys":282625,"value_size_bytes":137782272,"free_mem_bytes":14928144384,"tot_mem_bytes":18996527104,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":6.7118892897072784E10,"free_disk_bytes":47452258304,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.87200000000001,"system_load":0.75,"elapsed_time":656,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21095","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":5,"sys_cpu_%":5,"last_contact":1420008870314
30-Dec 22:54:30.980 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.183:53322","num_keys":282624,"value_size_bytes":136826880,"free_mem_bytes":8421508096,"tot_mem_bytes":12551454720,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.465590777316414E10,"free_disk_bytes":47515172864,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.942,"system_load":0.71,"elapsed_time":473,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"1752","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":2,"last_contact":1420008870498
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.184:53322","num_keys":282624,"value_size_bytes":136828928,"free_mem_bytes":9398592512,"tot_mem_bytes":12130975744,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.0845117595966354E10,"free_disk_bytes":47348449280,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":29.285000000000007,"system_load":6.12,"elapsed_time":735,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"705","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":36,"last_contact":1420008870237
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.185:53322","num_keys":282624,"value_size_bytes":136828928,"free_mem_bytes":7619120128,"tot_mem_bytes":12437684224,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":9.036797526532576E10,"free_disk_bytes":47473229824,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":31.119000000000003,"system_load":30.46,"elapsed_time":39,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"27587","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":22,"last_contact":1420008870933
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.186:53322","num_keys":282625,"value_size_bytes":136827904,"free_mem_bytes":7531493376,"tot_mem_bytes":18088984576,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.08090549127899E10,"free_disk_bytes":40319844352,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.00300000000001,"system_load":10.54,"elapsed_time":508,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"22122","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":5,"sys_cpu_%":20,"last_contact":1420008870464
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.187:53322","num_keys":282625,"value_size_bytes":136941568,"free_mem_bytes":7334728704,"tot_mem_bytes":11810111488,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.416037165434036E10,"free_disk_bytes":47467986944,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":18.577,"system_load":11.22,"elapsed_time":729,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"29270","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":23,"last_contact":1420008870244
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.188:53322","num_keys":282625,"value_size_bytes":136828928,"free_mem_bytes":8903839744,"tot_mem_bytes":16845897728,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.1003314023594894E10,"free_disk_bytes":47509929984,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.87200000000001,"system_load":0.73,"elapsed_time":412,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"7110","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":4,"sys_cpu_%":4,"last_contact":1420008870561
30-Dec 22:54:30.981 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.189:53322","num_keys":282624,"value_size_bytes":136827904,"free_mem_bytes":7144208384,"tot_mem_bytes":14533263360,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.599631726311291E10,"free_disk_bytes":47470084096,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.861999999999995,"system_load":0.58,"elapsed_time":189,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"24944","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":3,"last_contact":1420008870784
30-Dec 22:54:30.982 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.190:53322","num_keys":282624,"value_size_bytes":136826880,"free_mem_bytes":8431889408,"tot_mem_bytes":12709789696,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.570260489005854E10,"free_disk_bytes":47680847872,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.156000000000006,"system_load":0.61,"elapsed_time":773,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21689","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":2,"last_contact":1420008870200
30-Dec 22:54:30.982 172.16.2.181:53322 14015 # Session INFO WATER: ],"consensus":true,"locked":true
30-Dec 22:59:34.110 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:00:08.936 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Cloud.html
30-Dec 23:00:08.946 172.16.2.181:53322 14015 # Session INFO WATER: H2O Cloud Status:
30-Dec 23:00:08.950 172.16.2.181:53322 14015 # Session INFO WATER: "version":"2.9.0.99999","cloud_name":"ArnoDemo","node_name":"/172.16.2.181:53322","cloud_size":10,"cloud_uptime_millis":1652287,"cloud_healthy":true,"nodes":[
30-Dec 23:00:08.950 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.181:53322","num_keys":304710,"value_size_bytes":149069824,"free_mem_bytes":4620040192,"tot_mem_bytes":30541873152,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":6.5348076945909904E10,"free_disk_bytes":47421849600,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":33.748,"system_load":3.57,"elapsed_time":764,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"14015","fj_threads_lo":[3,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,0,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":4,"tcps_active":18,"open_fds":102,"my_cpu%":9,"sys_cpu_%":9,"last_contact":1420009208174
30-Dec 23:00:08.951 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.182:53322","num_keys":282625,"value_size_bytes":137782272,"free_mem_bytes":13573648384,"tot_mem_bytes":24553979904,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.529666661721057E10,"free_disk_bytes":47452258304,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.50299999999999,"system_load":1.73,"elapsed_time":74,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21095","fj_threads_lo":[1,31],"fj_queue_lo":[0,0],"fj_threads_hi":[30,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":101,"my_cpu%":7,"sys_cpu_%":6,"last_contact":1420009208864
30-Dec 23:00:08.951 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.183:53322","num_keys":282624,"value_size_bytes":136826880,"free_mem_bytes":6496364544,"tot_mem_bytes":12551454720,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":4.1902551729950226E10,"free_disk_bytes":47515172864,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.79099999999999,"system_load":0.24,"elapsed_time":869,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"1752","fj_threads_lo":[1,28],"fj_queue_lo":[0,0],"fj_threads_hi":[22,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":1,"sys_cpu_%":1,"last_contact":1420009208069
30-Dec 23:00:08.951 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.184:53322","num_keys":282624,"value_size_bytes":136828928,"free_mem_bytes":13222873088,"tot_mem_bytes":18658885632,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.4032599834714405E10,"free_disk_bytes":47348449280,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":14.229,"system_load":20.87,"elapsed_time":286,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"705","fj_threads_lo":[1,2],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,5,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":101,"my_cpu%":4,"sys_cpu_%":42,"last_contact":1420009208658
30-Dec 23:00:08.951 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.185:53322","num_keys":282624,"value_size_bytes":136828928,"free_mem_bytes":9249271808,"tot_mem_bytes":16776691712,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.3301786083628384E10,"free_disk_bytes":47473229824,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":32.206999999999994,"system_load":8.81,"elapsed_time":654,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"27587","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":36,"last_contact":1420009208291
30-Dec 23:00:08.951 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.186:53322","num_keys":282625,"value_size_bytes":136827904,"free_mem_bytes":7774970880,"tot_mem_bytes":28015853568,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":4.0538076891865776E10,"free_disk_bytes":40319844352,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.031,"system_load":11.44,"elapsed_time":2671,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"22122","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":6,"sys_cpu_%":52,"last_contact":1420009206274
30-Dec 23:00:08.952 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.187:53322","num_keys":282625,"value_size_bytes":136941568,"free_mem_bytes":5369607168,"tot_mem_bytes":11810111488,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.93657818301204E10,"free_disk_bytes":47467986944,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":12.491000000000001,"system_load":8.96,"elapsed_time":233,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"29270","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":1,"sys_cpu_%":41,"last_contact":1420009208712
30-Dec 23:00:08.952 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.188:53322","num_keys":282625,"value_size_bytes":136828928,"free_mem_bytes":10358898688,"tot_mem_bytes":21229469696,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.9826132981482723E10,"free_disk_bytes":47509929984,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.855,"system_load":0.69,"elapsed_time":215,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"7110","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":3,"last_contact":1420009208730
30-Dec 23:00:08.952 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.189:53322","num_keys":282624,"value_size_bytes":136827904,"free_mem_bytes":8870020096,"tot_mem_bytes":19819659264,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.4246984136037617E10,"free_disk_bytes":47470084096,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.477000000000004,"system_load":0.44,"elapsed_time":713,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"24944","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":4,"sys_cpu_%":4,"last_contact":1420009208233
30-Dec 23:00:08.952 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.190:53322","num_keys":282624,"value_size_bytes":136826880,"free_mem_bytes":9355334656,"tot_mem_bytes":15518400512,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.2277958283176273E10,"free_disk_bytes":47680847872,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.565999999999995,"system_load":0.77,"elapsed_time":43,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21689","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":2,"last_contact":1420009208903
30-Dec 23:00:08.952 172.16.2.181:53322 14015 # Session INFO WATER: ],"consensus":true,"locked":true
30-Dec 23:00:19.281 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:01:17.675 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:01:57.752 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:01:57.881 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: 2. tree was built 00:10:45.920 (Wall: 30-Dec 23:01:57.881)
30-Dec 23:01:58.914 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 23:01:58.915 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is 0.3534496757788199, with 2x2 trees (average of 183524.0 nodes)
30-Dec 23:01:58.915 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 273057 errors on 554974 rows, CM= [[140965, 136586], [136471, 140952]]
30-Dec 23:01:58.947 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Computation of variable importance with 2th-tree took: 00:00:00.031 (Wall: 30-Dec 23:01:58.946)
30-Dec 23:06:31.721 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:07:11.848 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:08:14.082 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:08:54.223 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:10:04.563 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:10:49.697 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:12:05.599 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:12:08.367 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: 3. tree was built 00:10:09.411 (Wall: 30-Dec 23:12:08.367)
30-Dec 23:12:08.700 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 23:12:08.700 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is 0.34286711256341296, with 3x2 trees (average of 180764.5 nodes)
30-Dec 23:12:08.700 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 345203 errors on 702925 rows, CM= [[178716, 173104], [172099, 179006]]
30-Dec 23:12:08.713 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Computation of variable importance with 3th-tree took: 00:00:00.013 (Wall: 30-Dec 23:12:08.713)
30-Dec 23:12:15.737 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:19:53.320 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:20:33.440 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:21:35.372 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:22:15.484 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:22:17.390 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: 4. tree was built 00:10:08.671 (Wall: 30-Dec 23:22:17.390)
30-Dec 23:22:18.567 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 23:22:18.571 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is 0.33250216397235116, with 4x2 trees (average of 174593.5 nodes)
30-Dec 23:22:18.571 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 393770 errors on 802071 rows, CM= [[203742, 197518], [196252, 204559]]
30-Dec 23:22:18.582 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Computation of variable importance with 4th-tree took: 00:00:00.011 (Wall: 30-Dec 23:22:18.582)
30-Dec 23:24:35.195 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Cloud.html
30-Dec 23:24:35.199 172.16.2.181:53322 14015 # Session INFO WATER: H2O Cloud Status:
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "version":"2.9.0.99999","cloud_name":"ArnoDemo","node_name":"/172.16.2.181:53322","cloud_size":10,"cloud_uptime_millis":3118545,"cloud_healthy":true,"nodes":[
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.181:53322","num_keys":304715,"value_size_bytes":154689536,"free_mem_bytes":15191811072,"tot_mem_bytes":30541873152,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.2082425368052208E10,"free_disk_bytes":47421849600,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.995000000000005,"system_load":1.87,"elapsed_time":403,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"14015","fj_threads_lo":[3,32],"fj_queue_lo":[0,0],"fj_threads_hi":[8,1,0,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":4,"tcps_active":18,"open_fds":102,"my_cpu%":7,"sys_cpu_%":7,"last_contact":1420010674793
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.182:53322","num_keys":282625,"value_size_bytes":137782272,"free_mem_bytes":21002746880,"tot_mem_bytes":27889500160,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.1127162440303364E10,"free_disk_bytes":47452258304,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.961000000000006,"system_load":2.79,"elapsed_time":1041,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21095","fj_threads_lo":[1,28],"fj_queue_lo":[0,0],"fj_threads_hi":[27,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":101,"my_cpu%":5,"sys_cpu_%":5,"last_contact":1420010674155
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.183:53322","num_keys":282624,"value_size_bytes":136826880,"free_mem_bytes":8964659200,"tot_mem_bytes":17817927680,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":6.579650394908313E10,"free_disk_bytes":47515172864,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":34.769000000000005,"system_load":0.23,"elapsed_time":754,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"1752","fj_threads_lo":[1,28],"fj_queue_lo":[0,0],"fj_threads_hi":[24,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":2,"last_contact":1420010674443
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.184:53322","num_keys":282625,"value_size_bytes":137789440,"free_mem_bytes":13877695488,"tot_mem_bytes":20246429696,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.578487268856157E10,"free_disk_bytes":47348449280,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":33.98,"system_load":1.88,"elapsed_time":16,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"705","fj_threads_lo":[1,30],"fj_queue_lo":[0,0],"fj_threads_hi":[30,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":36,"last_contact":1420010675181
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.185:53322","num_keys":282625,"value_size_bytes":137791488,"free_mem_bytes":10049432576,"tot_mem_bytes":14003208192,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.509292321658429E10,"free_disk_bytes":47473229824,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.172000000000004,"system_load":1.91,"elapsed_time":897,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"27587","fj_threads_lo":[1,32],"fj_queue_lo":[0,0],"fj_threads_hi":[31,1,1,1,2,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":39,"last_contact":1420010674301
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.186:53322","num_keys":282625,"value_size_bytes":136827904,"free_mem_bytes":26975673344,"tot_mem_bytes":30456414208,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.5022168626075672E10,"free_disk_bytes":40319844352,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":39.979000000000006,"system_load":5.32,"elapsed_time":77,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"22122","fj_threads_lo":[1,2],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,6,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":2,"tcps_active":18,"open_fds":101,"my_cpu%":5,"sys_cpu_%":28,"last_contact":1420010675121
30-Dec 23:24:35.203 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.187:53322","num_keys":282626,"value_size_bytes":137857024,"free_mem_bytes":9458949120,"tot_mem_bytes":18238930944,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.623815658149762E10,"free_disk_bytes":47467986944,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":29.205999999999996,"system_load":15.17,"elapsed_time":366,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"29270","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":9,"last_contact":1420010674832
30-Dec 23:24:35.204 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.188:53322","num_keys":282625,"value_size_bytes":136828928,"free_mem_bytes":8503310336,"tot_mem_bytes":24923078656,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":2.6415935446699924E10,"free_disk_bytes":47509929984,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.654,"system_load":1.77,"elapsed_time":388,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"7110","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":4,"last_contact":1420010674810
30-Dec 23:24:35.204 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.189:53322","num_keys":282626,"value_size_bytes":138783744,"free_mem_bytes":14498454528,"tot_mem_bytes":19672858624,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":1.976902289696217E10,"free_disk_bytes":47470084096,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":35.807,"system_load":1.69,"elapsed_time":742,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"24944","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":3,"sys_cpu_%":3,"last_contact":1420010674457
30-Dec 23:24:35.204 172.16.2.181:53322 14015 # Session INFO WATER: "name":"/172.16.2.190:53322","num_keys":282625,"value_size_bytes":137765888,"free_mem_bytes":8835400704,"tot_mem_bytes":16911433728,"max_mem_bytes":30541873152,"mem_bandwidth_bytes_/sec":3.770695098304075E10,"free_disk_bytes":47680847872,"max_disk_bytes":51470401536,"num_cpus":32,"linpack_gflops":36.611,"system_load":1.71,"elapsed_time":984,"node_healthy":true,"cpus_allowed":32,"nthreads":32,"PID":"21689","fj_threads_lo":[1,1],"fj_queue_lo":[0,0],"fj_threads_hi":[1,1,1,1,1,1,1],"fj_queue_hi":[0,0,0,0,0,0,0],"rpcs":0,"tcps_active":18,"open_fds":101,"my_cpu%":2,"sys_cpu_%":2,"last_contact":1420010674215
30-Dec 23:24:35.204 172.16.2.181:53322 14015 # Session INFO WATER: ],"consensus":true,"locked":true
30-Dec 23:27:15.935 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /LogView.html
30-Dec 23:29:36.927 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:30:17.031 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:31:14.458 172.16.2.181:53322 14015 #e Thread INFO CLEAN: Pausing to swap to disk; more memory may help
30-Dec 23:31:14.459 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /Jobs.html
30-Dec 23:31:19.513 172.16.2.181:53322 14015 #mCleaner INFO CLEAN: Continuing after swapping
30-Dec 23:31:19.749 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: 5. tree was built 00:09:01.158 (Wall: 30-Dec 23:31:19.749)
30-Dec 23:31:20.022 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: ==============================================================
30-Dec 23:31:20.022 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Mean Squared Error is 0.3223262189526788, with 5x2 trees (average of 172378.5 nodes)
30-Dec 23:31:20.023 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Total of 425396 errors on 868224 rows, CM= [[220978, 213235], [212161, 221850]]
30-Dec 23:31:20.034 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Computation of variable importance with 5th-tree took: 00:00:00.011 (Wall: 30-Dec 23:31:20.034)
30-Dec 23:31:20.076 172.16.2.181:53322 14015 FJ-0-55 INFO DRF__: Modeling done in 00:51:12.291 (Wall: 30-Dec 23:31:20.076)
30-Dec 23:31:22.034 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: setting training_start_time to: 1420008007785 for Model: DRF_aebc83401c431197c435d774a3b142ff (DRFModel@2111158339)
30-Dec 23:31:22.050 172.16.2.181:53322 14015 FJ-0-55 INFO WATER: setting training_duration_in_ms to: 3074265 for Model: DRF_aebc83401c431197c435d774a3b142ff (DRFModel@2111158339)
30-Dec 23:32:10.564 172.16.2.181:53322 14015 # Session INFO HTTPD: GET /LogView.html

Assignee

Cliff Click

Reporter

Arno Candel

Labels

None

CustomerVisible

Yes

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

None

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

None

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

Components

Priority

Major
Configure