BiTel

scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:01 ERROR [pool-2-thread-3] LogCalculator - Tree not found for contract id: 6
scheduler 06-16/09:00:02 ERROR [pool-2-thread-3] LogCalculator - Detected errors => 2402

06-16/09:10:09  INFO [main] DefaultServerSetup - Init DB connection pools
06-16/09:10:10  INFO [main] DefaultServerSetup - Init trash pools..
06-16/09:10:10  INFO [main] DefaultServerSetup - Init trash pool trash_1
06-16/09:10:10  INFO [main] TaskExecuter - Start TaskExecuter
06-16/09:10:10  INFO [Thread-5] TaskExecuter - TaskExecuter => reloadTasks()
06-16/09:10:10  INFO [Thread-10] TaskRunProcessor - Running Task: bitel.billing.server.npay.Recalculator@7114460
06-16/09:10:10  INFO [pool-1-thread-1] Recalculator - PaymentRecalculator time: 01.06.2009 00
06-16/09:10:10  INFO [Thread-10] TaskRunProcessor - Running Task: bitel.billing.server.npay.Recalculator@42ef83d3
06-16/09:10:10  INFO [pool-1-thread-2] Recalculator - PaymentRecalculator time: 01.06.2009 00
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.ipn.IPNTestGates
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.contract.LimitRestorer
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.mail.UserDispatchSender
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.contract.DebtMailSender
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.contract.ContractStatusSetter
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.mail.MailSender
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.ipn.LogCalculator
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.ipn.LogCalculator
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.npay.Calculator
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.script.TimerEventGenerate
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.cerbercrypt.CardContractSetter
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.cerbercrypt.StatusSetter
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.cerbercrypt.Calculator
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.cerbercrypt.Locker
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.cerbercrypt.Synchronizer
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.npay.Calculator
06-16/09:10:10  INFO [Thread-5] TaskExecuter - Task: bitel.billing.server.npay.Calculator
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Calculate current month
06-16/09:10:10  INFO [pool-1-thread-1] Setup - Load config for mid=11
06-16/09:10:10  INFO [Thread-10] TaskRunProcessor - Running Task: bitel.billing.server.npay.Recalculator@5983e7de
06-16/09:10:10  INFO [pool-1-thread-3] Recalculator - PaymentRecalculator time: 01.06.2009 00
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Calculate current month
06-16/09:10:10  INFO [pool-1-thread-3] Setup - Load config for mid=11
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Calculate current month
06-16/09:10:10  INFO [pool-1-thread-2] Setup - Load config for mid=11
06-16/09:10:10  INFO [Thread-10] TaskRunProcessor - Running Task: bitel.billing.server.npay.Recalculator@112c3327
06-16/09:10:10  INFO [pool-1-thread-4] Recalculator - PaymentRecalculator time: 01.06.2009 00
06-16/09:10:10  INFO [Thread-10] TaskRunProcessor - Running Task: bitel.billing.server.npay.Recalculator@1118fa47
06-16/09:10:10  INFO [pool-1-thread-5] Recalculator - PaymentRecalculator time: 01.06.2009 00
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Memory total: 18 546 688; max: 477 233 152; free: 13 286 328
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Memory total: 18 546 688; max: 477 233 152; free: 13 286 328
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Calculate current month
06-16/09:10:10  INFO [pool-1-thread-5] Calculator - Calculate current month
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Memory total: 18 546 688; max: 477 233 152; free: 11 011 896
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Memory total: 18 546 688; max: 477 233 152; free: 10 879 608
06-16/09:10:10  INFO [pool-1-thread-5] Calculator - Memory total: 18 546 688; max: 477 233 152; free: 10 089 992
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Setting balances for independ and subs..
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Setting balances for independ and subs..
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Selected.
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '2009-6-2719-47' for key 'PRIMARY'
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
        at java.lang.reflect.Constructor.newInstance(Unknown Source)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.Util.getInstance(Util.java:381)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:233)
        at bitel.billing.server.contract.bean.BalanceUtils.setAccount(BalanceUtils.java:933)
        at bitel.billing.server.npay.Calculator.executeTask(Calculator.java:462)
        at bitel.billing.server.task.TaskBase.run(TaskBase.java:50)
        at bitel.billing.server.task.TaskBase.startTask(TaskBase.java:72)
        at bitel.billing.server.npay.Recalculator.run(Recalculator.java:45)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Selecting sub account
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Selecting sub account
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Setting balance for super..
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Setting balance for super..
06-16/09:10:10  INFO [pool-1-thread-5] Calculator - Task finished time=75 ms.
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Setting balances for independ and subs..
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-1] Calculator - Task finished time=434 ms.
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Selecting sub account
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Setting balances for independ and subs..
06-16/09:10:10  INFO [pool-1-thread-3] Calculator - Task finished time=269 ms.
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Setting balance for super..
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Selecting sub account
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Setting balance for super..
06-16/09:10:10  INFO [pool-1-thread-2] Calculator - Task finished time=347 ms.
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Selected.
06-16/09:10:10  INFO [pool-1-thread-4] Calculator - Task finished time=188 ms.

./scheduler_status.sh
Sending 'status' on 127.0.0.1:9066
Result: TaskExecuter working
Started: 16.06.2009 09:10:09    Uptime: 0 d 02:04:41
Memory total: 177 995 776; max: 477 233 152; free: 62 851 352
Trees in cache: 53
Connections pool to Master status Idle: 12; Active: 0; maxActive: 300; maxIdle: 20
Connections pool to Trash "trash_1" status Idle: 0; Active: 0; maxActive: 300; maxIdle: 20

./scheduler_status.sh
Sending 'status' on 127.0.0.1:9066
Result: TaskExecuter working
Started: 16.06.2009 21:46:19    Uptime: 0 d 11:15:09
Memory total: 448 331 776; max: 477 233 152; free: 440 749 792
Trees in cache: 36
Connections pool to Master status Idle: 9; Active: 1; maxActive: 300; maxIdle: 20
Connections pool to Trash "trash_1" status Idle: 0; Active: 0; maxActive: 300; maxIdle: 20

jstack -l 8645
2009-06-17 09:04:18
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.3-b02 mixed mode):

"Attach Listener" daemon prio=10 tid=0x0000000040d3f800 nid=0x50d1 waiting on condition [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"pool-2-thread-7" prio=10 tid=0x0000000040f38000 nid=0x3454 waiting on condition [0x0000000005f3e000..0x0000000005f3eb70]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000000ca3f400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"pool-2-thread-5" prio=10 tid=0x0000000040d1d800 nid=0x2618 waiting on condition [0x000000003c0fc000..0x000000003c0fcb70]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000000ca3f400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"pool-2-thread-4" prio=10 tid=0x0000000040f38c00 nid=0x24fa waiting on condition [0x0000000003523000..0x0000000003523bf0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000000ca3f400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"pool-2-thread-2" prio=10 tid=0x0000000040ed2400 nid=0x23e7 waiting on condition [0x0000000003220000..0x00000000032209f0]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000000ca3f400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"pool-2-thread-1" prio=10 tid=0x0000000040dc5c00 nid=0x23e6 waiting on condition [0x000000003c5ad000..0x000000003c5adb70]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000000ca3f400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(Unknown Source)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
        at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"DestroyJavaVM" prio=10 tid=0x00007f95d03b6c00 nid=0x21c9 waiting on condition [0x0000000000000000..0x0000000000213ce0]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Thread-12" prio=10 tid=0x00007f95d03a4400 nid=0x21e2 waiting on condition [0x000000000311f000..0x000000000311faf0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at bitel.billing.server.admin.eventbus.BillingEventBus$ListenerThread.run(BillingEventBus.java:229)

   Locked ownable synchronizers:
        - None

"Thread-5" prio=10 tid=0x00007f95d04bb800 nid=0x21e1 waiting on condition [0x000000000301e000..0x000000000301ec70]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at bitel.billing.server.TaskExecuter.run(TaskExecuter.java:194)

   Locked ownable synchronizers:
        - None

"Thread-11" prio=10 tid=0x00007f95d0157800 nid=0x21e0 waiting on condition [0x000000003fd90000..0x000000003fd90bf0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at bitel.billing.server.admin.errorlog.AlarmSender.run(AlarmSender.java:115)

   Locked ownable synchronizers:
        - None

"Thread-8" prio=10 tid=0x0000000040dc8000 nid=0x21de runnable [0x0000000002f1d000..0x0000000002f1dcf0]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(Unknown Source)
        - locked <0x000000000ca42998> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(Unknown Source)
        at java.net.ServerSocket.accept(Unknown Source)
        at bitel.billing.server.util.manager.ProcessManager.run(ProcessManager.java:33)

   Locked ownable synchronizers:
        - None

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007f95d03c0c00 nid=0x21dd in Object.wait() [0x0000000002df4000..0x0000000002df4a70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000000ca42ec0> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Unknown Source)
        - locked <0x000000000ca42ec0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"Thread-2" daemon prio=10 tid=0x00007f95d0307400 nid=0x21dc waiting on condition [0x0000000002bd5000..0x0000000002bd59f0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

   Locked ownable synchronizers:
        - None

"Low Memory Detector" daemon prio=10 tid=0x00007f95d000b400 nid=0x21d8 runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"CompilerThread1" daemon prio=10 tid=0x00007f95d0008000 nid=0x21d7 waiting on condition [0x0000000000000000..0x00000000026c3600]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"CompilerThread0" daemon prio=10 tid=0x00007f95d0006400 nid=0x21d6 waiting on condition [0x0000000000000000..0x00000000025c2580]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" daemon prio=10 tid=0x0000000040bc3c00 nid=0x21d5 runnable [0x0000000000000000..0x00000000024c2700]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" daemon prio=10 tid=0x0000000040ba5c00 nid=0x21d4 in Object.wait() [0x00000000023c1000..0x00000000023c19f0]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000000caa3428> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x000000000caa3428> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

   Locked ownable synchronizers:
        - None

"Reference Handler" daemon prio=10 tid=0x0000000040ba3c00 nid=0x21d3 in Object.wait() [0x00000000022c0000..0x00000000022c0b70]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000000caa73c8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x000000000caa73c8> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
        - None

"VM Thread" prio=10 tid=0x0000000040b9b400 nid=0x21d2 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040b3dc00 nid=0x21ca runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040b3f800 nid=0x21cb runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040b41000 nid=0x21cc runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000040b42c00 nid=0x21cd runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000040b44400 nid=0x21ce runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000040b45c00 nid=0x21cf runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000040b47800 nid=0x21d0 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000040b49000 nid=0x21d1 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f95d0015c00 nid=0x21d9 waiting on condition

JNI global references: 846