| forum.bitel.ru http://forum.bitel.ru/ |
|
| Тормозит аккаунтинг http://forum.bitel.ru/viewtopic.php?f=44&t=7175 |
Страница 1 из 1 |
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 05:33 ] |
| Заголовок сообщения: | Тормозит аккаунтинг |
Последнее время в логах аккаунтинга стали появляться вот такие записи Цитата: radius 08-29/16:41:53 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full! radius 08-29/16:42:23 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full! radius 08-29/16:42:53 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full! В логаз NAS-а соответственно Цитата: Aug 29 23:25:24.545: %RADIUS-4-RADIUS_DEAD: RADIUS server 172.16.20.10:1812,1813 is not responding. Aug 29 23:26:24.545: %RADIUS-4-RADIUS_ALIVE: RADIUS server 172.16.20.10:1812,1813 is being marked alive. Почитав форум понял что проблема была, и решалась сменой движка mysql на Innodb, но у нас уже innodb стоит. Цитата: time mysqladmin processlist > /dev/null real 0m0.030s user 0m0.007s sys 0m0.011s Загрузка сервака Цитата: cat /proc/loadavg 1.11 0.96 0.82 2/743 29048 То-есть все в порядке нормы. Далее сделали clear rad sta и через 1 минуту посмотрели Цитата: sh rad sta Auth. Acct. Both Maximum inQ length: NA NA 3 Maximum waitQ length: NA NA 8 Maximum doneQ length: NA NA 2 Total responses seen: 44 358 402 Packets with responses: 44 358 402 Packets without responses: 0 0 0 Access Rejects : 40 Average response delay(ms): 6 13 12 Maximum response delay(ms): 50 365 365 Number of Radius timeouts: 0 10 10 Duplicate ID detects: 0 0 0 Buffer Allocation Failures: 0 0 0 Maximum Buffer Size (bytes): 231 532 532 Source Port Range: (2 ports only) 1645 - 1646 Last used Source Port/Identifier: 1645/216 1646/22 Elapsed time since counters last cleared: 1m Перерыли уже много всего в поисках проблемы, подскажите куда копать ? Maximum response delay(ms) растет достаточно быстро до таймаута. Number of Radius timeouts соответственно тоже растет. При этом все связано только с аккаунтингом. |
|
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 05:37 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Через 6 минут Цитата: sh rad sta
Auth. Acct. Both Maximum inQ length: NA NA 3 Maximum waitQ length: NA NA 9 Maximum doneQ length: NA NA 2 Total responses seen: 164 1460 1624 Packets with responses: 164 1460 1624 Packets without responses: 0 0 0 Access Rejects : 153 Average response delay(ms): 8 781 703 Maximum response delay(ms): 225 173296 173296 Number of Radius timeouts: 0 38 38 Duplicate ID detects: 0 0 0 Buffer Allocation Failures: 0 0 0 Maximum Buffer Size (bytes): 242 535 535 Source Port Range: (2 ports only) 1645 - 1646 Last used Source Port/Identifier: 1645/80 1646/125 Elapsed time since counters last cleared: 6m |
|
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 07:44 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Цитата: sh rad sta Данная статистика показывает что акцэсс данные "затупы" не касаются.
Auth. Acct. Both Maximum inQ length: NA NA 4 Maximum waitQ length: NA NA 12 Maximum doneQ length: NA NA 3 Total responses seen: 2096 20729 22825 Packets with responses: 2096 20729 22825 Packets without responses: 0 0 0 Access Rejects : 1946 Average response delay(ms): 8 640 582 Maximum response delay(ms): 351 175126 175126 Number of Radius timeouts: 0 458 458 Duplicate ID detects: 0 0 0 Buffer Allocation Failures: 0 0 0 Maximum Buffer Size (bytes): 242 548 548 Source Port Range: (2 ports only) 1645 - 1646 Last used Source Port/Identifier: 1645/32 1646/45 Elapsed time since counters last cleared: 1h19m |
|
| Автор: | Amir [ 30 авг 2012, 13:03 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Покажите вывод accounting_status.sh. Вывод mysql SHOW PROCESSLIST. Какую-то свою предобработку запросов делали? Обработку onAccountingStart onAccountingStop? Конфигурация activemq обновлена viewtopic.php?f=1&t=6695&hilit=activemq ? |
|
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 13:32 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Amir писал(а): Покажите вывод accounting_status.sh. Вывод mysql SHOW PROCESSLIST. Какую-то свою предобработку запросов делали? Обработку onAccountingStart onAccountingStop? Конфигурация activemq обновлена viewtopic.php?f=1&t=6695&hilit=activemq ? Цитата: ./accounting_status.sh Java Runtime: Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM [1.6.0_30] /opt/java/jdk1.6.0_30/jre Runtime name: 16928@bgbilling Java endorsed dirs: /lib/endorsed:/opt/java/jdk/lib/endorsed OS: Linux 2.6.32-71.29.1.el6.x86_64 [amd64], file.encoding: UTF-8, user.name: root Heap sizes: current=74240k free=73464k max=1101376k Kernel version 5.2 build 1282 from 17.08.2012 17:41:32 Inet version 5.2 build 963 from 22.08.2012 20:13:32 Radius: accounting-requests per minute start: 1; stop: 1; update: 296 access-requests per minute accept: 0; reject: 0 ignore per minute access-request: 0; accounting-update: 0 Flow listener [null:2001] flow packets in current minute: 4046 Started: 28.08.2012 19:31:45 Uptime: 1 d 21:00:21 Memory total: 33 816 576; max: 238 616 576; free: 15 005 200 Memory pools: Non-heap memory[Code Cache]: max: 50 331 648; used: 5 019 520; peek: 5 055 744 Heap memory[PS Eden Space]: max: 88 997 888; used: 5 895 544; peek: 88 932 352 Heap memory[PS Survivor Space]: max: 196 608; used: 163 840; peek: 3 495 376 Heap memory[PS Old Gen]: max: 178 978 816; used: 12 751 992; peek: 52 582 392 Non-heap memory[PS Perm Gen]: max: 85 983 232; used: 30 983 288; peek: 30 983 288 Thread count: 52 Trees in cache: 0 Connections pool to Master status Idle: 2; Active: 3; maxActive: 300; maxIdle: 20 Цитата: mysql> SHOW PROCESSLIST; +----+------+-----------------+-----------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+------+-----------------+-----------+---------+------+-------+------------------+ | 2 | bill | 127.0.0.1:42590 | bgbilling | Sleep | 10 | | NULL | | 3 | bill | 127.0.0.1:42591 | bgbilling | Sleep | 16 | | NULL | | 4 | bill | 127.0.0.1:42592 | bgbilling | Sleep | 2 | | NULL | | 5 | bill | 127.0.0.1:42593 | bgbilling | Sleep | 14 | | NULL | | 6 | bill | 127.0.0.1:42594 | bgbilling | Sleep | 9 | | NULL | | 7 | bill | 127.0.0.1:42595 | bgbilling | Sleep | 6 | | NULL | | 8 | bill | 127.0.0.1:42596 | bgbilling | Sleep | 5 | | NULL | | 9 | bill | 127.0.0.1:42597 | bgbilling | Sleep | 1 | | NULL | | 10 | bill | 127.0.0.1:42598 | bgbilling | Sleep | 11 | | NULL | | 11 | bill | 127.0.0.1:36023 | bgbilling | Sleep | 14 | | NULL | | 12 | bill | 127.0.0.1:36024 | bgbilling | Sleep | 5 | | NULL | | 20 | bill | 127.0.0.1:41284 | bgbilling | Sleep | 6 | | NULL | | 21 | bill | 127.0.0.1:41285 | bgbilling | Sleep | 3 | | NULL | | 22 | bill | 127.0.0.1:45136 | bgbilling | Sleep | 6 | | NULL | | 23 | bill | 127.0.0.1:41167 | bgbilling | Sleep | 9 | | NULL | | 24 | bill | 127.0.0.1:41168 | bgbilling | Sleep | 21 | | NULL | | 25 | bill | 127.0.0.1:44172 | bgbilling | Sleep | 12 | | NULL | | 26 | bill | 127.0.0.1:53043 | bgbilling | Sleep | 10 | | NULL | | 27 | bill | 127.0.0.1:53044 | bgbilling | Sleep | 10 | | NULL | | 28 | bill | 127.0.0.1:53045 | bgbilling | Sleep | 33 | | NULL | | 29 | bill | 127.0.0.1:53046 | bgbilling | Sleep | 10 | | NULL | | 30 | root | localhost | NULL | Query | 0 | NULL | SHOW PROCESSLIST | +----+------+-----------------+-----------+---------+------+-------+------------------+ 22 rows in set (0.01 sec) Обработка только на акцэсс, то что постили в вики. |
|
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 13:42 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Обновил конфиг для ActiveMQ Цитата: #sh rad sta
Auth. Acct. Both Maximum inQ length: NA NA 4 Maximum waitQ length: NA NA 9 Maximum doneQ length: NA NA 3 Total responses seen: 106 1350 1456 Packets with responses: 106 1350 1456 Packets without responses: 0 0 0 Access Rejects : 97 Average response delay(ms): 206 235 233 Maximum response delay(ms): 4936 287168 287168 Number of Radius timeouts: 0 7 7 Duplicate ID detects: 0 0 0 Buffer Allocation Failures: 0 0 0 Maximum Buffer Size (bytes): 237 537 537 Source Port Range: (2 ports only) 1645 - 1646 Last used Source Port/Identifier: 1645/121 1646/148 Elapsed time since counters last cleared: 4m |
|
| Автор: | Amir [ 30 авг 2012, 13:47 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Покажите вывод jstack <pid> И inet-accounting.xml |
|
| Автор: | [LTC]Dimonyga [ 30 авг 2012, 13:54 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Код: 2012-08-30 16:52:40 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode): "ActiveMQ Session Task" prio=10 tid=0x00007fcd88010800 nid=0x52b7 waiting on condition [0x00007fcd62dec000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f0297bd8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) "Attach Listener" daemon prio=10 tid=0x00007fcdbc103000 nid=0x5294 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "InactivityMonitor WriteCheck" daemon prio=10 tid=0x00007fcd88003000 nid=0x49a0 in Object.wait() [0x00007fcdc1b67000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f0a4bde0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "InactivityMonitor ReadCheck" daemon prio=10 tid=0x00007fcd88002800 nid=0x499f in Object.wait() [0x00007fcd62bea000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f0a4c298> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=10 tid=0x00007fcd9c001800 nid=0x499c runnable [0x00007fcdc0f54000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527) at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58) at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269) at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186) at java.lang.Thread.run(Thread.java:662) "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64017800 nid=0x4354 in Object.wait() [0x00007fcdc1965000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f0988890> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00000000409f8000 nid=0x197f in Object.wait() [0x00007fcd626e5000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x00000000f095c010> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64016800 nid=0x16a9 in Object.wait() [0x00007fcd622e1000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f094c338> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64009000 nid=0x167b in Object.wait() [0x00007fcd62fee000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f0935b70> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "flow-p-8-t-10" prio=10 tid=0x00007fcd7c00f000 nid=0xa32 waiting on condition [0x00007fcd630ef000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-9" prio=10 tid=0x00007fcd7c00d800 nid=0xa2f waiting on condition [0x00007fcd631f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-8" prio=10 tid=0x00007fcd7c00c000 nid=0xa2c waiting on condition [0x00007fcd632f1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-7" prio=10 tid=0x00007fcd7c00a800 nid=0xa2a waiting on condition [0x00007fcd633f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-6" prio=10 tid=0x00007fcd7c009000 nid=0xa27 waiting on condition [0x00007fcd634f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-5" prio=10 tid=0x00007fcd7c007800 nid=0xa1d waiting on condition [0x00007fcd635f4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-4" prio=10 tid=0x00007fcd7c006000 nid=0xa17 waiting on condition [0x00007fcd636f5000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-3" prio=10 tid=0x00007fcd7c004800 nid=0xa11 waiting on condition [0x00007fcd637f6000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-10" prio=10 tid=0x00007fcd80010000 nid=0xa10 waiting on condition [0x00007fcd638f7000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-9" prio=10 tid=0x00007fcd8000e800 nid=0xa0e waiting on condition [0x00007fcd639f8000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-8" prio=10 tid=0x00007fcd8000d000 nid=0xa0d waiting on condition [0x00007fcd63af9000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-2" prio=10 tid=0x00007fcd7c003000 nid=0xa08 waiting on condition [0x00007fcd63bfa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-7" prio=10 tid=0x00007fcd8000b800 nid=0xa06 waiting on condition [0x00007fcd63cfb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-6" prio=10 tid=0x00007fcd8000a000 nid=0xa00 waiting on condition [0x00007fcd63dfc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-5" prio=10 tid=0x00007fcd80008800 nid=0x9f9 waiting on condition [0x00007fcd63efd000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-4" prio=10 tid=0x00007fcd80006800 nid=0x9f6 waiting on condition [0x00007fcd63ffe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-3" prio=10 tid=0x00007fcd80004800 nid=0x9f2 waiting on condition [0x00007fcdc0146000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "flow-p-8-t-1" prio=10 tid=0x00007fcd7c002000 nid=0x9f1 waiting on condition [0x00007fcdc0247000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-2" prio=10 tid=0x00007fcd80003000 nid=0x9f0 waiting on condition [0x00007fcdc0348000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "rdsLstnr-p-6-t-1" prio=10 tid=0x00007fcd80001800 nid=0x9ef waiting on condition [0x00007fcdc0449000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "DestroyJavaVM" prio=10 tid=0x00007fcde8007000 nid=0x731 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "" prio=10 tid=0x00007fcde83d6800 nid=0x9ee runnable [0x00007fcdc054a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f030ba28> (a sun.nio.ch.Util$2) - locked <0x00000000f030ba18> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f030b810> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at ru.bitel.bgbilling.kernel.network.flow.FlowListener.run(FlowListener.java:87) at java.lang.Thread.run(Thread.java:662) "InetRadiusListener" prio=10 tid=0x00007fcde8519000 nid=0x9ed runnable [0x00007fcdc064b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f0328708> (a sun.nio.ch.Util$2) - locked <0x00000000f03286f8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f03284f0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at ru.bitel.common.io.DatagramChannelListener.run(DatagramChannelListener.java:193) at java.lang.Thread.run(Thread.java:662) "nas-ex-service-p-15-t-1" prio=10 tid=0x00007fcde8420800 nid=0x9c9 waiting on condition [0x00007fcdc1d69000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f05a02d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "event-proc-p-2-t-1" prio=10 tid=0x00007fcd6c001800 nid=0x968 waiting on condition [0x00007fcdc084d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f02971c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "worker-p-13-t-1" prio=10 tid=0x00007fcde843c000 nid=0x936 waiting on condition [0x00007fcdc0a4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f05a0a30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "accwrkr-1-p-12-t-1" prio=10 tid=0x00007fcde82a8000 nid=0x935 waiting on condition [0x00007fcdc0b50000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f05a51e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "accwrkr-2-p-11-t-1" prio=10 tid=0x00007fcde852b800 nid=0x934 runnable [0x00007fcdc0c50000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114) at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161) at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189) - locked <0x00000000f094f318> (a com.mysql.jdbc.util.ReadAheadInputStream) at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2526) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2968) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3516) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1986) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620) - locked <0x00000000f094f600> (a java.lang.Object) at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1663) - locked <0x00000000f094f600> (a java.lang.Object) at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334) at ru.bitel.bgbilling.server.util.PoolGuardConnectionWrapper.commit(PoolGuardConnectionWrapper.java:73) at ru.bitel.bgbilling.modules.inet.accounting.SessionFlushingManager.runImpl(SessionFlushingManager.java:203) at ru.bitel.common.worker.WorkerTask.run(WorkerTask.java:86) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "accwrkr-3-p-10-t-1" prio=10 tid=0x00007fcde851b800 nid=0x933 waiting on condition [0x00007fcdc0d52000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f05a8eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "hrlydtlggr-p-7-t-1" prio=10 tid=0x00007fcde8462000 nid=0x8b6 waiting on condition [0x00007fcdc1257000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f0327f40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "hrlydtlggr-p-5-t-1" prio=10 tid=0x00007fcde83ca000 nid=0x8a8 waiting on condition [0x00007fcdc1c68000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f032c9c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "setup-sched-p-1-t-1" prio=10 tid=0x00007fcde839a800 nid=0x8a7 waiting on condition [0x00007fcdc1864000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f02952f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "evpool-pblsh-p-4-t-2" prio=10 tid=0x00007fcde8424000 nid=0x84a waiting on condition [0x00007fcdc1460000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f0321310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "evpool-pblsh-p-4-t-1" prio=10 tid=0x00007fcde83d9800 nid=0x849 waiting on condition [0x00007fcdc1561000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f0321310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "convergenceBalance-p-3-t-1" prio=10 tid=0x00007fcde839f800 nid=0x848 waiting on condition [0x00007fcdc1662000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f00aafc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40) "ActiveMQ Scheduler" daemon prio=10 tid=0x00007fcd98005000 nid=0x7e8 in Object.wait() [0x00007fcdc1763000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x00000000f01962c0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Timer-0" daemon prio=10 tid=0x00007fcde8360000 nid=0x761 in Object.wait() [0x00007fcdc1f6b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f025c030> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Thread-3" daemon prio=10 tid=0x00007fcde8222000 nid=0x751 waiting on condition [0x00007fcdc206c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104) "Dispatcher-Thread-4" daemon prio=10 tid=0x00007fcde8268800 nid=0x750 in Object.wait() [0x00007fcde41d2000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at ru.bitel.common.logging.Log4jAsyncAppender$Dispatcher.run(Log4jAsyncAppender.java:56) - locked <0x00000000f02c0ef8> (a java.util.ArrayList) at java.lang.Thread.run(Thread.java:662) "Thread-2" prio=10 tid=0x00007fcde8269800 nid=0x74e runnable [0x00007fcdc216d000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <0x00000000f02e2c38> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at ru.bitel.bgbilling.kernel.application.server.CommandPortListener.run(CommandPortListener.java:195) "Low Memory Detector" daemon prio=10 tid=0x00007fcde8091000 nid=0x749 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007fcde808e800 nid=0x748 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007fcde808b800 nid=0x747 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007fcde8089800 nid=0x746 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007fcde806d800 nid=0x73e in Object.wait() [0x00007fcde49f8000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x00000000f00925a0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x00007fcde806b800 nid=0x73d in Object.wait() [0x00007fcde4af9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x00000000f0092560> (a java.lang.ref.Reference$Lock) "VM Thread" prio=10 tid=0x00007fcde8064800 nid=0x73c runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fcde801a000 nid=0x736 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fcde801b800 nid=0x737 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fcde801d800 nid=0x738 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fcde801f800 nid=0x739 runnable "VM Periodic Task Thread" prio=10 tid=0x00007fcde8093800 nid=0x74a waiting on condition JNI global references: 1191 Код: <?xml version="1.0" encoding="UTF-8"?>
<application context="accounting"> <param name="app.name" value="BGInetAccounting"/> <param name="app.id" value="2"/> <param name="moduleId" value="1"/> <param name="db.driver" value="com.mysql.jdbc.Driver"/> <param name="db.url" value="jdbc:mysql://127.0.0.1/bgbilling?useUnicode=true&characterEncoding=Cp1251&allowUrlInLocalInfile=true&zeroDateTimeBehavior=convertToNull&jdbcCompliantTruncation=false&queryTimeoutKillsConnection=true"/> <param name="db.user" value="bill"/> <param name="db.pswd" value="********"/> <param name="mq.url" value="failover:(tcp://localhost:61616)"/> <param name="mq.user" value="bill"/> <param name="mq.pswd" value="*********"/> <param name="rootDeviceId" value="1"/> <param name="commonIdentifierName" value="rootDeviceId"/> <bean name="accounting" class="ru.bitel.bgbilling.modules.inet.accounting.Accounting"/> <param name="datalog.radius.dir" value="data/radius" /> <param name="datalog.flow.dir" value="data/flow" /> <context name="radius"> <bean name="processor" class="ru.bitel.bgbilling.modules.inet.radius.InetRadiusProcessor"/> <scheduledExecutorService name="hrlydtlggr" corePoolSize="1"/> <bean name="radiusDataLogger" class="ru.bitel.bgbilling.modules.inet.radius.RadiusHourlyDataLogger"> <param name="scheduledExecutor">hrlydtlggr</param> </bean> <bean name="radiusListener" class="ru.bitel.bgbilling.modules.inet.radius.InetRadiusListener"> <constructor> <param name="host" value=""/> <param name="port" value="1813"/> <param name="byteBufferCapacity">512 * 1024</param> <param name="processor">processor</param> <param name="mode">RadiusListener.Mode.accounting</param> <param name="setup">setup</param> <param name="dataLogger">radiusDataLogger</param> </constructor> </bean> </context> <context name="collector"> <scheduledExecutorService name="hrlydtlggr" corePoolSize="1"/> <bean name="flowDataLogger" class="ru.bitel.bgbilling.modules.inet.collector.IPHourlyDataLogger"> <param name="scheduledExecutor">hrlydtlggr</param> </bean> <bean name="flowListener" class="ru.bitel.bgbilling.modules.inet.collector.InetFlowListener"> <constructor factoryMethod="newInstance"> <param name="type" value="netflow"/> <param name="host" value=""/> <param name="port" value="2001"/> <param name="threadCount" value="10"/> <param name="recvBufferSize">2 * 1024 * 1024</param> <param name="soRCVBUF">512 * 1024</param> <param name="agentDeviceIds" value="1"/> </constructor> </bean> </context> </application> |
|
| Автор: | [LTC]Dimonyga [ 03 сен 2012, 08:18 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
UP |
|
| Автор: | Amir [ 03 сен 2012, 10:35 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
По логам как-будто ничего не делает, кроме сброса в базу. Может быть очень часто настроен сброс? Или на одном договоре очень много сервисов? Так же посмотрите лог аккаунтинга - по таймстампам - действительно ли долго происходит обработка одного апдейта. Может быть проблема ери отправке. Если причина не найдется, нужен доступ. |
|
| Автор: | [LTC]Dimonyga [ 04 сен 2012, 08:50 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Amir писал(а): По логам как-будто ничего не делает, кроме сброса в базу. Может быть очень часто настроен сброс? Или на одном договоре очень много сервисов? Так же посмотрите лог аккаунтинга - по таймстампам - действительно ли долго происходит обработка одного апдейта. Может быть проблема ери отправке. Если причина не найдется, нужен доступ. Как я понимаю нужен этот лог ? Цитата: 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=319 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613605:677987 Add traffic 3=48 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=100 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 3=131 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 598825:677875 Add traffic 3=52 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 3=131 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617394:678558 Add traffic 3=134 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add traffic 3=131 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Update time to 1346723431279 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add time 1 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add traffic 0=1 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613605:677987 Add traffic 3=52 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=315 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613547:677889 Add traffic 3=122 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 4=315 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616607:677838 Add traffic 3=131 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617182:678346 Add traffic 3=129 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 6=72 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Update time to 1346723431279 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add time 5 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 0=5 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616553:677914 Add traffic 3=129 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 5=527 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617182:678346 Add traffic 4=313 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616607:677838 Add traffic 4=315 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 598825:677875 Add traffic 4=41 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617243:678407 Add traffic 4=131 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617051:678215 Add traffic 4=132 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617051:678215 Add traffic 3=99 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616720:677946 Add traffic 3=48 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 4=129 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617418:678582 Add traffic 4=170 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 6=72 09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617243:678407 Add traffic 3=296 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add time 25 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add traffic 0=25 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add traffic 0=25 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add time 26 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add traffic 0=26 09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add traffic 0=26 Много сервисов на одном договоре - нет. Апдэйты в базу - 1 раз в минуту. |
|
| Автор: | Amir [ 05 сен 2012, 15:51 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Цитата: Как я понимаю нужен этот лог ? Там где RADIUS-пакеты приходят.Пока так и не понятно, в чем проблема. |
|
| Автор: | [LTC]Dimonyga [ 07 сен 2012, 07:29 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Такое ощущение что аккаунтинг не может пережевать нэтфлоу данные. Код: collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 686026 for direction: 2 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624423:686026 Add traffic 3=40 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:37.214.123.6:10577 / 0:213.108.186.207:1318 40 ToS/DiffServ: 0 nextHop: 0.0.0.0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 686005 for direction: 2 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624402:686005 Add traffic 3=40 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 0:213.108.187.81:3365 / 10:178.63.105.200:123 76 ToS/DiffServ: 0 nextHop: 217.150.48.202 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685921 for direction: 1 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 4 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624318:685921 Add traffic 4=76 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:89.105.158.206:50533 / 0:213.108.187.93:6881 93 ToS/DiffServ: 0 nextHop: 0.0.0.0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685641 for direction: 2 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 615449:685641 Add traffic 3=93 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:83.59.175.52:0 / 0:213.108.186.224:771 159 ToS/DiffServ: 0 nextHop: 0.0.0.0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685540 for direction: 2 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 612664:685540 Add traffic 3=159 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 0:213.108.186.170:35578 / 10:128.74.135.183:42809 93 ToS/DiffServ: 0 nextHop: 217.150.48.202 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685661 for direction: 1 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 4 collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 620779:685661 Add traffic 4=93 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10 collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface mq 09-07/10:30:28 DEBUG [evpool-pblsh-p-4-t-1] EventProcessor - Publish: Event[pool:ru.bitel.bgbilling.modules.inet.accounting.event.InetAccountingEvent] timestamp: 1346977828916; moduleId: 1; pluginId: -1; cid: -1; scid: -1; userId: -1 Во время вот таких вот сообщений в логе пролетают "Radius timeouts" Но циска мне не сказала что радиус деад, думаю это нужно ловить в вечернее время, когда много народу. |
|
| Автор: | Amir [ 07 сен 2012, 17:45 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
А режим логирования TRACE недавно поставили или давно стоит? Можно конфиг устройства Accounting? |
|
| Автор: | [LTC]Dimonyga [ 09 сен 2012, 07:17 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
TRACE стоял изначально. Цитата: bras1.loctelecom.ru#sh ip flow export verbose Flow export v5 is enabled for main cache Export source and destination details : VRF ID : Default Destination(1) 172.16.20.1 (2001) Version 5 flow records 90832844 flows exported in 5218395 udp datagrams 0 flows failed due to lack of export packet 0 export packets were sent up to process level 0 export packets were dropped due to no fib 0 export packets were dropped due to adjacency issues 0 export packets were dropped due to fragmentation failures 0 export packets were dropped due to encapsulation fixup failures 0 export packets were dropped enqueuing for the RP 0 export packets were dropped due to IPC rate limiting 0 export packets were dropped due to Card not being able to export Собственно конфиг заключается в следующем: Цитата: ip flow-export destination 172.16.20.1 2001 и Цитата: ip flow ingress на interface Virtual-Template1
ip flow egress |
|
| Автор: | Amir [ 10 сен 2012, 00:00 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Попробуйте переключить в режим INFO и посмотреть, есть ли разница. Я имел ввиду конфигурацию устройства (Accounting|Access+Accounting|Process group) в модуле. |
|
| Автор: | [LTC]Dimonyga [ 12 сен 2012, 09:22 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Переключили - толку нет, сейчас попробуем заменить активное оборудование, если не поможет - отпишусь и подниму тему, появилось предположение что iscsi на который сбрасывается NetFlow и общение BRAS-а с биллингом забивают канал, предположение конечно попахивает абсурдом, но проверить стоит. |
|
| Автор: | Amir [ 13 сен 2012, 02:37 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Можете дать доступ, посмотреть? |
|
| Автор: | [LTC]Dimonyga [ 13 сен 2012, 08:55 ] |
| Заголовок сообщения: | Re: Тормозит аккаунтинг |
Amir писал(а): Можете дать доступ, посмотреть? После замены оборудования если не поможет - заведу тикет в хэлпдэске и там - же напишу данные для доступа. Так как это оборудование ядра сети - заменить его не так просто. |
|
| Страница 1 из 1 | Часовой пояс: UTC + 5 часов [ Летнее время ] |
| Powered by phpBB® Forum Software © phpBB Group http://www.phpbb.com/ |
|