forum.bitel.ru • Просмотр темы - Тормозит аккаунтинг

Сообщения без ответов | Активные темы

Список форумов » BGBilling » Модули Интернета » Модуль Inet

Часовой пояс: UTC + 5 часов [ Летнее время ]

Тормозит аккаунтинг

Страница 1 из 1

[ Сообщений: 19 ]

Версия для печати

Пред. тема | След. тема

Автор

Сообщение

[LTC]Dimonyga

Заголовок сообщения: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 05:33

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Последнее время в логах аккаунтинга стали появляться вот такие записи

Цитата:

radius 08-29/16:41:53 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full!
radius 08-29/16:42:23 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full!
radius 08-29/16:42:53 ERROR [InetRadiusListener] RadiusListener - RadiusListener accounting queue is full!

В логаз NAS-а соответственно

Цитата:

Aug 29 23:25:24.545: %RADIUS-4-RADIUS_DEAD: RADIUS server 172.16.20.10:1812,1813 is not responding.
Aug 29 23:26:24.545: %RADIUS-4-RADIUS_ALIVE: RADIUS server 172.16.20.10:1812,1813 is being marked alive.

Почитав форум понял что проблема была, и решалась сменой движка mysql на Innodb, но у нас уже innodb стоит.

Цитата:

time mysqladmin processlist > /dev/null
real 0m0.030s
user 0m0.007s
sys 0m0.011s

Загрузка сервака

Цитата:

cat /proc/loadavg
1.11 0.96 0.82 2/743 29048

То-есть все в порядке нормы.
Далее сделали clear rad sta
и через 1 минуту посмотрели

Цитата:

sh rad sta
Auth. Acct. Both
Maximum inQ length: NA NA 3
Maximum waitQ length: NA NA 8
Maximum doneQ length: NA NA 2
Total responses seen: 44 358 402
Packets with responses: 44 358 402
Packets without responses: 0 0 0
Access Rejects : 40
Average response delay(ms): 6 13 12
Maximum response delay(ms): 50 365 365
Number of Radius timeouts: 0 10 10
Duplicate ID detects: 0 0 0
Buffer Allocation Failures: 0 0 0
Maximum Buffer Size (bytes): 231 532 532
Source Port Range: (2 ports only)
1645 - 1646
Last used Source Port/Identifier:
1645/216
1646/22

Elapsed time since counters last cleared: 1m

Перерыли уже много всего в поисках проблемы, подскажите куда копать ?
Maximum response delay(ms) растет достаточно быстро до таймаута.
Number of Radius timeouts соответственно тоже растет.
При этом все связано только с аккаунтингом.

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 05:37

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Через 6 минут

Цитата:

sh rad sta
Auth. Acct. Both
Maximum inQ length: NA NA 3
Maximum waitQ length: NA NA 9
Maximum doneQ length: NA NA 2
Total responses seen: 164 1460 1624
Packets with responses: 164 1460 1624
Packets without responses: 0 0 0
Access Rejects : 153
Average response delay(ms): 8 781 703
Maximum response delay(ms): 225 173296 173296
Number of Radius timeouts: 0 38 38
Duplicate ID detects: 0 0 0
Buffer Allocation Failures: 0 0 0
Maximum Buffer Size (bytes): 242 535 535
Source Port Range: (2 ports only)
1645 - 1646
Last used Source Port/Identifier:
1645/80
1646/125

Elapsed time since counters last cleared: 6m

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 07:44

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Цитата:

sh rad sta
Auth. Acct. Both
Maximum inQ length: NA NA 4
Maximum waitQ length: NA NA 12
Maximum doneQ length: NA NA 3
Total responses seen: 2096 20729 22825
Packets with responses: 2096 20729 22825
Packets without responses: 0 0 0
Access Rejects : 1946
Average response delay(ms): 8 640 582
Maximum response delay(ms): 351 175126 175126
Number of Radius timeouts: 0 458 458
Duplicate ID detects: 0 0 0
Buffer Allocation Failures: 0 0 0
Maximum Buffer Size (bytes): 242 548 548
Source Port Range: (2 ports only)
1645 - 1646
Last used Source Port/Identifier:
1645/32
1646/45

Elapsed time since counters last cleared: 1h19m

Данная статистика показывает что акцэсс данные "затупы" не касаются.

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 13:03

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

Покажите вывод accounting_status.sh. Вывод mysql SHOW PROCESSLIST.
Какую-то свою предобработку запросов делали? Обработку onAccountingStart onAccountingStop?
Конфигурация activemq обновлена viewtopic.php?f=1&t=6695&hilit=activemq ?

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 13:32

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Amir писал(а):

Цитата:

./accounting_status.sh
Java Runtime: Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM [1.6.0_30] /opt/java/jdk1.6.0_30/jre
Runtime name: 16928@bgbilling
Java endorsed dirs: /lib/endorsed:/opt/java/jdk/lib/endorsed
OS: Linux 2.6.32-71.29.1.el6.x86_64 [amd64], file.encoding: UTF-8, user.name: root
Heap sizes: current=74240k free=73464k max=1101376k
Kernel version 5.2 build 1282 from 17.08.2012 17:41:32
Inet version 5.2 build 963 from 22.08.2012 20:13:32
Radius:
accounting-requests per minute start: 1; stop: 1; update: 296
access-requests per minute accept: 0; reject: 0
ignore per minute access-request: 0; accounting-update: 0
Flow listener [null:2001]
flow packets in current minute: 4046

Started: 28.08.2012 19:31:45 Uptime: 1 d 21:00:21
Memory total: 33 816 576; max: 238 616 576; free: 15 005 200
Memory pools:
Non-heap memory[Code Cache]: max: 50 331 648; used: 5 019 520; peek: 5 055 744
Heap memory[PS Eden Space]: max: 88 997 888; used: 5 895 544; peek: 88 932 352
Heap memory[PS Survivor Space]: max: 196 608; used: 163 840; peek: 3 495 376
Heap memory[PS Old Gen]: max: 178 978 816; used: 12 751 992; peek: 52 582 392
Non-heap memory[PS Perm Gen]: max: 85 983 232; used: 30 983 288; peek: 30 983 288
Thread count: 52
Trees in cache: 0
Connections pool to Master status Idle: 2; Active: 3; maxActive: 300; maxIdle: 20

Цитата:

mysql> SHOW PROCESSLIST;
+----+------+-----------------+-----------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------------+-----------+---------+------+-------+------------------+
| 2 | bill | 127.0.0.1:42590 | bgbilling | Sleep | 10 | | NULL |
| 3 | bill | 127.0.0.1:42591 | bgbilling | Sleep | 16 | | NULL |
| 4 | bill | 127.0.0.1:42592 | bgbilling | Sleep | 2 | | NULL |
| 5 | bill | 127.0.0.1:42593 | bgbilling | Sleep | 14 | | NULL |
| 6 | bill | 127.0.0.1:42594 | bgbilling | Sleep | 9 | | NULL |
| 7 | bill | 127.0.0.1:42595 | bgbilling | Sleep | 6 | | NULL |
| 8 | bill | 127.0.0.1:42596 | bgbilling | Sleep | 5 | | NULL |
| 9 | bill | 127.0.0.1:42597 | bgbilling | Sleep | 1 | | NULL |
| 10 | bill | 127.0.0.1:42598 | bgbilling | Sleep | 11 | | NULL |
| 11 | bill | 127.0.0.1:36023 | bgbilling | Sleep | 14 | | NULL |
| 12 | bill | 127.0.0.1:36024 | bgbilling | Sleep | 5 | | NULL |
| 20 | bill | 127.0.0.1:41284 | bgbilling | Sleep | 6 | | NULL |
| 21 | bill | 127.0.0.1:41285 | bgbilling | Sleep | 3 | | NULL |
| 22 | bill | 127.0.0.1:45136 | bgbilling | Sleep | 6 | | NULL |
| 23 | bill | 127.0.0.1:41167 | bgbilling | Sleep | 9 | | NULL |
| 24 | bill | 127.0.0.1:41168 | bgbilling | Sleep | 21 | | NULL |
| 25 | bill | 127.0.0.1:44172 | bgbilling | Sleep | 12 | | NULL |
| 26 | bill | 127.0.0.1:53043 | bgbilling | Sleep | 10 | | NULL |
| 27 | bill | 127.0.0.1:53044 | bgbilling | Sleep | 10 | | NULL |
| 28 | bill | 127.0.0.1:53045 | bgbilling | Sleep | 33 | | NULL |
| 29 | bill | 127.0.0.1:53046 | bgbilling | Sleep | 10 | | NULL |
| 30 | root | localhost | NULL | Query | 0 | NULL | SHOW PROCESSLIST |
+----+------+-----------------+-----------+---------+------+-------+------------------+
22 rows in set (0.01 sec)

Обработка только на акцэсс, то что постили в вики.

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 13:42

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Обновил конфиг для ActiveMQ

Цитата:

#sh rad sta
Auth. Acct. Both
Maximum inQ length: NA NA 4
Maximum waitQ length: NA NA 9
Maximum doneQ length: NA NA 3
Total responses seen: 106 1350 1456
Packets with responses: 106 1350 1456
Packets without responses: 0 0 0
Access Rejects : 97
Average response delay(ms): 206 235 233
Maximum response delay(ms): 4936 287168 287168
Number of Radius timeouts: 0 7 7
Duplicate ID detects: 0 0 0
Buffer Allocation Failures: 0 0 0
Maximum Buffer Size (bytes): 237 537 537
Source Port Range: (2 ports only)
1645 - 1646
Last used Source Port/Identifier:
1645/121
1646/148

Elapsed time since counters last cleared: 4m

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 13:47

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

Покажите вывод jstack <pid>
И inet-accounting.xml

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 30 авг 2012, 13:54

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Код:

2012-08-30 16:52:40
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode):

"ActiveMQ Session Task" prio=10 tid=0x00007fcd88010800 nid=0x52b7 waiting on condition [0x00007fcd62dec000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0297bd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"Attach Listener" daemon prio=10 tid=0x00007fcdbc103000 nid=0x5294 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"InactivityMonitor WriteCheck" daemon prio=10 tid=0x00007fcd88003000 nid=0x49a0 in Object.wait() [0x00007fcdc1b67000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f0a4bde0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"InactivityMonitor ReadCheck" daemon prio=10 tid=0x00007fcd88002800 nid=0x499f in Object.wait() [0x00007fcd62bea000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f0a4c298> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=10 tid=0x00007fcd9c001800 nid=0x499c runnable [0x00007fcdc0f54000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
        at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:527)
        at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
        at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:512)
        at java.io.DataInputStream.readInt(DataInputStream.java:370)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:269)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:211)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:186)
        at java.lang.Thread.run(Thread.java:662)

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64017800 nid=0x4354 in Object.wait() [0x00007fcdc1965000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f0988890> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00000000409f8000 nid=0x197f in Object.wait() [0x00007fcd626e5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x00000000f095c010> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64016800 nid=0x16a9 in Object.wait() [0x00007fcd622e1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f094c338> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007fcd64009000 nid=0x167b in Object.wait() [0x00007fcd62fee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f0935b70> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"flow-p-8-t-10" prio=10 tid=0x00007fcd7c00f000 nid=0xa32 waiting on condition [0x00007fcd630ef000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-9" prio=10 tid=0x00007fcd7c00d800 nid=0xa2f waiting on condition [0x00007fcd631f0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-8" prio=10 tid=0x00007fcd7c00c000 nid=0xa2c waiting on condition [0x00007fcd632f1000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-7" prio=10 tid=0x00007fcd7c00a800 nid=0xa2a waiting on condition [0x00007fcd633f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-6" prio=10 tid=0x00007fcd7c009000 nid=0xa27 waiting on condition [0x00007fcd634f3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-5" prio=10 tid=0x00007fcd7c007800 nid=0xa1d waiting on condition [0x00007fcd635f4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-4" prio=10 tid=0x00007fcd7c006000 nid=0xa17 waiting on condition [0x00007fcd636f5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-3" prio=10 tid=0x00007fcd7c004800 nid=0xa11 waiting on condition [0x00007fcd637f6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-10" prio=10 tid=0x00007fcd80010000 nid=0xa10 waiting on condition [0x00007fcd638f7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-9" prio=10 tid=0x00007fcd8000e800 nid=0xa0e waiting on condition [0x00007fcd639f8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-8" prio=10 tid=0x00007fcd8000d000 nid=0xa0d waiting on condition [0x00007fcd63af9000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-2" prio=10 tid=0x00007fcd7c003000 nid=0xa08 waiting on condition [0x00007fcd63bfa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-7" prio=10 tid=0x00007fcd8000b800 nid=0xa06 waiting on condition [0x00007fcd63cfb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-6" prio=10 tid=0x00007fcd8000a000 nid=0xa00 waiting on condition [0x00007fcd63dfc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-5" prio=10 tid=0x00007fcd80008800 nid=0x9f9 waiting on condition [0x00007fcd63efd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-4" prio=10 tid=0x00007fcd80006800 nid=0x9f6 waiting on condition [0x00007fcd63ffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-3" prio=10 tid=0x00007fcd80004800 nid=0x9f2 waiting on condition [0x00007fcdc0146000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"flow-p-8-t-1" prio=10 tid=0x00007fcd7c002000 nid=0x9f1 waiting on condition [0x00007fcdc0247000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f030bcd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-2" prio=10 tid=0x00007fcd80003000 nid=0x9f0 waiting on condition [0x00007fcdc0348000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"rdsLstnr-p-6-t-1" prio=10 tid=0x00007fcd80001800 nid=0x9ef waiting on condition [0x00007fcdc0449000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032d1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"DestroyJavaVM" prio=10 tid=0x00007fcde8007000 nid=0x731 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"" prio=10 tid=0x00007fcde83d6800 nid=0x9ee runnable [0x00007fcdc054a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00000000f030ba28> (a sun.nio.ch.Util$2)
        - locked <0x00000000f030ba18> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f030b810> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at ru.bitel.bgbilling.kernel.network.flow.FlowListener.run(FlowListener.java:87)
        at java.lang.Thread.run(Thread.java:662)

"InetRadiusListener" prio=10 tid=0x00007fcde8519000 nid=0x9ed runnable [0x00007fcdc064b000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00000000f0328708> (a sun.nio.ch.Util$2)
        - locked <0x00000000f03286f8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000f03284f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
        at ru.bitel.common.io.DatagramChannelListener.run(DatagramChannelListener.java:193)
        at java.lang.Thread.run(Thread.java:662)

"nas-ex-service-p-15-t-1" prio=10 tid=0x00007fcde8420800 nid=0x9c9 waiting on condition [0x00007fcdc1d69000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f05a02d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"event-proc-p-2-t-1" prio=10 tid=0x00007fcd6c001800 nid=0x968 waiting on condition [0x00007fcdc084d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f02971c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"worker-p-13-t-1" prio=10 tid=0x00007fcde843c000 nid=0x936 waiting on condition [0x00007fcdc0a4f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f05a0a30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"accwrkr-1-p-12-t-1" prio=10 tid=0x00007fcde82a8000 nid=0x935 waiting on condition [0x00007fcdc0b50000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f05a51e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"accwrkr-2-p-11-t-1" prio=10 tid=0x00007fcde852b800 nid=0x934 runnable [0x00007fcdc0c50000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
        at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
        at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
        - locked <0x00000000f094f318> (a com.mysql.jdbc.util.ReadAheadInputStream)
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2526)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2979)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2968)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3516)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1986)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2140)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620)
        - locked <0x00000000f094f600> (a java.lang.Object)
        at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1663)
        - locked <0x00000000f094f600> (a java.lang.Object)
        at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)
        at ru.bitel.bgbilling.server.util.PoolGuardConnectionWrapper.commit(PoolGuardConnectionWrapper.java:73)
        at ru.bitel.bgbilling.modules.inet.accounting.SessionFlushingManager.runImpl(SessionFlushingManager.java:203)
        at ru.bitel.common.worker.WorkerTask.run(WorkerTask.java:86)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"accwrkr-3-p-10-t-1" prio=10 tid=0x00007fcde851b800 nid=0x933 waiting on condition [0x00007fcdc0d52000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f05a8eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"hrlydtlggr-p-7-t-1" prio=10 tid=0x00007fcde8462000 nid=0x8b6 waiting on condition [0x00007fcdc1257000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0327f40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"hrlydtlggr-p-5-t-1" prio=10 tid=0x00007fcde83ca000 nid=0x8a8 waiting on condition [0x00007fcdc1c68000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f032c9c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"setup-sched-p-1-t-1" prio=10 tid=0x00007fcde839a800 nid=0x8a7 waiting on condition [0x00007fcdc1864000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f02952f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"evpool-pblsh-p-4-t-2" prio=10 tid=0x00007fcde8424000 nid=0x84a waiting on condition [0x00007fcdc1460000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0321310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"evpool-pblsh-p-4-t-1" prio=10 tid=0x00007fcde83d9800 nid=0x849 waiting on condition [0x00007fcdc1561000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0321310> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"convergenceBalance-p-3-t-1" prio=10 tid=0x00007fcde839f800 nid=0x848 waiting on condition [0x00007fcdc1662000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f00aafc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)
        at ru.bitel.common.worker.WorkerThread.run(WorkerThread.java:40)

"ActiveMQ Scheduler" daemon prio=10 tid=0x00007fcd98005000 nid=0x7e8 in Object.wait() [0x00007fcdc1763000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x00000000f01962c0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=10 tid=0x00007fcde8360000 nid=0x761 in Object.wait() [0x00007fcdc1f6b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00000000f025c030> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Thread-3" daemon prio=10 tid=0x00007fcde8222000 nid=0x751 waiting on condition [0x00007fcdc206c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

"Dispatcher-Thread-4" daemon prio=10 tid=0x00007fcde8268800 nid=0x750 in Object.wait() [0x00007fcde41d2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at ru.bitel.common.logging.Log4jAsyncAppender$Dispatcher.run(Log4jAsyncAppender.java:56)
        - locked <0x00000000f02c0ef8> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:662)

"Thread-2" prio=10 tid=0x00007fcde8269800 nid=0x74e runnable [0x00007fcdc216d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
        - locked <0x00000000f02e2c38> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:462)
        at java.net.ServerSocket.accept(ServerSocket.java:430)
        at ru.bitel.bgbilling.kernel.application.server.CommandPortListener.run(CommandPortListener.java:195)

"Low Memory Detector" daemon prio=10 tid=0x00007fcde8091000 nid=0x749 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007fcde808e800 nid=0x748 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007fcde808b800 nid=0x747 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007fcde8089800 nid=0x746 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007fcde806d800 nid=0x73e in Object.wait() [0x00007fcde49f8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00000000f00925a0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00007fcde806b800 nid=0x73d in Object.wait() [0x00007fcde4af9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00000000f0092560> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00007fcde8064800 nid=0x73c runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fcde801a000 nid=0x736 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fcde801b800 nid=0x737 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fcde801d800 nid=0x738 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fcde801f800 nid=0x739 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007fcde8093800 nid=0x74a waiting on condition

JNI global references: 1191

Код:

<?xml version="1.0" encoding="UTF-8"?>
<application context="accounting">
        <param name="app.name" value="BGInetAccounting"/>
        <param name="app.id" value="2"/>

        <param name="moduleId" value="1"/>

        <param name="db.driver" value="com.mysql.jdbc.Driver"/>
        <param name="db.url" value="jdbc:mysql://127.0.0.1/bgbilling?useUnicode=true&amp;characterEncoding=Cp1251&amp;allowUrlInLocalInfile=true&amp;zeroDateTimeBehavior=convertToNull&amp;jdbcCompliantTruncation=false&amp;queryTimeoutKillsConnection=true"/>
        <param name="db.user" value="bill"/>
        <param name="db.pswd" value="********"/>

        <param name="mq.url" value="failover:(tcp://localhost:61616)"/>
        <param name="mq.user" value="bill"/>
        <param name="mq.pswd" value="*********"/>

        <param name="rootDeviceId" value="1"/>

        <param name="commonIdentifierName" value="rootDeviceId"/>

        <bean name="accounting" class="ru.bitel.bgbilling.modules.inet.accounting.Accounting"/>

        <param name="datalog.radius.dir" value="data/radius" />
        <param name="datalog.flow.dir" value="data/flow" />

        <context name="radius">
                <bean name="processor" class="ru.bitel.bgbilling.modules.inet.radius.InetRadiusProcessor"/>

                <scheduledExecutorService name="hrlydtlggr" corePoolSize="1"/>

                <bean name="radiusDataLogger" class="ru.bitel.bgbilling.modules.inet.radius.RadiusHourlyDataLogger">
                        <param name="scheduledExecutor">hrlydtlggr</param>
                </bean>

                <bean name="radiusListener" class="ru.bitel.bgbilling.modules.inet.radius.InetRadiusListener">
                        <constructor>
                                <param name="host" value=""/>
                                <param name="port" value="1813"/>
                                <param name="byteBufferCapacity">512 * 1024</param>
                                <param name="processor">processor</param>
                                <param name="mode">RadiusListener.Mode.accounting</param>
                                <param name="setup">setup</param>
                                <param name="dataLogger">radiusDataLogger</param>
                        </constructor>
                </bean>
        </context>

        <context name="collector">
                <scheduledExecutorService name="hrlydtlggr" corePoolSize="1"/>

                <bean name="flowDataLogger" class="ru.bitel.bgbilling.modules.inet.collector.IPHourlyDataLogger">
                        <param name="scheduledExecutor">hrlydtlggr</param>
                </bean>

                <bean name="flowListener" class="ru.bitel.bgbilling.modules.inet.collector.InetFlowListener">
                        <constructor factoryMethod="newInstance">
                                <param name="type" value="netflow"/>
                                <param name="host" value=""/>
                                <param name="port" value="2001"/>
                                <param name="threadCount" value="10"/>
                                <param name="recvBufferSize">2 * 1024 * 1024</param>
                                <param name="soRCVBUF">512 * 1024</param>
                                <param name="agentDeviceIds" value="1"/>
                        </constructor>
                </bean>
        </context>
</application>

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 03 сен 2012, 08:18

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 03 сен 2012, 10:35

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

По логам как-будто ничего не делает, кроме сброса в базу. Может быть очень часто настроен сброс?
Или на одном договоре очень много сервисов?
Так же посмотрите лог аккаунтинга - по таймстампам - действительно ли долго происходит обработка одного апдейта. Может быть проблема ери отправке.
Если причина не найдется, нужен доступ.

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 04 сен 2012, 08:50

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Amir писал(а):

Как я понимаю нужен этот лог ?

Цитата:

09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=319
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613605:677987 Add traffic 3=48
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=100
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 3=131
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 598825:677875 Add traffic 3=52
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 3=131
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617394:678558 Add traffic 3=134
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add traffic 3=131
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Update time to 1346723431279
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add time 1
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617286:678450 Add traffic 0=1
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613605:677987 Add traffic 3=52
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613785:677940 Add traffic 4=315
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 613547:677889 Add traffic 3=122
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 4=315
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616607:677838 Add traffic 3=131
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617182:678346 Add traffic 3=129
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 6=72
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Update time to 1346723431279
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add time 5
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 0=5
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616553:677914 Add traffic 3=129
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 5=527
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617182:678346 Add traffic 4=313
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616607:677838 Add traffic 4=315
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 598825:677875 Add traffic 4=41
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617243:678407 Add traffic 4=131
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617051:678215 Add traffic 4=132
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617051:678215 Add traffic 3=99
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 616720:677946 Add traffic 3=48
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617369:678533 Add traffic 4=129
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617418:678582 Add traffic 4=170
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617412:678576 Add traffic 6=72
09-04/11:50:31 DEBUG [flow-p-8-t-5] connection - 617243:678407 Add traffic 3=296
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add time 25
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add traffic 0=25
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-7] connection - 606630:677842 Add traffic 0=25
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add time 26
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add traffic 0=26
09-04/11:50:31 DEBUG [rdsLstnr-p-6-t-9] connection - 617254:678418 Add traffic 0=26

Много сервисов на одном договоре - нет.
Апдэйты в базу - 1 раз в минуту.

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 05 сен 2012, 15:51

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

Цитата:

Как я понимаю нужен этот лог ?

Там где RADIUS-пакеты приходят.
Пока так и не понятно, в чем проблема.

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 07 сен 2012, 07:29

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Такое ощущение что аккаунтинг не может пережевать нэтфлоу данные.

Код:

collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 686026 for direction: 2
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624423:686026 Add traffic 3=40
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:37.214.123.6:10577 / 0:213.108.186.207:1318 40 ToS/DiffServ: 0 nextHop: 0.0.0.0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 686005 for direction: 2
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624402:686005 Add traffic 3=40
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 0:213.108.187.81:3365 / 10:178.63.105.200:123 76 ToS/DiffServ: 0 nextHop: 217.150.48.202
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685921 for direction: 1
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 4
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 624318:685921 Add traffic 4=76
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:89.105.158.206:50533 / 0:213.108.187.93:6881 93 ToS/DiffServ: 0 nextHop: 0.0.0.0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685641 for direction: 2
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 615449:685641 Add traffic 3=93
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 10:83.59.175.52:0 / 0:213.108.186.224:771 159 ToS/DiffServ: 0 nextHop: 0.0.0.0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685540 for direction: 2
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 3
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 612664:685540 Add traffic 3=159
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Processing Flow 0:213.108.186.170:35578 / 10:128.74.135.183:42809 93 ToS/DiffServ: 0 nextHop: 217.150.48.202
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 0
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found session 685661 for direction: 1
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowAgentInterface - Found trafficType 4
collector 09-07/10:30:28 DEBUG [flow-p-8-t-10] connection - 620779:685661 Add traffic 4=93
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Not found iface = 10
collector 09-07/10:30:28 TRACE [flow-p-8-t-10] FlowListenerWorker - Session not found on default iface
mq 09-07/10:30:28 DEBUG [evpool-pblsh-p-4-t-1] EventProcessor - Publish: Event[pool:ru.bitel.bgbilling.modules.inet.accounting.event.InetAccountingEvent] timestamp: 1346977828916; moduleId: 1; pluginId: -1; cid: -1; scid: -1; userId: -1

Во время вот таких вот сообщений в логе пролетают "Radius timeouts"
Но циска мне не сказала что радиус деад, думаю это нужно ловить в вечернее время, когда много народу.

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 07 сен 2012, 17:45

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

А режим логирования TRACE недавно поставили или давно стоит?
Можно конфиг устройства Accounting?

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 09 сен 2012, 07:17

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

TRACE стоял изначально.

Цитата:

bras1.loctelecom.ru#sh ip flow export verbose
Flow export v5 is enabled for main cache
Export source and destination details :
VRF ID : Default
Destination(1) 172.16.20.1 (2001)
Version 5 flow records
90832844 flows exported in 5218395 udp datagrams
0 flows failed due to lack of export packet
0 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped due to fragmentation failures
0 export packets were dropped due to encapsulation fixup failures
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting
0 export packets were dropped due to Card not being able to export

Собственно конфиг заключается в следующем:

Цитата:

ip flow-export destination 172.16.20.1 2001

Цитата:

ip flow ingress
ip flow egress

на interface Virtual-Template1

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 10 сен 2012, 00:00

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

Попробуйте переключить в режим INFO и посмотреть, есть ли разница.
Я имел ввиду конфигурацию устройства (Accounting|Access+Accounting|Process group) в модуле.

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 12 сен 2012, 09:22

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Переключили - толку нет, сейчас попробуем заменить активное оборудование, если не поможет - отпишусь и подниму тему, появилось предположение что iscsi на который сбрасывается NetFlow и общение BRAS-а с биллингом забивают канал, предположение конечно попахивает абсурдом, но проверить стоит.

Вернуться к началу

Amir

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 13 сен 2012, 02:37

Разработчик

Зарегистрирован: 19 дек 2006, 21:04
Сообщения: 5970
Карма: 256

Можете дать доступ, посмотреть?

Вернуться к началу

[LTC]Dimonyga

Заголовок сообщения: Re: Тормозит аккаунтинг

Добавлено: 13 сен 2012, 08:55

Клиент

Зарегистрирован: 12 янв 2012, 16:12
Сообщения: 56
Откуда: г. Новосибирск
Карма: 8

Amir писал(а):

Можете дать доступ, посмотреть?

После замены оборудования если не поможет - заведу тикет в хэлпдэске и там - же напишу данные для доступа. Так как это оборудование ядра сети - заменить его не так просто.

Вернуться к началу

Страница 1 из 1

[ Сообщений: 19 ]

Список форумов » BGBilling » Модули Интернета » Модуль Inet

Часовой пояс: UTC + 5 часов [ Летнее время ]

Кто сейчас на конференции

Сейчас этот форум просматривают: нет зарегистрированных пользователей и гости: 1

Вы не можете начинать темы
Вы не можете отвечать на сообщения
Вы не можете редактировать свои сообщения
Вы не можете удалять свои сообщения
Вы не можете добавлять вложения

Перейти:

BiTel

Тормозит аккаунтинг

Кто сейчас на конференции