BiTel

Добрый день,

BGBilling v7.0. В базе были проблемы, восстановили бэкап за тот день когда все работало нормально.
На данный момент имеется следующая проблема - биллинг периодически не отвечает циске, в результате чего с интервалом 10-15 минут на ней появляется определенное количество unauthen-сессий.
На циске - все в порядке, в базе смотрим iostat, load average, used memory, mysql check и тд - тоже все ок.
На сервере с брокером (ActiveMQ) все в норме. Number of Pending Messages по нулям.


root@deneb:/usr/local/BGInetAccess# ./access_status.sh
Java Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM [1.8.0_73] /opt/java/jdk1.8.0_73/jre
Runtime name: 29247@deneb.azdata.net
Java endorsed dirs: /lib/endorsed:/opt/java/jdk/lib/endorsed
OS: Linux 3.2.0-4-amd64 [amd64], file.encoding: UTF-8, user.name: root
Heap sizes: current=123904k free=121282k max=1824256k
Kernel version 7.0.1383 / 19.06.2018 20:44:10
Inet version 7.0.895 / 19.06.2018 20:44:21

IP pool [radius.realm.default.ipCategories,dhcp.ipCategories] ipCategoryIds: [1], size: 61, occupied: 45, reserved: 0, localRsrvd: 16, fetchSize: 180
IP pool [dhcp.ipCategories] ipCategoryIds: [33], size: 247, occupied: 181, reserved: 33, localRsrvd: 32, fetchSize: 33
IP pool [radius.disable.ipCategories] ipCategoryIds: [2], size: 0, occupied: 0, reserved: 0, localRsrvd: 0, fetchSize: 50
IP pool [dhcp.ipCategories] ipCategoryIds: [3], size: 0, occupied: 0, reserved: 0, localRsrvd: 0, fetchSize: 50
IP pool [dhcp.ipCategories] ipCategoryIds: [5], size: 631, occupied: 526, reserved: 41, localRsrvd: 80, fetchSize: 40
IP pool [dhcp.ipCategories] ipCategoryIds: [37], size: 186, occupied: 151, reserved: 35, localRsrvd: 35, fetchSize: 180
IP pool [dhcp.ipCategories] ipCategoryIds: [6], size: 631, occupied: 493, reserved: 114, localRsrvd: 112, fetchSize: 56
IP pool [dhcp.ipCategories] ipCategoryIds: [39], size: 186, occupied: 165, reserved: 3, localRsrvd: 21, fetchSize: 180
IP pool [dhcp.ipCategories] ipCategoryIds: [8], size: 1265, occupied: 1062, reserved: 132, localRsrvd: 126, fetchSize: 123
IP pool [dhcp.ipCategories] ipCategoryIds: [41], size: 1012, occupied: 893, reserved: 65, localRsrvd: 61, fetchSize: 57
IP pool [dhcp.ipCategories] ipCategoryIds: [9], size: 250, occupied: 201, reserved: 29, localRsrvd: 28, fetchSize: 30
IP pool [dhcp.ipCategories] ipCategoryIds: [43], size: 506, occupied: 403, reserved: 46, localRsrvd: 42, fetchSize: 50
IP pool [dhcp.ipCategories] ipCategoryIds: [47], size: 567, occupied: 512, reserved: 57, localRsrvd: 55, fetchSize: 31
IP pool [dhcp.ipCategories] ipCategoryIds: [49], size: 506, occupied: 410, reserved: 97, localRsrvd: 96, fetchSize: 87
IP pool [dhcp.ipCategories] ipCategoryIds: [20], size: 186, occupied: 146, reserved: 19, localRsrvd: 18, fetchSize: 25
IP pool [dhcp.ipCategories] ipCategoryIds: [21], size: 378, occupied: 349, reserved: 25, localRsrvd: 28, fetchSize: 92
IP pool [dhcp.ipCategories] ipCategoryIds: [23], size: 125, occupied: 99, reserved: 26, localRsrvd: 26, fetchSize: 180
IP pool [dhcp.ipCategories] ipCategoryIds: [25], size: 253, occupied: 184, reserved: 53, localRsrvd: 51, fetchSize: 77
IP pool [dhcp.ipCategories] ipCategoryIds: [27], size: 314, occupied: 243, reserved: 36, localRsrvd: 71, fetchSize: 37
IP pool [dhcp.ipCategories] ipCategoryIds: [29], size: 186, occupied: 157, reserved: 0, localRsrvd: 29, fetchSize: 125
IP pool [dhcp.ipCategories] ipCategoryIds: [31], size: 250, occupied: 192, reserved: 59, localRsrvd: 58, fetchSize: 39

Radius:
accounting-requests per minute start: 0; stop: 0; update: 0
access-requests per minute accept: 8; reject: 3
ignore per minute access-request: 0; accounting: 0; accounting-update: 0
Antispam ban count: 0; used per minute: 0
DhcpListener:
dhcp-requests per minute (1m/10m/1h): 179.00/211.40/270.47


Started: 04.11.2019 13:19:01 Uptime: 1 d 00:58:51
Memory total: 972 554 240; max: 2 863 661 056; free: 485 185 496
Memory pools:
Non-heap memory[Code Cache]: max: 251 658 240; used: 37 363 968; peek: 37 363 968
Non-heap memory[Metaspace]: max: -1; used: 42 670 112; peek: 42 916 184
Non-heap memory[Compressed Class Space]: max: 1 073 741 824; used: 4 640 920; peek: 4 763 136
Heap memory[PS Eden Space]: max: 1 048 576 000; used: 31 469 440; peek: 1 024 458 752
Heap memory[PS Survivor Space]: max: 12 582 912; used: 4 227 296; peek: 67 606 208
Heap memory[PS Old Gen]: max: 2 147 483 648; used: 451 672 008; peek: 542 407 480
Buffer pools:
[direct]: count: 1322; memoryUsed: 242 612 710
[mapped]: count: 2; memoryUsed: 134
Thread count: 1553
Trees in cache: 0
Connections pool to Master status Idle: 16; Active: 4; maxActive: 300; maxIdle: 20

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------




root@deneb:/usr/local/BGInetAccounting# ./accounting_status.sh
Java Runtime: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM [1.8.0_73] /opt/java/jdk1.8.0_73/jre
Runtime name: 29284@deneb.azdata.net
Java endorsed dirs: /lib/endorsed:/opt/java/jdk/lib/endorsed
OS: Linux 3.2.0-4-amd64 [amd64], file.encoding: UTF-8, user.name: root
Heap sizes: current=123904k free=121282k max=1824256k
Kernel version 7.0.1383 / 19.06.2018 20:44:10
Inet version 7.0.895 / 19.06.2018 20:44:21

IP pool [radius.realm.default.ipCategories] ipCategoryIds: [1], size: 61, occupied: 45, reserved: 0, localRsrvd: 0, fetchSize: 50
IP pool [radius.disable.ipCategories] ipCategoryIds: [2], size: 0, occupied: 0, reserved: 0, localRsrvd: 0, fetchSize: 50
Connections:
[signal] count: 12985; start per minute (1m/10m/1h): 128.00/60.60/57.60; stop per minute (1m/10m/1h): 145.00/58.60/54.80
[traffic] count: 0; start per minute: 0.00; stop per minute: 0.00

Radius:
accounting-requests per minute start: 40; stop: 47; update: 4356
access-requests per minute accept: 0; reject: 0
ignore per minute access-request: 0; accounting: 0; accounting-update: 0

Started: 04.11.2019 13:18:51 Uptime: 1 d 00:59:40
Memory total: 232 259 584; max: 1 908 932 608; free: 30 323 752
Memory pools:
Non-heap memory[Code Cache]: max: 251 658 240; used: 33 633 856; peek: 33 742 592
Non-heap memory[Metaspace]: max: -1; used: 40 725 448; peek: 40 732 040
Non-heap memory[Compressed Class Space]: max: 1 073 741 824; used: 4 480 768; peek: 4 584 984
Heap memory[PS Eden Space]: max: 714 604 544; used: 4 628 792; peek: 698 875 904
Heap memory[PS Survivor Space]: max: 524 288; used: 262 144; peek: 39 338 464
Heap memory[PS Old Gen]: max: 1 431 830 528; used: 197 048 768; peek: 283 148 600
Buffer pools:
[direct]: count: 284; memoryUsed: 35 656 584
[mapped]: count: 0; memoryUsed: 0
Thread count: 311
Trees in cache: 0
Connections pool to Master status Idle: 20; Active: 0; maxActive: 300; maxIdle: 20


----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Подскажите, пожалуйста, откуда начать копать? Какие логи в первую очередь посмотреть?