При попытке подключится вылетает
QuoteRequest timed out
В то же время в логе сервера появляется:
Quote
[25-Feb-2014 08:00:00.951] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:00.951] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:02.243] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[25-Feb-2014 08:00:02.243] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:32.242] [DEBUG] [CLSN-0] RecvNXCPMessageEx failed (0)
[25-Feb-2014 08:00:43.402] [DEBUG] [CLSN-1] Server time zone: UTC+00UTC
[25-Feb-2014 08:00:43.402] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:01:13.401] [DEBUG] [CLSN-1] RecvNXCPMessageEx failed (0)
upd. Нашел рекомендацию:
https://www.netxms.org/forum/general-support/database-connection-pool-exhausted-fallback-connection-used/
Остановил netxms через service netxmsd stop.
При запуске $ nxdbmgr check
оказалось что база заблокирована :(
При том что ps aux | grep netxmsd
ничего не выводит.
Перезапустил после проверки.
Та же ошибка...
см аттач.
Остановил netxmsd через nxadm -c down.
С первого раза остановка не удалась...
Через минут повторил команду - служба завершилась.
После этого
~$ nxdbmgr check
Выдал
Checking database (excluding collected data):
Database is locked by server
upd.
Решил вместо рестарта службы ребутнуть сервер.
Подключился нормально.
Debian 7 + Oracle 11
Сегодня опять то же самое...
Quote
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
Пробую:
Quote
netxmsd: down
Proceeding with server shutdown...
Жду минут 5...
$ ps ax | grep netxms
2877 ? Ssl 1:16 /usr/bin/netxmsd -d -D 5
$ tail /var/log/netxms
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.392] [DEBUG] Starting status poll for node PI0-500-6340.it.bank.gov.ua (ID: 1262)
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.392] [DEBUG] Starting status poll for node i0vc8950.it.bank.gov.ua (ID: 1264)
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.657] [DEBUG] Finished caching of DCI values
[26-Feb-2014 09:09:24.663] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:57.248] [DEBUG] Updating maps...
[26-Feb-2014 09:09:57.248] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:10:16.357] [INFO ] NetXMS Server stopped
Пробуем дальше:
$ sudo service netxmsd stop
Процесс ликвидирован.
Все конекты к БД ищезли.
nxdbmgr check
NetXMS Database Manager Version 1.2.12 Build 7861
Checking database (excluding collected data):
Database is locked by server ....
Do you wish to force database unlock? (Y/N) y
Database lock removed
* Checking zone objects... [PASSED]
* Checking node objects... [PASSED]
* Checking interface objects... [PASSED]
* Checking network service objects... [PASSED]
* Checking cluster objects... [PASSED]
* Checking template to node mapping... [PASSED]
* Checking object properties... [PASSED]
* Checking event processing policy... [PASSED]
* Checking network map links... [PASSED]
Database doesn't contain any errors
Database check completed
При старте:
[26-Feb-2014 09:28:37.055] [DEBUG] WARNING: duplicate interface IP address ...
[26-Feb-2014 09:28:37.556] [DEBUG] WARNING: duplicate interface IP address ...
[26-Feb-2014 09:28:38.828] [DEBUG] WARNING: duplicate interface IP address ...
..................
И чуть дальше та же ошибка:
[26-Feb-2014 09:31:04.089] [DEBUG] Job Manager worker thread started
[26-Feb-2014 09:31:04.090] [DEBUG] Node poller started
[26-Feb-2014 09:31:04.091] [DEBUG] Event storm detector thread stopped because event storm detection is off
[26-Feb-2014 09:31:04.098] [DEBUG] PollManager: 77 pollers to start
[26-Feb-2014 09:31:04.122] [DEBUG] New DB connection opened: handle=0x5f2a550
[26-Feb-2014 09:31:04.131] [DEBUG] New DB connection opened: handle=0x6377020
[26-Feb-2014 09:31:04.157] [DEBUG] New DB connection opened: handle=0x28e2f60
[26-Feb-2014 09:31:04.184] [DEBUG] New DB connection opened: handle=0x6359860
[26-Feb-2014 09:31:04.218] [DEBUG] New DB connection opened: handle=0x6bd8b0
[26-Feb-2014 09:31:04.244] [DEBUG] New DB connection opened: handle=0x631c400
[26-Feb-2014 09:31:04.282] [DEBUG] New DB connection opened: handle=0x62c1650
[26-Feb-2014 09:31:04.310] [DEBUG] New DB connection opened: handle=0x62c3d50
[26-Feb-2014 09:31:04.340] [DEBUG] New DB connection opened: handle=0x6282990
[26-Feb-2014 09:31:04.371] [DEBUG] New DB connection opened: handle=0x6263020
[26-Feb-2014 09:31:04.398] [DEBUG] New DB connection opened: handle=0x623f1e0
[26-Feb-2014 09:31:04.426] [DEBUG] New DB connection opened: handle=0x5fafa30
[26-Feb-2014 09:31:04.658] [DEBUG] New DB connection opened: handle=0x61fcf50
[26-Feb-2014 09:31:04.684] [DEBUG] New DB connection opened: handle=0x61b4b60
[26-Feb-2014 09:31:04.713] [DEBUG] New DB connection opened: handle=0x618ea90
[26-Feb-2014 09:31:04.748] [DEBUG] New DB connection opened: handle=0x615c150
[b][26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.749] [DEBUG] Database Connection Pool exhausted, fallback connection used[/b]
[26-Feb-2014 09:31:04.749] [DEBUG] Syncer thread started, sync_interval = 60
При попыте подключится клиентом:
[26-Feb-2014 09:33:05.192] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[26-Feb-2014 09:33:05.192] [DEBUG] Database Connection Pool exhausted, fallback connection used
.....
[26-Feb-2014 09:33:35.191] [DEBUG] [CLSN-0] RecvNXCPMessageEx failed (0)
Пробую остановить повторно:
$ sudo service netxmsd stop
$ tail -n100 /var/log/netxms
[26-Feb-2014 09:35:04.256] [INFO ] NetXMS Server stopped
[26-Feb-2014 09:35:04.256] [DEBUG] DB connection 0x5f2a550 closed
[26-Feb-2014 09:35:04.256] [DEBUG] Syncer thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] UptimeCalculator thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Client keep-alive thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Item poller thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Watchdog thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Job Manager worker thread stopped
[26-Feb-2014 09:35:04.256] [DEBUG] PollManager: main thread terminated
[26-Feb-2014 09:35:05.258] [DEBUG] All threads was notified, continue with shutdown
[26-Feb-2014 09:35:05.259] [DEBUG] All objects saved to database
[26-Feb-2014 09:35:05.259] [DEBUG] All users saved to database
[26-Feb-2014 09:35:05.259] [DEBUG] DB connection 0x6105120 closed
[26-Feb-2014 09:35:05.259] [DEBUG] DB connection 0x5d95b00 closed
[26-Feb-2014 09:35:05.259] [DEBUG] Database writer stopped
[26-Feb-2014 09:35:05.464] [DEBUG] SNMP Trap Receiver terminated
проверяю:
$ ps aux | grep netxms
Пусто....
Проверяю БД:
$ nxdbmgr check
NetXMS Database Manager Version 1.2.12 Build 7861
Checking database (excluding collected data):
Database is locked by server ...
Do you wish to force database unlock? (Y/N)
...
Забллокировано :(
Отправляю в перезагрузку...
Странно ошибка появляется:
[26-Feb-2014 09:43:19.050] [DEBUG] Database Connection Pool initialized
[26-Feb-2014 09:43:19.050] [DEBUG] Database Connection Pool maintenance thread started
[26-Feb-2014 09:46:13.126] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.736] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.746] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.788] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:15.864] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.433] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.511] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.619] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.673] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.512] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.516] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.516] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.579] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.607] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:18.669] [DEBUG] Database Connection Pool exhausted, fallback connection used
Но клиент подключается нормально.
[26-Feb-2014 09:50:44.869] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[26-Feb-2014 09:50:45.081] [DEBUG] [CLSN-0] User [email protected] authenticated
[26-Feb-2014 09:50:48.382] [DEBUG] [CLSN-0] listLibraryImages: category=*ANY*
Попробуйте увеличить кол-во соединений в пуле поменяв параметр ConnectionPoolMaxSize. И пришлите пожалуйста результат команды
nxadm -c "show dbcp"
в момент когда много ошибок "pool exhausted".
Quote
Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.736] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.746] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.788] [DEBUG] Database Connection Pool exhausted, fallback connection used
Аналогичная проблема. увеличение ConnectionPoolMaxSize не помогло..
Видел вот такое
Quote0x8651a30 27.Feb.2014 12:25:39 config.cpp:221
0x86549a0 27.Feb.2014 12:25:39 config.cpp:221
0x8655148 27.Feb.2014 12:25:39 config.cpp:221
0x8655760 27.Feb.2014 12:25:39 config.cpp:221
0x8674108 27.Feb.2014 12:25:39 config.cpp:221
0xb5707c08 27.Feb.2014 12:25:39 config.cpp:221
0xb5e0f418 27.Feb.2014 12:25:39 config.cpp:221
0x96ba978 27.Feb.2014 12:25:39 config.cpp:221
0xb5f00b88 27.Feb.2014 12:25:39 config.cpp:221
0xb6207c18 27.Feb.2014 12:25:39 config.cpp:221
0xb6503c00 27.Feb.2014 12:25:39 config.cpp:221
0xb620a498 27.Feb.2014 12:25:39 config.cpp:221
0xb573bc10 27.Feb.2014 12:25:39 config.cpp:221
0xb5d82228 27.Feb.2014 12:25:39 config.cpp:221
0x8a5fca8 27.Feb.2014 12:25:39 config.cpp:221
0xab9e1250 27.Feb.2014 12:25:39 config.cpp:221
0xab415b88 27.Feb.2014 12:25:39 config.cpp:221
17 database connections in use
Quote from: inspirit on February 27, 2014, 10:33:48 AM
Видел вот такое
Quote0x8651a30 27.Feb.2014 12:25:39 config.cpp:221
0x86549a0 27.Feb.2014 12:25:39 config.cpp:221
0x8655148 27.Feb.2014 12:25:39 config.cpp:221
0x8655760 27.Feb.2014 12:25:39 config.cpp:221
0x8674108 27.Feb.2014 12:25:39 config.cpp:221
0xb5707c08 27.Feb.2014 12:25:39 config.cpp:221
0xb5e0f418 27.Feb.2014 12:25:39 config.cpp:221
0x96ba978 27.Feb.2014 12:25:39 config.cpp:221
0xb5f00b88 27.Feb.2014 12:25:39 config.cpp:221
0xb6207c18 27.Feb.2014 12:25:39 config.cpp:221
0xb6503c00 27.Feb.2014 12:25:39 config.cpp:221
0xb620a498 27.Feb.2014 12:25:39 config.cpp:221
0xb573bc10 27.Feb.2014 12:25:39 config.cpp:221
0xb5d82228 27.Feb.2014 12:25:39 config.cpp:221
0x8a5fca8 27.Feb.2014 12:25:39 config.cpp:221
0xab9e1250 27.Feb.2014 12:25:39 config.cpp:221
0xab415b88 27.Feb.2014 12:25:39 config.cpp:221
17 database connections in use
У меня в логе по запросам
andrey@i0debnxms:/var/log$ grep 'database connections' netxms*
andrey@i0debnxms:/var/log$ grep 'config.cpp' netxms*
пусто...
Это не лог netxms. Это я в цикле запустил nxadm -c 'show dbcp'
Ошибка в 1.2.14 не появляется.