Ошибка подключения клиента

Started by andrey--k, February 25, 2014, 10:06:10 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions

andrey--k

February 25, 2014, 10:06:10 AM Last Edit: February 25, 2014, 10:34:10 AM by andrey--k
При попытке подключится вылетает
QuoteRequest timed out

В то же время в логе сервера появляется:
Quote
[25-Feb-2014 08:00:00.951] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:00.951] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:02.243] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[25-Feb-2014 08:00:02.243] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:00:32.242] [DEBUG] [CLSN-0] RecvNXCPMessageEx failed (0)
[25-Feb-2014 08:00:43.402] [DEBUG] [CLSN-1] Server time zone: UTC+00UTC
[25-Feb-2014 08:00:43.402] [DEBUG] Database Connection Pool exhausted, fallback connection used
[25-Feb-2014 08:01:13.401] [DEBUG] [CLSN-1] RecvNXCPMessageEx failed (0)
upd. Нашел рекомендацию:
https://www.netxms.org/forum/general-support/database-connection-pool-exhausted-fallback-connection-used/

andrey--k

#1
February 25, 2014, 10:09:16 AM
Остановил netxms через service netxmsd stop.

При запуске $ nxdbmgr check
оказалось что база заблокирована :(

При том что ps aux | grep netxmsd
ничего не выводит.

andrey--k

#2
February 25, 2014, 10:22:10 AM
Перезапустил после проверки.
Та же ошибка...
см аттач.

andrey--k

#3
February 25, 2014, 10:33:01 AM Last Edit: February 25, 2014, 10:42:31 AM by andrey--k
Остановил netxmsd через nxadm -c down.
С первого раза остановка не удалась...
Через минут повторил команду - служба завершилась.
После этого
~$ nxdbmgr check
Выдал
Checking database (excluding collected data):
Database is locked by server

upd.
Решил вместо рестарта службы ребутнуть сервер.
Подключился нормально.

andrey--k

#4
February 26, 2014, 11:56:11 AM
Debian 7 + Oracle 11

Сегодня опять то же самое...
Quote
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used

Пробую:
Quote
netxmsd: down
Proceeding with server shutdown...
Жду минут 5...
Code Select

$ ps ax | grep netxms
2877 ?        Ssl    1:16 /usr/bin/netxmsd -d -D 5

$ tail /var/log/netxms
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.392] [DEBUG] Starting status poll for node PI0-500-6340.it.bank.gov.ua (ID: 1262)
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.392] [DEBUG] Starting status poll for node i0vc8950.it.bank.gov.ua (ID: 1264)
[26-Feb-2014 09:09:11.392] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:11.657] [DEBUG] Finished caching of DCI values
[26-Feb-2014 09:09:24.663] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:09:57.248] [DEBUG] Updating maps...
[26-Feb-2014 09:09:57.248] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:10:16.357] [INFO ] NetXMS Server stopped


Пробуем дальше:
Code Select
$ sudo service netxmsd stop


Процесс ликвидирован.
Все конекты к БД ищезли.

Code Select
nxdbmgr check
NetXMS Database Manager Version 1.2.12 Build 7861

Checking database (excluding collected data):
Database is locked by server ....
Do you wish to force database unlock? (Y/N) y
Database lock removed
* Checking zone objects...                                            [PASSED]
* Checking node objects...                                            [PASSED]
* Checking interface objects...                                       [PASSED]
* Checking network service objects...                                 [PASSED]
* Checking cluster objects...                                         [PASSED]
* Checking template to node mapping...                                [PASSED]
* Checking object properties...                                       [PASSED]
* Checking event processing policy...                                 [PASSED]
* Checking network map links...                                       [PASSED]
Database doesn't contain any errors
Database check completed


При старте:
Code Select

[26-Feb-2014 09:28:37.055] [DEBUG] WARNING: duplicate interface IP address ...
[26-Feb-2014 09:28:37.556] [DEBUG] WARNING: duplicate interface IP address ...
[26-Feb-2014 09:28:38.828] [DEBUG] WARNING: duplicate interface IP address ...
..................

И чуть дальше та же ошибка:
Code Select


[26-Feb-2014 09:31:04.089] [DEBUG] Job Manager worker thread started
[26-Feb-2014 09:31:04.090] [DEBUG] Node poller started
[26-Feb-2014 09:31:04.091] [DEBUG] Event storm detector thread stopped because event storm detection is off
[26-Feb-2014 09:31:04.098] [DEBUG] PollManager: 77 pollers to start
[26-Feb-2014 09:31:04.122] [DEBUG] New DB connection opened: handle=0x5f2a550
[26-Feb-2014 09:31:04.131] [DEBUG] New DB connection opened: handle=0x6377020
[26-Feb-2014 09:31:04.157] [DEBUG] New DB connection opened: handle=0x28e2f60
[26-Feb-2014 09:31:04.184] [DEBUG] New DB connection opened: handle=0x6359860
[26-Feb-2014 09:31:04.218] [DEBUG] New DB connection opened: handle=0x6bd8b0
[26-Feb-2014 09:31:04.244] [DEBUG] New DB connection opened: handle=0x631c400
[26-Feb-2014 09:31:04.282] [DEBUG] New DB connection opened: handle=0x62c1650
[26-Feb-2014 09:31:04.310] [DEBUG] New DB connection opened: handle=0x62c3d50
[26-Feb-2014 09:31:04.340] [DEBUG] New DB connection opened: handle=0x6282990
[26-Feb-2014 09:31:04.371] [DEBUG] New DB connection opened: handle=0x6263020
[26-Feb-2014 09:31:04.398] [DEBUG] New DB connection opened: handle=0x623f1e0
[26-Feb-2014 09:31:04.426] [DEBUG] New DB connection opened: handle=0x5fafa30
[26-Feb-2014 09:31:04.658] [DEBUG] New DB connection opened: handle=0x61fcf50
[26-Feb-2014 09:31:04.684] [DEBUG] New DB connection opened: handle=0x61b4b60
[26-Feb-2014 09:31:04.713] [DEBUG] New DB connection opened: handle=0x618ea90
[26-Feb-2014 09:31:04.748] [DEBUG] New DB connection opened: handle=0x615c150
[b][26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.748] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:31:04.749] [DEBUG] Database Connection Pool exhausted, fallback connection used[/b]
[26-Feb-2014 09:31:04.749] [DEBUG] Syncer thread started, sync_interval = 60


При попыте подключится клиентом:
Code Select

[26-Feb-2014 09:33:05.192] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[26-Feb-2014 09:33:05.192] [DEBUG] Database Connection Pool exhausted, fallback connection used
.....
[26-Feb-2014 09:33:35.191] [DEBUG] [CLSN-0] RecvNXCPMessageEx failed (0)


Пробую остановить повторно:
Code Select

$ sudo service netxmsd stop

$ tail -n100 /var/log/netxms
[26-Feb-2014 09:35:04.256] [INFO ] NetXMS Server stopped
[26-Feb-2014 09:35:04.256] [DEBUG] DB connection 0x5f2a550 closed
[26-Feb-2014 09:35:04.256] [DEBUG] Syncer thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] UptimeCalculator thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Client keep-alive thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Item poller thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Watchdog thread terminated
[26-Feb-2014 09:35:04.256] [DEBUG] Job Manager worker thread stopped
[26-Feb-2014 09:35:04.256] [DEBUG] PollManager: main thread terminated
[26-Feb-2014 09:35:05.258] [DEBUG] All threads was notified, continue with shutdown
[26-Feb-2014 09:35:05.259] [DEBUG] All objects saved to database
[26-Feb-2014 09:35:05.259] [DEBUG] All users saved to database
[26-Feb-2014 09:35:05.259] [DEBUG] DB connection 0x6105120 closed
[26-Feb-2014 09:35:05.259] [DEBUG] DB connection 0x5d95b00 closed
[26-Feb-2014 09:35:05.259] [DEBUG] Database writer stopped
[26-Feb-2014 09:35:05.464] [DEBUG] SNMP Trap Receiver terminated

проверяю:
Code Select

$ ps aux | grep netxms

Пусто....

Проверяю БД:
Code Select

$ nxdbmgr check
NetXMS Database Manager Version 1.2.12 Build 7861

Checking database (excluding collected data):
Database is locked by server ...
Do you wish to force database unlock? (Y/N)
...

Забллокировано :(

Отправляю в перезагрузку...

Странно ошибка появляется:
Code Select

[26-Feb-2014 09:43:19.050] [DEBUG] Database Connection Pool initialized
[26-Feb-2014 09:43:19.050] [DEBUG] Database Connection Pool maintenance thread started
[26-Feb-2014 09:46:13.126] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.736] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.746] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.788] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:15.864] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.433] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.511] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.619] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:16.673] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.446] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.447] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.512] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.516] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.516] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.579] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:17.607] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:18.669] [DEBUG] Database Connection Pool exhausted, fallback connection used


Но клиент подключается нормально.
Code Select

[26-Feb-2014 09:50:44.869] [DEBUG] [CLSN-0] Server time zone: UTC+00UTC
[26-Feb-2014 09:50:45.081] [DEBUG] [CLSN-0] User [email protected] authenticated
[26-Feb-2014 09:50:48.382] [DEBUG] [CLSN-0] listLibraryImages: category=*ANY*

Victor Kirhenshtein

#5
February 26, 2014, 02:43:54 PM
Попробуйте увеличить кол-во соединений в пуле поменяв параметр ConnectionPoolMaxSize. И пришлите пожалуйста результат команды

nxadm -c "show dbcp"

в момент когда много ошибок "pool exhausted".

inspirit

#6
February 27, 2014, 09:45:18 AM
Quote
Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.127] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.736] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.746] [DEBUG] Database Connection Pool exhausted, fallback connection used
[26-Feb-2014 09:46:13.788] [DEBUG] Database Connection Pool exhausted, fallback connection used

Аналогичная проблема. увеличение ConnectionPoolMaxSize не помогло..

inspirit

#7
February 27, 2014, 10:33:48 AM
Видел вот такое

Quote0x8651a30 27.Feb.2014 12:25:39 config.cpp:221
0x86549a0 27.Feb.2014 12:25:39 config.cpp:221
0x8655148 27.Feb.2014 12:25:39 config.cpp:221
0x8655760 27.Feb.2014 12:25:39 config.cpp:221
0x8674108 27.Feb.2014 12:25:39 config.cpp:221
0xb5707c08 27.Feb.2014 12:25:39 config.cpp:221
0xb5e0f418 27.Feb.2014 12:25:39 config.cpp:221
0x96ba978 27.Feb.2014 12:25:39 config.cpp:221
0xb5f00b88 27.Feb.2014 12:25:39 config.cpp:221
0xb6207c18 27.Feb.2014 12:25:39 config.cpp:221
0xb6503c00 27.Feb.2014 12:25:39 config.cpp:221
0xb620a498 27.Feb.2014 12:25:39 config.cpp:221
0xb573bc10 27.Feb.2014 12:25:39 config.cpp:221
0xb5d82228 27.Feb.2014 12:25:39 config.cpp:221
0x8a5fca8 27.Feb.2014 12:25:39 config.cpp:221
0xab9e1250 27.Feb.2014 12:25:39 config.cpp:221
0xab415b88 27.Feb.2014 12:25:39 config.cpp:221
17 database connections in use

andrey--k

#8
February 27, 2014, 11:25:07 AM
Quote from: inspirit on February 27, 2014, 10:33:48 AM
Видел вот такое

Quote0x8651a30 27.Feb.2014 12:25:39 config.cpp:221
0x86549a0 27.Feb.2014 12:25:39 config.cpp:221
0x8655148 27.Feb.2014 12:25:39 config.cpp:221
0x8655760 27.Feb.2014 12:25:39 config.cpp:221
0x8674108 27.Feb.2014 12:25:39 config.cpp:221
0xb5707c08 27.Feb.2014 12:25:39 config.cpp:221
0xb5e0f418 27.Feb.2014 12:25:39 config.cpp:221
0x96ba978 27.Feb.2014 12:25:39 config.cpp:221
0xb5f00b88 27.Feb.2014 12:25:39 config.cpp:221
0xb6207c18 27.Feb.2014 12:25:39 config.cpp:221
0xb6503c00 27.Feb.2014 12:25:39 config.cpp:221
0xb620a498 27.Feb.2014 12:25:39 config.cpp:221
0xb573bc10 27.Feb.2014 12:25:39 config.cpp:221
0xb5d82228 27.Feb.2014 12:25:39 config.cpp:221
0x8a5fca8 27.Feb.2014 12:25:39 config.cpp:221
0xab9e1250 27.Feb.2014 12:25:39 config.cpp:221
0xab415b88 27.Feb.2014 12:25:39 config.cpp:221
17 database connections in use

У меня в логе по запросам
Code Select

andrey@i0debnxms:/var/log$ grep 'database connections' netxms*
andrey@i0debnxms:/var/log$ grep 'config.cpp' netxms*

пусто...

inspirit

#9
February 27, 2014, 11:39:45 AM
Это не лог netxms. Это я в цикле запустил nxadm -c 'show dbcp'

andrey--k

#10
June 10, 2014, 10:19:42 AM
Ошибка в 1.2.14 не появляется.
Print
Go Up Pages1
User actions