POLL - How should we hanlde the case, when DB server is gone away?

Hello.

Recently, one of our customers run into the following problem: the
server does not reconnect to DB, if the connection is lost. Moreover,
everything is working, except flushing changes into the database (so,
after restart you will lose all changes made since that disconnect). It
is fixed in the current cvs (and will be available as a new release
after a couple of days, when some additional testing will be done), but
we still can't decide the best way for handling this situation.

We see three ways of solving the problem (currently, we use #1):

1) When the connection is lost, the sever keeps all changes in memory
and constantly tries to reconnect and flush them. If you shutdown the
server, changes are gone (we don't save queues at the moment). Works
good, but this solution has a major drawback: memory. While it's perfect
for small installations, it can be painful with big systems (around
2mb/minute on a system with 1000 nodes, 10 DCIs on each, and 1 minute
polling interval).

2) Shutdown the server when DB is gone, leaving only a thread to
reconnect. When the connection is restored - this thread fires up the core.
Drawbacks:
a) monitoring is disabled (in case #1 events will be processed, so you
will get alarms, but they will not be stored in DB)
b) we use multiple connections, it's not-so-clear to me, how we should
handle cases like "one connection closed and can't reconnect, two still
working".

3) Mix: use the first solution until queue size limit is reached, then
shutdown the server and wait for reconnection.

Any suggestions?

wbr, alex.
Received on Mon Aug 21 2006 - 12:55:21 EEST