Node Down/Up Alerts not sending

Started by redlam, August 30, 2024, 06:51:07 PM

Previous topic - Next topic

redlam

I am attempting to have netxms generate a slack message for when nodes go down and come up. 

I have created a slack alert notification channel that works when hitting the "send notification" button.
I have created a slack alert action.
I have enabled the slack alert server action in the event processing policy for "show alarm when node is down" and "terminate node down alarms when node is up."

From my understanding this should generate a slack message when a node goes down, and generate a second slack message when a node comes up. 

Currently I get no slack alerts when a node goes down or comes up again. What am I missing?

Thanks

Filipp Sudanov

First thing to check, just to make sure, is if you've pressed "save" button in epp editor.

Check Logs -> Server Action Executions and Logs -> Notifications, if your Slack is listed there.

For testing you can add Slack notification actions to maintenance mode enter and leave rules in EPP, then you can just take any node and take it in and out of maintenance.

If it still does not work, pls show screenshots of your EPP rule configuration.

redlam

Thanks for the reply.

I did press Save in the epp editor. In the Notifications Log it does show the slack alerts. I have a device on the bench that I am using to test node down / node up. 

Screenshots attached. 

Filipp Sudanov

I wonder that Recipient column in the log is empty. How is your action configured, recipient is specified in it.

redlam

#4
It's using slack webhooks so no recipient is specified.  Its very strange because if I send notification from within the Notification Channels menu, it works perfectly.

Filipp Sudanov

Ok, but what about message field, it also should be empty?

Anyways, let's add some debug. In Tools -> Server debug console issue

debug ncd.* 7

this will set debug level 7 for notification channel driver related debug. Then check server log file

redlam

Not noticing any change to the log after enabling the debug command. 

Filipp Sudanov

It should be in log file, /var/log/netxmsd on Linux and C:\NetXMS\log\netxmsd.log on Windows

redlam

I see the action in the client gui log, but the log text file is just full of these token issued lines. 


2024.09.05 04:49:38.385 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 04:52:38.445 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 04:55:38.517 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 04:58:38.554 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:01:38.627 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:04:38.665 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:07:38.704 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:10:38.760 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:13:38.829 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:16:38.893 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:19:38.940 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:22:38.991 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:25:39.038 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:28:39.077 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:31:39.112 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:34:39.160 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:37:39.211 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:40:39.276 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:43:39.334 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:46:39.376 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:49:39.418 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:52:39.466 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:55:39.533 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 05:58:39.581 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:01:39.625 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:04:39.681 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:07:39.714 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:10:39.786 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:13:39.826 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:16:39.881 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:19:39.925 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:22:39.955 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:25:40.016 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:28:40.059 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:31:40.124 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:34:40.171 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:37:40.226 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:40:40.288 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:43:40.334 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:46:40.381 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:49:40.426 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:52:40.471 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:55:40.520 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 06:58:40.553 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:01:40.628 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:04:40.697 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:07:40.754 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:10:40.772 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:13:40.821 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:16:40.854 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:19:40.908 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:22:40.948 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:25:41.012 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:28:41.062 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:31:41.089 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:34:41.144 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:37:41.212 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:40:41.274 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:43:41.326 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:46:41.402 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:49:41.445 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:50:46.080 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:52:41.500 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:53:46.584 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.05 07:55:41.553 *I* [auth              ] New ephemeral authentication token issued for user admin [1]

Filipp Sudanov

Ok, just in case, set debug level for these tags:

debug ncd.* 7
debug ncd 7
debug nc.* 7
debug nc 7

they issue

debug

command and check that debug levels were actually set.

redlam

Ok adding those helped. The log now shows this:


2024.09.09 08:49:19.368 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.09 08:49:45.416 *I* [logger            ] Debug level for tag "ncd.*" set to 7
2024.09.09 08:49:52.473 *I* [logger            ] Debug level for tag "ncd" set to 7
2024.09.09 08:49:57.138 *I* [logger            ] Debug level for tag "nc.*" set to 7
2024.09.09 08:50:02.590 *I* [logger            ] Debug level for tag "nc" set to 7
2024.09.09 08:50:53.104 *D* [nc                ] SendNotification: sending message to "" via channel "Slack"
2024.09.09 08:50:53.104 *D* [nc                ] Driver for channel Slack requires recipient, but message has no recipient (message dropped)
2024.09.09 08:51:26.293 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.09 08:52:19.404 *I* [auth              ] New ephemeral authentication token issued for user admin [1]
2024.09.09 08:54:04.207 *D* [nc                ] SendNotification: sending message to "" via channel "Slack"
2024.09.09 08:54:04.207 *D* [nc                ] Driver for channel Slack requires recipient, but message has no recipient (message dropped)
2024.09.09 08:54:09.219 *D* [nc                ] SendNotification: sending message to "" via channel "Slack"
2024.09.09 08:54:09.219 *D* [nc                ] Driver for channel Slack requires recipient, but message has no recipient (message dropped)
2024.09.09 08:54:26.392 *I* [auth              ] New ephemeral authentication token issued for user admin [1]


Added "admin" for the recipient and I am finally getting notifications in slack!!! 

Thank you to all for your help!