NXShell in Docker, host using different tunnels/MTU -Dnetxms.encryptSession ??

Started by broski, August 29, 2016, 04:42:24 PM

Previous topic - Next topic

broski

Hello,

I am able to reproduce an issue with my Docker NetXMS scripts. The method of reproducing is simply done by keeping the encrption flag on when establishing to NetXMS daemon. The main issue is that it fails to connect 95% of the time.

I -do not- always get this response:
java.net.SocketTimeoutException: connect timed out

It is almost as though the connection is -stuck- or the state of the encryption is not completed (fragmented or MTU issue?)

When encryption is turned off, it works 100%

Question:
Is there a mandatory/recommended MTU setting for incoming NetXMS / NXShell clients?


At any given time, on any given mobile-network, the MTU may be 1400, 1304 or 1500. It also may be a combination of the three due to multiple tunnels.
I have tried MSS Advert on the interface and a few other things, but.. It's difficult to repeat on -all- systems because their MTU is dynamic etc.. hard to diagnose. I have a combo of cjdns, ipsec and tls tunnels at any given time. The common denominator is it always works when nxshell -Dnetxms.encryptSession=false .. temporary fix is to keep the nxshell command in loop until it is successful.

NetXMS is amazingly fast, efficient and flexible. Thank you for the hard work, blood, sweat, tears etc..

-broski

Alex Kirhenshtein

can you please provide result of tcpdump running inside container (during failed attempt with encryption)?

broski

## From within container
## Encryption: OFF


15:15:33.659869 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [S], seq 627089738, win 29200, options [mss 1460,sackOK,TS val 412269349 ecr 0,nop,wscale 7], length 0
E..<.o@[email protected]=....]%`.J......r..C.........
...%........
15:15:33.671474 IP 99.99.99.99.4701 > 172.17.0.2.58016: Flags [S.], seq 330609110, ack 627089739, win 28960, options [mss 1460,sackOK,TS val 904440320 ecr 412269349,nop,wscale 7], length 0
E..<[email protected]=......]......%`.K..q ...........
5......%....
15:15:33.671514 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [.], ack 1, win 229, options [nop,nop,TS val 412269352 ecr 904440320], length 0
E..4.p@.@..?.....U=....]%`.K.........;.....
...(5...
15:15:33.695110 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [P.], seq 1:17, ack 1, win 229, options [nop,nop,TS val 412269357 ecr 904440320], length 16
E..D.q@[email protected]=....]%`.K.........K.....
...-5....g..............
15:15:33.706422 IP 99.99.99.99.4701 > 172.17.0.2.58016: Flags [.], ack 17, win 227, options [nop,nop,TS val 904440328 ecr 412269357], length 0
[email protected]=......]......%`.[...........
5......-
15:15:33.706756 IP 99.99.99.99.4701 > 172.17.0.2.58016: Flags [P.], seq 1:625, ack 17, win 227, options [nop,nop,TS val 904440328 ecr 412269357], length 624
[email protected]=......]......%`.[...........
5......-.......p...........................y.......
.2...0...5..........+...H1.....z......................./.................../........................................%.7.].LX.l.k...y.QX.02...te.j8.,.....U...;[email protected]..:.!.~.v..f'*..W........H`].mVe..H.a...3(a............E".....Ul...C.}z.w+............k5F.Y..jOHiU.hZ..q.J..3....3N...8.......f>.A..$.i.8X..j....K..s8u`...L.-.|.....o4Tu..N[[email protected].....*I.CI........^....W..........4.........E.S.T.-.0.4.E.D.T.............<.h.t.t.p.:././.t.i.l.e...o.p.e.n.s.t.r.e.e.t.m.a.p...o.r.g./.............d.d...M.M...y.y.y.y.............H.H.:.m.m.:.s.s...............
.H.H.:.m.m..
15:15:33.706854 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [.], ack 625, win 238, options [nop,nop,TS val 412269360 ecr 904440328], length 0
E..4.r@.@..=.....U=....]%`.[...G.....;.....
...05...
15:15:33.713156 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [P.], seq 17:257, ack 625, win 238, options [nop,nop,TS val 412269362 ecr 904440328], length 240
E..$.s@[email protected]=....]%`.[...G.....+.....
...25..............................,.L.i.n.u.x. .4...4...0.-.3.4.-.g.e.n.e.r.i.c.............u.s.e.r.-.f.o.o...............
.2...0...4...............p.a.s.s.-.f.o.o.................................n.x.j.c.l.i.e.n.t./.2...0...4...................e.n
15:15:33.724672 IP 99.99.99.99.4701 > 172.17.0.2.58016: Flags [P.], seq 625:657, ack 257, win 235, options [nop,nop,TS val 904440333 ecr 412269362], length 32
[email protected]=......].....G%`.K...........
5......2....... ........................
15:15:33.763461 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [.], ack 657, win 238, options [nop,nop,TS val 412269375 ecr 904440333], length 0
E..4.t@.@..;.....U=....]%`.K...g.....;.....
...?5..
15:15:34.054997 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [F.], seq 257, ack 657, win 238, options [nop,nop,TS val 412269447 ecr 904440333], length 0
E..4.u@.@..:.....U=....]%`.K...g.....;.....
....5..
15:15:34.066583 IP 99.99.99.99.4701 > 172.17.0.2.58016: Flags [F.], seq 657, ack 258, win 235, options [nop,nop,TS val 904440418 ecr 412269447], length 0
[email protected]=......].....g%`.L...........
5..b....
15:15:34.066617 IP 172.17.0.2.58016 > 99.99.99.99.4701: Flags [.], ack 658, win 238, options [nop,nop,TS val 412269450 ecr 904440418], length 0
E..4.v@[email protected]=....]%`.L...h.....;.....
....5..b


## From within container
## Encryption: ON


15:12:05.849190 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [S], seq 2077716537, win 29200, options [mss 1460,sackOK,TS val 412217396 ecr 0,nop,wscale 7], length 0
E..<..@.@.$......U=....]{.p9......r..C.........
...4........
15:12:05.859866 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [S.], seq 3178845353, ack 2077716538, win 28960, options [mss 1460,sackOK,TS val 904388367 ecr 412217396,nop,wscale 7], length 0
E..<[email protected]=......]...yT.{.p:..q /..........
5......4....
15:12:05.859908 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [.], ack 1, win 229, options [nop,nop,TS val 412217399 ecr 904388367], length 0
E..4..@.@.$......U=....]{.p:.yT......;.....
...75...
15:12:05.884530 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [P.], seq 1:17, ack 1, win 229, options [nop,nop,TS val 412217405 ecr 904388367], length 16
E..D..@.@.$......U=....]{.p:.yT......K.....
...=5....g..............
15:12:05.894934 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [.], ack 17, win 227, options [nop,nop,TS val 904388376 ecr 412217405], length 0
[email protected].'..U=......]...yT.{.pJ...........
5......=
15:12:05.895221 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [P.], seq 1:625, ack 17, win 227, options [nop,nop,TS val 904388376 ecr 412217405], length 624
[email protected].%..U=......]...yT.{.pJ....)......
5......=.......p...........................y.......
.2...0...5..........+...H1.....z......................./.................../...................................... _,.).._06.*[email protected],..?...*..+.=...;.xp...5.=.......}...../....+...!.A_./.D...,.&.g.D"c......I).T.....G..._n..e.....1.nW....dI..h.1..0...aOV.;.\.$5...3
.....'yp.e...Zv..'...".y~...8.&..,D=.Y.......#DpiU....CL...m...K...............^....W..........4.........E.S.T.-.0.4.E.D.T.............<.h.t.t.p.:././.t.i.l.e...o.p.e.n.s.t.r.e.e.t.m.a.p...o.r.g./.............d.d...M.M...y.y.y.y.............H.H.:.m.m.:.s.s...............
.H.H.:.m.m..
15:12:05.895349 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [.], ack 625, win 238, options [nop,nop,TS val 412217407 ecr 904388376], length 0
E..4..@.@.$......U=....]{.pJ.yW......;.....
...?5...
15:12:05.900621 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [P.], seq 17:41, ack 625, win 238, options [nop,nop,TS val 412217409 ecr 904388376], length 24
E..L..@.@.$......U=....]{.pJ.yW......S.....
...A5......................?....
15:12:05.911313 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [P.], seq 625:961, ack 41, win 227, options [nop,nop,TS val 904388380 ecr 412217409], length 336
[email protected].&..U=......]...yW.{.pb...........
5......A.......P...........z.......;...............$0.. 0.. *.H.............0.........?.W..TA2..y..AZ..N]..9L..%.<..'dry........4t..?wH..W.'<?....\...H.*...$".[f..)4E...)z..b*.....H...0..e.N.|N....f..T......D4.. .<.e.z0.........r~......1qo.7./.RU.....!..H...2h.F.0..e...........bD/.6......jL.|.QX.4...vS...Xe....z...!......|.s.Fh.`,..[+V.......
15:12:05.947412 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [.], ack 961, win 248, options [nop,nop,TS val 412217421 ecr 904388380], length 0
E..4..@.@.$......U=....]{.pb.yXj.....;.....
...M5...
15:12:35.901649 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [F.], seq 41, ack 961, win 248, options [nop,nop,TS val 412224909 ecr 904388380], length 0
E..4..@.@.$......U=....]{.pb.yXj.....;.....
....5...
15:12:35.911606 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [P.], seq 961:993, ack 41, win 227, options [nop,nop,TS val 904395880 ecr 412217421], length 32
[email protected].'..U=......]...yXj{.pb...........
5..h...M....... ........................
15:12:35.911660 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [R], seq 2077716578, win 0, length 0
E..(oW@[email protected]=....]{.pb....P...8...
15:12:35.912270 IP 99.99.99.99.4701 > 172.17.0.2.58002: Flags [F.], seq 993, ack 42, win 227, options [nop,nop,TS val 904395880 ecr 412224909], length 0
[email protected].'..U=......]...yX.{.pc.....&.....
5..h...
15:12:35.912297 IP 172.17.0.2.58002 > 99.99.99.99.4701: Flags [R], seq 2077716579, win 0, length 0
E..(oX@[email protected]=....]{.pc....P...8...





broski

Exactly -- Right?

How does the session-encryption=true change the authentication process? I can start looking at the code, but would like to ask here first for obvious reasons. I ask because the only think I can think of is a stateless vs. stateful issue I am having in the network setup with arbitrary tunnels/interfaces.

Thanks in advance.