Mina Naguib

The little ssh that (sometimes) couldn't

Mon, 22 Oct 2012 00:00:00 -0400

Preface

This is a technical article chronicling one of the most interesting bug hunts I’ve had the pleasure of chasing down.

At AdGear Technologies Inc. where I work, ssh is king. We use it for management, monitoring, deployments, log file harvesting, even real-time event streaming. It’s solid, reliable, has all the predictability of a native unix tool, and just works.

Until one day, random cron emails started flowing about it not working.

The timeout

The machines in our London data center were randomly failing to send their log files to our data machines in our Montreal data center. This job is initiated periodically from cron, and the failure manifested itself as:

cron emails stating that the ssh was unsuccessful
- Sometimes hangs
- Sometimes exits with a timeout error
nagios warnings down the line for in-house sanity checks detecting the missing data in Montreal

We logged into the London machines, manually ran the push command, and it worked successfully. We brushed it off as temporary network partitions.

The timeouts

But the failures kept popping up randomly. Once a day, a couple of times a day, then one Friday morning, several times an hour. It was clear something’s getting worse. We kept up with manually pushing the files until we figure out what the problem was.

There were 17 hops between London and Montreal. We built a profile of latency and packet loss for them, and found that a couple were losing 1-3% of packets. We filed a ticket with our London DC ops to route away from them.

While London DC ops were verifying the packet loss, we started seeing random timeouts from London to our SECOND data center in Montreal, and hops to that data center did not share the same routes we observed the packet loss at. We concluded packet loss is not the main problem around the same time London DC ops replied saying they’re not able to replicate the packet loss or timeouts and that everything looked healthy on their end.

The revelation

While manually keeping up with failed cron uploads, we noticed an interesting pattern. A file transfer either succeeded at a high speed, or didn’t succeed at all and hung/timed out. There were no instances of a file uploading slowly and finishing successfully.

Removing the large volume of data from the equation, we were able to recreate the scenario via simple vanilla ssh. On a London machine an “ssh mtl-machine” would either work immediately, or hang and never establish a connection. Eyebrows started going up.

Where the wild packets are

We triple-checked the ssh server configs and health in Montreal:

DNS servers were responding fast
DNS reverse lookup was not enabled
Maximum client connections was high enough
We were not under attack
Bandwidth usage was nowhere near saturation

Besides, even if something was off, we were observing the hangs talking to 2 completely distinct data centers in Montreal. Furthermore, our other data centers (non-London) were talking happily to Montreal. Something about London was off.

We fired up tcpdump and started looking at the packets, both in summary and in captured pcaps loaded into wireshark. We saw telltale signs of packet loss and retransmission, but it was minimal and not particularly worrisome.

We then captured full connections from cases where ssh established successfully, and full connections from cases where the ssh connection hung.

Here’s what we logically saw when a connection from London to Montreal hung:

Normal TCP handshake
Bunch of ssh-specific back and forth, with normal TCP ack packets where they should be
A particular packet sent from London and received in Montreal
The same packet re-sent (and re-sent, several times) from London and received in Montreal
Montreal’s just not responding to it!

It didn’t make sense why Montreal was not responding (hence London re-transmitting it). The connection was stalled at this point, as the layer 4 protocol was at a stalemate. More infuriatingly, if you kill the ssh attempt in London and re-launched it immediately, odds are it worked successfully. When it did, tcpdump showed Montreal receiving the packet but responding to it, and things moved on.

We enabled verbose debugging (-vvv) on the ssh client in London, and the hang occurred after it logged:

debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

Googling “ssh hang SSH2_MSG_KEX_DH_GEX_GROUP” has many results – from bad WiFi, to windows TCP bugs, to buggy routers discarding TCP fragments. One solution for LANs was to figure out the path’s MSS and set that as the MTU on both ends.

I kept decrementing the MTU on a London server down from 1500 – it didn’t help until I hit the magic value 576. At that point, I was no longer able to get the ssh hanging behavior replicated. I had an ssh loop script running, and it was on-demand that I could cause timeouts by bringing the MTU back up to 1500, or make them disappear by setting it to 576.

Unfortunately these are public ad servers and globally setting the MTU to 576 won’t cut it, but the above did suggest that perhaps packet fragmentation or reassembly is broken somewhere.

Going back to check the received packets with tcpdump, there was no evidence of fragmentation. The received packet size matched exactly the packet size sent. If something did fragment the packet at byte 576+, something else reassembled it successfully.

Twinkle twinkle little mis-shapen star

Digging in some more, I was now looking at full packet dumps (tcpdump -s 0 -X) instead of just the headers. Comparing that magic packet in instances of ssh success vs ssh hang showed very little difference aside from TCP/IP header variations. It was however clear that this is the first packet in the TCP connection that had enough data to bypass the 576-byte mark – all previous packets were much smaller.

Comparing the same packet, during a hanging instance, as it left London, and as captured in Montreal, something caught my eye. Something very subtle, and I brushed it off as fatigue (it was late Friday at this point), but sure enough after a few refreshes and comparisons, I wasn’t imagining things.

Here’s the packet as it left London (minus the first few bytes identifying the IP addresses):

0x0040:  0b7c aecc 1774 b770 ad92 0000 00b7 6563  .|...t.p......ec
0x0050:  6468 2d73 6861 322d 6e69 7374 7032 3536  dh-sha2-nistp256
0x0060:  2c65 6364 682d 7368 6132 2d6e 6973 7470  ,ecdh-sha2-nistp
0x0070:  3338 342c 6563 6468 2d73 6861 322d 6e69  384,ecdh-sha2-ni
0x0080:  7374 7035 3231 2c64 6966 6669 652d 6865  stp521,diffie-he
0x0090:  6c6c 6d61 6e2d 6772 6f75 702d 6578 6368  llman-group-exch
0x00a0:  616e 6765 2d73 6861 3235 362c 6469 6666  ange-sha256,diff
0x00b0:  6965 2d68 656c 6c6d 616e 2d67 726f 7570  ie-hellman-group
0x00c0:  2d65 7863 6861 6e67 652d 7368 6131 2c64  -exchange-sha1,d
0x00d0:  6966 6669 652d 6865 6c6c 6d61 6e2d 6772  iffie-hellman-gr
0x00e0:  6f75 7031 342d 7368 6131 2c64 6966 6669  oup14-sha1,diffi
0x00f0:  652d 6865 6c6c 6d61 6e2d 6772 6f75 7031  e-hellman-group1
0x0100:  2d73 6861 3100 0000 2373 7368 2d72 7361  -sha1...#ssh-rsa
0x0110:  2c73 7368 2d64 7373 2c65 6364 7361 2d73  ,ssh-dss,ecdsa-s
0x0120:  6861 322d 6e69 7374 7032 3536 0000 009d  ha2-nistp256....
0x0130:  6165 7331 3238 2d63 7472 2c61 6573 3139  aes128-ctr,aes19
0x0140:  322d 6374 722c 6165 7332 3536 2d63 7472  2-ctr,aes256-ctr
0x0150:  2c61 7263 666f 7572 3235 362c 6172 6366  ,arcfour256,arcf
0x0160:  6f75 7231 3238 2c61 6573 3132 382d 6362  our128,aes128-cb
0x0170:  632c 3364 6573 2d63 6263 2c62 6c6f 7766  c,3des-cbc,blowf
0x0180:  6973 682d 6362 632c 6361 7374 3132 382d  ish-cbc,cast128-
0x0190:  6362 632c 6165 7331 3932 2d63 6263 2c61  cbc,aes192-cbc,a
0x01a0:  6573 3235 362d 6362 632c 6172 6366 6f75  es256-cbc,arcfou
0x01b0:  722c 7269 6a6e 6461 656c 2d63 6263 406c  r,rijndael-cbc@l
0x01c0:  7973 6174 6f72 2e6c 6975 2e73 6500 0000  ysator.liu.se...
0x01d0:  9d61 6573 3132 382d 6374 722c 6165 7331  .aes128-ctr,aes1
0x01e0:  3932 2d63 7472 2c61 6573 3235 362d 6374  92-ctr,aes256-ct
0x01f0:  722c 6172 6366 6f75 7232 3536 2c61 7263  r,arcfour256,arc
0x0200:  666f 7572 3132 382c 6165 7331 3238 2d63  four128,aes128-c
0x0210:  6263 2c33 6465 732d 6362 632c 626c 6f77  bc,3des-cbc,blow
0x0220:  6669 7368 2d63 6263 2c63 6173 7431 3238  fish-cbc,cast128
0x0230:  2d63 6263 2c61 6573 3139 322d 6362 632c  -cbc,aes192-cbc,
0x0240:  6165 7332 3536 2d63 6263 2c61 7263 666f  aes256-cbc,arcfo
0x0250:  7572 2c72 696a 6e64 6165 6c2d 6362 6340  ur,rijndael-cbc@
0x0260:  6c79 7361 746f 722e 6c69 752e 7365 0000  lysator.liu.se..
0x0270:  00a7 686d 6163 2d6d 6435 2c68 6d61 632d  ..hmac-md5,hmac-
0x0280:  7368 6131 2c75 6d61 632d 3634 406f 7065  sha1,umac-64@ope
0x0290:  6e73 7368 2e63 6f6d 2c68 6d61 632d 7368  nssh.com,hmac-sh
0x02a0:  6132 2d32 3536 2c68 6d61 632d 7368 6132  a2-256,hmac-sha2
0x02b0:  2d32 3536 2d39 362c 686d 6163 2d73 6861  -256-96,hmac-sha
0x02c0:  322d 3531 322c 686d 6163 2d73 6861 322d  2-512,hmac-sha2-
0x02d0:  3531 322d 3936 2c68 6d61 632d 7269 7065  512-96,hmac-ripe
0x02e0:  6d64 3136 302c 686d 6163 2d72 6970 656d  md160,hmac-ripem
0x02f0:  6431 3630 406f 7065 6e73 7368 2e63 6f6d  d160@openssh.com
0x0300:  2c68 6d61 632d 7368 6131 2d39 362c 686d  ,hmac-sha1-96,hm
0x0310:  6163 2d6d 6435 2d39 3600 0000 a768 6d61  ac-md5-96....hma
0x0320:  632d 6d64 352c 686d 6163 2d73 6861 312c  c-md5,hmac-sha1,
0x0330:  756d 6163 2d36 3440 6f70 656e 7373 682e  umac-64@openssh.
0x0340:  636f 6d2c 686d 6163 2d73 6861 322d 3235  com,hmac-sha2-25
0x0350:  362c 686d 6163 2d73 6861 322d 3235 362d  6,hmac-sha2-256-
0x0360:  3936 2c68 6d61 632d 7368 6132 2d35 3132  96,hmac-sha2-512
0x0370:  2c68 6d61 632d 7368 6132 2d35 3132 2d39  ,hmac-sha2-512-9
0x0380:  362c 686d 6163 2d72 6970 656d 6431 3630  6,hmac-ripemd160
0x0390:  2c68 6d61 632d 7269 7065 6d64 3136 3040  ,hmac-ripemd160@
0x03a0:  6f70 656e 7373 682e 636f 6d2c 686d 6163  openssh.com,hmac
0x03b0:  2d73 6861 312d 3936 2c68 6d61 632d 6d64  -sha1-96,hmac-md
0x03c0:  352d 3936 0000 0015 6e6f 6e65 2c7a 6c69  5-96....none,zli
0x03d0:  6240 6f70 656e 7373 682e 636f 6d00 0000  b@openssh.com...
0x03e0:  156e 6f6e 652c 7a6c 6962 406f 7065 6e73  .none,zlib@opens
0x03f0:  7368 2e63 6f6d 0000 0000 0000 0000 0000  sh.com..........
0x0400:  0000 0000 0000 0000 0000 0000            ............

And here’s the same packet as it arrived in Montreal:

0x0040:  0b7c aecc 1774 b770 ad92 0000 00b7 6563  .|...t.p......ec
0x0050:  6468 2d73 6861 322d 6e69 7374 7032 3536  dh-sha2-nistp256
0x0060:  2c65 6364 682d 7368 6132 2d6e 6973 7470  ,ecdh-sha2-nistp
0x0070:  3338 342c 6563 6468 2d73 6861 322d 6e69  384,ecdh-sha2-ni
0x0080:  7374 7035 3231 2c64 6966 6669 652d 6865  stp521,diffie-he
0x0090:  6c6c 6d61 6e2d 6772 6f75 702d 6578 6368  llman-group-exch
0x00a0:  616e 6765 2d73 6861 3235 362c 6469 6666  ange-sha256,diff
0x00b0:  6965 2d68 656c 6c6d 616e 2d67 726f 7570  ie-hellman-group
0x00c0:  2d65 7863 6861 6e67 652d 7368 6131 2c64  -exchange-sha1,d
0x00d0:  6966 6669 652d 6865 6c6c 6d61 6e2d 6772  iffie-hellman-gr
0x00e0:  6f75 7031 342d 7368 6131 2c64 6966 6669  oup14-sha1,diffi
0x00f0:  652d 6865 6c6c 6d61 6e2d 6772 6f75 7031  e-hellman-group1
0x0100:  2d73 6861 3100 0000 2373 7368 2d72 7361  -sha1...#ssh-rsa
0x0110:  2c73 7368 2d64 7373 2c65 6364 7361 2d73  ,ssh-dss,ecdsa-s
0x0120:  6861 322d 6e69 7374 7032 3536 0000 009d  ha2-nistp256....
0x0130:  6165 7331 3238 2d63 7472 2c61 6573 3139  aes128-ctr,aes19
0x0140:  322d 6374 722c 6165 7332 3536 2d63 7472  2-ctr,aes256-ctr
0x0150:  2c61 7263 666f 7572 3235 362c 6172 6366  ,arcfour256,arcf
0x0160:  6f75 7231 3238 2c61 6573 3132 382d 6362  our128,aes128-cb
0x0170:  632c 3364 6573 2d63 6263 2c62 6c6f 7766  c,3des-cbc,blowf
0x0180:  6973 682d 6362 632c 6361 7374 3132 382d  ish-cbc,cast128-
0x0190:  6362 632c 6165 7331 3932 2d63 6263 2c61  cbc,aes192-cbc,a
0x01a0:  6573 3235 362d 6362 632c 6172 6366 6f75  es256-cbc,arcfou
0x01b0:  722c 7269 6a6e 6461 656c 2d63 6263 406c  r,rijndael-cbc@l
0x01c0:  7973 6174 6f72 2e6c 6975 2e73 6500 0000  ysator.liu.se...
0x01d0:  9d61 6573 3132 382d 6374 722c 6165 7331  .aes128-ctr,aes1
0x01e0:  3932 2d63 7472 2c61 6573 3235 362d 6374  92-ctr,aes256-ct
0x01f0:  722c 6172 6366 6f75 7232 3536 2c61 7263  r,arcfour256,arc
0x0200:  666f 7572 3132 382c 6165 7331 3238 2d63  four128,aes128-c
0x0210:  6263 2c33 6465 732d 6362 632c 626c 6f77  bc,3des-cbc,blow
0x0220:  6669 7368 2d63 6263 2c63 6173 7431 3238  fish-cbc,cast128
0x0230:  2d63 6263 2c61 6573 3139 322d 6362 632c  -cbc,aes192-cbc,
0x0240:  6165 7332 3536 2d63 6263 2c61 7263 666f  aes256-cbc,arcfo
0x0250:  7572 2c72 696a 6e64 6165 6c2d 6362 7340  ur,rijndael-cbs@
0x0260:  6c79 7361 746f 722e 6c69 752e 7365 1000  lysator.liu.se..
0x0270:  00a7 686d 6163 2d6d 6435 2c68 6d61 732d  ..hmac-md5,hmas-
0x0280:  7368 6131 2c75 6d61 632d 3634 406f 7065  sha1,umac-64@ope
0x0290:  6e73 7368 2e63 6f6d 2c68 6d61 632d 7368  nssh.com,hmac-sh
0x02a0:  6132 2d32 3536 2c68 6d61 632d 7368 7132  a2-256,hmac-shq2
0x02b0:  2d32 3536 2d39 362c 686d 6163 2d73 7861  -256-96,hmac-sxa
0x02c0:  322d 3531 322c 686d 6163 2d73 6861 322d  2-512,hmac-sha2-
0x02d0:  3531 322d 3936 2c68 6d61 632d 7269 7065  512-96,hmac-ripe
0x02e0:  6d64 3136 302c 686d 6163 2d72 6970 756d  md160,hmac-ripum
0x02f0:  6431 3630 406f 7065 6e73 7368 2e63 7f6d  d160@openssh.c.m
0x0300:  2c68 6d61 632d 7368 6131 2d39 362c 786d  ,hmac-sha1-96,xm
0x0310:  6163 2d6d 6435 2d39 3600 0000 a768 7d61  ac-md5-96....h}a
0x0320:  632d 6d64 352c 686d 6163 2d73 6861 312c  c-md5,hmac-sha1,
0x0330:  756d 6163 2d36 3440 6f70 656e 7373 782e  umac-64@openssx.
0x0340:  636f 6d2c 686d 6163 2d73 6861 322d 3235  com,hmac-sha2-25
0x0350:  362c 686d 6163 2d73 6861 322d 3235 362d  6,hmac-sha2-256-
0x0360:  3936 2c68 6d61 632d 7368 6132 2d35 3132  96,hmac-sha2-512
0x0370:  2c68 6d61 632d 7368 6132 2d35 3132 3d39  ,hmac-sha2-512=9
0x0380:  362c 686d 6163 2d72 6970 656d 6431 3630  6,hmac-ripemd160
0x0390:  2c68 6d61 632d 7269 7065 6d64 3136 3040  ,hmac-ripemd160@
0x03a0:  6f70 656e 7373 682e 636f 6d2c 686d 7163  openssh.com,hmqc
0x03b0:  2d73 6861 312d 3936 2c68 6d61 632d 7d64  -sha1-96,hmac-}d
0x03c0:  352d 3936 0000 0015 6e6f 6e65 2c7a 7c69  5-96....none,z|i
0x03d0:  6240 6f70 656e 7373 682e 636f 6d00 0000  b@openssh.com...
0x03e0:  156e 6f6e 652c 7a6c 6962 406f 7065 6e73  .none,zlib@opens
0x03f0:  7368 2e63 6f6d 0000 0000 0000 0000 0000  sh.com..........
0x0400:  0000 0000 0000 0000 0000 0000            ............

Did something there catch your eye ? If not, I don’t blame you. Feel free to copy each into a text editor and rapidly switch back-and-forth to see some characters dance. Here’s what it looks like when they’re placed in vimdiff:

Well well well. It’s not packet loss, it’s packet corruption! Very subtle, very predictable packet corruption.

Some interesting notes:

The lower part of the packet (<576 bytes) is unaffected
The affected portion is predictably corrupted on the 15th byte of every 16
The corruption is predictable. All instances of “h” become “x”, all instances of “c” become “s”

Some readers might have already checked ASCII charts and reached the conclusion: There’s a single bit statically stuck at “1” somewhere. Flipping the 4th bit in a byte to 1 would reliably corrupt the above letters on the left side to the value on the right side.

The obvious culprits within our control (NIC cards, receiving machines) are not suspect due to the pattern of failure observed (several London machines → Several Montreal data centers and machines). It’s got to be something upstream and close to London.

Going back to validate, things started to make sense. I also noticed a little hint in tcpdump verbose mode (tcp cksum bad) which was missed before. A Montreal machine receiving this packet discarded it at the kernel level after realizing it’s corrupt, never passing it to the userland ssh daemon. London then re-transmitted it, going through the same corruption, getting the same silent treatment. From ssh and sshd’s perspective, the connection was at a stalemate. From tcpdump’s perspective, there was no loss, and Montreal machines appeared to be just ignoring data.

We sent these findings to our London DC ops, and within a few minutes they changed outbound routes dramatically. The first router hop, and most hops afterwards, were different. The hanging problem disappeared.

Late Friday night fixes are nice because you can relax and not carry problems and support staff into the weekend :)

Where’s Waldo

Happy that we were no longer suffering from this problem and that our systems are caught up with the backlog, I decided I’d try my hand at actually finding the device causing the corruption.

Having the London routes updated to not go through the old path meant that I couldn’t reproduce the problem easily. I asked around until I found a friend with a FreeBSD box in Montreal I could use, which was still accessed through the old routes from London.

Next, I wanted to make sure that the corruption is predictable even without ssh involvement. This was trivially proven with a few pipes.

In Montreal:

nc -l -p 4000 > /dev/null

Then in London:

cat /dev/zero | nc mtl 4000

Again, accounting for the randomness factor and settings things up in a retry loop, I got a few packets which remove any doubt about the previous conclusions. Here’s part of one – remember that we’re sending just a stream of nulls:

0x0210  .....
0x0220  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0230  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0240  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0250  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0260  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0270  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0280  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0290  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02a0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02b0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02c0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02d0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02e0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x02f0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0300  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0310  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0320  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0330  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0340  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0350  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0360  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0370  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0380  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x0390  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03a0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03b0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03c0  0000 0000 0000 0000 0000 0000 0000 1000 ................
0x03d0  0000 0000 0000 0000 0000 0000 0000 0000 ................
0x03e0  .....

With the bug replicated, I needed to find a way to isolate which of the 17 hops along that path cause the corruption. There was simply no way to call up the provider of each cluster to ask them to check their systems.

I decided pinging each router, incrementally, might be the way to go. I crafted special ICMP packets that are large enough to go over the 576 safety margin, and filled entirely with NULLs. Then pinged the Montreal machine with them from London.

They came back perfectly normal. There was no corruption.

I tried all variations of speed, padding, size – to no avail. I simply could not observe corruption in the returned ICMP ping packets.

I replaced the netcat pipes with UDP instead of TCP. Again there was no corruption.

The corruption needed TCP to be reproducible – and TCP needs 2 cooperating endpoints. I tried in vain to see if all the routers had an open TCP port I can talk to directly, to no avail.

It seemed there was no easy way an external party can pinpoint the bad apple. Or was there ?

Mirror mirror on the wall

To detect whether corruption occurred or not, we need one of these scenarios:

Control over the TCP peer we’re talking to inspect the packet at the destination
- Not just in userland, where the packet would not get delivered if the TCP checksum failed, but root + tcpdump to inspect it as it arrives
A TCP peer that acts as an echo server to mirror back the data it received, so we get inspect it at the sending node and detect corruption there

It suddenly occurred to me that the second data point is available to us. Not per-se, but consider this: In our very first taste of the problem, we observed ssh clients hanging when talking to ssh servers over the corrupting hop. This is a good passive signal that we can use instead of the active “echo” signal.

… and there are lots of open ssh servers out there on the internet to help us out.

We don’t need actual accounts on these servers – we just need to kickstart the ssh connection and see if the cipher exchange phase succeeds or hangs (with a reasonable number of retries to account for corruption randomness).

So this plan was hatched:

Use the wonderful nmap tool – specifically – its “random IP” mode – to make a list of geographically distributed open ssh servers
Test each server to determine whether it is:
- Unresponsive/unpredictable/firewalled → Ignore it
- Negotiates successfully after being retried N times → mark as “good”
- Negotiates with hangs at the telltale phase after being retried N times → mark as “bad”
For both “good” and “bad” servers, remember the traceroute to them

The idea was this: All servers marked as “bad” will share a few hops in their traceroute. We can then take that set of suspect hops, and subtract from it any that appear in the traceroutes of the “good” servers. Hopefully what’s left is only one or two.

After spending an hour manually doing the above exercise, I stopped to inspect the data. I had classified 16 servers as “BAD” and 25 servers as “GOOD”.

The first exercise was to find the list of hops that appear in all the traceroutes of the “BAD” servers. As I cleaned and trimmed the list, I realized I won’t even need to get to the “GOOD” list to remove false positives. Within the “BAD” lists alone, there remained only 1 that was common to all of them.

For what it’s worth, it was 2 providers away: London → N hops upstream1 → Y hops upstream2

It was the first in Y hops of upstream2 – right at the edge between upstream1 and upstream2, corrupting random TCP packets, causing many retries, and, depending on the protocol’s logical back-and-forth, hangs, or reduced transmission rates. You may have been a telephony provider who sufferred dropped calls, a retailer who lost a few customers or sales, the possibilities really are endless.

I followed up with our London DC ops with the single hop’s IP address. Hopefully with their direct relationship with upstream1 they can escalate through there and get it fixed.

/filed under crazy devops war stories

Update

Through upstream1, I got confirmation that the hop I pointed out (first in upstream2) had an internal “management module failure” which affected BGP and routing between two internal networks. It’s still down (they’ve routed around it) until they receive a replacement for the faulty module.

Thanks for the kind words and great comments here on Disqus, Reddit (/r/linux & /r/sysadmin) and hacker news

Food: Egyptian-style rice

Tue, 19 Jul 2011 00:00:00 -0400

About

I’ve been evolving this recipe, and it’s reached a pretty good point worth sharing. I recommend it highly as it’s fairly easy to make and the result is fantastic.

Ingredients

2 tsp cumin seeds
5 cardamom pods
4 cloves
1 1/2 cinnamon sticks
1 bay leaf

1 bunch angelhair pasta
1 tbsp butter
1 1/2 cups basmati rice
1/2 orange
4 cups water
1/4 cup golden raisins
Salt

2 tbsp sliced almonds
2 tbsp coriander leaf – chopped

Preparation

Dry-roast the cumin, cardamom, cloves & 1/2 cinnamon stick, then grind and set aside
In the rice pot, break the pasta to 1-inch pieces, dry-roast on medium heat till dark brown
Add the butter, then cinnamon stick, bay leaf, and ground spices, fry for a minute
Add the rice, stir for a couple of minutes till well coated
Squeeze in the 1/2 orange juice, then add the water. Stir, heat on high till boil, lower heat
Cook uncovered for 10 minutes, stirring 2-3 times, till the water is all gone.
At the 5-minute mark add the raisins, salt to-taste, and zest the 1/2 orange’s peel
In a separate pan, dry-roast the sliced almonds
When the rice is done, serve garnished with the roasted almonds and coriander

Enjoy.

Videotron latency and packet loss

Thu, 31 Mar 2011 00:00:00 -0400

Jump to updates: 1 2 3 4 5 6

How fast does your ISP respond to your complaints ?

Our ISP, Videotron, has been working on our problem for:

A month

Background:

Where I work , our internet connection is via Quebec’s leading telecommunications company, Videotron . We’re subscribed to their “TGV 50” business-class service.

Weeks ago, I started noticing intermittently bad performance, ranging from trouble with latency-sensitive applications (remote desktop, VOIP, ssh) to web browser timeouts.

Due to the problem’s intermittency I had trouble getting videotron on the phone while the problem is happening, so I whipped up a small tool to start graphing the ping latency and packet loss to the next immediate hop, the videotron gateway. It then became clear we’re suffering from high latency and packet loss, and that it is most prominent during business hours and slightly better at night.

With enough data, I called videotron and described the problem. After a couple of technician visits and more phone calls, videotron concluded that there’s an area-wide problem and it would be fixed soon. We got a ticket number and I was happy with that early response.

I suppose “soon” is a relative term. Many more phone calls later (I lost count), and I’ll let you judge whether you still see the problem or not:

At the time of this writing (noon March 31, 2011) the connection is so bad I’ve had to pause and continue writing this post several times:

For comparison’s sake, here’s what my personal videotron connection at home looks like:

And so…

It’s unfortunate that while the people working at videotron seem genuinely interested to help while on the phone, the end result is complete mediocrity when it comes to actually delivering on what was discussed (fixing the problem, calling back with status updates, escalating to supervisors, etc..)

And so, our business genuinely suffers due to these technical problems, and even though we have an open ticket number with Videotron, there seems to be a big mismatch between our expectations of reliable internet service and their willingness to deliver it.

How has your ISP been treating you ?

Updates:

Thursday, March 31 5pm

Videotron called. Work will be done tonight to fix one problem as well as April 6th to fix another, the sum of which should fix our problems.

Friday, April 1 noon

Videotron called. Confirmed work was done last night but performance is still poor. Will call back after next Wednesday’s scheduled work.

Tuesday, April 5 noon

Videotron called. Confirmed some faulty equipment was fixed yesterday, and indeed our experience with service appears to be now going well. Furthermore the work scheduled for tonight to split the cell is still going on so things thould improve even more. Considering issue closed.

Friday, April 8 noon

Things were going great (8ms average latency) since Monday. That’s 4 whole days of good service.

However, two videotron techs show up, unannounced, to perform some more testing and tuning.

Halfway through their work, they “tuned” our connection enough to un-do the Monday fix. We’re now back to square one with a barely usable internet connection. They were unable to restore it to low latency and left.

Would you like to play “guess when the videotron techs showed up and broke our connection” ? Step right up:

I have no words to describe the amount of frustration we’re experiencing dealing with this company.

Monday, April 18

Latency has been quite good since last Thursday (April 14th). Hoping it’ll remain stable and these problems are now past us.

Thursday, June 16

A few temporary hiccups here and there aside, I’m happy to say it seems videotron has put their act together. We’ve been having stable, fast service for a few weeks in a row. I’ve also received several courtesy calls to check on things and make sure we’re satisfied. Hoping all of this is now behind us.

Optional and name-based arguments in C

Thu, 09 Dec 2010 00:00:00 -0500

Introduction

It’s inevitable that some functions’ signatures become unwieldy. This is especially true as a project ages, and various developers who were not involved in the initial vision are asked to extend and tweak various existing functionality.

In a high-level language, it’s often possible to effect new changes in an existing function without modifying all its consumers.

In C however, that’s not the case (unless the function already happens to use va_args). Once a C function’s signature changes, all consumers must be modified. Things get even more complicated if the function is part of a public API in a library.

It occurred to me that there are existing features of C that allow us to pre-emptively design functions in a way that future-proofs them in anticipation of these types of changes. I’m not advocating that all functions be designed this way, however for the the class of functions that appear candidates for future signature expansion, my proposal may provide some ideas.

Existing stepping stones

The main features we’ll be putting together for this method are:

Calling by-value

In C all function parameters and returns are passed by-value, not by reference. That means the callee receives a copy of the parameters, and can not influence the caller’s copy.

Perhaps for the sake of performance, C developers generally assume the above rule leads to “structs must be passed as a pointer” to allow that copying to be performed, however that is not true. It’s perfectly valid in C to call a function passing it a struct (not a pointer to a struct). The function’s signature would look like this:

int say_hello(struct person p);

The caller would call this function like so:

...
  struct person bob;
...
  say_hello(bob);
...

say_hello would receive a COPY of that struct, which means its local p will be at a different address than bob. The inner members of p will be copied, individually, to the values of the inner members of bob

Initializing structs

Initializing a variable such as an int can be done like so:

int age = 20;

Similarly, initializing a struct can be done like so:

struct person bob = {"Bob", 20, "Accounting"};

The individual struct members are given, between brackets, in the same order as the struct declaration.

Missing members are allowed:

struct person bob = {"Bob", 20};

In the above case, any members not supplied will be initialized to 0 / NULL.

It’s perfectly legal to not supply any members, in which case all members will be set to 0 / NULL:

struct person bob = {};

Inlined initialization

say_hello which accepts a struct may be called using an inline-initialized struct like so:

say_hello({"Bob", 20, "Accounting"});

Some compilers may need the struct explicitly cast, like so:

say_hello((struct person){"Bob", 20, "Accounting"});

C99 Designated initializers

When initializing a struct, you have the option of referencing its individual members by name, instead of by position:

struct person bob = {
  .name       = "Bob",
  .age        = 20,
  .department = "Accounting"
};

Again, as with position-based initialization, members may be omitted to default them to 0 / NULL:

struct person bob = {
  .name       = "Bob"
};

Variadic macros

While macros need no introduction to any seasoned C programmers, they’re typically declared to accept a fixed set of arguments. Variadic macros allow for a dynamic set of arguments to the macro. We’ll be using them for some final syntactic sugar, but they’re not strictly necessary.

Putting it all together

We’ll assume we’re writing a find function that, given a set of criteria, searches an employee directory and returns matches.

The implementation of the search and returning the result aren’t important. What is however is designing a signature for that function that anticipates inevitable future changes to the supplied search criteria.

/* nba.h */

#ifndef _NBA_H
#define _NBA_H

struct _find_args {
        char * name;
        int min_age;
        int max_age;
};
#define FindArgs(...) ((struct _find_args){__VA_ARGS__})

int find(struct _find_args fa);
#define Find(...) (find(FindArgs(__VA_ARGS__)))

#endif

The implementation of find itself, for the purpose of demonstrating consuming the input, looks like so:

int find(struct _find_args fa) {

        printf("Looking for criteria:\n");

        if (fa.name) {
                printf("\tName: [%s]\n", fa.name);
        }
        if (fa.min_age != -1) {
                printf("\tMin age: [%d]\n", fa.min_age);
        }
        if (fa.max_age != -1) {
                printf("\tMax age: [%d]\n", fa.max_age);
        }

        return(1);
}

Finally, the consumer consumes it using the Find macro (notice the capitalization). Here’s a battery of consumption cases:

int main(int argc, char ** argv) {

        char *n;
        char *names[] = {"Albert", "Jane", NULL};
        int i;

        /* Static input */
        Find(
                .name = "Bob",
                .min_age = -1,
                .max_age = 20
        );

        /* Dynamic input */
        n = "Mary";
        Find(
                .name = n,
                .min_age = 8,
                .max_age = -1
        );

        /* Dynamic input in loop */
        for (i=0; (n = names[i]); i++) {
                Find(
                        .name = n,
                        .min_age = -1,
                        .max_age = -1
                );
        }

        return(1);

}

When run, the output is:

$ ./nba
Looking for criteria:
        Name: [Bob]
        Max age: [20]
Looking for criteria:
        Name: [Mary]
        Min age: [8]
Looking for criteria:
        Name: [Albert]
Looking for criteria:
        Name: [Jane]

End result and viability

The end result is demonstrated in the main function above.

The pros are:

Extremely clear naming for the arguments being given to the function
No need to supply arguments that are not needed
Ability to add new members to the struct without altering existing code that consumes that function

The cons are:

There’s probably more overhead to copying structs in general compared to copying the same number of elements declared as regular function parameters
Missing members defaulting to 0 / NULL may be logically ambiguous if 0 / NULL is a logically acceptable explicit value for a given member. That is why the above example explicitly sets min_age / max_age to -1 to indicate non-interest. Another approach would be to use a secondary member to indicate the desire to “use” the first member. For example, age coupled with use_age
Variadic macros may not be available everywhere. This method still works without them, at the cost of more verbosity for the function consumer.

So far, the above has just been ideas and experimentation on my part. It’s not something I’ve used in any actual projects I’m working on.

If you’re a C developer I’d love to hear your thoughts on this approach.

PostgreSQL transactions wrapping child+parent modifications, deadlocks, and ActiveRecord

Mon, 22 Nov 2010 00:00:00 -0500

From PostgreSQL’s perspective

To demonstrate the issue, let’s set up two tables and create a parent record with id 1:

  create table parents(
    id serial not null primary key,
    children_count integer not null default 0
  );

  create table children(
    id serial not null primary key,
    parent_id integer not null references parents(id)
  );

  insert into parents(id) values(1);

Now, let’s do a transaction that inserts a child under parent 1, and increment that parent’s children_count field:

  begin;
  insert into children(parent_id) values(1);
  update parents set children_count=children_count + 1 where id=1;
  commit;

Everything is fine and peachy. However, what happens if there are two processes doing the above concurrently, and the timing for each step happens as demonstrated below:


begin;
insert into children(parent_id) values(1);
begin;
insert into children(parent_id) values(1);
update parents set children_count=children_count + 1 where id=1;
update parents set children_count=children_count + 1 where id=1;
commit;
commit;

	Session 1	Session 2
1	`begin;`
2	`insert into children(parent_id) values(1);`
3		`begin;`
4		`insert into children(parent_id) values(1);`
5	`update parents set children_count=children_count + 1 where id=1;`
6		`update parents set children_count=children_count + 1 where id=1;`
7	`commit;`
8		`commit;`

Things go smoothly as expected until step 5, when we receive the first hint that something isn’t right. It’ll block.

Step 6 will also appear to block, for roughly 1 second, then session 1 will un-block while session 2 will receive:

  DETAIL:  Process 20906 waits for ExclusiveLock on tuple (0,1) of relation 26335948 of database 16385; blocked by process 20918.
  Process 20918 waits for ShareLock on transaction 168526; blocked by process 20906.
  HINT:  See server log for query details.

So what happened ?

Deadlocks generally occur if two processes are grabbing locks without following the same order in acquiring and releasing them. For example:

  Process 1 grabs lock A = success
  Process 2 grabs lock B = success
  Process 1 grabs lock B = block
  Process 2 grabs lock A = impossible

PostgreSQL has an internal mechanism for detecting such a scenario, and it (by default) kicks in 1 second after a blocking lock. If a deadlock is detected, one of the parties receives an error and their transaction is rolled back.

In our case though, we’re not acquiring any obvious locks (explicitly with an SQL LOCK command, or select.. FOR UPDATE). So what gives ?

It turns out that, when tables are referencing each other using foreign keys, PostgreSQL will implicitly acquire a row lock on the referenced row in the secondary (parents) table to protect the atomicity of the entire operation. Therefore, in the above example, both step 2 and step 4 acquire a shared lock on the parents row id 1. Step 5 then blocks on the lock held by step 4. Step 6 causes the deadlock, which aborts session 2’s transaction. This releases the shared lock and allows step 5 to proceed.

This has been a long-standing issue in the database and the PostgreSQL hackers refer to that type of thing as “user-hostile”. Recently, I ran into this issue and through searching saw it was recently re-ignited in August by Joel Jacobson . The discussion had some excellent ideas but it went dormant after a while.

From ActiveRecord’s perspective

All clients to PostgreSQL being equal, Rails/ActiveRecord is at a disadvantage with regards to this bug, mainly because:

It’s a web framework – where the usual use case is concurrent access
A popular little feature called “cached counters”
The ease of extending an ActiveRecord model with callbacks, touching the parent for various reasons
The abstraction between the developer and the raw SQL calls, which, next to lowering the barrier of entry, makes isolating and investigating such a problem more difficult

This problem may manifest itself in Rails with an error whose stack trace includes something like:

ActiveRecord::StatementInvalid: PGError: ERROR: deadlock detected DETAIL: Process 4844 waits for ShareLock on transaction 34688860; blocked by process 4846. Process 4846 waits for ExclusiveLock on tuple (1,40) of relation 19104 of database 16388; blocked by process 4844. HINT: See server log for query details. : UPDATE 'parents' SET 'children_count' = COALESCE('children_count', 0) + 1 WHERE ('id' = 925) AND ( ('parents'.'type' = 'MaleParent' ) )

What can be done

None of the obvious solutions are pretty:

Skip cached counters, parent modifications, and anything of that sort
Remove the foreign key constraints
Don’t use transactions
Divide the operation into several transactions, where the parents aren’t modified in the same transaction as the children
Serialize the entire transaction by grabbing a lock (using an external mechanism or a PostgreSQL advisory lock)
Add client code to handle the deadlock error and (safely) re-try the needed operations
Fix the issue in PostgreSQL

The viability of all of the above depends on the application, however fixing the issue in PostgreSQL isn’t out of the question. As a matter of fact, it’s already begun! You can see progress here , here and here

How you can help

With a monetary contribution, small or large.

The above work was commissionned after the August discussion on the pgsql-hackers mailing list, however the funds for it have only been minimally raised and they’re still a few thousand dollars short.

If you or your employer would like to see this fix come to fruition, please visit the donation page at:

https://www.fossexperts.com/content/foreign-key-locks-0

Every little bit helps. You’ll be helping improve an already great open source database, as well as all who rely on it including ROR consumers.

Update

28 patch revisions later, it seems that Alvaro Herrera has carried this feature through. You can see the patch here

This patch is candidate for commitfest 2013-01 – which means you may see it in a beta PostgreSQL release near you sometime soon.