SSH does not properly erase keys from the volatile memory

Below is a report from lab testing requirements for RHEL Common Criteria certification and they think there is a problem with ssh component not properly erasing keys from the volatile memory.

While testing FCS_CKM_EXT.4 for sshd, we are finding keys left in RAM. Our test procedures are as follows:

Build and load the fmem kernel module on the TOE https://github.com/NateBrune/fmem
Examine /proc/iomem to determine all “System RAM” addresses
Establish an SSH connection to the TOE with a client that is instrumented to print the encryption keys in debug output.
Terminate the SSH connection
Use dd to save all “System RAM” addresses to disk, using /dev/fmem to access the memory
Search the memory for the encryption keys from Step 3

We have tried this with the following configurations:

normal sshd daemon, running normally
normal sshd daemon, stopping it after step 4
manually running /usr/sbin/sshd -d -r
We were trying to minimize forking behaviror

In step 6, we took a closer look at the memory that matched the keys. It looks like it is a pair of newkeys structs serialized using newkeys_to_blob (the actual keys are highlighted):

00000f60: 0000 0000 0000 0000 0000 0000 0000 0013 ................

00000f70: 5353 482d 322e 302d 4f70 656e 5353 485f SSH-2.0-OpenSSH_

00000f80: 382e 3800 0000 1353 5348 2d32 2e30 2d4f 8.8....SSH-2.0-O

00000f90: 7065 6e53 5348 5f38 2e37 0000 0030 0ae6 penSSH_8.7...0..

00000fa0: 6e81 ab6c 9263 8c09 bcfe d3dc 0b46 db09 n..l.c.......F..

00000fb0: b1b7 787e 0353 ccde 6878 0cf4 8029 ab74 ..x~.S..hx...).t

00000fc0: f53b f0d5 b33f ce95 d862 00d4 a7f7 0000 .;...?...b......

00000fd0: 000c 0000 0093 0000 000a 6165 7332 3536 ..........aes256

00000fe0: 2d63 7472 0000 0000 0000 0010 0000 0020 -ctr...........

00000ff0: 0557 c8cf af96 49a1 ccfb b4f2 0a26 8bd0 .W....I......&..

00001000: e68d 3f53 0e73 4e50 8faf 64f8 cc76 2f74 ..?S.sNP..d..v/t

00001010: 0000 0010 f461 7363 61ea dc07 2068 2f13 .....asca... h/.

00001020: 7015 e1eb 0000 000d 686d 6163 2d73 6861 p.......hmac-sha

00001030: 322d 3235 3600 0000 0100 0000 20b0 8348 2-256....... ..H

00001040: 8972 5971 ace5 768a 3454 ee6c e86d fc44 .rYq..v.4T.l.m.D

00001050: aaf5 a168 c7cf aab6 acde bccc 8800 0000 ...h............

00001060: 0000 0000 046e 6f6e 6500 0000 9300 0000 .....none.......

00001070: 0a61 6573 3235 362d 6374 7200 0000 0000 .aes256-ctr.....

00001080: 0000 1000 0000 20fc c7a6 a148 e77b 7b24 ...... ....H.{{$

00001090: 328e 61d2 1db2 a7d3 6c82 7cb7 a42e 7183 2.a.....l.|...q.

000010a0: 38f6 09b6 76f5 c100 0000 101a bee6 2b0d 8...v.........+.

000010b0: b78f 1e24 79fe 50fe 911d a700 0000 0d68 ...$y.P........h

000010c0: 6d61 632d 7368 6132 2d32 3536 0000 0001 mac-sha2-256....

000010d0: 0000 0020 b176 4737 0a93 1c13 1a48 0093 ... .vG7.....H..

000010e0: 8cc2 f362 e4b7 dd60 d616 3c17 8654 036b ...b...`..<..T.k

000010f0: 18e5 ed09 0000 0000 0000 0004 6e6f 6e65 ............none

The offsets are from the specific memory dump file which starts at address 0x100000000. I don’t think the exact addresses useful since they varied with each run.

We were not able to track done exactly which process the memory was allocated to.

This finding was very surprising to us, because we know openssh is normally very good about security. We tried looking through the code to see where/how a copy of the keys could be left in memory, but we were not able to identify any likely issues.