scnotes

[scnotes]

Day 7 - enums, structs, unions in SDF & /dev/crash officially writeable

I added support for parsing enums, structs and unions in SDF and wrote a small utility called tprint (short for "type print" and in the same nature of sizeof) that to print them out. Here is what the code for that looks like:

$ cat -n cmd/tprint/tprint.go 1 package main 2 3 import ( 4 "fmt" 5 "os" 6 "sdimitro/dbg/debug/sdf" 7 ) 8 9 func printAggregateType(_sdf *sdf.Sdf, typeName string, aggregateType string) { 10 fields, err := _sdf.Members(typeName) 11 if err != nil { 12 fmt.Fprintln(os.Stderr, err) 13 os.Exit(1) 14 } 15 16 fmt.Printf("%s %s {\n", aggregateType, typeName) 17 for _, nv := range fields { 18 fmt.Printf("\t%s %s\n", nv.TypeName, nv.Name) 19 } 20 fmt.Println("}") 21 } 22 23 func main() { 24 if len(os.Args) != 3 { 25 fmt.Printf("usage: %s <type name> <file>\n", os.Args[0]) 26 os.Exit(2) 27 } 28 29 typeName := os.Args[1] 30 filePath := os.Args[2] 31 32 _sdf, err := sdf.New([]string{filePath}) 33 if err != nil { 34 fmt.Fprintln(os.Stderr, err) 35 os.Exit(1) 36 } 37 38 tag, err := _sdf.GetSdfTag(typeName) 39 if err != nil { 40 fmt.Fprintln(os.Stderr, err) 41 os.Exit(1) 42 } 43 44 switch tag { 45 case sdf.SdfBaseTag: 46 fmt.Println(typeName, "is scalar") 47 case sdf.SdfUnionTag: 48 printAggregateType(_sdf, typeName, "union") 49 case sdf.SdfStructTag: 50 printAggregateType(_sdf, typeName, "struct") 51 case sdf.SdfEnumTag: 52 enums, err := _sdf.EnumValues(typeName) 53 if err != nil { 54 fmt.Fprintln(os.Stderr, err) 55 os.Exit(1) 56 } 57 58 fmt.Printf("enum %s {\n", typeName) 59 for _, nv := range enums { 60 fmt.Printf("\t%s = %X\n", nv.Name, nv.Value) 61 } 62 fmt.Println("}") 63 case sdf.SdfInvalid: 64 fmt.Fprintf(os.Stderr, "Invalid tag was returned for type '%s'", 65 typeName) 66 os.Exit(1) 67 default: 68 fmt.Fprintf(os.Stderr, "Unknown Tag %d\n", tag) 69 os.Exit(1) 70 } 71 72 os.Exit(0) 73 }

Here are a few examples:

$ tprint zfs_space_check zfs-with-dwarf.ko enum zfs_space_check { ZFS_SPACE_CHECK_NORMAL = 0 ZFS_SPACE_CHECK_RESERVED = 1 ZFS_SPACE_CHECK_EXTRA_RESERVED = 2 ZFS_SPACE_CHECK_DESTROY = 2 ZFS_SPACE_CHECK_ZCP_EVAL = 2 ZFS_SPACE_CHECK_NONE = 3 ZFS_SPACE_CHECK_DISCARD_CHECKPOINT = 3 } $ tprint fletcher_4_ctx zfs.dbg union fletcher_4_ctx { zio_cksum_t scalar superscalar sse avx avx512 } $ tprint spa_t zfs.dbg struct spa { spa_name spa_comment avl_node_t spa_avl spa_config spa_config_syncing spa_config_splitting spa_load_info uint64_t spa_config_txg int spa_sync_pass pool_state_t spa_state ...

Future work: As you can see there is still work to do because for now Sdf doesn't know yet about pointers, arrays, and consts. I've left that as future work for now. I've also added unit tests for enumerations but not for structs or unions yet. I also have only tested the above with the zfs kernel module. I still need to do the final test with the linux kernel binary. I've also left a lot of XXXs around the code, especially around generating the type names of members in structs and unions on-demand. Furthermore, I'd like to add the ability in tprint to print typedefs and get passed arguments to print out the whole path of the typedef being resolved. Finally, I'd like to rewrite part of the parsing logic where the SDF module tries all parsers (ELF + DWARF, DWARF and SDF dump so far) to look more like the loadDwarfData() where we just print at the end that we have an "unrecognized binary format".

Announcement: I managed to get my code for writing /dev/crash upstreamed in the official crash repo.

[scnotes]

Day 6 - /dev/crash memory module

I find it annoying that even though we can read from /proc/kcore we will never be able to write to it. That’s why I decided to give /dev/crash a chance out of curiosity.

I was able to read from it through my sample program from the addendum of Day 5. The actual offset I got again from using gdb on crash:

crash> rd zfs_flags ffffffffc096d6e8: 0000000300000000 crash> sym zfs_flags ffffffffc096d6e8 (b) zfs_flags [zfs] Breakpoint 2, read_memory_device (fd=4, bufptr=0x7fffffffd558, cnt=8, addr=18446744072645695208, paddr=9138570984) at memory.c:2528 2528 if (pc->curcmd_flags & XEN_MACHINE_ADDR) (gdb) p/x 18446744072645695208 $1 = 0xffffffffc096d6e8 (gdb) p/x 9138570984 $2 = 0x220b386e8 // <---- the paddr was used for the lseek()

As for how we got the paddr, I just stepped down one frame:

#0 read_memory_device (fd=4, bufptr=0x7fffffffd558, cnt=8, addr=<optimized out>, paddr=9138570984) at memory.c:2537 #1 0x0000555555675121 in readmem (addr=18446744072645695208, memtype=1, buffer=<optimized out>, size=8, type=0x7fffffffd570 "64-bit KVADDR", error_handle=1) at memory.c:2254 #2 0x0000555555692e49 in display_memory (addr=<optimized out>, count=1, flag=1168, memtype=1, opt=<optimized out>) at memory.c:1570

the code:

2201 while (size > 0) { 2202 switch (memtype) 2203 { 2204 case UVADDR: 2205 if (!uvtop(CURRENT_CONTEXT(), addr, &paddr, 0)) { 2206 if (PRINT_ERROR_MESSAGE) 2207 error(INFO, INVALID_UVADDR, addr, type); 2208 goto readmem_error; 2209 } 2210 break; 2211 2212 case KVADDR: // <------ I think this is memtype == 1 2213 if (!kvtop(CURRENT_CONTEXT(), addr, &paddr, 0)) { 2214 if (PRINT_ERROR_MESSAGE) 2215 error(INFO, INVALID_KVADDR, addr, type); 2216 goto readmem_error; 2217 } 2218 break;

so that kvtop()[kernel virtual to physical] did the translation.

I was able to read zfs_flags from it:

$ sudo ./sample attempting to read from /dev/crash ... read 8 bytes - value: 300000000

Then I looked at the source code of /dev/crash and even though I could find code for reading memory I couldn't find any code for writing memory (which is weird because there is a function called write_memory_device in crash's code that seems to do that, but I'm not sure how it would plug in to the /dev/crash module if there is no write callback implemented:

static struct file_operations crash_fops = { .owner = THIS_MODULE, .llseek = crash_llseek, .read = crash_read, .unlocked_ioctl = crash_ioctl, .open = crash_open, .release = crash_release, };

Just for kicks I decided to change the module myself and add a write callback (it looked very similar to crash_read() but did the opposite):

static ssize_t crash_write(struct file *file, const char *buf, size_t count, loff_t *poff) { void *vaddr; struct page *page; u64 offset; ssize_t written; char *buffer = file->private_data; offset = *poff; if (offset >> PAGE_SHIFT != (offset+count-1) >> PAGE_SHIFT) return -EINVAL; vaddr = map_virtual(offset, &page); if (!vaddr) return -EFAULT; /* * Use bounce buffer to bypass the CONFIG_HARDENED_USERCOPY * kernel text restriction. */ if (copy_from_user(buffer, buf, count)) { unmap_virtual(page); return -EFAULT; } if (probe_kernel_write(buffer, vaddr, count)) { unmap_virtual(page); return -EFAULT; } unmap_virtual(page); written = count; *poff += written; return written; } ... static struct file_operations crash_fops = { .owner = THIS_MODULE, .llseek = crash_llseek, .read = crash_read, .write = crash_write, // <--- Makis in da house! .unlocked_ioctl = crash_ioctl, .open = crash_open, .release = crash_release, };

It actually compiled in the first try, which was surprising. So I went ahead and loaded that module and then changed my sample program to read zfs_flags then change one bit in it and then read it back. Sadly this is what I got back:

$ sudo ./sample attempting to read from /dev/crash ... read 8 bytes - value: 300000000 wrote 8 bytes - value: 310000000 read 8 bytes - value: 300000000

The value didn't change and I have no idea why. I decided to trace my new function in the crash module with bpftrace to see if it is even run and it seems like we do get there:

$ sudo trace-bpfcc '::crash_write "%x %x %d", arg1, arg2, arg3' PID TID COMM FUNC - 11483 11483 sample crash_write e2796300 7ee58 8

Then I decided to ensure that it returns what I expect (it should be 8 - written):

$ sudo trace-bpfcc 'r::crash_write "%d", retval' PID TID COMM FUNC - 12388 12388 sample crash_write 8

... OK OK OK ... while writing this I realized that I am an idiot .. I changed this simple thing in the code of my crash_write() function:

188 // if (probe_kernel_write(buffer, vaddr, count)) { 189 if (probe_kernel_write(vaddr, buffer, count)) {

and tried again:

$ sudo ./sample attempting to read from /dev/crash ... read 8 bytes - value: 300000000 wrote 8 bytes - value: 310000000 read 8 bytes - value: 310000000

Now we are talking!

So the next step would be to write a proper ReaderWriter interface for kernel memory around the crash module. Within that the main question would be to do a proper translation of virtual to physical addresses. Then once this is done, the next step would be to create an interface that given a namelist we return the address of that symbol in memory.

[scnotes]

[Addendum] Day 5 - Reading kernel memory

I spent some time figuring out where crash reads the kernel’s memory from and how it does it. Booting crash right away we can find the source:

$ sudo crash /usr/lib/debug/boot/vmlinux-4.15.0-43-generic ... <truncated startup output> ... WARNING: kernel relocated [962MB]: patching 99464 gdb minimal_symbol values KERNEL: /usr/lib/debug/boot/vmlinux-4.15.0-43-generic DUMPFILE: /proc/kcore CPUS: 2 ... <truncated startup output> ... crash>

Besides /proc/kcore above I've also heard about /dev/mem and /dev/crash. The latter is a crash specific driver in case both former options are protected. Reading through the source code of crash it seems like it first looks at /dev/mem, then /proc/kcore, and finaly /dev/crash.

I specifically care about the first two options. So I attempted to make a small program in Go that tries to read something from them. In order to know that the value read is valid though I need to use a legitimate example and verify its value. I decided to pick the zfs_flags global.

Using crash we can find the value of zfs_flags on my running system and its virtual address in memory:

crash> rd zfs_flags ffffffffc07ac6e8: 0000000300000000

I doubt that ffffffffc07ac6e8 will be the offset of any of our files above though, thus I decided to use gdb on crash and find where we seek /proc/kcore. After some reading I found the relevant source code:

/* * Read from /proc/kcore. */ int read_proc_kcore(int fd, void *bufptr, int cnt, ulong addr, physaddr_t paddr) { ... if (offset == UNINITIALIZED) return SEEK_ERROR; if (lseek(fd, offset, SEEK_SET) != offset) perror("lseek"); if (read(fd, bufptr, readcnt) != readcnt) return READ_ERROR; return cnt; }

Then I put a breakpoint in that function with gdb and printed the value of the offset variable right before the lseek:

$ sudo gdb /usr/bin/crash (gdb) b cmd_rd ... (gdb) b read_proc_kcore ... run /usr/lib/debug/boot/vmlinux-4.15.0-43-generic ... <after a few continues and steps > ... (gdb) step 4205 if (lseek(fd, offset, SEEK_SET) != offset) (gdb) p offset $4 = 140736422668008 (<-- which is 0x7fffc07ae6e8 in hex)

(Keep in mind that for all of the above I had to download and compile crash from source in order to get the debug info).

With all of the above in mind then I wrote my Go test program:

$ cat -n sample.go 1 package main 2 3 import ( 4 "encoding/binary" 5 "fmt" 6 "io" 7 "os" 8 ) 9 10 func readZfsFlagsFrom(path string) error { 11 fmt.Printf("attempting to read from %s ...\n", path) 12 13 f, err := os.Open(path) 14 if err != nil { 15 return err 16 } 17 defer f.Close() 18 19 // the offset I pulled from `crash` 20 offset := int64(0x7fffc07ae6e8) 21 22 _, err = f.Seek(offset, io.SeekStart) 23 if err != nil { 24 return err 25 } 26 27 buf := make([]byte, 8) 28 n, err := f.Read(buf) 29 if err != nil { 30 return err 31 } 32 33 value := binary.LittleEndian.Uint64(buf) 34 fmt.Printf("read %d bytes - value: %x\n", n, value) 35 36 return nil 37 } 38 39 func main() { 40 for _, path := range []string{"/proc/kcore", "/dev/mem"} { 41 if err := readZfsFlagsFrom(path); err != nil { 42 fmt.Fprintln(os.Stderr, err) 43 os.Exit(1) 44 } 45 } 46 }

Running that program:

$ sudo ./sample attempting to read from /proc/kcore ... read 8 bytes - value: 300000000 attempting to read from /dev/mem ... read /dev/mem: bad address

Plainly reading from/proc/kcore works! (and from /dev/mem fails as expected due to some kernel compilation configuration).

The next step now, would be to figure out two things: [1] How to get the address of zfs_flags from the running kernel based on its namelist (and potentially KASLR). [2] Figure out how to translate that to its respective offset in /proc/kcore.

[scnotes]

Day 5 - Testing, Coverage, Benchmarking, Profiling

This was a lot easier than what I expected. Just by quickly writing unit tests in table-driven style, I got coverage, benchmarking, and profiling for free.

$ go test -v === RUN TestSizeOf --- PASS: TestSizeOf (2.59s) === RUN TestDumpedSizeOf --- PASS: TestDumpedSizeOf (3.31s) PASS ok sdimitro/dbg/debug/sdf 5.925s

A sneak peak at my testing table:

type apiTest struct { typeName string expectApiError bool expectedSize int64 } type testGroup struct { file string expectParseError bool tests []apiTest } var testDir = "testdata/" var testTable = []testGroup{ {"minimal-no-syms", true, []apiTest{}}, {"minimal-with-syms", false, []apiTest{ {"bogus", true, -1}, {"char", false, 1}, {"int", false, 4}, {"uint8_t", false, 1}, {"boolean", false, 4}, {"boolean_t", false, 4}, {"complex", false, 1040}, {"complex_t", false, 1040}, }}, {"zfs-stripped.ko", true, []apiTest{}}, {"zfs-with-dwarf.ko", false, []apiTest{ {"bogus", true, -1}, {"int", false, 4}, {"uint64_t", false, 8}, {"boolean_t", false, 4}, {"zfs_space_check", false, 4}, {"zfs_space_check_t", false, 4}, {"spa", false, 6696}, {"spa_t", false, 6696}, }}, ..etc

As for some preliminary benchmarking:

func BenchmarkZfsKernelModuleLoadAndSize(b *testing.B) { for i := 0; i < b.N; i++ { _sdf, err := New([]string{"testdata/zfs-with-dwarf.ko"}) if err != nil { b.Error(err) } size, err := _sdf.SizeOf("spa_t") if err != nil { b.Error(err) } if size == -1 { b.Error("returned size: -1") } } }

and running it:

$ go test -bench=. goos: darwin goarch: amd64 pkg: sdimitro/dbg/debug/sdf BenchmarkZfsKernelModuleLoadAndSize-8 1 1180891330 ns/op PASS ok sdimitro/dbg/debug/sdf 7.022s

Kind of annoying that this is consistenly in nanoseconds but ok.

$ go test -coverprofile coverage.out PASS coverage: 84.2% of statements ok sdimitro/dbg/debug/sdf 5.911s

The remaining 16% are error cases for when the DWARF format of our target is invalid. Also using the cover tool we can get a pretty line by line analysis of what was covered.

As for CPU and Memory profiling I can gather results from the benchmarking and get beautiful reports with the pprof tool. Memory is something that I am very much concerned, especially when converting DWARF to SDF and unfortunately I don't fully understand the results from the memory reports just yet. I'm about to start reading this and a few other resources to get acquainted.

So what is next? I think I'll leave pprint as a future TODO and work on KmemReader.

[scnotes]

Day 4 - gzip compression + code polishing

This is not really a big entry. I polished the code a bit more on my repository and after that I decided to add compression. I decided to go with gzip for no particular reason really. Here is the diff:

diff --git a/debug/sdf/sdf.go b/debug/sdf/sdf.go index bbecd1d..21a71c4 100644 --- a/debug/sdf/sdf.go +++ b/debug/sdf/sdf.go @@ -1,6 +1,8 @@ package sdf import ( + "bytes" + "compress/gzip" "debug/dwarf" "debug/elf" "encoding/gob" @@ -383,15 +472,27 @@ func gobRegisterConcreteTypes() { } func (s *Sdf) loadSdfFile(path string) error { + var buf bytes.Buffer + file, err := os.Open(path) if err != nil { return err } defer file.Close() - gobRegisterConcreteTypes() + zr, err := gzip.NewReader(file) + if err != nil { + return err + } + defer zr.Close() + + _, err = io.Copy(&buf, zr) + if err != nil { + return err + } - dec := gob.NewDecoder(file) + gobRegisterConcreteTypes() + dec := gob.NewDecoder(&buf) err = dec.Decode(&s.Sources) if err != nil { return err @@ -441,23 +546,46 @@ func (s *Sdf) Dump(path string) error { func (s *Sdf) Dump(path string) error { + var buf bytes.Buffer + + gobRegisterConcreteTypes() + enc := gob.NewEncoder(&buf) + err := enc.Encode(s.Sources) + if err != nil { + return err + } + err = enc.Encode(s.TypeMap) + if err != nil { + return err + } + err = enc.Encode(s.TypeNameMap) + if err != nil { + return err + } + err = enc.Encode(s.ConflictMap) + if err != nil { + return err + } + file, err := os.Create(path) if err != nil { return err } defer file.Close() - gobRegisterConcreteTypes() + zw := gzip.NewWriter(file) + defer zw.Close() - enc := gob.NewEncoder(file) - enc.Encode(s.Sources) - enc.Encode(s.TypeMap) - enc.Encode(s.TypeNameMap) - enc.Encode(s.ConflictMap) + _, err = zw.Write(buf.Bytes()) + if err != nil { + return err + } return nil }

If you take out the error-handling that I added for the gob in Dump() (I had forgotten to add it the first when I first implemented that feature), the lines of code that I introduced to add compression were 18. That's 18 lines using the standard library to add compression. Not bad really.

As for the impact of compression: Before compression, using sdfdump to generate the SDF gob from the Linux Kernel's DWARF data, and loading the gob with sizeof we get the following runtimes and gob size:

$ time ./sdfdump /usr/lib/debug/boot/vmlinux-4.15.0-43-generic real 0m27.813s user 0m30.389s sys 0m1.353s $ ls -lh dump.sdf -rw-r--r-- 1 ####### staff 77M Dec 31 06:02 dump.sdf $ time ../sizeof/sizeof cma dump.sdf 72 real 0m2.264s user 0m2.293s sys 0m0.156s

Doing the same with compression:

$ time ./sdfdump /usr/lib/debug/boot/vmlinux-4.15.0-43-generic real 0m29.942s user 0m32.411s sys 0m1.719s $ ls -lh dump.sdf -rw-r--r-- 1 ####### staff 20M Dec 31 06:02 dump.sdf $ time ../sizeof/sizeof cma dump.sdf 72 real 0m2.638s user 0m2.517s sys 0m0.224s

We get a 77% reduction in the size of the gob, while the runtime is increased by ~7% for sdfdump and 1~2% for sizeof. I believe that the tradeoff is worth it.

Next:

Document what is required for full uniquification, and then let these comments rot, until I decide to finish it off

Add unit and blackbox tests

Implement pprint and add some test cases for it too

KmemReader <3

[scnotes]

Day 3 - Gobs of Data, sdfdump, uniquification steps

I made the initial steps towards uniquification of types. Currently sdf can handle uniquifying types from all compilation units within a specific object. The good news is that with the current structure it should be easy to add uniquification among multiple objects. The only set back is that I haven't exactly thought of a good way to exposing the choice to the consumer between the different types that appear under the same name yet (I'll probably spit out the different options and accept the module'compilationUnit'name notation for the choice - but with backticks).

The uniquification steps added some overhread but thankfully not orders of magnitute. Here is the time of the sizeof utility before:

$ time sizeof int ~/linux-crash/vmlinux-4.13.0-16-generic 4 real 0m23.720s user 0m22.059s sys 0m3.030s

And here it is with uniquification:

$ time ./sizeof cma /usr/lib/debug/boot/vmlinux-4.15.0-43-generic 72 real 0m24.496s user 0m27.970s sys 0m1.126s

In terms of computational complexity the current design, goes through all the DWARF entries once to aggregate all the type names and build up the intermediate data for the uniquification. The amount of these intermediate data have a worst-case boundary that is equal to the number of the DWARF entries and we iterate over them once to finish up uniquification. Thus the boundary of computational complexity still stays at O(n) where n is the number of DWARF entries.

The above runtimes still bother me though and that's why I decided to give Go's gob encoding a try. I implemented the Dump() API call and added some more code in the Load() call to handle this encoding. Then I created the sdfdump command that basically eats DWARF data and spits out a gob. It's source code looks like this:

$ cat cmd/sdfdump/sdfdump.go package main import ( "fmt" "os" "sdimitro/dbg/debug/sdf" ) func main() { if len(os.Args) != 2 { fmt.Println("usage: %s ", os.Args[0]) os.Exit(2) } filepath := os.Args[1] _sdf, err := sdf.New([]string{filepath}) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } err = _sdf.Dump("dump.sdf") if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } os.Exit(0) }

WIth the debug info for the linux kernel being 400~600MB the generated gob is 77MB, which is actually still bad in my opinion. At least using the sizeof utility on the gob we get a substantial speedup:

$ time ./sizeof cma ../sdfdump/dump.sdf 72 real 0m2.440s user 0m2.340s sys 0m0.268s

That's not bad, assuming this is the startup time for our hypothetical debugger. Especially if you take into account how simplistic and small the code (Load/Dump for gob is less than 100 lines). Compressing the gob with tar (gzip) we can get it down to 20MB. With this in mind it may be worth considering adding a few more lines (less than 10 really) to compress and decompress the gob, as long as the runtime doesn't get a hit.

Even though far from our reach it is worth pointing out that the CTF data for our current illumos kernel in Delphix OS are 1~2MB. For now this is far from reach (we should also keep in mind though that the Linux Kernel has 2~4 times the number of types of the illumos kernel).

I think from here I can try out a few different directions:

Give a try at the compression idea above.

Enable reading of pointer, array, const, and volatile type DWARF data. This should be straighforward.

With the sizeof utility out of the way, the pprint utility should be next. The utility should include pretty-printing function signatures including the argument types.

Polish the code, even though the are ~500 LoC in the library there is some code duplication, and a few comments could be written for some tricky parts of the code.

Add more code in the test suite which is currently in a state of atrophy.

Start implementing the next phase of the dbg project which is the KmemReader functionality. Once that is in place, I'll be able to look at kernel global variables at runtime. If getting writting permissions is straightforward too, implementing KmemWriter should be very similar to the KmemReader interface. That would be a significant milestone as we'll no longer need the zfs-params hacks from ZoL.

[scnotes]

Day 2 - Parsing DWARF info

The SDF interface is slowly improving (current version is 0.4). I implemented the sizeof command line utility which for now only works with basic types.

Its source looks like this:

$ cat cmd/sizeof/sizeof.go package main import ( "fmt" "os" "sdimitro/dbg/debug/sdf" ) func main() { if len(os.Args) != 3 { // ... show usage ... return } typename := os.Args[1] filepath := os.Args[2] _sdf, _ := sdf.New([]string{filepath}) size, _ := _sdf.SizeOf(typename) fmt.Println(size) }

The call to New() parses the target file and constructs internal state for quick access to data types. It's runtime complexity is presumably relative to the size of the target file in terms of debugging sections and entries within those sections. The call to SizeOf() should take constant time.

So here is the problem:

$ ls -lh ~/linux-crash/vmlinux-4.13.0-16-generic -rw-r--r-- 1 sdimitropoulos staff 539M Mar 1 2018 /home/serapheim/linux-crash/vmlinux-4.13.0-16-generic $ time sizeof int ~/linux-crash/vmlinux-4.13.0-16-generic 4 real 0m23.720s user 0m22.059s sys 0m3.030s

For some, having a lower bound of 24 seconds to start a program may seem unacceptable. Currently SDF goes through all the DWARF entries that it finds, which ideally should not happen as we don't care about children entries of functions and some other types of entries that don't have to do with data types. Thus I decided to experiment and temporarily changed SDF to always skip child entries regardless of what entry type it is reading. This would place an upper bound in the amount of time that I can save by skipping irrelevant entries.

$ time sizeof int ~/linux-crash/vmlinux-4.13.0-16-generic 4 real 0m13.023s user 0m10.270s sys 0m3.054s

So I can at most half the runtime if I'm lucky, and maybe reduce it only slightly more as I'm getting more familiar with Go. Regardless, I bet that if I saved the internal state of SDF after it has parsed the DWARF info, dump it all in a file potentially compressed and reuse it (ala CTF style) the improvements should be more dramatic. The most straightforward implementation and quick in terms of time would be to use something like gobs (see https://blog.golang.org/gobs-of-data).

That is not something that I think I want to focus for now though. Loading the zfs kernel module is not that bad:

$ time sizeof "unsigned char" /lib/modules/4.15.0-34-generic/extra/zfs/zfs.ko 1 real 0m1.965s user 0m1.738s sys 0m0.279s

Since the basis for SDF has been set, I think the next task would be creating a library that reads linux crash dumps.

[scnotes]

Day 1 - SDF interface (version 0.2)

SDF what?!?

I think a relatively simple interface for programmatic access to type information would be the following:

Load(path string) sdf, error Dump(path string) error Clear() SizeOf(typename string) uint64, error EnumValues(typename string) []{string, int32}, error StructureFields(typename string) []{string, string}, error FunctionParameters(funcname string) []{string, string}, error FunctionReturnType(funcname string) string, error

I envision the usage of the API to be the following:

[1] Call Load() at some binary with debugging info, and all the type info are parsed into internal structures by the time the call returns.

[2] Call SizeOf() to find a type's size. This size can be used by a debugger later to issue a read or a write from a memory dump or binary.

[3] EnumValues() used to return an enumeration's member names and values, this can be used to pretty print enumeration from the debugger. Similarly for structures with StructureFields().

[4] FunctionParameters() & FunctionReturnType() - used for displaying useful stack traces.

[5] Dump() to dump all the state & Clear() to get rid of it.

Some important notes:

[I] Load() should work with all of the following:

all types of binaries that have debugging info (e.g. executables, shared objects, kernel modules, kernel images, etc..)

Pure DWARF files from any version

Pure SDF dumps

Multiple units with naming conflicts

[II] All the function that take a typename parameter will resolve typedefs automatically, all the way down to base types.

[III] When reading DWARF data any entries that are children of a SubProgram (e.g. function) are skipped.

Open questions:

[I] How should we deal with type info that don't resolve all the way? What about external or volatile types?

[II] What about typedef's of function types?

A progression of toy utilities are to be written in order to realize the strengths of the library:

[1] The sizeof utility -

sizeof (base type) (file) sizeof (typedef'd type) (file) sizeof (struct type) (file) sizeof (enum type) (file) sizeof (array type) (file)

[2] The resolve utility -

resolve (typedef'd type) (file)

[3] The pprint utility -

pprint (enum type) (file) pprint (struct type) (file) pprint (function) (file) pprint (inlined) (file)

In all of the above examples file could be:

a toy executable with DWARF data and the minimal use-cases.

zfs.ko

a toy DWARF file

Linux Kernel's debug symbols file

an sdf dump

Where am I going with this?

SDF will have the type info, bundling that together with the ELF library, we get symbol names + type info. Then all you need is a special purpose Reader() implementation that can read /proc/kcore pr /dev/kmem and takes care of all the KASLR complexities for you. Add a layer of an API that uses reflection on top of these to write things like DCMDs ala mdb_ctf_read() and/or write an evaluator for ACID ASTs.

[scnotes]

Day 0 - ELF, DWARF, /dev/mem

For various reasons I decided to use Go and make small prototypes of what would otherwise be mdb dcmds for Linux. I decided to try implementing something like ::zfs_dbgmsg as a first try, because I'd be able to double check my results with the already existing /proc/spl/kstat/zfs/dbgmsg. The first steps towards such a goal would be:

Be able to read ELF data

Be able to read DWARF data

Be able to read from live memory

Fortunately the existing libraries in Go have a very nice API for reading ELF files. I made a rudimentary nm clone with the following code:

package main import ( "debug/elf" "fmt" "os" ) func main() { if len(os.Args) < 2 { fmt.Println("Usage: nmgo elf_file") os.Exit(1) } f, err := os.Open(os.Args[1]) if err != nil { panic(err) } _elf, err := elf.NewFile(f) if err != nil { panic(err) } symbols, err := _elf.Symbols() if err != nil { panic(err) } for _, sym := range symbols { fmt.Println(sym.Name) } }

I tested this on a kernel binary and compared the output to that of the original nm. Looking at the number of entries from both outputs and eye-balling some of them, the results were approximately the same.

Then I came up with the following program below that goes through all the DWARF data and only prints the basic data types:

package main import ( "debug/elf" "fmt" "io" "os" ) func main() { if len(os.Args) < 2 { fmt.Println("Usage: nmgo elf_file") os.Exit(1) } f, err := os.Open(os.Args[1]) if err != nil { panic(err) } _elf, err := elf.NewFile(f) if err != nil { panic(err) } dData, err := _elf.DWARF() if err != nil { panic(err) } dReader := dData.Reader() if err != nil { panic(err) } for { entry, err := dReader.Next() if err != nil { if err != io.EOF { panic(err) } break } if entry == nil { break } if entry.Tag == 0x24 { fmt.Println(entry) } } }

The API seemed more bare for DWARF data, but I believe this is due to the format itself. What I need in order to reach my goal is an API where give the name of a global symbol it would spit back one of these two structures:

{offset, type, size} if it is a variable or constant

{offset, size, list of parameters and their types} if it is a function

So basically, I'd probably want to read both ELF and DWARF data and construct something a Map that maps names to structures like the ones described above. Interesting implementation details that I foresee is dealing with typedefs and naming conflicts where two symbols have the same name.

As for reading memory from a live system, I consulted the source code of crash(8) which can be found here. Basically for live system analysis the physical memory source is one of the following:

1) /dev/mem/

2) /dev/kcore

3) /dev/crash

4) /dev/kmem/ under certain circumstances

According to the documentation from crash(8), if the system was configured with CONFIG_STRICT_DEVMEM or CONFIG_HARDENED_USERCOPY, then /dev/mem/ cannot be used. In addition, if the system was configured without CONFIG_PROC_CORE or /proc/kcore is not functional (whatever that means..), then it cannot be used either. /dev/crash apparently is a driver specifically made for crash to deal with systems that have the above troubles.

Taking a quick look of the source with regards to reading from /dev/mem/ (assuming this is the simplest case), according to memory_source_init all you've got to do is open("/dev/mem", O_RDWR) or open("/dev/mem/", O_RDONLY) if the former fails. Then in the idea case according to read_dev_mem() just lseek(fd, , SEEK_SET) and then read(fd, buf, cnt). Seems like there is work to be done.

[scnotes]

Hello, World!

Notes from the ambitious journey of making the Linux kernel more debuggable.