[panda-users] Tracing instructions from kernel space

[Jayashree Mohan]

Great!
And I really appreciate your quick response. Thanks a lot for helping
me debug this. Will let you know if the tlb flush solves my problem.

Thanks,
Jayashree Mohan


On Fri, Nov 9, 2018 at 11:42 PM Brendan Dolan-Gavitt <brendandg at nyu.edu> wrote:
>
> The memory tracing has two parts:
>
> - Modifications to tcg/i386/tcg-target.inc.c to change which function
> is called for guest memory access, and to disable the TLB lookup
> https://github.com/panda-re/panda/blob/master/tcg/i386/tcg-target.inc.c#L1258-L1277
> https://github.com/panda-re/panda/blob/master/tcg/i386/tcg-target.inc.c#L1363-L1366
>
> - Modifications to the load/store functions in softmmu_template.h to
> add PANDA callbacks:
> https://github.com/panda-re/panda/blob/master/softmmu_template.h#L439-L441
>
> We don't change the architecture-specific parts of translation (which
> would be in target/i386/translate.c) – but as far as we know all guest
> memory accesses end up going through the softmmu_template.h functions
> as long as the TLB lookup is disabled.
>
> -Brendan
>
> On Sat, Nov 10, 2018 at 12:33 AM, Jayashree Mohan
> <jayashree2912 at gmail.com> wrote:
> > In the record path, do you track all memory accesses by translating
> > the mov, movnti,... instructions? Could you point me to where this
> > tracing happens for x86 architecture?
> >
> > Thanks,
> > Jayashree Mohan
> >
> >
> > On Fri, Nov 9, 2018 at 11:26 PM Brendan Dolan-Gavitt <brendandg at nyu.edu> wrote:
> >>
> >> Hmm, is this being done on a live VM rather than a replay? It's
> >> possible you need to flush the translation cache with
> >> panda_do_flush_tb() in your init_plugin. Enabling memory callbacks
> >> changes the way guest code is translated (specifically, it ensures
> >> that every memory access goes through the "slow path" which is
> >> instrumented, rather than using a TLB lookup which is not
> >> instrumented) – so code that executed before your plugin was loaded
> >> may be cached without instrumentation.
> >>
> >> -Brendan
> >>
> >> On Sat, Nov 10, 2018 at 12:09 AM, Jayashree Mohan
> >> <jayashree2912 at gmail.com> wrote:
> >> > To reproduce the problem I am facing try this :
> >> >
> >> > I've enabled tracking the entire memory region in the plugin here :
> >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_williewillus_panda-5Fscratchpad_blob_jaya-5Fdev_personal-5Fplugins_panda_plugins_writetracker_writetracker.cpp&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=vdDK3h0JrBAVEQOA5NPJJzN37RqUrQep0dSoDS6Spp8&s=g2YXZwKwkUfasXrWZZOXh-W3Emu3QmZgzd8ZfnucDj8&e=
> >> >
> >> > and the test workload file I am using is here :
> >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_williewillus_panda-5Fscratchpad_blob_jaya-5Fdev_communication_write-2Dcacheline.c&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=vdDK3h0JrBAVEQOA5NPJJzN37RqUrQep0dSoDS6Spp8&s=smaGGWv4JFQLrJUzg2YZKBjwuTOTHNl-kH7eF33682w&e=
> >> >
> >> > It simply writes 64 B of data into the file.
> >> >
> >> > What I am trying to do is -
> >> > 1. Create a pmem device / ramdisk  and mount it on ext4-dax at /mnt/pmem0
> >> > 2. Now start up the VM on QEMU and load the writetracker plugin above
> >> > 3. On the VM, execute the workload above, which writes 64B, followed by a
> >> > fsync
> >> > 4. Unload the write tracker plugin. This would have traced memory writes
> >> > along with the address and data in the wt.out file in the host.
> >> > 5. You can simply parse the file to see the address and data by using the
> >> > src/reader.cpp file in the repo.
> >> >
> >> > On doing the above steps, I see the behaviour I described. When I write 64B,
> >> > I don't see the data, which is a series of RRRRR... in the parsed wt.out
> >> > file. Rather, if I just change this to write 63 B instead, I can see it.
> >> >
> >> > Sample snippet from the parsed file when I write 63B :
> >> >
> >> > [pc 0xffffffff81456121] write to VA 41411000, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411008, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411010, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411018, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411020, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411028, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456121] write to VA 41411030, size 8, Data : RRRRRRRR
> >> > [pc 0xffffffff81456149] write to VA 41411038, size 4, Data : RRRR
> >> > [pc 0xffffffff8145615d] write to VA 4141103c, size 1, Data : R
> >> > [pc 0xffffffff8145615d] write to VA 4141103d, size 1, Data : R
> >> > [pc 0xffffffff8145615d] write to VA 4141103e, size 1, Data : R
> >> >
> >> > Though it says VA, the physical address is being printed, which is something
> >> > around 1GB. My emulated pmem device occupies 1GB- 1GB+128MB in the physical
> >> > memory - so this makes sense.
> >> >
> >> > Irrespective of the address, even if I run the workload on a ramdisk, I
> >> > would expect to see 'RRRR....'  in the trace file, which I don't if I write
> >> > multiples of aligned 64B.
> >> >
> >> > Hope this helps reproduce the issue. Let me know if you need more details or
> >> > if I am doing some mistake here.
> >> >
> >> > Thanks,
> >> > Jayashree Mohan
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Jayashree Mohan
> >> >
> >> >
> >> >
> >> > On Fri, Nov 9, 2018 at 10:19 PM Brendan Dolan-Gavitt <brendandg at nyu.edu>
> >> > wrote:
> >> >>
> >> >> Hmm, something definitely seems odd – with this test program:
> >> >>
> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_moyix_ed0d6dde9bc8164ff5e58030282d72af&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=vdDK3h0JrBAVEQOA5NPJJzN37RqUrQep0dSoDS6Spp8&s=1oFxSLFP6Gl5iPdBGajVymKqzp2c6JOHd0gYtOcbqoY&e=
> >> >>
> >> >> and then testing with -panda stringsearch:str="averylongstring" I can
> >> >> see writes in the kernel in __copy_from_user_ll_nozero – but only if I
> >> >> do a memcpy to a different userspace buffer first!
> >> >>
> >> >> Could you share the userland program you're using to test so I can
> >> >> compare?
> >> >>
> >> >> -Brendan
> >> >>
> >> >> On Fri, Nov 9, 2018 at 9:06 PM, Jayashree Mohan <jayashree2912 at gmail.com>
> >> >> wrote:
> >> >> > Hi Brendan,
> >> >> >
> >> >> > We verified by enabling tracing the entire memory region rather than
> >> >> > confining it to 1-2G. However, the writes still cannot be traced. The
> >> >> > behaviour we see if rather interesting. When we write something less
> >> >> > than a cacheline(64B) using a write system call followed by fsync, it
> >> >> > gets traced by PANDA during the PANDA_CB_VIRT_MEM_AFTER_WRITE
> >> >> > callback. However, when we write anything in multiples of aligned
> >> >> > cachelines, we don't see any memory write traces. For example, if I
> >> >> > write 258B into a file, I can see the last two bytes of data alone.
> >> >> > This seems weird as PANDA is not tracing full aligned cachelines. Do
> >> >> > you seem to understand why this could be happening?
> >> >> >
> >> >> > Thanks,
> >> >> > Jayashree Mohan
> >> >> >
> >> >> > On Fri, Nov 9, 2018 at 11:41 AM Brendan Dolan-Gavitt <brendandg at nyu.edu>
> >> >> > wrote:
> >> >> >>
> >> >> >> The only thing I can think of from looking at your code briefly is
> >> >> >> your use of the physical address range to restrict it to only log
> >> >> >> writes in the 1-2GB range. Could it be that the kernel does
> >> >> >> copy_from_user at the start and copies it into someplace outside that
> >> >> >> range, then writes it back with copy_to_user at the end?
> >> >> >>
> >> >> >> On Fri, Nov 9, 2018 at 11:01 AM, Jayashree Mohan
> >> >> >> <jayashree2912 at gmail.com> wrote:
> >> >> >> > Hi Brendan,
> >> >> >> >
> >> >> >> > Thanks for the reply.
> >> >> >> >
> >> >> >> > Take a look at the plugin here :
> >> >> >> >
> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_williewillus_panda-5Fscratchpad_blob_master_personal-5Fplugins_panda_plugins_writetracker_writetracker.cpp&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=kHkbR-cTusA4zz6euYKwT5Pzf8nrwZP1idV_9NTfZaA&s=BDZjQuADJrBGUzNIgo2EW_xVvAN7aN-caR1Rl0PqNgo&e=
> >> >> >> >
> >> >> >> > We load this plugin, and in the VM, write a simple program that
> >> >> >> > writes to
> >> >> >> > the pmem device mounted within the memory region being tracked. I see
> >> >> >> > memcpy
> >> >> >> > writes being traced, but not the ones due to write system call.
> >> >> >> >
> >> >> >> > I'll try checking if any of my writes originate in the kernel.
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Jayashree Mohan
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Jayashree Mohan
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Nov 9, 2018 at 9:56 AM Brendan Dolan-Gavitt
> >> >> >> > <brendandg at nyu.edu>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Yes, it should definitely be tracing memory accesses in the kernel
> >> >> >> >> (it
> >> >> >> >> traces all memory accesses on the system) – could you post your
> >> >> >> >> plugin
> >> >> >> >> code?
> >> >> >> >>
> >> >> >> >> You may also want to simply log all memory accesses, along with the
> >> >> >> >> current program counter and (optionally) whether or not they
> >> >> >> >> originate
> >> >> >> >> in the kernel (using the panda_in_kernel API) to debug.
> >> >> >> >>
> >> >> >> >> On Fri, Nov 9, 2018 at 10:35 AM, Jayashree Mohan
> >> >> >> >> <jayashree2912 at gmail.com> wrote:
> >> >> >> >> > Hi all,
> >> >> >> >> >
> >> >> >> >> > I am using PANDA to trace all store instructions in an emulated
> >> >> >> >> > pmem
> >> >> >> >> > device. I do this by writing a plugin that issues calbacks on
> >> >> >> >> > "PANDA_CB_VIRT_MEM_AFTER_WRITE" events. If I run a simple workload
> >> >> >> >> > that does write() system calls followed by mmap and memcpy, I can
> >> >> >> >> > see
> >> >> >> >> > the callbacks being triggered for the user-space memcpy calls to a
> >> >> >> >> > file, but not anytime during the write system call. Does PANDA
> >> >> >> >> > allow
> >> >> >> >> > tracing instructions from the kernel space?
> >> >> >> >> >
> >> >> >> >> > Thanks,
> >> >> >> >> > Jayashree Mohan
> >> >> >> >> > _______________________________________________
> >> >> >> >> > panda-users mailing list
> >> >> >> >> > panda-users at mit.edu
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.mit.edu_mailman_listinfo_panda-2Dusers&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=t_XD-sNNGDpfuGLf63sp5f-I-OP6dhEVNn-r9F-giQU&s=o4Ml-SG3gwaAZ7JrRz3N2W7BvJdTyZvua-jgEyicY2Q&e=
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Brendan Dolan-Gavitt
> >> >> >> >> Assistant Professor, Department of Computer Science and Engineering
> >> >> >> >> NYU Tandon School of Engineering
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Brendan Dolan-Gavitt
> >> >> >> Assistant Professor, Department of Computer Science and Engineering
> >> >> >> NYU Tandon School of Engineering
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Brendan Dolan-Gavitt
> >> >> Assistant Professor, Department of Computer Science and Engineering
> >> >> NYU Tandon School of Engineering
> >>
> >>
> >>
> >> --
> >> Brendan Dolan-Gavitt
> >> Assistant Professor, Department of Computer Science and Engineering
> >> NYU Tandon School of Engineering
>
>
>
> --
> Brendan Dolan-Gavitt
> Assistant Professor, Department of Computer Science and Engineering
> NYU Tandon School of Engineering