<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">To reproduce the problem I am facing try this :<br><br><div>I've enabled tracking the entire memory region in the plugin here : <a href="https://github.com/williewillus/panda_scratchpad/blob/jaya_dev/personal_plugins/panda/plugins/writetracker/writetracker.cpp">https://github.com/williewillus/panda_scratchpad/blob/jaya_dev/personal_plugins/panda/plugins/writetracker/writetracker.cpp</a></div><div><br></div><div>and the test workload file I am using is here : <a href="https://github.com/williewillus/panda_scratchpad/blob/jaya_dev/communication/write-cacheline.c">https://github.com/williewillus/panda_scratchpad/blob/jaya_dev/communication/write-cacheline.c</a></div><div><br></div><div>It simply writes 64 B of data into the file. <br></div><div><br></div><div>What I am trying to do is - <br></div><div>1. Create a pmem device / ramdisk and mount it on ext4-dax at /mnt/pmem0</div><div>2. Now start up the VM on QEMU and load the writetracker plugin above</div><div>3. On the VM, execute the workload above, which writes 64B, followed by a fsync</div><div>4. Unload the write tracker plugin. This would have traced memory writes along with the address and data in the wt.out file in the host. <br></div><div>5. You can simply parse the file to see the address and data by using the src/reader.cpp file in the repo.</div><div><br></div><div>On doing the above steps, I see the behaviour I described. When I write 64B, I don't see the data, which is a series of RRRRR... in the parsed wt.out file. Rather, if I just change this to write 63 B instead, I can see it.</div><div><br></div><div>Sample snippet from the parsed file when I write 63B :</div><div><br></div><div>[pc 0xffffffff81456121] write to VA 41411000, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411008, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411010, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411018, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411020, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411028, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456121] write to VA 41411030, size 8, Data : RRRRRRRR<br>[pc 0xffffffff81456149] write to VA 41411038, size 4, Data : RRRR<br>[pc 0xffffffff8145615d] write to VA 4141103c, size 1, Data : R<br>[pc 0xffffffff8145615d] write to VA 4141103d, size 1, Data : R<br>[pc 0xffffffff8145615d] write to VA 4141103e, size 1, Data : R<br></div></div><div dir="ltr"><br></div><div>Though it says VA, the physical address is being printed, which is something around 1GB. My emulated pmem device occupies 1GB- 1GB+128MB in the physical memory - so this makes sense. <br></div><div><br></div><div>Irrespective of the address, even if I run the workload on a ramdisk, I would expect to see 'RRRR....' in the trace file, which I don't if I write multiples of aligned 64B.</div><div><br></div><div>Hope this helps reproduce the issue. Let me know if you need more details or if I am doing some mistake here.<br></div><div dir="ltr"><br>Thanks,<br>Jayashree Mohan<br><br></div></div></div></div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><br>Thanks,<br>Jayashree Mohan<br><br></div></div><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 9, 2018 at 10:19 PM Brendan Dolan-Gavitt <<a href="mailto:brendandg@nyu.edu">brendandg@nyu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hmm, something definitely seems odd – with this test program:<br>
<br>
<a href="https://gist.github.com/moyix/ed0d6dde9bc8164ff5e58030282d72af" rel="noreferrer" target="_blank">https://gist.github.com/moyix/ed0d6dde9bc8164ff5e58030282d72af</a><br>
<br>
and then testing with -panda stringsearch:str="averylongstring" I can<br>
see writes in the kernel in __copy_from_user_ll_nozero – but only if I<br>
do a memcpy to a different userspace buffer first!<br>
<br>
Could you share the userland program you're using to test so I can compare?<br>
<br>
-Brendan<br>
<br>
On Fri, Nov 9, 2018 at 9:06 PM, Jayashree Mohan <<a href="mailto:jayashree2912@gmail.com" target="_blank">jayashree2912@gmail.com</a>> wrote:<br>
> Hi Brendan,<br>
><br>
> We verified by enabling tracing the entire memory region rather than<br>
> confining it to 1-2G. However, the writes still cannot be traced. The<br>
> behaviour we see if rather interesting. When we write something less<br>
> than a cacheline(64B) using a write system call followed by fsync, it<br>
> gets traced by PANDA during the PANDA_CB_VIRT_MEM_AFTER_WRITE<br>
> callback. However, when we write anything in multiples of aligned<br>
> cachelines, we don't see any memory write traces. For example, if I<br>
> write 258B into a file, I can see the last two bytes of data alone.<br>
> This seems weird as PANDA is not tracing full aligned cachelines. Do<br>
> you seem to understand why this could be happening?<br>
><br>
> Thanks,<br>
> Jayashree Mohan<br>
><br>
> On Fri, Nov 9, 2018 at 11:41 AM Brendan Dolan-Gavitt <<a href="mailto:brendandg@nyu.edu" target="_blank">brendandg@nyu.edu</a>> wrote:<br>
>><br>
>> The only thing I can think of from looking at your code briefly is<br>
>> your use of the physical address range to restrict it to only log<br>
>> writes in the 1-2GB range. Could it be that the kernel does<br>
>> copy_from_user at the start and copies it into someplace outside that<br>
>> range, then writes it back with copy_to_user at the end?<br>
>><br>
>> On Fri, Nov 9, 2018 at 11:01 AM, Jayashree Mohan<br>
>> <<a href="mailto:jayashree2912@gmail.com" target="_blank">jayashree2912@gmail.com</a>> wrote:<br>
>> > Hi Brendan,<br>
>> ><br>
>> > Thanks for the reply.<br>
>> ><br>
>> > Take a look at the plugin here :<br>
>> > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_williewillus_panda-5Fscratchpad_blob_master_personal-5Fplugins_panda_plugins_writetracker_writetracker.cpp&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=kHkbR-cTusA4zz6euYKwT5Pzf8nrwZP1idV_9NTfZaA&s=BDZjQuADJrBGUzNIgo2EW_xVvAN7aN-caR1Rl0PqNgo&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_williewillus_panda-5Fscratchpad_blob_master_personal-5Fplugins_panda_plugins_writetracker_writetracker.cpp&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=kHkbR-cTusA4zz6euYKwT5Pzf8nrwZP1idV_9NTfZaA&s=BDZjQuADJrBGUzNIgo2EW_xVvAN7aN-caR1Rl0PqNgo&e=</a><br>
>> ><br>
>> > We load this plugin, and in the VM, write a simple program that writes to<br>
>> > the pmem device mounted within the memory region being tracked. I see memcpy<br>
>> > writes being traced, but not the ones due to write system call.<br>
>> ><br>
>> > I'll try checking if any of my writes originate in the kernel.<br>
>> ><br>
>> > Thanks,<br>
>> > Jayashree Mohan<br>
>> ><br>
>> ><br>
>> ><br>
>> > Thanks,<br>
>> > Jayashree Mohan<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Fri, Nov 9, 2018 at 9:56 AM Brendan Dolan-Gavitt <<a href="mailto:brendandg@nyu.edu" target="_blank">brendandg@nyu.edu</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Yes, it should definitely be tracing memory accesses in the kernel (it<br>
>> >> traces all memory accesses on the system) – could you post your plugin<br>
>> >> code?<br>
>> >><br>
>> >> You may also want to simply log all memory accesses, along with the<br>
>> >> current program counter and (optionally) whether or not they originate<br>
>> >> in the kernel (using the panda_in_kernel API) to debug.<br>
>> >><br>
>> >> On Fri, Nov 9, 2018 at 10:35 AM, Jayashree Mohan<br>
>> >> <<a href="mailto:jayashree2912@gmail.com" target="_blank">jayashree2912@gmail.com</a>> wrote:<br>
>> >> > Hi all,<br>
>> >> ><br>
>> >> > I am using PANDA to trace all store instructions in an emulated pmem<br>
>> >> > device. I do this by writing a plugin that issues calbacks on<br>
>> >> > "PANDA_CB_VIRT_MEM_AFTER_WRITE" events. If I run a simple workload<br>
>> >> > that does write() system calls followed by mmap and memcpy, I can see<br>
>> >> > the callbacks being triggered for the user-space memcpy calls to a<br>
>> >> > file, but not anytime during the write system call. Does PANDA allow<br>
>> >> > tracing instructions from the kernel space?<br>
>> >> ><br>
>> >> > Thanks,<br>
>> >> > Jayashree Mohan<br>
>> >> > _______________________________________________<br>
>> >> > panda-users mailing list<br>
>> >> > <a href="mailto:panda-users@mit.edu" target="_blank">panda-users@mit.edu</a><br>
>> >> ><br>
>> >> > <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.mit.edu_mailman_listinfo_panda-2Dusers&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=t_XD-sNNGDpfuGLf63sp5f-I-OP6dhEVNn-r9F-giQU&s=o4Ml-SG3gwaAZ7JrRz3N2W7BvJdTyZvua-jgEyicY2Q&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.mit.edu_mailman_listinfo_panda-2Dusers&d=DwICAg&c=slrrB7dE8n7gBJbeO0g-IQ&r=A4wu5Zmpus3hDmokNWeJTO0SLjrxguzCAxn30Hc-o48&m=t_XD-sNNGDpfuGLf63sp5f-I-OP6dhEVNn-r9F-giQU&s=o4Ml-SG3gwaAZ7JrRz3N2W7BvJdTyZvua-jgEyicY2Q&e=</a><br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> Brendan Dolan-Gavitt<br>
>> >> Assistant Professor, Department of Computer Science and Engineering<br>
>> >> NYU Tandon School of Engineering<br>
>><br>
>><br>
>><br>
>> --<br>
>> Brendan Dolan-Gavitt<br>
>> Assistant Professor, Department of Computer Science and Engineering<br>
>> NYU Tandon School of Engineering<br>
<br>
<br>
<br>
-- <br>
Brendan Dolan-Gavitt<br>
Assistant Professor, Department of Computer Science and Engineering<br>
NYU Tandon School of Engineering<br>
</blockquote></div>