[panda-users] Replay of legacy records

aicardi@eurecom.fr aicardi at eurecom.fr
Thu Sep 7 03:32:56 EDT 2017 

Yes, that is exactly what I did, I needed to get a notification every  
time a new process starts/ends (because I need to track the execution  
of a specific process) so I thought it was enough to enable the  
'on_new_process' and 'on_finished_process' callbacks.

Is there another way to keep track of processes?

P.s. just to mention it, I am using those two callbacks in the new  
version of panda (panda2) and they are pretty much stable for now.


Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:

> Hmm, that code seems to be in the osi "proc events" functionality,
> which should be disabled by default (it was never developed to a
> completely stable state). Did you change the build process to enable
> it somehow? (e.g., by adding -DOSI_PROC_EVENTS to the build flags)
>
> -Brendan
>
> On Wed, Sep 6, 2017 at 11:47 AM, <aicardi at eurecom.fr> wrote:
>
>> Oh sorry for that, I didn't realize it was another problem. Here is the
>> actual backtrace (I hope):
>>
>> Stopped reason: SIGSEGV
>> ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>, ps=0x0,
>> in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at
>> osi_proc_events.cpp:78
>> 78              for (unsigned int i=0; i<ps->num; i++) {
>> gdb-peda$ bt
>> #0  ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>, ps=0x0,
>> in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at
>> osi_proc_events.cpp:78
>> #1  0x00007fffd6c8029d in procstate_update (ps=<optimized out>,   
>> in=in at entry=0x7fff62c60298,
>> out=out at entry=0x7fff62c602a0) at osi_proc_events.cpp:141
>> #2  0x00007fffd6c7f57d in vmi_pgd_changed (env=0x1c53660,
>> oldval=<optimized out>, newval=<optimized out>) at os_intro.c:120
>> #3  0x00000000006a3697 in cpu_x86_update_cr3 (env=0x1c53660,
>> new_cr3=0x185000) at /home/samaicardi/panda1/qemu/target-i386/helper.c:526
>> #4  0x00007fffa46fbf03 in ?? ()
>> #5  0x00000000ffffffff in ?? ()
>> #6  0x00007ffff6f24360 in pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
>> #7  0x0000000000000000 in ?? ()
>>
>> gdb-peda$
>>
>>
>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>
>> You need to do "handle SIG38 nostop noprint" first -- otherwise QEMU's
>>> alarm timer will cause gdb to stop constantly.
>>>
>>> On Wed, Sep 6, 2017 at 11:05 AM <aicardi at eurecom.fr> wrote:
>>>
>>> Yes sure:
>>>>
>>>> Stopped reason: SIG38
>>>> pthread_cond_wait@@GLIBC_2.3.2 () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>>>> 185     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such
>>>> file or directory.
>>>> gdb-peda$ bt
>>>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>>>> #1  0x00000000006170d9 in qemu_cond_wait (cond=cond at entry=0x1812420
>>>> <qemu_io_proceeded_cond>, mutex=mutex at entry=0x1ba2940
>>>> <qemu_global_mutex>) at qemu-thread-posix.c:113
>>>> #2  0x0000000000682b0f in qemu_tcg_wait_io_event () at
>>>> /home/samaicardi/panda1/qemu/cpus.c:745
>>>> #3  qemu_tcg_cpu_thread_fn (arg=<optimized out>) at
>>>> /home/samaicardi/panda1/qemu/cpus.c:824
>>>> #4  0x00007ffff6f1e6ba in start_thread (arg=0x7fff62c63700) at
>>>> pthread_create.c:333
>>>> #5  0x00007fffe0b803dd in clone () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>>>> gdb-peda$
>>>>
>>>>
>>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>>
>>>> > That same command line and replay work fine for me. Could you provide a
>>>> > backtrace from gdb of the crash?
>>>> >
>>>> > On Wed, Sep 6, 2017 at 4:37 AM, <aicardi at eurecom.fr> wrote:
>>>> >
>>>> >> Thank you for the suggestions.
>>>> >> Yes, I did include 'assert(init_osi_api());' right after
>>>> >> 'panda_require("osi");'. The problem is that the execution segfaults
>>>> at
>>>> the
>>>> >> instruction 'panda_require("osi");', it doesn't reach the following
>>>> >> instructions.
>>>> >>
>>>> >> When I try to execute the following command:
>>>> >> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> -replay
>>>> >> 6bb0dca4-0cea-465c-be82-5a39c1fed8ef -panda osi -panda win7x86intro
>>>> >> -panda asidstory:width=180 -os windows-32-7 -m 1G
>>>> >>
>>>> >> I get:
>>>> >> loading snapshot
>>>> >> Unknown savevm section or instance 'vmmouse' 0
>>>> >> ... done.
>>>> >> opening nondet log for read :   ./6bb0dca4-0cea-465c-be82-5a3
>>>> >> 9c1fed8ef-rr-nondet.log
>>>> >> max_instr = 75766907229
>>>> >> Segmentation fault (core dumped)
>>>> >>
>>>> >>
>>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>> >>
>>>> >> Yep, good advice :) Also, you may want to verify that an existing
>>>> plugin
>>>> >>> that uses osi, such as asidstory, works with the replay.
>>>> >>>
>>>> >>> On Tue, Sep 5, 2017 at 3:03 PM, Bridgey theGeek <
>>>> bridgeythegeek at gmail.com
>>>> >>> >
>>>> >>> wrote:
>>>> >>>
>>>> >>> Off the top of my head, did you include:
>>>> >>>>
>>>> >>>> assert(init_osi_api());
>>>> >>>>
>>>> >>>> immediately after:
>>>> >>>> panda_require("osi");
>>>> >>>> ?
>>>> >>>>
>>>> >>>> For example: https://github.com/panda-re/panda/blob/
>>>> >>>> 060e90693f2ceb30b9c461a5835701e5c463b87a/panda/plugins/
>>>> >>>> asidstory/asidstory.cpp#L359
>>>> >>>> (The same in PANDA 1.0 and 2.0)
>>>> >>>>
>>>> >>>> HTH,
>>>> >>>> Adam
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, 5 Sep 2017 at 10:40 <aicardi at eurecom.fr> wrote:
>>>> >>>>
>>>> >>>> Could you please tell me how to execute those recordings with
>>>> >>>>> qemu-system-x86_64 in 32-bit mode? I've tried to load the 'osi'
>>>> plugin
>>>> >>>>> on several recordings but every time I got a segmentation fault.
>>>> >>>>>
>>>> >>>>> The way I execute them is:
>>>> >>>>> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64
>>>> -replay
>>>> >>>>> <replay_name> -panda syscalls2:profile=windows7_x86 -panda
>>>> >>>>> <my_plugin>:<my_plugin_params> -os windows-32-7 -m 1G
>>>> >>>>>
>>>> >>>>> and in my_plugin I call:
>>>> >>>>> panda_require("osi");
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Thank you in advance,
>>>> >>>>> -samaicardi
>>>> >>>>>
>>>> >>>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>> >>>>>
>>>> >>>>> > The vast majority of those recordings are from Windows 7 32-bit,
>>>> so
>>>> >>>>> osi
>>>> >>>>> > will work on them. They were recorded on an emulated x86_64
>>>> machine
>>>> >>>>> running
>>>> >>>>> > in 32-bit mode.
>>>> >>>>> >
>>>> >>>>> > On Mon, Sep 4, 2017 at 5:10 AM, <aicardi at eurecom.fr> wrote:
>>>> >>>>> >
>>>> >>>>> >> Thank you for the information, it worked.
>>>> >>>>> >>
>>>> >>>>> >> Is it possible to use the 'osi' plugin on those recordings? I've
>>>> seen
>>>> >>>>> the
>>>> >>>>> >> introspection implemented only for windows 32 bit.
>>>> >>>>> >>
>>>> >>>>> >> -samaicardi
>>>> >>>>> >>
>>>> >>>>> >>
>>>> >>>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>> >>>>> >>
>>>> >>>>> >> The malware recordings use 1GB of RAM, so you need to pass "-m
>>>> 1G" on
>>>> >>>>> the
>>>> >>>>> >>> command line when replaying.
>>>> >>>>> >>>
>>>> >>>>> >>> Also you may want to instead use the panda1 repository found
>>>> here:
>>>> >>>>> >>>
>>>> >>>>> >>> https://github.com/moyix/panda
>>>> >>>>> >>>
>>>> >>>>> >>> As I think I've done a couple bugfixes to the old branch since
>>>> we
>>>> >>>>> migrated
>>>> >>>>> >>> the repository to the new version of QEMU.
>>>> >>>>> >>>
>>>> >>>>> >>> -Brendan
>>>> >>>>> >>>
>>>> >>>>> >>> On Thu, Aug 31, 2017 at 11:56 AM, <aicardi at eurecom.fr> wrote:
>>>> >>>>> >>>
>>>> >>>>> >>> Ok I got it, thanks for the explanation.
>>>> >>>>> >>>>
>>>> >>>>> >>>> I have another problem actually, I tried to replay several
>>>> records
>>>> >>>>> (from
>>>> >>>>> >>>> http://panda.gtisc.gatech.edu/malrec/) with the
>>>> qemu-system-x86_64
>>>> >>>>> >>>> compiled from the branch called 'panda1' that I found here:
>>>> >>>>> >>>> https://github.com/panda-re/panda/tree/panda1
>>>> >>>>> >>>> I always get the following error:
>>>> >>>>> >>>> $> ~/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>>>> >>>>> >>>> logs/rr/7d114620-3e3c-4193-96ce-4689fd9efde3
>>>> >>>>> >>>>
>>>> >>>>> >>>> (process:1475): GLib-WARNING **:
>>>> /build/glib2.0-prJhLS/glib2.0-
>>>> >>>>> >>>> 2.48.2/./glib/gmem.c:483:
>>>> >>>>> >>>> custom memory allocation vtable not supported
>>>> >>>>> >>>> loading snapshot
>>>> >>>>> >>>> Block expected 134217728, found 1073741824, total 1082589184,
>>>> >>>>> system
>>>> >>>>> >>>> total
>>>> >>>>> >>>> 143065088
>>>> >>>>> >>>> qemu: warning: error while loading state for instance 0x0 of
>>>> device
>>>> >>>>> 'ram'
>>>> >>>>> >>>> qemu-system-x86_64: Error -22 while loading VM state
>>>> >>>>> >>>> ... done.
>>>> >>>>> >>>> opening nondet log for read :   logs/rr/7d114620-3e3c-4193-96
>>>> >>>>> >>>> ce-4689fd9efde3-rr-nondet.log
>>>> >>>>> >>>> Infinite loop detected during replay, aborting.
>>>> >>>>> >>>> {guest_instr_count=0 pc=0x0000fff0, secondary=0x00000000}
>>>> >>>>> >>>> 7d114620-3e3c-4193-96ce-4689fd9efde3:           0 (  0.00%)
>>>> >>>>> instrs.
>>>> >>>>> >>>> 1.00 sec.  0.03 GB ram.
>>>> >>>>> >>>> total_instr in replay: 15418486377 <(541)%20848-6377>
>>>> >>>>>
>>>> >>>>> >>>> ERROR: replay failed!
>>>> >>>>> >>>> Time taken was: 0 seconds.
>>>> >>>>> >>>> max_queue_len = 1
>>>> >>>>> >>>> 0 items on recycle list, 0 bytes total
>>>> >>>>> >>>> ERROR: replay failed!
>>>> >>>>> >>>> Aborted (core dumped)
>>>> >>>>> >>>>
>>>> >>>>> >>>> Do you possibly know why every record seems to generate an
>>>> infinite
>>>> >>>>> loop?
>>>> >>>>> >>>>
>>>> >>>>> >>>> Thanks in advance,
>>>> >>>>> >>>> samaicardi
>>>> >>>>> >>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>> >>>>> >>>>
>>>> >>>>> >>>> Unfortunately the new version is unlikely to ever be able to
>>>> replay
>>>> >>>>> old
>>>> >>>>> >>>>
>>>> >>>>> >>>>> recordings; too much in QEMU has changed, most notably the
>>>> >>>>> underlying
>>>> >>>>> >>>>> default machine model (and hence the set of devices included
>>>> in
>>>> >>>>> the
>>>> >>>>> >>>>> snapshot). We also took the opportunity to change some of the
>>>> >>>>> >>>>> record/replay
>>>> >>>>> >>>>> log entry types to better match QEMU's new memory API.
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> It is frustrating, since we have 91,000 malware recordings
>>>> now
>>>> and
>>>> >>>>> it
>>>> >>>>> >>>>> would
>>>> >>>>> >>>>> be cool to use them in panda2, but for now malware-related
>>>> work
>>>> >>>>> has
>>>> >>>>> to
>>>> >>>>> >>>>> use
>>>> >>>>> >>>>> panda1. I will be switching malrec over to panda2 as soon as
>>>> I
>>>> >>>>> have
>>>> >>>>> some
>>>> >>>>> >>>>> free time, though.
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> -Brendan
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> On Thu, Aug 31, 2017 at 4:50 AM, <aicardi at eurecom.fr> wrote:
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> Hello everyone,
>>>> >>>>> >>>>>
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> I am writing a plugin for the new version of panda
>>>> >>>>> >>>>>> (https://github.com/panda-re/panda) and I would like to
>>>> test
>>>> it
>>>> >>>>> with
>>>> >>>>> >>>>>> several malware records that can be found here:
>>>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> I followed the guidelines explained here:
>>>> >>>>> >>>>>> https://irfanulhaq.info/2015/12/09/replay-panda-malware-
>>>> >>>>> recordings/
>>>> >>>>> >>>>>> but I'm having troubles in starting the replays.
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> When I try to execute one of those records I get the
>>>> following
>>>> >>>>> error
>>>> >>>>> >>>>>> message:
>>>> >>>>> >>>>>> $> ~/panda2/x86_64-softmmu/qemu-system-x86_64 -replay
>>>> >>>>> >>>>>> ~/replays/malrec/logs/rr/bb67f
>>>> d7e-7baa-437d-9333-9999b15f5fde
>>>> >>>>> >>>>>> > loading snapshot
>>>> >>>>> >>>>>> > qemu-system-x86_64: Unsupported migration stream version
>>>> >>>>> >>>>>> > Failed to load vmstate
>>>> >>>>> >>>>>> > Failed to start replay
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> If I understood it properly, the 'problem' of those records
>>>> is
>>>> >>>>> that
>>>> >>>>> >>>>>> they have been recorded starting from one of the snapshots
>>>> that
>>>> >>>>> can be
>>>> >>>>> >>>>>> found here:
>>>> http://panda.gtisc.gatech.edu/malrec/rr/references/
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> These snapshots were taken using the old version of panda
>>>> >>>>> >>>>>> (https://github.com/moyix/panda).
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> By analyzing the code of the new panda
>>>> >>>>> (include/migration/migration.h)
>>>> >>>>> >>>>>> I saw that there's the following line:
>>>> >>>>> >>>>>> #define QEMU_VM_FILE_VERSION         0x00000003
>>>> >>>>> >>>>>> which is different from what was declared in the old panda
>>>> >>>>> >>>>>> (qemu/savevm.c):
>>>> >>>>> >>>>>> #define QEMU_VM_SECTION_FULL         0x04
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> That difference is causing the error I am getting and I may
>>>> infer
>>>> >>>>> >>>>>> there are other differences between the two versions (for
>>>> what
>>>> >>>>> >>>>>> concerns the procedure of saving a snapshot).
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> My question is, since the two versions of panda take
>>>> snapshots in
>>>> >>>>> >>>>>> different ways (they write different metadata I guess), is
>>>> there
>>>> >>>>> a
>>>> >>>>> way
>>>> >>>>> >>>>>> to replay records (from http://panda.gtisc.gatech.edu/
>>>> malrec/
>>>> )
>>>> >>>>> with
>>>> >>>>> >>>>>> the new version of panda?
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> Or, is it possible to 'patch' the vm snapshots (from
>>>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/rr/references/) to
>>>> make
>>>> >>>>> them
>>>> >>>>> work
>>>> >>>>> >>>>>> with the new version of panda?
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> Thank you in advance for any suggestions you may have!
>>>> >>>>> >>>>>> samaicardi
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> ------------------------------
>>>> ------------------------------
>>>> >>>>> >>>>>> -------------------
>>>> >>>>> >>>>>> This message was sent using EURECOM Webmail:
>>>> >>>>> http://webmail.eurecom.fr
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>> _______________________________________________
>>>> >>>>> >>>>>> panda-users mailing list
>>>> >>>>> >>>>>> panda-users at mit.edu
>>>> >>>>> >>>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>>
>>>> >>>>> >>>>>
>>>> >>>>> >>>>> --
>>>> >>>>> >>>>> Brendan Dolan-Gavitt
>>>> >>>>> >>>>> Assistant Professor, Department of Computer Science and
>>>> >>>>> Engineering
>>>> >>>>> >>>>> NYU Tandon School of Engineering
>>>> >>>>> >>>>>
>>>> >>>>> >>>>>
>>>> >>>>> >>>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>> ------------------------------------------------------------
>>>> >>>>> >>>> -------------------
>>>> >>>>> >>>> This message was sent using EURECOM Webmail:
>>>> >>>>> http://webmail.eurecom.fr
>>>> >>>>> >>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>>
>>>> >>>>> >>>
>>>> >>>>> >>> --
>>>> >>>>> >>> Brendan Dolan-Gavitt
>>>> >>>>> >>> Assistant Professor, Department of Computer Science and
>>>> Engineering
>>>> >>>>> >>> NYU Tandon School of Engineering
>>>> >>>>> >>>
>>>> >>>>> >>>
>>>> >>>>> >>
>>>> >>>>> >>
>>>> >>>>> >> ------------------------------------------------------------
>>>> >>>>> >> -------------------
>>>> >>>>> >> This message was sent using EURECOM Webmail:
>>>> >>>>> http://webmail.eurecom.fr
>>>> >>>>> >>
>>>> >>>>> >>
>>>> >>>>> >
>>>> >>>>> >
>>>> >>>>> > --
>>>> >>>>> > Brendan Dolan-Gavitt
>>>> >>>>> > Assistant Professor, Department of Computer Science and
>>>> Engineering
>>>> >>>>> > NYU Tandon School of Engineering
>>>> >>>>> >
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> ------------------------------------------------------------
>>>> >>>>> -------------------
>>>> >>>>> This message was sent using EURECOM Webmail:
>>>> http://webmail.eurecom.fr
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> _______________________________________________
>>>> >>>>> panda-users mailing list
>>>> >>>>> panda-users at mit.edu
>>>> >>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>>> >>>>>
>>>> >>>>>
>>>> >>>> _______________________________________________
>>>> >>>> panda-users mailing list
>>>> >>>> panda-users at mit.edu
>>>> >>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>> --
>>>> >>> Brendan Dolan-Gavitt
>>>> >>> Assistant Professor, Department of Computer Science and Engineering
>>>> >>> NYU Tandon School of Engineering
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >> ------------------------------------------------------------
>>>> >> -------------------
>>>> >> This message was sent using EURECOM Webmail:
>>>> http://webmail.eurecom.fr
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> > --
>>>> > Brendan Dolan-Gavitt
>>>> > Assistant Professor, Department of Computer Science and Engineering
>>>> > NYU Tandon School of Engineering
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> -------------------
>>>> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>>>
>>>> --
>>>>
>>> Brendan Dolan-Gavitt
>>> Assistant Professor, Department of Computer Science and Engineering
>>> NYU Tandon School of Engineering
>>>
>>>
>>
>>
>> ------------------------------------------------------------
>> -------------------
>> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>
>>
>
>
> --
> Brendan Dolan-Gavitt
> Assistant Professor, Department of Computer Science and Engineering
> NYU Tandon School of Engineering
>



-------------------------------------------------------------------------------
This message was sent using EURECOM Webmail: http://webmail.eurecom.fr

More information about the panda-users mailing list