[panda-users] Replay of legacy records

[Brendan Dolan-Gavitt]

Hmm, that code seems to be in the osi "proc events" functionality,
which should be disabled by default (it was never developed to a
completely stable state). Did you change the build process to enable
it somehow? (e.g., by adding -DOSI_PROC_EVENTS to the build flags)

-Brendan

On Wed, Sep 6, 2017 at 11:47 AM, <aicardi at eurecom.fr> wrote:

> Oh sorry for that, I didn't realize it was another problem. Here is the
> actual backtrace (I hope):
>
> Stopped reason: SIGSEGV
> ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>, ps=0x0,
> in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at
> osi_proc_events.cpp:78
> 78              for (unsigned int i=0; i<ps->num; i++) {
> gdb-peda$ bt
> #0  ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>, ps=0x0,
> in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at
> osi_proc_events.cpp:78
> #1  0x00007fffd6c8029d in procstate_update (ps=<optimized out>, in=in at entry=0x7fff62c60298,
> out=out at entry=0x7fff62c602a0) at osi_proc_events.cpp:141
> #2  0x00007fffd6c7f57d in vmi_pgd_changed (env=0x1c53660,
> oldval=<optimized out>, newval=<optimized out>) at os_intro.c:120
> #3  0x00000000006a3697 in cpu_x86_update_cr3 (env=0x1c53660,
> new_cr3=0x185000) at /home/samaicardi/panda1/qemu/target-i386/helper.c:526
> #4  0x00007fffa46fbf03 in ?? ()
> #5  0x00000000ffffffff in ?? ()
> #6  0x00007ffff6f24360 in pthread_cond_wait@@GLIBC_2.3.2 () at
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
> #7  0x0000000000000000 in ?? ()
>
> gdb-peda$
>
>
> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>
> You need to do "handle SIG38 nostop noprint" first -- otherwise QEMU's
>> alarm timer will cause gdb to stop constantly.
>>
>> On Wed, Sep 6, 2017 at 11:05 AM <aicardi at eurecom.fr> wrote:
>>
>> Yes sure:
>>>
>>> Stopped reason: SIG38
>>> pthread_cond_wait@@GLIBC_2.3.2 () at
>>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>>> 185     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such
>>> file or directory.
>>> gdb-peda$ bt
>>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>>> #1  0x00000000006170d9 in qemu_cond_wait (cond=cond at entry=0x1812420
>>> <qemu_io_proceeded_cond>, mutex=mutex at entry=0x1ba2940
>>> <qemu_global_mutex>) at qemu-thread-posix.c:113
>>> #2  0x0000000000682b0f in qemu_tcg_wait_io_event () at
>>> /home/samaicardi/panda1/qemu/cpus.c:745
>>> #3  qemu_tcg_cpu_thread_fn (arg=<optimized out>) at
>>> /home/samaicardi/panda1/qemu/cpus.c:824
>>> #4  0x00007ffff6f1e6ba in start_thread (arg=0x7fff62c63700) at
>>> pthread_create.c:333
>>> #5  0x00007fffe0b803dd in clone () at
>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>>> gdb-peda$
>>>
>>>
>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>
>>> > That same command line and replay work fine for me. Could you provide a
>>> > backtrace from gdb of the crash?
>>> >
>>> > On Wed, Sep 6, 2017 at 4:37 AM, <aicardi at eurecom.fr> wrote:
>>> >
>>> >> Thank you for the suggestions.
>>> >> Yes, I did include 'assert(init_osi_api());' right after
>>> >> 'panda_require("osi");'. The problem is that the execution segfaults
>>> at
>>> the
>>> >> instruction 'panda_require("osi");', it doesn't reach the following
>>> >> instructions.
>>> >>
>>> >> When I try to execute the following command:
>>> >> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64
>>> -replay
>>> >> 6bb0dca4-0cea-465c-be82-5a39c1fed8ef -panda osi -panda win7x86intro
>>> >> -panda asidstory:width=180 -os windows-32-7 -m 1G
>>> >>
>>> >> I get:
>>> >> loading snapshot
>>> >> Unknown savevm section or instance 'vmmouse' 0
>>> >> ... done.
>>> >> opening nondet log for read :   ./6bb0dca4-0cea-465c-be82-5a3
>>> >> 9c1fed8ef-rr-nondet.log
>>> >> max_instr = 75766907229
>>> >> Segmentation fault (core dumped)
>>> >>
>>> >>
>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>
>>> >> Yep, good advice :) Also, you may want to verify that an existing
>>> plugin
>>> >>> that uses osi, such as asidstory, works with the replay.
>>> >>>
>>> >>> On Tue, Sep 5, 2017 at 3:03 PM, Bridgey theGeek <
>>> bridgeythegeek at gmail.com
>>> >>> >
>>> >>> wrote:
>>> >>>
>>> >>> Off the top of my head, did you include:
>>> >>>>
>>> >>>> assert(init_osi_api());
>>> >>>>
>>> >>>> immediately after:
>>> >>>> panda_require("osi");
>>> >>>> ?
>>> >>>>
>>> >>>> For example: https://github.com/panda-re/panda/blob/
>>> >>>> 060e90693f2ceb30b9c461a5835701e5c463b87a/panda/plugins/
>>> >>>> asidstory/asidstory.cpp#L359
>>> >>>> (The same in PANDA 1.0 and 2.0)
>>> >>>>
>>> >>>> HTH,
>>> >>>> Adam
>>> >>>>
>>> >>>>
>>> >>>> On Tue, 5 Sep 2017 at 10:40 <aicardi at eurecom.fr> wrote:
>>> >>>>
>>> >>>> Could you please tell me how to execute those recordings with
>>> >>>>> qemu-system-x86_64 in 32-bit mode? I've tried to load the 'osi'
>>> plugin
>>> >>>>> on several recordings but every time I got a segmentation fault.
>>> >>>>>
>>> >>>>> The way I execute them is:
>>> >>>>> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64
>>> -replay
>>> >>>>> <replay_name> -panda syscalls2:profile=windows7_x86 -panda
>>> >>>>> <my_plugin>:<my_plugin_params> -os windows-32-7 -m 1G
>>> >>>>>
>>> >>>>> and in my_plugin I call:
>>> >>>>> panda_require("osi");
>>> >>>>>
>>> >>>>>
>>> >>>>> Thank you in advance,
>>> >>>>> -samaicardi
>>> >>>>>
>>> >>>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>>>>
>>> >>>>> > The vast majority of those recordings are from Windows 7 32-bit,
>>> so
>>> >>>>> osi
>>> >>>>> > will work on them. They were recorded on an emulated x86_64
>>> machine
>>> >>>>> running
>>> >>>>> > in 32-bit mode.
>>> >>>>> >
>>> >>>>> > On Mon, Sep 4, 2017 at 5:10 AM, <aicardi at eurecom.fr> wrote:
>>> >>>>> >
>>> >>>>> >> Thank you for the information, it worked.
>>> >>>>> >>
>>> >>>>> >> Is it possible to use the 'osi' plugin on those recordings? I've
>>> seen
>>> >>>>> the
>>> >>>>> >> introspection implemented only for windows 32 bit.
>>> >>>>> >>
>>> >>>>> >> -samaicardi
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>>>> >>
>>> >>>>> >> The malware recordings use 1GB of RAM, so you need to pass "-m
>>> 1G" on
>>> >>>>> the
>>> >>>>> >>> command line when replaying.
>>> >>>>> >>>
>>> >>>>> >>> Also you may want to instead use the panda1 repository found
>>> here:
>>> >>>>> >>>
>>> >>>>> >>> https://github.com/moyix/panda
>>> >>>>> >>>
>>> >>>>> >>> As I think I've done a couple bugfixes to the old branch since
>>> we
>>> >>>>> migrated
>>> >>>>> >>> the repository to the new version of QEMU.
>>> >>>>> >>>
>>> >>>>> >>> -Brendan
>>> >>>>> >>>
>>> >>>>> >>> On Thu, Aug 31, 2017 at 11:56 AM, <aicardi at eurecom.fr> wrote:
>>> >>>>> >>>
>>> >>>>> >>> Ok I got it, thanks for the explanation.
>>> >>>>> >>>>
>>> >>>>> >>>> I have another problem actually, I tried to replay several
>>> records
>>> >>>>> (from
>>> >>>>> >>>> http://panda.gtisc.gatech.edu/malrec/) with the
>>> qemu-system-x86_64
>>> >>>>> >>>> compiled from the branch called 'panda1' that I found here:
>>> >>>>> >>>> https://github.com/panda-re/panda/tree/panda1
>>> >>>>> >>>> I always get the following error:
>>> >>>>> >>>> $> ~/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>>> >>>>> >>>> logs/rr/7d114620-3e3c-4193-96ce-4689fd9efde3
>>> >>>>> >>>>
>>> >>>>> >>>> (process:1475): GLib-WARNING **:
>>> /build/glib2.0-prJhLS/glib2.0-
>>> >>>>> >>>> 2.48.2/./glib/gmem.c:483:
>>> >>>>> >>>> custom memory allocation vtable not supported
>>> >>>>> >>>> loading snapshot
>>> >>>>> >>>> Block expected 134217728, found 1073741824, total 1082589184,
>>> >>>>> system
>>> >>>>> >>>> total
>>> >>>>> >>>> 143065088
>>> >>>>> >>>> qemu: warning: error while loading state for instance 0x0 of
>>> device
>>> >>>>> 'ram'
>>> >>>>> >>>> qemu-system-x86_64: Error -22 while loading VM state
>>> >>>>> >>>> ... done.
>>> >>>>> >>>> opening nondet log for read :   logs/rr/7d114620-3e3c-4193-96
>>> >>>>> >>>> ce-4689fd9efde3-rr-nondet.log
>>> >>>>> >>>> Infinite loop detected during replay, aborting.
>>> >>>>> >>>> {guest_instr_count=0 pc=0x0000fff0, secondary=0x00000000}
>>> >>>>> >>>> 7d114620-3e3c-4193-96ce-4689fd9efde3:           0 (  0.00%)
>>> >>>>> instrs.
>>> >>>>> >>>> 1.00 sec.  0.03 GB ram.
>>> >>>>> >>>> total_instr in replay: 15418486377 <(541)%20848-6377>
>>> >>>>>
>>> >>>>> >>>> ERROR: replay failed!
>>> >>>>> >>>> Time taken was: 0 seconds.
>>> >>>>> >>>> max_queue_len = 1
>>> >>>>> >>>> 0 items on recycle list, 0 bytes total
>>> >>>>> >>>> ERROR: replay failed!
>>> >>>>> >>>> Aborted (core dumped)
>>> >>>>> >>>>
>>> >>>>> >>>> Do you possibly know why every record seems to generate an
>>> infinite
>>> >>>>> loop?
>>> >>>>> >>>>
>>> >>>>> >>>> Thanks in advance,
>>> >>>>> >>>> samaicardi
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>>>> >>>>
>>> >>>>> >>>> Unfortunately the new version is unlikely to ever be able to
>>> replay
>>> >>>>> old
>>> >>>>> >>>>
>>> >>>>> >>>>> recordings; too much in QEMU has changed, most notably the
>>> >>>>> underlying
>>> >>>>> >>>>> default machine model (and hence the set of devices included
>>> in
>>> >>>>> the
>>> >>>>> >>>>> snapshot). We also took the opportunity to change some of the
>>> >>>>> >>>>> record/replay
>>> >>>>> >>>>> log entry types to better match QEMU's new memory API.
>>> >>>>> >>>>>
>>> >>>>> >>>>> It is frustrating, since we have 91,000 malware recordings
>>> now
>>> and
>>> >>>>> it
>>> >>>>> >>>>> would
>>> >>>>> >>>>> be cool to use them in panda2, but for now malware-related
>>> work
>>> >>>>> has
>>> >>>>> to
>>> >>>>> >>>>> use
>>> >>>>> >>>>> panda1. I will be switching malrec over to panda2 as soon as
>>> I
>>> >>>>> have
>>> >>>>> some
>>> >>>>> >>>>> free time, though.
>>> >>>>> >>>>>
>>> >>>>> >>>>> -Brendan
>>> >>>>> >>>>>
>>> >>>>> >>>>> On Thu, Aug 31, 2017 at 4:50 AM, <aicardi at eurecom.fr> wrote:
>>> >>>>> >>>>>
>>> >>>>> >>>>> Hello everyone,
>>> >>>>> >>>>>
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> I am writing a plugin for the new version of panda
>>> >>>>> >>>>>> (https://github.com/panda-re/panda) and I would like to
>>> test
>>> it
>>> >>>>> with
>>> >>>>> >>>>>> several malware records that can be found here:
>>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> I followed the guidelines explained here:
>>> >>>>> >>>>>> https://irfanulhaq.info/2015/12/09/replay-panda-malware-
>>> >>>>> recordings/
>>> >>>>> >>>>>> but I'm having troubles in starting the replays.
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> When I try to execute one of those records I get the
>>> following
>>> >>>>> error
>>> >>>>> >>>>>> message:
>>> >>>>> >>>>>> $> ~/panda2/x86_64-softmmu/qemu-system-x86_64 -replay
>>> >>>>> >>>>>> ~/replays/malrec/logs/rr/bb67f
>>> d7e-7baa-437d-9333-9999b15f5fde
>>> >>>>> >>>>>> > loading snapshot
>>> >>>>> >>>>>> > qemu-system-x86_64: Unsupported migration stream version
>>> >>>>> >>>>>> > Failed to load vmstate
>>> >>>>> >>>>>> > Failed to start replay
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> If I understood it properly, the 'problem' of those records
>>> is
>>> >>>>> that
>>> >>>>> >>>>>> they have been recorded starting from one of the snapshots
>>> that
>>> >>>>> can be
>>> >>>>> >>>>>> found here:
>>> http://panda.gtisc.gatech.edu/malrec/rr/references/
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> These snapshots were taken using the old version of panda
>>> >>>>> >>>>>> (https://github.com/moyix/panda).
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> By analyzing the code of the new panda
>>> >>>>> (include/migration/migration.h)
>>> >>>>> >>>>>> I saw that there's the following line:
>>> >>>>> >>>>>> #define QEMU_VM_FILE_VERSION         0x00000003
>>> >>>>> >>>>>> which is different from what was declared in the old panda
>>> >>>>> >>>>>> (qemu/savevm.c):
>>> >>>>> >>>>>> #define QEMU_VM_SECTION_FULL         0x04
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> That difference is causing the error I am getting and I may
>>> infer
>>> >>>>> >>>>>> there are other differences between the two versions (for
>>> what
>>> >>>>> >>>>>> concerns the procedure of saving a snapshot).
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> My question is, since the two versions of panda take
>>> snapshots in
>>> >>>>> >>>>>> different ways (they write different metadata I guess), is
>>> there
>>> >>>>> a
>>> >>>>> way
>>> >>>>> >>>>>> to replay records (from http://panda.gtisc.gatech.edu/
>>> malrec/
>>> )
>>> >>>>> with
>>> >>>>> >>>>>> the new version of panda?
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> Or, is it possible to 'patch' the vm snapshots (from
>>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/rr/references/) to
>>> make
>>> >>>>> them
>>> >>>>> work
>>> >>>>> >>>>>> with the new version of panda?
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> Thank you in advance for any suggestions you may have!
>>> >>>>> >>>>>> samaicardi
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> ------------------------------
>>> ------------------------------
>>> >>>>> >>>>>> -------------------
>>> >>>>> >>>>>> This message was sent using EURECOM Webmail:
>>> >>>>> http://webmail.eurecom.fr
>>> >>>>> >>>>>>
>>> >>>>> >>>>>>
>>> >>>>> >>>>>> _______________________________________________
>>> >>>>> >>>>>> panda-users mailing list
>>> >>>>> >>>>>> panda-users at mit.edu
>>> >>>>> >>>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>> >>>>> >>>>>>
>>> >>>>> >>>>>>
>>> >>>>> >>>>>>
>>> >>>>> >>>>>
>>> >>>>> >>>>> --
>>> >>>>> >>>>> Brendan Dolan-Gavitt
>>> >>>>> >>>>> Assistant Professor, Department of Computer Science and
>>> >>>>> Engineering
>>> >>>>> >>>>> NYU Tandon School of Engineering
>>> >>>>> >>>>>
>>> >>>>> >>>>>
>>> >>>>> >>>>>
>>> >>>>> >>>>
>>> >>>>> >>>> ------------------------------------------------------------
>>> >>>>> >>>> -------------------
>>> >>>>> >>>> This message was sent using EURECOM Webmail:
>>> >>>>> http://webmail.eurecom.fr
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>>
>>> >>>>> >>>
>>> >>>>> >>> --
>>> >>>>> >>> Brendan Dolan-Gavitt
>>> >>>>> >>> Assistant Professor, Department of Computer Science and
>>> Engineering
>>> >>>>> >>> NYU Tandon School of Engineering
>>> >>>>> >>>
>>> >>>>> >>>
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >> ------------------------------------------------------------
>>> >>>>> >> -------------------
>>> >>>>> >> This message was sent using EURECOM Webmail:
>>> >>>>> http://webmail.eurecom.fr
>>> >>>>> >>
>>> >>>>> >>
>>> >>>>> >
>>> >>>>> >
>>> >>>>> > --
>>> >>>>> > Brendan Dolan-Gavitt
>>> >>>>> > Assistant Professor, Department of Computer Science and
>>> Engineering
>>> >>>>> > NYU Tandon School of Engineering
>>> >>>>> >
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> ------------------------------------------------------------
>>> >>>>> -------------------
>>> >>>>> This message was sent using EURECOM Webmail:
>>> http://webmail.eurecom.fr
>>> >>>>>
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> panda-users mailing list
>>> >>>>> panda-users at mit.edu
>>> >>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>> >>>>>
>>> >>>>>
>>> >>>> _______________________________________________
>>> >>>> panda-users mailing list
>>> >>>> panda-users at mit.edu
>>> >>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> Brendan Dolan-Gavitt
>>> >>> Assistant Professor, Department of Computer Science and Engineering
>>> >>> NYU Tandon School of Engineering
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------
>>> >> -------------------
>>> >> This message was sent using EURECOM Webmail:
>>> http://webmail.eurecom.fr
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Brendan Dolan-Gavitt
>>> > Assistant Professor, Department of Computer Science and Engineering
>>> > NYU Tandon School of Engineering
>>> >
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> -------------------
>>> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>>
>>> --
>>>
>> Brendan Dolan-Gavitt
>> Assistant Professor, Department of Computer Science and Engineering
>> NYU Tandon School of Engineering
>>
>>
>
>
> ------------------------------------------------------------
> -------------------
> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>
>


-- 
Brendan Dolan-Gavitt
Assistant Professor, Department of Computer Science and Engineering
NYU Tandon School of Engineering
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/panda-users/attachments/20170906/065a128e/attachment-0001.html