[panda-users] Replay of legacy records

aicardi@eurecom.fr aicardi at eurecom.fr
Wed Sep 6 11:47:00 EDT 2017 

Oh sorry for that, I didn't realize it was another problem. Here is  
the actual backtrace (I hope):

Stopped reason: SIGSEGV
ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>, ps=0x0,  
in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at  
osi_proc_events.cpp:78
78		for (unsigned int i=0; i<ps->num; i++) {
gdb-peda$ bt
#0  ProcState::update (this=this at entry=0x7fffd6e821d0 <pstate>,  
ps=0x0, in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at  
osi_proc_events.cpp:78
#1  0x00007fffd6c8029d in procstate_update (ps=<optimized out>,  
in=in at entry=0x7fff62c60298, out=out at entry=0x7fff62c602a0) at  
osi_proc_events.cpp:141
#2  0x00007fffd6c7f57d in vmi_pgd_changed (env=0x1c53660,  
oldval=<optimized out>, newval=<optimized out>) at os_intro.c:120
#3  0x00000000006a3697 in cpu_x86_update_cr3 (env=0x1c53660,  
new_cr3=0x185000) at  
/home/samaicardi/panda1/qemu/target-i386/helper.c:526
#4  0x00007fffa46fbf03 in ?? ()
#5  0x00000000ffffffff in ?? ()
#6  0x00007ffff6f24360 in pthread_cond_wait@@GLIBC_2.3.2 () at  
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#7  0x0000000000000000 in ?? ()
gdb-peda$


Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:

> You need to do "handle SIG38 nostop noprint" first -- otherwise QEMU's
> alarm timer will cause gdb to stop constantly.
>
> On Wed, Sep 6, 2017 at 11:05 AM <aicardi at eurecom.fr> wrote:
>
>> Yes sure:
>>
>> Stopped reason: SIG38
>> pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> 185     ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: No such
>> file or directory.
>> gdb-peda$ bt
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1  0x00000000006170d9 in qemu_cond_wait (cond=cond at entry=0x1812420
>> <qemu_io_proceeded_cond>, mutex=mutex at entry=0x1ba2940
>> <qemu_global_mutex>) at qemu-thread-posix.c:113
>> #2  0x0000000000682b0f in qemu_tcg_wait_io_event () at
>> /home/samaicardi/panda1/qemu/cpus.c:745
>> #3  qemu_tcg_cpu_thread_fn (arg=<optimized out>) at
>> /home/samaicardi/panda1/qemu/cpus.c:824
>> #4  0x00007ffff6f1e6ba in start_thread (arg=0x7fff62c63700) at
>> pthread_create.c:333
>> #5  0x00007fffe0b803dd in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
>> gdb-peda$
>>
>>
>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>
>> > That same command line and replay work fine for me. Could you provide a
>> > backtrace from gdb of the crash?
>> >
>> > On Wed, Sep 6, 2017 at 4:37 AM, <aicardi at eurecom.fr> wrote:
>> >
>> >> Thank you for the suggestions.
>> >> Yes, I did include 'assert(init_osi_api());' right after
>> >> 'panda_require("osi");'. The problem is that the execution segfaults at
>> the
>> >> instruction 'panda_require("osi");', it doesn't reach the following
>> >> instructions.
>> >>
>> >> When I try to execute the following command:
>> >> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>> >> 6bb0dca4-0cea-465c-be82-5a39c1fed8ef -panda osi -panda win7x86intro
>> >> -panda asidstory:width=180 -os windows-32-7 -m 1G
>> >>
>> >> I get:
>> >> loading snapshot
>> >> Unknown savevm section or instance 'vmmouse' 0
>> >> ... done.
>> >> opening nondet log for read :   ./6bb0dca4-0cea-465c-be82-5a3
>> >> 9c1fed8ef-rr-nondet.log
>> >> max_instr = 75766907229
>> >> Segmentation fault (core dumped)
>> >>
>> >>
>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>> >>
>> >> Yep, good advice :) Also, you may want to verify that an existing plugin
>> >>> that uses osi, such as asidstory, works with the replay.
>> >>>
>> >>> On Tue, Sep 5, 2017 at 3:03 PM, Bridgey theGeek <
>> bridgeythegeek at gmail.com
>> >>> >
>> >>> wrote:
>> >>>
>> >>> Off the top of my head, did you include:
>> >>>>
>> >>>> assert(init_osi_api());
>> >>>>
>> >>>> immediately after:
>> >>>> panda_require("osi");
>> >>>> ?
>> >>>>
>> >>>> For example: https://github.com/panda-re/panda/blob/
>> >>>> 060e90693f2ceb30b9c461a5835701e5c463b87a/panda/plugins/
>> >>>> asidstory/asidstory.cpp#L359
>> >>>> (The same in PANDA 1.0 and 2.0)
>> >>>>
>> >>>> HTH,
>> >>>> Adam
>> >>>>
>> >>>>
>> >>>> On Tue, 5 Sep 2017 at 10:40 <aicardi at eurecom.fr> wrote:
>> >>>>
>> >>>> Could you please tell me how to execute those recordings with
>> >>>>> qemu-system-x86_64 in 32-bit mode? I've tried to load the 'osi'
>> plugin
>> >>>>> on several recordings but every time I got a segmentation fault.
>> >>>>>
>> >>>>> The way I execute them is:
>> >>>>> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64
>> -replay
>> >>>>> <replay_name> -panda syscalls2:profile=windows7_x86 -panda
>> >>>>> <my_plugin>:<my_plugin_params> -os windows-32-7 -m 1G
>> >>>>>
>> >>>>> and in my_plugin I call:
>> >>>>> panda_require("osi");
>> >>>>>
>> >>>>>
>> >>>>> Thank you in advance,
>> >>>>> -samaicardi
>> >>>>>
>> >>>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>> >>>>>
>> >>>>> > The vast majority of those recordings are from Windows 7 32-bit, so
>> >>>>> osi
>> >>>>> > will work on them. They were recorded on an emulated x86_64 machine
>> >>>>> running
>> >>>>> > in 32-bit mode.
>> >>>>> >
>> >>>>> > On Mon, Sep 4, 2017 at 5:10 AM, <aicardi at eurecom.fr> wrote:
>> >>>>> >
>> >>>>> >> Thank you for the information, it worked.
>> >>>>> >>
>> >>>>> >> Is it possible to use the 'osi' plugin on those recordings? I've
>> seen
>> >>>>> the
>> >>>>> >> introspection implemented only for windows 32 bit.
>> >>>>> >>
>> >>>>> >> -samaicardi
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>> >>>>> >>
>> >>>>> >> The malware recordings use 1GB of RAM, so you need to pass "-m
>> 1G" on
>> >>>>> the
>> >>>>> >>> command line when replaying.
>> >>>>> >>>
>> >>>>> >>> Also you may want to instead use the panda1 repository found
>> here:
>> >>>>> >>>
>> >>>>> >>> https://github.com/moyix/panda
>> >>>>> >>>
>> >>>>> >>> As I think I've done a couple bugfixes to the old branch since we
>> >>>>> migrated
>> >>>>> >>> the repository to the new version of QEMU.
>> >>>>> >>>
>> >>>>> >>> -Brendan
>> >>>>> >>>
>> >>>>> >>> On Thu, Aug 31, 2017 at 11:56 AM, <aicardi at eurecom.fr> wrote:
>> >>>>> >>>
>> >>>>> >>> Ok I got it, thanks for the explanation.
>> >>>>> >>>>
>> >>>>> >>>> I have another problem actually, I tried to replay several
>> records
>> >>>>> (from
>> >>>>> >>>> http://panda.gtisc.gatech.edu/malrec/) with the
>> qemu-system-x86_64
>> >>>>> >>>> compiled from the branch called 'panda1' that I found here:
>> >>>>> >>>> https://github.com/panda-re/panda/tree/panda1
>> >>>>> >>>> I always get the following error:
>> >>>>> >>>> $> ~/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>> >>>>> >>>> logs/rr/7d114620-3e3c-4193-96ce-4689fd9efde3
>> >>>>> >>>>
>> >>>>> >>>> (process:1475): GLib-WARNING **:  /build/glib2.0-prJhLS/glib2.0-
>> >>>>> >>>> 2.48.2/./glib/gmem.c:483:
>> >>>>> >>>> custom memory allocation vtable not supported
>> >>>>> >>>> loading snapshot
>> >>>>> >>>> Block expected 134217728, found 1073741824, total 1082589184,
>> >>>>> system
>> >>>>> >>>> total
>> >>>>> >>>> 143065088
>> >>>>> >>>> qemu: warning: error while loading state for instance 0x0 of
>> device
>> >>>>> 'ram'
>> >>>>> >>>> qemu-system-x86_64: Error -22 while loading VM state
>> >>>>> >>>> ... done.
>> >>>>> >>>> opening nondet log for read :   logs/rr/7d114620-3e3c-4193-96
>> >>>>> >>>> ce-4689fd9efde3-rr-nondet.log
>> >>>>> >>>> Infinite loop detected during replay, aborting.
>> >>>>> >>>> {guest_instr_count=0 pc=0x0000fff0, secondary=0x00000000}
>> >>>>> >>>> 7d114620-3e3c-4193-96ce-4689fd9efde3:           0 (  0.00%)
>> >>>>> instrs.
>> >>>>> >>>> 1.00 sec.  0.03 GB ram.
>> >>>>> >>>> total_instr in replay: 15418486377 <(541)%20848-6377>
>> >>>>>
>> >>>>> >>>> ERROR: replay failed!
>> >>>>> >>>> Time taken was: 0 seconds.
>> >>>>> >>>> max_queue_len = 1
>> >>>>> >>>> 0 items on recycle list, 0 bytes total
>> >>>>> >>>> ERROR: replay failed!
>> >>>>> >>>> Aborted (core dumped)
>> >>>>> >>>>
>> >>>>> >>>> Do you possibly know why every record seems to generate an
>> infinite
>> >>>>> loop?
>> >>>>> >>>>
>> >>>>> >>>> Thanks in advance,
>> >>>>> >>>> samaicardi
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>> >>>>> >>>>
>> >>>>> >>>> Unfortunately the new version is unlikely to ever be able to
>> replay
>> >>>>> old
>> >>>>> >>>>
>> >>>>> >>>>> recordings; too much in QEMU has changed, most notably the
>> >>>>> underlying
>> >>>>> >>>>> default machine model (and hence the set of devices included in
>> >>>>> the
>> >>>>> >>>>> snapshot). We also took the opportunity to change some of the
>> >>>>> >>>>> record/replay
>> >>>>> >>>>> log entry types to better match QEMU's new memory API.
>> >>>>> >>>>>
>> >>>>> >>>>> It is frustrating, since we have 91,000 malware recordings now
>> and
>> >>>>> it
>> >>>>> >>>>> would
>> >>>>> >>>>> be cool to use them in panda2, but for now malware-related work
>> >>>>> has
>> >>>>> to
>> >>>>> >>>>> use
>> >>>>> >>>>> panda1. I will be switching malrec over to panda2 as soon as I
>> >>>>> have
>> >>>>> some
>> >>>>> >>>>> free time, though.
>> >>>>> >>>>>
>> >>>>> >>>>> -Brendan
>> >>>>> >>>>>
>> >>>>> >>>>> On Thu, Aug 31, 2017 at 4:50 AM, <aicardi at eurecom.fr> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>> Hello everyone,
>> >>>>> >>>>>
>> >>>>> >>>>>>
>> >>>>> >>>>>> I am writing a plugin for the new version of panda
>> >>>>> >>>>>> (https://github.com/panda-re/panda) and I would like to test
>> it
>> >>>>> with
>> >>>>> >>>>>> several malware records that can be found here:
>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/
>> >>>>> >>>>>>
>> >>>>> >>>>>> I followed the guidelines explained here:
>> >>>>> >>>>>> https://irfanulhaq.info/2015/12/09/replay-panda-malware-
>> >>>>> recordings/
>> >>>>> >>>>>> but I'm having troubles in starting the replays.
>> >>>>> >>>>>>
>> >>>>> >>>>>> When I try to execute one of those records I get the following
>> >>>>> error
>> >>>>> >>>>>> message:
>> >>>>> >>>>>> $> ~/panda2/x86_64-softmmu/qemu-system-x86_64 -replay
>> >>>>> >>>>>> ~/replays/malrec/logs/rr/bb67fd7e-7baa-437d-9333-9999b15f5fde
>> >>>>> >>>>>> > loading snapshot
>> >>>>> >>>>>> > qemu-system-x86_64: Unsupported migration stream version
>> >>>>> >>>>>> > Failed to load vmstate
>> >>>>> >>>>>> > Failed to start replay
>> >>>>> >>>>>>
>> >>>>> >>>>>> If I understood it properly, the 'problem' of those records is
>> >>>>> that
>> >>>>> >>>>>> they have been recorded starting from one of the snapshots
>> that
>> >>>>> can be
>> >>>>> >>>>>> found here:
>> http://panda.gtisc.gatech.edu/malrec/rr/references/
>> >>>>> >>>>>>
>> >>>>> >>>>>> These snapshots were taken using the old version of panda
>> >>>>> >>>>>> (https://github.com/moyix/panda).
>> >>>>> >>>>>>
>> >>>>> >>>>>> By analyzing the code of the new panda
>> >>>>> (include/migration/migration.h)
>> >>>>> >>>>>> I saw that there's the following line:
>> >>>>> >>>>>> #define QEMU_VM_FILE_VERSION         0x00000003
>> >>>>> >>>>>> which is different from what was declared in the old panda
>> >>>>> >>>>>> (qemu/savevm.c):
>> >>>>> >>>>>> #define QEMU_VM_SECTION_FULL         0x04
>> >>>>> >>>>>>
>> >>>>> >>>>>> That difference is causing the error I am getting and I may
>> infer
>> >>>>> >>>>>> there are other differences between the two versions (for what
>> >>>>> >>>>>> concerns the procedure of saving a snapshot).
>> >>>>> >>>>>>
>> >>>>> >>>>>> My question is, since the two versions of panda take
>> snapshots in
>> >>>>> >>>>>> different ways (they write different metadata I guess), is
>> there
>> >>>>> a
>> >>>>> way
>> >>>>> >>>>>> to replay records (from http://panda.gtisc.gatech.edu/malrec/
>> )
>> >>>>> with
>> >>>>> >>>>>> the new version of panda?
>> >>>>> >>>>>>
>> >>>>> >>>>>> Or, is it possible to 'patch' the vm snapshots (from
>> >>>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/rr/references/) to make
>> >>>>> them
>> >>>>> work
>> >>>>> >>>>>> with the new version of panda?
>> >>>>> >>>>>>
>> >>>>> >>>>>> Thank you in advance for any suggestions you may have!
>> >>>>> >>>>>> samaicardi
>> >>>>> >>>>>>
>> >>>>> >>>>>> ------------------------------------------------------------
>> >>>>> >>>>>> -------------------
>> >>>>> >>>>>> This message was sent using EURECOM Webmail:
>> >>>>> http://webmail.eurecom.fr
>> >>>>> >>>>>>
>> >>>>> >>>>>>
>> >>>>> >>>>>> _______________________________________________
>> >>>>> >>>>>> panda-users mailing list
>> >>>>> >>>>>> panda-users at mit.edu
>> >>>>> >>>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>> >>>>> >>>>>>
>> >>>>> >>>>>>
>> >>>>> >>>>>>
>> >>>>> >>>>>
>> >>>>> >>>>> --
>> >>>>> >>>>> Brendan Dolan-Gavitt
>> >>>>> >>>>> Assistant Professor, Department of Computer Science and
>> >>>>> Engineering
>> >>>>> >>>>> NYU Tandon School of Engineering
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>
>> >>>>> >>>> ------------------------------------------------------------
>> >>>>> >>>> -------------------
>> >>>>> >>>> This message was sent using EURECOM Webmail:
>> >>>>> http://webmail.eurecom.fr
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>> >>> --
>> >>>>> >>> Brendan Dolan-Gavitt
>> >>>>> >>> Assistant Professor, Department of Computer Science and
>> Engineering
>> >>>>> >>> NYU Tandon School of Engineering
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> ------------------------------------------------------------
>> >>>>> >> -------------------
>> >>>>> >> This message was sent using EURECOM Webmail:
>> >>>>> http://webmail.eurecom.fr
>> >>>>> >>
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > Brendan Dolan-Gavitt
>> >>>>> > Assistant Professor, Department of Computer Science and Engineering
>> >>>>> > NYU Tandon School of Engineering
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> ------------------------------------------------------------
>> >>>>> -------------------
>> >>>>> This message was sent using EURECOM Webmail:
>> http://webmail.eurecom.fr
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> panda-users mailing list
>> >>>>> panda-users at mit.edu
>> >>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>> >>>>>
>> >>>>>
>> >>>> _______________________________________________
>> >>>> panda-users mailing list
>> >>>> panda-users at mit.edu
>> >>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Brendan Dolan-Gavitt
>> >>> Assistant Professor, Department of Computer Science and Engineering
>> >>> NYU Tandon School of Engineering
>> >>>
>> >>>
>> >>
>> >>
>> >> ------------------------------------------------------------
>> >> -------------------
>> >> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>> >>
>> >>
>> >
>> >
>> > --
>> > Brendan Dolan-Gavitt
>> > Assistant Professor, Department of Computer Science and Engineering
>> > NYU Tandon School of Engineering
>> >
>>
>>
>>
>>
>> -------------------------------------------------------------------------------
>> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>
>> --
> Brendan Dolan-Gavitt
> Assistant Professor, Department of Computer Science and Engineering
> NYU Tandon School of Engineering
>



-------------------------------------------------------------------------------
This message was sent using EURECOM Webmail: http://webmail.eurecom.fr

More information about the panda-users mailing list