[panda-users] Replay of legacy records

Wed Sep 6 04:37:21 EDT 2017

Thank you for the suggestions.
Yes, I did include 'assert(init_osi_api());' right after  
'panda_require("osi");'. The problem is that the execution segfaults  
at the instruction 'panda_require("osi");', it doesn't reach the  
following instructions.

When I try to execute the following command:
/home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay  
6bb0dca4-0cea-465c-be82-5a39c1fed8ef -panda osi -panda win7x86intro  
-panda asidstory:width=180 -os windows-32-7 -m 1G

I get:
loading snapshot
Unknown savevm section or instance 'vmmouse' 0
... done.
opening nondet log for read  
:	./6bb0dca4-0cea-465c-be82-5a39c1fed8ef-rr-nondet.log
max_instr = 75766907229
Segmentation fault (core dumped)

Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:

> Yep, good advice :) Also, you may want to verify that an existing plugin
> that uses osi, such as asidstory, works with the replay.
>
> On Tue, Sep 5, 2017 at 3:03 PM, Bridgey theGeek <bridgeythegeek at gmail.com>
> wrote:
>
>> Off the top of my head, did you include:
>>
>> assert(init_osi_api());
>>
>> immediately after:
>> panda_require("osi");
>> ?
>>
>> For example: https://github.com/panda-re/panda/blob/
>> 060e90693f2ceb30b9c461a5835701e5c463b87a/panda/plugins/
>> asidstory/asidstory.cpp#L359
>> (The same in PANDA 1.0 and 2.0)
>>
>> HTH,
>> Adam
>>
>>
>> On Tue, 5 Sep 2017 at 10:40 <aicardi at eurecom.fr> wrote:
>>
>>> Could you please tell me how to execute those recordings with
>>> qemu-system-x86_64 in 32-bit mode? I've tried to load the 'osi' plugin
>>> on several recordings but every time I got a segmentation fault.
>>>
>>> The way I execute them is:
>>> /home/samaicardi/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>>> <replay_name> -panda syscalls2:profile=windows7_x86 -panda
>>> <my_plugin>:<my_plugin_params> -os windows-32-7 -m 1G
>>>
>>> and in my_plugin I call:
>>> panda_require("osi");
>>>
>>>
>>> Thank you in advance,
>>> -samaicardi
>>>
>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>>
>>> > The vast majority of those recordings are from Windows 7 32-bit, so osi
>>> > will work on them. They were recorded on an emulated x86_64 machine
>>> running
>>> > in 32-bit mode.
>>> >
>>> > On Mon, Sep 4, 2017 at 5:10 AM, <aicardi at eurecom.fr> wrote:
>>> >
>>> >> Thank you for the information, it worked.
>>> >>
>>> >> Is it possible to use the 'osi' plugin on those recordings? I've seen
>>> the
>>> >> introspection implemented only for windows 32 bit.
>>> >>
>>> >> -samaicardi
>>> >>
>>> >>
>>> >> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>
>>> >> The malware recordings use 1GB of RAM, so you need to pass "-m 1G" on
>>> the
>>> >>> command line when replaying.
>>> >>>
>>> >>> Also you may want to instead use the panda1 repository found here:
>>> >>>
>>> >>> https://github.com/moyix/panda
>>> >>>
>>> >>> As I think I've done a couple bugfixes to the old branch since we
>>> migrated
>>> >>> the repository to the new version of QEMU.
>>> >>>
>>> >>> -Brendan
>>> >>>
>>> >>> On Thu, Aug 31, 2017 at 11:56 AM, <aicardi at eurecom.fr> wrote:
>>> >>>
>>> >>> Ok I got it, thanks for the explanation.
>>> >>>>
>>> >>>> I have another problem actually, I tried to replay several records
>>> (from
>>> >>>> http://panda.gtisc.gatech.edu/malrec/) with the qemu-system-x86_64
>>> >>>> compiled from the branch called 'panda1' that I found here:
>>> >>>> https://github.com/panda-re/panda/tree/panda1
>>> >>>> I always get the following error:
>>> >>>> $> ~/panda1/qemu/x86_64-softmmu/qemu-system-x86_64 -replay
>>> >>>> logs/rr/7d114620-3e3c-4193-96ce-4689fd9efde3
>>> >>>>
>>> >>>> (process:1475): GLib-WARNING **:  /build/glib2.0-prJhLS/glib2.0-
>>> >>>> 2.48.2/./glib/gmem.c:483:
>>> >>>> custom memory allocation vtable not supported
>>> >>>> loading snapshot
>>> >>>> Block expected 134217728, found 1073741824, total 1082589184, system
>>> >>>> total
>>> >>>> 143065088
>>> >>>> qemu: warning: error while loading state for instance 0x0 of device
>>> 'ram'
>>> >>>> qemu-system-x86_64: Error -22 while loading VM state
>>> >>>> ... done.
>>> >>>> opening nondet log for read :   logs/rr/7d114620-3e3c-4193-96
>>> >>>> ce-4689fd9efde3-rr-nondet.log
>>> >>>> Infinite loop detected during replay, aborting.
>>> >>>> {guest_instr_count=0 pc=0x0000fff0, secondary=0x00000000}
>>> >>>> 7d114620-3e3c-4193-96ce-4689fd9efde3:           0 (  0.00%) instrs.
>>> >>>> 1.00 sec.  0.03 GB ram.
>>> >>>> total_instr in replay: 15418486377 <(541)%20848-6377>
>>> >>>> ERROR: replay failed!
>>> >>>> Time taken was: 0 seconds.
>>> >>>> max_queue_len = 1
>>> >>>> 0 items on recycle list, 0 bytes total
>>> >>>> ERROR: replay failed!
>>> >>>> Aborted (core dumped)
>>> >>>>
>>> >>>> Do you possibly know why every record seems to generate an infinite
>>> loop?
>>> >>>>
>>> >>>> Thanks in advance,
>>> >>>> samaicardi
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> Quoting Brendan Dolan-Gavitt <brendandg at nyu.edu>:
>>> >>>>
>>> >>>> Unfortunately the new version is unlikely to ever be able to replay
>>> old
>>> >>>>
>>> >>>>> recordings; too much in QEMU has changed, most notably the
>>> underlying
>>> >>>>> default machine model (and hence the set of devices included in the
>>> >>>>> snapshot). We also took the opportunity to change some of the
>>> >>>>> record/replay
>>> >>>>> log entry types to better match QEMU's new memory API.
>>> >>>>>
>>> >>>>> It is frustrating, since we have 91,000 malware recordings now and
>>> it
>>> >>>>> would
>>> >>>>> be cool to use them in panda2, but for now malware-related work has
>>> to
>>> >>>>> use
>>> >>>>> panda1. I will be switching malrec over to panda2 as soon as I have
>>> some
>>> >>>>> free time, though.
>>> >>>>>
>>> >>>>> -Brendan
>>> >>>>>
>>> >>>>> On Thu, Aug 31, 2017 at 4:50 AM, <aicardi at eurecom.fr> wrote:
>>> >>>>>
>>> >>>>> Hello everyone,
>>> >>>>>
>>> >>>>>>
>>> >>>>>> I am writing a plugin for the new version of panda
>>> >>>>>> (https://github.com/panda-re/panda) and I would like to test it
>>> with
>>> >>>>>> several malware records that can be found here:
>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/
>>> >>>>>>
>>> >>>>>> I followed the guidelines explained here:
>>> >>>>>> https://irfanulhaq.info/2015/12/09/replay-panda-malware-
>>> recordings/
>>> >>>>>> but I'm having troubles in starting the replays.
>>> >>>>>>
>>> >>>>>> When I try to execute one of those records I get the following
>>> error
>>> >>>>>> message:
>>> >>>>>> $> ~/panda2/x86_64-softmmu/qemu-system-x86_64 -replay
>>> >>>>>> ~/replays/malrec/logs/rr/bb67fd7e-7baa-437d-9333-9999b15f5fde
>>> >>>>>> > loading snapshot
>>> >>>>>> > qemu-system-x86_64: Unsupported migration stream version
>>> >>>>>> > Failed to load vmstate
>>> >>>>>> > Failed to start replay
>>> >>>>>>
>>> >>>>>> If I understood it properly, the 'problem' of those records is that
>>> >>>>>> they have been recorded starting from one of the snapshots that
>>> can be
>>> >>>>>> found here: http://panda.gtisc.gatech.edu/malrec/rr/references/
>>> >>>>>>
>>> >>>>>> These snapshots were taken using the old version of panda
>>> >>>>>> (https://github.com/moyix/panda).
>>> >>>>>>
>>> >>>>>> By analyzing the code of the new panda
>>> (include/migration/migration.h)
>>> >>>>>> I saw that there's the following line:
>>> >>>>>> #define QEMU_VM_FILE_VERSION         0x00000003
>>> >>>>>> which is different from what was declared in the old panda
>>> >>>>>> (qemu/savevm.c):
>>> >>>>>> #define QEMU_VM_SECTION_FULL         0x04
>>> >>>>>>
>>> >>>>>> That difference is causing the error I am getting and I may infer
>>> >>>>>> there are other differences between the two versions (for what
>>> >>>>>> concerns the procedure of saving a snapshot).
>>> >>>>>>
>>> >>>>>> My question is, since the two versions of panda take snapshots in
>>> >>>>>> different ways (they write different metadata I guess), is there a
>>> way
>>> >>>>>> to replay records (from http://panda.gtisc.gatech.edu/malrec/)
>>> with
>>> >>>>>> the new version of panda?
>>> >>>>>>
>>> >>>>>> Or, is it possible to 'patch' the vm snapshots (from
>>> >>>>>> http://panda.gtisc.gatech.edu/malrec/rr/references/) to make them
>>> work
>>> >>>>>> with the new version of panda?
>>> >>>>>>
>>> >>>>>> Thank you in advance for any suggestions you may have!
>>> >>>>>> samaicardi
>>> >>>>>>
>>> >>>>>> ------------------------------------------------------------
>>> >>>>>> -------------------
>>> >>>>>> This message was sent using EURECOM Webmail:
>>> http://webmail.eurecom.fr
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> _______________________________________________
>>> >>>>>> panda-users mailing list
>>> >>>>>> panda-users at mit.edu
>>> >>>>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Brendan Dolan-Gavitt
>>> >>>>> Assistant Professor, Department of Computer Science and Engineering
>>> >>>>> NYU Tandon School of Engineering
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>> ------------------------------------------------------------
>>> >>>> -------------------
>>> >>>> This message was sent using EURECOM Webmail:
>>> http://webmail.eurecom.fr
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> Brendan Dolan-Gavitt
>>> >>> Assistant Professor, Department of Computer Science and Engineering
>>> >>> NYU Tandon School of Engineering
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------
>>> >> -------------------
>>> >> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>> >>
>>> >>
>>> >
>>> >
>>> > --
>>> > Brendan Dolan-Gavitt
>>> > Assistant Professor, Department of Computer Science and Engineering
>>> > NYU Tandon School of Engineering
>>> >
>>>
>>>
>>>
>>> ------------------------------------------------------------
>>> -------------------
>>> This message was sent using EURECOM Webmail: http://webmail.eurecom.fr
>>>
>>>
>>> _______________________________________________
>>> panda-users mailing list
>>> panda-users at mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>>
>>
>> _______________________________________________
>> panda-users mailing list
>> panda-users at mit.edu
>> http://mailman.mit.edu/mailman/listinfo/panda-users
>>
>>
>
>
> --
> Brendan Dolan-Gavitt
> Assistant Professor, Department of Computer Science and Engineering
> NYU Tandon School of Engineering
>

-------------------------------------------------------------------------------
This message was sent using EURECOM Webmail: http://webmail.eurecom.fr