Ero Carrera's blog

Polip and entry point obfuscation

2009-07-01T19:37:00.007+02:00

A while ago, on a visit to an Anti-Virus lab, we started playing with some Polip samples. One of the analysts mentioned how tedious was in some cases to find the obfuscated entry-point in files infected with Polip.

We looked into a few samples of the malware, observing how the transfer of control to Polip code happens. As we read through the code, I started seeing some patterns.
In the cases we looked at, Polip always added a new section to the end of the infected executable. Then it chose some call within the original application and modified it to jump to the virus code, later on resuming execution in the original target of the call. Keeping, in that way, the original functionality of the infected application, just with a small detour.

Needless to say, it would be very tedious to manually find the redirected call(s) within an infected executable, but really easy with some scripting. Using IDAPython and the knowledge about how the jumps to the malicious code look like, we can easily come up with something that will quickly list them for us.

Let's take a look at the structure of the executable. The code of the standard application will, in most cases, reside in a single segment named CODE, .text or something along the lines. The code of the virus will reside in a segment appended to the end of the file.

Hence, if we could make a simple script that checks every single code reference that crosses segment boundaries, we would be able to list the transfers of control to the virus. Some other references might come up depending on the executable, but with some additional filtering, we will get just a few, with Polip's entry point(s) among them.

The idea in pseudo-code could look something like the following:

for each segment in the executable:
  for each function if the segment:
    for each instruction in the function:
      for each code reference from the instruction:
        if the reference points to another segment and both source and target segments are marked executable then
          print 'Possible obfuscated entry point found'

And in IDAPython... well, not all that different:

for segment in idautils.Segments():
  for func_start in idautils.Functions(segment, idc.SegEnd(segment)) :
    for head in Heads(func_start, FindFuncEnd(func_start)):
      for ref in list( CodeRefsFrom(head, 0) ):
        if SegName(ref) != SegName(func_start) and GetSegmentAttr( ref, SEGATTR_PERM ) & 0x1 and GetSegmentAttr( func_start, SEGATTR_PERM ) & 0x1:
          print '%08x: intersegment reference to %08x' % (head, ref)

Polip also finds cavities within the "standard" text section and places chunks of itself there. For those cases this simple idea of looking for inter-segment code references won't yield anything. Fortunately, most of the code lies in the extra section and studying the references from that code is almost trivial to find the chunks that Polip inserted in the cavities... it's just a few more lines of IDAPython left as an exercise for the reader... ;-)

Just want to remind anyone interested that BlackHat Vegas is coming in a few weeks and Pedram Amini and I will be teaching our training, "Reverse Engineering on Windows: Application in Malicious Code Analysis " . If you want learn about how to build this kind of automation among other things, we would love to have you in our class.

Thanks Fravia! Rest In Peace

2009-05-04T19:32:00.004+02:00

Yesterday, May 3rd, the great Fravia passed away.
He ran his legendary fravia pages and searchlores. A lot of people in the reverse engineering world are (and will be) definitely indebted to him for his teachings. He will definitely will be missed.

Some friends notes on the sad news here, here and here

More syscall ordinals

2009-01-23T18:44:00.003+01:00

Daniel Reynaud has improved on the script I described a while ago and has posted on his blog the system call ordinals for Windows XP SP2 x64.

pefile and LOAD_CONFIG

2009-01-23T01:58:00.002+01:00

Following a conversation in twitter I've noticed pefile was lacking support for parsing one data directory in the PE format that is rather interesting, the IMAGE_LOAD_CONFIG_DIRECTORY.

I've added support for it and fixed a few small bugs and released it as pefile-1.2.10-60

Now one can access this structure's fields like, for instance, pe.DIRECTORY_ENTRY_LOAD_CONFIG.struct.SecurityCookie or pe.DIRECTORY_ENTRY_LOAD_CONFIG.struct.SEHandlerTable and also modify their values and write the result to a new PE file, all the usual handling & mangling that pefile allows.

Tips and tricks

2009-01-08T17:39:00.004+01:00

A couple of interesting things I've found out lately:

When packaging the latest pefile I noticed the dot-underscore files in the tar.gz. If one extracts the contents there are no such files to be found (if you're working on OSX) while they will show up in other operating systems. Those dot-underscore files are OSX way of storing the resource fork (metainformaion). While it might be handy to keep it around when moving files between Macs, it's not nice to have such dot-underscore files show up in other systems. How to get rid of them is not too well documented.

There are two oddly named environment variables (they changed between OSX versions) that control the creation of such files. Setting the following envionment variables to 'true' will make tar not create those dot-underscore files when archiving a file with a resource fork.

COPYFILE_DISABLE
COPY_EXTENDED_ATTRIBUTES_DISABLE

One can just set them in the python setup.py script, so when the source distribution is created, no resource forks are dumped into those files.

import os
os.environ['COPY_EXTENDED_ATTRIBUTES_DISABLE'] = 'true'
os.environ['COPYFILE_DISABLE'] = 'true'

In my case it was TextMate that was using a resource fork to store some metainformation about the Python files I was working with.

Now a useful tip for subversion. I always knew CVS and subversion had to have such feature but never was able to find how to use, I finally tracked it down.

This might not be knew to anyone that has spent some time with svn... but was to me. The things is, I was sure there had to be a comfortable way of having SVN automatically add the revision number to the source code. That would allow to have version numbers with a revision appended to them automatically, which would make lots of things much nicer, like tracking errors with specific versions.

To achieve that, one can use subversion keywords. SVN will replace those keywords with the appropriate information. In this case the cool one is "$LastChangedRevision$", whereever we write it, it will get replaced by "$LastChangedRevision: XXX $" where XXX is the revision number.

You need to tell subversion you want it to replace that keyword in a given file(s). To do that just issue a: svn propset svn:keywords "Rev" path/to/the/file to set the property on that file.

A practical example for Python code would be:

__revision__ = "$LastChangedRevision$"
__version__ = '%d' % int( __revision__[21:-2] )

The keyword would be replaced as described above and then we can fetch the revision number and add it to the version number transparently. Subversion will handle it cleanly.

Updated pefile

2009-01-07T17:58:00.003+01:00

I've just released pefile-1.2.10-56 which besides some new functionality it also fixes bugs for a few extreme cases and incorporates some performance improvements, the biggest of which is the one in the generation of the textual representation of the file. dump_info() is now much faster than it used to be.
This version adds onto the features provided by version 1.2.9 which introduced the ability to test and generate checksums for the PE file among a few other things.

Please refer to pefile's homepage for a detailed list of the changes.

Also, to get started with pefile it's often useful to take a look at usage examples. There's a wiki page in the project's page showing a few different recipes on how to go about doing different tasks. Some as simple as, for instance, extracing a DLL's exported symbols...

import pefile
pe = pefile.PE(‘/path/to/pefile.exe’)

for exp in pe.DIRECTORY_ENTRY_EXPORT.symbols:
print hex(pe.OPTIONAL_HEADER.ImageBase + exp.address), exp.name, exp.ordinal

An all the way into more complex examples.

Enjoy!

Thoughts on "Using dual-mappings to evade automated unpackers"

2008-10-21T14:29:00.001+02:00

Uninformed 10 was released recently. On it Skape brings about a simple, yet beautiful and powerful idea. The paper itself is short and concise but a quick summary would be along the lines of: some generic unpackers use, either as their main technique or as an heuristic, the tracking of memory writes and whenever the execution flow hits those written areas an assumption is made that unpacked (or self-modifying code) has been reached. This is an over-simplification because of multi-staged unpackers and other details, but will suffice for the sake of the discussion. Skape basically introduces a technique by which he's able to write into a range of virtual addresses an execute from another range, both pointing to the same real data.

This is a well supported technique and nothing strange of itself. It is its use what is creative and rather amusing as it breaks the assumption mentioned earlier. Given that no writes are seen in the area of memory that will be executed. The technique relies on the possibility of having several virtual addresses refer the same physical memory. The Memory Management Unit (MMU) allows pages of virtual memory to map to common physical locations in order to avoid, for instance, the need of having multiple copies of shared components between different processes.

Some of the tools having trouble with this trick can't really do much about it but other tools he mentions in the paper should have no issue handling it in some way or another.

I've been working with Bochs for a few years. The technique should be (and is) easily defeatable by tools providing a bird's eye perspective like Bochs or by tools running in kernel mode.

Analyzing the generally one-to-one mapping of virtual-to-physical memory to find a one-to-many relationship is not all that difficult, and the case where a set of virtually mapped pages receive writes and not much more and another set pointing to the same physical ranges is only (or mostly just) executed should be easy to determine. That'd be a simple heuristic but attempting to defeat it by writing to the execute-only mapping would break down the whole idea...

It's definitely a beatiful idea but it's easy to detect if you're on the right spot.

One can take a quick look at it if you have Windbg lying around. Just connect to your test machine running a test process that implements the code that Skape outlines.

Do a quick process listing to find it by issuing:

!process 0 0

Then, take a look at the Cid of the process of interest and get its details with:

!process <Cid>

In that listing the page directory base DirBase will be given. With that we can tell Windbg to do the mapping of a virtual address using the virtual-to-physical mapping of that process by pasing the directory base to the !vtop command. In my case the DirBase of my process was f5a and I'll instruct Windbg to give me the corresponding physical address to which the virtual addresses 0x419000 and 0x519000 are mapped (those addresses are specific to the example code I wrote implementing skape's idea)

lkd> !vtop f5a 0x419000
Pdi 1 Pti 19
00419000 14a79000 pfn(14a79)
lkd> !vtop f5a 0x519000
Pdi 1 Pti 119
00519000 14a79000 pfn(14a79)

As you can see both lead to the same physical location 14a79000. So it's easy to see this from kernel mode by walking the page directory of a process under surveillance.

If you want to see all physical-virtual mappings the following command provides that:

!ptov f5a

I've also implemented detection of this trick in some Python scripts I've developed for my Python-weaponized Bochs environment. On that front. I have an up-to-date patch for Bochs (improved from what I introduced at HitB in Dubai). While I hope to be able to eventually work with the Bochs developers to merge it anyone that wants to give it a shot it welcome to shoot me an email...

I'll be talking on packers and their techniques on the forthcoming Hack in the Box in Kuala Lumpur. I'll be there next week and later in November at Power Of Community in Korea.

Twitter in Dubai

2008-04-16T15:29:00.002+02:00

This is how it looks...

Funnily enough, it seemed to be reachable through Dubai's airport wifi. Also the iPhone's Twinkle application can get to it, but I guess it's using some API and not accessing the main site.

I think it's the first time I'm in a country blocking a site like Twitter... I'd have thought other sites such as Myspace would also be blocked but apparently aren't.

Running around

2008-04-05T19:31:00.004+02:00

It's been a few crazy weeks for me. A few weeks ago I finally moved back to the wonderful Barcelona and trying to get settled in between trips to SOURCE Boston, BlackHat Amsterdam and now RSA in San Francisco... and next week is going to be HitB in Dubai where I'll be showing a new tool I've put together.

SOURCE Boston was a really interesting event, impeccably organized and with really great speakers and atmosphere. The technical level of the talks I could see was great, but missed the first day of conference because of tight scheduling. The materials will be coming out here. It was the first installment and sure hope will be the first of many, as it was really fun and enjoyable.

In BlackHat Amsterdam I was teaching the training with Pedram Amini. We got some good feedback and the course should be seeing some good updates in Las Vegas later in the summer.

Digging up system call ordinals

2008-03-06T21:12:00.005+01:00

Today I was hacking a small tool and I needed a list of all the system call ordinals corresponding to the APIs exported by NTDLL.DLL. A bit of googling didn't come up with anything too interesting so I wrote a small IDAPython script to harvest them out of a disassembly of NTDLL.DLL.
The script will simply iterate through every segment and every function and try to find the byte pattern corresponding to the prolog of API functions calling the stub doing the SYSENTER, SYSCALL or INT 2Eh.
At least in Windows XP SP2 they will have the form:

MOV eax, XX	where XX is the syscall ordinal
MOV edx, 7FFE0300h	the stub doing the transition to kernel mode, the actual code reached depends on the underlying processor
CALL [edx]

Those instructions correspond to the byte sequence 'B8 ? 00 00 00 BA 00 03 FE 7F'. I'll just tell IDAPython to look for it at the beginning of each function and, if found, I'll extract the value of the system call ordinal and the name of the function and print a list of them:

syscall_ordinal_code = 'b8 ? 00 00 00 ba 00 03 fe 7f'

for seg in Segments():
  for func in Functions(seg, SegEnd(seg)):
    address = FindBinary(func, SEARCH_DOWN, syscall_ordinal_code)
    if address == func:
      print '%08x: Syscall ordinal %04x for %s (%s)' % (
        func, Dword(func+1), Name(func), Comment(func))

And the outcome of running the script on IDA with NTDLL.DLL looks like this:

7c90d379: Syscall ordinal 0000 for ZwAcceptConnectPort (NtAcceptConnectPort)
7c90d38e: Syscall ordinal 0001 for ZwAccessCheck (NtAccessCheck)
7c90d3a3: Syscall ordinal 0002 for ZwAccessCheckAndAuditAlarm (NtAccessCheckAndAuditAlarm)
7c90d3b8: Syscall ordinal 0003 for ZwAccessCheckByType (NtAccessCheckByType)
7c90d3cd: Syscall ordinal 0004 for ZwAccessCheckByTypeAndAuditAlarm (NtAccessCheckByTypeAndAuditAlarm)
7c90d3e2: Syscall ordinal 0005 for ZwAccessCheckByTypeResultList (NtAccessCheckByTypeResultList)
7c90d3f7: Syscall ordinal 0006 for ZwAccessCheckByTypeResultListAndAuditAlarm (NtAccessCheckByTypeResultListAndAuditAlarm)
7c90d40c: Syscall ordinal 0007 for ZwAccessCheckByTypeResultListAndAuditAlarmByHandle (NtAccessCheckByTypeResultListAndAuditAlarmByHandle)
7c90d421: Syscall ordinal 0008 for ZwAddAtom (NtAddAtom)
7c90d436: Syscall ordinal 0009 for ZwAddBootEntry (NtAddBootEntry)
7c90d44b: Syscall ordinal 000a for ZwAdjustGroupsToken (NtAdjustGroupsToken)
7c90d460: Syscall ordinal 000b for ZwAdjustPrivilegesToken (NtAdjustPrivilegesToken)
7c90d475: Syscall ordinal 000c for ZwAlertResumeThread (NtAlertResumeThread)
7c90d48a: Syscall ordinal 000d for ZwAlertThread (NtAlertThread)
7c90d49f: Syscall ordinal 000e for ZwAllocateLocallyUniqueId (NtAllocateLocallyUniqueId)
7c90d4b4: Syscall ordinal 000f for ZwAllocateUserPhysicalPages (NtAllocateUserPhysicalPages)
7c90d4c9: Syscall ordinal 0010 for ZwAllocateUuids (NtAllocateUuids)
7c90d4de: Syscall ordinal 0011 for ZwAllocateVirtualMemory (NtAllocateVirtualMemory)
7c90d4f3: Syscall ordinal 0012 for ZwAreMappedFilesTheSame (NtAreMappedFilesTheSame)
7c90d508: Syscall ordinal 0013 for ZwAssignProcessToJobObject (NtAssignProcessToJobObject)
7c90d51d: Syscall ordinal 0014 for ZwCallbackReturn (NtCallbackReturn)
7c90d532: Syscall ordinal 0015 for ZwCancelDeviceWakeupRequest (NtCancelDeviceWakeupRequest)
7c90d547: Syscall ordinal 0016 for ZwCancelIoFile (NtCancelIoFile)
7c90d55c: Syscall ordinal 0017 for ZwCancelTimer (NtCancelTimer)
7c90d571: Syscall ordinal 0018 for ZwClearEvent (NtClearEvent)
7c90d586: Syscall ordinal 0019 for ZwClose (NtClose)
7c90d59b: Syscall ordinal 001a for ZwCloseObjectAuditAlarm (NtCloseObjectAuditAlarm)
7c90d5b0: Syscall ordinal 001b for ZwCompactKeys (NtCompactKeys)
7c90d5c5: Syscall ordinal 001c for ZwCompareTokens (NtCompareTokens)
7c90d5da: Syscall ordinal 001d for ZwCompleteConnectPort (NtCompleteConnectPort)
7c90d5ef: Syscall ordinal 001e for ZwCompressKey (NtCompressKey)
7c90d604: Syscall ordinal 001f for ZwConnectPort (NtConnectPort)
7c90d619: Syscall ordinal 0020 for ZwContinue (NtContinue)
7c90d62e: Syscall ordinal 0021 for ZwCreateDebugObject (NtCreateDebugObject)
7c90d643: Syscall ordinal 0022 for ZwCreateDirectoryObject (NtCreateDirectoryObject)
7c90d658: Syscall ordinal 0023 for ZwCreateEvent (NtCreateEvent)
7c90d66d: Syscall ordinal 0024 for ZwCreateEventPair (NtCreateEventPair)
7c90d682: Syscall ordinal 0025 for ZwCreateFile (NtCreateFile)
7c90d697: Syscall ordinal 0026 for ZwCreateIoCompletion (NtCreateIoCompletion)
7c90d6ac: Syscall ordinal 0027 for ZwCreateJobObject (NtCreateJobObject)
7c90d6c1: Syscall ordinal 0028 for ZwCreateJobSet (NtCreateJobSet)
7c90d6d6: Syscall ordinal 0029 for ZwCreateKey (NtCreateKey)
7c90d6eb: Syscall ordinal 002a for ZwCreateMailslotFile (NtCreateMailslotFile)
7c90d700: Syscall ordinal 002b for ZwCreateMutant (NtCreateMutant)
7c90d715: Syscall ordinal 002c for ZwCreateNamedPipeFile (NtCreateNamedPipeFile)
7c90d72a: Syscall ordinal 002d for ZwCreatePagingFile (NtCreatePagingFile)
7c90d73f: Syscall ordinal 002e for ZwCreatePort (NtCreatePort)
7c90d754: Syscall ordinal 002f for ZwCreateProcess (NtCreateProcess)
7c90d769: Syscall ordinal 0030 for ZwCreateProcessEx (NtCreateProcessEx)
7c90d77e: Syscall ordinal 0031 for ZwCreateProfile (NtCreateProfile)
7c90d793: Syscall ordinal 0032 for ZwCreateSection (NtCreateSection)
7c90d7a8: Syscall ordinal 0033 for ZwCreateSemaphore (NtCreateSemaphore)
7c90d7bd: Syscall ordinal 0034 for ZwCreateSymbolicLinkObject (NtCreateSymbolicLinkObject)
7c90d7d2: Syscall ordinal 0035 for ZwCreateThread (NtCreateThread)
7c90d7e7: Syscall ordinal 0036 for ZwCreateTimer (NtCreateTimer)
7c90d7fc: Syscall ordinal 0037 for ZwCreateToken (NtCreateToken)
7c90d811: Syscall ordinal 0038 for ZwCreateWaitablePort (NtCreateWaitablePort)
7c90d826: Syscall ordinal 0039 for ZwDebugActiveProcess (NtDebugActiveProcess)
7c90d83b: Syscall ordinal 003a for ZwDebugContinue (NtDebugContinue)
7c90d850: Syscall ordinal 003b for ZwDelayExecution (NtDelayExecution)
7c90d865: Syscall ordinal 003c for ZwDeleteAtom (NtDeleteAtom)
7c90d87a: Syscall ordinal 003d for ZwDeleteBootEntry (NtDeleteBootEntry)
7c90d88f: Syscall ordinal 003e for ZwDeleteFile (NtDeleteFile)
7c90d8a4: Syscall ordinal 003f for ZwDeleteKey (NtDeleteKey)
7c90d8b9: Syscall ordinal 0040 for ZwDeleteObjectAuditAlarm (NtDeleteObjectAuditAlarm)
7c90d8ce: Syscall ordinal 0041 for ZwDeleteValueKey (NtDeleteValueKey)
7c90d8e3: Syscall ordinal 0042 for ZwDeviceIoControlFile (NtDeviceIoControlFile)
7c90d8f8: Syscall ordinal 0043 for ZwDisplayString (NtDisplayString)
7c90d90d: Syscall ordinal 0044 for ZwDuplicateObject (NtDuplicateObject)
7c90d922: Syscall ordinal 0045 for ZwDuplicateToken (NtDuplicateToken)
7c90d937: Syscall ordinal 0046 for ZwEnumerateBootEntries (NtEnumerateBootEntries)
7c90d94c: Syscall ordinal 0047 for ZwEnumerateKey (NtEnumerateKey)
7c90d961: Syscall ordinal 0048 for ZwEnumerateSystemEnvironmentValuesEx (NtEnumerateSystemEnvironmentValuesEx)
7c90d976: Syscall ordinal 0049 for ZwEnumerateValueKey (NtEnumerateValueKey)
7c90d98b: Syscall ordinal 004a for ZwExtendSection (NtExtendSection)
7c90d9a0: Syscall ordinal 004b for ZwFilterToken (NtFilterToken)
7c90d9b5: Syscall ordinal 004c for ZwFindAtom (NtFindAtom)
7c90d9ca: Syscall ordinal 004d for ZwFlushBuffersFile (NtFlushBuffersFile)
7c90d9df: Syscall ordinal 004e for ZwFlushInstructionCache (NtFlushInstructionCache)
7c90d9f4: Syscall ordinal 004f for ZwFlushKey (NtFlushKey)
7c90da09: Syscall ordinal 0050 for ZwFlushVirtualMemory (NtFlushVirtualMemory)
7c90da1e: Syscall ordinal 0051 for ZwFlushWriteBuffer (NtFlushWriteBuffer)
7c90da33: Syscall ordinal 0052 for ZwFreeUserPhysicalPages (NtFreeUserPhysicalPages)
7c90da48: Syscall ordinal 0053 for ZwFreeVirtualMemory (NtFreeVirtualMemory)
7c90da5d: Syscall ordinal 0054 for ZwFsControlFile (NtFsControlFile)
7c90da72: Syscall ordinal 0055 for ZwGetContextThread (NtGetContextThread)
7c90da87: Syscall ordinal 0056 for ZwGetDevicePowerState (NtGetDevicePowerState)
7c90da9c: Syscall ordinal 0057 for ZwGetPlugPlayEvent (NtGetPlugPlayEvent)
7c90dab1: Syscall ordinal 0058 for ZwGetWriteWatch (NtGetWriteWatch)
7c90dac6: Syscall ordinal 0059 for ZwImpersonateAnonymousToken (NtImpersonateAnonymousToken)
7c90dadb: Syscall ordinal 005a for ZwImpersonateClientOfPort (NtImpersonateClientOfPort)
7c90daf0: Syscall ordinal 005b for ZwImpersonateThread (NtImpersonateThread)
7c90db05: Syscall ordinal 005c for ZwInitializeRegistry (NtInitializeRegistry)
7c90db1a: Syscall ordinal 005d for ZwInitiatePowerAction (NtInitiatePowerAction)
7c90db2f: Syscall ordinal 005e for ZwIsProcessInJob (NtIsProcessInJob)
7c90db44: Syscall ordinal 005f for ZwIsSystemResumeAutomatic (NtIsSystemResumeAutomatic)
7c90db59: Syscall ordinal 0060 for ZwListenPort (NtListenPort)
7c90db6e: Syscall ordinal 0061 for ZwLoadDriver (NtLoadDriver)
7c90db83: Syscall ordinal 0062 for ZwLoadKey (NtLoadKey)
7c90db98: Syscall ordinal 0063 for ZwLoadKey2 (NtLoadKey2)
7c90dbad: Syscall ordinal 0064 for ZwLockFile (NtLockFile)
7c90dbc2: Syscall ordinal 0065 for ZwLockProductActivationKeys (NtLockProductActivationKeys)
7c90dbd7: Syscall ordinal 0066 for ZwLockRegistryKey (NtLockRegistryKey)
7c90dbec: Syscall ordinal 0067 for ZwLockVirtualMemory (NtLockVirtualMemory)
7c90dc01: Syscall ordinal 0068 for ZwMakePermanentObject (NtMakePermanentObject)
7c90dc16: Syscall ordinal 0069 for ZwMakeTemporaryObject (NtMakeTemporaryObject)
7c90dc2b: Syscall ordinal 006a for ZwMapUserPhysicalPages (NtMapUserPhysicalPages)
7c90dc40: Syscall ordinal 006b for ZwMapUserPhysicalPagesScatter (NtMapUserPhysicalPagesScatter)
7c90dc55: Syscall ordinal 006c for ZwMapViewOfSection (NtMapViewOfSection)
7c90dc6a: Syscall ordinal 006d for ZwModifyBootEntry (NtModifyBootEntry)
7c90dc7f: Syscall ordinal 006e for ZwNotifyChangeDirectoryFile (NtNotifyChangeDirectoryFile)
7c90dc94: Syscall ordinal 006f for ZwNotifyChangeKey (NtNotifyChangeKey)
7c90dca9: Syscall ordinal 0070 for ZwNotifyChangeMultipleKeys (NtNotifyChangeMultipleKeys)
7c90dcbe: Syscall ordinal 0071 for ZwOpenDirectoryObject (NtOpenDirectoryObject)
7c90dcd3: Syscall ordinal 0072 for ZwOpenEvent (NtOpenEvent)
7c90dce8: Syscall ordinal 0073 for ZwOpenEventPair (NtOpenEventPair)
7c90dcfd: Syscall ordinal 0074 for ZwOpenFile (NtOpenFile)
7c90dd12: Syscall ordinal 0075 for ZwOpenIoCompletion (NtOpenIoCompletion)
7c90dd27: Syscall ordinal 0076 for ZwOpenJobObject (NtOpenJobObject)
7c90dd3c: Syscall ordinal 0077 for ZwOpenKey (NtOpenKey)
7c90dd51: Syscall ordinal 0078 for ZwOpenMutant (NtOpenMutant)
7c90dd66: Syscall ordinal 0079 for ZwOpenObjectAuditAlarm (NtOpenObjectAuditAlarm)
7c90dd7b: Syscall ordinal 007a for ZwOpenProcess (NtOpenProcess)
7c90dd90: Syscall ordinal 007b for ZwOpenProcessToken (NtOpenProcessToken)
7c90dda5: Syscall ordinal 007c for ZwOpenProcessTokenEx (NtOpenProcessTokenEx)
7c90ddba: Syscall ordinal 007d for ZwOpenSection (NtOpenSection)
7c90ddcf: Syscall ordinal 007e for ZwOpenSemaphore (NtOpenSemaphore)
7c90dde4: Syscall ordinal 007f for ZwOpenSymbolicLinkObject (NtOpenSymbolicLinkObject)
7c90ddf9: Syscall ordinal 0080 for ZwOpenThread (NtOpenThread)
7c90de0e: Syscall ordinal 0081 for ZwOpenThreadToken (NtOpenThreadToken)
7c90de23: Syscall ordinal 0082 for ZwOpenThreadTokenEx (NtOpenThreadTokenEx)
7c90de38: Syscall ordinal 0083 for ZwOpenTimer (NtOpenTimer)
7c90de4d: Syscall ordinal 0084 for ZwPlugPlayControl (NtPlugPlayControl)
7c90de62: Syscall ordinal 0085 for ZwPowerInformation (NtPowerInformation)
7c90de77: Syscall ordinal 0086 for ZwPrivilegeCheck (NtPrivilegeCheck)
7c90de8c: Syscall ordinal 0087 for ZwPrivilegeObjectAuditAlarm (NtPrivilegeObjectAuditAlarm)
7c90dea1: Syscall ordinal 0088 for ZwPrivilegedServiceAuditAlarm (NtPrivilegedServiceAuditAlarm)
7c90deb6: Syscall ordinal 0089 for ZwProtectVirtualMemory (NtProtectVirtualMemory)
7c90decb: Syscall ordinal 008a for ZwPulseEvent (NtPulseEvent)
7c90dee0: Syscall ordinal 008b for ZwQueryAttributesFile (NtQueryAttributesFile)
7c90def5: Syscall ordinal 008c for ZwQueryBootEntryOrder (NtQueryBootEntryOrder)
7c90df0a: Syscall ordinal 008d for ZwQueryBootOptions (NtQueryBootOptions)
7c90df1f: Syscall ordinal 008e for ZwQueryDebugFilterState (NtQueryDebugFilterState)
7c90df34: Syscall ordinal 008f for ZwQueryDefaultLocale (NtQueryDefaultLocale)
7c90df49: Syscall ordinal 0090 for ZwQueryDefaultUILanguage (NtQueryDefaultUILanguage)
7c90df5e: Syscall ordinal 0091 for ZwQueryDirectoryFile (NtQueryDirectoryFile)
7c90df73: Syscall ordinal 0092 for ZwQueryDirectoryObject (NtQueryDirectoryObject)
7c90df88: Syscall ordinal 0093 for ZwQueryEaFile (NtQueryEaFile)
7c90df9d: Syscall ordinal 0094 for ZwQueryEvent (NtQueryEvent)
7c90dfb2: Syscall ordinal 0095 for ZwQueryFullAttributesFile (NtQueryFullAttributesFile)
7c90dfc7: Syscall ordinal 0096 for ZwQueryInformationAtom (NtQueryInformationAtom)
7c90dfdc: Syscall ordinal 0097 for ZwQueryInformationFile (NtQueryInformationFile)
7c90dff1: Syscall ordinal 0098 for ZwQueryInformationJobObject (NtQueryInformationJobObject)
7c90e006: Syscall ordinal 0099 for ZwQueryInformationPort (NtQueryInformationPort)
7c90e01b: Syscall ordinal 009a for ZwQueryInformationProcess (NtQueryInformationProcess)
7c90e030: Syscall ordinal 009b for ZwQueryInformationThread (NtQueryInformationThread)
7c90e045: Syscall ordinal 009c for ZwQueryInformationToken (NtQueryInformationToken)
7c90e05a: Syscall ordinal 009d for ZwQueryInstallUILanguage (NtQueryInstallUILanguage)
7c90e06f: Syscall ordinal 009e for ZwQueryIntervalProfile (NtQueryIntervalProfile)
7c90e084: Syscall ordinal 009f for ZwQueryIoCompletion (NtQueryIoCompletion)
7c90e099: Syscall ordinal 00a0 for ZwQueryKey (NtQueryKey)
7c90e0ae: Syscall ordinal 00a1 for ZwQueryMultipleValueKey (NtQueryMultipleValueKey)
7c90e0c3: Syscall ordinal 00a2 for ZwQueryMutant (NtQueryMutant)
7c90e0d8: Syscall ordinal 00a3 for ZwQueryObject (NtQueryObject)
7c90e0ed: Syscall ordinal 00a4 for ZwQueryOpenSubKeys (NtQueryOpenSubKeys)
7c90e102: Syscall ordinal 00a5 for ZwQueryPerformanceCounter (NtQueryPerformanceCounter)
7c90e117: Syscall ordinal 00a6 for ZwQueryQuotaInformationFile (NtQueryQuotaInformationFile)
7c90e12c: Syscall ordinal 00a7 for ZwQuerySection (NtQuerySection)
7c90e141: Syscall ordinal 00a8 for ZwQuerySecurityObject (NtQuerySecurityObject)
7c90e156: Syscall ordinal 00a9 for ZwQuerySemaphore (NtQuerySemaphore)
7c90e16b: Syscall ordinal 00aa for ZwQuerySymbolicLinkObject (NtQuerySymbolicLinkObject)
7c90e180: Syscall ordinal 00ab for ZwQuerySystemEnvironmentValue (NtQuerySystemEnvironmentValue)
7c90e195: Syscall ordinal 00ac for ZwQuerySystemEnvironmentValueEx (NtQuerySystemEnvironmentValueEx)
7c90e1aa: Syscall ordinal 00ad for ZwQuerySystemInformation (NtQuerySystemInformation
RtlGetNativeSystemInformation)
7c90e1bf: Syscall ordinal 00ae for ZwQuerySystemTime (NtQuerySystemTime)
7c90e1d4: Syscall ordinal 00af for ZwQueryTimer (NtQueryTimer)
7c90e1e9: Syscall ordinal 00b0 for ZwQueryTimerResolution (NtQueryTimerResolution)
7c90e1fe: Syscall ordinal 00b1 for ZwQueryValueKey (NtQueryValueKey)
7c90e213: Syscall ordinal 00b2 for ZwQueryVirtualMemory (NtQueryVirtualMemory)
7c90e228: Syscall ordinal 00b3 for ZwQueryVolumeInformationFile (NtQueryVolumeInformationFile)
7c90e23d: Syscall ordinal 00b4 for ZwQueueApcThread (NtQueueApcThread)
7c90e252: Syscall ordinal 00b5 for ZwRaiseException (NtRaiseException)
7c90e267: Syscall ordinal 00b6 for ZwRaiseHardError (NtRaiseHardError)
7c90e27c: Syscall ordinal 00b7 for ZwReadFile (NtReadFile)
7c90e291: Syscall ordinal 00b8 for ZwReadFileScatter (NtReadFileScatter)
7c90e2a6: Syscall ordinal 00b9 for ZwReadRequestData (NtReadRequestData)
7c90e2bb: Syscall ordinal 00ba for ZwReadVirtualMemory (NtReadVirtualMemory)
7c90e2d0: Syscall ordinal 00bb for ZwRegisterThreadTerminatePort (NtRegisterThreadTerminatePort)
7c90e2e5: Syscall ordinal 00bc for ZwReleaseMutant (NtReleaseMutant)
7c90e2fa: Syscall ordinal 00bd for ZwReleaseSemaphore (NtReleaseSemaphore)
7c90e30f: Syscall ordinal 00be for ZwRemoveIoCompletion (NtRemoveIoCompletion)
7c90e324: Syscall ordinal 00bf for ZwRemoveProcessDebug (NtRemoveProcessDebug)
7c90e339: Syscall ordinal 00c0 for ZwRenameKey (NtRenameKey)
7c90e34e: Syscall ordinal 00c1 for ZwReplaceKey (NtReplaceKey)
7c90e363: Syscall ordinal 00c2 for ZwReplyPort (NtReplyPort)
7c90e378: Syscall ordinal 00c3 for ZwReplyWaitReceivePort (NtReplyWaitReceivePort)
7c90e38d: Syscall ordinal 00c4 for ZwReplyWaitReceivePortEx (NtReplyWaitReceivePortEx)
7c90e3a2: Syscall ordinal 00c5 for ZwReplyWaitReplyPort (NtReplyWaitReplyPort)
7c90e3b7: Syscall ordinal 00c6 for ZwRequestDeviceWakeup (NtRequestDeviceWakeup)
7c90e3cc: Syscall ordinal 00c7 for ZwRequestPort (NtRequestPort)
7c90e3e1: Syscall ordinal 00c8 for ZwRequestWaitReplyPort (NtRequestWaitReplyPort)
7c90e3f6: Syscall ordinal 00c9 for ZwRequestWakeupLatency (NtRequestWakeupLatency)
7c90e40b: Syscall ordinal 00ca for ZwResetEvent (NtResetEvent)
7c90e420: Syscall ordinal 00cb for ZwResetWriteWatch (NtResetWriteWatch)
7c90e435: Syscall ordinal 00cc for ZwRestoreKey (NtRestoreKey)
7c90e44a: Syscall ordinal 00cd for ZwResumeProcess (NtResumeProcess)
7c90e45f: Syscall ordinal 00ce for ZwResumeThread (NtResumeThread)
7c90e474: Syscall ordinal 00cf for ZwSaveKey (NtSaveKey)
7c90e489: Syscall ordinal 00d0 for ZwSaveKeyEx (NtSaveKeyEx)
7c90e49e: Syscall ordinal 00d1 for ZwSaveMergedKeys (NtSaveMergedKeys)
7c90e4b3: Syscall ordinal 00d2 for ZwSecureConnectPort (NtSecureConnectPort)
7c90e4c8: Syscall ordinal 00d3 for ZwSetBootEntryOrder (NtSetBootEntryOrder)
7c90e4dd: Syscall ordinal 00d4 for ZwSetBootOptions (NtSetBootOptions)
7c90e4f2: Syscall ordinal 00d5 for ZwSetContextThread (NtSetContextThread)
7c90e507: Syscall ordinal 00d6 for ZwSetDebugFilterState (NtSetDebugFilterState)
7c90e51c: Syscall ordinal 00d7 for ZwSetDefaultHardErrorPort (NtSetDefaultHardErrorPort)
7c90e531: Syscall ordinal 00d8 for ZwSetDefaultLocale (NtSetDefaultLocale)
7c90e546: Syscall ordinal 00d9 for ZwSetDefaultUILanguage (NtSetDefaultUILanguage)
7c90e55b: Syscall ordinal 00da for ZwSetEaFile (NtSetEaFile)
7c90e570: Syscall ordinal 00db for ZwSetEvent (NtSetEvent)
7c90e585: Syscall ordinal 00dc for ZwSetEventBoostPriority (NtSetEventBoostPriority)
7c90e59a: Syscall ordinal 00dd for ZwSetHighEventPair (NtSetHighEventPair)
7c90e5af: Syscall ordinal 00de for ZwSetHighWaitLowEventPair (NtSetHighWaitLowEventPair)
7c90e5c4: Syscall ordinal 00df for ZwSetInformationDebugObject (NtSetInformationDebugObject)
7c90e5d9: Syscall ordinal 00e0 for ZwSetInformationFile (NtSetInformationFile)
7c90e5ee: Syscall ordinal 00e1 for ZwSetInformationJobObject (NtSetInformationJobObject)
7c90e603: Syscall ordinal 00e2 for ZwSetInformationKey (NtSetInformationKey)
7c90e618: Syscall ordinal 00e3 for ZwSetInformationObject (NtSetInformationObject)
7c90e62d: Syscall ordinal 00e4 for ZwSetInformationProcess (NtSetInformationProcess)
7c90e642: Syscall ordinal 00e5 for ZwSetInformationThread (NtSetInformationThread)
7c90e657: Syscall ordinal 00e6 for ZwSetInformationToken (NtSetInformationToken)
7c90e66c: Syscall ordinal 00e7 for ZwSetIntervalProfile (NtSetIntervalProfile)
7c90e681: Syscall ordinal 00e8 for ZwSetIoCompletion (NtSetIoCompletion)
7c90e696: Syscall ordinal 00e9 for ZwSetLdtEntries (NtSetLdtEntries)
7c90e6ab: Syscall ordinal 00ea for ZwSetLowEventPair (NtSetLowEventPair)
7c90e6c0: Syscall ordinal 00eb for ZwSetLowWaitHighEventPair (NtSetLowWaitHighEventPair)
7c90e6d5: Syscall ordinal 00ec for ZwSetQuotaInformationFile (NtSetQuotaInformationFile)
7c90e6ea: Syscall ordinal 00ed for ZwSetSecurityObject (NtSetSecurityObject)
7c90e6ff: Syscall ordinal 00ee for ZwSetSystemEnvironmentValue (NtSetSystemEnvironmentValue)
7c90e714: Syscall ordinal 00ef for ZwSetSystemEnvironmentValueEx (NtSetSystemEnvironmentValueEx)
7c90e729: Syscall ordinal 00f0 for ZwSetSystemInformation (NtSetSystemInformation)
7c90e73e: Syscall ordinal 00f1 for ZwSetSystemPowerState (NtSetSystemPowerState)
7c90e753: Syscall ordinal 00f2 for ZwSetSystemTime (NtSetSystemTime)
7c90e768: Syscall ordinal 00f3 for ZwSetThreadExecutionState (NtSetThreadExecutionState)
7c90e77d: Syscall ordinal 00f4 for ZwSetTimer (NtSetTimer)
7c90e792: Syscall ordinal 00f5 for ZwSetTimerResolution (NtSetTimerResolution)
7c90e7a7: Syscall ordinal 00f6 for ZwSetUuidSeed (NtSetUuidSeed)
7c90e7bc: Syscall ordinal 00f7 for ZwSetValueKey (NtSetValueKey)
7c90e7d1: Syscall ordinal 00f8 for ZwSetVolumeInformationFile (NtSetVolumeInformationFile)
7c90e7e6: Syscall ordinal 00f9 for ZwShutdownSystem (NtShutdownSystem)
7c90e7fb: Syscall ordinal 00fa for ZwSignalAndWaitForSingleObject (NtSignalAndWaitForSingleObject)
7c90e810: Syscall ordinal 00fb for ZwStartProfile (NtStartProfile)
7c90e825: Syscall ordinal 00fc for ZwStopProfile (NtStopProfile)
7c90e83a: Syscall ordinal 00fd for ZwSuspendProcess (NtSuspendProcess)
7c90e84f: Syscall ordinal 00fe for ZwSuspendThread (NtSuspendThread)
7c90e864: Syscall ordinal 00ff for ZwSystemDebugControl (NtSystemDebugControl)

Update: As somebody pointed out in the comments, there's a really good compilation of system call ordinals up at Metasploit's site.

Recon 2008

2008-03-04T03:37:00.004+01:00

So, it seems that after a year off Recon is coming back in 2008. I attended last time it was held and was one of the best conferences I've been to.

Old interviews with members of 29A

2008-03-02T22:03:00.003+01:00

The guys at Hispasec managed to dig up some old interviews with members GriYo and MrSandman of the legendary group 29A that recently announced was closing shop. Definitely worth a read if you can handle spanish.

badass debugger + badass toy = geek pr0n

2008-02-19T18:54:00.005+01:00

Today I finally got working a hacked-together minimal version of the iPhone debugger client for BinNavi. It's heavily based on Patrick Walton's (with HD's updates) weasel debugger. Once tied to BinNavi debug client framework the whole client-server interaction is trivial.

It feels just right, the best looking debugger together with the slickest device.. recipe for fun.. ;-)

The test application is telnet on the iPhone. On the iPhone's screen is the debug output from BinNavi's debug client. telnet is launched from an ssh session in OSX, where BinNavi is running.

For anybody trying to link Mach's debugging interface with a C++ iPhone application, remember the extern "C" when defining boolean_t exc_server(mach_msg_header_t *in, mach_msg_header_t *out); (which is not defined in the header files, as pointed in weasel's source code). Otherwise you'll get a nasty "Undefined symbols" message when linking.

extern "C" is also needed for catch_exception_raise(...) so exc_server can call it to handle exceptions. Documented here.
(I've used the standard iPhone toolchain on Debian, this is running on the firmware 1.1.3)

pydot 1.0.2 ... that took long

2008-02-14T23:53:00.002+01:00

Finally! it was long due. Here it is pydot 1.0.2

Some weeks ago I started updating the code to support all the attributes and enhancements in GraphViz 2.16. In attempting to make it pass all the regression tests some severe shortcomings it had became apparent.
pydot users had also provided with insight into how to improve performance by redesigning the way the data for the objects is stored internally. All in all, the limitations I was facing led me to rewrite the whole core of pydot, which took much longer than I wanted but I feel it was well worth it as it's orders of magnitude better than the last release 0.9.

Performance-wise the new pydot stores graphs and their objects using a hierarchy of nested dictionaries and lists. Graph, Node, Edge objects are mere proxies to the data and are created on demand. So that now it's possible to have a graph with a 1 million edges and there will not be a single Edge instance (only if requested, then they will be created on demand, mapping the data and providing with all the methods to act on the data in the global dictionary).
Storing a graph with 1 million edges in pydot 1.0 has approximately the same memory requirements (~813MiB) as dealing with one with only 40.000 edges in pydot 0.9 (~851MiB), the 40.000 edges graph needs ~35MiB in pydot 1.0 . Handling graphs should be much faster, as no linear searches are performed in pydot 1.0.2

Exe_Dump_Utility, a web-enabled pefile

2007-12-19T21:46:00.000+01:00

Gregory Piñero has put together Exe_Dump_Utility, a web-based version of pefile. Now it's possible to obtain the whole set of information processed by pefile online, without the need to install it. Neat!

xkcd: Python

2007-12-05T13:00:00.000+01:00

So true

Take Two: Packers, Time and Google Groups

2007-11-30T00:03:00.000+01:00

I just had to do it... This morning I read about chronoscope in a post in the Google Code Blog and I could not help myself from wanting to tinker with it.

I wrote a Mathematica function to export a time-series of the format (timestamp, value) into the dataset format used by chronoscope.

Epoch[date_] :=
  ToString[AbsoluteTime[DateList[ToString[date]]] -
  AbsoluteTime[DateList["1970"]]];

ChronoscopeJsExport = Function[ {datasetName, id, label, axis, data},
  jsData = datasetName <>
  " = {\nId: \"" <> ToString[id] <> "\", \n" <> "domain: [" <>
  StringJoin[ Riffle[ Map[ Epoch, data[[All, 1]] ], ", "] ] <>
  "], \n" <> "range: [" <>
  StringJoin[
    Riffle[ Map[ ToString, data[[All, 2]] ], ", "] ] <> "], \n" <>
  "label: \"" <> ToString[label] <> "\", \n" <>
  "axis: \"" <> ToString[axis] <> "\"\n};";
  jsData
];

And ran it through the packer time-series I harvested from Google Groups. Then I picked some widget demo code and put it all together in a mash-up. The results of the quick hack are here... much nicer to visualize than in the previous post. (and it's interactive!)

Use the mouse-wheel to zoom
Drag the plot left/right to browse around different date ranges
You can pick any packer and the data will be plotted against the previously selected one

xkcd: Network

2007-11-28T09:52:00.000+01:00

Simply brilliant.

pefile 1.2.8

2007-11-25T23:56:00.000+01:00

And yet another one. pefile 1.2.8 comes with the usual few bugfixes and a slew of enhancements. Some of them are:

One can now "relocate" the image by invoking relocate_image(ImageBase) with a new ImageBase the PE file's relocations will be applied to produce the relocated image.

Section entropy is computed faster (thanks to Gergely)

MD5, SHA-1, SHA-256, SHA-512 hashes are calculated on a per-section basis (thanks Jim Clausing for the suggestion)

Improved (rather fixed) handling of Unicode strings when parsing the resources information

For more details and downloads head to pefile's project page.

Right, Left, Right, Right, Left... and the Dancing Girl

2007-11-25T17:10:00.000+01:00

Lately a animation of a woman has been going around. The animation shows a rotating silhouette, the catch is that it can be perceived to be rotating clockwise or counter-clockwise. It tends to be a bit hard to change the perception of the direction of rotation once one particular direction has been recognized (at least in my personal case), I've read that for some people it switches direction more or less randomly, after looking at it for a while.
I was curious as to why it works, whether I could reproduce the trick and if I could make myself see her rotating in one direction or the other at will.

The why it works is relatively straightforward. Whether the rotation is clockwise or counter-clockwise is impossible to say if it happens in the same plane as where the viewer's viewpoint lays and there's no feeling of depth. The brain needs the perspective in order to tell the direction for sure, perspective will make the objects that are father look smaller and the ones closer bigger, that will help the brain discriminate one direction over the other. The dancing woman has been created in such way that it appears to have some perspective, yet it's still ambiguous (and you can see things jumping strangely at rotation as a result of this composition, just pay attention at the magical stretch of the arm closer to the body when it passes in front/behind)

It's easy to reproduce, just look at this example I quickly put together. With perspective it can be easily said whether it rotates in one direction or the other. We can either display it by setting the viewpoint above or directly in front with with a large aperture angle that exaggerates the perspective.

Then, if we now we set the viewpoint in front, yet so far that the projection lines become nearly parallel so that we lose the sense of perspective. It becomes much harder to tell the direction of motion and it's even possible to see it going both directions.

Then regarding the choosing at will of one direction over the other... I figured out that given that the only thing preventing my brain from deciding is the ambiguity caused by lack of information that would bias some layer of my neural networks to decide clockwise/counter-clockwise... I went really high tech and starting moving my finger in front of the dancing woman in the direction I wanted to see her to rotate... that seems to solve the ambiguity and I can make her turn one way or another at will...

I wonder if the trick works for other people too.

Packers, Time and Google Groups

2007-11-21T22:44:00.000+01:00

The other day I was talking with a friend and the discussion went into when certain anti-disassembly, anti-debug, etc. techniques might have appeared. That's bound to be difficult because tricks are usually simultaneously discovered by different people.

So I though, a trick will usually be regarded as "common" once it gets implemented in some packer, as those try to make analysis difficult and will attempt to embedded whichever tricks are good/popular within the underground at the time in order to make the reverse engineering process as cumbersome as possible. Therefore if I could somehow place packers in time I'd have a starting point...

That led me to remember about Google Groups. It's possible to make queries restricted to date ranges and the archives go back to 1981. I quickly put together a script to scan with a one-month window through 1981 to 2007 for a set of popular packers.

The most painful part of the whole process was to fool Google... they sure do not like robots... whenever they get a bunch of very simply automated queries they'll server back a "403 Forbidden" telling queries look like coming from a virus or spyware app...
But my script is good, it's no evil spyware... so I got into the mood of working my way around the checks. I needed to do quite some queries (> 10K) so I better make it believe I'm not a robot. Besides finding the right timing for the queries (too often will make Google sad) I had to distribute the search over a few hosts, randomize headers and User-Agents and the query itself (just throw in some randomized, "orthogonal" (nothing to do with your query) search terms). After that the script was good to go...

So, after mining the news groups for popular packer names ( the search string was, most of the time, " exe" plus the "randomized" terms ) I got a cute small data set to throw into Mathematica...

The results will have some inaccuracies, as it's possible some of the terms appeared in some news post not related to the packers. Yet I think they look plausible. When the volume of hits is high enough or constant over time it feels like it would indicate the approximate release date of the packer in question, or at least the first public discussion about it which, I would tend to think, will not necessarily be too far apart.
If someone can either corroborate or refute the data I'll be glad to hear.

I also did some test overlaying virus release times in order to try to spot correlations between big outbreaks and news-posts about packers, but I couldn't see anything particularly significant.

OpenRCE.org goodies

2007-10-12T13:47:00.000+02:00

By popular demand I've added OpenRCE.org T-Shirts and other goodies to the online store.

Hope you like them.

More iPhone hacking by HD Moore

2007-10-12T10:21:00.000+02:00

He's wrote more on hacking the iPhone on Metasplot's blog. Definitely worth a read. I'm dying to read part two.

Metasploit on the iPhone

2007-09-26T10:03:00.000+02:00

A nice write-up on the iPhone has been posted in Metasploit's blog.

My favorite point...

Every process runs as root. MobileSafari, MobileMail, even the Calculator, all run with full root privileges. Any security flaw in any iPhone application can lead to a complete system compromise. A rootkit takes on a whole new meaning when the attacker has access to the camera, microphone, contact list, and phone hardware. Couple this with "always-on" internet access over EDGE and you have a perfect spying device.

Hex-Rays unleashed

2007-09-18T09:19:00.000+02:00

Hex-Rays, Ilfak Guilfanov's decompiler, has been unleashed. I have had the chance of playing a bit with the beta and it is really impressive, to say the least. This will save so many hours to reverse engineers...

Un-bricking the Garmin GPSmap 60CS

2007-09-17T12:58:00.000+02:00

A while ago I managed to brick my trustworthy GPSmap 60CS when I was trying to update its firmware (don't try doing it from within a virtual machine if you can avoid it, unless you have a feeling for adventure ;) ).
It was bad enough that it would not even give any signs of life when trying to turn it on.

So I called Garmin Europe to see how much would it cost to get it repaired... and it was steep enough that I had to Google for a solution for a bit longer... and luckily enough I found a sneaky way of reloading the firmware that actually worked!

Now it's back alive!

Reverse engineering a compiler-produced artifact

2007-09-09T23:40:00.000+02:00

In our training, Pedram and I deal with some very simple compiler optimizations or artifacts. Although they represent the same semantics that the programmer defined in the original source code, those optimizations are sometimes cumbersome to convert back to a meaningful high-level representation.

The other day I was just studying a piece of code and bumped into a relatively common pattern. The code I was looking at was supposed to represent a division of a function's argument by a constant. But in the disassembled code I was studying I could only see a multiplication. This can be slightly confusing unless one has seen a bit more assembly than what is healthy and remembers some of the compiler-produced fun that goes on...

A couple of things to remember:

Compilers love to work with multiples of 2. The processor can can just shift registers left and right (shifting is incredibly fast, that is moving the contents of a register left or right padding with o or 1 as appropriate). Shifting to the left for multiplication by 2 and towards the right for division by 2 (this is akin to having a number in base 10 and multiplying by 10 by adding zeros to the right and dividing by by removing the rightmost digit).
Compilers hate to use the division instruction. The division takes a lot of steps, or cycles, for the CPU to complete. Hence they will avoid to use it at all cost.

The code looked like this:
(irrelevant interleaved code left out)

mov ecx, [esp+4+arg_4]
mov eax, 66666667h
imul ecx
sar edx, 3

In the snippet we can see function argument being multiplied by 0x66666667, and the result being stored as a 64 bit value in EDX:EAX (topmost 32 bits in EDX, the lower 32 in EAX)
Then the top 32 bits are shifted ("arithmetically") to the right. That is, divided by 2 thrice, same as 2^3 = 8. Effectively dividing the value by 8.
But the division is applied only to the top 32 bits, ignoring the lower 32. That could be understood to also mean that, by taking the topmost 32 bits and ignoring the bottom ones, the result of the multiplication is implicitly being divided by 2^32. (That's only guessed by the subsequent usage of the value just obtained, there's never again a reference to the lower 32bits, so I assume that they are discarded)

What do we have so far?

[ (Value * 0x66666667) / 2^32 ] / 2^3 ]

But, what's that 0x66666667? why to multiply by something so large and then divide?
The reason is that such computation allows the processor to keep most of the precision of the division it is trying to perform, still obtaining an integer in the end but without having to resort to using floating point arithmetic (which is far slower)

Let's do an example in base 10. Imagine that you only can multiply and divide by 10 (shifting numbers left and right) and we want to divide a number by 30. By shifting we can only divide by 10, 100, 1000, etc

But we have that: Value/30 = value * 1/3 * 1/10

Given that, represented as an integer, 1/3 would produce 0 we can "scale" it by multiplying by a large constant that later, once we are done, we divide by to get the value we're after. Given that the easiest for us is to multiply/divide by 10, we can "scale" 1/3 and make it 100000/3 which approximately equals 33333, which is a nice integer value. We would want to make this value as large as it fits in our registers in order to be as precise as possible. The bigger it is the more precision it will retain for subsequent operations.

Value/30 = ( Value*33333 ) / 1000000

Hence, we now have a clue now of where that 0x66666667 value might be coming from. Given that the processor works in base 2. We can assume that it's going to prefer multiples of 2. Also, given that it will try to obtain the largest value that fits in a 32bit register, that gives us an idea of the range of the power-of-two in use. We can get there with a bit of trial and error (We want to obtain an integer as a result of dividing a power of two by 0x66666667).

2.0^33/0x66666667 = 4.9999999982537702 ~= 5

Therefore:

0x66666667 ~= 2^33/5

So, in the end we get to

( [ (Value * 2^33)/5] /2^32 ) / 2^3

And with some algebra it simplifies to:

Value / (5*2^2) = Value/20

Effectively dividing the value by 20, without actually using the division instruction. That's to the extent that compilers will go to avoid using the division instruction...

Reverse engineering is fun isn't it?

Update: Given that this is a relatively old and well known optimization strategy it's only natural that it had been discussed before. It was just brought to my attention that Ilfak had blogged about a similar optimization and this chapter (PDF) from Hacker's Delight provides more details.

Visualizations of the Portable Executable Format

2007-08-28T00:32:00.000+02:00

I've always found that clear diagramming and laying out of complicated information makes it much more accessible and understandable.
When I started looking into the Portable Executable format I found it really helpful to lay out all the headers and structures I was trying to understand, to visualize how they relate to each other and the information they contain.
The resulting diagrams have been available under the corresponding section in OpenRCE for a some time already.

Now, given the feedback I received about some of those, I decided to put them up in an online store so people can get the real posters, high-resolution, updated and redesigned versions of those diagrams.

The Portable Executable Format poster has been updated to display the modified structures of the PE32+ format, while the Portable Executable Format. A File Walkthrough depicts the basic headers on top of a raw byte representation of an actual executable file.

PyDbg hacks

2007-08-23T18:30:00.000+02:00

Pedram just posted on his OpenRCE blog some awesome PyDBG hacks.

Google Sky

2007-08-22T16:49:00.000+02:00

Google has just released Google Earth 4.2 and it comes with an extremely cool new feature, Google Sky.
It provides layers for all kind of astronomical objects and astrophysical features like gravitational lenses. It even has the Ultra Deep Field image sets from the Hubble. This is simply gorgeous, I could play with it for hours...

pefile 1.2.7

2007-08-22T01:25:00.000+02:00

Just pushed out an updated version of pefile with some minor enhancements and fixes:

Added additional IMAGE_SUBSYSTEM_* flags
Added processing of the Optional Header's DllCharacteristics
Time/date fileds are now reported as UTC times
Added warning message for suspicious entry point addresses
Several minor parsing bugs fixed

Great Python overview

2007-08-21T20:29:00.000+02:00

Alex Martelli gave a talk in the Baypiggies meeting providing a great overview of Python. Check out the video and slides.

Black Hat Slides

2007-08-10T02:05:00.000+02:00

Although originally Halvar Flake and I were supposed to present together in a quick turbo-talk at Black Hat in Las Vegas, he unfortunately couldn't make it to the conference for reasons that have been already discussed.

I ended up sticking mostly to the original plan for the talk and presented some Python tools to automate reverse engineering and analysis processes.

I've just put the slides up here.

pefile 1.2.6

2007-08-10T01:04:00.001+02:00

It's finally here! Took longer than I expected because of all the enhancements and because I decided to move pefile to Google Code.

Besides access to the source code through their subversion server, they also have a really cute wiki.

I've added documentation and examples and it should make it easier for people to contribute ideas and improvements.

I introduced some of the new features of pefile in my turbo-talk in the last Black Hat in Las Vegas.

Besides some bugfixes, pefile-1.2.6 can now parse PEiD's signatures, it will report on the entropy of each section and will display more warnings for suspicious values found when parsing PE files. Just check the example dumps of Tiny PE and 0x90.exe

Information on how to use the PEiD signature matching can be found here. I posted a while ago on what can be done with the signature parsing.

Supercomputing done with style

2007-07-25T12:02:00.000+02:00

I was just reading through a list of the top 7 supercomputers and just saw this:

That's some style putting together a supercomputer, what a location!
It's the MareNostrum in Barcelona.

BlackHat Vegas is nearly here...

2007-07-14T20:46:00.000+02:00

I'll be there, teaching with Pedram a couple of rounds (weekend and week) of our training, Reverse Engineering on Windows: Application in Malicious Code Analysis. And then ranting together with Halvar in a turbo talk, 4 x 5

Pedram and myself "live on stage"

For the people more into cutting edge vulnerability research, Halvar will also be doing his Analyzing Software for Security Vulnerabilities. Feel free to grab any of us during the conference if you have any questions regarding BinDiff, BinNavi or VxClass.

And now that I am in the mood of advertising things, be sure to check OpenRCE's event calendar, you can even subscribe to the iCal feed. I try to keep it up to date with whatever events fall into my ears. If anyone knows of more, please let me know.

Windows XP and Bochs

2007-07-07T22:41:00.000+02:00

This might be of interest for anyone out there attempting to get Windows XP to install inside Bochs with no luck.

I had not been able to get it to install in any recent version, for one reason or another it always failed during install with a problem regarding the "catalogs" (error message was along the lines of "Setup failed to install the product catalogs. This is a fatal error."). No matter what options I had compiled in Bochs.

I had got it to install in the past... a long long time ago, so I figured out that I might get lucky with other versions of Bochs, so I started trying. 1.4.1 nothing, 2.0.2 nothing, 2.1.1 nothing (trying all of them with different configuration options, that took a while)... but finally got to 2.2.6 and bingo! it made it through with no errors! Once installed the image runs just fine in the latest incarnation of Bochs. Here it is, Bochs running Windows XP on Fedora running inside Parallels on my OS X...

Virtualization(emulation) = madness^2 ?

I hope this saves somebody from hours of compiling, recompiling and reinstalling...

Scanning data for entropy anomalies II

2007-07-06T02:59:00.000+02:00

Recently Phantal (aka Brian) left some comments on my blog and in OpenRCE on some calculations he did following up on my post Scanning data for entropy anomalies.

He develops the algorithm aiming at improving the execution speed of the entropy "scanner" example I had shown. I ran through his steps and arrived to the same conclusions he did on his latest comment. I just thought it'd be worth showing his work as a separate post rather than just a comment.

His idea is, by looking into the standard definition of entropy , to isolate all that doesn't change in the expression when the window slides and just update the entropy, instead of blindly recalculating it's value from scratch for each offset of the scan window.

Shannon' s entropy, usually represented as H, takes the following form if we work with the 256 possible byte values as the symbols :

where p(b) is the probability of the occurrence of a given byte.

H, the entropy, will tend towards its maximum value, 8, if the data has the maximum possible entropy. In such case the probability of each byte occurring would be the same which produces

Note that, although this is usually thought of as measuring the "amount of randomness", it is not that much the case. A sequence of bytes starting at 0 and increasing until 255 going through all the values in order would reach the maximum entropy value 8, even that it is all but random.

The probability of a given byte appearing in our window can also be expressed as . being the number of times the byte appears within the window and the width of the window.

The expression for the entropy can be expanded as follows

The entropy after sliding the window, , will have the same sum expansion except for two terms, the ones of the bytes going out and entering the window. We can then just update those and recalculate the expression by first removing the old values for the incoming and outgoing bytes and then adding the new values for both, after updating their count.

and that's all. Now on to some implementations in Mathematica and Python (but creating a Mathematica function with Pythonika). His implementation in C can be found in the comments of the previous post.

EntropyScan = Function[{Data, WindowScanSize},

  SummationTerm [Prob_] := If[Prob > 0, Prob Log[2, Prob], 0];

  (* Get the initial chunk and calculate the entropy *)
  CurrentChunk = Data[[ Range[1, WindowScanSize] ]];

  (* Calculate initial byte count and probabilities *)
  ByteCounts = Table[Count[CurrentChunk, i - 1], {i, 1, 256}];
  ByteProbs = Table[ByteCounts[[i]]/WindowScanSize, {i, 1, 256}];

  FilteredByteProbs = Select[ByteProbs, # > 0 &];
  H = - Total[
    Table[FilteredByteProbs[[i]] Log[2, FilteredByteProbs[[i]]],
    {i, 1, Length[FilteredByteProbs]}]];
  Entropies = {H};

  (* Slide the window and recalculate for incoming and outgoing bytes *)
  For[offset = 1, offset + WindowScanSize <= Length[Data], offset++,

    (* Get incoming and outgoing bytes *)
    ByteOut = Data[[offset]] + 1;
    ByteIn = Data[[offset + WindowScanSize]] + 1;

    (* Get the old probabilities *)
    OldValByteOut = SummationTerm[ByteProbs[[ByteOut]]];
    OldValByteIn = SummationTerm[ByteProbs[[ByteIn]]];

    (* Update counters and values *)
    ByteCounts[[ByteOut]]--;
    ByteCounts[[ByteIn]]++;
    ByteProbs[[ByteOut]] = ByteCounts[[ByteOut]]/WindowScanSize;
    ByteProbs[[ByteIn]] = ByteCounts[[ByteIn]]/WindowScanSize;

    (* Get the new probabilities *)
    ValByteOut = SummationTerm[ByteProbs[[ByteOut]]];
    ValByteIn = SummationTerm[ByteProbs[[ByteIn]]];

    (* Update the entropy *)
    H = H + OldValByteOut + OldValByteIn - ValByteIn - ValByteOut;

    Entropies = Append[Entropies, H];
  ];

  Entropies
];

Py["import math"]

EntropyScanPython = PyFunction["\<
def entropy_scan(args):
  data = args[0]
  window_size = float(args[1])

  summation_term = lambda p: p*math.log(p,2) if p>0 else 0

  current_chunk = data[:int(window_size)]
  byte_counts = [
    len(filter(lambda a:a==i, current_chunk))
    for i in range(256)]
  byte_probs = [byte_counts[i]/window_size for i in range(256)]

  H = -sum(
    [byte_probs[i]*math.log(byte_probs[i], 2)
      for i in range(256) if byte_probs[i]>0])
  entropies = [H]

  for offset in range(len(data)-window_size):
    byte_out, byte_in = data[offset], data[int(offset+window_size)]

    old_val_byte_out = summation_term(byte_probs[byte_out])
    old_val_byte_in = summation_term(byte_probs[byte_in])

    byte_counts[byte_out] -= 1;
    byte_counts[byte_in] += 1;
    byte_probs[byte_out] = byte_counts[byte_out]/window_size;
    byte_probs[byte_in] = byte_counts[byte_in]/window_size;

    val_byte_out = summation_term(byte_probs[byte_out])
    val_byte_in = summation_term(byte_probs[byte_in])

    H = H + old_val_byte_out + old_val_byte_in - val_byte_out - val_byte_in
    entropies.append(H)
  return entropies
\>"];

iPhone restore image on the loose

2007-07-04T10:34:00.001+02:00

Just getting my morning coffee and browsing through the news feeds I bumped into a post pointing to the iPhone's restore image. Apparently it's been making the rounds for a couple of days already.

On July 2nd there was a thread on Full Disclosure already discussing the contents. Of special interest seems the password protected 82MB image "694-5262-39.dmg", likely to contain the whole of iPhone's software. People seem to be already attempting to crack it open.

Also a couple of UNIX passwords for two user accounts were cracked but that might be of limited use.

The photos are from the iPhone launch in San Francisco on the 28th of June. That was something worth seeing.

Talk and visualization of third world statistics by Hans Rosling

2007-07-03T23:53:00.000+02:00

A friend just pointed me to a great talk, titled "Debunking third-world myths with the best stats you've ever seen", by Hans Rosling, the founder of Gapminder.

I like their dynamic visualization of statistical data. In my opinion, adding the time dimension to the data definitely allows to extract a better understanding of the evolution of the systems under study, and this is an excellent example.

The Powerset Demo Day

2007-06-29T07:58:00.000+02:00

I just had the luck of being invited, together with forty other people, to the first public demo of what the guys at Powerset are building up. The demo was in their headquarters in San Francisco. It was fairly impressive, to put it mildly.

They showed demos of all what they've been blogging about, like this or this and some other new applications.

I also had a chance to talk to some of the core Ruby developers, the search ranking engineers and the linguists working there and it's really an incredible team. One can smell the excitement floating in the air about what they do.

Steve Newcomb and others commented on different sides of their technology and the company itself. Mark Johnson introduced a just out of the oven Powerlabs, to give a taste of what's coming in September. It'll be possible to generate mash-ups using their natural language processing and understanding technology which, in my humble opinion, I think it's going to truly open the doors to a new generation of clever semantic tools. They aim to being really open about their system and to let people interact with it. Another side in which they also want to be open is in their contribution back to the open source community from which they build a lot of their infrastructure.

From seeing their demos, it really looks like they are are taking search, among other things, to a new level by not indexing keywords but actually indexing "concepts" that they extract by semantically parsing the searchable data. If they find "dog" in a sentence they will associate also "mammal", "animal", "pet" allowing for real abstraction when performing searches.
They combine that with normal search ranking techniques to obtain impressive results.

It really creates new ways of interacting with the data to be searched. Queries like "What politician died from a disease?" or "What disease killed a politician?" work flawlessly even when there are no references to die or kill in the text. Their natural language engine understands that "died from" equals to "killed by" and relates to "deceased" or "pass away". It really knows about those concepts abstractly and the semantic relations in the search query.

Powerlabs, the social site aimed at letting people play with their technology that will be launched in September, already has more that 10000 members signed up , so definitely there's a growing community interested in their developments.

During the Q&A sessions some very interesting topics popped up, like support for multiple languages and detection and resistance to spam (or text created by different models in order to appear human generated). Also, the "understanding" that they obtain from parsing a sentece could allow to better spam filtering, not just by spotting more or less likely-to-be-spam words, but actually detecting incoherent meanings or just uninteresting topics... just imagine having messages in your inbox automatically clustered by their real meanings, without having to specify a single rule (emails dealing with this, emails dealing with that...). The applications are endless...

I'm dying to play with it...

Powerset and the garden path

2007-06-22T19:15:00.000+02:00

I've recently bumped again into Powerset. I had previously heard about them when they got some people from PARC (if I remember correctly) and went into attempting to build something that I had always dreamed about. The guys at Powerset are tackling one of the hardest and most interesting (in my opinion) problems currently known, that is, helping computers process and "understand" natural language and use those results to make information more accessible. From my humble amateur-linguistic-aficionado point of view, they are doing it a great work there. Soon I will have a chance to see it live, first hand, and I can't wait.

In one of their latest posts they discuss some ambiguities that arise from using words with several meanings in contexts where the least used of the meanings is taken into use, leading to misunderstandings.

To put it in other terms, the problems arise when using the less known meanings of words in a way that the brain is misled when starting to read a sentence and leads to misunderstand the subsequent words (which can also have several meanings which depend on how one understood the start of the sentence) .
Normally, once the sentence has been read several times, the brain finally "switches" into the right interpretation of the different meanings of those words in a way that the whole construct becomes coherent.
I personally see it as resembling the visual phenomena where the brain interprets specially crafted images in different ways, switching back and forth between their different interpretations, like in the Young Girl-Old Woman Illusion or the Rabbit-Duck one.

In the case of these garden path sentences, as they are commonly called, the brain gets confused because of the dependencies between the words and their meanings.

As the brain starts reading a sentence, it will attempt to predict what follows, and it's amazingly good at that. The trick is to throw it off track by using words with multiple meanings.

In the example that they have as their post title "Search Engines Leaking Oil for Holes" the brain is tricked by taking the most common meaning of the first two words (a composite noun or collocation) and attempting to interpret it in a way that later becomes rather confusing when reaching "leaking oil".

Re-reading the sentence can lead to a second interpretation

In their post they ask how hard would be to find an automated way of generating such garden path sentences and they describe a pseudo-algorithm like the following:

You can make your own garden path sentences by following a few simple heuristics (...). The trick is to choose words that can act as both nouns and verbs, or as both adjectives and nouns, words like store, search, and post. Then follow the ambiguous word by another word that can take on more than one form. The hard part is to then add on another noun phrase that makes sense with the less common interpretation of the second word.

Trying to follow their heuristics, the first thing to do would be to find sets of words that can be both a noun and a verb or and adjective and a noun. Thanks to WordNet, PyWordNet and the mash-up of those and more provided by the guys from NodeBox that's not such a hard task as it would have otherwise been without such toolset.

Sets of words fulfilling those requirements can be build in a few lines of Python.

# Collect nouns, verbs and adjectives
verbs = set( wordnet.V.keys() )
nouns = set( wordnet.N.keys() )
adjectives = set( wordnet.ADJ.keys() )

# Pick the ones that can work both as nouns and verbs or as nouns and adjectives
noun_verbs = verbs.intersection(nouns)
noun_adjectives = adjectives.intersection(nouns)

print 'Found % d words that are both verbs and nouns' % len(noun_verbs)
print 'Found % d words that are both adjectives and nouns' % len(noun_adjectives)

Found 4096 words that are both verbs and nouns
Found 3138 words that are both adjectives and nouns

I will also need to have some means of knowing which words are more likely to follow a given one. For that I will reach into some datasets I collected years ago for some computational linguistics experiments I did. Using a small corpora of 2.071.007 sentences built out of books from the Project Gutenberg and parsing it through some Python code I obtained 16.057.624 word pairs, 2.365.383 of them unique. That will provide me with some numbers on what words are likely to follow others.

I can now look for frequently used words that can be both nouns and verbs. In the following line "occurrences" is a list containing all the words and the number of times they appear. They are filtered to only show the ones that are both nouns and verbs.

print [word for word in occurrences[:300] if word[0] in noun_verbs]

{{"be", 10070}, {"have", 7827}, {"like", 6577}, {"will", 6201}, {"out", 5422}, {"still", 4136}, {"even", 4049}, {"man", 3957}, {"can", 3866}, {"down", 3376}, {"see", 3104}, {"do", 3097}, {"time", 2729}, {"people", 2663}, {"well", 2602}, {"last", 2581}, {"back", 2337}, {"white", 2250}, {"make", 2088}, {"till", 2083}, {"come", 2048}, {"black", 2021}, {"general", 2004}, {"found", 1935}, {"light", 1918}, {"round", 1910}, {"go", 1880}, {"better", 1815}, {"face", 1755}, {"saw", 1742}, {"lay", 1740}, {"work", 1682}, {"form", 1678}, {"let", 1673}, {"right", 1654}, {"set", 1647}, {"lord", 1621}, {"look", 1579}, {"take", 1577}, {"hand", 1574}, {"head", 1546}, {"full", 1544}, {"best", 1538}, {"put", 1534}, {"state", 1531}, {"party", 1522}, {"love", 1517}, {"place", 1493}, {"house", 1491}, {"say", 1440}, {"get", 1401}, {"part", 1386}, {"water", 1385}, {"name", 1384}, {"second", 1370}, {"give", 1344}, {"felt", 1342}, {"present", 1327}, {"fell", 1320}, {"land", 1319}, {"use", 1311}}

Now given a word it's possible to find other words that would often follow it and can also have several functions. For instance, lets see what comes out for "look":

# Pick words following 'look' that can be both nouns and verbs
succeeding_words = [p for p in word_sparse['look'].items () if p[0] in noun_verbs]
# Sort them by the most frequently used to the least
succeeding_words.sort ( lambda a, b : -1 if a[1] > b[1] else 0 if a[1] == b[0] else 1)
print succeeding_words[: 100]

"[('like', 255), ('out', 185), ('down', 124), ('back', 115), ('forward', 82), ('round', 49), ('well', 42), ('pale', 26), ('better', 16), ('black', 9), ('right', 7), ('full', 6), ('white', 5), ('blue', 5), ('grave', 5), ('even', 4), ('still', 4), ('double', 4), ('cross', 4), ('close', 3)]

And the results for "form"

[('name', 185), ('part', 18), ('can', 8), ('saint', 8), ('like', 7), ('see', 5), ('will', 5), ('state', 4), ('ice', 3), ('till', 3), ('lay', 3), ('french', 3), ('people', 3), ('found', 2), ('out', 2), ('put', 2), ('well', 2), ('note', 2), ('black', 2), ('starch', 2)]

Although not being a native English speaker makes this a tiny bit more challenging, I can see how one could play with combinations like "look, like", "look, still", "look, well", "form, name", "form, like", etc. to build slightly confusing sentences.

Collocations also are great to mislead the brain whenever one of the words has more than a meaning ("visitor center", "search engines", "meeting point") .
A quick hack to try to spot some automatically could be to look for pairs of words often appearing together and having the desired properties of fulfilling more than one function.
But given the low quality results in the shown next; one could, for instance, also take into account the relative frequency of a noun-noun compound as compared to other pairings of the nouns, to try to see how much more often those two words appear together than with others. There's extensive literature on how to improve this and this was meant as a short-ish blog post after all.

print [ p for p in word_pairs_occurrences[:10000] if en.is_noun(p[0][0]) and en.is_verb(p[0][0]) and en.is_noun(p[0][1]) and en.is_verb(p[0][1]) ]

{{will, be}, {can, be}, {labor, force}, {will, have}, {be, found}, {can, do}, {come, back}, {exchange, rate}, {will, do}, {will, make}, {can, read}, {prime, minister}, {will, go}, {will, give}, {have, come}, {come, out}, {go, back}, {right, hand}, {set, out}, {go, out}, {find, out}, {can, see}, {will, come}, {come, down}, {will, take}, {have, found}, {short, form}, {will, tell}, {birth, total}, {get, out}, {go, down}, {land, use}, {be, put}, {can, tell}, {father, brown}, {will, find}, {white, man}, {put, out}, {take, care}, {can, get}, {dare, say}, {will, see}, {can, make}, {be, well}, {short, time}, {can, have}, {found, out}, {lay, down}, {second, time}, {be, better}, {be, read}, {can, think}, {go, home}, {lord, will}, {birth, rate}, {hoist, side}, {meter, gauge}, {ftp, program}, {be, true}, {be, like}, {last, time}, {look, like}, {will, say}, {man, can}, {set, down}, {license, fee}, {come, home}, {can, find}, {make, out}, {put, down}, {give, notice}, {can, say}, {be, cut}, {take, place}, {low, voice}, {will, try}, {cast, out}, {get, index}, {have, put}, {lie, down}, {can, go}, {radio, relay}, {still, be}, {will, get}, {be, ready}, {well, be}, {wait, till}, {get, back}, {tax, return}, {free, copyright}, {fell, down}, {can, copy}, {set, bin}, {have, felt}, {look, out}, {be, out}, {form, name}, {satellite, earth}, {burst, out}, {will, keep}, {be, free}, {can, give}, {double, track}, {people, have}, {cut, down}, {will, show}, {fish, catch}, {turn, out}, {carry, out}, {well, have}, {work, force}, {be, set}, {have, set}, {miss, garland}, {will, put}, {can, take}, {do, well}, {let, go}, {mine, hand}, {earth, station}, {fell, back}, {take, heed}, {short, distance}, {air, force}, {can, help}, {will, help}, {cry, out}, {will, let}, {free, state}, {feel, like}, {will, cause}, {present, time}, {will, think}, {be, present}, {will, return}, {cast, down}, {black, man}, {narrow, gauge}, {bulletin, board}, {man, be}, {be, right}, {dry, tree}, {will, set}, {be, back}, {point, out}, {right, side}, {can, come}, {look, down}, {will, call}, {run, down}, {file, size}, {major, transport}, {labor, party}, {be, content}, {will, leave}, {man, will}, {will, look}, {can, use}, {need, be}}

Definitely the problem is very challenging with current tools, but it's always fun to give it a spin. With a few hours and limited tools I could only get to think of some ways to find good candidate words for garden path sentences. Definitely nowhere close to actually completing full sentences.

It would be great to expand on this toy research and make it actually useful and interesting. Using larger data sets (like this Google data set) from which to extract word relationships would be a good way to start. Having statistics for trigrams, fourgrams, etc. of words would make things better, having more contextual information would be possible to get more meaningful constructs by ensuring that the chosen words occur close within a small context.

I can think of more ways of improving it, most of them involving large datasets and lots of computational power... gosh, I'm getting carried away thinking about this...

Looking forward to Powerset letting people play with their tools, I' m sure that implementing ideas like the one discussed in this rant will become much easier.

Safari 3.0.1 for Windows

2007-06-14T23:31:00.000+02:00

Apple released today an update for the recently unleashed Safari for Windows attempting to fix some of the problems promptly uncovered after the inital release.

I wanted to take a look at all the changes in the release. After comparing the hashes of all executable modules the following were identical to the ones included in Safari 3.0.0:

coregraphics.dll, icudt36.dll, icuin36.dll, icuuc36.dll, libtidy.dll, libxml2.dll, libxslt.dll, javaplugin.jar, pthreadvc2.dll, sqlite3.dll, zlib1.dll

Of all the remaining, whose hashes were different, the following proved to be structurally identical after running them through BinDiff:

safariresources.dll, cfnetwork.dll, safaritheme.dll, pubsubdll.dll, npjavaplugin.dll, corefoundation.dll

So that leaves us with the stuff to focus on:

safari.exe and webkit.exe

The dependencies between this modules are show in the following graph (green=identical hash, black=hash changed but structurally identical, red=the meat)

Within those two modules, BinDiff finds a handful of functions changed, what could possibly have been fixed... ? ;)

BinNavi: Simplifying code II. The implementation

2007-06-14T07:27:00.000+02:00

Following up the ideas presented on a previous post, this one will discuss their implementation.

We wanted to write some code that would help us understand the meaning of certain operations encoded in assembly instructions. We wanted to do that by going through the instructions and building extended expressions representing the relations between the input and output values of a basic block.

In order to implement the desired functionality, we will write some code to replace parts of the operand trees with existing, known values, from the already analyzed code.

The code will iterate through the operand tree until it finds leafs (labeled as ’l’)
The leafs will then be substituted if their contents are known
The new, augmented, operand tree will be returned

The main part of this simple operand expression "builder" is to parse instructions and keep track of the values of the operands for the current instruction. If any values can be augmented by including the operations performed in previous instructions, those will be replaced by the result of such operations.

The function will take a dictionary (aka associative array) instance that will map the registers and memory addresses to their known contents
It will parse each instruction and update the dictionary according to the semantics of the opcode which in this example won't be totally accurate but will suffice to illustrate the process
It will return an updated dictionary with the results of tracking the values

This first function will retrieve an operand tree and return it as a nested list.
(The code presented in this post makes use of BinNavi's Python scripting to access operands, instructions, etc)

# Retrieve an operand as nested lists
#
def get_optree(op, parent=None):
    if not parent:
        return get_optree(op, op.get_roots()[0])[0]
    tree = []
    children = op.get_children(parent)
    for child in children:
        subtree = get_optree(op, child)
        if subtree:
            tree.append( [str(child), subtree] )
        else:
            tree.append( [’l’, str(child)] )
    return tree

And the result of running it on the following instructions will be:

004A4703: lea b4 ecx,b4 ss [ebp+4294967240]
004A4706: push b4 26
004A4708: sub b4 eax,b4 ecx
004A470A: pop b4 ecx
004A470B: add b4 eax,b4 13

[[’l’, ’ecx’],
[’ss’, [[’[’, [
[’+’, [[’l’, ’ebp’], [’l’, ’4294967240’]]]]]]]]
[[’l’, ’26’]]
[[’l’, ’eax’], [’l’, ’ecx’]]
[[’l’, ’ecx’]]
[[’l’, ’eax’], [’l’, ’13’]]

The following function is in charge of substituting any values in the tree for which their content is known. For example, if ecx is used as a source operand but in a previous instruction is was assigned a value of 10, we would substitute ecx in the tree with 10.

# Substitute the leafs in the operand tree
# with the last values obtained during the
# analysis
#
def substitute_leafs(tree, operand_trees):
    if not isinstance(tree, list):
        return tree
    if isinstance(tree[0], str) and tree[0]==’l’:
        return operand_trees.get(str(tree), tree)
    else:
        for idx, elm in enumerate(tree):
            tree[idx] = substitute_leafs(elm, operand_trees)
    return tree

The next function will be responsible for parsing the semantics of the instructions (in a merely illustrative way, it's far from implementing strictly the semantics of the Intel instructions used here). Instructions responsible for basic arithmetic and assignment operations will be processed and their results stored in the operand tree that we are processing.

# Parse instructions

def parse_instruction(insn, stack, operand_trees=dict()):
    # Process the MOV instruction
    if insn.mnem == 'mov':
        # Get the operands
        dst, src = map(get_optree, insn.operands)
        # Perform substitutions for any known values
        src = substitute_leafs(src, operand_trees)
        # Assing the source operand to the destination
        operand_trees[str(dst)] = src
    elif insn.mnem == 'add':
        dst, src = map(get_optree, insn.operands)
        dst_expr = operand_trees.get(str(dst), dst)
        src = substitute_leafs(src, operand_trees)
        src_expr = operand_trees.get(str(src), src)
        operand_trees[str(dst)] = ['+', [dst_expr, src_expr]]
    elif insn.mnem == 'sub':
        dst, src = map(get_optree, insn.operands)
        dst_expr = operand_trees.get(str(dst), dst)
        src = substitute_leafs(src, operand_trees)
        src_expr = operand_trees.get(str(src), src)
        operand_trees[str(dst)] = ['+', [dst_expr, ['-', [src_expr]]]]
    elif insn.mnem == 'lea':
        dst, src = map(get_optree, insn.operands)
        src = substitute_leafs(src, operand_trees)
        src_expr = operand_trees.get(str(src), src)
        operand_trees[str(dst)] = src[1][0][1][0]
    elif insn.mnem == 'push':
        dst, src = ['l', 'STACK_TOP'], get_optree(insn.operands[0])
        src = substitute_leafs(src, operand_trees)
        src_expr = operand_trees.get(str(src), src)
        stack.append(src)
    elif insn.mnem == 'pop':
        dst, src = get_optree(insn.operands[0]), ['l', 'STACK_TOP']
        src = stack.pop()
        operand_trees[str(dst)] = src
    elif insn.mnem == 'cdq':
        pass
    elif insn.mnem == 'idiv':
        src = get_optree(insn.operands[0])
        # dst is always eax
        dst = ['l', 'eax']
        src = substitute_leafs(src, operand_trees)
        src_expr = operand_trees.get(str(src), src)
        orig_src = operand_trees.get(str(dst), dst)
        operand_trees[str(dst)] = ['*',
            [orig_src, ['INV', [src_expr]]] ]
        operand_trees[str(['l', 'edx'])] = ['%',
            [orig_src, ['mod', [src_expr]]] ]
    return operand_trees

After running the previous functions on a simple basic block, we will obtain operand trees representing the operations performed by the instructions on their operands.

In the end we can ask the value of a register and we will see, in a single extended expression, the value of the register as a result of all operations taking place in the basic block.

>>> operand_trees["['l', 'ecx']"]
['l', '(0x1A, 26)']

>>> operand_trees["['l', 'al']"]
['ss',
  [['[',
    [['+',
      [['l', 'ebp'],
      ['%',
        [['+',
          [['+',
            [['l', 'eax'],
            ['-', [['+',
              [['l', 'ebp'],
              ['l', 'lowercase_chars']]]]]]],
          ['l', '13']]],
        ['mod', [['l', '(0x1A, 26)']]]]],
      ['l', 'lowercase_chars']]]]]]]

We can also graphically represent the result of the operand composition in BinNavi.
The following function will traverse the operand tree and add the nodes to the graph instance provided.

# Convert nested lists into a graph object
# that can be visualized
#
def tree_to_graph(tree, g=Graph(), parent=None):
    if isinstance(tree[1], list):
        node_str = tree[0]
        node = g.add_node(str(node_str))
        if parent:
            g.add_edge(parent, node)
        for child in tree[1]:
            tree_to_graph(child, g, node)
    else:
        leaf = g.add_node(str(tree[1]))
        if parent:
            g.add_edge(parent, leaf)
    return g

And running it on one of the composed operand trees results in:

tree_to_graph(operand_trees["['l', 'al']"], Graph()).show()

As a last step we can represent the expression textually and add it to the node.

# Convert nested lists into a single expression as
# a string
#

def tree_to_expression(tree):
    root = tree[0]
    children = tree[1]
    if isinstance(children, list):
        parsed_children = [tree_to_expression(child) for child in children]
    else:
        return children
    if root == '*':
        return '(' + '*'.join(parsed_children) + ')'
    elif root == '+':
        return '(' + '+'.join(parsed_children) + ')'
    elif root == '-':
        return '(-' + parsed_children[0] + ')'
    elif root == '[':
        return '[' + ' '.join(parsed_children) + ']'
    elif root == 'INV':
        return 'INV('+parsed_children[0]+')'
    elif root == 'mod':
        return parsed_children[0]
    elif root == '%':
        return '(' + parsed_children[0]+'%'+parsed_children[1] + ')'
    elif root == 'l':
        return children
    elif root == 'ss':
        return parsed_children[0]

And running the following in BinNavi's Python interpreter will generate the expression and add it to the basic block as a comment in BinNavi's view.

fgv = Graphview.get_open_views()[0]
expr = tree_to_expression(operand_trees["['l', 'al']"])
fgv.set_node_comment(0x4a4703L, expr)

The resulting expression represents the ROT13 operation for lowercase characters. (The code belongs to the Mydoom.D worm)
ebp+lowercase_chars points to a string containing the alphabet. eax is the pointer to the character to encode/decode within the alphabet, and its position is calculated by subtracting the beginning address of the alphabet from eax. Then 13 is added to it and wrapped around by getting the value modulo 26 and added again to the beginning of the alphabet to obtain the final decoded/encoded character.
This expression is a nice and condensed form of expressing the functionality implemented in the basic block.

This post illustrates the versatility of a properly abstracted representation of the operands and instructions and the level of automation that can be achieved in a relatively simple manner with the help of the interactive Python scripting in BinNavi.

VxClass. Automated executable classification

2007-06-11T19:02:00.000+02:00

Here at Sabre Security, we have been putting together a variety of technologies nurtured and developed over years of reverse engineering and malware analysis. All of it has taken form in VxClass, which is finally shaping up. We are quite proud and happy to see such a complex project (and one I've personally long dreamed about) actually working.

The first incarnation of VxClass is already able to automatically handle (unpack, analyze and classify) a wide range of Windows malware. The results are nearly addictive to look at. Accompanying this post are some screen-shots of the Web interface showing a listing of files and the automatically generated cluster of families.

VxClass will allow analysts or other tools to communicate with it and submit executable files. Those will be unpacked, analyzed and classified automatically according to their structural properties. The classification results as well as analysis databases can be retrieved through either an XMLRPC or a Web interface.

If you or your company would be interested on evaluating it or discussing if it might be an applicable technology for you, don't hesitate in dropping us a mail.

Also, I'm in the Bay Area for most of June and August (also in Black Hat in Las Vegas). So if you want to have me demo it or chat about it, drop me a line.

pefile and packer detection

2007-06-09T00:19:00.000+02:00

I've always wanted some tool that I could run over large collections of executable files and would tell me what's packed and what's not and, ideally, also the packer. PEiD has wonderful signature libraries but my ideal tool would be easier to integrate with other components and not restricted to Windows.
The guys at OffensiveComputing had put together some code to, by making use of PEiD signatures and pefile, recognize packers.

I've decided that it's time for pefile to have such functionality by default and I've reimplemented the signature parsing and matching. The next version of pefile should include this new code.
I've also found some pretty extensive signature libraries and here are some of results of the test runs in some files I've laying around.

Of the 48.025 files (all malware) that I scanned, in ~42% no packer could be found using the current signature database. In the remaining ~58% the tests found 227 different packers and compiler signatures.

A more extense listing of the most frequently found packers looks like:

Given that I've run pefile in several tens of thousands of pieces of malware with all kind of exotic PE format contortions, I've managed to find and fix a couple of obscure bugs. The forthcoming release will be even stronger when facing files that push the limits of the PE format well-formedness.

Cool train

2007-06-08T11:20:00.000+02:00

Yesterday, on the way back from an exam in Cologne, my seat was just behind the cockpit on one of the ICE trains. They look pretty sleek elsewhere in the train but the cockpit is nothing short of Star Trek. Although this trip was just from Cologne to Bochum and just takes an hour, in longer journeys (to Frankfurt for instance) this trains get to go nearly at 300 Km/h (~186 mph).

As well, they seem to be running trials of a wireless service and they had this cool live map with the current location.

Parallels Desktop 3.0

2007-06-01T11:13:00.001+02:00

The Parallels guys just announced the soon to come version 3.0. This one will have, among a good set of new and long awaited features, a snapshot manager. I can't wait for that, it's going to make my life so much easier.

Useful VB IDC script

2007-05-31T12:28:00.000+02:00

Pierre just posted in Datarescue's forum about an IDC script for IDA contributed by Reginald Wong.
The script will run through Visual Basic executables and parse a good deal of object and form information. This will probably be interesting for anybody having to reverse engineer VB applications in IDA.

The script also points to some interesting resources:

Visual Basic Image Internal Structure Format by Alex Ionescu

Visual Basic Reversed - A decompiling approach by Andrea Geddon

Google talk about reverse engeering to find security vulnerabilities

2007-05-24T15:52:00.000+02:00

Great video of a Google talk by Alexander Sotirov on reverse engineering to find security vulnerabilities.

Inkling market on the GDP growth of China vs. Germany

2007-05-21T22:45:00.000+02:00

One of the markets from all the interesting ones in inkling markets tries to predict whether China's GDP will grow beyond Germany's by 2008 (not adjusted for PPP). My first reaction based on a very generic feeling I got from reading general media was of slight surprise that it wasn't already bigger, given all the incredible economic metrics China keeps pushing out.

But taking a look at some historic data and doing some projections actually makes it seem quite difficult to happen, as Germany's economy is supposedly slowly picking up and rumors have it that China's economic growth will eventually cool down and flatten out. Even if all were to continue at its current rate, it does not seem all that likely that China's GDP would surpass Germany still for a few years.

ida2sql, exporting IDA's dissasemblies to SQL

2007-05-20T16:33:00.000+02:00

Because BinNavi nowadays reads all the disassembly information from a SQL database, we needed some means of exporting the information to it. ida2sql is the result, it is a monster set of Python scripts I wrote (all nicely wrapped in a couple of files for easy installation) that will export the information from an IDB (only Intel, ARM and PPC so far. The latter two in experimental mode) into a MySQL database.
It's available for download from my site together with installation and usage instructions. It needs the IDAPython plug-in to run.

Any feedback is welcome.

I posted a while ago about the database schema. One can do pretty neat things when having the dissasembly in such form...

pefile-1.2.5 released

2007-05-20T16:09:00.000+02:00

Besides some small fixes the new release of pefile is able to report suspicious or malformed entries encountered while parsing. Any time that a non-critical (something that wouldn't prevent the file from running) problem is found it's added to a list of warnings that can be retrieved get_warnings() or shown show_warnings()

Example warning messages:

Suspicious NumberOfRvaAndSizes in the Optional Header. Normal values are never larger than 0x10, the value is: 0xdfffddde

Error parsing the import table. Invalid data at RVA: 0x400000

Error parsing the Import directory. Invalid Import data at RVA: 0x60

Error parsing export directory at RVA: 0x6c6c642e

BinNavi: Simplifying code

2007-05-15T19:43:00.000+02:00

This in a slightly expanded example of what Halvar and I showed in the Sabre-Security trainings last October. With it I illustrate how to, in BinNavi, build a small dataflow analyzer in Python so that it can do some work for us.

Lets imagine that there's a basic block like the following, which is part of the ROT13 routine in the Mydoom worm.

Ideally we would like to have such tool what would spare us from having to figure out what the assembly code in that block does, regardless of how simple it might be in this case. Such functionality would make analysis much less tedious and less error prone.

Given that BinNavi offers an interactive Python interpreter with access to every single function, basic block, instruction, operand and expressions within an operand, this is a problem that we could try to address with some scripting.

The dataflow reconstruction

Assembly instructions move data around and modify it by performing different operations on it. Values are fetched from registers, memory or specified explicitly and the results are stored back in registers or memory.

So, if we would like to be able to transform the operations into something more readable we could track all the assignments and operations and compose some sort of combined expression. That's actually much easier than it might sound like.

Taking advantage of Python's dictionaries (aka hashes, aka associative arrays, aka maps) we could proceed as follows.

We process individually each instruction. As each has different semantics, we have to deal with them individually (some groups of similar instructions can be handled somehow generically tho.).

First the instruction might push or pop something from the stack and we could emulate that behaviour with a simple list.

Second the instruction might get data from a source register/memory/immediate and, upon operating on it, assign it to a destination operand. We will just consider these two cases for the time being, this is only an exercise after all ;)

In this case we could store in our dictionary the source item using as key the destination register or memory location. That would allow us to keep the current state of the registers and memory locations. For instance: lea eax, [edi+2]

The destination of the lea operation is eax the source/result is [edi+2]. Hence we could have an entry in our dictionary that maps eax to edi+2, there are no brackets here because lea will store the value of the addition, not its contents, so we don't need the brackets (which stand for memory-dereference)

If a register is accessed later on by another instruction and we know where the value comes from (because we stored the previous assignment operation in our dictionary) we could compose it together, having a combined expression that tracks all data. Following the example, imagine the instruction mov bl, [eax+esi], that will put into bl the value pointed to by [eax+esi], in our dictionary we would map bl to have the value [eax+esi]

But, if we would check if we have any of the expression parts already in our dictionary (remember that eax maps to edi+2) then we can expand [eax+esi] into [edi+2+esi]. One might be able to infer now that this procedure can be taken further, composing longer expressions that would reduce the amount and complexity of the assembly code to look at.

As another illustrative example, one could assign the the AL key in the dictionary the nested lists representing the expression tree for the memory reference as seen in the following figure.

As just seen, operands are treated as graphs by BinNavi. That allows for a versatile manipulation of their contents. We can transform the tree to nested lists and operate on those.

The form of the nested lists is

root

child_1

child_2

Where each children has the same form

For instance, the following two operand trees

would take the following from as nested lists:

[b4, [ecx]]

[b4, [ss, ['[', [+, [ebp], [4294967240]]]]]

In short, in order to do basic reconstruction of the data flow the following operations need to be performed:

Keep track of all assigned values using a dictionary

Substituting leafs with values already in our dictionary

Perform substitutions and assign SRCs to DSTs hence updating the dictionary to the last state

In a following post, I'll show an implementation using BinNavi's embedded Python interpreter.

Scanning data for entropy anomalies

2007-05-12T14:54:00.000+02:00

l0re just asked the following question in the OpenRCE forums:

I'm currently searching for a tool that does an entropy analyse. I want it to use it for finding a RSA key in a binary file. I have seen a tool that could do this on a workshop but unfortunately I don't know the name of tool and I can't find it with help of google. Does any one know the name of the tool or a tool that could do this?

I'm don't know of such tool from the top of my head although PEiD and OllyDBG both do statistical tests in order to detect possibly compressed/packed executables.

But having to come up with such things is one of the reasons why I love Python and Mathematica+Pythonika. With both it's possible to put together, in a few minutes the desired functionality.

So, the idea is to spot the typical high entropy that should be exhibited by something like a RSA key stored in binary form. Assuming that it's stored within data with significantly lower entropy, such as a standard executable file (that is, not packed or compressed itself), it should be easy to spot visually. Let's check...

First we need a function that calculates the entropy of a given chunk of data. The following code will take a Python string and calculate it's byte entropy, returning a real number in the range 0.0 and 8.0.
Values close to 8.0 would indicate a high entropy, hence the likelihood of compressed or otherwise highly random data. Low values would indicate low complexity data such as text or executable instructions or any other data exhibiting clear patterns.

import math

def H(data):
  if not data:
    return 0
  entropy = 0
  for x in range(256):
    p_x = float(data.count(chr(x)))/len(data)
    if p_x > 0:
      entropy += - p_x*math.log(p_x, 2)
  return entropy

Next we want to be able to take a chunk of data and run the entropy calculation function all across it, on byte increments, with a defined block size. Starting from the byte at offset 0, we will calculate the entropy of each data chunk of the given size and return it's value. The function is an iterator so that we can easily get a list of entropies for all offsets that we can next feed into a plotting function.

def entropy_scan (data, block_size) :
  for block in (
    data[x:block_size+x]
    for x in range (len (data) - block_size) ):

    yield H (block)

Now we need some test data, the following code will generate a low-entropy chunk of data 1024 bytes long, followed by a high-entropy one (assuming the random generator is good enough, which is the case for the example) also 1024 bytes long and closing with 1024 bytes more of low entropy data.

data = ''.join (
  [chr (random.randint (0, 64)) for x in xrange (1024)] +
  [chr (random.randint (0, 255)) for x in xrange (1024)] +
  [chr (random.randint (0, 64)) for x in xrange (1024)] )

If we run the Python code within Mathematica

ListPlot[ Py["\<
list(
entropy_scan( data, 256 ) )
\>"] ]

we obtain the following plot

displaying a noticeable bump in the region where the higher entropy data lies within our test data.

Update:

Deadhacker has posted an augmented version of my hack that does not rely on Mathematica in addition of being able to run on arbitrary files passed as arguments to his script.

Training in BlackHat Tokyo

2007-05-09T22:37:00.000+02:00

Pedram and I will be teaching our Reverse Engineering on Windows: Application in Malicious Code Analysis training in the Black Hat Japan Spring Training 2007 in Tokyo on May 28th and 29th.

The Annapurna circuit the Google Earth way

2007-05-07T23:46:00.000+02:00

I was playing a bit with Google Earth this evening.

Threw in the GPS data I recorded during the trip, some pictures from the Google Earth community and got something like this.

Back from Nepal

2007-05-07T15:11:00.000+02:00

What an incredible trip. Nepal and trekking along the Annapurna Circuit has been one of the greatest trips I've done.

The scenery, the people, the contrasts, the physical challenge and the remoteness made it quite an unforgettable experience. The pleasure of being away and cutting off from the daily routines, and everything in general, in such a drastic manner is something wonderful and truly difficult to value enough.

Some random highlights and impressions of the trip were:

Kathmandu is chaotic beyond belief (I've been told Delhi is even worse, need to see it to believe it)

Pokhara is an active "little" city, more pleasant than Kathmandu in my opinion, and very touristic (lots of treks start and end there)

The mountains are big enough to make words have a hard time to describe them

The Thorong La pass is hard but totally doable if reasonably fit (the views of the Mustang valley during the descent are nothing short of breathtaking)

When they say it's windy in the stretch from Muktinath to Jomsom they really mean it!

Tatopani (meaning "hot water" for its hot springs) is a gorgeous, lovely little town, one must stop there

Catching the sunrise from Poon Hill is a must-do if one takes the path up to Gorepani

Paragliding in Pokhara after the trek is a good way of wrapping up the trip

Also... on the way back we had a long layover, 13 hours, in Bahrain (Gulf Air were cool enough to give free hotel and transportation to Manama for the day!). The National Museum is fairly good, the exhibit on the Dilmun civilization is really enlightening.

There are other curious pieces, like some mathematics book from the 18th century, apparently covering some geometry concepts.

I've got hooked, now I must go back and do the Everest circuit trek...

Trekking in Nepal

2007-04-08T00:50:00.000+02:00

In a matter of hours, after a probably short night sleep, I'll be traveling to Nepal, taking nearly a month off to go trekking in the Annapurna region. I'm sure it's going to be an incredible experience.

I'll be back on the beginning of May.

IDAPython 0.9.0

2007-04-03T22:50:00.000+02:00

Gergely Erdélyi has just put out the last release of IDAPython, 0.9.0

It can be found in his site together with some brief release notes. This release supports Python 2.5 among a good deal of other enhancements and additions.

A PE trick, the Thread Local Storage

2007-03-20T09:06:00.000+01:00

In our training, when discussing packer's tricks and the PE file format, Pedram and I talk about different ways of executing code in a PE executable before the entry point (the one pointed to by the AddressOfEntryPoint field of the Optional header) is given control by the Windows loader.

One possible way of achieving it is to use the TLS directory entry of the PE file format's headers. TLS stands for Thread Local Storage and it's meant to be used to allocate storage for thread-specific data. The TLS structure, IMAGE_TLS_DIRECTORY, pointed to by the TLS directory entry has a small number of fields. The one of special interest is the one pointing to a list of callbacks, AddressOfCallBacks.

During the class I got asked how one would implement the functionality required to get code to run, called from the TLS callbacks. I had a rough an idea how the implementation would go but had never tried implementing it myself. So before the training was over I started to look into it and finally, a few days later, got it to work.

So, in order to put this together the first thing I did was to dig up the definition of the IMAGE_TLS_DIRECTORY structure.

typedef struct _IMAGE_TLS_DIRECTORY {
    UINT32 StartAddressOfRawData;
    UINT32 EndAddressOfRawData;
    PUINT32 AddressOfIndex;
    PIMAGE_TLS_CALLBACK *AddressOfCallBacks;
    UINT32 SizeOfZeroFill;
    UINT32 Characteristics;
} IMAGE_TLS_DIRECTORY, *PIMAGE_TLS_DIRECTORY;

Then I started hacking it together with a hexeditor, editing a harmless test PE file in order to have a TLS directory entry that would point to my manually hex-crafted structure.

I then added some placeholder code

90
90
90
C2 0C 00

nop
nop
nop
retn 0x0c

which would handle the stack as a TLS callback is expected to

typedef void (MODENTRY *PIMAGE_TLS_CALLBACK) ( PTR DllHandle, UINT32 Reason, PTR Reserved );

and pointed to it from the callback list I created. I then pointed the callback field, AddressOfCallBacks, in the TLS structure to my callback list.

And everything should be fine according to the plans... but nope!! and this is where I was stuck for a few days. The best I could do was to get my TLS callback code run on program unload, but never before the entry point was given control. Puzzling...

I dug out a file with a working TLS. Ilfak wrote a while ago about TLS here and had a nice, small example. (And, by the way, IDA does read and handle TLS just fine and marks them as entry points as well. Very convenient!)

I was set to get mine working, so I started looking at what his was doing different from mine (besides him just being sane and not doing it with a hex-editor ;-) )

I took a look at all the PE headers but none of the differences seemed to have anything to do with my sample not working.

I started to grow slightly uncomfortable and decided to bring in the artillery. Taking a look at how the windows loader (residing in NTDLL.DLL) handles both files and seeing what's affecting my TLS callback not being called should help. So I brought up BinNavi and traced the execution path of both binaries being loaded by Windows.

First thing was to trace the execution of Ilfak's example, I wanted to see all functions visited in the windows loader as his executable was being loaded. The TLS callbacks would have to be called by one of these.

I then recorded the execution path of my test executable and took a look at what functions were being visited in both traces. (All the nodes in the following graph are visited by the working example, the green ones are the ones visited by mine, so there's a lot of superfluous code I can skip looking at)

Eventually spotted a function called when processing both binaries, _LdrpRunInitializeRoutines, that looked like a good candidate to be the one calling the TLS callbacks and took a look at the execution traces within that specific function.

In the following graph each node represents a basic block, the red one is where the TLS callbacks are called from. That's the node reached in the working example but not in mine. The green nodes are all the ones visited in the case the execution flow reaches the red basic block. The darker ones are the execution trace of my test. Hence I need to figure out which conditions are diverting the execution flow and how they are related to things I could change in my test program.

Now I could see the common parts of the execution path and a couple of branches that were taken differently. Given the visual output, it's extremely easy to see what branches were different and I could now check what affected the flow.

The TLS callbacks were ran immediately after the following condition

Which, tracing it back, comes for an initial check at the beginning of the function

The following article helped me when I was trying to figure out what was going on. According to it, the function _LdrpClearLoadInProgress returns the number of DLLs currently loaded. That's the value that gets assigned to the variable that gets compared to zero and makes the flow of my test program diverge from Ilfak's working example. Therefore TLS callbacks only get run when a given amount of DLLs have already been loaded and that was the reason my test didn't run on load... I only needed to add one mode DLL to the import table for it to work. Fortunately it was easy to spot with BinNavi.

Thank you to cailin for the proofreading.

Tiny (and crazy) PE

2007-03-12T23:21:00.000+01:00

I did prepare a couple of new graphics for the last training I taught with Pedram in BlackHat DC.

One of them was to illustrate a bit the structure of the header mess that leads to the small footprint of the executables in Solar Eclipse's solution to the Tiny PE challenge.

I think it's a good example of how flexible and tolerant the Windows loader is and why loading PE files is something that tends to break most tools when a file pushes the limits.

I'll comment on some of the slides on this post without going into too much detail on how or why things work, that's well explained in Solar Eclipse's page. The general format is the same I use when doing the walk-through of the PE format's main headers in a sane file in order to illustrate how the headers are laid out on the file itself.

The red zeros in the following pictures mean data beyond the file size. That data is zeroed by Windows when the file is mapped in memory and Tiny PE relies on those zeros being there, as windows will try to access data in memory at that location, beyond the end of the file. If the memory wasn't zeroed first and contained random data it would be much harder to cook up the headers in the current "compressed" layout.

This first shot just shows the DOS header and the e_lfanew field. Which points to the the start of the NT headers.

The e_lfanew field contains 4, which is the offset within the file where the NT headers can be found. That's in the middle of what would otherwise be the DOS header. In the shot of NT headers we can see some of its fields.

The NT headers contain the File and Optional headers, the next picture shows some of the fields constituting the Optional header.

As the last entry in the Optional Header one can find the array of data directories.

Now, the loader would need to locate the section headers. These normally follow after the directories of the Optional header. But in this case, as it's illustrated in the following picture, they lay in what would be the middle of the Optional Header. The location of the section headers is calculated by adding the size of the Optional header (4) to its offset (0x1C). Amusingly enough, the Windows loader does not take into account the reported size of the optional header when it reads the header itself, but it does in order to find what follows.

And here are the fields of the section header...

I haven't taken the time to illustrate the import directory and the couple of additional details missing (It's just left as an exercise of mental contortion for the reader...). The original text by Solar Eclipse provides with the rest of the info for the interested souls.

pefile: parsing version information from the resources directory

2007-02-22T14:10:00.000+01:00

A while ago I got some inquiries on how to go about reading the version information stored in PE files.

I had an idea of it being just a bunch of unicode strings without much of a structure but to follow along the rest of the PE file format, it does indeed have some structure. The only inconvenient was to find proper resources on how to parse it as Microsoft's docs mainly amount to, understandably, "just use the API". I eventually found a couple of references where a parser for the version information stored with a Portable Executable's resources directory was implemented.

After finally understanding how that information was stored, I added support in pefile so now a dictionary is conveniently returned whenever parseable version information exists in a PE file.

Some of the links in which I based my parsing implementation are:

Full parser in C

VS_VERSIONINFO structure definition

VS_FIXEDFILEINFO structure definition

VarFileInfo structure definition

StringFileInfo structure definition

From those last two links one can follow into definitions for the other structures.

Now, before I forget how this all goes. The version info structure in composed of a list of substructures. Those substructures can be of StringFileInfo or VarFileInfo type. The former contains the usual textual information that can be seen on the Version tab on the Properties dialog for a PE image. The later specifies version information in a way that does not depend on the language and codepage.

StringFileInfo contains a list of StringTable structures and each of those contains a String structure. This last structure contains the Key, Value pairs that make for the textual version information.
VarFileInfo contains a list of Var structures (although normally is only one) and each of those contains a list of pairs of Word values with version information.

VS_VERSIONINFO(VS_FIXEDFILEINFO)
- StringFileInfo
  - StringTable (LangID)
    - String
- VarFileInfo
  - Var
    - WORD, WORD

Example

If the file has version information, the following attributes will exist in the PE instance returned.

VS_VERSIONINFO will contain the first three fields of the main structure: 'Length', 'ValueLength', and 'Type'

VS_FIXEDFILEINFO will hold the rest of the fields, accessible as sub-attributes: 'Signature', 'StrucVersion', 'FileVersionMS', 'FileVersionLS',
'ProductVersionMS', 'ProductVersionLS', 'FileFlagsMask', 'FileFlags', 'FileOS', 'FileType', 'FileSubtype', 'FileDateMS', 'FileDateLS'

FileInfo is a list of all StringFileInfo and VarFileInfo structures.

StringFileInfo structures will have a list as an attribute named 'StringTable' containing all the StringTable structures. Each of those structures contains a dictionary 'entries' with all the key/value version information string pairs.

VarFileInfo structures will have a list as an attribute named 'Var' containing all Var structures. Each Var structure will have a dictionary as an attribute named 'entry' which will contain the name and value of the Var.

print hex(pe.VS_VERSIONINFO.Length)
print hex(pe.VS_VERSIONINFO.Type)
print hex(pe.VS_VERSIONINFO.ValueLength)

print hex(pe.VS_FIXEDFILEINFO.Signature)
print hex(pe.VS_FIXEDFILEINFO.FileFlags)
print hex(pe.VS_FIXEDFILEINFO.FileOS)

for fileinfo in pe.FileInfo:

if fileinfo.Key == 'StringFileInfo':
for st in fileinfo.StringTable:
for entry in st.entries.items():
print '%s: %s' % (entry[0], entry[1])

if fileinfo.Key == 'VarFileInfo':
for var in fileinfo.Var:
print '%s: %s' % var.entry.items()[0]

0x35c
0x0
0x34

0xfeef04bdL
0x0
0x4

LegalCopyright: Mozilla Corporation
InternalName: Firefox
FileVersion: 1.8.1: 2006101023
CompanyName: Mozilla Corporation
LegalTrademarks: Firefox is a Trademark of The Mozilla Foundation.
Comments:
ProductName: Firefox
ProductVersion: 2.0
FileDescription: Firefox
OriginalFilename: firefox.exe

Translation: 0x0000 0x04b0

This should come quite handy, for instance, to people interested in creating databases of version information of collections of DLLs and EXEs...

pefile 1.2.2

2007-02-22T13:50:00.000+01:00

I've just released an update to pefile. This new release includes:

pefile-1.2.2 can now correctly parse the files from the Tiny PE challenge, which pushes the limits of valid parsing

Added support for parsing the version information structures in the resources directory

Additional information and download links can be found in pefile's page.

BinNavi database format

2007-02-06T16:16:00.001+01:00

In the latest version of BinNavi we moved to SQL for storage of the disassembly information.

We spent a fair amount of time thinking about how to best store the disassembly information in a way that would be as little architecture-dependent as possible and allow for fast querying while, at the same time, trying to not make it too hard to use directly through SQL.

We also wanted to be able to have a central repository of all disassembly data, doing away with the need of keeping local databases that easily get out of sync. A central repository has other advantages like allowing for different users to look at the same project.

The disassembly data is currently exported from IDA into the database via a exporter , ida2sql, written in Python and requiring IDAPython. This exporter is included with BinNavi and also made available separately. If you want to play with it just drops us a line at Sabre.

One can already perform interesting analysis by just using SQL queries directly, although we provide a lot of the functionality in a more convenient form through BinNavi's integrated Python interface.

The core set of tables looks like this:

The basic table layout is:

modules, which holds the information about all the disassemblies in the database, the name of the file, the date it was imported and a comment field

For each of the following the module id will be appended to the table name.

functions, containing all the functions in the disassembly and specifying their address name and type

basic_blocks, all the basic blocks in the disassembly and their parent function

instructions, the instructions making the basic blocks. Contains the data making up the instruction together with its address, mnemonic and parent basic block

callgraph relates all callers and callees

control_flow_graph expresses all the links between basic blocks

operand_strings contains the operand strings as shown by IDA

expression_tree represents all expressions composing the operands as a tree

operand_tuples maps addresses to the operands used by the instruction at such location

expression_substitutions allows to replace any part of the expression tree of an operand with a string, variable names are handled through this table

operand_expressions relates the operands to the expressions composing them

address_references contains all references, both to code and data labeled with their type

sections holds the raw data for the section composing the binary source for the disassembly

Currently only MySQL has been thoroughly tested. Some of the advantages the schema brings are:

Supports multiple modules in a single database

Instruction operands are stored as trees, which enables it to support a variety of architectures, efficient storage and advanced querying

Use of SQL statements to perform advanced analysis and data-mining

The exporter module from IDA to the SQL schema currently supports METAPC(x86), PPC and ARM exporting (the latter two are still in beta). More will be added in the future.

Some query examples

Listing the modules in the database:

SELECT * FROM modules

Counting the number of functions:

SELECT count(address) FROM functions

Counting the number of basic blocks (blocks shared by functions will be counted multiple times):

SELECT count(address) FROM basic_blocks

and without counting the shared ones:

SELECT count(DISTINCT(address)) FROM basic_blocks;

How about a histogram showing the of mnemonic distribution?

SELECT mnemonic, COUNT(mnemonic) as mnem_count FROM instructions GROUP BY mnemonic ORDER BY mnem_count;

.
.
or, 275
inc, 361
movzx, 377
leave, 392
sub, 403
stos, 429
.
.

or something a bit more elaborate like getting all addresses in a disassembly where a specific register is used:

SELECT
    HEX(instructions.address), mnemonic
FROM
    instructions
JOIN
    (operand_tuples, operand_expressions, expression_tree)
ON
    instructions.address = operand_tuples.address AND
    operand_tuples.operand_id = operand_expressions.operand_id AND
    expression_tree.id = operand_expressions.expr_id
WHERE
    symbol='ebp';

.
.
71CF14D3, mov
71CF14D6, mov
71CF14E9, pop
71CF15E1, push
71CF15E2, mov
71CF15E6, mov
71CF1617, pop
.
.

for queries like the last one or more complex ones it's usually a good idea to move to the embedded Python interpreter in BinNavi before getting lost in SQL...

In New York

2007-01-21T22:43:00.000+01:00

I'll be in New York from the 27th until the 2nd of February. If someone wants to meet and go out for a beer or something and talk about BinNavi, BinDiff and what we do at Sabre-Security or just rant... drop me a line to ero.carrera at gmail

Uninformed 6

2007-01-21T22:30:00.000+01:00

The latest Uninformed issue makes for a good read.

Exploiting 802.11 Wireless Driver Vulnerabilities on Windows is truly interesting. Considering how highly-sensitive something like wireless drivers are, it's just sad how breakable appear to be.

Subverting PatchGuard Version 2 is Skywing's the lastest effort on how to overcome Microsoft's integrity checking technology. It's nice to see how far Microsoft have come and yet quite amusing to see the holes they leave open...

It also includes a nice article from skape on relocation tricks one can play with PE files. Although some of those tricks have been around in malware for a while this is the first time I've seen a good write-up about how they work.

Playing with relocations is a trick that Pedram and I always comment on in our training when teaching the PE file format. Speaking of which, Pedram and I will be teaching our training, Reverse Engineering on Windows: Application in Malicious Code Analysis, in BlackHat DC on February 26-27

BinNavi's basic block handling

2007-01-20T22:49:00.000+01:00

A while back I talked about the problem of highly optimized code and the resulting problems when we want to store it in a database while allowing for all possible constructs.
In this post I'll show how the recently released BinNavi 1.2 handles some cases where code is shared among functions and basic blocks exhibit non-typical characteristics.

Note: I could not get the Microsoft or Intel compilers to produce compiled code with functions sharing basic blocks as an example for this post. Using PGO (Profiling Guided Optimization) and other optimizations proved fun but the furthest I could go was producing multi-chunked functions. I might need to play more with it in order to get chunks to be shared... Anyway, I just picked a Microsoft DLL for the example. Specifically wkssvc.dll.

In this DLL there are two functions sharing some blocks on their exit paths. The shared code is shown in green in this graphic exported from BinNavi.

Interestingly enough, those blocks share exactly the same code but one function has six of them while the other has seven. I previously commented on the issue here and happends because of a reference in one of the functions that targets the shared code and causes a block to split.

Here one can see the shared code grouped and highlighted. Click on the image for a larger view.

Now a zoomed version of the block on the left:

And the one on the right:

Within the database, BinNavi handles this by allowing instructions to belong to multiple basic blocks, as well as basic blocks to belong to multiple functions. And it just works...

Multi-chunked functions and IDA

2006-12-12T01:27:00.000+01:00

In my post a few days back I did mention the problem with shared basic blocks across functions and made the assumption that IDA could not handle them. It is the case, as clarified by Ilfak, that IDA can actually handle them, yet they are not currently assigned to multiple functions. IDA's plugin API even allows to iterate through the chunk's parent functions.

Intel binaries for Pythonika

2006-12-11T00:43:00.001+01:00

I've just uploaded new Pythonika packages (tar.gz and zip) to my site. The only change is that there are now compiled versions for Intel of the MathLink module for OS X. For Python 2.3 and 2.5.

Simply blocks, basically...

2006-12-08T16:38:00.000+01:00

A few days ago I bumped into something I was not really counting on seeing. Compiler optimizations have surely gone a long way.

Some background first...

After having seeing functions split over non-contiguous basic blocks for some time now, it was quite natural to think that some of those basic blocks could be shared among functions ( obviously, only the ones leading to the function exit points as once the shared code is reached, there's no way of getting flow back to basic blocks not shared by those functions).

Then we have that functions can be split with their blocks in different parts of a binary and some of those blocks shared. The reason for the splitting comes from doing profiling in normal use-cases of the applications and trying to group frequently accessed code into as few pages of the executable as possible, so that a minimum set of those need to be mapped at one time in memory. Only when infrequently visited code is reached some new pages new to be mapped. The following figures illustrates the concept.

UPDATE: Just got told that the reason for the splitting is more likely to be there to take advantage of the internal CPU instruction cache than of memory paging. Keeping the frequently traversed code together will result in less instructions being fetched from RAM (slower) for that code area. Also will allow to fit more code in the code-cache by moving away the less used blocks.

Here we can see the blocks being laid out continuously in memory. As can be normally seen in non-optimized code.

This would be how the same function would be laid out if profiling information is incorporated, so that frequently traversed paths are together within the code (in the same memory page if possible, in order to reduce memory footprint and paging).

Once one has the splitting, the idea of sharing comes naturally.

This results in that, from the disassembler point of view, one has to allow for those chunks and also for those chunks to be assigned to an arbitrary number of "owning" or parent functions.

What is more interesting, and the subject of this post, is the fact that instructions can also belong to different basic blocks. At least, under one view. This arises from cases where extensive optimizations are used.

A couple of days ago I was looking into an optimized binary (the craziest I have seen in a while) and how it was mapping into the SQL representation we are using at Sabre, there were some problems when exporting the information from IDA. (IDA can't really handle too well (yet) heavily-"chunked" code, so I have to account for that and build intelligence that analyzes the code for cases like the one I'm discussing here)

The problem was with two functions sharing a number of basic blocks, the funny side was that, depending which function one analyzes the flow among the shared blocks will look different. And the cause is fairly obvious too once one realizes why the problem appears.

A conditional branch from the non-shared code in one of the functions targeting the shared code will cause a split in the flow. A split which is not present from the other function's point of view. The following figure shows the result of a branch into shared code from only one of the sharing functions.

There are two solutions for this problem. One would be to represent the same basic blocks all over the binary, which would introduce a non-natural spit in a function, the other way would be allowing to have different "views" of the code, using the basic blocks simply as a representation of the underlying model (the disassembled instructions), so that different basic blocks would contain the same instructions and those basic blocks would accurately represent the flow in the two functions...
In the next figure, the colored basic blocks contain the same instructions in both functions, but the flow is different because of the branching.

I'm leaning towards the second approach (the one in the previous figure), our SQL schema should support it trivially, which is fairly neat.

Some useful OS X apps

2006-12-04T19:21:00.001+01:00

I recently tried SuperDuper! and found it to be quite speedy and convenient way of keeping a mirror of my laptop as a backup. It's much faster than Retrospect (which aims at being a full featured backup system, as oposed to SuperDuper!) which sadly still is not available as an Universal application.

Another cool application is ScreenRecycler, which allows to extend OSX's desktop over other computers over the network using VNC. It does need a pretty fast network to work in a totally transparent fashion, but it's a nice idea and works. (I've only tried it in a tri-head configuration; the laptop's display, a second physical screen attached to the MacBook and the third one using ScreenRecycler)

Latest Parallels beta is impressive

2006-12-04T19:05:00.001+01:00

The last beta of Parallels Desktop has a really cute feature, that is the Coherence mode. It allows to display all the windows of applications running within the virtualized OS among the windows in OS X. It's a really freaky feeling to have Windows XP windows among OS X apps. Seeing IDA next to Safari and XCode is quite an experience.
When one wonders how they did it, it comes as something that should not be specially complicated to implement. I haven't looked at all into it, but my guess would be that they have Parallels walk the window list, get the geometry information and just capture those regions and display them as individual windows in OS X... it's just really cool.

Interesting tracing tool

2006-11-29T21:26:00.001+01:00

I just read in Dailydave about UMSS, a tool developed by McAfee labs and hosted in Sourceforge.

I haven't had time to play with it but seems to provide support for single-instruction tracing. It does rely on an external disassembler so it won't be useful for self modifying code in its current state (the first use that jumped to my head). But sounds quite interesting nonetheless.

OS X binary protection

2006-11-28T19:49:00.000+01:00

Just came across a nice article (from the book Mac OS X Internals: A Systems Approach) about OS X's binary protection. It details how Apple implemented encryption of the code section of certain binaries.

NumPy arrays and Pythonika

2006-11-16T19:21:00.000+01:00

If someone tries to pass NumPy's ndarray objects into Mathematica with something like:

Py["numpy.array([[1,2,3],[4,5,6],[7,8,9]])"]

the following error will appear:

Object type can't be converted!

That's due to the fact that Pythonika doesn't know what to do when finding objects of such type (or any type that it's not one of Python's basic types).

In order to get around that, one can do something like:

Py["numpy.array([[1,2,3],[4,5,6],[7,8,9]]).tolist()"]

Which will return the expected nested lists in Mathematica:

{{1, 2, 3}, {4, 5, 6}, {7, 8, 9}}

Mac OS X security report

2006-11-15T17:27:00.000+01:00

Symantec has put out a nice summary of the state of OS X security. It lists some of the vulnerabilities discovered to date and goes over some of the existing malware and rootkits. Worth a read.

pefile 1.2

2006-11-06T10:10:00.000+01:00

pefile has just gotten some nice updates.

- Added support for PE32+ files
- Merged the patches from the Offensive Computing people and other contributors
- Added support for writing changes back to the PE file. This should be used with care
- Miscellaneous other bugfixes and enhancements

As usual, more information and download links can be found in pefile's page.

Pythonika

2006-11-06T10:02:00.000+01:00

I have finally managed to release Pythonika! I wrote Pythonika quite a while ago and was never getting around to push it out.

Pythonika is a MathLink module for Mathematica that makes it possible to write Python code within Mathematica's notebooks. It handles the conversion of Python and Mathematica objects transparently and allows to use all of Python's standard modules.

I'm a big fan of Python and I've been using Pythonika for a while. I hope more people will find it useful.

Pythonika is available at:

http://dkbza.org/pythonika.html

(an example notebook is available on the previous link as well as in the downloaded package)

The download includes source code and binaries for OSX/Windows/Linux.

Google rocks

2006-10-05T19:36:00.000+02:00

Gotta love these guys

http://www.google.com/codesearch/advanced_code_search
Some quick tests even find virus source code.. awesome. Also fun: fun search one, fun search two, fun search three

Reverse Engineering Training

2006-06-22T08:05:00.000+02:00

Pedram Amini and I will be conducting a training in Black Hat Las Vegas in a bit over a month. We will be covering reverse engineering tools and techniques with a focus on Windows malware. We have built an extensive Python toolset nowadays, and together with other tools, the material we will cover will also give insight on how use those and others in order to automate analysis techniques.

We will also include the latest on the results of the research I have been mentioning in my posts and more. I think people will enjoy it.

Packer tracing

2006-06-22T02:55:00.000+02:00

One of the things I talked about in one of Recon's lightning talks was some quick research I had done tracing packers. Using some internal tools I've written I traced the "behaviour" of different packers. I thought it would be nice to just show those results here too. These are presented as plots. Time is in the horizontal axes with the vertical one expressing the address where the traced event occurs. The colors encode memory writes and EIP location using green and blue respectively.
The packers were traced as they were unpacking Window XP's Notepad.exe. It's possible to see in some of the graphs a blue dot in the rightmost side, indicating the EIP jumping to the original entry point of the unpacked application.

I think these graphs are quite informative and give ideas on possible heuristics to tackle the problem of generic unpacking. The datasets used to generate them are rather large, for some packers I collected tens of millions of points which were plot as seen.

Specially interesting are the peculiar plots of tElock and Yoda's packers, where it's possible to see the EIP(blue) going through addresses which had been previously written to (green), indicating multi-stage unpacking taking place.

Note: In some graphs there are gaps in the EIP trace. That is obviously impossible as the execution is continuous, the reason for the gaps is that the EIP was outside the plotted range, for instance, in DLL code.

ASPack 2.12

Petite 2.2

UPX 1.95

FSG v2.0

tElock 0.98

Yoda's Protector v1.02

Yoda's Crypter v1.3

Recon 2006

2006-06-22T01:49:00.000+02:00

Last weekend I had the pleasure of assisting to the reverse engineering conference Recon in Montreal. Recon has probably been one of the best events I had the luck to assist to so far. The quality of the speakers was really high and the people assisting were incredible. The whole event was very well organized, with no hiccups.
I had many really enjoyable talks and met great people.

Regarding the talks I specially enjoyed Bunie's Disassembling And Patching Hardware. I did study electronics some years back, although never to the level of what Bunnie was showing. I found the talk really interesting.
Alex Ionescu's Subverting Windows 2003 SP1 Kernel Integrity Protection was tremendously informative too, going really deep into w2k3 protection mechanisms.
Skype's talk by Fabrice Desclaux and Kostya Kortchinsky was truly amusing, they really took apart every single part of Skype.
Spoonm showcased IDARub, now Ruby lovers have a tool as good as IDAPython, if not better. IDARub's network support is something that IDAPython will surely catch up with soon.
Pedram Amini released PaiMei(download), a tool that surely will be talked about a lot. A full reverse engineering framework in Python. It really looked mighty powerful.

Also, I ended up cooking up two quick (ended up being not-so-quick) turbo-talks; Win32 Static Analysis In Python showing two Python modules I wrote a while ago pefile and pydasm and Unpacking Bird's Eye View showing some results on tracing unpackers using some internal tools. More on that last one on another post...

2006-06-15T07:35:00.000+02:00

Tonight I dropped by the taping of Diggnation. They were doing a special live show because of their first anniversary and I was lucky enough to catch it while I'm in San Francisco. These guys are awesome, even better live!

pefile-1.1

2006-05-30T03:42:00.000+02:00

I just released pefile-1.1. This release brings some new functionality besides some bugs fixed. A detailed list of changes is available here.

ph-neutral

2006-05-30T00:17:00.000+02:00

I came back yesterday from spending the weekend in ph-neutral in Berlin. Again the Phenoelit, DarkLab and C-Base people managed to put together an amazing event. I met really great people and saw old friends, all of them I'd like to thank for really great conversations/brainstorming and the fun I had partying there. To all I'd like send greetings from here! Looking forward to next year's.

Stevey's Blog Rants: Math For Programmers

2006-03-20T11:50:00.000+01:00

Really good post on learning maths. If only more people would think like that, specially people in charge of teaching it.

Stevey's Blog Rants: Math For Programmers

DSL bandwidth and cooking my dinner are related?

2006-03-01T22:04:00.000+01:00

What's the correlation between the act of getting my dinner heated and BitTorrent download speeds?

I experienced today something that was rather obvious after taking a look at the frequency spectrum. I was rather amused to see two of my laptops suddenly experience some sort of simultaneous network connectivity problems. In one, Azureus was showing a drop on my combined downloaded speeds and in the other the Internet radio I was listening started to break down and buffer repeatedly... I just then realized that had been few minutes since I had started heating some food in the microwave. The kitchen stands about half the way from my living room to the wireless access point and it kind of made sense that the microwave could be the reason. Sure enough, once it was done the speeds peaked up again.

Some reading later on showed that indeed, operating microwaves can impair the signal quality in wireless networks. I just thought that microwaves irradiated much less. On the other hand, probably the signal levels are so low that small inference easily degrades the signal. I found it amusing.

Microwave Owens: 2.450 GHz

US/Canada 802.11b/g Channel Nominal Frequencies go from 2.412 GHz to 2.462 GHz in steps of 5 MHz
(Europe seems to use very similar frequencies if not the same.)

Old "recordings", just incredible

2006-02-19T10:55:00.000+01:00

Archaeologists get ancient audio from grooves on Pompeii pottery. Sound got recorded in old pottery's grooves. It's not that difficult to believe that it could happen (although I don't know the exact mechanics of the process they were using those times) , but it's impressive that someone even thought of the idea of it happening and managed to recover it. From the recording it does seem like they did. (Via Digg)

OS X malware

2006-02-16T12:39:00.000+01:00

The following entry in MacRumors, The First Mac OS X Virus? (A New OS X Trojan), mentions of a recently discovered piece of malware targeting PPC OS X. I hope I can get my hands into a sample of it, I'd love to check it out...

Video of the Multi-Touch Interaction Research

2006-02-08T23:18:00.000+01:00

This is really impressive, there are countless things I would want to do with a system with an interface like this: Video of the Multi-Touch Interaction Research. Just imagine the amount of information and the speed and precision with which it can be handled. (Via Digg)

oh no!

2006-01-25T20:47:00.000+01:00

This is real teaching

2006-01-21T21:25:00.000+01:00

I can only wish math would be taught this way more often... simply beautiful

The Socratic Method used on 3rd graders to teach them binary arithmatic: "The experiment was to see whether I could teach these students binary arithmetic (arithmetic using only two numbers, 0 and 1) only by asking them questions." (via Digg)

Visual Complexity

2006-01-19T23:07:00.000+01:00

This is just beautiful. A set of visualizations of graphs depicting complex systems.

Ilfaks's: Return to the sources?

2006-01-19T14:55:00.000+01:00

Ilfak's Return to the sources? makes for an interesting read on the problem of decompilation. I agree with such views.
I've always been thinking of decompilation in terms of generating a higher level representation of some lower level code. Information is lost in the process of compiling, so expecting to retrieve code which resembles the original is wishful thinking to say the least. Unless the compilation process in known and an injective map from a set of high level constructs to a set low level ones exists, in that case one could use that information in order to reconstruct those higher level constructs. Although, nowadays, with the amount of optimization such as branchless logic, code scheduling, etc such occurrence is rare.

However, as Ilfak mentions, trying to extract higher lever features from a mathematical representation of the lower level code might actually produce results and I personally think that's the way to go...

IDAPython 0.8.0 released

2006-01-19T14:27:00.000+01:00

Dyce has released IDAPython 0.8.0. I know of plenty of people who have been waiting for this one, me for one. Also, the Windows version is linked against Python 2.4.

pefile is out!!

2005-12-27T04:52:00.000+01:00

Finally, after many delays I've managed to find time to release a newer version of pefile (was pype). The name change comes from the fact that another project called PyPE already exists and was using the name earlier. So now it's pefile.

It can now parse Delayed Imports, thanks to Adam Morrison, and multiple fixes have been added. It should be able to take really corrupted and malformed PE files.

Some Python loving

2005-12-27T03:21:00.000+01:00

This post has very good points on Python and Java. Having played with Eclipse myself and coding lots of Python with no IDE (SubEthaEdit rules!) I could hardly agree more with him.

Eyecandy...

2005-12-26T04:48:00.000+01:00

Having and huge backlog of stuff, being xmas night away of any known humans and a terrible jetlag does help my blog... it's nearly 5am and still gonna be kicking around for a while... to the topic:

Some days ago some very nice guys gave a nice collection of malware, mainly bots. Nice because it was in the order of thousands... the wet dream of an automation fanatic...
I love to run stats on large collections of data (Statistical Natural Language Processing on large corpora is extreme fun) and malware is something I've had some encounters with ;)

I haven't even starting doing real crunching but the other day I ran some Mathematica SOM (Self Organizing Network) code I had written a while ago on some values extracted from a subset of those samples. SOMs are simple neural networks which have the advantage of making clustering of high-dimensional data really easy. In this case I just went for using 3-dimensional vectors with the ImageBase, AddressOfEntryPoint members of the PE header and the first section's size. I ran that through the SOM which gave pretty nice clusters. The following scatterplot shows that indeed those (normalized) values do indeed have quite some structure.

I went a bit further then and ran some code I wrote using pydasm which disassembles the binaries and gives me a 36-dimension vector (originally spits out a 6x6 matrix) representing specific characteristics and patterns within the code.

I then run that data through the SOM code and again produced defined clusters. Not that surprising since this approach captures patterns in the code and most of those binaries share commons packers (UPX and the likes). So, some more eyecandy...

Those last two pictures show certain areas with a higher density of points. I then went on making a 2D histogram by counting how many samples fell into each bucket. The following pictures show the spread of those. (They are the same data, the second plot is logarithmic though.)

Dunno if it's of any particular use, but sure it's fun...

Some (hopefully) informative graphs

2005-12-26T04:23:00.000+01:00

Long due but I've finally been able to finish and upload two PDFs I wanted to push out for quite a while.

The can be found in my OpenRCE file repository. One is on the PE file format and the other on the Windows memory layout.