... In Which It Is Revealed How An AHCI Bug Makes One's Insyde(s) Freeze

I found this code in Intel's AHCI Option ROM from the Insyde BIOS. It appears to be code to build the Translated Device Parameter Table (which is a slightly different¹ implementation than the one documented in the Enhanced Working T13 Draft 1126DT).

Psuedo code  (version: Serial ATA AHCI BIOS, Version iSrc 1.20_E.0019 07092009):

Function Create TDPT for drive
    Read partition table with INT13 0x201
    If read fails, or 0xAA55 signature isn't present, goto Calculate
    Get head,sectors from FIRST Partition table entry, Ending CHS values
    heads = head+1  (because of 255 limit in partition table)
    For each partition entry
        If BOOTABLE (entry[0] == 0x80) goto UsePartition
    EndFor
    For each partition entry
        if (entry[0] == 0) and (entry[4] != 0) goto UsePartition
    EndFor


Calculate
    Call CalculateCHS - using DPT and physical size
    if no need for translation, return GOOD
    Goto CreateTDPT with cylinders, heads, sectors


UsePartition:
    Read first sector of partition with INT13 0x4200
    If the word at offset 0x1A is less than 0x100,
      and the word at offset 0x18 is less than 0x40
    then
        set heads to the byte at offset 0x1A
        set sectors to the byte at offset 0x18
    fi
    tracksize = heads * sectors
    if tracksize == 0, Goto Calculate
    DWORD size = (DPT[heads]*DPT[sectors])*DPT[cylinders]
    WORD cylinders = size / tracksize <--- Bad!
    -- if the result is greater than 65536, a divide overflow occurs

    -- which isn't handled by the BIOSes

    if (cylinders > 1024) cylinders = 1024
    if ((heads == DPT[heads]) && (sectors == DPT[sectors])) return GOOD


CreateTDPT:
    WORD at DPT[8]  = DPT[0] - Save original cylinders
    BYTE at DPT[10] = DPT[2] - Save original heads
    BYTE at DPT[7]  = DPT[3] - Save original sectors
    WORD at DPT[0]  = cylinders
    BYTE at DPT[2]  = heads
    BYTE at DPT[3]  = sectors
    BYTE at DPT[5]  = 8 if heads greater than 8, otherwise 0
    BYTE at DPT[4]  = 0xA0
    BYTE at DPT[15] = SUM( DPT[0] .. DPT[14] )

So, what goes wrong? When it breaks, it starts with bad values from the partition table, and tries to fix it with values from the boot parameter block, if it finds "valid" numbers there for heads and sectors (that is, less than or equal to 0xFF and 0x3F, respectively). When these values aren't right,  due to full disk encryption, an operating system other than Microsoft Windows, or malicious intent:

• It uses the ending head/sector of the first partition to size the translation layer.
• Windows 7 with 100MB partition results in unexpected values for INT13, FUNCTION=8 (eg, 0x13 heads).
• It stores those values into the Translated Device Parameter Table..  and then some other code comes along and uses those values. While I can't find where those values are causing the exception, anything doing C/H/S translation will be unhappy.

Looking back at version Serial ATA AHCI BIOS, Version iSrc 1.20E (Gigabyte Desktop Motherboard), I found that it doesn't read from the BPB at all.  I speculate the extra read of the NTFS boot-sector was to workaround a problem on Insyde BIOS.   This version can be crashed if the two bytes in the partition table are small enough and will hang with error code 23. Award BIOS will function OK with the other unexpected values, but Insyde BIOS will still crash if it sees them.

Finally, one HP system with an Insyde BIOS has the latest(?) 'fixed' version (Serial ATA AHCI BIOS, Version iSrc 1.20_E.0024 12212009), which reads from both the partition table and the BPB, also adding  still more checks. Unfortunately, it seems as though someone messed up and added a further bug, as it doesn't actually use any of the values it reads, but rather discards them all.

New and improved UsePartition (Serial ATA AHCI BIOS, Version iSrc 1.20_E.0024 12212009):

Read first sector of partition with INT13 0x4200

if the word at offset 0x1FE is not equal 0xAA55

   and the byte at offset 0 is not equal 0xEB

   and the word at offset 0x1A is less than 0x100

then

    set heads to the byte at offset 0x1A

fi

if (tracks == 0) or ((sectors & 0x3F) == 0) Goto Calculate

-- New bug: Since sectors can be at most 0x3F from partition table

-- the newer version ALWAYS goes off to Calculate the CHS

if (sectors & 0xC0) == 0)  Goto Calculate

tracksize = heads * sectors

if tracksize == 0, Goto Calculate

DWORD size = (DPT[heads]*DPT[sectors])*DPT[cylinders]

WORD cylinders = size / tracksize <-- Uber dangerous

-- if the result is greater than 65536, a divide overflow occurs

-- which isn't handled by the BIOSes.

if (cylinders > 1024) cylinders = 1024

if ((heads == DPT[heads]) && (sectors == DPT[sectors])) return GOOD

Goto CreateTDPT

A year later and neither Acer nor Gigabyte are providing fixed BIOSes.

¹Expected Final TDPT Values from a 60GB SSD:

          WORD Logical Cylinders   0x400

    BYTE Heads               0xFF

    BYTE Sectors             0x3F

    BYTE Signature           0xA0

    BYTE HeadsAbove8Flag     0x08

    BYTE Ignored             0x00

    BYTE Physical Sectors    0x3F

    WORD Physical Cylinders  0x3FFF

    BYTE Physical Heads      0x10

    BYTE Ignored[4]          0x0

    BYTE Checksum            0x89

Acer 5810tz's "Secret" BIOS Menu

As described in http://marcansoft.com/blog/2009/06/enabling-intel-vt-on-the-aspire-8930g/ and on NotebookReview's Forum, there are hidden menus in Insyde's Acer  BIOS.  Some folks modify their BIOSes to enable them; however,  on the Acer 5810, it's possible to enable some of these menus with just a setup change, which is, of course, safer than having to flash your whole BIOS just to modify a single boolean.

To do so, you need:

•  a DOS-bootable USB stick
• the flashit.exe binary from the BIOS update
• grdb from http://ladsoft.tripod.com/grdb.htm

Copy the utilities to the usb disk and create a batchfile (ex. modify.bat) and add the following:

flashit Setup A04A27F4-DF00-4D42-B552-39511302113D /rb:setup

grdb setup

flashit Setup A04A27F4-DF00-4D42-B552-39511302113D /wb:setup

     

Reboot to DOS on the USB stick, run the batchfile and in GRDB type:
e 31a 1 

w 

q

When you reboot, you should have an extra menu "Intel" below "D2D Recovery"

The "hidden setup" is usually located immediately after the "D2D recovery" variable.  If you wish to locate it on a different version of Acer Insyde bios, you can write out the setup variable ( the /RB command-line) with and without D2D recovery enabled.  The location one past that will be what you need. For example:

fc /b setup1 setup2

Comparing files SETUP1 and SETUP2

00000219: 00 01

Which is 21A (or 31a if you are using GRDB, because GRDB pretends it's a .COM file )

Although its unlikely that messing up the setup variables will cause the laptop to not boot, prudence would be to know how to use the FN-ESC "Crisis Recovery mode" and know what the filename is for your particular laptop.  (Mine is JM41X64.fd)

Possibly Useful Reference:

Post On Lenovo Forum With FlashIt Parameters

Pictures:

"Secret" menu available. 

Intel->

Intel->Power

Intel->Advanced

Intel->Advanced->Boot

The Byte That Bit Me Insyde

Since Insyde doesn't seem interested in patching its BIOS, I thought I'd share a neat way to make a laptop with an Insyde BIOS hang on boot by changing a single byte.

Caution: Back up any important data, if you're silly enough to do this on a live system; while this shouldn't mangle any of your bits, it's certainly possible it could.

Also, if you're not terribly fond of grasping naked sectors and pushing values into unfilled gaps, you might want to bail out now.

The Prerequisites:

• Laptop with buggy Insyde EFI BIOS.
• SSD or HDD (physical medium unimportant)
• Two or more partitions (which is the default Windows 7 configuration, and tends to be the default Linux installation as well).
• The SATA port configured as AHCI in BIOS.

My hardware configuration:
Acer 5810tz Timeline Notebook, with InsydeBIOS Release 1.35.

Now, with your personal favorite disk editor of choice (I'm using the very excellent, free and portable HxD):

HxD with Physical Disk 1

The Byte To Change

• Select the first partition of your first physical disk (labelled Physical Drive 1 in HxD).
• Go to sector 2048.    
• Go to offset 0x19 (25 decimal) and change from 0 to any value.
• Reboot
• Watch your laptop with an Insyde BIOS freeze.

Don't panic!  Simply:

•  disconnect the laptop drive,
•  bop into BIOS (F2)
•  change the SATA mode from AHCI to IDE.
•  Reconnect your hard drive, and boot.  You'll hit the usual Windows BSOD complaining about you trying to boot with the wrong set of disk drivers; boot into the 32 bit recovery console (since many utilities don't yet have 64bit equivalents and the WOW64 subsystem probably won't be available (especially if you're booting a mini-xp environment off a thumb drive instead))
•  Run your sector editor, and change byte at offset 0x19 back to 0.
•  Reboot normally.

So, what's going on here?

Sector 2048's where Windows 7 puts the start of its first partition.  From the partition boot sector layout we can see the particular byte is part of the boot parameter block, specifically the high byte of the Sectors Per Track word, which happens to be ignored by Windows 7.

What's a BIOS doing, caring about this?

Near as I can tell, this could be an attempt at an AHCI optimization, and the BIOS code simply fails to do a sanity check on the range.

Q. Except for the occasional black hat looking for a chuckle, who'd want to hang their laptop?

A. Anyone installing Linux, BSD, or full disk encryption such as Truecrypt and PGPwde.

These, as part of their normal operation, can change that special byte, giving much excitement and hair pulling to the lucky person whose BIOS, in an effort to be a most helpful puppy, manages to decorate the newspaper with excrement after having a chew with it.

I'm so glad EFI has led us away from incompabitle, buggy BIOSes!

Tools mentioned:

HxD Hex Editor (used for sector editing)

References:
Homepage of Insyde

Others probably running into this specific issue:
Seen on HPs, blamed on encryption
And Lenovos, blamed on Truecrypt.
Seen again, attributed to AHCI+Truecrypt conflict.
Even with BSD!

Splitting a single flac file into mp3s.

I might need to stop avoiding .flac files, as now I found how easy it is to do something with them; historically, I've just been lazy and ripped my CDs to mp3s.

However, it might be prudent for me to start converting to flac, and have lossless audio I can just extract the specific tracks I want for a given rotation.

Split2flac - A beautiful shell script

References:
https://bbs.archlinux.org/viewtopic.php?id=90278
http://stillstup.blogspot.com/2008/07/split-lossless-audio-ape-flac-wv-wav-by.html

Linux VMWare BIOS Extraction

More of a note to self than a blog...

objcopy /usr/lib/vmware/bin/vmware-vmx -O binary -j bios440 --set-section-flags bios440=a b.z
perl -e 'use Compress::Zlib; my $v; read STDIN, $v, 217218; $v = uncompress($v); print $v;' < b.z > bios440_mod.rom

cp oembios440.rom /srv/projects/vm/nýttkjöt/
echo "bios440.filename = \"oembios440.rom\"" >> /srv/projects/vm/nýttkjöt/nýjar_vm.vmx

Reference:
VMware BIOS modification - for Linux users
Modifying The VMware BIOS
Persistent BIOS Infection

Exporting Email From Myspace With Google Gears in Firefox on Linux

Myspace doesn't appear to have a way of natively allowing exporting messages.  What follows is a proof-of-concept kludged script to do so.

I desired to retrieve 50 email messages from my now defunct Myspace inbox.

I didn't have any messages in sub folders, nor did I care for the single message I had in my 'sent' folder.

For whatever reason, it doesn't appear that Sent messages are being saved in the Gears database, though it ought parse those in sub folders.

I make no effort to parse and scrape user images; however, doing so appears as though it would be trivial.

Of further note: for a user that has an account number of 12345678, a user id of jdoe, and a display name as "Johnny D0e LOLZ!!!!", Myspace appears to store another display name ("John Doe") that it shows under a user's picture in email; this name is not accessible via the Gears database, as far as I can tell.  (The example above is based on an actual Myspace 'friend')

Regarding Myspace's formatting, I found the Firefox add-on "SQLite Manager" exceptionally useful for advancing my understanding of how Myspace was saving the messages; just using sqlite's .dump function didn't make things like rowid clear for a sql-language amateur such as myself;  anyone attempting to perhaps pursue the creation of a real parser would likely benefit from the use of sqlite manager.

Please note, the CreatedData appears to have milliseconds tacked on; hence the weirdness I do with trimming.  My timezone is MDT; please adjust the +/- for hour accordingly.

This all is done in Firefox on Xubuntu, as Google Gears was not available for Google Chrome on Linux.

Steps:

1. Install Google Gears: Google Gears
2. Log in to Myspace and go to your inbox; obviously, give Gears permission to sync.  Wait until messages are saved to hdd.
3. Created a directory under your home directory, perhaps "myspace_emails" or similar would be sensible.
4. With your editor of choice (or cat >> scrapemyspace.sh for the lazy/efficient) create file and copy/paste my script below into it. Give it +x.
5. Now simply run script (sh name_you_used.sh); if you wish to have all emails in ONE file, in addition to the separate ones created, do: script.sh >> allmyspaceemails.txt or whatever; this was what I needed, and why I used tee.  The dump to the sqlite3 file is for debugging, and contains the sql commands to rebuild the database (for example, in mysql...). Comment out if not required.
6. Done!

If the location where your Gears data is stored changes, simply change the path variables at the top; also useful if you want to test, by copying the local database to your home directory.

The *.default may need to be changed (~/.mozilla/firefox/profiles.ini) for multiple users.

There is NO error checking in my script: if this makes you uneasy, please feel free to add some.

You may use this script however you wish, as long as you give me attribution, but the following disclaimer also applies:

This script is provided "AS IS", without warranty of any kind, stated or implied, including without limitation the warranties of merchantability, fitness for a particular purpose and non-infringement. The entire risk as to the quality and performance of this script is borne by you. Should the script prove defective, you and not me assume the entire cost of any service and repair. etc. etc. (If you don't understand what this script does, please find someone who does, and ask them if it's okay for you to run it, instead of trusting a random person on the intrawebs.)

Final disclaimer: I know less about bash scripting than I do about SQL. Beware.

That said, enjoy!

# Script to parse Google Gear sqlite3 db with Myspace Emails

# Copyright (C) 2010, Seth Stahlman. All rights reserved

#!/bin/bash

dbpath=~/.mozilla/firefox/*.default/Google*/messaging.myspace.com/http_80/

dbfile='myspace.messaging.database\#database'

db="${dbpath}${dbfile}"

cut='cut -d, -s'
sqlite3 $db ".dump" >> myspace_email.sqlite3.sql

authors=`sqlite3 -csv $db "select rowid, * from AuthorData_content;"`

echo "$authors" |

while read a

do

        uid=`echo $a | $cut -f1`

        dn=`echo $a | $cut -f2`

        un=`echo $a | $cut -f3`

        echo "Messages from $dn ($un) (Account ID: $uid)\n" | tee -a "${uid}_${un}"

        msgids=`sqlite3 -list -separator , $db "select MessageId, \

                datetime(substr(CreatedDate, 1, 10), 'unixepoch', '+1 hour') as date from MessageMetaData where AuthorId = $uid;"`

        echo "$msgids" |

        while read m

        do

                messageid=`echo $m | $cut -f1`

                messagedate=`echo $m | $cut -f2`

                subject=`sqlite3 -list $db "select c0Subject from MessageData_Content where rowid = $messageid;"`

                body=`sqlite3 -list $db "select c1Body from MessageData_Content where rowid = $messageid;"`

                echo "Message ID: $messageid\nDate: $messagedate\nSubject: $subject\n\n$body\n" | tee -a "${uid}_${un}"

        done

done