# HG changeset patch # User Mychaela Falconia # Date 1583705757 0 # Node ID b6b8307d195b1ab4647e88a6a514e5b2d5754a9d # Parent 8b1e86dcc3ac79db59c8717153988b9bf25ccbc6 doc: new articles Binary-file-formats and Flash-programming diff -r 8b1e86dcc3ac -r b6b8307d195b doc/Binary-file-formats --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Binary-file-formats Sun Mar 08 22:15:57 2020 +0000 @@ -0,0 +1,121 @@ +In FreeCalypso we use 3 different file formats for Calypso binary images, i.e., +code images to be loaded into either flash or RAM or retrieved flash dumps. +These 3 different file formats are straight binary (*.bin), moko-style m0 (*.m0) +and little-endian S-records (*.srec). + +Straight binary (*.bin) +======================= + +Straight binary is our preferred format for flash dumps. It is written in the +native little-endian byte order of the Calypso ARM7 processor, i.e., the order +of bytes in the raw binary file directly corresponds to incrementing byte +addresses as visible to the ARM7 - any ASCII strings in the image thus appear +naturally. We also use the same straight binary format in native LE byte order +for flashable code images generated with the gcc+binutils toolchain (as opposed +to TI's TMS470), generated with arm-elf-objcopy -O binary - although we don't +have too many such code images currently given that neither FC Citrine nor FC +Selenite ever achieved production quality. + +Another unrelated use of this straight binary format is for RAM-loadable code +images that are fed to Compal's bootloader (Motorola C1xx and Sony Ericsson +J100) as opposed to Calypso boot ROM. Our generally preferred image format for +RAM-loadable code pieces is little-endian S-records (*.srec, see below), but +for Compal's bootloader we use straight binary instead because of the way this +bootloader protocol works. + +moko-style m0 +============= + +This format is a variant of Motorola hex (S-records), a variant invented by TI +rather than by us. This format is produced by TI's hex470 tool when run with +-m -memwidth 16 -romwidth 16 options, which is the configuration used by TI in +the Calypso program, and is read by TI's flash programming tool called FLUID. +TI used this format not only for flashable firmware images, but also for various +RAM-loadable code pieces, particularly those that comprise the target-side +component of FLUID. + +The special quirk of this S-record variant format is its peculiar byte order. +TI viewed it as "16-bit hex", meaning that the image is logically viewed as +consisting of 16-bit words rather than 8-bit bytes, each S3 record carries an +even number of bytes to be loaded at an even address, and each 16-bit word +(4 hex digits) appears in these S3 records with its most-significant hex nibble +toward the left, just like the address field of each S-record. But if this +image gets interpreted by some more naive tool (for example, objcopy from GNU +binutils) as bytes rather than 16-bit words, the result will be a reversed byte +order, with all strings etc messed up. + +In FreeCalypso we use this moko-style m0 format (our new name for what TI called +16-bit hex) only for flashable firmware images built with TI's TMS470 toolchain +(can be our own FC Magnetite or historical ones built by Openmoko or other +similar historical vendors), but never for any RAM-loadable code pieces - we use +little-endian SREC for the latter as explained below. + +And what about the name? Why do we call it moko-style m0 rather than just m0? +The reason is because Compal muddied our waters by introducing their own *.m0 +files that were generated with -memwidth 8 -romwidth 8 instead of -memwidth 16 +-romwidth 16, producing 8-bit hex instead of 16-bit hex. We do not support +Compal's different *.m0 files at all, and we needed some name to specifically +identify TI-style m0 files rather than Compal-style. We ended up with the name +moko-style rather than TI-style because we already had our mokosrec2bin program +going back to 2013-04-15, one of the very first programs written in the +FreeCalypso family of projects: our very first encounter with this file format +were mokoN firmware images put out in this *.m0 format by That Company. + +Little-endian S-records (*.srec) +================================ + +Back at the beginning of FreeCalypso in the spring/summer of 2013 I (Mother +Mychaela) decided to use S-records instead of straight binary for our +RAM-loadable code pieces, i.e., code that is loaded into RAM either through the +Calypso boot ROM (fc-iram) or by chain-loading via loadagent (fc-xram). I made +this decision based on two factors: + +1) ARM code generated by common toolchains (both TI's TMS470 and gcc+binutils) + without special contortions is not position-independent: a code image that + was linked for a given address needs to be loaded at that specific address, + not some other. + +2) An S-record image has its load address embedded in the image itself, whereas + a raw binary naturally does not carry any such extra metadata. + +With SREC as the standardized hand-off format from code generation tools to +loadtools, the choice of load address is made entirely on the code generation +side; loadtools do not impose a fixed load address, nor do they require it to +be communicated via extra command line arguments or options. + +However, the variant of SREC we use for RAM-loadable code pieces is not the same +as moko-style m0 - the byte order is the opposite, with our RAM-loadable code +pieces using the native little-endian byte order of the ARM7 processor as the +byte order within S3 records. Prior to the introduction of RAM-loadable FC +Magnetite fw images for Pirelli DP-L10 in late 2016 (and then likewise for our +own FCDEV3B), the only RAM-loadable code pieces we have had were built with +gcc+binutils, not with TMS470, and GNU binutils got a different take on the +S-record format than TI did: they generate byte-oriented SREC files, with the +byte order being the same as it would be in a straight binary file, matching +the target processor's memory byte addressing order. Thus GNU-style SREC has +been adopted as the format for our RAM-loadable code images for both fc-iram +and fc-xram, as opposed to TI-style SREC aka moko-style m0. The convention we +have adopted is that *.m0 filename suffix means TI-style aka moko-style, +whereas *.srec means GNU-style. + +Besides the S3 record byte order, there is one other difference between TI-built +*.m0 code images and GNU-built *.srec ones: the final S7 record carries the +entry point address in GNU-built *.srec images, whereas TI's *.m0 images always +have a zero dummy address in there. Our fc-iram and fc-xram tools require the +real entry point address in the S7 record. + +How do we generate ramimage.srec RAM-loadable images for fc-xram in FC +Magnetite? Answer: FC Magnetite build system includes a special ad hoc +converter program that reads ramimage.m0 produced by TI's hex470 tool and +produces ramimage.srec: it reverses the order of bytes, adds another S3 record +that writes the boot-ROM-redirected interrupt and exception vectors and +generates an S7 record with the right entry point address. + +This little-endian *.srec format is actively used only for RAM-loadable code +pieces in FreeCalypso, not for anything that goes into or gets read from flash. +We do have flash dump2srec and flash program-srec commands in fc-loadtool, they +were implemented back in the founding stage of FreeCalypso in 2013 for the sake +of completeness and symmetry (it seemed right to support both binary and +S-record formats), but they never got any practical use: if you are making a +flash dump, you would normally want to examine it afterward, and any such +examination almost always needs a straight binary image, not S-records. diff -r 8b1e86dcc3ac -r b6b8307d195b doc/Flash-programming --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Flash-programming Sun Mar 08 22:15:57 2020 +0000 @@ -0,0 +1,199 @@ +Our Calypso device flash programming tool fc-loadtool can be used in several +different paradigms; this article is an attempt to clarify the many available +modes of usage. You should also read the companion article Binary-file-formats +for further background. + +Flashing firmware release images +================================ + +In conventional forward engineering environments where you develop or maintain +firmware for hardware made by your own company (meaning no reverse eng, no +"illicit" aftermarket tinkering on hw made by some other company aeons ago), +you have a firmware build system that produces fw build images (some of which +may subsequently be blessed as releases), and you have a tool that flashes +these fw build images into your hardware, operating as efficiently as possible, +automated as much as possible, requiring minimal user action for the boring +repetitive task of flashing a new fw image every time you build one. And if +you become lucky enough to produce your hardware in volume, the same objectives +of maximal efficiency and automation carry over into the production line +environment as well. + +In TI's environment the standardized format for firmware build images which are +then flashable into hardware targets was a variant of Motorola SREC written in +*.m0 files, a variant which we have named moko-style m0 after its most famous +user. The special quirk of this particular SREC variant is its peculiar byte +order. TI's firmware build system produces a *.m0 S-record image as its final +build product, and TI's official Calypso flash programming tool (FLUID) takes +these *.m0 files as its input. + +Since the beginning of FreeCalypso we have had two ways of flashing TI-built +firmware images into suitable targets (initially OM GTA02 modem, then many +others including our own FCDEV3B): + +1) Our fc-loadtool has had a flash program-m0 command from the beginning, + programming device flash with bits from an m0 file directly and natively. + However, prior to fc-host-tools-r12 this command was poorly supported: it + ran significantly slower than flash program-bin, had poorer progress + indication and did not perform CRC-32 verification at the end, which is an + important integrity check. Also this original flash program-m0 command (as + opposed to flash e-program-m0 added in fc-host-tools-r13) does not include a + built-in erase step, thus prior to fc-host-tools-r13 the user had to have + outside knowledge of how many sectors to erase first with a separate flash + erase command. + + The new flash e-program-m0 command added in fc-host-tools-r13 is m0 image + flashing finally done right. It reads in the specified S-record image in + moko-style m0 format, builds a map of potentially discontiguous flash + regions into which the image deposits bits, erases the set of flash sectors + which need to be erased before programming these regions, then programs the + new image bits into flash, exactly like TI's own FLUID. + +2) The alternative way is to first convert the *.m0 S-record image produced by + TI's hex470 post-linker tool into straight binary (*.bin) with a FreeCalypso + tool called mokosrec2bin, then program the binary fw image into flash with + fc-loadtool command flash program-bin. This method is the one we've been + using since 2017, and our FC Magnetite firmware build system is now set up + to produce not only fwimage.m0, but also fwimage.bin (it runs mokosrec2bin), + and it also generates an fc-loadtool command script (a text file named + flash-script) with two commands in it: a flash erase command with a + calculated sector address range and a flash program-bin command to program + the accompanying fwimage.bin image. + +As of fc-host-tools-r13 both methods work equally well: if you have an official +FreeCalypso firmware release (containing fwimage.m0, fwimage.bin and +flash-script files) which you need to flash into a device such as our own +FCDEV3B or OM GTA02 (but *not* Mot C1xx!), you can execute either +'exec flash-script' or 'flash e-program-m0 fwimage.m0' at the loadtool> prompt, +and both ways will produce exactly the same result with equal performance and +reliability. And if you need a more special operation such as erasing the +entire flash (factory production lines) or erasing and reprogramming only a +certain part of the normally affected sector range, that's what custom command +scripting ability is for. + +For the sake of symmetry, we also have a flash e-program-bin command that is a +binary image format counterpart to flash e-program-m0: it first erases the +sectors into which new bits will be programmed, then programs the new bits. +Thus a third equally good way to flash a new FreeCalypso fw release into a +target such as FCDEV3B or GTA02 is to execute +'flash e-program-bin 0 fwimage.bin' - but don't *ever* do it on a Mot C1xx +phone! + +Flash backup and restore +======================== + +A completely different paradigm takes place on alien targets such as Motorola +C1xx and Pirelli DP-L10, made by alien manufacturers, meaning not FreeCalypso, +not Openmoko and not TI. The most important flash operation on these alien +targets is making a flash dump; these dumps can then be used for forensics, +reverse engineering, or simply as a backup. When we subsequently write to +flash on these alien targets (after having saved a backup first), we are not +flashing an m0 fw image or a binary image made from one with mokosrec2bin, +instead the most common operations are: + +* Flashing a backup image back into the same device it was originally made + from (flash restore); + +* Changing a device from one firmware version to a different one by programming + its flash with firmware bits that were originally read out from some fw-donor + unit; + +* Surgical manipulations such as erasing FFS sectors or rewriting one specific + part of the flash based on reverse-engineered understanding of its structure. + +This different paradigm leads to a different mode of usage for fc-loadtool: +instead of needing a maximally-automated operation that flashes a firmware +release image with as little user thought involvement as possible, our flash +manipulations need to be of a more manual peek-n-poke manner. We provide a +flash dump2bin command for making and saving flash dumps first and foremost, +allowing any part of the flash to be dumped and saved selectively if desired, +including the second flash bank on the Pirelli DP-L10 and likewise on our own +FCDEV3B. When it comes to flash write operations, we provide a manual flash +erase command that allows (and requires) the operator to specify exactly which +sector range should be erased and a manual flash program-bin command that +allows any range of 16-bit words to be programmed at any flash address, with +the bits to be programmed coming from a binary file, either the whole file or +any specified subrange. + +These manual flash erase and flash program-bin commands give full control to +the operator, allowing every possible flash manipulation which the hardware +itself allows, at the expense of requiring the operator to think about which +flash addresses, offsets and lengths need to be operated on, and either enter +long commands manually or write a command script. + +Given our historical origins (long before we got to the point of producing our +own hardware, we started out by exploring the forbidden GSM realm of devices +made by alien manufacturers who were hostile to our cause), our original flash +manipulation support in fc-loadtool had been centered around the manual +peek-n-poke paradigm, with elementary flash erase and flash program-bin commands +as our main staple, and no thought had been given originally to producing +functionality that would work like FLUID or like our current flash e-program-m0 +and e-program-bin commands. But all actively maintained software evolves, and +as our FreeCalypso family of projects has matured over the years, we now offer +richer functionality covering a wider range of use cases. + +Binary vs. S-records +==================== + +(Please read the companion article Binary-file-formats for background, then + come back here.) + +If you are exploring and manipulating the flash content of a GSM device in an +aftermarket fashion, as opposed to flashing your own fw builds into your own hw +design produced by your own company like Openmoko did in the late 2000s and +like we do currently at FreeCalypso HQ, then binary is the generally preferred +format: you make dumps with flash dump2bin, and when you selectively program +these images back into devices, you use flash program-bin with the right offsets +and length, along with appropriate flash erase commands. + +We also have flash dump2srec and flash program-srec commands in fc-loadtool, +they were implemented back in the founding stage of FreeCalypso in 2013 for the +sake of completeness and symmetry (it seemed right to support both binary and +S-record formats), but they never got any practical use: if you are making a +flash dump, you would normally want to examine it afterward, and any such +examination almost always needs a straight binary image, not S-records. +Furthermore, our flash program-bin command allows you to selectively program +just a particular portion of a binary image file into flash, at any arbitrary +flash address, but we don't have the same flexibility with flash program-srec - +the latter command is really just a sibling of program-m0 with the opposite +byte order. + +Thus the short summary is as follows: + +* If you are flashing an official firmware release image into your device, you + need to use flash e-program-bin or flash e-program-m0 depending on whether + the image is provided in *.bin or *.m0 format, or alternatively our older + flash program-bin or flash program-m0 commands preceded by a separate flash + erase command with the right sector range, possibly packaged in a supplied + fc-loadtool command script. + +* If you are restoring a flash dump made with flash dump2bin or performing + aftermarket flash manipulations on Mot C1xx or Pirelli DP-L10 phones or other + such alien devices, you need to use binary-format-based flash manipulations + commands; the specific commands will depend on exactly what you are seeking + to do. + +* flash program-srec and e-program-srec commands do not currently have a valid + use case. + +Special considerations for Compal phones +======================================== + +Motorola C1xx and Sony Ericsson J100 phones made by Compal have brickable flash: +the right kind of flash-resident bootloader must always be present at the +beginning of the flash, or else the phone is unrecoverably bricked. We have +special support in fc-loadtool for minimizing the bricking vulnerability window +when operating on these phones, but this special support requires user +cooperation, meaning that you must limit your flash manipulations on these +phones to a narrower subset: + +* flash program-m0, program-srec, e-program-m0 and e-program-srec commands are + not appropriate for these brickable phones - do not use any of these commands + on these targets. + +* Flash sector 0 must be manipulated only with the special + flash erase-program-boot command, not any of the regular erase or program + commands. + +* Regular flash erase, flash program-bin and flash e-program-bin commands can + and should be used for the rest of the flash starting at offset 0x10000 - but + you still need to understand what you are doing.