FreeCalypso > hg > freecalypso-sw
changeset 250:3d88461d8284
beginning of TIFFS IVA documentation
| author | Michael Spacefalcon <msokolov@ivan.Harhan.ORG> |
|---|---|
| date | Fri, 31 Jan 2014 07:37:39 +0000 |
| parents | 66a6f1652909 |
| children | 63cd64625597 |
| files | README doc/TIFFS doc/TIFFS-Overview ffstools/README ffstools/Usage |
| diffstat | 5 files changed, 487 insertions(+), 370 deletions(-) [+] |
line wrap: on
line diff
--- a/README Mon Jan 27 08:05:43 2014 +0000 +++ b/README Fri Jan 31 07:37:39 2014 +0000 @@ -6,8 +6,15 @@ modems and basic phones, consisting of Nucleus RTOS, RiViera and GPF frameworks, TI's BSP/driver and L1 code, Condat G23 GSM/GPRS protocol stack, AT command interpreter and UI layers. - The current code is in the earliest embryonic stages of - development, and does not do anything useful yet. + + At the present, none of the actual GSM code (L1 or G23) has + been integrated yet, but most of the underlying RTOS environment + is present and working: one can exercise RVT, ETM and FFS. + L1 integration will begin soon. + +ffstools Here you will find tools for "in vitro" examination of FFS + (flash file system) images read out of TI-based GSM devices. + See doc/TIFFS-Overview for more information. loadtools The suite of tools which run on a development host machine (normally PC/Linux desktop or laptop) and communicate with
--- a/doc/TIFFS Mon Jan 27 08:05:43 2014 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,368 +0,0 @@ -All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon) -implement some kind of flash file system, or FFS. Several different FFS code -implementations, and correspondingly several different on-flash data formats, -have been used throughout the history of TI's involvement in the wireless -terminal business. The FFS incarnation of primary interest to the FreeCalypso -project is the one invented by Mads Meisner-Jensen at TI in the early 2000s -(at least according to the comments in the sources available to us), and it is -relevant to us in the following ways: - -* When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to - work with the original FFS from the factory (call it MokoFFS), the same FFS - as used by the mokoN firmwares: this FFS contains the IMEI and the RF - calibration values from the factory, which we most certainly don't want to go - without. - -* The Leonardo firmware semi-src which we are using as the reference for - building our own full source, multi-target GSM fw contains a turnkey-working - implementation of this very FFS, using the on-flash format in question and - providing run-time APIs expected by the rest of the GSM fw suite. Following - the principle of ``if it ain't broke, don't fix it'', we can use this FFS not - only on the gtamodem target, but also on other targets, including those where - we would be starting from a blank state and thus have the freedom to use - whatever FFS we like. - -* The original proprietary fw on the Pirelli DP-L10 phone also happens to use - an FFS in the same format. Pirelli's FFS does *not* contain the IMEI or any - of the RF calibration values though, and trying to reuse it directly for our - own FC GSM fw seems to be more trouble than benefit - so we'll probably have - our fw start with a blank TIFFS instead - but there is still insight to be - gained from in-vitro examination of captured Pirelli FFS images. - -Naming -====== - -I have previously referred to the FFS format in question as Mokopir-FFS or -MPFFS, from "Moko" and "Pirelli". I was originally hesitant to call it TIFFS, -as lacking the source code, I had no way of knowing whether the FFS format and -implementation were of TI's own invention, or something that TI licensed as a -black box from one of their many proprietary software partners. (I was unable -to identify it as any well-known, industry-standard FFS format, but absence of -evidence is not evidence of absence.) But now that we have TI's original source -code which implements this FFS (first the MV100-0.1.rar source, then the full -Leonardo one), complete with comments and a HISTORY file, we know that our FFS -was invented and implemented by someone named Mads Meisner-Jensen at TI - I'm -guessing in the SSA group in Nice, France. - -I am now making a naming transition from MPFFS to TIFFS: there is really no -link between this FFS format and the Openmoko+Pirelli duo, other than the -happenstance of me having first encountered this FFS on these two GSM device -brands, and the name TIFFS is more neutrally-descriptive. - -What it is -========== - -In a rare departure from TI's norm (most of TI's GSM firmware and associated -development tools suffer from heavy Windows poisoning), what I call TIFFS is -very Unixy. It is a file system with a hierarchical directory tree structure -and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics -of "what is a file" and "what is a directory" are exactly the same as in UNIX; -and TIFFS even supports symlinks, although that support is a little under- -developed, and apparently no FFS symlinks were ever used in any production GSM -device. Thus the FFS implemented in TI-based GSM devices (modems and -"dumbphones") is really no different from, for example, JFFS2 in embedded Linux -systems. - -(The only traditional UNIX file system features which are missing in TIFFS are - the creation/modification/access timestamps and the ownership/permission - fields.) - -The FFS in a GSM device typically stores two kinds of content: - -* Factory data: IMEI, RF calibration values, device make/model/revision - ID strings etc. These files are expected to be programmed on the factory - production line and not changed afterward. - -* Dynamic data written into the FFS in normal device operation: when you use a - "dumbphone" running TI-based firmware, every time you store something "on the - phone" or in "non-volatile memory", that item is actually stored in the FFS. - (Where else, if you think of it?) That includes contacts and received SMS - stored "on the phone" instead of the SIM, any selections you make in the - settings/preferences menus which persist across reboots (power cycles), call - history etc. - -It needs to be noted that the "dynamic data" aspect of FFS usage applies not -only to complete phones, but also to modems like the one used in the GTA01/02. -One would naively think that non-volatile storage of data in flash outside of -factory programming would be needed only in a device with its own UI, and that -a modem subservient to external AT commands would be completely stateless -across reboot/power cycles; but that is not the case in actuality. TI's GSM -firmwares, including the Openmoko ones (the "standard" mokoN), are designed to -always "mount" their FFS with read/write access; TI's FFS implementation in the -firmware has no concept of a "read-only mount". - -I am still investigating just what kinds of data are routinely written into the -non-volatile FFS by the firmware in normal operation on devices like the GTA0x -modem, but there most definitely are some. - -There is no hard separation between "static" and "dynamic" data in the file -system structure; TIFFS is thus akin to an embedded Linux system with just a -single root file system containing both "static" files like userland binaries -and "dynamic" ones like configuration files under /etc which the user is -expected to edit with vi after logging into the box, or log and similar files -created by the system itself under /var, for example. - -Where it lives -============== - -The type of flash memory used in Calypso GSM modems and "dumbphones" is called -NOR flash. This NOR flash memory is physically divided (by the design of the -flash chip itself) into units called "sectors" or more descriptively, erase -blocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from -64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the -S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total). -The key physical property is that any bit may be changed from a '1' to a '0' at -any time, in any combination, but resetting of '0' bits back to ones can be -done only on the granularity of these largish sectors, in an operation called -"sector erase". - -The location of TIFFS within the flash memory of a given GSM device is defined -by the firmware design of that device, but is always some integral number of -contiguous flash sectors. Some examples: - -* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at - flash offset 0x380000. - -* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies - 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning - of the 2nd flash chip select (0x02000000 in the ARM7 address space). - -* The smallest real FFS configuration called for by the table in dev.c in TI's - original Leonardo fw source is 3 sectors of 64 KiB each; the same table also - sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in - RAM without real flash). - -* The largest FFS configuration that has been envisioned by the original - designers seems to be somewhere around 128 sectors. - -Each flash sector used for TIFFS begins with this 6-byte signature: - -46 66 73 23 10 02 - -The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the -format version number of 0x0210 in little-endian byte order. The following two -bytes give a count of how many times that sector has been erased and rewritten -(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates -that block's role and status in the FFS life cycle. - -How it works -============ - -Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to -recover gracefully from any possible power failure or crash: one can yank the -battery from the GSM device (or induce a firmware crash) at the most mis- -opportune moment in the middle of an FFS write operation, and the FFS is -expected to recover on the next boot cycle. I won't be able to document here -all gory details of exactly how this goal is achieved, partly because I haven't -studied the code to the requisite level of depth myself yet, but all of the -responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source -tree; feel free to study it. - -In its "normal" or "clean" state (i.e., when not in the middle of a write -operation or recovery from an ungracefully interrupted one), a TIFFS instance -consists of the following 3 types of blocks: - -* One block containing inode records, indicated by AB in its type/flags/status - byte in the block header; -* N-2 blocks (where N is the total number of flash sectors allocated for the - FFS) containing (or waiting to be filled with) data chunks - indicated by BD - in the type/flags/status byte; -* One "free" block, indicated by BF - destined to become a new AB or a new BD - at some point. - -Each object written into the FFS (file, directory or symlink) consists of a -16-byte inode record written into the AB block and a data chunk written into -one of the BD blocks. The data chunk includes the name of the object, hence -one is required even for directories. Data chunks are contiguous, uncompressed, -and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS -configuration. Files larger than this limit are stored in a "segmented" form, -giving rise to a 4th inode or object type (after file, directory and symlink): -segment. Each segment of a segmented file consists of not only a data chunk, -but also an inode record for the segment, which gives the location of the data -chunk and ties the segment object into the overall FFS structure, making it -accessible. - -Because aside from complete sector erasure, flash memory bits can only -transition from '1' to '0' but not the other way around, overwriting an existing -file with some new content (an operation which any reasonable file system must -implement in some way) cannot be done in place. Instead like most flash file -systems, TIFFS implements this common operation by writing the new version of -the file to a new location (previously blank flash) and then invalidating the -old version - and doing all that while keeping in mind the possibility of an -ungraceful crash or powerdown at any moment, and the requirement of recovering -gracefully from any such event. - -Of course as an FFS receives more write activity, even if one keeps overwriting -some existing files with new content of the same size, without adding to the -visible total content size (think du(1) command), eventually all remaining blank -flash space will fill up. At that point (or at some earlier point, depending -on the FFS design and/or configuration) the FFS has to invoke a compaction or -reclamation or garbage collection procedure: any "mixed" blocks containing both -valid and stale data are transitioned into a "stale-only" state by having the -active data moved to a new block, and then the "all stale" blocks are subjected -to sector erasure, becoming new blank sectors. The logic responsible for these -operations once again needs to be resilient to the possibility of a crash or -powerdown occurring at the most mis-opportune moment, and it also needs to -implement flash wear leveling: there is a physical limit to how many times a -given flash sector can be erased and rewritten before it goes bad. - -All of the above are common and well-known principles, successfully implemented -in well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutely -no different in this regard; for the implementation details, read the source -code. - -How this FFS comes into being -============================= - -(This section is only relevant to you if you plan on physically producing your - own GSM phones or modems on your own factory production line, like this author - fancies doing in the not-too-distant future, or if you simply enjoy knowing - how it is done.) - -To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the -embedded Linux world, which would produce a TIFFS image complete with some -initial directory and file content "in vitro". Instead it appears that the FFS -instances found in shipped products such as Openmoko phones have been created -"in vivo" by TI's firmware running on the device itself during the "production -test" phase. - -The process seems to go like this: - -* When the printed circuit board is physically populated with components such - as the Calypso chip and the flash chip, the latter can be blank - if the - board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM - (Openmoko and Pirelli both good on this one, but shame on Compal!), there is - no need to preprogram the flash chip with anything prior to populating it on - the board, and the device remains fully unbrickable at all times afterward. - -* When the assembled board is powered up for the first time, with completely - blank flash, the Calypso boot ROM will sit there and patiently wait for a - code download on either of its two UARTs. - -* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's - fc-loadtool free replacement, the factory production station loads the main - firmware image into the flash. Note, it is just the firmware image at this - step, and the FFS sectors remain blank. - -* The board is commanded to reboot (or power-cycled), and the firmware image - boots for the first time. - -* TI's FFS implementation code in their standard firmware reacts to all blank - flash in the FFS sectors as follows: it performs what they call the preformat - operation, writing the TIFFS signature and a BF state byte into every FFS - sector, but the main "format" operation, which sets up the AB/BD block roles, - creates the root inode and makes the FFS ready to accept the creation of its - first directories and files, is not done automatically. - -In order to perform the FFS format operation and then fill the new FFS with -whatever directories and files are deemed needed to be present in "fresh" -shipping products, the factory production station connects to the just-booted -firmware running on the target via the RVT/ETM protocol (see the RVTMUX -write-up), and sends "test mode" commands to this running firmware. These -"FFS test mode" (or TMFFS) commands include the format operation, an mkdir -operation to create directories, and a "file write" operation akin to doing -'cat > /dir/whatever/file', creating files in FFS and storing any desired data -in them. - -The IMEI is assigned and written into FFS in this process, but it is not the -only data item that will be unique for each individual device made. Much more -important are the RF calibration values: I have yet to learn exactly what is -being (or needs to be) measured, how these measurements are performed (under -what conditions; what external test equipment is needed), and how these measured -and recorded RF calibration values affect GSM device operation, but this TI -presentation gives some clues: - -ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf - -All of these calibration values are stored in a bunch of files under the -/gsm/rf subtree, and these files seem to be "owned" by the L1 code. The latter -has RAM data structures which correspond to these files; upon normal boot the -initialization code looks in FFS, and if it finds any of the RF calibration -files, it reads each present file into the corresponding RAM data structure, -overwriting the compiled-in defaults. It appears (slightly uncertain because I -have not yet reintegrated the code in question into our own gsm-fw) that the RF -calibration files in FFS come into being as follows: - -* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the - measurements and stores results in its RAM data structures as commanded by - the production test station through the "test mode" interface; - -* A final test mode command directs the above L1 code to write its RAM data - structures into FFS. - -Once I actually learn this RF calibration process properly in connection with -building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what -it would take to recreate these RF calibration values if they are lost. But -until then the only advice I can give is to make a backup copy of your modem -FFS with fc-loadtool, and to save it securely. - -FreeCalypso support for TIFFS -============================= - -Aside from implementing and using it in our own gsm-fw, FreeCalypso will offer -the following support for TIFFS: - -1. A tiffs host utility is being written which will allow a user to list and - extract content from saved FFS images (read out of flash with fc-loadtool) - "in vitro". It will be a restructured and (hopefully) improved version of - the mpffs-* tools released back in the summer of SE52 (A.D. 2013); the latter - already perform the advertised function, but I seek to integrate some other - functionality which I developed in an ad hoc side project ("pirollback"), - and I'm taking the opportunity to make the MPFFS->TIFFS renaming. - -(The mpffs-* tools mentioned above have been written based on reverse eng only, - before I found any source code for TI's FFS firmware implementation! Now that - we have the source, some terminological and other inevitable misunderstandings - can be corrected.) - -2. A number of FC tools may be strung together into a kit for editing the FFS - content of a GSM device, e.g., for changing the IMEI. The following pieces - will be involved: - -* What is destined to eventually become our totally free GSM fw (the gsm-fw - source subtree at the top of freecalypso-sw) does not contain any of the - actual GSM protocol stack (or even L1) functionality yet, but it already - contains both the FFS code and those components (ETM and TMFFS[12]) which - are needed for interfacing an external "test mode shell" to this FFS - implementation through the RVTMUX interface. And when our gsm-fw does gain - the actual GSM functionality, the ability to build a minimal FFS+ETM-only - configuration will still be retained. - -* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs - entirely from RAM via fc-xram, no flashing), and run on a physical device - such as the GTA0x GSM modem via the fc-xram host utility; - -* After loading the ramImage, fc-xram will immediately exec our rvinterf host - utility described in the RVTMUX write-up; - -* Once the GSM device is running what is effectively an FFS editing agent out - of RAM, accessed via rvinterf over the serial channel, the user will be able - to run fc-tmsh (or perhaps the FFS operations will be implemented in some - other utility, we'll see), and that "test mode shell" will provide commands - for writing things to FFS exactly like one would do in the factory production - line environment for which TI taylored their tools. - -The "in vivo" method of editing the FFS content of a GSM device described above -will probably sound very convoluted, and you may find yourself asking for a way -to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit -that image "in vitro" with some utility on your PC, and then use fc-loadtool -again to program it back into your device. But consider that an "in vitro" FFS -modification would involve erasing and rewriting all sectors of your FFS, -whereas an "in vivo" modification of some small file like the IMEI would be -just a short flash write operation without any erasures at all, i.e., kinder -on the flash. - -In any case, the "in vivo" method will definitely be available soon because all -of the components involved therein are also needed for other development uses -in the FreeCalypso project, whereas developing a fully-functional "in vitro" -alternative (one that can create an FFS image "de novo" from a tree of files -and directories a la mkfs.jffs2, or add new files to an existing TIFFS image -etc) would be a good amount of extra work which we otherwise don't need - hence -the latter is not very likely to be written any time soon. - -However, if the "in vitro" modification you seek is something trivial like -changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without -changing its length, you will be able to use the "in vitro, read-only" tiffs -host utility to find the exact byte location of the file data within the TIFFS -image, and use your favourite hex editor to whack whatever new byte content you -like at that offset.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/TIFFS-Overview Fri Jan 31 07:37:39 2014 +0000 @@ -0,0 +1,363 @@ +All TI GSM firmwares known to this author (FreeCalypso developer Space Falcon) +implement some kind of flash file system, or FFS. Several different FFS code +implementations, and correspondingly several different on-flash data formats, +have been used throughout the history of TI's involvement in the wireless +terminal business. The FFS incarnation of primary interest to the FreeCalypso +project is the one invented by Mads Meisner-Jensen at TI in the early 2000s +(at least according to the comments in the sources available to us), and it is +relevant to us in the following ways: + +* When targeting the GSM modem in Openmoko's GTA01/02 smartphones, we need to + work with the original FFS from the factory (call it MokoFFS), the same FFS + as used by the mokoN firmwares: this FFS contains the IMEI and the RF + calibration values from the factory, which we most certainly don't want to go + without. + +* The Leonardo firmware semi-src which we are using as the reference for + building our own full source, multi-target GSM fw contains a turnkey-working + implementation of this very FFS, using the on-flash format in question and + providing run-time APIs expected by the rest of the GSM fw suite. Following + the principle of ``if it ain't broke, don't fix it'', we can use this FFS not + only on the gtamodem target, but also on other targets, including those where + we would be starting from a blank state and thus have the freedom to use + whatever FFS we like. + +* The original proprietary fw on the Pirelli DP-L10 phone also happens to use + an FFS in the same format. Pirelli's FFS does *not* contain the IMEI or any + of the RF calibration values though, and trying to reuse it directly for our + own FC GSM fw seems to be more trouble than benefit - so we'll probably have + our fw start with a blank TIFFS instead - but there is still insight to be + gained from in-vitro examination of captured Pirelli FFS images. + +Naming +====== + +I have previously referred to the FFS format in question as Mokopir-FFS or +MPFFS, from "Moko" and "Pirelli". I was originally hesitant to call it TIFFS, +as lacking the source code, I had no way of knowing whether the FFS format and +implementation were of TI's own invention, or something that TI licensed as a +black box from one of their many proprietary software partners. (I was unable +to identify it as any well-known, industry-standard FFS format, but absence of +evidence is not evidence of absence.) But now that we have TI's original source +code which implements this FFS (first the MV100-0.1.rar source, then the full +Leonardo one), complete with comments and a HISTORY file, we know that our FFS +was invented and implemented by someone named Mads Meisner-Jensen at TI - I'm +guessing in the SSA group in Nice, France. + +I am now making a naming transition from MPFFS to TIFFS: there is really no +link between this FFS format and the Openmoko+Pirelli duo, other than the +happenstance of me having first encountered this FFS on these two GSM device +brands, and the name TIFFS is more neutrally-descriptive. + +What it is +========== + +In a rare departure from TI's norm (most of TI's GSM firmware and associated +development tools suffer from heavy Windows poisoning), what I call TIFFS is +very Unixy. It is a file system with a hierarchical directory tree structure +and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics +of "what is a file" and "what is a directory" are exactly the same as in UNIX; +and TIFFS even supports symlinks, although that support is a little under- +developed, and apparently no FFS symlinks were ever used in any production GSM +device. Thus the FFS implemented in TI-based GSM devices (modems and +"dumbphones") is really no different from, for example, JFFS2 in embedded Linux +systems. + +(The only traditional UNIX file system features which are missing in TIFFS are + the creation/modification/access timestamps and the ownership/permission + fields.) + +The FFS in a GSM device typically stores two kinds of content: + +* Factory data: IMEI, RF calibration values, device make/model/revision + ID strings etc. These files are expected to be programmed on the factory + production line and not changed afterward. + +* Dynamic data written into the FFS in normal device operation: when you use a + "dumbphone" running TI-based firmware, every time you store something "on the + phone" or in "non-volatile memory", that item is actually stored in the FFS. + (Where else, if you think of it?) That includes contacts and received SMS + stored "on the phone" instead of the SIM, any selections you make in the + settings/preferences menus which persist across reboots (power cycles), call + history etc. + +It needs to be noted that the "dynamic data" aspect of FFS usage applies not +only to complete phones, but also to modems like the one used in the GTA01/02. +One would naively think that non-volatile storage of data in flash outside of +factory programming would be needed only in a device with its own UI, and that +a modem subservient to external AT commands would be completely stateless +across reboot/power cycles; but that is not the case in actuality. TI's GSM +firmwares, including the Openmoko ones (the "standard" mokoN), are designed to +always "mount" their FFS with read/write access; TI's FFS implementation in the +firmware has no concept of a "read-only mount". + +I am still investigating just what kinds of data are routinely written into the +non-volatile FFS by the firmware in normal operation on devices like the GTA0x +modem, but there most definitely are some. + +There is no hard separation between "static" and "dynamic" data in the file +system structure; TIFFS is thus akin to an embedded Linux system with just a +single root file system containing both "static" files like userland binaries +and "dynamic" ones like configuration files under /etc which the user is +expected to edit with vi after logging into the box, or log and similar files +created by the system itself under /var, for example. + +Where it lives +============== + +The type of flash memory used in Calypso GSM modems and "dumbphones" is called +NOR flash. This NOR flash memory is physically divided (by the design of the +flash chip itself) into units called "sectors" or more descriptively, erase +blocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from +64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the +S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total). +The key physical property is that any bit may be changed from a '1' to a '0' at +any time, in any combination, but resetting of '0' bits back to ones can be +done only on the granularity of these largish sectors, in an operation called +"sector erase". + +The location of TIFFS within the flash memory of a given GSM device is defined +by the firmware design of that device, but is always some integral number of +contiguous flash sectors. Some examples: + +* On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at + flash offset 0x380000. + +* On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies + 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning + of the 2nd flash chip select (0x02000000 in the ARM7 address space). + +* The smallest real FFS configuration called for by the table in dev.c in TI's + original Leonardo fw source is 3 sectors of 64 KiB each; the same table also + sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in + RAM without real flash). + +* The largest FFS configuration that has been envisioned by the original + designers seems to be somewhere around 128 sectors. + +Each flash sector used for TIFFS begins with this 6-byte signature: + +46 66 73 23 10 02 + +The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the +format version number of 0x0210 in little-endian byte order. The following two +bytes give a count of how many times that sector has been erased and rewritten +(FF FF in "fresh" or "virgin" FFS images), and the following byte indicates +that block's role and status in the FFS life cycle. + +How it works +============ + +Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to +recover gracefully from any possible power failure or crash: one can yank the +battery from the GSM device (or induce a firmware crash) at the most mis- +opportune moment in the middle of an FFS write operation, and the FFS is +expected to recover on the next boot cycle. I won't be able to document here +all gory details of exactly how this goal is achieved, partly because I haven't +studied the code to the requisite level of depth myself yet, but all of the +responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source +tree; feel free to study it. + +In its "normal" or "clean" state (i.e., when not in the middle of a write +operation or recovery from an ungracefully interrupted one), a TIFFS instance +consists of the following 3 types of blocks: + +* One block containing inode records, indicated by AB in its type/flags/status + byte in the block header; +* N-2 blocks (where N is the total number of flash sectors allocated for the + FFS) containing (or waiting to be filled with) data chunks - indicated by BD + in the type/flags/status byte; +* One "free" block, indicated by BF - destined to become a new AB or a new BD + at some point. + +Each object written into the FFS (file, directory or symlink) consists of a +16-byte inode record written into the AB block and a data chunk written into +one of the BD blocks. The data chunk includes the name of the object, hence +one is required even for directories. Data chunks are contiguous, uncompressed, +and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS +configuration. Files larger than this limit are stored in a "segmented" form, +giving rise to a 4th inode or object type (after file, directory and symlink): +segment. Each segment of a segmented file consists of not only a data chunk, +but also an inode record for the segment, which gives the location of the data +chunk and ties the segment object into the overall FFS structure, making it +accessible. + +Because aside from complete sector erasure, flash memory bits can only +transition from '1' to '0' but not the other way around, overwriting an existing +file with some new content (an operation which any reasonable file system must +implement in some way) cannot be done in place. Instead like most flash file +systems, TIFFS implements this common operation by writing the new version of +the file to a new location (previously blank flash) and then invalidating the +old version - and doing all that while keeping in mind the possibility of an +ungraceful crash or powerdown at any moment, and the requirement of recovering +gracefully from any such event. + +Of course as an FFS receives more write activity, even if one keeps overwriting +some existing files with new content of the same size, without adding to the +visible total content size (think du(1) command), eventually all remaining blank +flash space will fill up. At that point (or at some earlier point, depending +on the FFS design and/or configuration) the FFS has to invoke a compaction or +reclamation or garbage collection procedure: any "mixed" blocks containing both +valid and stale data are transitioned into a "stale-only" state by having the +active data moved to a new block, and then the "all stale" blocks are subjected +to sector erasure, becoming new blank sectors. The logic responsible for these +operations once again needs to be resilient to the possibility of a crash or +powerdown occurring at the most mis-opportune moment, and it also needs to +implement flash wear leveling: there is a physical limit to how many times a +given flash sector can be erased and rewritten before it goes bad. + +All of the above are common and well-known principles, successfully implemented +in well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutely +no different in this regard; for the implementation details, read the source +code. + +How this FFS comes into being +============================= + +(This section is only relevant to you if you plan on physically producing your + own GSM phones or modems on your own factory production line, like this author + fancies doing in the not-too-distant future, or if you simply enjoy knowing + how it is done.) + +To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the +embedded Linux world, which would produce a TIFFS image complete with some +initial directory and file content "in vitro". Instead it appears that the FFS +instances found in shipped products such as Openmoko phones have been created +"in vivo" by TI's firmware running on the device itself during the "production +test" phase. + +The process seems to go like this: + +* When the printed circuit board is physically populated with components such + as the Calypso chip and the flash chip, the latter can be blank - if the + board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM + (Openmoko and Pirelli both good on this one, but shame on Compal!), there is + no need to preprogram the flash chip with anything prior to populating it on + the board, and the device remains fully unbrickable at all times afterward. + +* When the assembled board is powered up for the first time, with completely + blank flash, the Calypso boot ROM will sit there and patiently wait for a + code download on either of its two UARTs. + +* Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's + fc-loadtool free replacement, the factory production station loads the main + firmware image into the flash. Note, it is just the firmware image at this + step, and the FFS sectors remain blank. + +* The board is commanded to reboot (or power-cycled), and the firmware image + boots for the first time. + +* TI's FFS implementation code in their standard firmware reacts to all blank + flash in the FFS sectors as follows: it performs what they call the preformat + operation, writing the TIFFS signature and a BF state byte into every FFS + sector, but the main "format" operation, which sets up the AB/BD block roles, + creates the root inode and makes the FFS ready to accept the creation of its + first directories and files, is not done automatically. + +In order to perform the FFS format operation and then fill the new FFS with +whatever directories and files are deemed needed to be present in "fresh" +shipping products, the factory production station connects to the just-booted +firmware running on the target via the RVT/ETM protocol (see the RVTMUX +write-up), and sends "test mode" commands to this running firmware. These +"FFS test mode" (or TMFFS) commands include the format operation, an mkdir +operation to create directories, and a "file write" operation akin to doing +'cat > /dir/whatever/file', creating files in FFS and storing any desired data +in them. + +The IMEI is assigned and written into FFS in this process, but it is not the +only data item that will be unique for each individual device made. Much more +important are the RF calibration values: I have yet to learn exactly what is +being (or needs to be) measured, how these measurements are performed (under +what conditions; what external test equipment is needed), and how these measured +and recorded RF calibration values affect GSM device operation, but this TI +presentation gives some clues: + +ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf + +All of these calibration values are stored in a bunch of files under the +/gsm/rf subtree, and these files seem to be "owned" by the L1 code. The latter +has RAM data structures which correspond to these files; upon normal boot the +initialization code looks in FFS, and if it finds any of the RF calibration +files, it reads each present file into the corresponding RAM data structure, +overwriting the compiled-in defaults. It appears (slightly uncertain because I +have not yet reintegrated the code in question into our own gsm-fw) that the RF +calibration files in FFS come into being as follows: + +* The RF calibration code in L1 (i.e., part of the main GSM fw) performs the + measurements and stores results in its RAM data structures as commanded by + the production test station through the "test mode" interface; + +* A final test mode command directs the above L1 code to write its RAM data + structures into FFS. + +Once I actually learn this RF calibration process properly in connection with +building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what +it would take to recreate these RF calibration values if they are lost. But +until then the only advice I can give is to make a backup copy of your modem +FFS with fc-loadtool, and to save it securely. + +FreeCalypso support for TIFFS +============================= + +Aside from implementing and using it in our own gsm-fw, FreeCalypso offers +the following support for TIFFS: + +1. We have a utility for "in vitro" examination of FFS images read out of GSM + devices with fc-loadtool. This tiffs utility (along with mokoffs and pirffs + wrappers) lives in the ffstools top-level directory of the freecalypso-sw + source tree. This TIFFS "in vitro analyzer" utility supplants the earlier + mpffs-* tools, and adds some additional examination functionality. It is + strictly a "read only" tool, however - it is not designed for "in vitro" + editing" of TIFFS images. + +2. A number of FC tools may be strung together into a kit for editing the FFS + content of a GSM device, e.g., for changing the IMEI. The following pieces + will be involved: + +* What is destined to eventually become our totally free GSM fw (the gsm-fw + source subtree at the top of freecalypso-sw) does not contain any of the + actual GSM protocol stack (or even L1) functionality yet, but it already + contains both the FFS code and those components (ETM and TMFFS[12]) which + are needed for interfacing an external "test mode shell" to this FFS + implementation through the RVTMUX interface. And when our gsm-fw does gain + the actual GSM functionality, the ability to build a minimal FFS+ETM-only + configuration will still be retained. + +* The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs + entirely from RAM via fc-xram, no flashing), and run on a physical device + such as the GTA0x GSM modem via the fc-xram host utility; + +* After loading the ramImage, fc-xram will immediately exec our rvinterf host + utility described in the RVTMUX write-up; + +* Once the GSM device is running what is effectively an FFS editing agent out + of RAM, accessed via rvinterf over the serial channel, the user will be able + to run fc-tmsh (or perhaps the FFS operations will be implemented in some + other utility, we'll see), and that "test mode shell" will provide commands + for writing things to FFS exactly like one would do in the factory production + line environment for which TI taylored their tools. + +The "in vivo" method of editing the FFS content of a GSM device described above +will probably sound very convoluted, and you may find yourself asking for a way +to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit +that image "in vitro" with some utility on your PC, and then use fc-loadtool +again to program it back into your device. But consider that an "in vitro" FFS +modification would involve erasing and rewriting all sectors of your FFS, +whereas an "in vivo" modification of some small file like the IMEI would be +just a short flash write operation without any erasures at all, i.e., kinder +on the flash. + +In any case, the "in vivo" method will definitely be available soon because all +of the components involved therein are also needed for other development uses +in the FreeCalypso project, whereas developing a fully-functional "in vitro" +alternative (one that can create an FFS image "de novo" from a tree of files +and directories a la mkfs.jffs2, or add new files to an existing TIFFS image +etc) would be a good amount of extra work which we otherwise don't need - hence +the latter is not very likely to be written any time soon. + +However, if the "in vitro" modification you seek is something trivial like +changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without +changing its length, you can use the existing "in vitro, read-only" tiffs host +utility to find the exact byte location of the file data within the TIFFS image, +and then use your favourite hex editor to whack whatever new byte content you +like at that offset.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/ffstools/README Fri Jan 31 07:37:39 2014 +0000 @@ -0,0 +1,31 @@ +You are looking at the source for the TIFFS In Vitro Analyzer utility. You may +have downloaded it either as a separate package or as part of the larger +freecalypso-sw suite. + +See TIFFS-Overview (in ../doc if you are working with the full freecalypso-sw +source tree) for a general description of what TIFFS is and why it matters. + +The utility contained in the present package runs on a general purpose GNU/Linux +(or other Unix) host and enables "in vitro" examination of Flash File System +images read out of TI-based GSM devices. Using this utility, you can list the +directory and file content of an FFS image, cat any individual file in the FFS, +or extract the complete FFS content into your regular Unix file system. Some +"forensic" operations are also supported: by listing the inode array, one can +deduce the order in which the present FFS content got created, and see what +files have been overwritten or deleted in the span of still-visible history. +One can then cat the old byte content of those overwritten or deleted files, +if those data chunks are still in the FFS image (i.e., if the flash sector in +question has not been reclaimed yet). + +Compilation and installation are straightforward: run 'make' to compile the +source; you should get 3 executable binaries named tiffs, mokoffs and pirffs; +then run 'make install' as root to install them in /usr/local/bin. The binary +named tiffs is the main program; mokoffs and pirffs are wrappers that simplify +the most common current use cases. + +To install somewhere other than /usr/local/bin, edit the INSTBIN= setting in +the subdirectory Makefiles. You will also need to edit +tiffs-wrappers/installpath.c accordingly, as the mokoffs and pirffs wrappers +are designed to exec tiffs by its absolute installed pathname. + +See Usage for the usage instructions.
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/ffstools/Usage Fri Jan 31 07:37:39 2014 +0000 @@ -0,0 +1,84 @@ +The generic tiffs utility needs to be invoked as follows: + +tiffs [global-options] <imgfile> <org> <cmd> [command-args] + +The first 3 non-optional arguments are the filename of the TIFFS image under +examination, the FFS organization being examined, and the operation to be +performed. The present utility is designed in the classic Unix manner in that +each invokation performs a single operation and exits, such that invokations of +tiffs (or one of the wrappers described below) may be plumbed into pipes and +the like. + +The 2nd argument to tiffs after the FFS image filename describes how the TIFFS +instance under study is organized in terms of flash sectors. The syntax of +this argument is KxN, where K is the flash sector size in KiB and N is the +number of sectors occupied by the FFS. For MokoFFS images the correct +organization argument is 64x7 (7 sectors of 64 KiB each); for Pirelli's FFS +images it is 256x18 (18 sectors of 256 KiB each). + +The following global options may be given before the image filename argument: + +-a num + + Use the specified flash block (sector) as the inode array block. + +-o offset + + The FFS image begins at the specified offset within the file, rather + than at the beginning. This option is useful when working with complete + device flash dumps of which FFS is only a part, starting somewhere + other than at 0. + +-r ino + + Use the specified inode as the root. Per Falcon's convention, TIFFS + inode numbers are always given in hex, hence this argument is + interpreted as hex without needing a 0x prefix. + +The invokation syntax for mokoffs and pirffs wrappers is the same as for tiffs, +except that the FFS organization argument (64x7 or 256x18) is omitted; the +wrapper fills that argument in before passing the command to the main tiffs +program. The only other difference is that instead of the generic -o global +option, mokoffs takes a -f global option (no argument) which indicates that one +is working with a complete flash dump image, rather than just the FFS portion; +mokoffs -f gets translated into tiffs -o0x380000. (pirffs has no such option +at all because Pirelli's FFS starts at offset 0 within its respective flash +chip select.) + +The next argument after the FFS organization for tiffs (or after the image +filename for mokoffs/pirffs) is the command (or operation) to be performed. +The following tiffs commands are currently available: + +Standard listing/extraction commands +==================================== + +These commands list or extract the normally-visible content of the FFS, i.e., +the content which is visible when the FFS is "mounted" normally, and which the +FFS promises to preserve - as opposed to deleted or overwritten content. + +ls [-v[v]] [pathname...] + +Tiffs ls without additional arguments yields a listing of the complete FFS +directory tree, akin to tar tv. Example output fragment: + +fr 4096 /.journal +d /gsm +d /gsm/rf +d /gsm/rf/tx +f 512 /gsm/rf/tx/ramps.900 +f 128 /gsm/rf/tx/levels.900 +f 128 /gsm/rf/tx/calchan.900 + +The first character is 'f' for files or 'd' for directories. An 'r' following +immediately afterward means that the object has the read-only attribute set. +For files the listing includes the content size in bytes, and the last part is +the pathname of the object within the FFS. + +With a single -v option added after ls, the output will include verbose +information as to the segmentation structure of each file. With two -v options +or with -vv, this additional output will also include the byte offset of each +data chunk, relative to the beginning of the FFS image. + +Tiffs ls with a pathname argument yields information about the specified FFS +object; -v and -vv options act as already described, but are arguably more +useful when listing single files.