FreeCalypso > hg > freecalypso-sw

comparison doc/TIFFS @ 227:1852900ce9ea

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc/TIFFS write-up
author Michael Spacefalcon <msokolov@ivan.Harhan.ORG>
date Tue, 14 Jan 2014 02:31:09 +0000
parents 2900fe603f8a
children
comparison
equal deleted inserted replaced
226:4d706a4134b0 227:1852900ce9ea
46 46
47 I am now making a naming transition from MPFFS to TIFFS: there is really no 47 I am now making a naming transition from MPFFS to TIFFS: there is really no
48 link between this FFS format and the Openmoko+Pirelli duo, other than the 48 link between this FFS format and the Openmoko+Pirelli duo, other than the
49 happenstance of me having first encountered this FFS on these two GSM device 49 happenstance of me having first encountered this FFS on these two GSM device
50 brands, and the name TIFFS is more neutrally-descriptive. 50 brands, and the name TIFFS is more neutrally-descriptive.
51
52 What it is
53 ==========
54
55 In a rare departure from TI's norm (most of TI's GSM firmware and associated
56 development tools suffer from heavy Windows poisoning), what I call TIFFS is
57 very Unixy. It is a file system with a hierarchical directory tree structure
58 and with Unixy forward-slash-separated, case-sensitive pathnames; the semantics
59 of "what is a file" and "what is a directory" are exactly the same as in UNIX;
60 and TIFFS even supports symlinks, although that support is a little under-
61 developed, and apparently no FFS symlinks were ever used in any production GSM
62 device. Thus the FFS implemented in TI-based GSM devices (modems and
63 "dumbphones") is really no different from, for example, JFFS2 in embedded Linux
64 systems.
65
66 (The only traditional UNIX file system features which are missing in TIFFS are
67 the creation/modification/access timestamps and the ownership/permission
68 fields.)
69
70 The FFS in a GSM device typically stores two kinds of content:
71
72 * Factory data: IMEI, RF calibration values, device make/model/revision
73 ID strings etc. These files are expected to be programmed on the factory
74 production line and not changed afterward.
75
76 * Dynamic data written into the FFS in normal device operation: when you use a
77 "dumbphone" running TI-based firmware, every time you store something "on the
78 phone" or in "non-volatile memory", that item is actually stored in the FFS.
79 (Where else, if you think of it?) That includes contacts and received SMS
80 stored "on the phone" instead of the SIM, any selections you make in the
81 settings/preferences menus which persist across reboots (power cycles), call
82 history etc.
83
84 It needs to be noted that the "dynamic data" aspect of FFS usage applies not
85 only to complete phones, but also to modems like the one used in the GTA01/02.
86 One would naively think that non-volatile storage of data in flash outside of
87 factory programming would be needed only in a device with its own UI, and that
88 a modem subservient to external AT commands would be completely stateless
89 across reboot/power cycles; but that is not the case in actuality. TI's GSM
90 firmwares, including the Openmoko ones (the "standard" mokoN), are designed to
91 always "mount" their FFS with read/write access; TI's FFS implementation in the
92 firmware has no concept of a "read-only mount".
93
94 I am still investigating just what kinds of data are routinely written into the
95 non-volatile FFS by the firmware in normal operation on devices like the GTA0x
96 modem, but there most definitely are some.
97
98 There is no hard separation between "static" and "dynamic" data in the file
99 system structure; TIFFS is thus akin to an embedded Linux system with just a
100 single root file system containing both "static" files like userland binaries
101 and "dynamic" ones like configuration files under /etc which the user is
102 expected to edit with vi after logging into the box, or log and similar files
103 created by the system itself under /var, for example.
104
105 Where it lives
106 ==============
107
108 The type of flash memory used in Calypso GSM modems and "dumbphones" is called
109 NOR flash. This NOR flash memory is physically divided (by the design of the
110 flash chip itself) into units called "sectors" or more descriptively, erase
111 blocks. The typical NOR flash sector size (in Calypso GSM devices) ranges from
112 64 KiB in the GTA02 modem's NOR flash (4 MiB total) to 256 KiB in the
113 S71PL129NC0 flash+RAM chip used in the Pirelli DP-L10 (16 MiB of flash total).
114 The key physical property is that any bit may be changed from a '1' to a '0' at
115 any time, in any combination, but resetting of '0' bits back to ones can be
116 done only on the granularity of these largish sectors, in an operation called
117 "sector erase".
118
119 The location of TIFFS within the flash memory of a given GSM device is defined
120 by the firmware design of that device, but is always some integral number of
121 contiguous flash sectors. Some examples:
122
123 * On the GTA01/02 GSM modem, FFS occupies 7 sectors of 64 KiB each, starting at
124 flash offset 0x380000.
125
126 * On the Pirelli DP-L10, the FFS used by the original proprietary fw occupies
127 18 sectors of 256 KiB each (for 4.5 MiB in total), starting at the beginning
128 of the 2nd flash chip select (0x02000000 in the ARM7 address space).
129
130 * The smallest real FFS configuration called for by the table in dev.c in TI's
131 original Leonardo fw source is 3 sectors of 64 KiB each; the same table also
132 sports a 4 KiB x 4 configuration for RAM-based testing (emulation of FFS in
133 RAM without real flash).
134
135 * The largest FFS configuration that has been envisioned by the original
136 designers seems to be somewhere around 128 sectors.
137
138 Each flash sector used for TIFFS begins with this 6-byte signature:
139
140 46 66 73 23 10 02
141
142 The first 4 bytes are 'Ffs#' in ASCII, and the following two bytes are the
143 format version number of 0x0210 in little-endian byte order. The following two
144 bytes give a count of how many times that sector has been erased and rewritten
145 (FF FF in "fresh" or "virgin" FFS images), and the following byte indicates
146 that block's role and status in the FFS life cycle.
147
148 How it works
149 ============
150
151 Just like JFFS2 and other high-quality flash file systems, TIFFS is designed to
152 recover gracefully from any possible power failure or crash: one can yank the
153 battery from the GSM device (or induce a firmware crash) at the most mis-
154 opportune moment in the middle of an FFS write operation, and the FFS is
155 expected to recover on the next boot cycle. I won't be able to document here
156 all gory details of exactly how this goal is achieved, partly because I haven't
157 studied the code to the requisite level of depth myself yet, but all of the
158 responsible code lives under gsm-fw/services/ffs in this freecalypso-sw source
159 tree; feel free to study it.
160
161 In its "normal" or "clean" state (i.e., when not in the middle of a write
162 operation or recovery from an ungracefully interrupted one), a TIFFS instance
163 consists of the following 3 types of blocks:
164
165 * One block containing inode records, indicated by AB in its type/flags/status
166 byte in the block header;
167 * N-2 blocks (where N is the total number of flash sectors allocated for the
168 FFS) containing (or waiting to be filled with) data chunks - indicated by BD
169 in the type/flags/status byte;
170 * One "free" block, indicated by BF - destined to become a new AB or a new BD
171 at some point.
172
173 Each object written into the FFS (file, directory or symlink) consists of a
174 16-byte inode record written into the AB block and a data chunk written into
175 one of the BD blocks. The data chunk includes the name of the object, hence
176 one is required even for directories. Data chunks are contiguous, uncompressed,
177 and subject to an upper size limit of 2048 or 8192 bytes, depending on the FFS
178 configuration. Files larger than this limit are stored in a "segmented" form,
179 giving rise to a 4th inode or object type (after file, directory and symlink):
180 segment. Each segment of a segmented file consists of not only a data chunk,
181 but also an inode record for the segment, which gives the location of the data
182 chunk and ties the segment object into the overall FFS structure, making it
183 accessible.
184
185 Because aside from complete sector erasure, flash memory bits can only
186 transition from '1' to '0' but not the other way around, overwriting an existing
187 file with some new content (an operation which any reasonable file system must
188 implement in some way) cannot be done in place. Instead like most flash file
189 systems, TIFFS implements this common operation by writing the new version of
190 the file to a new location (previously blank flash) and then invalidating the
191 old version - and doing all that while keeping in mind the possibility of an
192 ungraceful crash or powerdown at any moment, and the requirement of recovering
193 gracefully from any such event.
194
195 Of course as an FFS receives more write activity, even if one keeps overwriting
196 some existing files with new content of the same size, without adding to the
197 visible total content size (think du(1) command), eventually all remaining blank
198 flash space will fill up. At that point (or at some earlier point, depending
199 on the FFS design and/or configuration) the FFS has to invoke a compaction or
200 reclamation or garbage collection procedure: any "mixed" blocks containing both
201 valid and stale data are transitioned into a "stale-only" state by having the
202 active data moved to a new block, and then the "all stale" blocks are subjected
203 to sector erasure, becoming new blank sectors. The logic responsible for these
204 operations once again needs to be resilient to the possibility of a crash or
205 powerdown occurring at the most mis-opportune moment, and it also needs to
206 implement flash wear leveling: there is a physical limit to how many times a
207 given flash sector can be erased and rewritten before it goes bad.
208
209 All of the above are common and well-known principles, successfully implemented
210 in well-known flash file systems such as JFFS2 in Linux. TIFFS is absolutely
211 no different in this regard; for the implementation details, read the source
212 code.
213
214 How this FFS comes into being
215 =============================
216
217 (This section is only relevant to you if you plan on physically producing your
218 own GSM phones or modems on your own factory production line, like this author
219 fancies doing in the not-too-distant future, or if you simply enjoy knowing
220 how it is done.)
221
222 To my knowledge, TI never used or produced a tool akin to mkfs.jffs2 in the
223 embedded Linux world, which would produce a TIFFS image complete with some
224 initial directory and file content "in vitro". Instead it appears that the FFS
225 instances found in shipped products such as Openmoko phones have been created
226 "in vivo" by TI's firmware running on the device itself during the "production
227 test" phase.
228
229 The process seems to go like this:
230
231 * When the printed circuit board is physically populated with components such
232 as the Calypso chip and the flash chip, the latter can be blank - if the
233 board design has the nIBOOT pin pulled low, enabling the Calypso boot ROM
234 (Openmoko and Pirelli both good on this one, but shame on Compal!), there is
235 no need to preprogram the flash chip with anything prior to populating it on
236 the board, and the device remains fully unbrickable at all times afterward.
237
238 * When the assembled board is powered up for the first time, with completely
239 blank flash, the Calypso boot ROM will sit there and patiently wait for a
240 code download on either of its two UARTs.
241
242 * Using TI's FLUID (Flash Loader Utility Independent of Device) or FreeCalypso's
243 fc-loadtool free replacement, the factory production station loads the main
244 firmware image into the flash. Note, it is just the firmware image at this
245 step, and the FFS sectors remain blank.
246
247 * The board is commanded to reboot (or power-cycled), and the firmware image
248 boots for the first time.
249
250 * TI's FFS implementation code in their standard firmware reacts to all blank
251 flash in the FFS sectors as follows: it performs what they call the preformat
252 operation, writing the TIFFS signature and a BF state byte into every FFS
253 sector, but the main "format" operation, which sets up the AB/BD block roles,
254 creates the root inode and makes the FFS ready to accept the creation of its
255 first directories and files, is not done automatically.
256
257 In order to perform the FFS format operation and then fill the new FFS with
258 whatever directories and files are deemed needed to be present in "fresh"
259 shipping products, the factory production station connects to the just-booted
260 firmware running on the target via the RVT/ETM protocol (see the RVTMUX
261 write-up), and sends "test mode" commands to this running firmware. These
262 "FFS test mode" (or TMFFS) commands include the format operation, an mkdir
263 operation to create directories, and a "file write" operation akin to doing
264 'cat > /dir/whatever/file', creating files in FFS and storing any desired data
265 in them.
266
267 The IMEI is assigned and written into FFS in this process, but it is not the
268 only data item that will be unique for each individual device made. Much more
269 important are the RF calibration values: I have yet to learn exactly what is
270 being (or needs to be) measured, how these measurements are performed (under
271 what conditions; what external test equipment is needed), and how these measured
272 and recorded RF calibration values affect GSM device operation, but this TI
273 presentation gives some clues:
274
275 ftp://ftp.ifctf.org/pub/GSM/Calypso/rf_calibration.pdf
276
277 All of these calibration values are stored in a bunch of files under the
278 /gsm/rf subtree, and these files seem to be "owned" by the L1 code. The latter
279 has RAM data structures which correspond to these files; upon normal boot the
280 initialization code looks in FFS, and if it finds any of the RF calibration
281 files, it reads each present file into the corresponding RAM data structure,
282 overwriting the compiled-in defaults. It appears (slightly uncertain because I
283 have not yet reintegrated the code in question into our own gsm-fw) that the RF
284 calibration files in FFS come into being as follows:
285
286 * The RF calibration code in L1 (i.e., part of the main GSM fw) performs the
287 measurements and stores results in its RAM data structures as commanded by
288 the production test station through the "test mode" interface;
289
290 * A final test mode command directs the above L1 code to write its RAM data
291 structures into FFS.
292
293 Once I actually learn this RF calibration process properly in connection with
294 building my own Calypso-based GSM "dumbphone", I'll be able to say exactly what
295 it would take to recreate these RF calibration values if they are lost. But
296 until then the only advice I can give is to make a backup copy of your modem
297 FFS with fc-loadtool, and to save it securely.
298
299 FreeCalypso support for TIFFS
300 =============================
301
302 Aside from implementing and using it in our own gsm-fw, FreeCalypso will offer
303 the following support for TIFFS:
304
305 1. A tiffs host utility is being written which will allow a user to list and
306 extract content from saved FFS images (read out of flash with fc-loadtool)
307 "in vitro". It will be a restructured and (hopefully) improved version of
308 the mpffs-* tools released back in the summer of SE52 (A.D. 2013); the latter
309 already perform the advertised function, but I seek to integrate some other
310 functionality which I developed in an ad hoc side project ("pirollback"),
311 and I'm taking the opportunity to make the MPFFS->TIFFS renaming.
312
313 (The mpffs-* tools mentioned above have been written based on reverse eng only,
314 before I found any source code for TI's FFS firmware implementation! Now that
315 we have the source, some terminological and other inevitable misunderstandings
316 can be corrected.)
317
318 2. A number of FC tools may be strung together into a kit for editing the FFS
319 content of a GSM device, e.g., for changing the IMEI. The following pieces
320 will be involved:
321
322 * What is destined to eventually become our totally free GSM fw (the gsm-fw
323 source subtree at the top of freecalypso-sw) does not contain any of the
324 actual GSM protocol stack (or even L1) functionality yet, but it already
325 contains both the FFS code and those components (ETM and TMFFS[12]) which
326 are needed for interfacing an external "test mode shell" to this FFS
327 implementation through the RVTMUX interface. And when our gsm-fw does gain
328 the actual GSM functionality, the ability to build a minimal FFS+ETM-only
329 configuration will still be retained.
330
331 * The minimal FFS+ETM subset of gsm-fw can be built into a ramImage (runs
332 entirely from RAM via fc-xram, no flashing), and run on a physical device
333 such as the GTA0x GSM modem via the fc-xram host utility;
334
335 * After loading the ramImage, fc-xram will immediately exec our rvinterf host
336 utility described in the RVTMUX write-up;
337
338 * Once the GSM device is running what is effectively an FFS editing agent out
339 of RAM, accessed via rvinterf over the serial channel, the user will be able
340 to run fc-tmsh (or perhaps the FFS operations will be implemented in some
341 other utility, we'll see), and that "test mode shell" will provide commands
342 for writing things to FFS exactly like one would do in the factory production
343 line environment for which TI taylored their tools.
344
345 The "in vivo" method of editing the FFS content of a GSM device described above
346 will probably sound very convoluted, and you may find yourself asking for a way
347 to do it "in vitro" instead: read the FFS out of flash with fc-loadtool, edit
348 that image "in vitro" with some utility on your PC, and then use fc-loadtool
349 again to program it back into your device. But consider that an "in vitro" FFS
350 modification would involve erasing and rewriting all sectors of your FFS,
351 whereas an "in vivo" modification of some small file like the IMEI would be
352 just a short flash write operation without any erasures at all, i.e., kinder
353 on the flash.
354
355 In any case, the "in vivo" method will definitely be available soon because all
356 of the components involved therein are also needed for other development uses
357 in the FreeCalypso project, whereas developing a fully-functional "in vitro"
358 alternative (one that can create an FFS image "de novo" from a tree of files
359 and directories a la mkfs.jffs2, or add new files to an existing TIFFS image
360 etc) would be a good amount of extra work which we otherwise don't need - hence
361 the latter is not very likely to be written any time soon.
362
363 However, if the "in vitro" modification you seek is something trivial like
364 changing the byte content of a file such as /pcm/IMEI or /gsm/com/rfcap without
365 changing its length, you will be able to use the "in vitro, read-only" tiffs
366 host utility to find the exact byte location of the file data within the TIFFS
367 image, and use your favourite hex editor to whack whatever new byte content you
368 like at that offset.