# HG changeset patch # User Mychaela Falconia # Date 1672620619 0 # Node ID 3152e23399a26161688be126bbc3aa398ed64f79 # Parent 74d284add54d6b3fc9e23431c46ec8590c294f64 document SE K2x0 FFS quirks and our support for them diff -r 74d284add54d -r 3152e23399a2 doc/SE-K2x0-FFS --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/SE-K2x0-FFS Mon Jan 02 00:50:19 2023 +0000 @@ -0,0 +1,184 @@ +Flash file system in Sony Ericsson K200/220 phones +================================================== + +SE K200/220 phones are based on our familiar Calypso chipset, their firmware is +based on the standard chipset reference fw from TI, and they use TIFFS as their +flexible data storage mechanism. Their TIFFS instance is located in the first +3328 KiB of the 2nd flash bank; the geometry is 256x13 or 64x52 depending on +which flash chip was used in each given phone. (Some specimen have Spansion +S71PL129NB0 chips with 256 KiB flash sectors, others feature Samsung K5L2931CAM +chips with 64 KiB flash sectors.) + +Calibration data and self-regenerating ability +============================================== + +SE's FFS life cycle design on these phones is interesting in that vital per-unit +calibration data (RF, AFC and MADC calibrations) are stored in two places: +(1) in the FFS in TI's standard format and locations, and (2) in a dedicated +flash sector outside of FFS that appears to be written once at the factory and +never touched afterward. + +SE's firmware design allows the FFS to be fully erased without breaking the +phone: whenever the fw boots in a state where the FFS area of the flash is fully +erased (you can of course erase those FFS sectors with fc-loadtool, but it +appears that some official tools also provide the same operation) but the +separate factory calibration data sector is valid, the firmware formats and +initializes a new FFS, and furthermore copies all essential calibration records +from SE's proprietary sector structure into their standard TIFFS locations! + +But it gets even more interesting - when we examine FFS images read out of +various SE K2x0 specimen, we see two different occurrences: + +1) In some specimen the FFS content is exactly the same as what we would get by + erasing the FFS with fc-loadtool and letting the firmware regenerate a fresh + one on its next boot. We can thus reasonably assume that the FFS in these + specimen is indeed the product of such intervention at some point in the + phone's history. + +2) In other specimen we see some additional files beyond those that exist with + the life cycle of the previous paragraph. One readily noticeable addition + is the presence of /gsm/rf/tx/ramps.* files, which are absent in the self- + regenerated FFS where the phone's fw copied calibration records from SE's + dedicated calibration data sector. How are these Tx ramp tables different + from all other RF calibration records? Answer: in the present SE K2x0 + design, and also in all other Calypso phone and modem designs known to this + Mother, Tx ramps are calibrated per design and not per unit. Because the + correct Tx ramp tables are compiled into the firmware, having them in FFS is + redundant. + +Based on the above evidence, I (Mother Mychaela) hereby make the following +reconstruction of how these two kinds of FFS found in the wild likely came into +being: + +* In the case of Compal (Mot C1xx and SE J100) and Pirelli DP-L10 we know by + simple deductive reasoning that those firmwares must have had their l1_cust.c + code modified to read calibration records from their respective proprietary + flash data structures instead of TI's original FFS files. However, in the + case of SE K2x0 it is plausible that their l1_cust.c did NOT have to be + likewise modified - they could plausibly still have TI's original l1_cust.c + code that reads RF tables from FFS on initialization and writes them back + into FFS upon those special MISC_ENABLE commands that are issued at the end + of TI's original calibration procedures. + +* If my hypothesis above is correct, then it follows that when SE or their ODM + did their factory production line calibration, the results were initially + written into FFS and not into SE's separate factory calibration data sector. + If they are using unmodified l1_cust.c code from TI, then the presence of + redundant /gsm/rf/tx/ramps.* files is explained - TI's standard calibration + procedure results in these files being written into FFS because the canonical + l1_cust.c code classifies them as "Tx calibration" rather than "Tx config" - + see our RF_tables article. + +* In a separate step after all standard calibration procedures, they must have + copied all important calibration records into their non-standard flash sector + at 0x01FD0000, specifically to produce a more durable copy that can persist + through full erasure of FFS. Tx ramp tables were not included because they + are relatively large and redundant, not actually calibrated per unit. + +* Those SE K2x0 specimen whose FFS contains /gsm/rf/tx/ramps.* files and some + other files that aren't found in auto-regenerated FFS probably exhibit the + original FFS from the factory production line that was never subjected to + being blown away and regenerated! + +SE's non-standard extension to TIFFS +==================================== + +There is one aspect of SE K2x0 FFS that creates a pain point for our tools: +they made their own non-standard extension to TIFFS in terms of extended UCS-2 +filenames, and given our natural desire to be able to use our tiffs tool (TIFFS +IVA) to examine FFS images from these SE phones, we had to extend our tool so +that it parses SE's extension instead of throwing up errors like it did before. + +As explained in our TIFFS-Overview article, classic TIFFS requires every +elementary filename to consist of printable ASCII characters from the sensibly +narrow set of [A-Za-z0-9_.,+%$#-], further limited to a maximum of 20 +characters. However, SE K2x0 FFS images often contain some files that violate +this constraint; deeper examination reveals that SE (or their ODM) devised a +feature to allow filenames composed of UCS-2 characters, implemented as follows: + +* The first "character" of the filename written into TIFFS is \x02, i.e., NOT a + printable ASCII character; + +* The next "character" in the TIFFS object name is another binary byte encoding + the length of the UCS-2 string: for example, if the "high-level" name consists + of 6 UCS-2 characters, this second TIFFS object name "character" is \x06; + +* These two non-printable-ASCII chars are followed by a long string of [0-9A-F] + ASCII characters, encoding the "high-level" UCS-2 name in hex; + +* The whole arrangement ends with a terminating NUL, allowing the rest of TIFFS + to work unchanged. + +Seen from TIFFS perspective, these extended filenames created by SE K2x0 fw +violate traditional canon in two ways: + +1) The overall length of the mostly-hex-encoded filename string usually exceeds + the traditional 20 character limit; + +2) The set of allowed characters is grossly violated: not an innocuous extension + of allowing some additional "sane" characters, but non-printable-ASCII binary + chars that are barely compatible with C string functions by virtue of not + containing zero bytes. + +Please see the new section titled "Support for SE K2x0 extended filenames" in +our TIFFS-IVA-usage article for the explanation of how we handle these extended +filenames in our TIFFS IVA tool. + +fc-fsio considerations +====================== + +In addition to "in vitro" analysis with our tiffs tool, SE K2x0 FFS can also be +accessed "in vivo" with fc-fsio in the following two scenarios: + +1) SE K2x0 firmware does not have RVTMUX enabled by default, but they have + non-volatile flags in their FFS (which can be set via a hidden menu entered + via secret MMI code #*87223564#) that enable RVTMUX on the MODEM UART that + is brought out, and their RVTMUX includes ETM and TMFFS2 protocols. + +2) We can run TI's original FFS code against SE K2x0 FFS bodies by adding the + necessary configuration to our fc-xram based FFS editor tool described in + our TIFFS-Overview article. + +The expected behaviour in scenario 2 can be predicted statically by studying +the code, whereas scenario 1 calls for experimentation. Experiments with +RVTMUX-enabled SE K2x0 original fw reveal that it does behave (at least as far +as visible behaviour over TMFFS2) exactly like TI's original FFS code in terms +of readdir output when listing directories with extended filenames in them: as +long as they fit into TMFFS2 string buffer, these extended filenames are +returned just as one would naively expect. + +In light of these observations, fc-fsio has also been patched up to behave +gracefully when faced with such previously unexpected readdir results from the +target. Specifically: + +* The short form of 'ls' command will print whatever the target firmware + returns, including extended filenames; any non-printable-ASCII characters are + escaped in \x hex form. + +* 'ls -l' or 'll' will throw up an error if one attempts to list a directory + that contains one or more extended filenames, caught either at the level of + the readdir result not fitting into the buffer sized for standard TIFFS + filenames or (for SE K2x0 extended filenames of 4 or fewer UCS-2 characters) + at the level of this readdir result containing non-printable-ASCII chars. + +Finally, the following limitations need to be kept in mind: + +* There is no way to address an FFS object with an extended filename by entering + a pathname in fc-fsio commands; + +* File systems or subtrees containing extended filenames cannot be read out with + fc-fsio cpout command: it will fail just like ls -l; + +* rm-subtree and cleandir commands will likewise fail and stop whenever they + encounter a filename longer than 20 characters (invalid for the purpose of + pathname generation based on fixed-length buffers) anywhere in the subtree to + be deleted. + +The last limitation (inability to delete "bad" files) may seem like a serious +flaw in the design of fc-fsio at first, but please realize that the primary +mission of FreeCalypso is to work with our own hardware and firmware, not alien +- and the original FFS code from TI, which is what we use in FreeCalypso, will +never allow file system objects with such bad filenames to be created in the +first place. Therefore, the limitation of fc-fsio being unable to manipulate +certain files to the extent of not even being able to delete them is specific +to the peculiar scenario of operating on *alien* FFS from Sony Ericsson. diff -r 74d284add54d -r 3152e23399a2 doc/TIFFS-IVA-usage --- a/doc/TIFFS-IVA-usage Sat Dec 31 22:55:23 2022 +0000 +++ b/doc/TIFFS-IVA-usage Mon Jan 02 00:50:19 2023 +0000 @@ -190,3 +190,27 @@ Just like regular cat, but takes an inode number instead of a pathname. Can be used to cat the old content of deleted or overwritten files. + +Support for SE K2x0 extended filenames +====================================== + +As explained in the SE-K2x0-FFS article, Sony Ericsson K200/220 phones use +TIFFS as their flexible data storage mechanism, but they also made their own +non-standard extension to TIFFS in terms of extended UCS-2 filenames. + +In order to allow listing full FFS trees from SE K2x0 phones with tiffs ls and +possibly also extracting them with tiffs xtr, we have made the following hackish +extension to our TIFFS IVA: whenever either of these two commands encounters an +"extended" filename (one which fails standard TIFFS filename validity checks but +has a terminating NUL that allows the name and data portions of the chunk to be +separated) in the visible tree walk, our tool replaces it (for the purpose of +listed pathname generation or extraction into host file system) with a +synthesized pseudonym of the form "!bogo%04x", where %04x is the FFS inode +number in hex. In the case of tiffs ls, the actual extended filename is then +printed on a separate line (immediately following the synthesized listing), with +non-printable-ASCII characters escaped in \x hex form. We've also extended the +individual-inode form of lsino command to list these extended filenames when +encountering such, and we've extended catino (which needs to skip past the name +to get to the beginning of the data portion of the head chunk) to not reject +file head chunks with extended filenames in them as long as the needed +separating NUL is found.