# HG changeset patch # User Mychaela Falconia # Date 1539028370 0 # Node ID fcd1cf531017a7a5d9883b7b2e2576b075649678 TCS211-fw-arch masterpiece written diff -r 000000000000 -r fcd1cf531017 Handset-UI-fw --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Handset-UI-fw Mon Oct 08 19:52:50 2018 +0000 @@ -0,0 +1,287 @@ +The present article is a companion to the TCS211 firmware architecture document +(TCS211-fw-arch). Those who are interested in the FreeCalypso phone handset +goal (which is currently a very distant goal) should read the TCS211-fw-arch +document first and then this article, whereas those who are more interested in +the rock solid FreeCalypso modem (as opposed to handset) solution which is +already available today and would like to understand our modem fw better only +need to read the TCS211-fw-arch document and can safely skip this handset UI +addendum. + +TI's offerings for handset UI +============================= + +Unlike their rock solid, full commercial product offering for AT-command- +controlled modems, TI never produced a reference phone handset firmware +implementation that could be used as-is with minimal or no changes: instead +they provided a very rough proof-of-concept implementation which is nowhere +close to usable, and left it up to their customers (handset manufacturers) to +whip it into even a minimally decent shape. Furthermore, they had several +different approaches over the years to what the GSM industry calls by the +sexist term "MMI": + +* They once had something called SMI, for "simple MMI" or "slim MMI". We have + what appears to be the complete source for this SMI component as part of the + MV100-0.1.rar and 94140216gsm.rar finds, but both of those finds are just + broken fragments, not a complete firmware for any target. It might be + possible to integrate this unknown-origin SMI source into Magnetite and get + it to work with the TCS2 version of ACI, but no such feat has been attempted + yet - it would be a mere curiosity, not something for practical use. + +* SMI later gave way to a successor called BMI for "basic MMI", which is much + bigger in terms of code size and complexity and consists of two layers: there + is a layer called MFW (mobile framework) that sits on top of ACI, and then + BMI proper sits on top of MFW. This incarnation of TI's demo/prototype phone + UI is the one which they officially supported in the TCS211 program, and our + copy of TCS211 from OM miraculously has these BMI+MFW sources included, even + though OM obviously didn't use this code and could not even compile it without + doing some surgery on the build system. Our other TCS3.2/LoCosto source also + has BMI and MFW matching that newer version of G23M ACI and PS components, + and we use this new TCS3 version of BMI+MFW in our TCS2/TCS3 hybrid config. + +* It appears that TI once had or were trying to develop some kind of Riviera- + based "MMI" code as an alternative to the Condat-based SMI and BMI. SMI and + BMI+MFW execute in the same "MMI" GPF task as ACI and communicate with it + through direct function calls; in contrast, the alternative Riviera MMI would + run somewhere in Riviera land and communicate with ACI through ATP and some + kind of ACI adapter. We never got any of this code, and it is not clear if + it was ever a reality beyond just the idea. + +In any case, it is clear from the code that TI's SSA group in France who +developed the drivers for various Calypso chipset peripherals including LCDs, +keypads and ringing buzzers on their development boards and the Condat UK group +who did SMI and BMI had very different visions and ideas. Some examples: + +* The Calypso DSP has a built-in mechanism for playing quite pleasant-sounding + ringtone melodies through a loudspeaker driven by the chipset's ABB, and the + SSA group developed their RiViera Audio Service front-end to these L1+DSP + audio services, but Condat's code makes absolutely minimal use of this RV + Audio Service, just enough to be compatible with it, and does not use any of + the melody functions, neither E1 nor E2. + +* In the original TCS211 architecture before LoCosto changes, the driver for + the physical LCD was/is R2D in the Riviera/SSA land. R2D provides a very rich + set of text and graphics drawing primitives, but Condat's display layer does + not use any of them: instead they obtain the raw framebuffer address from R2D + and do all drawing operations themselves. + +* The TCS211 code we got had a jaw-dropping bug in the code path for ringing + the piezoelectric buzzer: due to a miscommunication between the French folks + in charge of the actual buzzer driver in chipsetsw and the German or UK folks + in charge of Condat's audio layer, the buzzer always rang at some 99 Hz (its + lowest possible frequency, horrible on ears) no matter what tone frequency + was intended. Given that our copy of TCS211 dates from 2007 and considering + how old the buzzer function must be, it hurts to think for how many years + that bug sat there without anyone wondering why ringing sounds so horrible. + +In terms of the code architecture, none of Condat's UI code ever calls any of +the actual drivers in the SSA realm directly: instead everything goes through +the Condat drivers layer in condat/com/src/driver, and that layer provides a +very poor adaptation as highlighted above. + +LCD support +=========== + +TI's demo/prototype UI code never supported a wide variety of different display +sizes or keypad layouts - instead they only supported whatever existed on their +in-house development boards at each given point in their history. TI's C-Sample +and earlier development boards had 84x48 pixel B&W displays, whereas from +D-Sample onward they made the big jump to a 176x220 pixel color LCD. Both +versions of the UI we got (TCS211 targeting D-Sample and TCS3.2 targeting +I-Sample, TI's LoCosto board) were developed on 176x220 pixel color LCD +platforms, and that is the only display size which they intended to support. +There also exists a remnant of support for their earlier 84x48 pixel C-Sample +LCD, which we resurrected in FreeCalypso to see it run on Mot C139 hardware, +but that support was broken: it would not even compile without our fixes. In +our current FC Magnetite firmware we can build this C-Sample UI version for the +C139 target and it works in the demo / proof-of-concept sense, but it is likely +to be even more broken than the official 176x220 pixel version. + +Anyone interested in adding support for a different LCD will need to start with +the R2D driver under src/cs/drivers/drv_app/r2d. The principal LCD type +selection is done in r2d_config.h (C-Sample and D-Sample are the only options +supported by the version we got with TCS211), and this selection affects all of +R2D and all of its clients. Our change to this r2d_config.h LCD type selection +logic consists of selecting C-Sample instead of D-Sample when the build target +is C139. A secondary selection is made in r2d_inits.c and r2d_refresh.c where +the code snippets for the actual hardware initialization and output are +included: the way we currently support the C139 hw target is a very thorough +emulation of the C-Sample including its vertical B&W framebuffer format, all of +the code in R2D and in Condat's display driver sees a real C-Sample LCD, and +only the hardware-poking code is substituted. + +The direct implication of our C-Sample emulation approach for the C139 LCD is +that the full 96x64 pixel size of Motorola's LCD becomes completely +inaccessible, and all software sees a 84x48 pixel subset. The same happens +with color: because TI's C-Sample LCD was B&W, the color capabilities of the +real C139 LCD become inaccessible. Anyone who wishes to fix this shortcoming +would need to implement a new bona fide LCD type in R2D, and then adapt +Condat's display driver accordingly. + +Condat's display driver (condat/com/src/driver/display.c) is very messy and +very difficult to understand. The only change we have made to it in FreeCalypso +was fixing the support for the C-Sample LCD which was bitrotten: the bitrot and +our fix for it is not specific to the C139, it would affect a real C-Sample +board just as well. Understanding this code well enough to extend it to other +LCD geometries and framebuffer formats would be a much greater challenge. + +Above Condat's display driver lies the actual UI implementation in BMI and MFW. +This UI code supports only 3 possible configurations: + +* Both COLOURDISPLAY and LSCREEN defined: the display is 176x220 color; + +* LSCREEN is defined but not COLOURDISPLAY: the display is 176x220 B&W - TI + used this config when building "GSM Lite" fw for the D-Sample; + +* Neither LSCREEN nor COLOURDISPLAY defined: the old 84x48 pixel B&W UI from + the days of C-Sample and earlier. + +Note the lack of support for small color displays like the one on the C139. + +Text fonts +---------- + +The SSA group's R2D driver provides text display functions and contains its own +fonts, but Condat's stuff does not use those display functions or fonts - +instead BMI and MFW (and presumably SMI too) use a different font/text +implementation contained in the Condat drivers layer. + +Keypad support +============== + +The hardware keypad driver is KPD in Riviera/SSA land, residing in +chipsetsw/drivers/drv_app/kpd in TCS211 or in src/cs/drivers/drv_app/kpd in +Magnetite and Selenite. Unlike the display driver R2D, KPD is always included +even in modem firmware builds - but if there is no keypad connected to the +Calypso, it does nothing. TI's firmware architecture and UI code support only +traditional numeric keypads - there is no provision for supporting a full QWERTY +keyboard. However, if the keypad layout on a given phone handset is close +enough to what TI had on their development boards, modifying KPD for the +specific wiring is very easy: we have already added proper support for Mot C1xx +and Pirelli DP-L10 keypads. + +Battery monitoring and charging +=============================== + +TI had two incarnations of their battery charging and monitoring driver: first +PWR, then LCC. Both were implemented in Riviera/SSA land. LCC was not a good +fit for our targets of interest (Mot C1xx, Pirelli DP-L10 and our desired +FreeCalypso Libre Dumbphone hardware) while PWR had other problems, so we +retired both of them and wrote our own replacement called FCHG. + +There is a quirk, however: there is no connection in the TCS211 code as +delivered by TI between their demo/prototype UI (or rather between Condat's +power driver stub) and either of their battery driver implementations. At the +time of TCS211-2007 the original PWR driver had already been retired and only +LCC was supported, but even that LCC driver had no connection to the UI: one +could remove it from the fw build configuration and the UI code would still +compile and link just fine, which would not be possible if there were any calls +to LCC's API functions. The practical effect of this quirk is that there is no +need to resurrect PWR or LCC in order to run TI's UI code in its pristine +original form - using our own FCHG or no battery management driver at all is +just as fine. + +The proper way to get proper support for Mot C139 +================================================= + +I (Mother Mychaela) feel very strongly that the best way to produce practically +usable end user FreeCalypso firmware for the C139 target would be to do it +together with our own FreeCalypso Libre Dumbphone development, as opposed to +trying to support the C139 to the exclusion of our own FreeCalypso hardware. +Specifically, I propose the following order of steps: + +1) First build our own FreeCalypso UI development board, a derivative of the + FCDEV3B adding a 176x220 pixel color LCD and other miscellany to serve as a + replacement for TI's D-Sample. + +2) Retire the C-Sample UI configuration and our currently implemented C-Sample + emulation hack on the C139, and start running TI's UI code the way TI's own + developers ran it on the D-Sample. + +3) Change the "small screen" UI layout from 84x48 to 96x64 pixels (from 6 rows + of 14 characters to 8 rows of 16 characters with TI's existing font), and + fix the bugs related to displaying this "small screen" (!LSCREEN) UI on a + physically larger LCD - we would like to display it on our 176x220 pixel UI + development board. + +4) Extend the UI code to allow the possibility of COLOURDISPLAY && !LSCREEN, + i.e., a small (96x64 pixels) color display like on the C139. Have this + 96x64 pixel color UI displayed on our 176x220 pixel UI development board. + +5) Work on getting the actual UI into shape, keeping the two configurations of + 176x220 pixel color (future FreeCalypso Libre Dumbphone) and 96x64 pixel + color (Mot C139) as actively supported ones. Do all development on our + 176x220 pixel UI development board. + +6) Once the UI-enabled firmware works decently on our development board in both + 176x220 and 96x64 configurations, add native C139 LCD support (not C-Sample + emulation) to R2D, adapt Condat's display driver as necessary if we are still + using it (if we don't find a way to get rid of it by this point), and run + our 96x64 pixel color UI config on real C139 hardware. At this point we + should have practically usable end user FreeCalypso fw on the C139. + +7) While the less demanding and more casual phone users will be happy with the + feeble C139 if it runs our FreeCalypso fw, those of us who desire the + Ultimate Awesome Libre Dumbphone will be able to take our 176x220 pixel UI + development board and start turning it into an actual FreeCalypso Libre + Dumbphone handset. This will be the point when we can move the ringtone + output from the piezo buzzer to the loudspeaker (Melody E1) and start making + other changes for better-than-C139 hardware. + +Of course the biggest difficulty with the above plan is that it requires +designing and building a new piece of hardware as its very first critical step. +My personal plan is to kill two birds with one stone: design the board in such +a way that it can be used both as a development board for the above plan and as +a close-to-final prototype for my desired FC Libre Dumbphone handset - although +not completely final, as this board would absolutely need to be usable in its +bare form on a lab bench without plastics, which calls for a somewhat different +design than a final handset product. + +Anyone who disagrees with my approach, anyone who is against designing and +building new custom hardware at high cost and who is instead interested first +and foremost in pre-existing hardware targets like Mot C139 is more than welcome +to delve into the code themselves and try their own hand at fixing the code to +make it practically usable on the C139. However, I have to warn you: if you +spend a significant amount of time working with TI's code and develop an +affection for it, it is quite possible that you will start to feel the way I do +in terms of desiring a D-Sample-like platform for development. + +Why Mot C139? +============= + +Out of the known repertoire of pre-existing Calypso-based phones whose existence +has been discovered by OsmocomBB and for which we already have some foundations +of support in FreeCalypso, Mot C139 is the most suitable one for the purpose of +turning it into a Libre Dumbphone by way of aftermarket firmware. Here are the +problems with the other ones: + +* Pirelli DP-L10 is my absolute favourite with an end user hat on, but the + unwanted extra chips for the unwanted-for-us WLAN and camera functionality + are a killer. There are no schematics for the phone and no documentation for + any of these chips, and we don't know how to power them down. In a fully + quiescent state with all Calypso sleep modes enabled and with both LCD and + keypad backlights off, the board still draws about 87 mA from the battery, + which will kill a fully charged battery in about 10 hours of complete idle. + Furthermore, it is not possible to turn on the loudspeaker without going + through the W56940 ringtone chip, and the reset line for this chip comes from + a GPIO on the completely undocumented camera chip - thus without a way to + control this reset line we may not be able to program the W56940, and without + that programming we may not be able to turn on the loudspeaker, ruining all + usefulness of this phone. + +* Mot C11x/12x family is good in terms of not having any undocumented hardware, + but the tiny RAM capacity is a bummer. These lowest-end phones have only + 256 KiB of IRAM (Calypso internal RAM) and 256 KiB of XRAM (board-level RAM), + whereas the next-step-up C139/140 has 512 KiB of XRAM. It is a significant + difference for us: while we fit into C139's 512 KiB with no sweat, it would + require some very unpleasant and unrewarding work to trim the fat and + reshuffle our memory usage to fit into the 256+256 arrangement on C11x. + Furthermore, most C11x/12x phones have only 2 MiB of flash, which would again + require major contortions to fit into, whereas all C139/140 phones have 4 MiB. + +* Mot C155/156 seem nice, but there is a problem: these phones ring with a + loudspeaker driven by a ringtone generator chip instead of a piezo buzzer + driven by the Calypso, and their ringtone generator chip is Sunplus SPMA100B. + No documentation could be found for that chip, and if we can't program it, we + won't be able to make the phone ring or operate its loudspeaker in any other + way. Furthermore, the LCD controller in these phones is much less obvious + than the one in the C139/140. diff -r 000000000000 -r fcd1cf531017 TCS211-fw-arch --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/TCS211-fw-arch Mon Oct 08 19:52:50 2018 +0000 @@ -0,0 +1,1867 @@ +This document describes the architecture of TI's TCS211 firmware and that of +our FreeCalypso Magnetite and Selenite firmwares which are based on it. + +What is TCS211, and why we use it as our reference +================================================== + +TI were in the business of making GSM baseband chipsets for about a decade +from the late 1990s up until 2009, and over that time span both their silicon +and their firmware architecture had evolved in many different ways. All of our +work in the FreeCalypso family of projects is based on one fairly arbitrary +snapshot, a rather arbitrarily picked single point in that long evolutionary +line: we use the Calypso chipset as opposed to both the ones before and the +ones after, and we use TI's TCS211 firmware from 2007 as our golden reference, +as opposed to other equally valid ways of architecturing the fw that came +before and after our arbitrarily picked snapshot. + +Q: Why do we use the Calypso chipset as opposed to LoCosto or E-Costo or +whatever was TI's very last offering before they got out of that business? + +A: Because that's what Openmoko used: their Neo FreeRunner aka GTA02 smartphones +were our primary hardware target for many years before we gathered the money +and the courage to build our own board-level hardware starting from just chips +bought on the Chinese surplus market. + +Q: Why do we use TI's TCS211 firmware from 2007 and its architecture as our +golden reference, as opposed to any of the other infinitely many equally valid +ways of architecturing a working firmware implementation for the same Calypso +chipset? + +A: Because it works flawlessly, and is extremely stable as a commercial product. +The firmware which Openmoko got from TI had only a tiny difference from TI's +internal TCS211 mainline (TSPACT signal definitions in tpudrv12.h which are +different between the quadband RFFE on TI's internal reference hw and the +triband one in FIC's commercial implementation), and with only a few additional +changes related to our use of a newer flash chip that wasn't supported back in +TI's and Openmoko's days, this golden reference fw can run equally well on our +own FCDEV3B. + +Relation between TCS211 and FreeCalypso +======================================= + +The only "pure" TCS211 firmware we got is the one that has been salvaged from +the ruins of Openmoko. To the best of our knowledge, it is the world's only +surviving copy of any version of TCS211 - it is entirely possible that even TI +may not have it any more in any of their archives, given the length of time that +has passed and the total lack of interest in this "ancient junk". In its pure +form, this world's sole surviving copy of TI's TCS211 fw is laden with blobs +(many components exist only as binary object libraries with no corresponding +source), and it features a build system that is very thoroughly Windows-based. +And to top it off, that configuration and build system has many critical +components which also exist only as compiled binaries (Windows executables or +Java bytecode) with no corresponding source. + +We started by replacing the original configuration and build system of TCS211 +with our own one that is Unix-based rather than Windows-based, and implemented +in Bourne shell with a few C helpers instead of XML, Java and Perl. The result +was named FreeCalypso Magnetite. At first we changed only the configuration +and build system, but kept all of the original TCS211 code, including all of +the binary-only components. Then we deblobbed it gradually, replacing binary- +only components with source, one component at a time. Where did we get the +source for the pieces that came as binary objects with no corresponding source? +The answer is different for different components: + +* For GSM Layer 1 (a very critical and highly chipset-dependent component), we + did a painstaking reconstruction which you can see in the tcs211-l1-reconst + repository. That world's last surviving copy of TCS211 which we got only had + *.c files censored out, while all of the original *.h files were preserved - + and thanks to the preserved configuration and build system, we also got all + of the original compilation lines including compiler options, -D definitions + and -I include paths. For most of the missing *.c files we got a "wrong" + version from the TCS3/LoCosto source. The reconstruction proceeded by taking + these "wrong version" *.c files, putting them one module (one *.c file) at a + time into the TCS211 build environment, and massaging each individual *.c + file until it compiled into a perfect match to the original binary object. + Thus we have reconstructed a full C source for the L1 component which for all + practical purposes can be treated as if it were the lost original source. + +* For some small pieces like the tpudrv12 RF driver and the OSL and OSX + components of GPF it was more of a translation from disassembly to C: the C + code we use is of our own writing, but it faithfully matches the logic + implemented by the original blobs as recovered through disassembly. + +* The G23M protocol stack is a very large and complex component, and our copy + of TCS211 (the world's only surviving copy to the best of our knowledge) has + it in binary-only form. Trying to source-reconstruct it precisely like we + did with L1 would have been infeasible, hence we took a different approach: + we put together a TCS2/TCS3 hybrid in which we made a wholesale replacement + of all G23M components: we adopted the new version of G23M wholesale without + trying to recreate the old version. + +* Both TCS211 and TI's newer TCS3.2 fw for the LoCosto chipset are based on + Nucleus PLUS RTOS (different versions), and both firmwares have their Nucleus + only as binary object libraries, no source. However, we got another version + of Nucleus from about the same time frame (slightly newer than the one TI used + in TCS211, but slightly older than the one in TCS3.2) from a non-TI source + (it was posted on a Russian web forum by Comrade XVilka), and in FreeCalypso + Selenite we use this new Nucleus as a replacement for TCS211 original version + in the same manner as how we had earlier made a wholesale replacement of the + G23M protocol stack. + +With two major components (Nucleus and the G23M PS) replaced with non-TCS211 +versions, our Magnetite hybrid and Selenite firmwares are no longer TCS211, but +they still faithfully follow the _architecture_ of TCS211: in each case when we +replaced the code, we made the new code version fit perfectly into the original +architecture without any disruptive changes. Thus anyone who desires to +understand our current FreeCalypso firmwares (Magnetite and Selenite) needs to +first understand the original TCS211 architecture, as it is essentially +unchanged. + +Why not use the LoCosto chipset and its TCS3.2 firmware? +======================================================== + +We went the Calypso route and not the LoCosto route because of the circumstances +that surrounded the beginning of our family of projects. We did not get all of +the tools needed for working with LoCosto chips and TI's TCS3.2 fw (CSST and +SBuild) until the spring of 2015, and by that time we had invested too much into +the Calypso to throw it all away and restart anew in the uncharted waters of +LoCosto. Another factor is that the software for talking to LoCosto's ROM +bootloader (CSST) exists only as Windows binaries sans source, and it would +require some effort to reverse-engineer the protocol and implement a free and +Unix-based alternative - whereas for the Calypso this work was already done by +OsmocomBB folks before we entered the scene. Finally, in the case of the +Calypso we have read out the actual content of the ROMs (both the ARM boot ROM +and the DSP ROM) and the ARM boot ROM code has been disassembled and thoroughly +understood - whereas in the case of LoCosto it is not certain if we can even +read out the ROM content, as it is said to be protected against reading. + +If someone else desires to play with LoCosto, either by hacking a Peek device +or by building an I-Sample board from the available PADS PCB file, go for it! +But the FreeCalypso core team is sticking with the Calypso chipset for now, and +our actively maintained Magnetite and Selenite firmwares follow the architecture +of TCS211, not that of TCS3.2. + +Relation between the ARM and DSP cores in the Calypso +===================================================== + +The Calypso digital baseband processor chip has two processor cores in it: an +ARM7TDMI core that runs the main firmware and a C54x DSP core that performs the +more burdensome signal processing tasks. The DSP is subservient to the ARM: +only the ARM comes out of reset and starts executing code upon power-up, while +the DSP is held in reset (does not run) until and unless the ARM firmware starts +it running. + +The ARM core executes code from outside of the Calypso chip itself: in normal +operation (outside of development) there is a flash memory chip connected to +Calypso's external memory bus, and the Calypso's ARM core executes firmware +stored in this flash. There is an optional (enabled or disabled by a hardware +pin) ARM boot ROM inside the Calypso chip; when this boot ROM is enabled by +nIBOOT pin strapping on the board (like it is on Openmoko and FreeCalypso +hardware), the ARM core executes code from this boot ROM first upon power-up or +reset before jumping to external flash. The tiny piece of code that is hard- +cast in this mask ROM acts as an unbricking aid: it gives a certain time window +during which the boot process can be interrupted and diverted if certain magic +characters are sent into either of Calypso's two UARTs by an external +development host, and if nothing is received on either UART during that time +window (as would be the case in normal usage of a Calypso phone or modem), the +boot ROM transfers control to the firmware image in the external flash. The +end result is that the ARM core always runs code from outside of the Calypso +chip itself, either the firmware image in the flash or whatever code is fed by +an external development host to the boot ROM serially over a UART. + +There is also an internal RAM inside the Calypso from which the ARM can execute +code (512 KiB on the full Calypso version or 256 KiB on Calypso Lite silicon +used in some historical low-end phones); the primary purpose of this internal +RAM is to allow chosen sections of code to execute faster without the +performance penalty of the external memory bus, but it is volatile RAM, not ROM +or flash, hence it doesn't have any code in it until and unless loaded by the +firmware copying code from flash or via the serial boot protocol. + +In contrast, the DSP is very different. The DSP core can never execute any +code from outside the chip, and has no access to the Calypso chip's external +memory bus at all. Instead the only two memories accessible to the DSP are a +mask ROM and a fast internal RAM. The DSP's dedicated mask ROM is 128 Kwords; +the DSP's RAM is 28 Kwords, out of which 8 Kwords constitute the so-called API +RAM which is accessible to both ARM and DSP cores. (The C54x DSP addresses +memory by words instead of bytes, hence the memory sizes are given in Kwords +instead of KiB.) + +The main bulk of the DSP's operating program is already hard-cast in the silicon +in the 128 Kword mask ROM. The DSP ROM code is structured in such a way that +any part of it can be overridden by downloadable patch codes which get loaded +somewhere in the DSP's 28 Kword RAM, but because the RAM is significantly +smaller than the ROM, downloadable DSP code cannot replace the entirety of the +ROM code - instead the code needs to be patched very selectively only where +necessary to fix a bug that was discovered after the silicon was made or to +extend the DSP functionality with a new feature. + +The DSP ROM code in the Calypso silicon we are using has been successfully read +out, but it is only the executable binary code and data - we never found a copy +of the source for this DSP ROM code. And even if we had this source, we would +not be able to casually modify and recompile it without spending millions of +dollars to fab a new chip revision with a modified mask ROM. Having this source +would allow us to develop our own DSP patch codes and to understand and maintain +the existing ones, hence we need to make an effort to convince TI to release +the source for the DSP ROM if they have it in their archives, but if no +surviving copy of this source exists anywhere in the world, the fallback plan +would be to reverse-engineer the DSP ROM code by disassembly. The latter plan +has not been pursued yet because of the very high labor cost it would involve. + +It is possible to run the Calypso DSP without any patches, i.e., have it run +only the code that is already in the mask ROM. Our competitor OsmocomBB +operates in this manner, and we have also built and run modified versions of +our TCS211-based FreeCalypso firmware with DSP patch loading disabled as an +experiment. However, all ARM-side firmwares that have been officially released +by TI for production use including our TCS211-20070608 golden reference do apply +downloadable patches to the DSP, and are designed to run with this patched DSP; +running them with DSP patching disabled results in unstable operation. + +DSP patch codes that are included in ARM-side Calypso firmwares take the form +of const char arrays initialized with hex bytes; these C source files with hex +char arrays inside were apparently produced from C54x COFF files with a tool +called coff2c, but we never got any of those COFF files or whatever source (C +or assembly) they were built from. At the present time in the FreeCalypso +family of projects we use the DSP patch codes (hex char arrays) which we got +with our copy of TCS211 from 20070608, and we treat the entire DSP block (the +combination of mask ROM plus patches) as a functional black box. + +Having to treat the DSP as a black box is certainly a major shortcoming of our +FreeCalypso solution. However, I (Mother Mychaela) would much rather have a +phone or modem in which only the DSP is a black box while I get to maintain all +of the upper layers with full freedom, as opposed to the status quo alternative +of a very high-level black box with FOTA backdoors. Unlike the ubiquitous +high-level black boxes from the likes of Qualcomm, the DSP in the Calypso cannot +be backdoored: it has no access to the ARM address space, thus no access to the +flash (cannot surreptitiously modify the firmware) and no access to any of the +higher-level radio protocol state maintained by the ARM, all it can do is +modulate and demodulate bursts and run voice codecs _as commanded by the ARM_. +Furthermore, the DSP has no access to the Calypso chip's TPU (Time Processing +Unit, the block that controls board-level RF hardware) and thus has no direct +control over any of the RF hardware: it cannot initiate radio transmission or +even reception on its own, instead the ARM firmware has to configure the RF +hardware via the TPU for each and every Rx or Tx time window. + +Finally, if anyone is truly paranoid about the possibility of backdoors in the +DSP, the DSP ROM code has been read out - you are welcome to hire a professional +reverser of your choice to disassemble and audit it as thoroughly as you like. +This code is unchangeable by virtue of being hard-cast in a mask ROM in the +silicon. + +The rest of this document covers the firmware that runs on the ARM core; it +controls the DSP via its API RAM, a form of shared memory interface. + +High-level structure of TCS211 firmware +======================================= + +The code base that makes up TI's TCS211 firmware consists of 3 main divisions: +chipset software, Condat G23M (GSM and GPRS L23 protocol stacks, ACI and +optional handset UI layers) and GPF. Let us look at them in turn: + +chipsetsw division +------------------ + +In the original TCS211 delivery there was a top-level directory named chipsetsw +(chipset software), containing code that is specific to TI's chipsets in +particular and was never intended to run on any other hardware. This code +division has been retained intact in our FreeCalypso Magnetite and Selenite +firmwares, taken in its entirety from our TCS211 golden reference, although we +have shortened the name: this code division now resides under src/cs in +Magnetite and Selenite. Aside from a few bits of system glue, this chipsetsw +breaks down into two further subdivisions: the L1+drivers core and the SSA +division. + +L1+drivers core +--------------- + +This division resides under chipsetsw/layer1 and chipsetsw/drivers/drv_core, or +under src/cs/layer1 and src/cs/drivers/drv_core in our version. The most +important piece here is L1 (GSM Layer 1): this code drives the DSP and the RF +hardware, and thereby makes the Calypso function as a GSM MS (mobile station) +and not merely as a general purpose microprocessor platform. This code can be +considered to be the most important part of the entire firmware. + +At one time TI had a so-called standalone L1 configuration, selected by the +OP_L1_STANDALONE C preprocessor symbol. We don't have the bits that are needed +to build this configuration (they were probably never released outside of TI at +all), but it appears that this fw build configuration consisted of just Nucleus, +L1, the drivers under drv_core, the OSL and OSX parts of GPF without the rest, +and some stubs for the few higher-level functions that are intertied with L1. + +The drivers under chipsetsw/drivers are divided into drv_core and drv_app: the +former are the most essential or fundamental ones, used by L1 and/or needed for +the OP_L1_STANDALONE config; the latter belong to the higher-level SSA division +described below. + +SSA division +------------ + +TI had a group called System Software and Applications (SSA), and they supplied +those parts of the firmware that are neither L1+drv_core nor Condat G23M. The +more interesting pieces here include the flash file system (FFS), the debug +trace facility (RVT), the Enhanced Test Mode (ETM) facility that allows +external development and production tools to poke at the firmware, RiViera Audio +Service (playing various beeps and ringtones through the DSP, a front-end to L1 +audio functions), LCD and keypad drivers for Calypso-based handsets, and various +supportive functions implemented via the Iota ABB: switch-on and switch-off +logic, battery monitoring and charging, backlight LED control. + +All firmware components in the SSA division are built on top of a framework +called RiViera - more will be said about it later. Everything under +chipsetsw/drivers/drv_app, chipsetsw/riviera and chipsetsw/services (or under +src/cs/drivers/drv_app, src/cs/riviera and src/cs/services in our version) +belongs to the SSA realm. + +Condat G23M division +-------------------- + +At the beginning of TI's involvement in the GSM baseband chipset business, they +only developed and maintained their own L1 code, which eventually grew into the +larger chipsetsw division described above, while the rest of the protocol stack +(which is hardware-independent) was licensed from another company called Condat. +Later Condat as a company was fully acquired by TI, and the once-customer of +this code became its owner. The name of TI/Condat's implementation of GSM +layers 2&3 for the MS side is G23M, and it forms its own major division of the +overall fw architecture. + +The overall Condat code realm can be further subdivided into GSM and GPRS L23 +protocol stacks, the Application Control Interface (ACI) which includes the AT +command interpreter (ATI), and additional phone UI layers which are only +included in handset but not modem firmwares. + +We don't know exactly how TI maintained this software internally: given that it +is mostly hardware-independent aside from integration details and some minor +features which may be present on one hw platform but not on another, it would +have made the most sense for TI to maintain a single internal mainline common +to both Calypso and LoCosto, and then integrate the code from this mainline into +chipset-specific customer releases. We have no way of knowing if TI indeed +followed this approach or not, but when we took the version of G23M from the +TCS3.2 source for the LoCosto chipset and grafted it onto the chipsetsw +foundation from TCS211 for the Calypso to produce our TCS2/TCS3 hybrid, the +integration went surprisingly smoothly. The full-source version of G23M which +we took from TCS3/LoCosto is newer than the binary-only version featured in the +world's last surviving copy of TCS211 from Openmoko. + +GPF island of stability +----------------------- + +Underlying the G23M protocol stack is a special layer called GPF, which was +originally Condat's Generic Protocol stack Framework. Apparently Condat were +in the business of developing and maintaining a whole bunch of protocol stacks: +GSM MS side, GSM network side, TETRA and who knows what else. GPF was their +common underpinning for all of their protocol stack projects, which ran on top +of many different OS environments: Nucleus, pSOS, VxWorks, Unix/Linux, Win32 +and who knows what else. + +In the case of TI/FreeCalypso GSM fw, both the protocol stack and the underlying +OS environment are fixed: GSM and Nucleus, respectively. But GPF is still a +critically important layer in the firmware architecture: in addition to serving +as the glue between the G23M stack and Nucleus, it provides some important +support infrastructure for the protocol stack. + +However, what makes GPF very special is the way in which it relates to the rest +of the firmware architecture. GPF remained common and unchanged across TI's +many different projects, and it is so independent from the rest of the firmware +and its build configuration that TI were able to make company-wide GPF library +builds and then plop them into multiple fw projects which used them as +configuration-independent prebuilt libraries. All TI firmware (semi-)sources +we've got use GPF in prebuilt library form and are not set up to recompile any +part of it from source. + +Our FC Magnetite firmware uses the original binary libs from TCS211-Openmoko +for its GPF component, but for FC Selenite the project requirement is to be +completely blob-free, hence we had to reconstruct the source for GPF. The +original source for most parts of GPF was found between TCS3.2 from Peek/FGW +and TCS211 from OM (the former had the source for the core "frame" modules and +the latter had the source for misc and tst), but we never got the source for the +OSL and OSX components, hence we had to reconstruct them from disassembly. OSL +is the glue layer between GPF and Nucleus, OSX is the glue layer between GPF +and L1. + +Firmware boot process +===================== + +As already mentioned earlier, the Calypso chip itself includes an ARM boot ROM +in the silicon that serves as an unbricking aid: it provides a certain time +window during which the boot process can be interrupted and diverted if certain +magic characters are sent into either of Calypso's two UARTs by an external +host, and if nothing is received on either UART during that time window, the +boot ROM transfers control to the firmware image in the external flash. As we +understand it, Calypso was TI's first DBB (digital baseband processor) chip to +include this boot ROM, and their previous DBB chips did not have such: they +would always execute code directly from external flash immediately out of reset. + +TI's TCS211 and earlier firmwares are structured in such a way that they boot +and run exactly the same way whether the Calypso boot ROM is present and +enabled, present but disabled, or not present at all. They put magic constant +0x00000001 in the 32-bit word at flash address 0x2000, which tells the Calypso +boot ROM (if it is present and enabled) to boot the flash fw image in legacy +mode: after providing the unbricking time window, the boot ROM moves itself out +of the way (sets two bits in the FFFF:FB10 register which tell the chip to unmap +the boot ROM and to map external memory at address 0) and induces a watchdog +reset, causing the chip to re-execute the reset vector, this time directly out +of external flash - thus the firmware boots as if the boot ROM weren't there, +but the ROM's unbricking function is retained. + +In order to make it easier to load new firmware images during development on +pre-Calypso platforms which didn't have a boot ROM, TI had developed a flash- +resident bootloader stage and included it in their fw architecture. This +bootloader stage is placed at the beginning of the flash at the reset vector, +and the rest of the firmware begins at an erase unit boundary. The bootloader +stage executes first, and before it jumps to the main firmware entry point +(_INT_Initialize) for normal boot, it offers an opportunity for the boot process +to be interrupted and diverted if an external host sends certain magic command +packets into either of the two UARTs during the allotted time window. If the +external host does interrupt and divert the boot process in this manner, it can +feed a code image to the bootloader to be written somewhere in target RAM, and +then command the bootloader to jump to it. It is exactly the same functionality +(though with different serial protocol specifics) as implemented in the Calypso +boot ROM. The ROM version is obviously superior because it is unbrickable, but +the flash-resident, built-with-firmware version is what TI used before they +came up with the idea of the boot ROM for the Calypso. + +When the boot-ROM-equipped Calypso came along, TI kept the flash-resident +bootloader in the firmware: it does no harm aside from adding a little bit of +delay to the boot process, it does not conflict with the ROM bootloader as the +two speak different serial protocols and respond to different interrupt-boot +sequences, and it allowed TI to keep the same firmware architecture for +platforms with and without a boot ROM. However, in our FreeCalypso firmwares +starting with Magnetite we have removed this extra bootloader stage for the +following reasons: + +* It is not useful to us on any of our hardware targets: on those devices that + have the Calypso boot ROM enabled, we use that boot ROM and get full + unbrickability, whereas on Mot C1xx phones we have to work with Mot/Compal's + own different bootloader and serial protocol at least initially, hence it + makes the most sense to stick with the same after the conversion to + FreeCalypso as well. + +* As delivered by TI with their full production TCS211 fw releases, their + firmware-resident bootloader works as intended only on hw platforms with + 13 MHz VCXOs like the original D-Sample (Clara RF), and is broken on platforms + like Rita RF (the only RF chip for which we have driver code!) with 26 MHz + VCXOs: there is no conditionally-compiled code anywhere in the bootloader + code path to set the VCLKOUT_DIV2 bit in the CNTL_CLK register on 26 MHz + platforms, thus the UARTs are fed with 26 MHz instead of the standard 13 MHz + clock expected in normal operation, and the intended baud rate of 115200 bps + turns into 230400. Because 230400 bps is a baud rate which Calypso UARTs + *cannot* produce in normal GSM operation (when the peripheral clock network + runs at the expected 13 MHz), tools that are designed to talk to Calypso GSM + devices are typically not designed to support this baud rate. In particular + for CP2102 USB-serial adapters, the precedent established by the factory + CP2102 EEPROM programming in the Pirelli DP-L10 phone is that the baud rate + entry for 230400 bps is replaced with 203125 bps, which is a valid baud rate + for Calypso UARTs running at 13 MHz. + +* We have no source for TI's firmware-resident bootloader, only linkable binary + objects that came with our world's last surviving copy of TCS211, which are + incompatible with our goal of blob-free firmware. + +Because this extra bootloader stage is ultimately unnecessary in our +environment, the deblobbing goal was easier accomplished by removing it +altogether instead of expending effort on a blob-free replacement. Because I +wasn't comfortable with modifying TMS470 assembly code and linker script magic, +the removal of the bootloader was accomplished by stubbing out its C body with +an empty function. In the gcc-built FC Selenite version it is removed +completely, without any leftover stubs. + +Finally, it needs to be noted for the sake of completeness that Compal's +bootloader used on Mot C1xx phones is a modified version based on TI's original +bootloader. However, this factoid matters only for historians and genealogists; +for all practical purposes it is an unrelated animal, as Mot/Compal's serial +protocol for interrupting and diverting the boot process is their own and bears +no resemblance to TI's version. And yes, Mot/Compal's version does set the +VCLKOUT_DIV2 bit in the CNTL_CLK register to adjust for the 26 MHz clock input +as its first order of business; it was probably the very first issue they had +to fix. + +When we build FC Magnetite or FC Selenite TMS470 firmware for Mot C1xx targets, +we use dd to strip off the first 64 KiB of the image produced by TI's linker +(the part where TI's bootloader resides, be it intact or stubbed out) and flash +the remaining image (the main body of the fw) starting at flash address 0x10000. +In the gcc-built Selenite version we natively link images that are designed to +be flashed at 0x10000 without any dirty hacks. Common to all FC firmwares for +C1xx targets, the bootloader image we put at 0 (in the brickable flash sector) +is a modified version based on one of Mot/Compal's originals: we have binary- +patched it to redirect the exception vectors from Mot/Compal's 0x20A0 to 0x10000 +and to move the main fw entry point from Mot/Compal's 0x20F8 to TI's 0x10058. + +None of this muckery applies to our own FreeCalypso hardware or to our +predecessor Openmoko's hw: on these good hw targets the complete fw image as +built is flashed at 0, and there is no possibility of bricking because we use +the boot ROM to gain access irrespective of what's in the flash. + +Main firmware entry point +------------------------- + +With the bootloader distraction out of the way, the main fw entry point is at +the _INT_Initialize symbol in the int.s assembly module, located in +src/cs/system/main/int.s in Magnetite and Selenite. The functional equivalent +for the gcc environment in Selenite is in src/cs/system/main/gcc/bootentry.S. +This assembly code performs some basic hardware initialization, sets up +sensible memory timings for the boot path phase before DPLL setup, copies the +IRAM code (the code that is intended to execute out of the fast internal RAM) +from flash to where it needs to be, zeros both IRAM and XRAM .bss regions, does +TI's cinit/auto_init business for initialized data in the TMS470 environment +(Selenite gcc version copies .data from flash to RAM instead), sets up the +system, IRQ, FIQ and exception stacks, does some assembly initialization for +Nucleus and finally jumps to Nucleus' C entry point INC_Initialize(). + +Further initialization takes place in the Init_Target() and Init_Drivers() +functions called from Application_Initialize(), which is the last function +called by INC_Initialize() before starting the Nucleus task scheduler. + +Nucleus environment +=================== + +Like all classic TI firmwares, ours is based on the Nucleus PLUS RTOS. Just +like TI's original code on which we are based, we use only a small subset of +the functionality provided by Nucleus - but because the latter is a library, +the pieces we don't use simply don't get pulled into the link. The main +function we get out of Nucleus is the scheduling of threads, or tasks as +Nucleus calls them. + +Aside from pre-stack-setup assembly init code and ARM exception handlers, every +piece of code in the firmware executes in one of the following contexts: + +* Application_Initialize(): this function and everything called from it execute + just before Nucleus' thread scheduler starts; at this point interrupts are + disabled at the ARM7 core level (in the CPSR) and must not be enabled; the + stack is Nucleus' "system stack" which is also used by the scheduler and LISRs + as explained below. + +* Regular threads or tasks: once Application_Initialize() finishes, all code + with the exception of interrupt handlers (LISRs and HISRs as explained below) + runs in the context of some Nucleus task. Whenever you are trying to debug + or simply understand some piece of code in the firmware, the first question + you should ask is "which task does this code execute in?". Most functional + components run in their own tasks, i.e., a given piece of code is only + intended to run within the Nucleus task that belongs to the component in + question. On the other hand, some components are implemented as APIs, + functions to be called from other components: these don't have their own task + associated with them, and instead they run in the context of whatever task + they were called from. Some only get called from one task: for example, the + "uartfax" driver API calls only get called from the protocol stack's UART + entity, which is its own task. Other component API functions like FFS and + trace can get called from just about any task in the system. Many components + have both their own task and some API functions to be called from other tasks, + and the API functions oftentimes post messages to the task to be worked on by + the latter; the just-mentioned FFS and trace functions work in this manner. + + In our TCS211-mimicking Magnetite and Selenite firmwares every Nucleus task is + created either through RiViera or through GPF, and not in any other way - see + the description of RiViera and GPF below. + +* LISRs (Low level Interrupt Service Routines): these are the interrupt handlers + that run immediately when an ARM IRQ or FIQ comes in. The code at the IRQ and + FIQ vector entry points calls Nucleus' magic stack switching function + (switches the CPU from IRQ/FIQ into SVC mode, saves the interrupted thread's + registers on that thread's stack, and switches to the "system" stack) and + then calls TI's IRQ dispatcher implemented in C. The latter figures out + which Calypso interrupt needs to be handled and calls the handler configured + in the compiled-in table. Nucleus' LISR registration framework is not used + by the GSM fw, but these interrupt handlers should be viewed as LISRs + nonetheless. + + There is one additional difference between canonical Nucleus and TI's version + (we've replicated the latter): canonical Nucleus was designed to support + nested LISRs, i.e., IRQs re-enabled in the magic stack switching function, + but in TI's version which we follow this IRQ re-enabling is removed: each LISR + runs with interrupts disabled and cannot be interrupted. (The corner case of + an FIQ interruping an IRQ remains to be looked at more closely as bugs may be + hiding there, but Calypso doesn't really use FIQ interrupts.) There is really + no need for LISR nesting in our GSM fw, as each LISR is very short: most LISRs + do nothing more than trigger the corresponding HISR. + +* HISRs (High level Interrupt Service Routines): these hold an intermediate + place between LISRs and tasks, similar to softirqs in the Linux kernel. A + HISR can be activated by a LISR calling NU_Activate_HISR(), and when the LISR + returns, the HISR will run before the interrupted task (or some higher + priority task, see below) can resume. HISRs run with CPU interrupts enabled, + thus more interrupts can occur, with their LISRs executing and possibly + triggering other HISRs. All triggered HISRs must complete and thereby go + "quiescent" before task scheduling resumes, i.e., all HISRs as a group have a + higher scheduling priority than tasks. + +Nucleus implements priority scheduling for tasks. Tasks have their priority set +when they are created (through RiViera or GPF, see below), and a higher priority +task will run until it gets blocked waiting for something, at which time lower +priority tasks will run. If a lower priority task sends a message to a higher +priority task, unblocking the latter which was waiting for incoming messages, +the lower priority task will effectively suspend itself immediately while the +higher priority task runs to process the message it was sent. + +HISRs oftentimes post messages to their associated tasks as well; if one of +these messages unblocks a higher priority task, that unblocked task will run +upon the completion of the HISR instead of the original lower priority task +that was interrupted by the LISR that triggered the HISR. Nucleus' scheduler +is fun! + +RiViera and GPF +=============== + +RiViera and GPF are two parallel/independent/competing wrappers around or +layers above Nucleus. GPF comes from Condat and is used by the G23M protocol +stack and indirectly by L1 (the peculiar way in which L1 ties in with the rest +of the firmware will be covered later), whereas RiViera is used by the fw +components from TI's SSA group: flash file system, debug trace, RiViera Audio +Service and so forth. + +At some point in their post-Calypso TCS3.x program TI decided to eliminate +RiViera as an independent framework and to reimplement RiViera APIs (used by +peripheral but necessary code such as FFS, ETM, various drivers etc) over GPF. +This arrangement is used in the TCS3.2 LoCosto firmware from which we have +lifted our source replacements for much of the code that came as binary objects +in our reference TCS211 version. However, our current Magnetite and Selenite +firmwares follow the architecture of TCS211, not that of TCS3.2, and because +the entire SSA division of the fw including the RiViera core came in full source +form in our copy of TCS211, it was only natural to keep this code and its +architecture. + +Start-up process continued +========================== + +As mentioned earlier, Nucleus calls the application's software init function +called Application_Initialize() after it initializes itself but before starting +the task scheduler. This function in TCS211 is just the following: + +Application_Initialize() +{ + Init_Target(); + Init_Drivers(); + Cust_Init_Layer1(); + Init_Serial_Flows(); + StartFrame(); + Init_Unmask_IT(); +} + +Cust_Init_Layer1() is in L1, StartFrame() is in GPF, and the remaining 4 init +functions live in the init.c module under src/cs/system/main. + +The Init_Target() function finishes the hardware initialization that was +started by the assembly code at the firmware boot entry point (int.s): among +other things, it sets up the final memory timings that will be used by the +running fw and configures the Calypso DPLL which provides multiplied internal +clocks to both ARM and DSP cores. On Calypso C035 silicon which is used on our +own FreeCalypso boards and on most of our pre-existing hw targets the DPLL and +the DSP run at 104 MHz and the ARM gets half of that, running at 52 MHz. +Init_Target() also calls AI_InitIOConfig(), the function that initializes +Calypso GPIO directions and initial outputs; both of these functions typically +need to be tweaked when adding support for a new Calypso board target. + +The Init_Drivers() function is primarily responsible for initializing RiViera +and FFS, although it also does a bit of init related to ABB and SIM drivers. + +I mentioned earlier that every Nucleus task in our firmware gets created and +started either through RiViera or through GPF. All GPF tasks are created and +placed into the runable state in the Application_Initialize() context: the work +is done by GPF init code in gpf/frame/frame.c, and the top level GPF init +function called from Application_Initialize() is StartFrame(). Thus when +Application_Initialize() finishes and the Nucleus thread scheduler starts +running for the first time, all GPF tasks are there to be scheduled. + +There is a compiled-in table of all protocol stack entities and the tasks in +which they need to run; in TCS211 these GPF config bits live under +g23m/condat/frame/config for the GSM+GPRS configuration and under +g23m/condat/com/src/config for the GSM-only config without GPRS. Canonically +each protocol stack entity runs in its own task, but sometimes two or more are +combined to run in the same task: for example, in the minimal GSM "voice only" +configuration (no CSD, fax or GPRS) CC, SMS and SS entities share the same task +named CM. Unlike RiViera, GPF does not support dynamic starting and stopping +of tasks. + +As each GPF task starts running (immediately upon entry into Nucleus' scheduling +loop as Application_Initialize() finishes), pf_TaskEntry() function in +gpf/frame/frame.c is the first code it runs. This function creates the queue +for messages to be sent to all entities running within the task in question, +calls each entity's pei_init() function (repeatedly until it succeeds: it will +fail until the other entities to which this entity needs to send messages have +created their message queues), and then falls into the main body of the task: +for all "regular" entities/tasks except L1, this main body consists of waiting +for messages (or signals or timeouts) to arrive on the queue and dispatching +each received message to the appropriate handler in the right entity. + +RiViera tasks get started in a different way. The responsible code lives in +src/cs/system/main/create_RVtasks.c, and the create_tasks() function found in +that module is called by Init_Drivers() in the Application_Initialize() context. +But this function does not directly create and start every configured RiViera +task like StartFrame() does for GPF. Instead it creates a special helper task +which will do this work once scheduled. Thus at the completion of +Application_Initialize() and the beginning of scheduling the set of runable +Nucleus tasks consists of all GPF ones plus the special RV starter task. Once +the RV starter task gets scheduled, it will call rvm_start_swe() to launch +every configured RiViera SWE (SoftWare Entity), which in turns entails creating +the tasks in which these SWEs are to run. + +Dynamic memory allocation +========================= + +All dynamic memory allocation (i.e., all RAM usage beyond statically allocated +variables and buffers) is once again done either through RiViera or through GPF, +and in no other way. Ultimately all areas of the physical RAM that will ever +be used by the fw in any way are allocated when the fw is compiled and linked: +the areas from which RiViera and GPF serve their dynamic memory allocations are +statically allocated as char arrays in the respective C modules and placed in +the appropriate IRAM or XRAM .bss section by the linker script; RiViera and GPF +then provide API functions that allocate memory dynamically from these +statically allocated large pools. + +RiViera and GPF have entirely separate memory pools from which they serve their +respective clients, hence there is no possibility of one affecting the other. +Riviera's memory allocation scheme is very much like the classic malloc&free: +there is one large unstructured pool from which all allocations are made, one +can allocate a chunk of any size, free chunks are merged when physically +adjacent, and fragmentation is an issue: a memory allocation request may fail +even when there is enough memory available in total if it is too fragmented. + +GPF's dynamic memory allocation facility is considerably more robust: while it +does maintain one or two (depending on configuration) memory pools of the +traditional "dynamic" kind (like malloc&free, susceptible to fragmentation), +most GPF memory allocation works on "partition" memory instead. Here GPF +maintains 3 separate groups of pools: PRIM, TEST and DMEM; each allocation +request must specify the appropriate pool group and cannot affect the others. +Within each pool there is a fixed number of partitions of a fixed size: for +example, in TI's TCS211 GSM+GPRS configuration the PRIM pool group consists of +190 partitions of 60 bytes, 110 partitions of 128 bytes, 50 partitions of 632 +bytes and 7 partitions of 1600 bytes. An allocation request from a given pool +group (e.g., PRIM) can request any arbitrary size in bytes, but it gets rounded +up to the nearest partition size and allocated out of the respective pool. If +no free partition is available, the requesting task is suspended until another +task frees one. Because these partitions are used primarily for intertask +communication, if none are free, it can only mean (assuming that the firmware +functions correctly) that all partitions have been allocated and sent to some +queue for some task to work on, hence eventually they will get freed. + +This scheme implemented in GPF is extremely robust in the opinion of this +author, and the other purely "dynamic" scheme is used (in the case of GPF) only +for init-time allocations which are never freed, such as task stacks - hence +the GPF-based part of the firmware is not suspectible at all to the problem of +memory fragmentation. But Riviera does suffer from this problem, and the +concern is more than just theoretical: one major user of Riviera-based dynamic +memory allocation is the trace facility (described in its own section below), +and my observation of the trace output from Pirelli's proprietary fw (which +appears to use the same architecture with separate Riviera and GPF) suggests +that after the fw has been running for a while, Riviera memory gets fragmented +to a point where many traces are being dropped. Replacing Riviera's poor +dynamic memory allocation scheme with a GPF-like partition-based one is a to-do +item for our project. + +Message-based intertask communication +===================================== + +Even though all entities of the G23M protocol stack are linked together into +one monolithic fw image and there is nothing to stop them from calling each +other's functions and accessing each other's variables, they don't work that +way. Instead all communication between entities is done through messages, just +as if they ran in separate address spaces or even on separate processors. +Buffers for this message exchange are allocated from a GPF partition pool: an +entity that needs to send a message to another entity allocates a buffer of the +needed size, fills it with the message to be sent, and posts it on the recipient +entity's message queue, all through GPF services. The other entity simply +processes the stream of messages that arrives on its message queue, freeing each +message (returning the buffer to the partition pool it came from) as it is +processed. + +Riviera-based tasks use a similar mechanism: unlike G23M protocol stack +entities, most Riviera-based functional modules provide APIs that are called as +functions from other tasks, but these API functions typically allocate a memory +buffer (through Riviera), fill it with the call parameters, and post it to the +associated task's message queue (also in the Riviera land) to be worked on. +Once the worker task gets the job done, it will either call a callback function +or post a response message back to the requestor - the latter option is only +possible if the requesting entity is also Riviera-based. + +A closer look at GPF +==================== + +There are certain sublayers within GPF which need to be pointed out. The 3 +major subdivisions within GPF are: + +* The meaty core of GPF: this part is the code under src/gpf/frame in our + Selenite GPF reconstruction, originating from gpf/FRAME in the TCS3.2 source + from Peek/FGW. It appears that this part was originally intended to be both + project-independent (same for GSM, TETRA etc) and OS-independent (same for + Nucleus, pSOS, VxWorks etc). This is the part of GPF that matters for the + G23M stack: all APIs called by PS entities are implemented here, and so are + all other PS-facing functions such as startup. (PS = protocol stack) + +* OS adaptation layer (OSL): this is the part of GPF that adapts it to a given + underlying OS, in our case Nucleus. + +* Test interface: see the code under gpf/tst in TCS211 from Openmoko or in + FC Selenite. This part handles the trace output from all entities that run + under GPF and the mechanism for sending external debug commands to the GPF+PS + subsystem. + +GPF was a difficult step in our GSM firmware deblobbing process because no +complete source for it could be found anywhere: apparently GPF was so stable +and so independent of firmware particulars (Calypso or LoCosto, GSM only or +GSM+GPRS, modem or complete phone with UI etc) that it appears to have been +used and distributed as prebuilt binary libraries even inside TI. All TI fw +(semi-)sources we've got use GPF in prebuilt library form and are not set up to +recompile any part of it from source. (They had to include all GPF header +files though, as most of them are included by G23M C modules, and it would be +too much hassle to figure out which ones are or aren't needed, hence all were +included.) + +Fortunately though, we were able to find the sources for most parts of GPF: + +* The LoCosto source in TCS3.2_N5.24_M18_V1.11_M23BTH_PSL1_src.zip features the + source for the "core" part of GPF under gpf/FRAME - these sources aren't + actually used by that fw's build system (it only uses the prebuilt binary + libs for GPF), but they are there. + +* Our TCS211 semi-src doesn't have any sources for the core part of GPF, but + instead it features the source for the test interface and some "misc" parts: + under gpf/MISC and gpf/tst in that source tree - these sources are not present + in the LoCosto version from Peek. + +The GPF frame, misc and tst sources we have found have been verified to match +the binary objects that came with TCS211 from OM: they can be compiled into a +bit-for-bit match. However, one critical piece was still missing: the OS +adaptation layer. It appears that the GPF core (vsi_??? modules) and OSL +(os_??? modules) were maintained and built together, ending up together in +frame_.lib files in the binary form used to build firmwares, but the +source for the "frame" part in the Peek find contained only vsi_*.c and others, +but not any of os_*.c. + +Our FC Magnetite firmware uses the original binary libs from TCS211-Openmoko +for its GPF component, but for FC Selenite the project requirement is to be +completely blob-free, hence we had to reconstruct the source for the OSL part +of GPF from disassembly. This work was originally done in 2014 in the context +of our first attempt at gcc-built blob-free GSM fw (FC Citrine, now deemed to +be a dead end and fully retired); this reconstruction was then dug up and +adapted for Selenite in 2018. As of this writing, this reconstruction is still +not 100% complete (one complex error handling function is stubbed out) and not +yet trusted to be fully correct, thus our fully deblobbed Selenite firmware is +currently considered experimental; our current production fw is still Magnetite +with blobs for GPF. + +A closer look at L1 +=================== + +The L1 code is remarkable in how little intertie it has with the rest of the +firmware it is linked into. It is almost entirely self-contained, expecting +only 4 functions to be provided by the underlying OS environment: + +os_alloc_sig -- allocate message buffer +os_free_sig -- free message buffer +os_send_sig -- send message to upper layers +os_receive_sig -- receive message from upper layers + +It helps to remember that at the beginning of TI's involvement in the GSM +baseband chipset business, L1 was the only thing they "owned", while Condat, +the maintainers of the higher level protocol stack, was a separate company. +TI's "turnkey" solution must have consisted of their own L1 code plus G23M code +(including GPF etc) licensed from Condat, but I'm guessing that TI probably +wanted to retain the ability to sell their chips with their L1 without being +entangled by Condat: let the customer use their own GSM L23 stack, or perhaps +work out their own independent licensing arrangements with Condat. I'm +guessing that L1 was maintained as its own highly independent and at least +conceptually portable entity for these reasons. + +The way in which L1 is intertied into the rest of the fw is the same in all TI +production firmwares we have seen, including both our TCS211 reference and the +TCS3.2 LoCosto version. There is a module called OSX, which is an extremely +thin adaptation layer that implements the APIs expected by L1 in terms of GPF. +Furthermore, this OSX layer provides header file isolation: the only "outside" +(non-L1) header included by L1 is cust_os.h, and it defines the necessary +interface to OSX *without* including any other headers (no GPF headers in +particular), using only the C language's native types. Apart from this +cust_os.h header, the entire OSX layer is implemented in one C module (osx.c, +which we had to reconstruct from osx.obj as the source was missing - but it's +very simple) which does include some GPF headers and implements the OSX API in +terms of GPF services. Thus in both TI's production firmwares and our own ones, +L1 does sit on top of GPF, but very indirectly. + +More specifically, the "production" version of OSX implements its API in terms +of *high-level* GPF functions, i.e., VSI. However, they also had an interesting +OP_L1_STANDALONE configuration which omitted not only all of G23M, but also the +core of GPF and possibly the Riviera environment as well. We don't have a way +to recreate this configuration exactly as it existed inside TI because we don't +have the source bits specific to this configuration, but we do have a little +bit of insight into how it worked. + +It appears that TI's OP_L1_STANDALONE build used a special "gutted" version of +GPF in which the "meaty core" (VSI etc) was removed. The OS layer (os_??? +modules implementing os_*() functions) that interfaces to Nucleus was kept, and +so was OSX used by L1 - but this time the OSX API functions were implemented in +terms of os_*() ones (low-level wrappers around Nucleus) instead of the higher- +level VSI APIs provided by the "meaty core" of GPF. It is purely a guess on my +part, but perhaps this hack was also done in the days before TI's acquisition +of Condat, and by omitting the "meaty core" of GPF, TI could claim that their +OP_L1_STANDALONE configuration did not contain any of Condat's "intellectual +property". + +Run-time structure of L1 +======================== + +L1 consists of two major parts: L1S and L1A. L1S is the synchronous part where +the most time-critical functions are performed; it runs as a Nucleus HISR. The +hardware in the Calypso generates an interrupt on every TDMA frame (4.615 ms, +or more precisely 60/13 ms), and the LISR handler for this interrupt triggers +the L1S HISR. L1S communicates with L1A through a shared memory data structure, +and also sometimes allocates message buffers and posts them to L1A's incoming +message queue (both via OSX API functions, i.e., via GPF in disguise). + +L1A runs as a regular task under Nucleus, and includes a blocking call (to GPF +via OSX) to wait for incoming messages on its queue. It is one big loop that +waits for incoming messages, then processes each received message and commands +L1S to do most of the work. The entry point to L1A in the L1 code proper is +l1a_task(), although the responsibility for running it as a task falls on some +"glue" code outside of L1 proper. TI's production firmwares with G23M included +have an L1 protocol stack entity within G23M whose only job (aside from some +initialization) is to run l1a_task() in the Nucleus task created by GPF for +that protocol stack entity; we do the same in our firmwares. + +Communication between L1 and G23M +================================= + +It is remarkable that L1 and G23M don't have any header files in common: L1 +uses its own (almost fully self-contained), whereas the G23M+GPF realm is its +own world with its own header files. One has to ask then: how do they +communicate? OK, we know they communicate through primitives (messages in +buffers allocated from GPF's PRIM partition memory pool) passed via message +queues, but what about the data structures in these messages? Where are those +defined if there are no header files in common between L1 and G23M? + +The answer is that there are separate definitions of the L1<->G23M interface on +each side, and TI must have kept them in sync manually. Not exactly a +recommended programming or software maintenance practice for sure, but TI took +care of it, and the existing proprietary products based on TI's firmware are +rock solid, so it is not really our place to complain. + +TI's firmwares from the era we are working with (both our TCS211 golden +reference and the TCS3.2/LoCosto source from which we took the newer full-source +version of G23M for our TCS2/TCS3 hybrid) also include a component called ALR. +It resides in the G23M code realm: G23M coding style, uses Condat header files, +runs as its own protocol stack entity under GPF. This component appears to +serve as a glue layer between the rest of the G23M stack (which is supposed to +be truly hardware-independent) and TI's L1. + +Speaking of ALR, it is worth mentioning that there is a little naming +inconsistency here. ALR is known to the connect-by-name logic in GPF as "PL" +(physical layer, apparently), while the ACI entity (Application Control +Interface, the top level entity) is known to the same logic as "MMI". No big +deal really, but hopefully knowing this quirk will save someone some confusion. + +A closer look at our FreeCalypso TCS2/TCS3 hybrid +================================================= + +Because we don't have an official TI firmware release for the Calypso in full +source form and because I am not willing to throw away all of our Calypso work +and restart anew with LoCosto with its own host of unknowns, the only currently +available way for us to have blob-free production-quality GSM mobile station fw +is the TCS2/TCS3 hybrid implemented in FC Magnetite and Selenite. This hybrid +is made by taking the G23M version from TCS3/LoCosto and grafting it onto the +chipsetsw foundation from TCS211, including the original TCS211/Calypso version +of L1 which we have meticulously source-reconstructed. The version of GPF used +for this hybrid is also the TCS211 version in Magnetite or our source +reconstruction thereof in Selenite. + +The Condat G23M pieces have been hybridized as follows: + +* cdginc generated header files are a special hybrid version described below; + +* The include files under condat/com/inc and condat/com/include are the TCS3 + version, except for pwr.h and rtc.h for which we use the TCS2 version; + +* comlib is the TCS2 version, except for cl_rlcmac.c which is from TCS3; + +* config modules (condat/com/src/config and condat/frame/config) are the TCS2 + version, with some fixes for the needs of the TCS3 version of G23M PS and our + own FreeCalypso fixes; + +* Condat drivers (condat/com/src/driver) are the TCS2 version; + +* All G23M PS components are the TCS3 version by necessity, as this is the part + for which the source is missing in our TCS211 version, with the exception of + ALR - the original source for the TCS211 version of ALR has miraculously + survived, the ALR source in TCS211 from OM can be compiled into a perfect + match for the binary lib version; + +* We use the TCS2 version of ALR (the interface to our TCS211 L1) and not the + TCS3 version (a change from Citrine), but it is compiled with the same hybrid + cdginc headers as the rest of hybrid G23M, not the old TCS211 ones; + +* ACI is the TCS3 version - we have the source for both versions, but trying to + use the old TCS2 version of ACI on top of the new TCS3 version of the PS + would cause untold breakage; + +* The UI layers (MFW and BMI) for handset fw builds are handled like ACI: we + have the source for both versions, but we use the TCS3 version which works + with the TCS3 versions of ACI and cdginc; + +* The CST (Customer Specific Task) component is the TCS2 version - while it + logically belongs in the Condat realm, the code lives in the chipsetsw realm + under chipsetsw/services/cst (yes, it's under services with SSA stuff even + though it doesn't use RiViera) and thus our copy of TCS211 from OM has this + source preserved. + +With this hybrid arrangement the main splice point lies above ALR, and there +are many little splice points throughout the code where some upper-level code +from TCS3 needs to talk to lower-level code from TCS2. There are no inversions, +i.e., no places where TCS2 code sits on top of code from TCS3, although there +are a few instances where TCS2 C code uses some TCS3 header files. + +TCS3 feature flags +------------------ + +Our TCS3.2/LoCosto code from Peek/FGW from 20090327 supports several new GSM +features (apparently related to GSM release 99) which are not supported by our +TCS211-20070608 golden reference from OM. All of these new features can be +enabled or disabled with conditional compilation flags. Our TCS2/TCS3 hybrid +currently has all of these new features disabled: it was too difficult for me +to figure out if these new features require some support from the hardware or +the DSP which is present on LoCosto but not Calypso, and even if our hw and DSP +have all of the necessary capabilities, at least some of the new features +require adding some code to L1, which is incompatible with my approach of +reconstructing TCS211 L1 pristinely. + +In any case, the GSM functionality we get by using the new version of G23M with +new feature flags disabled on top of pristine TCS211 L1 cannot be any worse +than what we would have had if we had the full corresponding source for our +TCS211-20070608 golden reference, and it is probably a little better because we +are using a newer version of G23M code. + +cdginc headers +-------------- + +Much of the code in the Condat G23M realm makes heavy use of a set of machine- +generated C header files called cdginc. These header files contain various +definitions related both to the GSM air protocols being implemented and to G23M +protocol stack internals (interfaces and message structures between components), +and they are generated from a set of message definition files (*.mdf) and +primitive definition files (*.pdf) by a tool called ccdgen. The *.{mdf,pdf} +inputs to ccdgen are human-readable ASCII, and of course the generated C header +files are human-readable too, but we have no source for the ccdgen tool itself, +only a Windows binary which we can run under Wine. + +The ccdgen binary problem is yet another instance of so far incomplete +liberation of the GSM firmware. It is currently a very low-priority problem: +we do not casually edit any of the *.{mdf,pdf} inputs to ccdgen, and we don't +run ccdgen on every fw build - instead we have run ccdgen once and checked its +output files (generated C headers) into our Magnetite and Selenite trees as if +they were sources. If we are not able to convince TI to dig up and release the +source for ccdgen, there is a viable albeit costly alternative: hire a Windows +reverser to RE the ccdgen.exe binary (262144 bytes) and produce a C +reimplementation that replicates all of its logic. It is a Win32 console app, +no GUI, and it is a pure data processing application without any hardware access +or OS functions or any other muckery: it is probably pure ANSI C code that reads +and parses a bunch of ASCII input files, performs some business logic on the +data, and writes another bunch of ASCII text files as outputs. It is currently +a very low-priority task though; reversing the Calypso DSP ROM code should +probably be a higher priority. + +The set of cdginc headers for our TCS2/TCS3 hybrid has been generated as +follows: + +* All of the *.mdf files are the TCS3 version; + +* All of the *.pdf files except mphc.pdf and mphp.pdf are also the TCS3 version; + +* mphc.pdf and mphp.pdf are the TCS211 version - this is the interface to + TCS211 L1; + +* All new feature flags (see discussion above) are set to disabled. + +Condat Coder and Decoder (CCD) +------------------------------ + +CCD is a firmware component in the Condat G23M realm which I haven't really +studied yet. It consists of two parts: + +* A fixed portion which TI used to distribute in binary form and which various + firmware projects used as a prebuilt library like GPF - technically TI + considered it to be a part of GPF, although we prefer to treat it as its own + more independent entity; + +* The ccddata portion which needs to be compiled with cdginc headers for each + given project. + +We got the source for both parts of CCD only in the TCS3.2/LoCosto version, but +not in the TCS211 version, hence the decision was easy: we use the TCS3 version +of CCD (both parts) with the TCS3 version of cdginc with the TCS3 version of +the G23M PS. + +TCS3.2 GPF discrepancy +---------------------- + +A careful examination of the prebuilt GPF libraries under gpf/LIB in the TCS3.2 +LoCosto source tree has revealed that a few of the binary objects exhibit some +differences from the TCS211 version which we've been treating as our golden +reference: + +* The os_mis module (OSL miscellany) in the IRAM library implements a new + function called os_CheckQueueEvent() and defines a new global data object + named my_os_mis_Protect; + +* The os_tim module (OSL timer code) in the flash (XIP) library has some code + differences; + +* The vsi_tim module (VSI timer code) in the flash (XIP) library has some code + differences; + +* The vsi_tim module (VSI timer code) in the IRAM library has some code + differences and makes use of the new os_CheckQueueEvent() function. + +In the case of os_??? modules we have no corresponding source for either +version, but the vsi_tim difference is more bizarre: we got our vsi_tim.c source +(and the rest of vsi_*.c) from the TCS3.2/LoCosto source, but this source +matches the TCS211 binary version and not the newer and different binary version +used by the TCS3.2 build system! (Remember that none of TI's firmware build +systems that we have seen are set up to recompile any part of GPF from source, +they used it only as prebuilt libraries.) + +Because we have the corresponding source for the "old" version of GPF frame core +but not for the "new" version, we are continuing to treat the "old" TCS211 +version as our golden reference: we use the source pieces which we got, and we +use the "old" os_???.obj blobs as our basis for reconstruction via disassembly. + +Because the changes in the TCS3.2 binary version of GPF involve only the +implementation of a part of VSI but not its API (there are no changes to any +part of the GPF API presented to the G23M PS that I can see anywhere), I have +every good reason to believe that there is no problem with using the new TCS3.2 +version of G23M with the old version of GPF from TCS211: it should work no worse +than pure TCS211. + +It should also be noted that if we ever succeed in getting some more complete +GPF source out of TI (including the source for the OS adaptation layer which is +difficult to reconstruct), thanks to the great stability and independence of +GPF, we will be happy with *any* version, does not need to match either TCS211 +or TCS3.2. + +GPRS implementation differences +------------------------------- + +There is a visible difference between the way GPRS is implemented in the old +TCS211-20070608 blob version of G23M and the way it is implemented in the newer +TCS3.2/LoCosto version we are using for our hybrid. The new implementation adds +a new protocol stack entity named UPM (User Plane Manager), and the pre-existing +SM and SNDCP entities have been significantly changed to work with this UPM. +Because we are using the GPRS config modules (condat/frame/config) from TCS211, +we had to add a -DFF_UPM compilation flag to include UPM in the GPF +configuration for the GSM+GPRS protocol stack. + +A closer look at ACI +==================== + +The Application Control Interface (ACI) is the crown that sits on top of the +G23M protocol stack. It includes the AT command interpreter (ATI) component, +and this AT command interface is brought to the outside world via the UART +protocol stack entity. The UART entity implements the GSM 07.10 MUX, can +operate the physical UART in either multiplexed or non-multiplexed mode (the +latter is the default on boot for a plain ASCII AT command interface) as +commanded by ACI, and establishes 1 to 4 logical channels carrying AT commands +to ACI. When a CSD or fax call or a GPRS PPP session is in progress, the data +path is switched to run between the UART entity and the appropriate GSM or GPRS +protocol stack destination. In the case of modem products that are designed to +be controlled by an external host via AT commands, this combination of ACI and +UART entities provides the ultimate end function of the device. + +The set of implemented AT commands is defined in ati_cmd.c: this is the C file +where new AT commands get added; there is also an enum of command IDs in +aci_cmh.h which needs to be extended. For every AT command listed in the table +in ati_cmd.c there is a handler function: for example, for the AT+CFUN command +there is a setatPlusCFUN() function that handles setting and a queatPlusCFUN() +function that handles querying. For some simple AT commands like AT+CGxx +queries the function listed in ati_cmd.c does the entirety of the work, but for +most of the interesting GSM commands (including the AT+CFUN example just used) +the set and query functions implemented in the ATI layer only handle the parsing +of ASCII arguments and generation of ASCII output (if any), whereas the actual +command implementation happens in the CMH layer below. + +Below ATI but still within ACI lies the sublayer of command handlers (CMH). +For each AT command that does something to the GSM mobile station there is a +functional equivalent, a C function that performs the same operation as the +spec-defined AT command, but is designed to be used natively from C code, +without AT command string parsing or output formatting. For the AT+CFUN example +used above, the setatPlusCFUN() ATI function parses the arguments from ASCII +and then calls sAT_PlusCFUN() to perform the actual operation, whereas the +queatPlusCFUN() ATI function calls qAT_PlusCFUN() to retrieve the current state +and then prints it out in ASCII. This functional interface is used by TI's +demo/prototype phone UI implementation described in the Handset-UI-fw companion +document. + +Finally, at the bottom of ACI lies the sublayer of Protocol Stack Adapters +(PSA): these are pieces of code that execute within the ACI task and exchange +primitives with various G23M protocol stack entities below. + +We have the source for both TCS2 and TCS3 versions of ACI. The TCS2 version is +from Openmoko, containing OM's modifications, and we had to go through these +changes and additions by OM, reject the bogus ones and reimplement the sensible +ones in the new TCS3 version of ACI for our TCS2/TCS3 hybrid going forward. + +Flash file system +================= + +Every GSM device that is based on TI's firmware architecture contains not only +the firmware image proper, but also a flash file system that is separate from +the fw image and is maintained in a different part of the flash chip. The FFS +implementation code is a mandatory part of the firmware; in TCS211 it resides +in chipsetsw/drivers/drv_app/ffs and logically belongs to the SSA realm. This +code initializes early in the fw boot process in the Application_Initialize() +context before the start of Nucleus task scheduling; the responsible function +is ffs_main_init() called from Init_Drivers(). + +Flash driver support and FFS location +------------------------------------- + +Determining the location of the flash area allocated for FFS and the flash +driver to be used to write to it is a combination of autodetection and hard- +coding. The approach implemented in the original TCS211 code is as follows: +there is a piece of autodetection code that reads the flash chip ID, and the +autodetected ID is then looked up in a hard-coded table that gives the driver +and geometry details and the location of the FFS sectors for each supported +flash chip type. However, this approach has its limitations: + +* The sequence of write operations which TI's autodetection code issues in + order to put the flash chip into its Read ID mode worked for older flash + chips that were used by TI and Openmoko, but does not work for the newer + Spansion S71PL129NC0HFW4B flash chip which we (FreeCalypso) have copied from + the Pirelli DP-L10 phone. + +* While the physical flash chip used on a given phone or modem board is a + physical property that can be autodetected, the choice of which flash sectors + should be used for FFS is a matter of policy. Before we built our own + FreeCalypso hardware, we had to run our fw on some pre-existing "alien" hw + targets, and we still support such usage to a limited extent. When we run + our FreeCalypso fw on an alien hw target as an aftermarket deal, our + aftermarket FFS location needs to be chosen quite carefully. + +* Some flash chips have two chip select banks, and with such chips it is + generally desirable to put the FFS in the second bank. However, it is a + matter of board wiring whether that second flash chip select is connected to + Calypso chip select nCS2, nCS3 or nCS4 - thus FFS addresses in the second bank + have to be hard-coded with conditional compilation per board type and cannot + be autodetected. + +To support our new repertoire of possible hardware targets, the logic has been +changed as follows in our Magnetite and Selenite firmwares: + +* When the target is our own FC hardware family (CONFIG_TARGET_FCFAM) or the + Pirelli DP-L10 phone (CONFIG_TARGET_PIRELLI), flash chip type autodetection + is disabled and a strictly fixed hard-coded FFS configuration is used; + +* When the target is one of Mot C1xx subfamilies, the autodetection logic is + kept (several different flash chip types are found in these phones), but a + different table is used, giving our aftermarket FFS configurations for these + Mot C1xx flash chip types; + +* When the target is gtamodem (Openmoko GTA01/02 modem) or dsample (TI's + D-Sample board), we use the original autodetection logic and device table + which we got from TI/Openmoko. + +We have also changed the AMD multibank flash driver to issue write commands in +a way that is compatible with our new S71PL129NC0HFW4B chip. + +FFS life cycle +-------------- + +In products that have been built according to TI's original way, including +Openmoko GTA01/02 and our own FreeCalypso devices, the FFS is formatted and +initialized with some essential content at the time of device manufacture, and +this factory-created and factory-initialized FFS then persists for the lifetime +of the device. In our factory environment at FreeCalypso hardware manufacturing +we initialize the flash on our freshly assembled boards like this: + +flash erase 0 0x800000 +flash program-bin 0 fwimage.bin +flash2 erase 0 0x800000 + +This factory procedure (which should ONLY be executed at the factory and never +by any end users or even sw/fw developers and tinkerers) ensures that the flash +is completely blank everywhere except the fw image loaded at the time of +production, and when this fw image boots for the first time, it will see blank +flash in the FFS sectors. When TI's FFS code in ffs_main_init() sees this +condition, it performs what TI called a preformat: it writes a basic FFS block +header into each FFS sector, but does not automatically perform a full format - +instead the latter needs to be commanded explicitly by the production station +via one of TMFFS command packet protocols as described later in this article. +In FreeCalypso we have adopted TMFFS2 as our choice of Test Mode FFS access +protocol, our host side implementation of this protocol is fc-fsio, and we +format and initialize the FFS on our devices with an fc-fsio command script as +part of our factory procedure. + +FFS content and usage +--------------------- + +TI's firmware architecture uses the FFS for many purposes: + +* The IMEI is stored in the FFS - GSMA can proclaim all they want that it + "MUST" be stored in some kind of super-secure one-time programmable fuses, + but in TI's architecture and in FreeCalypso it is just a regular file in the + FFS. + +* A number of RF calibration tables are stored in FFS and read by the RF code + in L1. If you have a Rohde&Schwarz CMU200 instrument which is itself in good + repair and calibration standing and a metrology-grade RF cabling setup whose + insertion loss at the relevant GSM frequencies is precisely known, creating + or recreating these RF calibration values is as simple as executing one shell + script that takes a few minutes to run - this is how we do it at FreeCalypso + hw manufacturing - but if you are an ordinary user or sw/fw developer or + tinkerer without a professional calibration station setup, you need to use + the RF calibration values that have been written into the FFS by the device + manufacturer. These RF calibration tables live under /gsm/rf. + +* /gsm/com/rfcap tells the RR component in the G23M protocol stack (not L1!) + which frequency bands are supported on a given device - on our devices it is + a factory-programmed file distinguishing between tri900 and tri850 units and + telling the firmware which bands it should scan for possible GSM cells. + +* Manufacturer, model and revision ID strings may be written into /pcm/CGMI, + /pcm/CGMM and /pcm/CGMR, respectively, to be returned by the corresponding + AT+CGxx query commands. + +* The G23M protocol stack writes a number of dynamically updated files under + the /gsm hierarchy and under /pcm. + +* TI's demo/prototype UI code (see Handset-UI-fw companion document) writes its + persistent state in files under /mmi. + +* Audio mode configuration files are kept under /aud - see the Audio-mode-config + article in freecalypso-tools. + +* If a given product uses the Melody E1 mechanism, melody files to be played + through the RiViera Audio Service are kept in FFS - see the Melody_E1 article + in freecalypso-tools. + +Building firmware for different targets +======================================= + +TI's TCS3.2 firmware for their LoCosto chipset which was rejected by the Mother +for reasons described near the beginning of this article makes a complete break +from the past and has no possibility of supporting any pre-LoCosto chips such +as our beloved Calypso, but TI's previous evolutionary developments weren't so +drastic: the evolution to Calypso from previous chips such as Hercules and +Ulysse was smoother, and our reference TCS211 fw is littered with C preprocessor +conditionals supporting TI's earlier development boards prior to D-Sample and +DBB chips prior to Calypso. + +TI's configuration management architecture supported only TI's own development +boards and not any of the end product boards: unfortunately they did not follow +a development model like the Linux kernel where everyone is encouraged to +contribute their custom board support bits upstream and the mainline kernel +strives to support every hw target that was ever supported with a single source +tree, instead it was the divergent model where every end device manufacturer +would take TI's reference firmware source and hack it for their specific needs +with no concern for upstreamability or support for targets or applications +other than their own. TI's firmware build configuration model defined the +following C preprocessor symbols relating to support for different hw targets, +all numeric, i.e., each symbol is always defined to a number: + +BOARD identifies which board is to be targeted, with numbers assigned for +different development boards made by various TI groups, but generally not for +customer boards. The only Calypso-based BOARD number is 41, originally +assigned for the D-Sample but then also reused for the Leonardo; all other +BOARD numbers are for some other chipsets that aren't Calypso. The previous +board before D-Sample was C-Sample, which is BOARD 9, but I am not sure exactly +what chipset it had - perhaps it was Ulysse/Nausica/Clara. There is still +plenty of support for BOARD 9 and even earlier boards in the firmware source we +got. + +CHIPSET identifies the main DBB chip. The interesting numbers are 7 for the +very original Calypso C05 rev A, 8 for Calypso C05 rev B (found on the D-Sample +board which the Mother scored in 2015), 10 for Calypso C035 (the Calypso silicon +version we work with in FreeCalypso), 11 for Calypso Lite (same as the regular +Calypso except for smaller IRAM), 12 for Calypso+ (a short-lived intermediate +step between Calypso and LoCosto) and 15 for LoCosto. + +ANLG_FAM (previously ANALOG) identifies the ABB chip. The numbers are 1 for +Nausica, 2 for Iota (what we use) and 3 for Syren (typically used with Calypso+ +like on the E-Sample board). + +RF_FAM (previously just RF) identifies the RF hardware hooked up to the baseband +chipset. The interesting numbers are 10 for Clara (D-Sample) and 12 for Rita, +the latter being the only RF chip for which we have driver support. + +Naturally any code that cares about DBB register differences would use the +CHIPSET definition, ABB support code would use ANLG_FAM, RF support code would +use RF_FAM, and finally code that needs to know about board-level peripherals +like LCDs and keypads would use the BOARD symbol. This model worked fine up to +D-Sample: for example, the code for C-Sample vs. D-Sample LCDs and keypads is +cleanly conditionalized on BOARD 9 vs. BOARD 41. However, the waters got badly +muddied when TI introduced their Leonardo board and instead of giving it its +own BOARD number, reused BOARD number 41 from D-Sample. + +D-Sample was TI's primary internal development platform for the Calypso, +featuring Iota for the ABB and Clara for the RF part. It was a great solid +platform in every way except the RF part: the old Clara RF is inconvenient +(needs more external parts) and TI were marketing their newer Rita RF to real +end device manufacturers, but the D-Sample still worked great for development: +if you aren't working specifically on the RF part, it doesn't matter as long as +you have a working driver for it, which we lack. Then TI made another Calypso +development board called Leonardo, featuring the same Calypso+Iota baseband +plus the newer Rita RF. But this Leonardo never fully replaced the D-Sample +for any of the high-level development in the SSA and UI groups. + +Openmoko's modem is a direct derivative of the Leonardo, the only change being +the RFFE (for some reason FIC didn't like TI's quadband RFFE as implemented on +Leonardo and E-Sample boards and used their own slightly hobbled triband RFFE +instead), and the firmware build given to OM was TI's Leonardo fw with just a +few tweaks in tpudrv12.h to account for the RFFE control signal differences. +However, because Leonardo never got its own BOARD number and the BOARD symbol +is still set to 41, all of the SSA/UI code (LCD, keypad, battery charging etc) +is still built as if for D-Sample - but none of that code is used on a pure AT +command modem without UI functions or UI hardware, hence OM probably never +noticed anything odd. + +And it wasn't just Openmoko - it appears that TI used their Leonardo boards +mostly or perhaps even solely in the ACI configuration without UI layers +(MMI=0 build configuration), while all or most UI development was done on +D-Sample kits. Their TCS211 reference fw product officially supported both +D-Sample and Leonardo targets in both ACI and BMI+MFW configurations, but if +one were to build a high-end UI-enabled config for the Leonardo like pdt_2272, +it would target a 176x220 pixel color LCD, the LCD output driver would be the +one for the D-Sample (expecting memory-mapped LCD registers on nCS3), and the +keypad driver would expect D-Sample keypad wiring. Looking at the available +Leonardo schematics I see a serial (uWire) LCD interface instead and a more +basic keypad with different wiring, so I don't see how those Leonardo+UI +firmware builds could possibly work. Perhaps some other group at TI did some +UI work on Leonardo boards, but never made it into the internal mainline +from which TCS211 releases were cut - who knows... + +Finally, aside from the basic failure to distinguish properly between D-Sample +and Leonardo boards, this whole BOARD number system provides absolutely no +mechanism to distinguish between TI's development boards and end product boards +derived from them, or between end product boards of vendor A vs. vendor B, or +between end product model A and model B from the same vendor - it's always +BOARD 41 as far as TI's code is concerned. When TI had to modify their code +for OM to support FIC's different TSPACT signal wiring, they just edited the +definitions in tpudrv12.h without any conditionals, so one couldn't build +binaries for the original Leonardo vs. OM's hardware from the same source tree +in different configs. + +The build system of TCS211 produces a set of generated C header files named +*.cfg (instead of the more natural *.h); these generated config headers define +all of the C preprocessor symbols listed above and many more. They are included +sometimes as #include "board.cfg" and othertimes as #include "config/board.cfg" +(ditto for other *.cfg), thus the list of -I directories passed by the build +system on compiler invocation lines needs to include both the config directory +and its parent. In our Magnetite and Selenite build systems we likewise +generate these *.cfg headers; some of the symbols defined therein are variable +and originate from Bourne shell variables in our own configuration system, but +many others are fixed. See scripts/cfg-template in our Magnetite and Selenite +trees for the magic. + +The BOARD symbol is always fixed at 41 in all FreeCalypso firmwares, +corresponding to TI's D-Sample and Leonardo, and we use our own different +mechanism to distinguish among our supported targets. The solution adopted in +Magnetite and Selenite is as follows: we are supplementing TI's *.cfg files +with our own fc-target.cfg (included as #include "fc-target.cfg" or as +#include "config/fc-target.cfg" matching whatever existing TI code we are +gently extending), and this fc-target.cfg header is populated by the build +system by copying the appropriate targets/*.h header file. These targets/*.h +header snippets define C preprocessor symbols of our own invention like +CONFIG_TARGET_xxx, and whenever we need to know our target in C code, we +#include "fc-target.cfg" and use #ifdef logic based on these preprocessor +symbols of our own addition. + +RVTMUX debug and development interface +====================================== + +The Calypso chip has two UARTs, and TI's TCS211 firmware and its predecessors +are designed with the assumption that both of these UARTs are available. Per +TI's fw architecture, Calypso's MODEM UART presents the standard AT command +interface with GSM 07.10 MUX, CSD, fax and GPRS capabilities as described +earlier when we looked at ACI and ATI, whereas the other UART (called the IrDA +UART in hardware docs but not used for that purpose) presents a vitally +important debug, development and production interface called RVTMUX. This +RVTMUX interface can also be moved to the MODEM UART, in which case the standard +AT command interface is lost. + +RVTMUX is a binary packet interface, and it got its name because it is a MUX of +multiple logical channels managed by the RiViera Trace (RVT) firmware component. +RVTMUX is often thought of as being primarily a debug trace interface, as that +is the primary use to which it is put: in normal operation the firmware emits +quite voluminous debug trace output on the IrDA UART, encapsulated in 3 +different RVTMUX channels as explained below. However, it is also possible to +send a number of different debug and development commands to the firmware via +this interface, and this functionality is used as a critical component in +Calypso GSM device factory production line processes: this RVTMUX interface is +the only way by which the FFS can be initialized, RF calibration and tests can +be performed and the IMEI can be set at the factory. + +Communication with a running firmware over this RVTMUX interface in a +development or production setting (whether passively reading debug traces or +actively sending development or test commands to the running fw) requires +specialized host tools. TI originally had a suite of Windows-based tools for +this purpose, but we are not using them in FreeCalypso: we only got Windows +binaries without any sources, and even in the case of those binaries we only +got an incomplete set with some important tools missing. Instead we are using +our own Unix-based tools called FreeCalypso host tools; these tools have been +developed from scratch by Mother Mychaela after studying the firmware components +with which they need to communicate. + +Debug trace output +================== + +The firmware component that "owns" the physical UART channel assigned to RVTMUX +is RVT, contained in chipsetsw/riviera/rvt in TCS211 or in src/cs/riviera/rvt +in our Magnetite and Selenite firmwares. It is a Riviera-based component, +and it has a Nucleus task that is created and started through Riviera. All +calls to the actual driver for the UART are made from RVT. In the case of +output from the Calypso GSM device to an external host, all such output is +performed in the context of RVT's Nucleus task; this task drains RVT's message +queue and emits the content of allocated buffers posted to it, freeing them +afterward. (The dynamic memory allocation system in this case is Riviera's, +which is susceptible to fragmentation - see discussion earlier in this article.) +Therefore, every trace or other output packet emitted from a GSM device running +our fw (or any of the proprietary firmwares based on the same architecture) +appears as a result of a message in a dynamically allocated buffer having been +posted to RVT's queue. + +RVT exports several API functions that are intended to be called from other +tasks, it is by way of these functions that most output is submitted to RVT. +One can call rvt_send_trace_cpy() with a fully prepared output message, and +that function will allocate a buffer from Riviera's dynamic memory allocator +properly accounted to RVT, fill it and post it to the RVT task's queue. +Alternatively, one can call rvt_mem_alloc() to allocate the buffer, fill it in +and then pass it to rvt_send_trace_no_cpy(). + +At higher levels, there are a total of 3 kinds of debug traces that can be +emitted: + +* Riviera traces: these are generated by various components implemented in + Riviera land, although in reality any component can generate a trace of this + form by calling rvf_send_trace() - this function can be called from any task. + +* L1 traces: L1 has its own trace facility implemented in + src/cs/layer1/cfile/l1_trace.c; it generates its traces as ASCII messages and + sends them out via rvt_send_trace_cpy(). + +* GPF traces: code that runs in GPF/G23M land and uses those header files and + coding conventions etc can emit traces through GPF. GPF's trace functions + (implemented in gpf/frame/vsi_trc.c) allocate a memory partition from + GPF's TEST pool, format the trace into it, and send the trace primitive to + GPF's special test interface task. That task receives trace and other GPF + test interface primitives on its queue, performs some manipulations on them, + and ultimately generates RVT trace output, i.e., a new dynamic memory buffer + is allocated in the Riviera land, the trace is copied there, and the Riviera + buffer goes to the RVT task for the actual output. + +Trace masking +============= + +The RV trace facility invoked via rvf_send_trace() has a crude masking ability, +but by default all traces are enabled. In TI's standard firmwares most of the +trace output comes from L1: L1's trace output is very voluminous, and most of +it is fully enabled by default. + +On the other hand, GPF and therefore G23M traces are mostly disabled by default. +One can turn the trace verbosity level from any GPF-based entity up or down by +sending a "system primitive" command to the running fw, and another such command +can be used to save these masks in FFS, so that they will be restored on the +next boot cycle and be effective at the earliest possible time. Enabling *all* +GPF trace output for all entities is generally not useful though, as it is so +verbose that a developer trying to make sense of it will likely drown in it - +and it will also overwhelm the debug trace facility itself, causing most of +these far too voluminous traces to be lost. Therefore, a developer seeking to +debug an issue in the G23M protocol stack needs to enable traces very +judiciously. + +GPF compressed trace hack +========================= + +TI's Windows-based GSM firmware build systems include a hack called str2ind. +Seeking to reduce the fw image size by eliminating trace ASCII strings from it, +and seeking to reduce the load on the RVTMUX serial interface by eliminating +the transmission time of these strings, they passed their sources through an +ad hoc preprocessor that replaces these ASCII strings with numeric indices. +The compilation process with this str2ind hack becomes very messy: each source +file is first passed through the C preprocessor, then the intermediate form is +passed through str2ind, and finally the de-string-ified form is compiled, with +the compiler being told not to run the C preprocessor again. + +TI's str2ind tool maintains a table of correspondence between the original trace +ASCII strings and the indices they've been turned into, and a copy of this table +becomes essential for making sense of GPF trace output: the firmware now emits +only numeric indices which are useless without this str2ind.tab mapping table. + +Our FC Magnetite build system retains the option of using str2ind, but it is +disabled by default: str2ind significantly increases firmware compilation times, +the resulting fw image sizes without str2ind are fine (the slight increase does +not push us over any limits), and we haven't had any issues with ASCII strings +overloading the trace interface. However, there is an additional complication +stemming from the choice of two possible G23M PS versions, one of which is a +set of blob libraries: + +* If Magnetite is compiled in a pure TCS211 configuration using the original + blob version of G23M PS, these blobs already have str2ind indices baked into + them instead of trace ASCII strings, hence the frozen str2ind.tab file from + Openmoko that maps these indices back to strings needs to be used. + +* If Magnetite is compiled in a TCS2/TCS3 hybrid config without G23M blobs, + then unless you enable it explicitly with USE_STR2IND=1, no str2ind will be + used at all. + +Our blob-free FC Selenite firmware does not support str2ind at all - we shall +stick with full ASCII string traces until and unless we run into an actual (as +opposed to hypothetical) problem with either fw image size or serial interface +load. + +RVTMUX command input +==================== + +RVTMUX is not just debug trace output: it is also possible for an external host +to send commands to the running fw via RVTMUX. + +Inside the fw RVTMUX input is handled by the RVT entity by way of a Nucleus +HISR. This HISR gets triggered when Rx bytes arrive at the designated UART, +and it calls the UART driver to collect the input. RVT code running in this +HISR parses the message structure and figures out which fw component the +incoming message is addressed to. Any fw component can register to receive +RVTMUX packets, and provides a callback function with this registration; this +callback function is called in the context of the HISR. + +In the original TCS211 fw there are only two components that register to receive +external host commands via RVTMUX: ETM and GPF, hence these are the only command +packet types that can be sent to this original fw. In FreeCalypso we have kept +these, and we've also added some new RVTMUX channels of our own invention. + +Test Mode (TM) and Enhanced Test Mode (ETM) +=========================================== + +A major use of the RVTMUX interface is sending so-called Test Mode commands +from an external host to a running GSM device. Depending on the firmware +version, a GSM device can be commanded to do any of the following things +through this mechanism: + +* Exercise RF test modes, e.g., transmit continuously at a set frequency and + power level; +* Read and write arbitrary memory locations in the Calypso ARM7 address space; +* Read and write ABB chip registers; +* Reboot or power off; +* Access and manipulate the device's flash file system (FFS). + +In the segment of history of interest to us TI has produced two different +target firmware components that can receive, interpret and act upon Test Mode +command packets: + +* The original Test Mode component of Layer 1, called L1TM or TML1: this + component handles all RF test modes (needed for RF calibration on device + production lines), and originally it also implemented memory and ABB register + read and write commands, and provided access to TMFFS1 (see below). In the + original implementation this component registered itself as the handler for + the "TM" RVTMUX channel (RVT packet type 0x14), so it would receive all TM + packets sent to the device. + +* Enhanced Test Mode (ETM) is a later invention. It registers itself (instead + of the old TM in L1) with RVT as the handler for the "TM" RVTMUX channel, and + then provides a registration service of its own, such that various components + in the fw suite can register to receive external command packets passing + first through RVT, then through ETM, and can send responses passing through + ETM, then through RVT back to the external host. If a given fw version + contains both ETM and L1TM like TCS211 does, then L1TM registers itself with + ETM; an external host would send exactly the same binary command packets to + exercise RF test modes, but inside the firmware they now pass through ETM on + their way to L1TM. + +The ETM_CORE module contained within ETM itself provides some low-level debug +commands: by sending the right binary command packets to the GSM device via the +RVTMUX serial channel, an external host can examine or modify any memory +location and any hardware register, cause the device to reset, etc. Prior to +ETM some of these functions (but not all) could be exercised through older TM3 +commands, but in FreeCalypso we became familiar with the ETM versions of these +commands long before the older ones because we got the ETM component in full +source form, whereas the sole surviving copy of TCS211 that serves as our golden +reference came with L1TM in binary object form like the rest of L1, and we got +to source-reconstructing it only much later. + +ETM is implemented as a Riviera SWE and has its own Nucleus task; the callback +function that gets called from the RVT HISR posts received messages onto ETM's +own queue drained by its task. The ETM task gets scheduled, picks up the +command posted to its queue, executes it, and sends a response message back to +the external host through RVT. + +Because all ETM commands funnel through ETM's queue and task, and that task +won't start looking at a new command until it finished handling the previous +one, all ETM commands and responses are in strict lock-step: it is not possible +to send two commands and have their responses come in out of order, and it makes +no sense to send another ETM command prior to receiving the response to the +previous one. (But there can still be debug traces or other traffic intermixed +on RVTMUX in between an ETM command and the corresponding response!) + +L1TM commands get posted to the message queue of the L1A task and then executed +in that task's context. + +FFS access via TM/ETM +===================== + +One of the essential facilities provided in one form or another in all known +incarnations of the Test Mode mechanism (at least in TI's original architecture, +as opposed to Motorola's bastardized version) is the ability to access and +manipulate the GSM device's flash file system (FFS) that was described earlier +in this article. TI's TMFFS1 and TMFFS2 protocols provide a command and +response packet interface to the FFS API functions inside the fw, and enable an +external host connected to the GSM device via the RVTMUX channel to perform +arbitrary read and write operations on the device file system. + +In the segment of history of interest to us TI has produced two different +and entirely incompatible versions of the TMFFS protocol: TMFFS1 and TMFFS2. +Or rather, what is now called TMFFS1 was originally just TMFFS, and then came +TMFFS2. TMFFS2 works only through ETM, whereas TMFFS1 predates ETM: in the +original implementation the tm_ffs() function in the FFS code was called from +L1TM code. + +Our copy of TCS211 reference fw includes the source for both TMFFS1 and TMFFS2; +it is theoretically possible to build a firmware image that includes both TMFFS +versions (they won't conflict because they respond to different command +packets), but it is pretty clear that TI never intended to have both enabled +at the same time. Our copy of TCS211 came with TMFFS1 enabled and we didn't +change it when we made the moko12 (leo2moko-r1) fw release for the Openmoko +community (the previous proprietary mokoN firmwares also implement TMFFS1), +but we have subsequently switched to TMFFS2 for our current Magnetite and +Selenite firmwares. + +Our choice of TMFFS2 over TMFFS1 was driven by the need to develop our own host +tools to replace TI's original ones which we never got. We needed to develop +our own host tools for operating on GSM device FFS via one of the two TMFFS +protocols, and after studying the fw source implementing both, I (Mother +Mychaela) came to the conclusion that TMFFS2 is both more capable and more +reliable; my guess is that TMFFS1 was likely kept around only because some of +TI's crappy Weendoze host software depended on it. (See the implementation +code in chipsetsw/drivers/drv_app/ffs/board/tmffs.c in TCS211 if you would like +to judge for yourself.) Our host tool that speaks the TMFFS2 protocol is +fc-fsio. + +GPF external command input +========================== + +The other component that can receive external commands is GPF. GPF's test +interface can receive so-called "system primitives", which are ASCII string +commands parsed and acted upon by GPF, and also binary protocol stack +primitives. Remember how all entities in the G23M stack communicate by sending +messages to each other? Well, GPF's test interface allows such messages to be +injected externally as well, directed to any entity in the running fw. System +primitive commands can also be used to cause entities to send their outgoing +primitives to the test interface, either instead of or in addition to the +originally intended recipient. + +AT commands over RVTMUX +======================= + +There is one more use to which we put the RVTMUX debug serial interface that is +an original FreeCalypso invention: communicating with the AT command interpreter +(ATI). TI's original architecture assumes that if a product is to offer a +standard AT command interface (the product is either a GSM/GPRS modem for which +this AT command interface is the sole mode of usage or a feature phone that +offers a data port as one of its features), then it will be presented on a +dedicated UART separate from RVTMUX. + +However, in the case of our FreeCalypso family of projects about 2 years had +passed between our first functional GSM fw attempts in 2015 and us successfully +building our own development board in 2017; during this time we had to work on +various crippled pre-existing Calypso devices, and many of them had only one +UART practically accessible. In response to this situation we developed a way +to pass AT commands over RVTMUX. We created a new RVTMUX channel for this +interface and assigned it RVT packet type 0x1A. Packets sent from an external +host to the GSM device carry AT commands and SMS string input, whereas packets +flowing the other way carry ATI's responses to commands and asynchronous +notifications such as incoming calls. The host utility for talking AT commands +to a FreeCalypso GSM device via RVTMUX is fc-shell, described below. + +Now that we have built a proper FreeCalypso development board with two UARTs, +the use of this AT-over-RVTMUX hack is deprecated for general usage: this hack +does not support any data services (CSD or GPRS), and even for SMS it is +crippled because maximum-length messages cannot be sent in the more capable PDU +mode. However, it still comes in handy during certain casual testing sessions, +and it is required if one needs to run our FreeCalypso firmware on Mot C1xx or +Pirelli DP-L10 hardware. + +FC host tools for talking to firmwares via RVTMUX +================================================= + +The fundamental tool for talking to running firmwares via RVTMUX is a program +called rvinterf. It runs on a Unix/Linux host machine, opens a serial port +that is expected to be connected to the RVTMUX UART on the target, and then +speaks TI's binary packet protocol on that serial port. It then performs two +functions: + +* If rvinterf is run in the foreground in a terminal window (or more precisely, + if its default terminal output is not disabled), every packet received from + the target is decoded and printed on stdout in human-readable ASCII. For + some packets like TM/ETM responses this "human-readable" form is just a hex + dump, but the trace messages which the firmware emits on its own are printed + in truly human-readable form. This output can also be saved to a log file. + +* Rvinterf creates a local UNIX domain socket on the machine it is running on, + and other host tools can then connect to this socket to exchange packets with + the firmware. Client programs connected to rvinterf via this local socket + interface can register to receive copies of packets sent by the target on + specific RVTMUX channels, and they can also send arbitrary packets to the + target. + +Our main "client" programs for actively interacting with running firmwares via +rvinterf are: + +fc-tmsh This utility speaks the TM/ETM protocol. It supports almost + all ETM and L1TM commands that are supported by our reference + TCS211 fw with the important exception of TMFFS; support means + that fc-tmsh can issue these commands and decode the firmware's + responses to them. fc-tmsh operates asynchronously in that the + issuance of commands to the target and the display of firmware + responses are completely decoupled; this asynchronous model is + a good match for L1/RF test mode commands and simple ETM + operations, but is a poor fit for FFS manipulation. fc-tmsh's + companion fc-fsio implements FFS access via TMFFS2, and we + don't have a host side implementation for TI's older TMFFS1 + protocol. + +fc-fsio This utility speaks the TMFFS2 protocol over the TM/ETM RVTMUX + channel (same channel as used by fc-tmsh, so don't try to run + both at the same time) and implements fairly high-level FFS read + and write operations. fc-fsio is used to format and initialize + the FFS on newly made devices in our hardware manufacturing + environment, it can upload files or entire subtrees into target + device FFS, it has higher-level commands for writing some files + like the IMEI, rfcap and AT+CGxx ID strings, and it can list and + read out FFS content. Unlike fc-tmsh, fc-fsio is synchronous: + it is built on command-response (send a command and expect a + response) primitives, and a single user command can turn into a + large number of command-response exchanges on the RVTMUX + interface. fc-fsio also implements a few non-FFS commands + because they naturally fit into this ETM synchronous model. + +fc-shell This tool is asynchronous like fc-tmsh, but instead of talking + and listening on the TM/ETM RVTMUX channel, it talks and listens + on GPF's channel and on the new AT-over-RVTMUX channel which we + added in FreeCalypso. fc-shell can be used to issue system + primitive commands to GPF (and to see firmware responses to + them), and to talk AT commands via RVTMUX. + +Finally, if you only need to passively observe the firmware's debug trace output +and don't need to make any active pokes at the target, our rvtdump utility is a +stripped-down version of rvinterf (or historically its predecessor) that only +decodes and prints/logs the output from the target without sending anything to +it. + +Further reading +=============== + +Believe it or not, some of the documentation that was written by the original +vendors of the software in question and which we've been able to locate turns +out to be fairly relevant and helpful, such that I recommend reading it. + +Documentation for Nucleus PLUS RTOS: + + ftp://ftp.freecalypso.org/pub/embedded/Nucleus/nucleus_manuals.tar.bz2 + + Quite informative, and fits our version of Nucleus just fine. + +Riviera environment: + + ftp://ftp.freecalypso.org/pub/GSM/Calypso/riviera_preso.pdf + + It's in slide presentation form, not a detailed technical document, but + it covers a lot of points, and all that Riviera stuff described in the + preso *is* present in our fw for real, hence it should be considered + relevant. + +GPF documentation: + + https://www.freecalypso.org/LoCosto-docs/SW%20doc/frame_users_guide.pdf + https://www.freecalypso.org/LoCosto-docs/SW%20doc/vsipei_api.pdf + + Very good reading, helped me understand GPF when I first reached this + part of firmware reintegration. + +TCS3.x/LoCosto fw architecture: + + https://www.freecalypso.org/LoCosto-docs/SW%20doc/TCS2_1_to_3_2_Migration_v0_8.pdf + ftp://ftp.freecalypso.org/pub/GSM/LoCosto/LoCosto_Software_Architecture_Specification_Document.pdf + + These TI docs focus mostly on how they changed the fw architecture from + their TCS2.x program (Calypso) to their newer TCS3.x (LoCosto), but one + can still get a little insight into the "old" TCS211 architecture they + were moving away from, which is the architecture we've adopted for + FreeCalypso.