freecalypso-docs: TCS211-fw-arch comparison

comparison TCS211-fw-arch @ 0:fcd1cf531017

TCS211-fw-arch masterpiece written

author	Mychaela Falconia <falcon@freecalypso.org>
date	Mon, 08 Oct 2018 19:52:50 +0000
parents
children	5de1f72ce941

comparison

equal deleted inserted replaced

--1:000000000000
+:fcd1cf531017
+This document describes the architecture of TI's TCS211 firmware and that of
+our FreeCalypso Magnetite and Selenite firmwares which are based on it.
+What is TCS211, and why we use it as our reference
+==================================================
+TI were in the business of making GSM baseband chipsets for about a decade
+from the late 1990s up until 2009, and over that time span both their silicon
+and their firmware architecture had evolved in many different ways.  All of our
+work in the FreeCalypso family of projects is based on one fairly arbitrary
+snapshot, a rather arbitrarily picked single point in that long evolutionary
+line: we use the Calypso chipset as opposed to both the ones before and the
+ones after, and we use TI's TCS211 firmware from 2007 as our golden reference,
+as opposed to other equally valid ways of architecturing the fw that came
+before and after our arbitrarily picked snapshot.
+Q: Why do we use the Calypso chipset as opposed to LoCosto or E-Costo or
+whatever was TI's very last offering before they got out of that business?
+A: Because that's what Openmoko used: their Neo FreeRunner aka GTA02 smartphones
+were our primary hardware target for many years before we gathered the money
+and the courage to build our own board-level hardware starting from just chips
+bought on the Chinese surplus market.
+Q: Why do we use TI's TCS211 firmware from 2007 and its architecture as our
+golden reference, as opposed to any of the other infinitely many equally valid
+ways of architecturing a working firmware implementation for the same Calypso
+chipset?
+A: Because it works flawlessly, and is extremely stable as a commercial product.
+The firmware which Openmoko got from TI had only a tiny difference from TI's
+internal TCS211 mainline (TSPACT signal definitions in tpudrv12.h which are
+different between the quadband RFFE on TI's internal reference hw and the
+triband one in FIC's commercial implementation), and with only a few additional
+changes related to our use of a newer flash chip that wasn't supported back in
+TI's and Openmoko's days, this golden reference fw can run equally well on our
+own FCDEV3B.
+Relation between TCS211 and FreeCalypso
+=======================================
+The only "pure" TCS211 firmware we got is the one that has been salvaged from
+the ruins of Openmoko.  To the best of our knowledge, it is the world's only
+surviving copy of any version of TCS211 - it is entirely possible that even TI
+may not have it any more in any of their archives, given the length of time that
+has passed and the total lack of interest in this "ancient junk".  In its pure
+form, this world's sole surviving copy of TI's TCS211 fw is laden with blobs
+(many components exist only as binary object libraries with no corresponding
+source), and it features a build system that is very thoroughly Windows-based.
+And to top it off, that configuration and build system has many critical
+components which also exist only as compiled binaries (Windows executables or
+Java bytecode) with no corresponding source.
+We started by replacing the original configuration and build system of TCS211
+with our own one that is Unix-based rather than Windows-based, and implemented
+in Bourne shell with a few C helpers instead of XML, Java and Perl.  The result
+was named FreeCalypso Magnetite.  At first we changed only the configuration
+and build system, but kept all of the original TCS211 code, including all of
+the binary-only components.  Then we deblobbed it gradually, replacing binary-
+only components with source, one component at a time.  Where did we get the
+source for the pieces that came as binary objects with no corresponding source?
+The answer is different for different components:
+* For GSM Layer 1 (a very critical and highly chipset-dependent component), we
+did a painstaking reconstruction which you can see in the tcs211-l1-reconst
+repository.  That world's last surviving copy of TCS211 which we got only had
+*.c files censored out, while all of the original *.h files were preserved -
+and thanks to the preserved configuration and build system, we also got all
+of the original compilation lines including compiler options, -D definitions
+and -I include paths.  For most of the missing *.c files we got a "wrong"
+version from the TCS3/LoCosto source.  The reconstruction proceeded by taking
+these "wrong version" *.c files, putting them one module (one *.c file) at a
+time into the TCS211 build environment, and massaging each individual *.c
+file until it compiled into a perfect match to the original binary object.
+Thus we have reconstructed a full C source for the L1 component which for all
+practical purposes can be treated as if it were the lost original source.
+* For some small pieces like the tpudrv12 RF driver and the OSL and OSX
+components of GPF it was more of a translation from disassembly to C: the C
+code we use is of our own writing, but it faithfully matches the logic
+implemented by the original blobs as recovered through disassembly.
+* The G23M protocol stack is a very large and complex component, and our copy
+of TCS211 (the world's only surviving copy to the best of our knowledge) has
+it in binary-only form.  Trying to source-reconstruct it precisely like we
+did with L1 would have been infeasible, hence we took a different approach:
+we put together a TCS2/TCS3 hybrid in which we made a wholesale replacement
+of all G23M components: we adopted the new version of G23M wholesale without
+trying to recreate the old version.
+* Both TCS211 and TI's newer TCS3.2 fw for the LoCosto chipset are based on
+Nucleus PLUS RTOS (different versions), and both firmwares have their Nucleus
+only as binary object libraries, no source.  However, we got another version
+of Nucleus from about the same time frame (slightly newer than the one TI used
+in TCS211, but slightly older than the one in TCS3.2) from a non-TI source
+(it was posted on a Russian web forum by Comrade XVilka), and in FreeCalypso
+Selenite we use this new Nucleus as a replacement for TCS211 original version
+in the same manner as how we had earlier made a wholesale replacement of the
+G23M protocol stack.
+With two major components (Nucleus and the G23M PS) replaced with non-TCS211
+versions, our Magnetite hybrid and Selenite firmwares are no longer TCS211, but
+they still faithfully follow the _architecture_ of TCS211: in each case when we
+replaced the code, we made the new code version fit perfectly into the original
+architecture without any disruptive changes.  Thus anyone who desires to
+understand our current FreeCalypso firmwares (Magnetite and Selenite) needs to
+first understand the original TCS211 architecture, as it is essentially
+unchanged.
+Why not use the LoCosto chipset and its TCS3.2 firmware?
+========================================================
+We went the Calypso route and not the LoCosto route because of the circumstances
+that surrounded the beginning of our family of projects.  We did not get all of
+the tools needed for working with LoCosto chips and TI's TCS3.2 fw (CSST and
+SBuild) until the spring of 2015, and by that time we had invested too much into
+the Calypso to throw it all away and restart anew in the uncharted waters of
+LoCosto.  Another factor is that the software for talking to LoCosto's ROM
+bootloader (CSST) exists only as Windows binaries sans source, and it would
+require some effort to reverse-engineer the protocol and implement a free and
+Unix-based alternative - whereas for the Calypso this work was already done by
+OsmocomBB folks before we entered the scene.  Finally, in the case of the
+Calypso we have read out the actual content of the ROMs (both the ARM boot ROM
+and the DSP ROM) and the ARM boot ROM code has been disassembled and thoroughly
+understood - whereas in the case of LoCosto it is not certain if we can even
+read out the ROM content, as it is said to be protected against reading.
+If someone else desires to play with LoCosto, either by hacking a Peek device
+or by building an I-Sample board from the available PADS PCB file, go for it!
+But the FreeCalypso core team is sticking with the Calypso chipset for now, and
+our actively maintained Magnetite and Selenite firmwares follow the architecture
+of TCS211, not that of TCS3.2.
+Relation between the ARM and DSP cores in the Calypso
+=====================================================
+The Calypso digital baseband processor chip has two processor cores in it: an
+ARM7TDMI core that runs the main firmware and a C54x DSP core that performs the
+more burdensome signal processing tasks.  The DSP is subservient to the ARM:
+only the ARM comes out of reset and starts executing code upon power-up, while
+the DSP is held in reset (does not run) until and unless the ARM firmware starts
+it running.
+The ARM core executes code from outside of the Calypso chip itself: in normal
+operation (outside of development) there is a flash memory chip connected to
+Calypso's external memory bus, and the Calypso's ARM core executes firmware
+stored in this flash.  There is an optional (enabled or disabled by a hardware
+pin) ARM boot ROM inside the Calypso chip; when this boot ROM is enabled by
+nIBOOT pin strapping on the board (like it is on Openmoko and FreeCalypso
+hardware), the ARM core executes code from this boot ROM first upon power-up or
+reset before jumping to external flash.  The tiny piece of code that is hard-
+cast in this mask ROM acts as an unbricking aid: it gives a certain time window
+during which the boot process can be interrupted and diverted if certain magic
+characters are sent into either of Calypso's two UARTs by an external
+development host, and if nothing is received on either UART during that time
+window (as would be the case in normal usage of a Calypso phone or modem), the
+boot ROM transfers control to the firmware image in the external flash.  The
+end result is that the ARM core always runs code from outside of the Calypso
+chip itself, either the firmware image in the flash or whatever code is fed by
+an external development host to the boot ROM serially over a UART.
+There is also an internal RAM inside the Calypso from which the ARM can execute
+code (512 KiB on the full Calypso version or 256 KiB on Calypso Lite silicon
+used in some historical low-end phones); the primary purpose of this internal
+RAM is to allow chosen sections of code to execute faster without the
+performance penalty of the external memory bus, but it is volatile RAM, not ROM
+or flash, hence it doesn't have any code in it until and unless loaded by the
+firmware copying code from flash or via the serial boot protocol.
+In contrast, the DSP is very different.  The DSP core can never execute any
+code from outside the chip, and has no access to the Calypso chip's external
+memory bus at all.  Instead the only two memories accessible to the DSP are a
+mask ROM and a fast internal RAM.  The DSP's dedicated mask ROM is 128 Kwords;
+the DSP's RAM is 28 Kwords, out of which 8 Kwords constitute the so-called API
+RAM which is accessible to both ARM and DSP cores.  (The C54x DSP addresses
+memory by words instead of bytes, hence the memory sizes are given in Kwords
+instead of KiB.)
+The main bulk of the DSP's operating program is already hard-cast in the silicon
+in the 128 Kword mask ROM.  The DSP ROM code is structured in such a way that
+any part of it can be overridden by downloadable patch codes which get loaded
+somewhere in the DSP's 28 Kword RAM, but because the RAM is significantly
+smaller than the ROM, downloadable DSP code cannot replace the entirety of the
+ROM code - instead the code needs to be patched very selectively only where
+necessary to fix a bug that was discovered after the silicon was made or to
+extend the DSP functionality with a new feature.
+The DSP ROM code in the Calypso silicon we are using has been successfully read
+out, but it is only the executable binary code and data - we never found a copy
+of the source for this DSP ROM code.  And even if we had this source, we would
+not be able to casually modify and recompile it without spending millions of
+dollars to fab a new chip revision with a modified mask ROM.  Having this source
+would allow us to develop our own DSP patch codes and to understand and maintain
+the existing ones, hence we need to make an effort to convince TI to release
+the source for the DSP ROM if they have it in their archives, but if no
+surviving copy of this source exists anywhere in the world, the fallback plan
+would be to reverse-engineer the DSP ROM code by disassembly.  The latter plan
+has not been pursued yet because of the very high labor cost it would involve.
+It is possible to run the Calypso DSP without any patches, i.e., have it run
+only the code that is already in the mask ROM.  Our competitor OsmocomBB
+operates in this manner, and we have also built and run modified versions of
+our TCS211-based FreeCalypso firmware with DSP patch loading disabled as an
+experiment.  However, all ARM-side firmwares that have been officially released
+by TI for production use including our TCS211-20070608 golden reference do apply
+downloadable patches to the DSP, and are designed to run with this patched DSP;
+running them with DSP patching disabled results in unstable operation.
+DSP patch codes that are included in ARM-side Calypso firmwares take the form
+of const char arrays initialized with hex bytes; these C source files with hex
+char arrays inside were apparently produced from C54x COFF files with a tool
+called coff2c, but we never got any of those COFF files or whatever source (C
+or assembly) they were built from.  At the present time in the FreeCalypso
+family of projects we use the DSP patch codes (hex char arrays) which we got
+with our copy of TCS211 from 20070608, and we treat the entire DSP block (the
+combination of mask ROM plus patches) as a functional black box.
+Having to treat the DSP as a black box is certainly a major shortcoming of our
+FreeCalypso solution.  However, I (Mother Mychaela) would much rather have a
+phone or modem in which only the DSP is a black box while I get to maintain all
+of the upper layers with full freedom, as opposed to the status quo alternative
+of a very high-level black box with FOTA backdoors.  Unlike the ubiquitous
+high-level black boxes from the likes of Qualcomm, the DSP in the Calypso cannot
+be backdoored: it has no access to the ARM address space, thus no access to the
+flash (cannot surreptitiously modify the firmware) and no access to any of the
+higher-level radio protocol state maintained by the ARM, all it can do is
+modulate and demodulate bursts and run voice codecs _as commanded by the ARM_.
+Furthermore, the DSP has no access to the Calypso chip's TPU (Time Processing
+Unit, the block that controls board-level RF hardware) and thus has no direct
+control over any of the RF hardware: it cannot initiate radio transmission or
+even reception on its own, instead the ARM firmware has to configure the RF
+hardware via the TPU for each and every Rx or Tx time window.
+Finally, if anyone is truly paranoid about the possibility of backdoors in the
+DSP, the DSP ROM code has been read out - you are welcome to hire a professional
+reverser of your choice to disassemble and audit it as thoroughly as you like.
+This code is unchangeable by virtue of being hard-cast in a mask ROM in the
+silicon.
+The rest of this document covers the firmware that runs on the ARM core; it
+controls the DSP via its API RAM, a form of shared memory interface.
+High-level structure of TCS211 firmware
+=======================================
+The code base that makes up TI's TCS211 firmware consists of 3 main divisions:
+chipset software, Condat G23M (GSM and GPRS L23 protocol stacks, ACI and
+optional handset UI layers) and GPF.  Let us look at them in turn:
+chipsetsw division
+------------------
+In the original TCS211 delivery there was a top-level directory named chipsetsw
+(chipset software), containing code that is specific to TI's chipsets in
+particular and was never intended to run on any other hardware.  This code
+division has been retained intact in our FreeCalypso Magnetite and Selenite
+firmwares, taken in its entirety from our TCS211 golden reference, although we
+have shortened the name: this code division now resides under src/cs in
+Magnetite and Selenite.  Aside from a few bits of system glue, this chipsetsw
+breaks down into two further subdivisions: the L1+drivers core and the SSA
+division.
+L1+drivers core
+---------------
+This division resides under chipsetsw/layer1 and chipsetsw/drivers/drv_core, or
+under src/cs/layer1 and src/cs/drivers/drv_core in our version.  The most
+important piece here is L1 (GSM Layer 1): this code drives the DSP and the RF
+hardware, and thereby makes the Calypso function as a GSM MS (mobile station)
+and not merely as a general purpose microprocessor platform.  This code can be
+considered to be the most important part of the entire firmware.
+At one time TI had a so-called standalone L1 configuration, selected by the
+OP_L1_STANDALONE C preprocessor symbol.  We don't have the bits that are needed
+to build this configuration (they were probably never released outside of TI at
+all), but it appears that this fw build configuration consisted of just Nucleus,
+L1, the drivers under drv_core, the OSL and OSX parts of GPF without the rest,
+and some stubs for the few higher-level functions that are intertied with L1.
+The drivers under chipsetsw/drivers are divided into drv_core and drv_app: the
+former are the most essential or fundamental ones, used by L1 and/or needed for
+the OP_L1_STANDALONE config; the latter belong to the higher-level SSA division
+described below.
+SSA division
+------------
+TI had a group called System Software and Applications (SSA), and they supplied
+those parts of the firmware that are neither L1+drv_core nor Condat G23M.  The
+more interesting pieces here include the flash file system (FFS), the debug
+trace facility (RVT), the Enhanced Test Mode (ETM) facility that allows
+external development and production tools to poke at the firmware, RiViera Audio
+Service (playing various beeps and ringtones through the DSP, a front-end to L1
+audio functions), LCD and keypad drivers for Calypso-based handsets, and various
+supportive functions implemented via the Iota ABB: switch-on and switch-off
+logic, battery monitoring and charging, backlight LED control.
+All firmware components in the SSA division are built on top of a framework
+called RiViera - more will be said about it later.  Everything under
+chipsetsw/drivers/drv_app, chipsetsw/riviera and chipsetsw/services (or under
+src/cs/drivers/drv_app, src/cs/riviera and src/cs/services in our version)
+belongs to the SSA realm.
+Condat G23M division
+--------------------
+At the beginning of TI's involvement in the GSM baseband chipset business, they
+only developed and maintained their own L1 code, which eventually grew into the
+larger chipsetsw division described above, while the rest of the protocol stack
+(which is hardware-independent) was licensed from another company called Condat.
+Later Condat as a company was fully acquired by TI, and the once-customer of
+this code became its owner.  The name of TI/Condat's implementation of GSM
+layers 2&3 for the MS side is G23M, and it forms its own major division of the
+overall fw architecture.
+The overall Condat code realm can be further subdivided into GSM and GPRS L23
+protocol stacks, the Application Control Interface (ACI) which includes the AT
+command interpreter (ATI), and additional phone UI layers which are only
+included in handset but not modem firmwares.
+We don't know exactly how TI maintained this software internally: given that it
+is mostly hardware-independent aside from integration details and some minor
+features which may be present on one hw platform but not on another, it would
+have made the most sense for TI to maintain a single internal mainline common
+to both Calypso and LoCosto, and then integrate the code from this mainline into
+chipset-specific customer releases.  We have no way of knowing if TI indeed
+followed this approach or not, but when we took the version of G23M from the
+TCS3.2 source for the LoCosto chipset and grafted it onto the chipsetsw
+foundation from TCS211 for the Calypso to produce our TCS2/TCS3 hybrid, the
+integration went surprisingly smoothly.  The full-source version of G23M which
+we took from TCS3/LoCosto is newer than the binary-only version featured in the
+world's last surviving copy of TCS211 from Openmoko.
+GPF island of stability
+-----------------------
+Underlying the G23M protocol stack is a special layer called GPF, which was
+originally Condat's Generic Protocol stack Framework.  Apparently Condat were
+in the business of developing and maintaining a whole bunch of protocol stacks:
+GSM MS side, GSM network side, TETRA and who knows what else.  GPF was their
+common underpinning for all of their protocol stack projects, which ran on top
+of many different OS environments: Nucleus, pSOS, VxWorks, Unix/Linux, Win32
+and who knows what else.
+In the case of TI/FreeCalypso GSM fw, both the protocol stack and the underlying
+OS environment are fixed: GSM and Nucleus, respectively.  But GPF is still a
+critically important layer in the firmware architecture: in addition to serving
+as the glue between the G23M stack and Nucleus, it provides some important
+support infrastructure for the protocol stack.
+However, what makes GPF very special is the way in which it relates to the rest
+of the firmware architecture.  GPF remained common and unchanged across TI's
+many different projects, and it is so independent from the rest of the firmware
+and its build configuration that TI were able to make company-wide GPF library
+builds and then plop them into multiple fw projects which used them as
+configuration-independent prebuilt libraries.  All TI firmware (semi-)sources
+we've got use GPF in prebuilt library form and are not set up to recompile any
+part of it from source.
+Our FC Magnetite firmware uses the original binary libs from TCS211-Openmoko
+for its GPF component, but for FC Selenite the project requirement is to be
+completely blob-free, hence we had to reconstruct the source for GPF.  The
+original source for most parts of GPF was found between TCS3.2 from Peek/FGW
+and TCS211 from OM (the former had the source for the core "frame" modules and
+the latter had the source for misc and tst), but we never got the source for the
+OSL and OSX components, hence we had to reconstruct them from disassembly.  OSL
+is the glue layer between GPF and Nucleus, OSX is the glue layer between GPF
+and L1.
+Firmware boot process
+=====================
+As already mentioned earlier, the Calypso chip itself includes an ARM boot ROM
+in the silicon that serves as an unbricking aid: it provides a certain time
+window during which the boot process can be interrupted and diverted if certain
+magic characters are sent into either of Calypso's two UARTs by an external
+host, and if nothing is received on either UART during that time window, the
+boot ROM transfers control to the firmware image in the external flash.  As we
+understand it, Calypso was TI's first DBB (digital baseband processor) chip to
+include this boot ROM, and their previous DBB chips did not have such: they
+would always execute code directly from external flash immediately out of reset.
+TI's TCS211 and earlier firmwares are structured in such a way that they boot
+and run exactly the same way whether the Calypso boot ROM is present and
+enabled, present but disabled, or not present at all.  They put magic constant
+0x00000001 in the 32-bit word at flash address 0x2000, which tells the Calypso
+boot ROM (if it is present and enabled) to boot the flash fw image in legacy
+mode: after providing the unbricking time window, the boot ROM moves itself out
+of the way (sets two bits in the FFFF:FB10 register which tell the chip to unmap
+the boot ROM and to map external memory at address 0) and induces a watchdog
+reset, causing the chip to re-execute the reset vector, this time directly out
+of external flash - thus the firmware boots as if the boot ROM weren't there,
+but the ROM's unbricking function is retained.
+In order to make it easier to load new firmware images during development on
+pre-Calypso platforms which didn't have a boot ROM, TI had developed a flash-
+resident bootloader stage and included it in their fw architecture.  This
+bootloader stage is placed at the beginning of the flash at the reset vector,
+and the rest of the firmware begins at an erase unit boundary.  The bootloader
+stage executes first, and before it jumps to the main firmware entry point
+(_INT_Initialize) for normal boot, it offers an opportunity for the boot process
+to be interrupted and diverted if an external host sends certain magic command
+packets into either of the two UARTs during the allotted time window.  If the
+external host does interrupt and divert the boot process in this manner, it can
+feed a code image to the bootloader to be written somewhere in target RAM, and
+then command the bootloader to jump to it.  It is exactly the same functionality
+(though with different serial protocol specifics) as implemented in the Calypso
+boot ROM.  The ROM version is obviously superior because it is unbrickable, but
+the flash-resident, built-with-firmware version is what TI used before they
+came up with the idea of the boot ROM for the Calypso.
+When the boot-ROM-equipped Calypso came along, TI kept the flash-resident
+bootloader in the firmware: it does no harm aside from adding a little bit of
+delay to the boot process, it does not conflict with the ROM bootloader as the
+two speak different serial protocols and respond to different interrupt-boot
+sequences, and it allowed TI to keep the same firmware architecture for
+platforms with and without a boot ROM.  However, in our FreeCalypso firmwares
+starting with Magnetite we have removed this extra bootloader stage for the
+following reasons:
+* It is not useful to us on any of our hardware targets: on those devices that
+have the Calypso boot ROM enabled, we use that boot ROM and get full
+unbrickability, whereas on Mot C1xx phones we have to work with Mot/Compal's
+own different bootloader and serial protocol at least initially, hence it
+makes the most sense to stick with the same after the conversion to
+FreeCalypso as well.
+* As delivered by TI with their full production TCS211 fw releases, their
+firmware-resident bootloader works as intended only on hw platforms with
+13 MHz VCXOs like the original D-Sample (Clara RF), and is broken on platforms
+like Rita RF (the only RF chip for which we have driver code!) with 26 MHz
+VCXOs: there is no conditionally-compiled code anywhere in the bootloader
+code path to set the VCLKOUT_DIV2 bit in the CNTL_CLK register on 26 MHz
+platforms, thus the UARTs are fed with 26 MHz instead of the standard 13 MHz
+clock expected in normal operation, and the intended baud rate of 115200 bps
+turns into 230400.  Because 230400 bps is a baud rate which Calypso UARTs
+*cannot* produce in normal GSM operation (when the peripheral clock network
+runs at the expected 13 MHz), tools that are designed to talk to Calypso GSM
+devices are typically not designed to support this baud rate.  In particular
+for CP2102 USB-serial adapters, the precedent established by the factory
+CP2102 EEPROM programming in the Pirelli DP-L10 phone is that the baud rate
+entry for 230400 bps is replaced with 203125 bps, which is a valid baud rate
+for Calypso UARTs running at 13 MHz.
+* We have no source for TI's firmware-resident bootloader, only linkable binary
+objects that came with our world's last surviving copy of TCS211, which are
+incompatible with our goal of blob-free firmware.
+Because this extra bootloader stage is ultimately unnecessary in our
+environment, the deblobbing goal was easier accomplished by removing it
+altogether instead of expending effort on a blob-free replacement.  Because I
+wasn't comfortable with modifying TMS470 assembly code and linker script magic,
+the removal of the bootloader was accomplished by stubbing out its C body with
+an empty function.  In the gcc-built FC Selenite version it is removed
+completely, without any leftover stubs.
+Finally, it needs to be noted for the sake of completeness that Compal's
+bootloader used on Mot C1xx phones is a modified version based on TI's original
+bootloader.  However, this factoid matters only for historians and genealogists;
+for all practical purposes it is an unrelated animal, as Mot/Compal's serial
+protocol for interrupting and diverting the boot process is their own and bears
+no resemblance to TI's version.  And yes, Mot/Compal's version does set the
+VCLKOUT_DIV2 bit in the CNTL_CLK register to adjust for the 26 MHz clock input
+as its first order of business; it was probably the very first issue they had
+to fix.
+When we build FC Magnetite or FC Selenite TMS470 firmware for Mot C1xx targets,
+we use dd to strip off the first 64 KiB of the image produced by TI's linker
+(the part where TI's bootloader resides, be it intact or stubbed out) and flash
+the remaining image (the main body of the fw) starting at flash address 0x10000.
+In the gcc-built Selenite version we natively link images that are designed to
+be flashed at 0x10000 without any dirty hacks.  Common to all FC firmwares for
+C1xx targets, the bootloader image we put at 0 (in the brickable flash sector)
+is a modified version based on one of Mot/Compal's originals: we have binary-
+patched it to redirect the exception vectors from Mot/Compal's 0x20A0 to 0x10000
+and to move the main fw entry point from Mot/Compal's 0x20F8 to TI's 0x10058.
+None of this muckery applies to our own FreeCalypso hardware or to our
+predecessor Openmoko's hw: on these good hw targets the complete fw image as
+built is flashed at 0, and there is no possibility of bricking because we use
+the boot ROM to gain access irrespective of what's in the flash.
+Main firmware entry point
+-------------------------
+With the bootloader distraction out of the way, the main fw entry point is at
+the _INT_Initialize symbol in the int.s assembly module, located in
+src/cs/system/main/int.s in Magnetite and Selenite.  The functional equivalent
+for the gcc environment in Selenite is in src/cs/system/main/gcc/bootentry.S.
+This assembly code performs some basic hardware initialization, sets up
+sensible memory timings for the boot path phase before DPLL setup, copies the
+IRAM code (the code that is intended to execute out of the fast internal RAM)
+from flash to where it needs to be, zeros both IRAM and XRAM .bss regions, does
+TI's cinit/auto_init business for initialized data in the TMS470 environment
+(Selenite gcc version copies .data from flash to RAM instead), sets up the
+system, IRQ, FIQ and exception stacks, does some assembly initialization for
+Nucleus and finally jumps to Nucleus' C entry point INC_Initialize().
+Further initialization takes place in the Init_Target() and Init_Drivers()
+functions called from Application_Initialize(), which is the last function
+called by INC_Initialize() before starting the Nucleus task scheduler.
+Nucleus environment
+===================
+Like all classic TI firmwares, ours is based on the Nucleus PLUS RTOS.  Just
+like TI's original code on which we are based, we use only a small subset of
+the functionality provided by Nucleus - but because the latter is a library,
+the pieces we don't use simply don't get pulled into the link.  The main
+function we get out of Nucleus is the scheduling of threads, or tasks as
+Nucleus calls them.
+Aside from pre-stack-setup assembly init code and ARM exception handlers, every
+piece of code in the firmware executes in one of the following contexts:
+* Application_Initialize(): this function and everything called from it execute
+just before Nucleus' thread scheduler starts; at this point interrupts are
+disabled at the ARM7 core level (in the CPSR) and must not be enabled; the
+stack is Nucleus' "system stack" which is also used by the scheduler and LISRs
+as explained below.
+* Regular threads or tasks: once Application_Initialize() finishes, all code
+with the exception of interrupt handlers (LISRs and HISRs as explained below)
+runs in the context of some Nucleus task.  Whenever you are trying to debug
+or simply understand some piece of code in the firmware, the first question
+you should ask is "which task does this code execute in?".  Most functional
+components run in their own tasks, i.e., a given piece of code is only
+intended to run within the Nucleus task that belongs to the component in
+question.  On the other hand, some components are implemented as APIs,
+functions to be called from other components: these don't have their own task
+associated with them, and instead they run in the context of whatever task
+they were called from.  Some only get called from one task: for example, the
+"uartfax" driver API calls only get called from the protocol stack's UART
+entity, which is its own task.  Other component API functions like FFS and
+trace can get called from just about any task in the system.  Many components
+have both their own task and some API functions to be called from other tasks,
+and the API functions oftentimes post messages to the task to be worked on by
+the latter; the just-mentioned FFS and trace functions work in this manner.
+In our TCS211-mimicking Magnetite and Selenite firmwares every Nucleus task is
+created either through RiViera or through GPF, and not in any other way - see
+the description of RiViera and GPF below.
+* LISRs (Low level Interrupt Service Routines): these are the interrupt handlers
+that run immediately when an ARM IRQ or FIQ comes in.  The code at the IRQ and
+FIQ vector entry points calls Nucleus' magic stack switching function
+(switches the CPU from IRQ/FIQ into SVC mode, saves the interrupted thread's
+registers on that thread's stack, and switches to the "system" stack) and
+then calls TI's IRQ dispatcher implemented in C.  The latter figures out
+which Calypso interrupt needs to be handled and calls the handler configured
+in the compiled-in table.  Nucleus' LISR registration framework is not used
+by the GSM fw, but these interrupt handlers should be viewed as LISRs
+nonetheless.
+There is one additional difference between canonical Nucleus and TI's version
+(we've replicated the latter): canonical Nucleus was designed to support
+nested LISRs, i.e., IRQs re-enabled in the magic stack switching function,
+but in TI's version which we follow this IRQ re-enabling is removed: each LISR
+runs with interrupts disabled and cannot be interrupted.  (The corner case of
+an FIQ interruping an IRQ remains to be looked at more closely as bugs may be
+hiding there, but Calypso doesn't really use FIQ interrupts.)  There is really
+no need for LISR nesting in our GSM fw, as each LISR is very short: most LISRs
+do nothing more than trigger the corresponding HISR.
+* HISRs (High level Interrupt Service Routines): these hold an intermediate
+place between LISRs and tasks, similar to softirqs in the Linux kernel.  A
+HISR can be activated by a LISR calling NU_Activate_HISR(), and when the LISR
+returns, the HISR will run before the interrupted task (or some higher
+priority task, see below) can resume.  HISRs run with CPU interrupts enabled,
+thus more interrupts can occur, with their LISRs executing and possibly
+triggering other HISRs.  All triggered HISRs must complete and thereby go
+"quiescent" before task scheduling resumes, i.e., all HISRs as a group have a
+higher scheduling priority than tasks.
+Nucleus implements priority scheduling for tasks.  Tasks have their priority set
+when they are created (through RiViera or GPF, see below), and a higher priority
+task will run until it gets blocked waiting for something, at which time lower
+priority tasks will run.  If a lower priority task sends a message to a higher
+priority task, unblocking the latter which was waiting for incoming messages,
+the lower priority task will effectively suspend itself immediately while the
+higher priority task runs to process the message it was sent.
+HISRs oftentimes post messages to their associated tasks as well; if one of
+these messages unblocks a higher priority task, that unblocked task will run
+upon the completion of the HISR instead of the original lower priority task
+that was interrupted by the LISR that triggered the HISR.  Nucleus' scheduler
+is fun!
+RiViera and GPF
+===============
+RiViera and GPF are two parallel/independent/competing wrappers around or
+layers above Nucleus.  GPF comes from Condat and is used by the G23M protocol
+stack and indirectly by L1 (the peculiar way in which L1 ties in with the rest
+of the firmware will be covered later), whereas RiViera is used by the fw
+components from TI's SSA group: flash file system, debug trace, RiViera Audio
+Service and so forth.
+At some point in their post-Calypso TCS3.x program TI decided to eliminate
+RiViera as an independent framework and to reimplement RiViera APIs (used by
+peripheral but necessary code such as FFS, ETM, various drivers etc) over GPF.
+This arrangement is used in the TCS3.2 LoCosto firmware from which we have
+lifted our source replacements for much of the code that came as binary objects
+in our reference TCS211 version.  However, our current Magnetite and Selenite
+firmwares follow the architecture of TCS211, not that of TCS3.2, and because
+the entire SSA division of the fw including the RiViera core came in full source
+form in our copy of TCS211, it was only natural to keep this code and its
+architecture.
+Start-up process continued
+==========================
+As mentioned earlier, Nucleus calls the application's software init function
+called Application_Initialize() after it initializes itself but before starting
+the task scheduler.  This function in TCS211 is just the following:
+Application_Initialize()
+{
+	Init_Target();
+	Init_Drivers();
+	Cust_Init_Layer1();
+	Init_Serial_Flows();
+	StartFrame();
+	Init_Unmask_IT();
+}
+Cust_Init_Layer1() is in L1, StartFrame() is in GPF, and the remaining 4 init
+functions live in the init.c module under src/cs/system/main.
+The Init_Target() function finishes the hardware initialization that was
+started by the assembly code at the firmware boot entry point (int.s): among
+other things, it sets up the final memory timings that will be used by the
+running fw and configures the Calypso DPLL which provides multiplied internal
+clocks to both ARM and DSP cores.  On Calypso C035 silicon which is used on our
+own FreeCalypso boards and on most of our pre-existing hw targets the DPLL and
+the DSP run at 104 MHz and the ARM gets half of that, running at 52 MHz.
+Init_Target() also calls AI_InitIOConfig(), the function that initializes
+Calypso GPIO directions and initial outputs; both of these functions typically
+need to be tweaked when adding support for a new Calypso board target.
+The Init_Drivers() function is primarily responsible for initializing RiViera
+and FFS, although it also does a bit of init related to ABB and SIM drivers.
+I mentioned earlier that every Nucleus task in our firmware gets created and
+started either through RiViera or through GPF.  All GPF tasks are created and
+placed into the runable state in the Application_Initialize() context: the work
+is done by GPF init code in gpf/frame/frame.c, and the top level GPF init
+function called from Application_Initialize() is StartFrame().  Thus when
+Application_Initialize() finishes and the Nucleus thread scheduler starts
+running for the first time, all GPF tasks are there to be scheduled.
+There is a compiled-in table of all protocol stack entities and the tasks in
+which they need to run; in TCS211 these GPF config bits live under
+g23m/condat/frame/config for the GSM+GPRS configuration and under
+g23m/condat/com/src/config for the GSM-only config without GPRS.  Canonically
+each protocol stack entity runs in its own task, but sometimes two or more are
+combined to run in the same task: for example, in the minimal GSM "voice only"
+configuration (no CSD, fax or GPRS) CC, SMS and SS entities share the same task
+named CM.  Unlike RiViera, GPF does not support dynamic starting and stopping
+of tasks.
+As each GPF task starts running (immediately upon entry into Nucleus' scheduling
+loop as Application_Initialize() finishes), pf_TaskEntry() function in
+gpf/frame/frame.c is the first code it runs.  This function creates the queue
+for messages to be sent to all entities running within the task in question,
+calls each entity's pei_init() function (repeatedly until it succeeds: it will
+fail until the other entities to which this entity needs to send messages have
+created their message queues), and then falls into the main body of the task:
+for all "regular" entities/tasks except L1, this main body consists of waiting
+for messages (or signals or timeouts) to arrive on the queue and dispatching
+each received message to the appropriate handler in the right entity.
+RiViera tasks get started in a different way.  The responsible code lives in
+src/cs/system/main/create_RVtasks.c, and the create_tasks() function found in
+that module is called by Init_Drivers() in the Application_Initialize() context.
+But this function does not directly create and start every configured RiViera
+task like StartFrame() does for GPF.  Instead it creates a special helper task
+which will do this work once scheduled.  Thus at the completion of
+Application_Initialize() and the beginning of scheduling the set of runable
+Nucleus tasks consists of all GPF ones plus the special RV starter task.  Once
+the RV starter task gets scheduled, it will call rvm_start_swe() to launch
+every configured RiViera SWE (SoftWare Entity), which in turns entails creating
+the tasks in which these SWEs are to run.
+Dynamic memory allocation
+=========================
+All dynamic memory allocation (i.e., all RAM usage beyond statically allocated
+variables and buffers) is once again done either through RiViera or through GPF,
+and in no other way.  Ultimately all areas of the physical RAM that will ever
+be used by the fw in any way are allocated when the fw is compiled and linked:
+the areas from which RiViera and GPF serve their dynamic memory allocations are
+statically allocated as char arrays in the respective C modules and placed in
+the appropriate IRAM or XRAM .bss section by the linker script; RiViera and GPF
+then provide API functions that allocate memory dynamically from these
+statically allocated large pools.
+RiViera and GPF have entirely separate memory pools from which they serve their
+respective clients, hence there is no possibility of one affecting the other.
+Riviera's memory allocation scheme is very much like the classic malloc&free:
+there is one large unstructured pool from which all allocations are made, one
+can allocate a chunk of any size, free chunks are merged when physically
+adjacent, and fragmentation is an issue: a memory allocation request may fail
+even when there is enough memory available in total if it is too fragmented.
+GPF's dynamic memory allocation facility is considerably more robust: while it
+does maintain one or two (depending on configuration) memory pools of the
+traditional "dynamic" kind (like malloc&free, susceptible to fragmentation),
+most GPF memory allocation works on "partition" memory instead.  Here GPF
+maintains 3 separate groups of pools: PRIM, TEST and DMEM; each allocation
+request must specify the appropriate pool group and cannot affect the others.
+Within each pool there is a fixed number of partitions of a fixed size: for
+example, in TI's TCS211 GSM+GPRS configuration the PRIM pool group consists of
+190 partitions of 60 bytes, 110 partitions of 128 bytes, 50 partitions of 632
+bytes and 7 partitions of 1600 bytes.  An allocation request from a given pool
+group (e.g., PRIM) can request any arbitrary size in bytes, but it gets rounded
+up to the nearest partition size and allocated out of the respective pool.  If
+no free partition is available, the requesting task is suspended until another
+task frees one.  Because these partitions are used primarily for intertask
+communication, if none are free, it can only mean (assuming that the firmware
+functions correctly) that all partitions have been allocated and sent to some
+queue for some task to work on, hence eventually they will get freed.
+This scheme implemented in GPF is extremely robust in the opinion of this
+author, and the other purely "dynamic" scheme is used (in the case of GPF) only
+for init-time allocations which are never freed, such as task stacks - hence
+the GPF-based part of the firmware is not suspectible at all to the problem of
+memory fragmentation.  But Riviera does suffer from this problem, and the
+concern is more than just theoretical: one major user of Riviera-based dynamic
+memory allocation is the trace facility (described in its own section below),
+and my observation of the trace output from Pirelli's proprietary fw (which
+appears to use the same architecture with separate Riviera and GPF) suggests
+that after the fw has been running for a while, Riviera memory gets fragmented
+to a point where many traces are being dropped.  Replacing Riviera's poor
+dynamic memory allocation scheme with a GPF-like partition-based one is a to-do
+item for our project.
+Message-based intertask communication
+=====================================
+Even though all entities of the G23M protocol stack are linked together into
+one monolithic fw image and there is nothing to stop them from calling each
+other's functions and accessing each other's variables, they don't work that
+way.  Instead all communication between entities is done through messages, just
+as if they ran in separate address spaces or even on separate processors.
+Buffers for this message exchange are allocated from a GPF partition pool: an
+entity that needs to send a message to another entity allocates a buffer of the
+needed size, fills it with the message to be sent, and posts it on the recipient
+entity's message queue, all through GPF services.  The other entity simply
+processes the stream of messages that arrives on its message queue, freeing each
+message (returning the buffer to the partition pool it came from) as it is
+processed.
+Riviera-based tasks use a similar mechanism: unlike G23M protocol stack
+entities, most Riviera-based functional modules provide APIs that are called as
+functions from other tasks, but these API functions typically allocate a memory
+buffer (through Riviera), fill it with the call parameters, and post it to the
+associated task's message queue (also in the Riviera land) to be worked on.
+Once the worker task gets the job done, it will either call a callback function
+or post a response message back to the requestor - the latter option is only
+possible if the requesting entity is also Riviera-based.
+A closer look at GPF
+====================
+There are certain sublayers within GPF which need to be pointed out.  The 3
+major subdivisions within GPF are:
+* The meaty core of GPF: this part is the code under src/gpf/frame in our
+Selenite GPF reconstruction, originating from gpf/FRAME in the TCS3.2 source
+from Peek/FGW.  It appears that this part was originally intended to be both
+project-independent (same for GSM, TETRA etc) and OS-independent (same for
+Nucleus, pSOS, VxWorks etc).  This is the part of GPF that matters for the
+G23M stack: all APIs called by PS entities are implemented here, and so are
+all other PS-facing functions such as startup.  (PS = protocol stack)
+* OS adaptation layer (OSL): this is the part of GPF that adapts it to a given
+underlying OS, in our case Nucleus.
+* Test interface: see the code under gpf/tst in TCS211 from Openmoko or in
+FC Selenite.  This part handles the trace output from all entities that run
+under GPF and the mechanism for sending external debug commands to the GPF+PS
+subsystem.
+GPF was a difficult step in our GSM firmware deblobbing process because no
+complete source for it could be found anywhere: apparently GPF was so stable
+and so independent of firmware particulars (Calypso or LoCosto, GSM only or
+GSM+GPRS, modem or complete phone with UI etc) that it appears to have been
+used and distributed as prebuilt binary libraries even inside TI.  All TI fw
+(semi-)sources we've got use GPF in prebuilt library form and are not set up to
+recompile any part of it from source.  (They had to include all GPF header
+files though, as most of them are included by G23M C modules, and it would be
+too much hassle to figure out which ones are or aren't needed, hence all were
+included.)
+Fortunately though, we were able to find the sources for most parts of GPF:
+* The LoCosto source in TCS3.2_N5.24_M18_V1.11_M23BTH_PSL1_src.zip features the
+source for the "core" part of GPF under gpf/FRAME - these sources aren't
+actually used by that fw's build system (it only uses the prebuilt binary
+libs for GPF), but they are there.
+* Our TCS211 semi-src doesn't have any sources for the core part of GPF, but
+instead it features the source for the test interface and some "misc" parts:
+under gpf/MISC and gpf/tst in that source tree - these sources are not present
+in the LoCosto version from Peek.
+The GPF frame, misc and tst sources we have found have been verified to match
+the binary objects that came with TCS211 from OM: they can be compiled into a
+bit-for-bit match.  However, one critical piece was still missing: the OS
+adaptation layer.  It appears that the GPF core (vsi_??? modules) and OSL
+(os_??? modules) were maintained and built together, ending up together in
+frame_<blah>.lib files in the binary form used to build firmwares, but the
+source for the "frame" part in the Peek find contained only vsi_*.c and others,
+but not any of os_*.c.
+Our FC Magnetite firmware uses the original binary libs from TCS211-Openmoko
+for its GPF component, but for FC Selenite the project requirement is to be
+completely blob-free, hence we had to reconstruct the source for the OSL part
+of GPF from disassembly.  This work was originally done in 2014 in the context
+of our first attempt at gcc-built blob-free GSM fw (FC Citrine, now deemed to
+be a dead end and fully retired); this reconstruction was then dug up and
+adapted for Selenite in 2018.  As of this writing, this reconstruction is still
+not 100% complete (one complex error handling function is stubbed out) and not
+yet trusted to be fully correct, thus our fully deblobbed Selenite firmware is
+currently considered experimental; our current production fw is still Magnetite
+with blobs for GPF.
+A closer look at L1
+===================
+The L1 code is remarkable in how little intertie it has with the rest of the
+firmware it is linked into.  It is almost entirely self-contained, expecting
+only 4 functions to be provided by the underlying OS environment:
+os_alloc_sig	-- allocate message buffer
+os_free_sig	-- free message buffer
+os_send_sig	-- send message to upper layers
+os_receive_sig	-- receive message from upper layers
+It helps to remember that at the beginning of TI's involvement in the GSM
+baseband chipset business, L1 was the only thing they "owned", while Condat,
+the maintainers of the higher level protocol stack, was a separate company.
+TI's "turnkey" solution must have consisted of their own L1 code plus G23M code
+(including GPF etc) licensed from Condat, but I'm guessing that TI probably
+wanted to retain the ability to sell their chips with their L1 without being
+entangled by Condat: let the customer use their own GSM L23 stack, or perhaps
+work out their own independent licensing arrangements with Condat.  I'm
+guessing that L1 was maintained as its own highly independent and at least
+conceptually portable entity for these reasons.
+The way in which L1 is intertied into the rest of the fw is the same in all TI
+production firmwares we have seen, including both our TCS211 reference and the
+TCS3.2 LoCosto version.  There is a module called OSX, which is an extremely
+thin adaptation layer that implements the APIs expected by L1 in terms of GPF.
+Furthermore, this OSX layer provides header file isolation: the only "outside"
+(non-L1) header included by L1 is cust_os.h, and it defines the necessary
+interface to OSX *without* including any other headers (no GPF headers in
+particular), using only the C language's native types.  Apart from this
+cust_os.h header, the entire OSX layer is implemented in one C module (osx.c,
+which we had to reconstruct from osx.obj as the source was missing - but it's
+very simple) which does include some GPF headers and implements the OSX API in
+terms of GPF services.  Thus in both TI's production firmwares and our own ones,
+L1 does sit on top of GPF, but very indirectly.
+More specifically, the "production" version of OSX implements its API in terms
+of *high-level* GPF functions, i.e., VSI.  However, they also had an interesting
+OP_L1_STANDALONE configuration which omitted not only all of G23M, but also the
+core of GPF and possibly the Riviera environment as well.  We don't have a way
+to recreate this configuration exactly as it existed inside TI because we don't
+have the source bits specific to this configuration, but we do have a little
+bit of insight into how it worked.
+It appears that TI's OP_L1_STANDALONE build used a special "gutted" version of
+GPF in which the "meaty core" (VSI etc) was removed.  The OS layer (os_???
+modules implementing os_*() functions) that interfaces to Nucleus was kept, and
+so was OSX used by L1 - but this time the OSX API functions were implemented in
+terms of os_*() ones (low-level wrappers around Nucleus) instead of the higher-
+level VSI APIs provided by the "meaty core" of GPF.  It is purely a guess on my
+part, but perhaps this hack was also done in the days before TI's acquisition
+of Condat, and by omitting the "meaty core" of GPF, TI could claim that their
+OP_L1_STANDALONE configuration did not contain any of Condat's "intellectual
+property".
+Run-time structure of L1
+========================
+L1 consists of two major parts: L1S and L1A.  L1S is the synchronous part where
+the most time-critical functions are performed; it runs as a Nucleus HISR.  The
+hardware in the Calypso generates an interrupt on every TDMA frame (4.615 ms,
+or more precisely 60/13 ms), and the LISR handler for this interrupt triggers
+the L1S HISR.  L1S communicates with L1A through a shared memory data structure,
+and also sometimes allocates message buffers and posts them to L1A's incoming
+message queue (both via OSX API functions, i.e., via GPF in disguise).
+L1A runs as a regular task under Nucleus, and includes a blocking call (to GPF
+via OSX) to wait for incoming messages on its queue.  It is one big loop that
+waits for incoming messages, then processes each received message and commands
+L1S to do most of the work.  The entry point to L1A in the L1 code proper is
+l1a_task(), although the responsibility for running it as a task falls on some
+"glue" code outside of L1 proper.  TI's production firmwares with G23M included
+have an L1 protocol stack entity within G23M whose only job (aside from some
+initialization) is to run l1a_task() in the Nucleus task created by GPF for
+that protocol stack entity; we do the same in our firmwares.
+Communication between L1 and G23M
+=================================
+It is remarkable that L1 and G23M don't have any header files in common: L1
+uses its own (almost fully self-contained), whereas the G23M+GPF realm is its
+own world with its own header files.  One has to ask then: how do they
+communicate?  OK, we know they communicate through primitives (messages in
+buffers allocated from GPF's PRIM partition memory pool) passed via message
+queues, but what about the data structures in these messages?  Where are those
+defined if there are no header files in common between L1 and G23M?
+The answer is that there are separate definitions of the L1<->G23M interface on
+each side, and TI must have kept them in sync manually.  Not exactly a
+recommended programming or software maintenance practice for sure, but TI took
+care of it, and the existing proprietary products based on TI's firmware are
+rock solid, so it is not really our place to complain.
+TI's firmwares from the era we are working with (both our TCS211 golden
+reference and the TCS3.2/LoCosto source from which we took the newer full-source
+version of G23M for our TCS2/TCS3 hybrid) also include a component called ALR.
+It resides in the G23M code realm: G23M coding style, uses Condat header files,
+runs as its own protocol stack entity under GPF.  This component appears to
+serve as a glue layer between the rest of the G23M stack (which is supposed to
+be truly hardware-independent) and TI's L1.
+Speaking of ALR, it is worth mentioning that there is a little naming
+inconsistency here.  ALR is known to the connect-by-name logic in GPF as "PL"
+(physical layer, apparently), while the ACI entity (Application Control
+Interface, the top level entity) is known to the same logic as "MMI".  No big
+deal really, but hopefully knowing this quirk will save someone some confusion.
+A closer look at our FreeCalypso TCS2/TCS3 hybrid
+=================================================
+Because we don't have an official TI firmware release for the Calypso in full
+source form and because I am not willing to throw away all of our Calypso work
+and restart anew with LoCosto with its own host of unknowns, the only currently
+available way for us to have blob-free production-quality GSM mobile station fw
+is the TCS2/TCS3 hybrid implemented in FC Magnetite and Selenite.  This hybrid
+is made by taking the G23M version from TCS3/LoCosto and grafting it onto the
+chipsetsw foundation from TCS211, including the original TCS211/Calypso version
+of L1 which we have meticulously source-reconstructed.  The version of GPF used
+for this hybrid is also the TCS211 version in Magnetite or our source
+reconstruction thereof in Selenite.
+The Condat G23M pieces have been hybridized as follows:
+* cdginc generated header files are a special hybrid version described below;
+* The include files under condat/com/inc and condat/com/include are the TCS3
+version, except for pwr.h and rtc.h for which we use the TCS2 version;
+* comlib is the TCS2 version, except for cl_rlcmac.c which is from TCS3;
+* config modules (condat/com/src/config and condat/frame/config) are the TCS2
+version, with some fixes for the needs of the TCS3 version of G23M PS and our
+own FreeCalypso fixes;
+* Condat drivers (condat/com/src/driver) are the TCS2 version;
+* All G23M PS components are the TCS3 version by necessity, as this is the part
+for which the source is missing in our TCS211 version, with the exception of
+ALR - the original source for the TCS211 version of ALR has miraculously
+survived, the ALR source in TCS211 from OM can be compiled into a perfect
+match for the binary lib version;
+* We use the TCS2 version of ALR (the interface to our TCS211 L1) and not the
+TCS3 version (a change from Citrine), but it is compiled with the same hybrid
+cdginc headers as the rest of hybrid G23M, not the old TCS211 ones;
+* ACI is the TCS3 version - we have the source for both versions, but trying to
+use the old TCS2 version of ACI on top of the new TCS3 version of the PS
+would cause untold breakage;
+* The UI layers (MFW and BMI) for handset fw builds are handled like ACI: we
+have the source for both versions, but we use the TCS3 version which works
+with the TCS3 versions of ACI and cdginc;
+* The CST (Customer Specific Task) component is the TCS2 version - while it
+logically belongs in the Condat realm, the code lives in the chipsetsw realm
+under chipsetsw/services/cst (yes, it's under services with SSA stuff even
+though it doesn't use RiViera) and thus our copy of TCS211 from OM has this
+source preserved.
+With this hybrid arrangement the main splice point lies above ALR, and there
+are many little splice points throughout the code where some upper-level code
+from TCS3 needs to talk to lower-level code from TCS2.  There are no inversions,
+i.e., no places where TCS2 code sits on top of code from TCS3, although there
+are a few instances where TCS2 C code uses some TCS3 header files.
+TCS3 feature flags
+------------------
+Our TCS3.2/LoCosto code from Peek/FGW from 20090327 supports several new GSM
+features (apparently related to GSM release 99) which are not supported by our
+TCS211-20070608 golden reference from OM.  All of these new features can be
+enabled or disabled with conditional compilation flags.  Our TCS2/TCS3 hybrid
+currently has all of these new features disabled: it was too difficult for me
+to figure out if these new features require some support from the hardware or
+the DSP which is present on LoCosto but not Calypso, and even if our hw and DSP
+have all of the necessary capabilities, at least some of the new features
+require adding some code to L1, which is incompatible with my approach of
+reconstructing TCS211 L1 pristinely.
+In any case, the GSM functionality we get by using the new version of G23M with
+new feature flags disabled on top of pristine TCS211 L1 cannot be any worse
+than what we would have had if we had the full corresponding source for our
+TCS211-20070608 golden reference, and it is probably a little better because we
+are using a newer version of G23M code.
+cdginc headers
+--------------
+Much of the code in the Condat G23M realm makes heavy use of a set of machine-
+generated C header files called cdginc.  These header files contain various
+definitions related both to the GSM air protocols being implemented and to G23M
+protocol stack internals (interfaces and message structures between components),
+and they are generated from a set of message definition files (*.mdf) and
+primitive definition files (*.pdf) by a tool called ccdgen.  The *.{mdf,pdf}
+inputs to ccdgen are human-readable ASCII, and of course the generated C header
+files are human-readable too, but we have no source for the ccdgen tool itself,
+only a Windows binary which we can run under Wine.
+The ccdgen binary problem is yet another instance of so far incomplete
+liberation of the GSM firmware.  It is currently a very low-priority problem:
+we do not casually edit any of the *.{mdf,pdf} inputs to ccdgen, and we don't
+run ccdgen on every fw build - instead we have run ccdgen once and checked its
+output files (generated C headers) into our Magnetite and Selenite trees as if
+they were sources.  If we are not able to convince TI to dig up and release the
+source for ccdgen, there is a viable albeit costly alternative: hire a Windows
+reverser to RE the ccdgen.exe binary (262144 bytes) and produce a C
+reimplementation that replicates all of its logic.  It is a Win32 console app,
+no GUI, and it is a pure data processing application without any hardware access
+or OS functions or any other muckery: it is probably pure ANSI C code that reads
+and parses a bunch of ASCII input files, performs some business logic on the
+data, and writes another bunch of ASCII text files as outputs.  It is currently
+a very low-priority task though; reversing the Calypso DSP ROM code should
+probably be a higher priority.
+The set of cdginc headers for our TCS2/TCS3 hybrid has been generated as
+follows:
+* All of the *.mdf files are the TCS3 version;
+* All of the *.pdf files except mphc.pdf and mphp.pdf are also the TCS3 version;
+* mphc.pdf and mphp.pdf are the TCS211 version - this is the interface to
+TCS211 L1;
+* All new feature flags (see discussion above) are set to disabled.
+Condat Coder and Decoder (CCD)
+------------------------------
+CCD is a firmware component in the Condat G23M realm which I haven't really
+studied yet.  It consists of two parts:
+* A fixed portion which TI used to distribute in binary form and which various
+firmware projects used as a prebuilt library like GPF - technically TI
+considered it to be a part of GPF, although we prefer to treat it as its own
+more independent entity;
+* The ccddata portion which needs to be compiled with cdginc headers for each
+given project.
+We got the source for both parts of CCD only in the TCS3.2/LoCosto version, but
+not in the TCS211 version, hence the decision was easy: we use the TCS3 version
+of CCD (both parts) with the TCS3 version of cdginc with the TCS3 version of
+the G23M PS.
+TCS3.2 GPF discrepancy
+----------------------
+A careful examination of the prebuilt GPF libraries under gpf/LIB in the TCS3.2
+LoCosto source tree has revealed that a few of the binary objects exhibit some
+differences from the TCS211 version which we've been treating as our golden
+reference:
+* The os_mis module (OSL miscellany) in the IRAM library implements a new
+function called os_CheckQueueEvent() and defines a new global data object
+named my_os_mis_Protect;
+* The os_tim module (OSL timer code) in the flash (XIP) library has some code
+differences;
+* The vsi_tim module (VSI timer code) in the flash (XIP) library has some code
+differences;
+* The vsi_tim module (VSI timer code) in the IRAM library has some code
+differences and makes use of the new os_CheckQueueEvent() function.
+In the case of os_??? modules we have no corresponding source for either
+version, but the vsi_tim difference is more bizarre: we got our vsi_tim.c source
+(and the rest of vsi_*.c) from the TCS3.2/LoCosto source, but this source
+matches the TCS211 binary version and not the newer and different binary version
+used by the TCS3.2 build system!  (Remember that none of TI's firmware build
+systems that we have seen are set up to recompile any part of GPF from source,
+they used it only as prebuilt libraries.)
+Because we have the corresponding source for the "old" version of GPF frame core
+but not for the "new" version, we are continuing to treat the "old" TCS211
+version as our golden reference: we use the source pieces which we got, and we
+use the "old" os_???.obj blobs as our basis for reconstruction via disassembly.
+Because the changes in the TCS3.2 binary version of GPF involve only the
+implementation of a part of VSI but not its API (there are no changes to any
+part of the GPF API presented to the G23M PS that I can see anywhere), I have
+every good reason to believe that there is no problem with using the new TCS3.2
+version of G23M with the old version of GPF from TCS211: it should work no worse
+than pure TCS211.
+It should also be noted that if we ever succeed in getting some more complete
+GPF source out of TI (including the source for the OS adaptation layer which is
+difficult to reconstruct), thanks to the great stability and independence of
+GPF, we will be happy with *any* version, does not need to match either TCS211
+or TCS3.2.
+GPRS implementation differences
+-------------------------------
+There is a visible difference between the way GPRS is implemented in the old
+TCS211-20070608 blob version of G23M and the way it is implemented in the newer
+TCS3.2/LoCosto version we are using for our hybrid.  The new implementation adds
+a new protocol stack entity named UPM (User Plane Manager), and the pre-existing
+SM and SNDCP entities have been significantly changed to work with this UPM.
+Because we are using the GPRS config modules (condat/frame/config) from TCS211,
+we had to add a -DFF_UPM compilation flag to include UPM in the GPF
+configuration for the GSM+GPRS protocol stack.
+A closer look at ACI
+====================
+The Application Control Interface (ACI) is the crown that sits on top of the
+G23M protocol stack.  It includes the AT command interpreter (ATI) component,
+and this AT command interface is brought to the outside world via the UART
+protocol stack entity.  The UART entity implements the GSM 07.10 MUX, can
+operate the physical UART in either multiplexed or non-multiplexed mode (the
+latter is the default on boot for a plain ASCII AT command interface) as
+commanded by ACI, and establishes 1 to 4 logical channels carrying AT commands
+to ACI.  When a CSD or fax call or a GPRS PPP session is in progress, the data
+path is switched to run between the UART entity and the appropriate GSM or GPRS
+protocol stack destination.  In the case of modem products that are designed to
+be controlled by an external host via AT commands, this combination of ACI and
+UART entities provides the ultimate end function of the device.
+The set of implemented AT commands is defined in ati_cmd.c: this is the C file
+where new AT commands get added; there is also an enum of command IDs in
+aci_cmh.h which needs to be extended.  For every AT command listed in the table
+in ati_cmd.c there is a handler function: for example, for the AT+CFUN command
+there is a setatPlusCFUN() function that handles setting and a queatPlusCFUN()
+function that handles querying.  For some simple AT commands like AT+CGxx
+queries the function listed in ati_cmd.c does the entirety of the work, but for
+most of the interesting GSM commands (including the AT+CFUN example just used)
+the set and query functions implemented in the ATI layer only handle the parsing
+of ASCII arguments and generation of ASCII output (if any), whereas the actual
+command implementation happens in the CMH layer below.
+Below ATI but still within ACI lies the sublayer of command handlers (CMH).
+For each AT command that does something to the GSM mobile station there is a
+functional equivalent, a C function that performs the same operation as the
+spec-defined AT command, but is designed to be used natively from C code,
+without AT command string parsing or output formatting.  For the AT+CFUN example
+used above, the setatPlusCFUN() ATI function parses the arguments from ASCII
+and then calls sAT_PlusCFUN() to perform the actual operation, whereas the
+queatPlusCFUN() ATI function calls qAT_PlusCFUN() to retrieve the current state
+and then prints it out in ASCII.  This functional interface is used by TI's
+demo/prototype phone UI implementation described in the Handset-UI-fw companion
+document.
+Finally, at the bottom of ACI lies the sublayer of Protocol Stack Adapters
+(PSA): these are pieces of code that execute within the ACI task and exchange
+primitives with various G23M protocol stack entities below.
+We have the source for both TCS2 and TCS3 versions of ACI.  The TCS2 version is
+from Openmoko, containing OM's modifications, and we had to go through these
+changes and additions by OM, reject the bogus ones and reimplement the sensible
+ones in the new TCS3 version of ACI for our TCS2/TCS3 hybrid going forward.
+Flash file system
+=================
+Every GSM device that is based on TI's firmware architecture contains not only
+the firmware image proper, but also a flash file system that is separate from
+the fw image and is maintained in a different part of the flash chip.  The FFS
+implementation code is a mandatory part of the firmware; in TCS211 it resides
+in chipsetsw/drivers/drv_app/ffs and logically belongs to the SSA realm.  This
+code initializes early in the fw boot process in the Application_Initialize()
+context before the start of Nucleus task scheduling; the responsible function
+is ffs_main_init() called from Init_Drivers().
+Flash driver support and FFS location
+-------------------------------------
+Determining the location of the flash area allocated for FFS and the flash
+driver to be used to write to it is a combination of autodetection and hard-
+coding.  The approach implemented in the original TCS211 code is as follows:
+there is a piece of autodetection code that reads the flash chip ID, and the
+autodetected ID is then looked up in a hard-coded table that gives the driver
+and geometry details and the location of the FFS sectors for each supported
+flash chip type.  However, this approach has its limitations:
+* The sequence of write operations which TI's autodetection code issues in
+order to put the flash chip into its Read ID mode worked for older flash
+chips that were used by TI and Openmoko, but does not work for the newer
+Spansion S71PL129NC0HFW4B flash chip which we (FreeCalypso) have copied from
+the Pirelli DP-L10 phone.
+* While the physical flash chip used on a given phone or modem board is a
+physical property that can be autodetected, the choice of which flash sectors
+should be used for FFS is a matter of policy.  Before we built our own
+FreeCalypso hardware, we had to run our fw on some pre-existing "alien" hw
+targets, and we still support such usage to a limited extent.  When we run
+our FreeCalypso fw on an alien hw target as an aftermarket deal, our
+aftermarket FFS location needs to be chosen quite carefully.
+* Some flash chips have two chip select banks, and with such chips it is
+generally desirable to put the FFS in the second bank.  However, it is a
+matter of board wiring whether that second flash chip select is connected to
+Calypso chip select nCS2, nCS3 or nCS4 - thus FFS addresses in the second bank
+have to be hard-coded with conditional compilation per board type and cannot
+be autodetected.
+To support our new repertoire of possible hardware targets, the logic has been
+changed as follows in our Magnetite and Selenite firmwares:
+* When the target is our own FC hardware family (CONFIG_TARGET_FCFAM) or the
+Pirelli DP-L10 phone (CONFIG_TARGET_PIRELLI), flash chip type autodetection
+is disabled and a strictly fixed hard-coded FFS configuration is used;
+* When the target is one of Mot C1xx subfamilies, the autodetection logic is
+kept (several different flash chip types are found in these phones), but a
+different table is used, giving our aftermarket FFS configurations for these
+Mot C1xx flash chip types;
+* When the target is gtamodem (Openmoko GTA01/02 modem) or dsample (TI's
+D-Sample board), we use the original autodetection logic and device table
+which we got from TI/Openmoko.
+We have also changed the AMD multibank flash driver to issue write commands in
+a way that is compatible with our new S71PL129NC0HFW4B chip.
+FFS life cycle
+--------------
+In products that have been built according to TI's original way, including
+Openmoko GTA01/02 and our own FreeCalypso devices, the FFS is formatted and
+initialized with some essential content at the time of device manufacture, and
+this factory-created and factory-initialized FFS then persists for the lifetime
+of the device.  In our factory environment at FreeCalypso hardware manufacturing
+we initialize the flash on our freshly assembled boards like this:
+flash erase 0 0x800000
+flash program-bin 0 fwimage.bin
+flash2 erase 0 0x800000
+This factory procedure (which should ONLY be executed at the factory and never
+by any end users or even sw/fw developers and tinkerers) ensures that the flash
+is completely blank everywhere except the fw image loaded at the time of
+production, and when this fw image boots for the first time, it will see blank
+flash in the FFS sectors.  When TI's FFS code in ffs_main_init() sees this
+condition, it performs what TI called a preformat: it writes a basic FFS block
+header into each FFS sector, but does not automatically perform a full format -
+instead the latter needs to be commanded explicitly by the production station
+via one of TMFFS command packet protocols as described later in this article.
+In FreeCalypso we have adopted TMFFS2 as our choice of Test Mode FFS access
+protocol, our host side implementation of this protocol is fc-fsio, and we
+format and initialize the FFS on our devices with an fc-fsio command script as
+part of our factory procedure.
+FFS content and usage
+---------------------
+TI's firmware architecture uses the FFS for many purposes:
+* The IMEI is stored in the FFS - GSMA can proclaim all they want that it
+"MUST" be stored in some kind of super-secure one-time programmable fuses,
+but in TI's architecture and in FreeCalypso it is just a regular file in the
+FFS.
+* A number of RF calibration tables are stored in FFS and read by the RF code
+in L1.  If you have a Rohde&Schwarz CMU200 instrument which is itself in good
+repair and calibration standing and a metrology-grade RF cabling setup whose
+insertion loss at the relevant GSM frequencies is precisely known, creating
+or recreating these RF calibration values is as simple as executing one shell
+script that takes a few minutes to run - this is how we do it at FreeCalypso
+hw manufacturing - but if you are an ordinary user or sw/fw developer or
+tinkerer without a professional calibration station setup, you need to use
+the RF calibration values that have been written into the FFS by the device
+manufacturer.  These RF calibration tables live under /gsm/rf.
+* /gsm/com/rfcap tells the RR component in the G23M protocol stack (not L1!)
+which frequency bands are supported on a given device - on our devices it is
+a factory-programmed file distinguishing between tri900 and tri850 units and
+telling the firmware which bands it should scan for possible GSM cells.
+* Manufacturer, model and revision ID strings may be written into /pcm/CGMI,
+/pcm/CGMM and /pcm/CGMR, respectively, to be returned by the corresponding
+AT+CGxx query commands.
+* The G23M protocol stack writes a number of dynamically updated files under
+the /gsm hierarchy and under /pcm.
+* TI's demo/prototype UI code (see Handset-UI-fw companion document) writes its
+persistent state in files under /mmi.
+* Audio mode configuration files are kept under /aud - see the Audio-mode-config
+article in freecalypso-tools.
+* If a given product uses the Melody E1 mechanism, melody files to be played
+through the RiViera Audio Service are kept in FFS - see the Melody_E1 article
+in freecalypso-tools.
+Building firmware for different targets
+=======================================
+TI's TCS3.2 firmware for their LoCosto chipset which was rejected by the Mother
+for reasons described near the beginning of this article makes a complete break
+from the past and has no possibility of supporting any pre-LoCosto chips such
+as our beloved Calypso, but TI's previous evolutionary developments weren't so
+drastic: the evolution to Calypso from previous chips such as Hercules and
+Ulysse was smoother, and our reference TCS211 fw is littered with C preprocessor
+conditionals supporting TI's earlier development boards prior to D-Sample and
+DBB chips prior to Calypso.
+TI's configuration management architecture supported only TI's own development
+boards and not any of the end product boards: unfortunately they did not follow
+a development model like the Linux kernel where everyone is encouraged to
+contribute their custom board support bits upstream and the mainline kernel
+strives to support every hw target that was ever supported with a single source
+tree, instead it was the divergent model where every end device manufacturer
+would take TI's reference firmware source and hack it for their specific needs
+with no concern for upstreamability or support for targets or applications
+other than their own.  TI's firmware build configuration model defined the
+following C preprocessor symbols relating to support for different hw targets,
+all numeric, i.e., each symbol is always defined to a number:
+BOARD identifies which board is to be targeted, with numbers assigned for
+different development boards made by various TI groups, but generally not for
+customer boards.  The only Calypso-based BOARD number is 41, originally
+assigned for the D-Sample but then also reused for the Leonardo; all other
+BOARD numbers are for some other chipsets that aren't Calypso.  The previous
+board before D-Sample was C-Sample, which is BOARD 9, but I am not sure exactly
+what chipset it had - perhaps it was Ulysse/Nausica/Clara.  There is still
+plenty of support for BOARD 9 and even earlier boards in the firmware source we
+got.
+CHIPSET identifies the main DBB chip.  The interesting numbers are 7 for the
+very original Calypso C05 rev A, 8 for Calypso C05 rev B (found on the D-Sample
+board which the Mother scored in 2015), 10 for Calypso C035 (the Calypso silicon
+version we work with in FreeCalypso), 11 for Calypso Lite (same as the regular
+Calypso except for smaller IRAM), 12 for Calypso+ (a short-lived intermediate
+step between Calypso and LoCosto) and 15 for LoCosto.
+ANLG_FAM (previously ANALOG) identifies the ABB chip.  The numbers are 1 for
+Nausica, 2 for Iota (what we use) and 3 for Syren (typically used with Calypso+
+like on the E-Sample board).
+RF_FAM (previously just RF) identifies the RF hardware hooked up to the baseband
+chipset.  The interesting numbers are 10 for Clara (D-Sample) and 12 for Rita,
+the latter being the only RF chip for which we have driver support.
+Naturally any code that cares about DBB register differences would use the
+CHIPSET definition, ABB support code would use ANLG_FAM, RF support code would
+use RF_FAM, and finally code that needs to know about board-level peripherals
+like LCDs and keypads would use the BOARD symbol.  This model worked fine up to
+D-Sample: for example, the code for C-Sample vs. D-Sample LCDs and keypads is
+cleanly conditionalized on BOARD 9 vs. BOARD 41.  However, the waters got badly
+muddied when TI introduced their Leonardo board and instead of giving it its
+own BOARD number, reused BOARD number 41 from D-Sample.
+D-Sample was TI's primary internal development platform for the Calypso,
+featuring Iota for the ABB and Clara for the RF part.  It was a great solid
+platform in every way except the RF part: the old Clara RF is inconvenient
+(needs more external parts) and TI were marketing their newer Rita RF to real
+end device manufacturers, but the D-Sample still worked great for development:
+if you aren't working specifically on the RF part, it doesn't matter as long as
+you have a working driver for it, which we lack.  Then TI made another Calypso
+development board called Leonardo, featuring the same Calypso+Iota baseband
+plus the newer Rita RF.  But this Leonardo never fully replaced the D-Sample
+for any of the high-level development in the SSA and UI groups.
+Openmoko's modem is a direct derivative of the Leonardo, the only change being
+the RFFE (for some reason FIC didn't like TI's quadband RFFE as implemented on
+Leonardo and E-Sample boards and used their own slightly hobbled triband RFFE
+instead), and the firmware build given to OM was TI's Leonardo fw with just a
+few tweaks in tpudrv12.h to account for the RFFE control signal differences.
+However, because Leonardo never got its own BOARD number and the BOARD symbol
+is still set to 41, all of the SSA/UI code (LCD, keypad, battery charging etc)
+is still built as if for D-Sample - but none of that code is used on a pure AT
+command modem without UI functions or UI hardware, hence OM probably never
+noticed anything odd.
+And it wasn't just Openmoko - it appears that TI used their Leonardo boards
+mostly or perhaps even solely in the ACI configuration without UI layers
+(MMI=0 build configuration), while all or most UI development was done on
+D-Sample kits.  Their TCS211 reference fw product officially supported both
+D-Sample and Leonardo targets in both ACI and BMI+MFW configurations, but if
+one were to build a high-end UI-enabled config for the Leonardo like pdt_2272,
+it would target a 176x220 pixel color LCD, the LCD output driver would be the
+one for the D-Sample (expecting memory-mapped LCD registers on nCS3), and the
+keypad driver would expect D-Sample keypad wiring.  Looking at the available
+Leonardo schematics I see a serial (uWire) LCD interface instead and a more
+basic keypad with different wiring, so I don't see how those Leonardo+UI
+firmware builds could possibly work.  Perhaps some other group at TI did some
+UI work on Leonardo boards, but never made it into the internal mainline
+from which TCS211 releases were cut - who knows...
+Finally, aside from the basic failure to distinguish properly between D-Sample
+and Leonardo boards, this whole BOARD number system provides absolutely no
+mechanism to distinguish between TI's development boards and end product boards
+derived from them, or between end product boards of vendor A vs. vendor B, or
+between end product model A and model B from the same vendor - it's always
+BOARD 41 as far as TI's code is concerned.  When TI had to modify their code
+for OM to support FIC's different TSPACT signal wiring, they just edited the
+definitions in tpudrv12.h without any conditionals, so one couldn't build
+binaries for the original Leonardo vs. OM's hardware from the same source tree
+in different configs.
+The build system of TCS211 produces a set of generated C header files named
+*.cfg (instead of the more natural *.h); these generated config headers define
+all of the C preprocessor symbols listed above and many more.  They are included
+sometimes as #include "board.cfg" and othertimes as #include "config/board.cfg"
+(ditto for other *.cfg), thus the list of -I directories passed by the build
+system on compiler invocation lines needs to include both the config directory
+and its parent.  In our Magnetite and Selenite build systems we likewise
+generate these *.cfg headers; some of the symbols defined therein are variable
+and originate from Bourne shell variables in our own configuration system, but
+many others are fixed.  See scripts/cfg-template in our Magnetite and Selenite
+trees for the magic.
+The BOARD symbol is always fixed at 41 in all FreeCalypso firmwares,
+corresponding to TI's D-Sample and Leonardo, and we use our own different
+mechanism to distinguish among our supported targets.  The solution adopted in
+Magnetite and Selenite is as follows: we are supplementing TI's *.cfg files
+with our own fc-target.cfg (included as #include "fc-target.cfg" or as
+#include "config/fc-target.cfg" matching whatever existing TI code we are
+gently extending), and this fc-target.cfg header is populated by the build
+system by copying the appropriate targets/*.h header file.  These targets/*.h
+header snippets define C preprocessor symbols of our own invention like
+CONFIG_TARGET_xxx, and whenever we need to know our target in C code, we
+#include "fc-target.cfg" and use #ifdef logic based on these preprocessor
+symbols of our own addition.
+RVTMUX debug and development interface
+======================================
+The Calypso chip has two UARTs, and TI's TCS211 firmware and its predecessors
+are designed with the assumption that both of these UARTs are available.  Per
+TI's fw architecture, Calypso's MODEM UART presents the standard AT command
+interface with GSM 07.10 MUX, CSD, fax and GPRS capabilities as described
+earlier when we looked at ACI and ATI, whereas the other UART (called the IrDA
+UART in hardware docs but not used for that purpose) presents a vitally
+important debug, development and production interface called RVTMUX.  This
+RVTMUX interface can also be moved to the MODEM UART, in which case the standard
+AT command interface is lost.
+RVTMUX is a binary packet interface, and it got its name because it is a MUX of
+multiple logical channels managed by the RiViera Trace (RVT) firmware component.
+RVTMUX is often thought of as being primarily a debug trace interface, as that
+is the primary use to which it is put: in normal operation the firmware emits
+quite voluminous debug trace output on the IrDA UART, encapsulated in 3
+different RVTMUX channels as explained below.  However, it is also possible to
+send a number of different debug and development commands to the firmware via
+this interface, and this functionality is used as a critical component in
+Calypso GSM device factory production line processes: this RVTMUX interface is
+the only way by which the FFS can be initialized, RF calibration and tests can
+be performed and the IMEI can be set at the factory.
+Communication with a running firmware over this RVTMUX interface in a
+development or production setting (whether passively reading debug traces or
+actively sending development or test commands to the running fw) requires
+specialized host tools.  TI originally had a suite of Windows-based tools for
+this purpose, but we are not using them in FreeCalypso: we only got Windows
+binaries without any sources, and even in the case of those binaries we only
+got an incomplete set with some important tools missing.  Instead we are using
+our own Unix-based tools called FreeCalypso host tools; these tools have been
+developed from scratch by Mother Mychaela after studying the firmware components
+with which they need to communicate.
+Debug trace output
+==================
+The firmware component that "owns" the physical UART channel assigned to RVTMUX
+is RVT, contained in chipsetsw/riviera/rvt in TCS211 or in src/cs/riviera/rvt
+in our Magnetite and Selenite firmwares.  It is a Riviera-based component,
+and it has a Nucleus task that is created and started through Riviera.  All
+calls to the actual driver for the UART are made from RVT.  In the case of
+output from the Calypso GSM device to an external host, all such output is
+performed in the context of RVT's Nucleus task; this task drains RVT's message
+queue and emits the content of allocated buffers posted to it, freeing them
+afterward.  (The dynamic memory allocation system in this case is Riviera's,
+which is susceptible to fragmentation - see discussion earlier in this article.)
+Therefore, every trace or other output packet emitted from a GSM device running
+our fw (or any of the proprietary firmwares based on the same architecture)
+appears as a result of a message in a dynamically allocated buffer having been
+posted to RVT's queue.
+RVT exports several API functions that are intended to be called from other
+tasks, it is by way of these functions that most output is submitted to RVT.
+One can call rvt_send_trace_cpy() with a fully prepared output message, and
+that function will allocate a buffer from Riviera's dynamic memory allocator
+properly accounted to RVT, fill it and post it to the RVT task's queue.
+Alternatively, one can call rvt_mem_alloc() to allocate the buffer, fill it in
+and then pass it to rvt_send_trace_no_cpy().
+At higher levels, there are a total of 3 kinds of debug traces that can be
+emitted:
+* Riviera traces: these are generated by various components implemented in
+Riviera land, although in reality any component can generate a trace of this
+form by calling rvf_send_trace() - this function can be called from any task.
+* L1 traces: L1 has its own trace facility implemented in
+src/cs/layer1/cfile/l1_trace.c; it generates its traces as ASCII messages and
+sends them out via rvt_send_trace_cpy().
+* GPF traces: code that runs in GPF/G23M land and uses those header files and
+coding conventions etc can emit traces through GPF.  GPF's trace functions
+(implemented in gpf/frame/vsi_trc.c) allocate a memory partition from
+GPF's TEST pool, format the trace into it, and send the trace primitive to
+GPF's special test interface task.  That task receives trace and other GPF
+test interface primitives on its queue, performs some manipulations on them,
+and ultimately generates RVT trace output, i.e., a new dynamic memory buffer
+is allocated in the Riviera land, the trace is copied there, and the Riviera
+buffer goes to the RVT task for the actual output.
+Trace masking
+=============
+The RV trace facility invoked via rvf_send_trace() has a crude masking ability,
+but by default all traces are enabled.  In TI's standard firmwares most of the
+trace output comes from L1: L1's trace output is very voluminous, and most of
+it is fully enabled by default.
+On the other hand, GPF and therefore G23M traces are mostly disabled by default.
+One can turn the trace verbosity level from any GPF-based entity up or down by
+sending a "system primitive" command to the running fw, and another such command
+can be used to save these masks in FFS, so that they will be restored on the
+next boot cycle and be effective at the earliest possible time.  Enabling *all*
+GPF trace output for all entities is generally not useful though, as it is so
+verbose that a developer trying to make sense of it will likely drown in it -
+and it will also overwhelm the debug trace facility itself, causing most of
+these far too voluminous traces to be lost.  Therefore, a developer seeking to
+debug an issue in the G23M protocol stack needs to enable traces very
+judiciously.
+GPF compressed trace hack
+=========================
+TI's Windows-based GSM firmware build systems include a hack called str2ind.
+Seeking to reduce the fw image size by eliminating trace ASCII strings from it,
+and seeking to reduce the load on the RVTMUX serial interface by eliminating
+the transmission time of these strings, they passed their sources through an
+ad hoc preprocessor that replaces these ASCII strings with numeric indices.
+The compilation process with this str2ind hack becomes very messy: each source
+file is first passed through the C preprocessor, then the intermediate form is
+passed through str2ind, and finally the de-string-ified form is compiled, with
+the compiler being told not to run the C preprocessor again.
+TI's str2ind tool maintains a table of correspondence between the original trace
+ASCII strings and the indices they've been turned into, and a copy of this table
+becomes essential for making sense of GPF trace output: the firmware now emits
+only numeric indices which are useless without this str2ind.tab mapping table.
+Our FC Magnetite build system retains the option of using str2ind, but it is
+disabled by default: str2ind significantly increases firmware compilation times,
+the resulting fw image sizes without str2ind are fine (the slight increase does
+not push us over any limits), and we haven't had any issues with ASCII strings
+overloading the trace interface.  However, there is an additional complication
+stemming from the choice of two possible G23M PS versions, one of which is a
+set of blob libraries:
+* If Magnetite is compiled in a pure TCS211 configuration using the original
+blob version of G23M PS, these blobs already have str2ind indices baked into
+them instead of trace ASCII strings, hence the frozen str2ind.tab file from
+Openmoko that maps these indices back to strings needs to be used.
+* If Magnetite is compiled in a TCS2/TCS3 hybrid config without G23M blobs,
+then unless you enable it explicitly with USE_STR2IND=1, no str2ind will be
+used at all.
+Our blob-free FC Selenite firmware does not support str2ind at all - we shall
+stick with full ASCII string traces until and unless we run into an actual (as
+opposed to hypothetical) problem with either fw image size or serial interface
+load.
+RVTMUX command input
+====================
+RVTMUX is not just debug trace output: it is also possible for an external host
+to send commands to the running fw via RVTMUX.
+Inside the fw RVTMUX input is handled by the RVT entity by way of a Nucleus
+HISR.  This HISR gets triggered when Rx bytes arrive at the designated UART,
+and it calls the UART driver to collect the input.  RVT code running in this
+HISR parses the message structure and figures out which fw component the
+incoming message is addressed to.  Any fw component can register to receive
+RVTMUX packets, and provides a callback function with this registration; this
+callback function is called in the context of the HISR.
+In the original TCS211 fw there are only two components that register to receive
+external host commands via RVTMUX: ETM and GPF, hence these are the only command
+packet types that can be sent to this original fw.  In FreeCalypso we have kept
+these, and we've also added some new RVTMUX channels of our own invention.
+Test Mode (TM) and Enhanced Test Mode (ETM)
+===========================================
+A major use of the RVTMUX interface is sending so-called Test Mode commands
+from an external host to a running GSM device.  Depending on the firmware
+version, a GSM device can be commanded to do any of the following things
+through this mechanism:
+* Exercise RF test modes, e.g., transmit continuously at a set frequency and
+power level;
+* Read and write arbitrary memory locations in the Calypso ARM7 address space;
+* Read and write ABB chip registers;
+* Reboot or power off;
+* Access and manipulate the device's flash file system (FFS).
+In the segment of history of interest to us TI has produced two different
+target firmware components that can receive, interpret and act upon Test Mode
+command packets:
+* The original Test Mode component of Layer 1, called L1TM or TML1: this
+component handles all RF test modes (needed for RF calibration on device
+production lines), and originally it also implemented memory and ABB register
+read and write commands, and provided access to TMFFS1 (see below).  In the
+original implementation this component registered itself as the handler for
+the "TM" RVTMUX channel (RVT packet type 0x14), so it would receive all TM
+packets sent to the device.
+* Enhanced Test Mode (ETM) is a later invention.  It registers itself (instead
+of the old TM in L1) with RVT as the handler for the "TM" RVTMUX channel, and
+then provides a registration service of its own, such that various components
+in the fw suite can register to receive external command packets passing
+first through RVT, then through ETM, and can send responses passing through
+ETM, then through RVT back to the external host.  If a given fw version
+contains both ETM and L1TM like TCS211 does, then L1TM registers itself with
+ETM; an external host would send exactly the same binary command packets to
+exercise RF test modes, but inside the firmware they now pass through ETM on
+their way to L1TM.
+The ETM_CORE module contained within ETM itself provides some low-level debug
+commands: by sending the right binary command packets to the GSM device via the
+RVTMUX serial channel, an external host can examine or modify any memory
+location and any hardware register, cause the device to reset, etc.  Prior to
+ETM some of these functions (but not all) could be exercised through older TM3
+commands, but in FreeCalypso we became familiar with the ETM versions of these
+commands long before the older ones because we got the ETM component in full
+source form, whereas the sole surviving copy of TCS211 that serves as our golden
+reference came with L1TM in binary object form like the rest of L1, and we got
+to source-reconstructing it only much later.
+ETM is implemented as a Riviera SWE and has its own Nucleus task; the callback
+function that gets called from the RVT HISR posts received messages onto ETM's
+own queue drained by its task.  The ETM task gets scheduled, picks up the
+command posted to its queue, executes it, and sends a response message back to
+the external host through RVT.
+Because all ETM commands funnel through ETM's queue and task, and that task
+won't start looking at a new command until it finished handling the previous
+one, all ETM commands and responses are in strict lock-step: it is not possible
+to send two commands and have their responses come in out of order, and it makes
+no sense to send another ETM command prior to receiving the response to the
+previous one.  (But there can still be debug traces or other traffic intermixed
+on RVTMUX in between an ETM command and the corresponding response!)
+L1TM commands get posted to the message queue of the L1A task and then executed
+in that task's context.
+FFS access via TM/ETM
+=====================
+One of the essential facilities provided in one form or another in all known
+incarnations of the Test Mode mechanism (at least in TI's original architecture,
+as opposed to Motorola's bastardized version) is the ability to access and
+manipulate the GSM device's flash file system (FFS) that was described earlier
+in this article.  TI's TMFFS1 and TMFFS2 protocols provide a command and
+response packet interface to the FFS API functions inside the fw, and enable an
+external host connected to the GSM device via the RVTMUX channel to perform
+arbitrary read and write operations on the device file system.
+In the segment of history of interest to us TI has produced two different
+and entirely incompatible versions of the TMFFS protocol: TMFFS1 and TMFFS2.
+Or rather, what is now called TMFFS1 was originally just TMFFS, and then came
+TMFFS2.  TMFFS2 works only through ETM, whereas TMFFS1 predates ETM: in the
+original implementation the tm_ffs() function in the FFS code was called from
+L1TM code.
+Our copy of TCS211 reference fw includes the source for both TMFFS1 and TMFFS2;
+it is theoretically possible to build a firmware image that includes both TMFFS
+versions (they won't conflict because they respond to different command
+packets), but it is pretty clear that TI never intended to have both enabled
+at the same time.  Our copy of TCS211 came with TMFFS1 enabled and we didn't
+change it when we made the moko12 (leo2moko-r1) fw release for the Openmoko
+community (the previous proprietary mokoN firmwares also implement TMFFS1),
+but we have subsequently switched to TMFFS2 for our current Magnetite and
+Selenite firmwares.
+Our choice of TMFFS2 over TMFFS1 was driven by the need to develop our own host
+tools to replace TI's original ones which we never got.  We needed to develop
+our own host tools for operating on GSM device FFS via one of the two TMFFS
+protocols, and after studying the fw source implementing both, I (Mother
+Mychaela) came to the conclusion that TMFFS2 is both more capable and more
+reliable; my guess is that TMFFS1 was likely kept around only because some of
+TI's crappy Weendoze host software depended on it.  (See the implementation
+code in chipsetsw/drivers/drv_app/ffs/board/tmffs.c in TCS211 if you would like
+to judge for yourself.)  Our host tool that speaks the TMFFS2 protocol is
+fc-fsio.
+GPF external command input
+==========================
+The other component that can receive external commands is GPF.  GPF's test
+interface can receive so-called "system primitives", which are ASCII string
+commands parsed and acted upon by GPF, and also binary protocol stack
+primitives.  Remember how all entities in the G23M stack communicate by sending
+messages to each other?  Well, GPF's test interface allows such messages to be
+injected externally as well, directed to any entity in the running fw.  System
+primitive commands can also be used to cause entities to send their outgoing
+primitives to the test interface, either instead of or in addition to the
+originally intended recipient.
+AT commands over RVTMUX
+=======================
+There is one more use to which we put the RVTMUX debug serial interface that is
+an original FreeCalypso invention: communicating with the AT command interpreter
+(ATI).  TI's original architecture assumes that if a product is to offer a
+standard AT command interface (the product is either a GSM/GPRS modem for which
+this AT command interface is the sole mode of usage or a feature phone that
+offers a data port as one of its features), then it will be presented on a
+dedicated UART separate from RVTMUX.
+However, in the case of our FreeCalypso family of projects about 2 years had
+passed between our first functional GSM fw attempts in 2015 and us successfully
+building our own development board in 2017; during this time we had to work on
+various crippled pre-existing Calypso devices, and many of them had only one
+UART practically accessible.  In response to this situation we developed a way
+to pass AT commands over RVTMUX.  We created a new RVTMUX channel for this
+interface and assigned it RVT packet type 0x1A.  Packets sent from an external
+host to the GSM device carry AT commands and SMS string input, whereas packets
+flowing the other way carry ATI's responses to commands and asynchronous
+notifications such as incoming calls.  The host utility for talking AT commands
+to a FreeCalypso GSM device via RVTMUX is fc-shell, described below.
+Now that we have built a proper FreeCalypso development board with two UARTs,
+the use of this AT-over-RVTMUX hack is deprecated for general usage: this hack
+does not support any data services (CSD or GPRS), and even for SMS it is
+crippled because maximum-length messages cannot be sent in the more capable PDU
+mode.  However, it still comes in handy during certain casual testing sessions,
+and it is required if one needs to run our FreeCalypso firmware on Mot C1xx or
+Pirelli DP-L10 hardware.
+FC host tools for talking to firmwares via RVTMUX
+=================================================
+The fundamental tool for talking to running firmwares via RVTMUX is a program
+called rvinterf.  It runs on a Unix/Linux host machine, opens a serial port
+that is expected to be connected to the RVTMUX UART on the target, and then
+speaks TI's binary packet protocol on that serial port.  It then performs two
+functions:
+* If rvinterf is run in the foreground in a terminal window (or more precisely,
+if its default terminal output is not disabled), every packet received from
+the target is decoded and printed on stdout in human-readable ASCII.  For
+some packets like TM/ETM responses this "human-readable" form is just a hex
+dump, but the trace messages which the firmware emits on its own are printed
+in truly human-readable form.  This output can also be saved to a log file.
+* Rvinterf creates a local UNIX domain socket on the machine it is running on,
+and other host tools can then connect to this socket to exchange packets with
+the firmware.  Client programs connected to rvinterf via this local socket
+interface can register to receive copies of packets sent by the target on
+specific RVTMUX channels, and they can also send arbitrary packets to the
+target.
+Our main "client" programs for actively interacting with running firmwares via
+rvinterf are:
+fc-tmsh		This utility speaks the TM/ETM protocol.  It supports almost
+		all ETM and L1TM commands that are supported by our reference
+		TCS211 fw with the important exception of TMFFS; support means
+		that fc-tmsh can issue these commands and decode the firmware's
+		responses to them.  fc-tmsh operates asynchronously in that the
+		issuance of commands to the target and the display of firmware
+		responses are completely decoupled; this asynchronous model is
+		a good match for L1/RF test mode commands and simple ETM
+		operations, but is a poor fit for FFS manipulation.  fc-tmsh's
+		companion fc-fsio implements FFS access via TMFFS2, and we
+		don't have a host side implementation for TI's older TMFFS1
+		protocol.
+fc-fsio		This utility speaks the TMFFS2 protocol over the TM/ETM RVTMUX
+		channel (same channel as used by fc-tmsh, so don't try to run
+		both at the same time) and implements fairly high-level FFS read
+		and write operations.  fc-fsio is used to format and initialize
+		the FFS on newly made devices in our hardware manufacturing
+		environment, it can upload files or entire subtrees into target
+		device FFS, it has higher-level commands for writing some files
+		like the IMEI, rfcap and AT+CGxx ID strings, and it can list and
+		read out FFS content.  Unlike fc-tmsh, fc-fsio is synchronous:
+		it is built on command-response (send a command and expect a
+		response) primitives, and a single user command can turn into a
+		large number of command-response exchanges on the RVTMUX
+		interface.  fc-fsio also implements a few non-FFS commands
+		because they naturally fit into this ETM synchronous model.
+fc-shell	This tool is asynchronous like fc-tmsh, but instead of talking
+		and listening on the TM/ETM RVTMUX channel, it talks and listens
+		on GPF's channel and on the new AT-over-RVTMUX channel which we
+		added in FreeCalypso.  fc-shell can be used to issue system
+		primitive commands to GPF (and to see firmware responses to
+		them), and to talk AT commands via RVTMUX.
+Finally, if you only need to passively observe the firmware's debug trace output
+and don't need to make any active pokes at the target, our rvtdump utility is a
+stripped-down version of rvinterf (or historically its predecessor) that only
+decodes and prints/logs the output from the target without sending anything to
+it.
+Further reading
+===============
+Believe it or not, some of the documentation that was written by the original
+vendors of the software in question and which we've been able to locate turns
+out to be fairly relevant and helpful, such that I recommend reading it.
+Documentation for Nucleus PLUS RTOS:
+	ftp://ftp.freecalypso.org/pub/embedded/Nucleus/nucleus_manuals.tar.bz2
+	Quite informative, and fits our version of Nucleus just fine.
+Riviera environment:
+	ftp://ftp.freecalypso.org/pub/GSM/Calypso/riviera_preso.pdf
+	It's in slide presentation form, not a detailed technical document, but
+	it covers a lot of points, and all that Riviera stuff described in the
+	preso *is* present in our fw for real, hence it should be considered
+	relevant.
+GPF documentation:
+	https://www.freecalypso.org/LoCosto-docs/SW%20doc/frame_users_guide.pdf
+	https://www.freecalypso.org/LoCosto-docs/SW%20doc/vsipei_api.pdf
+	Very good reading, helped me understand GPF when I first reached this
+	part of firmware reintegration.
+TCS3.x/LoCosto fw architecture:
+	https://www.freecalypso.org/LoCosto-docs/SW%20doc/TCS2_1_to_3_2_Migration_v0_8.pdf
+	ftp://ftp.freecalypso.org/pub/GSM/LoCosto/LoCosto_Software_Architecture_Specification_Document.pdf
+	These TI docs focus mostly on how they changed the fw architecture from
+	their TCS2.x program (Calypso) to their newer TCS3.x (LoCosto), but one
+	can still get a little insight into the "old" TCS211 architecture they
+	were moving away from, which is the architecture we've adopted for
+	FreeCalypso.

FreeCalypso > hg > freecalypso-docs

comparison TCS211-fw-arch @ 0:fcd1cf531017