C139 calibration cracked

[Mychaela Falconia]

Hello FreeCalypso community,

While my primary interest is in building new FreeCalypso hardware
devices which meet my high quality standards, I am also continuing to
pursue the effort to run FreeCalypso firmware on those pre-existing
"alien" hw targets for which we already have some preliminary support,
namely, Motorola C139 and Pirelli DP-L10.  Both of these targets have
severe limitations: the C139 is just naturally limited by being an
ultra-low-end model with a very tiny LCD and no loudspeaker, whereas
the Pirelli is full of undocumented peripheral hw outside of the
Calypso core chipset, resulting in our current inability to turn on
the loudspeaker (neither for calls nor for ringing), ruining the
practical usability of this phone.

But at least in the case of the C139 there are no absolute barriers to
running liberated firmware on this hw, i.e., there is no reason why it
can't be done - it most certainly CAN be done, and is only a matter of
development effort and priorities.  For me personally the relative
priority of C139 work has been pretty low because the ultra-low-end
nature of this hw model makes it unattractive for me, but it is still
non-zero.  Given how much we all need at least *some* fully liberated
phone and given the irrefutable fact that doing such a feat on the
plentifully available and dirt-cheap (at least US-band versions) Mot
C139 hardware is possible, one can argue that those with the necessary
skills (which would be me) have a moral obligation to do it.

With this background in mind, I have just made one important
breakthrough on the road toward running end user liberated fw on the
C139.  Up until now, whenever we ran FreeCalypso fw (any version) on
Mot C139 or any other C1xx hardware, we've been running completely
uncalibrated, using the fw's compiled-in values (unchanged from TI's
Leonardo, early 2000s, predating Openmoko) for all of the required
calibration parameters.  We've been able to connect to live GSM
networks in this state because the VCXO used by Mot/Compal just
happens to be close enough to what TI must have used on the Leonardo
so that AFC still works with TI's Leonardo afcparams values and the
C139 running FreeCalypso fw succeeds in acquiring the frequency burst,
but the uncalibrated Tx power levels aren't going to be correct except
maybe by dumb luck, and the same holds for Rx gain control.

We've been running uncalibrated on these C1xx phones because even
though Mot/Compal did calibrate them on their factory production line
like all other manufacturers, we haven't been able to make use of
their factory calibration values because the latter are in a
proprietary non-TI format of Mot/Compal's invention which we hadn't
been able to grok.  In my opinion, directing end users to use a hacked
phone on live public networks while operating the radio without any
kind of calibration is a big no-no, hence this lack of calibration was
a major damper on the C139 libre fw effort.

I considered the possibility of recalibrating these phones anew on the
CMU200, but it would be a major effort even for me, despite already
having a working CMU200 - it is not clear to me exactly how to
disassemble the phone case non-destructively to get to the internal RF
test port, and that port is of a different type than what Openmoko
used, hence a different measurement cabling setup would be needed.
And because we are talking about precision RF metrology here, the
cabling setup involves a lot more than just buying a cable with the
right connectors off ebay - the exact insertion loss of this cable at
specific frequencies needs to be known with high precision, which is
no small feat.  And of course a CMU200 recalibration requirement would
leave out those users who won't be able to afford the services of a
FreeCalypso phone recalibration shop.

And here comes this week's breakthrough: I have cracked the format in
which the factory RF calibration values are stored in the flash memory
of Mot C1xx phones, and there is now a utility in the freecalypso-tools
repository (will go into the next FC host tools release) that extracts
these calibration values, converts them into classic TCS211 RF table
format for our FreeCalypso fw, and writes them into files which you
can then upload into the FFS of your FC-converted C139 with fc-fsio.

The parameters that are calibrated per unit and will thus vary from
one unit to the next include the Rx GMagic value (Rx gain calibration)
and the set of APC DAC values to produce the intended Tx power levels;
these are the two calibration data items which my c1xx-calextr utility
currently extracts and converts into TI/FreeCalypso format.  These
parameters have been calibrated at the center frequency of each band.
There are also frequency channel-dependent corrections for both Rx and
Tx, but those are currently dropped in the conversion because the way
in which Mot/Compal do these channel corrections is very different
from TI's canonical way, and the meaning of the respective bits in
their flash records is not clear.  But these channel corrections are
quite small, and I am reasonably confident that with us using just the
center-of-band calibration values from Mot/Compal's factory, our Tx
power levels will be within the tolerances given by the GSM 05.05
spec, and the Rx gain control will also be within the margin for good
Rx performance.

There are also a number of parameters which are calibrated per design,
rather than per unit.  Because they aren't calibrated per unit, they
are not present in the factory-written flash data records, but because
they differ from one hw design to the next, not having values that are
correct for the hardware you are running on is also bad.  The Tx ramp
templates are calibrated per design rather than per unit on every GSM
device I have seen (on our own FCDEV3B we use the numbers provided by
TI inside their l1_cust.obj binary blob, just like Openmoko did), and
both Compal and Pirelli phones use fixed (per-design rather than per-
unit) values in the afcparams table.  (On our own FC hardware we
calibrate the VCXO and generate this afcparams table per unit,
following Openmoko.)

To have the correct Tx ramp templates when running FC fw on Mot C139
hw, I have extracted the templates used by Mot's official fw on this
model (different from the C11x ones which can be extracted from that
one C11x fw image we found with symbols) by reading them out of the
running fw via the TX_TEMPLATE_READ Test Mode command, preceded by
tms 1 to shut off the fw's normal L1 operation and rfpw 7 to select
each band of interest.  It was definitely tricky, but I got the Tx
ramps tables for all 4 bands (i.e., for both EU-band and US-band
units) read out, and they are now used by FC Magnetite fw when built
for the C139 target.  And it's a similar deal with the AFC parameters:
our FC Magnetite fw now uses the same numbers as Compal's and Pirelli's
official firmwares when built for the respective hw targets.

The end result of all of the above is that if you run the new FC
Magnetite fw on a C139 and upload the per-unit RF calibration values
extracted with c1xx-calextr, the resulting radio operation should now
be the same as with the official fw, modulo the small channel
corrections which we are currently dropping.  On the Pirelli we
likewise run with exactly the same RF calibration values (Rx, Tx and
now the AFC parameters as well) as the official fw, but on this target
there are some additional unknowns: TI's Leonardo reference design
used a "plain" VCXO without built-in temperature compensation, Mot
C1xx, Openmoko and our FCDEV3B do likewise, hence TI's reference fw is
correct for these platforms, but the Pirelli DP-L10 uses an external
VCTCXO instead, and I simply don't know whether or not the firmware
should do something different to be properly correct on this hw.

With RF calibration on "alien" C139 & Pirelli hw finally taken care
of, my next plan is to work on battery charging.  More specifically,
my plan is to decouple the core charging control code from the UI, so
that the battery charging code can be included and functional in
Magnetite fw builds for C139 and Pirelli targets even when no UI
layers are included.  The result will be a "voice pseudo-modem" fw
that can be flashed or RAM-loaded (in the case of the Pirelli) into
the phone, is able to charge the battery, but still requires control
via AT commands from a connected computer, with the LCD remaining dark
and buttons doing nothing.  While this configuration won't be useful
for any end users, factoring and decoupling are important in software
architecture - TI's fw architecture is already way too tangled as it
is, and decoupling battery charging from the UI would be a welcome
step in the direction of modularity.

But the biggest difficulty is going to be with the UI.  With RF
calibration taken care of and with battery charging hopefully working
before the end of 2017 or in early 2018, the lack of a working and
usable UI will literally be the *only* thing standing in the way of
practically usable libre phone fw on the C139.

The problem with UI is that of sequentiality of work steps, with the
desired sequence (desired on my part, that is) not being in the favor
of a prospective FreeCalypso C139 end user.  The underlying problem is
that the UI code we got from TI is of a demo / prototype / proof of
concept kind, and is nowhere close to a usable product - in marked
contrast with the underlying modem layers which are rock solid.
Because TI were in the business of selling chips and supporting the
core modem functionality, rather than designing and maintaining phone
UIs, they never developed UI versions for different screen sizes in
pixels, instead their demo/prototype UI targets the 176x220 pixel LCD
on their D-Sample platform.  What we got working on the C139 right now
is a dirty hack that resurrects TI's even older C-Sample UI (84x48 pix
B&W), which is bitrotten and therefore even more broken than the
D-Sample one.

The LCD on the C139 is 96x64 pixels, 16 bits of color, and in order to
turn this hw into a libre phone, we need a 96x64 pixel color UI.  The
Pirelli DP-L10 has a bigger LCD of 128x128 pixels (same color depth),
but if we had a working 96x64 pixel UI, we could run it on the Pirelli
as well (using a subset of the available LCD space), and see if the
rest of the phone is usable (lots of unknowns there) before spending
the effort to design and implement a 128x128 pixel UI for the Pirelli.
Thus a 96x64 pixel UI (one that isn't bitrottent and broken like the
C-Sample one) is what we really need.

But given the extremely messy state of the UI code we got from TI, I
am not willing to jump directly into the implementation of this 96x64
pixel UI.  Instead I desire a working hardware platform with a 176x220
pixel LCD (has to be this size, not smaller) on which I can run the UI
code we got from TI in its native unhacked form, the way it once ran
on D-Sample boards inside TI.  I want to get this UI working properly
on its original 176x220 LCD target, fix the most immediate crashing
etc bugs, do some other important software maintainability work along
the lines of fully migrating to the hybrid config and retiring the
TCS211 blobs version of the G23M PS and the ACI that goes with it, and
only *then* work on building a reduced 96x64 pixel version.

The blocker is that I don't have a hardware platform that has the
right Calypso chipset version on which we can run FreeCalypso *and* a
176x220 pixel LCD.  I have a real TI-made D-Sample board with an
apparently-working LCD, but we are not currently able to build
FreeCalypso fw for the D-Sample target because we have no tpudrv10.c
or even tpudrv10.obj driver for Clara RF.  We have the ancient
20020917 fw image that came in the flash with this D-Sample board, but
no symbols for it.

Unless a miracle happens and someone finds a copy of the Clara RF
driver that used to be tpudrv10.c (to everyone reading, you know what
to search for), I see only two options for how to get the development
platform I need:

Option 1: expend a Herculean effort to reverse-eng the ancient 20020917
D-Sample fw in the absence of symbols, and try to locate the code that
came from tpudrv10.c, then try to understand that code and translate it
into C in a form that can be integrated with our late TCS211 version
of L1.  Absolutely not something I look forward to doing.

Option 2: design and build a new FreeCalypso hardware board based on
the FCDEV3B, but adding the needed 176x220 pixel LCD, as well as the
standard set of buttons and other handset peripherals, making my
desired Handset Motherboard Prototype (HSMBP).

Option 2 appeals to me a lot more, but it's going to take a lot of
time and money - and until then, we have no UI.  We can have
everything else working on Mot C139 hardware, but no usable UI...

Thus unless someone can find tpudrv10.c for Clara RF on the D-Sample,
those who wish to see libre phone fw on the C139 and possibly on the
Pirelli as well should support my effort to build the HSMBP as the
next FreeCalypso board, along with the prerequisites of first
resolving the still-outstanding hw issues on our current FCDEV3B.

Hasta la Victoria, Siempre,
Mychaela aka The Mother