FreeCalypso update

[Mychaela Falconia]

Hello FreeCalypso community,

As you can probably tell from me having been mostly inactive over the
past 3-4 months, I've been away from the FreeCalypso project, dealing
with a bunch of personal life issues, but I've been really missing the
fun of our project, and I am now getting back into it - and I have a
bunch of updates.

The first update is that I have found a semi-local (Colorado) Altium
expert who was willing to take on our FCDEV3B (FreeCalypso development
board, triband) project at a reasonable hourly rate which I can
afford.  As you may remember, the design of this board was done for us
by an anonymous company (they never told me who they were) on a barter
basis, in exchange for me giving them a ton of hand-holding support
with FreeCalypso sw/fw on the Openmoko GTA02.  Then they made a bunch
of broken promises (they said they were going to pay for the physical
fabrication of the first batch of boards and send some of those boards
to me for further distribution to the FC community, but I never got
any boards from them - I don't know whether or not they ever produced
any for themselves), and when I was looking into ordering the
fabrication of these boards on our own, I discovered that the design
we got had one last remaining critical defect: the copper pour
clearances were too small.

Fast-forwarding to the present, I have hired this fine gentleman in
Colorado (the Altium expert) to fix this copper pour clearance defect
in our FCDEV3B design, fix any other defects he can spot, and
regenerate new corrected gerber files - he is working on it now.  Once
this work is complete, I will launch a new crowdfunding campaign to
raise funds for the physical production of our FreeCalypso development
boards - we will need about $5000 USD, and as discussed on this list
back in May, anyone who puts down $450 gets one board - if we can get
10 people to preorder boards at risk, we'll be able to finally get
them built.

The next update is on the firmware front.  As you may remember from
June, replacing the L1 code in our FreeCalypso Citrine firmware (the
one built with gcc without any blobs) with a more properly reconstructed
version fixed the deep sleep bug and the EFR voice codec (AMR is still
broken), but also introduced a major regression: oftentimes the voice
uplink path is broken, i.e., the downlink audio sounds just fine, but
the other end of the call hears nothing.

The fundamental problem with our Citrine firmware is that far too many
variables have been changed all at once.  The differences between the
working TCS211 reference and the non-working Citrine fw are vast:

* An entirely different version of the G23M protocol stack is used:
  TCS211 has blobs, Citrine uses the version from LoCosto, hacked up
  quite a bit;
* TCS211 has FAX_AND_DATA and GPRS enabled, Citrine has them disabled;
* Entirely different compiler with different optimization, different
  choices of ARM vs. Thumb instructions and fast vs. slow memory,
  different ABI and different way of ARM/Thumb interworking;
* A different version of Nucleus PLUS RTOS is used to replace the blob
  version in TCS211;
* The entire code base has been reintegrated in a different way, too
  many changes to audit them all.

Thus I have come to feel that Citrine, our fw approach up until now,
may be a dead end after all, and that we may need to take a different
approach toward the goal of GSM firmware that is maintainable and
tweakable, but also working.  The approach I am now leaning toward is
much more conservative: instead of rebuilding the fw from the ground
up and hoping that it will "magically just work" in the end (already
tried it with very disappointing results), start with the working
TCS211 "golden" reference and *change one variable at a time*, testing
for breakage at every step.

Toward this plan I have just started an entirely new FreeCalypso
firmware project:

https://bitbucket.org/falconian/fc-magnetite

So far FreeCalypso Magnetite is essentially the same thing as TCS211:
it still builds with the same TI proprietary compiler under Wine, and
almost all of TCS211 blobs are still there.  But unlike Citrine, it
works.  However, the biggest change from our TCS211 starting point to
the present state of FreeCalypso Magnetite is the build system.

When building leo2moko or tcs211-c139 or any of our other TCS211-based
firmware trees, one invokes Wine at the very top, and then the entire
build process happens in the Windows environment: Windows versions of
make and so forth.  In FC Magnetite I have moved the Wine invokation
from the top (root) of the build tree to the leaves: in the new build
system Wine is invoked only to run the actual .exe binaries of the
TMS470 compiler toolchain (to compile each individual C file, to build
a .lib from .obj files and so forth), while the entire process of
orchestrating the build (hierarchy of Makefiles and helper scripts)
happens at the native Unix level, above Wine instead of below.

In both TCS211 and FC Magnetite build systems the build takes place in
its own directory separate from the source, using Makefiles which are
themselves generated.  But the system for generating these Makefiles
is entirely new in FC Magnetite, and so is the working of these
generated Makefiles themselves.  The build system of TCS211 (the
system that generates pdt_*.mak makefiles and the helper scripts then
used by these makefiles) is called BuSyB, and it consists of some
tools written in Java and distributed as jars (bytecode) sans source,
a ton of XML files and a few Perl script helpers.  Because this build
system is incomprehensible and uses what are effectively binary tools
sans source (Java jars), I have NOT attempted a port of this BuSyB
from Windows to Unix - instead I have replaced it with an entirely new
build system of my own.  The latter is where the real work was.

Instead of Java and XML, the new build system of FreeCalypso Magnetite
is implemented in Bourne shell - it is a *lot* of Bourne shell magic.
But it faithfully replicates the TCS211 build process even in some
very minute details, and the result is a hierarchy of Makefiles that
works entirely at the native Unix level, with TMS470 .exe tools
encapsulated under Wine wrappers at the leaves, and produces exactly
the same firmware build as TCS211's BuSyB.  Thus instead of the
massive pain of dealing with TCS211 pdt_*.mak makefiles which were
almost like blobs in themselves, we now have the full freedom to
support many different build configurations in the same source tree,
to change from blobs to recompilation with the granularity of
individual libs or even individual objects if need be, to have
different build configurations compile the same sources with different
-D options, to add and delete components to and from the build as
desired, and so forth.

Now remember what I just wrote about changing only one variable at a
time?  My next planned experiment will do just that: the first variable
I would like to try changing is the G23M protocol stack.  In my next
experiment I will try to compile the TCS3.2 (LoCosto) version of the
G23M protocol stack (full C source) in the FC Magnetite environment,
to link it with the remaining fw components that stay unchanged from
TCS211 (even the L1 blobs), and see what happens.  Rationale: L1 and
G23M are the two major code components that came as binary blobs in
our only available copy of TCS211.  In the case of L1 I have already
proven that we can deblob it (transition from the use of blobs to
recompilation from C) by doing one module at a time, massaging LoCosto
L1 C modules until they compile in the TCS211 environment (TCS211 L1
headers and all) into objects that match the original blobs.  Now we
have G23M left to be addressed.  Hence my idea - try swapping G23M,
replacing TCS211 blobs with TCS3.2 source.  It is certainly possible
because we've done it "successfully" in Citrine - but in the latter
we've changed too many variables at once, and the result is pretty
broken.  Hence the experiment idea: what if we swap out G23M, but keep
everything else the same: keep the L1 blobs, TI's compiler, the
config of having FAX_AND_DATA and GPRS enabled, all of the rest of
TCS211 - and see what happens.  Stay tuned!

M~