Another FreeCalypso update

[Mychaela Falconia]

Hello FC community,

I have several exciting news regarding the firmware side of FreeCalypso:

* I have found the bug that was causing voice calls in AMR mode to
  break - it was not in L1 or in the DSP or any other place we thought
  of before; instead it was a fallout from TI's addition of A5/3 support
  in their TCS3.2/LoCosto firmware from which we get our non-blob
  version of the G23M protocol stack.  Hardware support for A5/3
  exists on TI LoCosto but not on TI Calypso; whoever added support
  for this cipher to the TCS3.2/LoCosto code base expanded the
  structure field that carries the A5 cipher key from 8 to 16 bytes as
  needed for A5/3.  The problem is that this structure field lives in
  the middle of several structures which are passed from ALR down to
  L1 in messages, and you can probably guess now what happened: TCS3.2
  version of the G23M PS uses one struct definition with a 16-byte
  field, Calypso L1 expects an 8-byte field in there instead, all
  following fields are shifted, all parameters carried in those
  subsequent fields get garbled.  AMR parameters were among those.

* I have applied the fix for the just-described A5/3 bug to both
  Citrine and Magnetite firmwares (see below), and both firmwares are
  now able to make successful voice calls in AMR mode.  In the case of
  Citrine the advertisement of AMR capability to the network is still
  disabled by default: there was an intermittent issue with the DSP
  dynamic download mechanism hanging sometimes, so I disabled that one
  for the default config, and although experiments indicate that AMR
  seems to work fine even without this patch, I decided to err on the
  side of safety and disable the non-essential frills in the default
  configuration.  But you can add an ALLOW_AMR_CODEC=1 line to your
  build.conf when compiling Citrine and see for yourself that AMR does
  work.

* Citrine previously had an issue where voice calls were unreliable
  even with FR and EFR codecs: the downlink audio would sound just
  fine, but nothing would be transmitted in the call uplink.  The issue
  was intermittent, so I don't have positive proof that it is fixed,
  but I have reason to believe that this misbehaviour was also caused
  by the A5/3 struct mismatch bug which is now fixed, hence I shall
  consider this call uplink non-passing bug to be fixed until and
  unless it manifests again.

As I wrote here just under two weeks ago, we now have two different
FreeCalypso firmwares: Citrine and Magnetite.  As a reminder, they
live here:

https://bitbucket.org/falconian/freecalypso-citrine
https://bitbucket.org/falconian/fc-magnetite

FC Citrine is the direct continuation of what we used to call just
"FreeCalypso GSM firmware", started back in 2013.  It is built with
gcc and does not use any of TI's COFF binary blobs (they can't be used
with the GNU toolchain without doing some major work on the latter),
hence all functionality that is present is compiled from full source.
Up until a couple of days ago it was essentially a non-working
experimental fw, as the most basic voice call functionality was always
broken in one way or another, but after the most recent A5/3 bugfix
the voice call functionality now seems to be working reliably at last.

With this most recent fix FC Citrine can now be considered a viable
and reliable choice of Calypso firmware for the most basic voice+SMS
functionality with control via AT commands (no UI) - and because it is
compiled from full source with gcc (no blobs, no proprietary tools),
we can finally compete with OsmocomBB feature for feature. :)

As exciting as it is, however, Citrine still has the limitation of
severely reduced functionality compared to the blob-laden, Windows-built
TCS211: the latter has CSD, fax, GPRS and advanced audio functions,
none of which are present in Citrine.  We could close this gap in two
ways:

1. We could start incrementally adding bits of functionality to Citrine,

or

2. We can take the opposite approach: set Citrine aside, restart with
   TCS211 with its full functionality, and start deblobbing it gradually
   and incrementally, without major rearchitecturing like I did in the
   original gsm-fw project that later became Citrine.

The second approach is what I started 3 weeks ago with FC Magnetite,
at a time when I thought Citrine to be unfixable.  The fear that Citrine
was unfixable has now been disproven, but Magnetite has already been
created and advanced forward quite a bit in the meantime, hence both
firmwares will now be with us for a while.

Compared to Citrine, Magnetite has the downside of compiling with TI's
proprietary TMS470 compiler (TCS211 version) under Wine, but it has
the advantage of full TCS211 functionality: it's all there.  However,
unlike straight TCS211, Magnetite is NOT all blobs: very considerable
progress is being made on deblobbing in the Magnetite tree *without*
sacrificing functionality.

We had already deblobbed most of L1 in the TCS211 environment earlier
this year, but the most recent major deblobbing done in the Magnetite
environment is the G23M protocol stack.  Our copy of TCS211 came with
the G23M PS in blobs, hence our non-blob version of this PS comes from
the TCS3.2/LoCosto source instead.  We have used this LoCosto-based
G23M PS successfully in Citrine, but that one is the voice+SMS subset:
no FAX_AND_DATA, no GPRS.

My most recent proud accomplishment of the TCS2/TCS3 hybrid config in
the Magnetite environment takes it a step further: just like Citrine,
this new config uses the LoCosto source versions of all G23M components
- no more Sotovik blobs! - but it has not only the voice+SMS subset,
but also FAX_AND_DATA and GPRS!

Zooming out to the bigger picture, the FreeCalypso project overall has
two principal goals: a libre dumbphone and a FreeCalypso GSM/GPRS modem
like Openmoko's.  Although my decision in this regard will almost
certainly be unpopular, for budgetary reasons I have decided to pursue
the FC modem goal first, pushing the libre dumbphone goal further down.
Yes, the reasons are budgetary - let me explain.

The biggest issue with the libre dumbphone goal is that I personally
see both Mot C1xx and Pirelli DP-L10 targets as dead ends, and I do
not have the motivation to invest oodles of effort into making a libre
dumbphone out of one of these pre-existing phone hw units.  Instead
the only dumbphone firmware work (UI, battery management etc) which I
am personally interested in doing would have to run either on TI's
D-Sample kit or on our own hardware.  But the D-Sample has the wrong
version of the core chipset, a version for which we lack the corresponding
fw support, and the chances of us finding the source pieces that would
be needed to run our own fw on the D-Sample are vanishingly slim -
thus building our own hw with the Calypso+RF chipset version for which
we do have fw support is the only practically available option.

After we build our way-overdue FCDEV3B board, I already have plans for
the next FreeCalypso board: HSMBP, which stands for Handset Motherboard
Prototype - it will be exactly what the name says.  One of its key
features will be a 176x220 pix 16-bit color LCD exactly like on TI's
D-Sample and later development platforms - it will finally allow us to
see TI's D-Sample-targeting UI which we got with TCS211, and use it as
the starting point for our FreeCalypso UI work.

But of course there is an obvious budgetary problem with this plan: we
still have not got the funds to build our _first_ FreeCalypso board
(the FCDEV3B), hence the timeline for building our second FC board
*after* FCDEV3B can only be very indefinite.

But whereas the next HSMBP board is only a concept idea in my head at
the present, for the FCDEV3B we already have a ready-to-build PCB
design.  So much has already been invested into the FCDEV3B goal that
we absolutely *have* to build it - it is like a baby in my womb, and
it is time for me to give birth - no going back.  Thus even in the
most pessimistic case of no one else supporting this project and me
having to fund it entirely on my own after taking care of the family
commitments to which my funds are currently allocated, the board will
still get built no later than the end of 2017, my current worse case
guesstimate.

Given this situation where the FCDEV3B has to get built soon by hook
or by crook whereas the HSMBP is a further-down goal for possibly much
later, I have decided to set the firmware work priorities accordingly:
prioritize that fw work which can be productized on the FCDEV3B and
similar sans-UI modem targets, while pushing all handset UI and related
work way down priority-wise.

So this is where things stand currently.  Coming up next, I am going
to update our website with the new goal structure and the work we are
doing toward these goals, and then launch the no-time-limit
crowdfunding campaign for the FCDEV3B.

Happy hacking,
Mychaela