FC loadtools sped up with new binary protocols

[Mychaela Falconia]

Hello FreeCalypso community,

I have now implemented the new binary protocols between loadtools and
loadagent for fc-loadtool memory dumps, fc-loadtool flash programming
and fc-xram loading of run-from-RAM fw images, and the performance
improvement is as good as I was hoping for: flash dumps are just a bit
over 2x faster, flash programming is between 1.5x and 2x faster, and
fc-xram got the most dramatic speed-up: between 3.5x and 5x.  The new
code is in the freecalypso-tools Hg repository, along with updated
documentation.

I haven't done any direct performance comparisons between our new
fc-loadtool and TI's FLUID (I won't be able to make this comparison
until I first produce my own Unix/Linux port of FLUID, as we don't
have the source for Openmoko's port), but my expectation is that FLUID
will still be a little faster, although our current fc-loadtool should
definitely fare a lot better than our previous versions.  Our previous
fc-loadtool versions were a definite loser on any performance metric
because of our hex-based serial protocol (transferring two bytes for
every byte of payload); our current binary protocol is a definite
improvement, but FLUID is very aggressively optimized for speed, in
ways that would be quite difficult to replicate in our architecture.

Of course FLUID has plenty of other design flaws that make it a dead
end despite its good speed performance, the most notable limitation
being target support: FLUID will work great on TI's own D-Sample (I
have what may be the only such board remaining in the world), on
Caramel development boards (Das Signal and I have the last two of
those boards in the world) and on Openmoko's modem (the only FLUID-
supported Calypso device physically available to people other than me
and DS), but it cannot be made to work on our FCDEV3B without very
significantly changing its flash handling (chip detection, geometry
and write algorithm selection) to be more like ours, and of course
FLUID is totally unsuitable for Compal targets.

Given that switching to FLUID for production use is not a viable
option, I have been working to improve our fc-loadtool to a point
where it will compete fairly with FLUID on functionality and come
reasonably close in performance.  Functionality is an issue too, not
just performance: only very recently (fc-host-tools-r12 last week) our
flash program-m0 command was brought up to par, allowing us to program
a discontiguous m0 fw image with large gaps like FLUID does without
taking a big penalty hit one way or another, and only this weekend I
implemented the new e-program-m0 command (like program-m0, but with a
built-in erase of the needed sectors) that allows an m0 image to be
programmed with the same degree of user agnosticism as with FLUID.
The new doc/Flash-programming article in freecalypso-tools explains
the issues.

There are only two more code changes I would like to make before
putting out the new fc-host-tools-r13 release:

1) I need to extend fc-loadtool's batch mode a little bit.  Right now
fc-loadtool has a mode where it executes a command script and exits; I
need to extend this batch mode so that our new very high-level commands
e-program-bin, e-program-m0 and e-program-srec can be used in batch
mode directly from the command line, without needing a command script
file.  This functionality extension is needed in order to compete with
FLUID for automated batch-mode flash programming.

2) Both FLUID and our own loadtools programs wait forever for Calypso
boot ROM to respond to interrupt-boot beacons being sent continuously
every 13 to 15 ms.  This forever wait works great for manual usage
where the operator presses PWON or otherwise causes the Calypso target
to boot and is there to take appropriate action if the boot ROM entry
fails to happen, but is not good for automated environments with
target boot control - see doc/Target-boot-control in freecalypso-tools.
Openmoko ran into this problem with FLUID when they were building an
automated (not Calypso-intelligent-user-driven) process for firmware
updates, and the solution they ended up implementing for detecting
failures and retrying is gawdawful - just look at the script inside
their moko11 uSD card image.  I am going to implement an option in all
of loadtools programs (to be used for automated batch operation with
target boot control) to put a time limit on the wait for Calypso boot
ROM response, producing an error exit (nonzero process exit code)
instead of a hang when the Calypso target fails to behave as expected.

I need to make these two remaining changes and some more documentation
updates and other polish, but otherwise the current code is a release
candidate - please test if you can.  Please note that testing the
current not-yet-released code will require compiling the new version
of loadagent with the ARM7 gcc+binutils toolchain.

Hasta la Victoria, Siempre,
Mychaela aka The Mother

P.S. to Das Signal: if you are going to play with FLUID on your Caramel
board, please please please make a flash backup from your board with
fc-loadtool (not FLUID) first, and save it very securely.  The RF
calibration values on your current board are going to be different from
the first board which you sent to me and whose flash we've already
saved.