From laforge at gnumonks.org  Tue Sep  3 15:45:16 2019
From: laforge at gnumonks.org (Harald Welte)
Date: Tue, 3 Sep 2019 17:45:16 +0200
Subject: GTM900-like FreeCalypso modem module idea
In-Reply-To: <CA+uuBqa60zSBWYLh+45cYXEMJnjQ8PD-NrpFC1OgnowSyn2QWA@mail.gmail.com>
References: <CA+uuBqa60zSBWYLh+45cYXEMJnjQ8PD-NrpFC1OgnowSyn2QWA@mail.gmail.com>
Message-ID: <20190903154516.GF1841@nataraja>

Hi Mychaela,

On Fri, Aug 23, 2019 at 06:35:37PM -0800, Mychaela Falconia wrote:
> I have a new hardware idea: we can create a FreeCalypso modem module
> in the form factor copied from Huawei GTM900.  

While I praise your enthusiasm, my personal opinion on this is that it
looks like it's rather futile.  If the GTM900-B can be obtained at
incredibly low prices (one-digit USD figures), people who just want
to play with Calypso based devices are unlikely going to invest
a considerably larger (probably by two orders of magnitude) amount
of money in a "clone" just for MCSI and tri-band RF.

But by all means, don't let my negativity keep you from doing anything.

I personally just don't see the use case, sorry.  Interest in GSM is
decreasing very rapidly anyways, as are networks/deployments.  And
with plenty of inexpensive, old hardware around.

Just my two cents,
	Harald
-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

From mychaela.falconia at gmail.com  Tue Sep  3 21:05:17 2019
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Tue, 3 Sep 2019 13:05:17 -0800
Subject: GTM900-like FreeCalypso modem module idea
In-Reply-To: <20190903154516.GF1841@nataraja>
References: <CA+uuBqa60zSBWYLh+45cYXEMJnjQ8PD-NrpFC1OgnowSyn2QWA@mail.gmail.com>
 <20190903154516.GF1841@nataraja>
Message-ID: <CA+uuBqaVS=JzQcOxtwGJGe+-zX8O2AOrLwV2y6gSL_Y-m7JgCg@mail.gmail.com>

Hi Harald!

> > I have a new hardware idea: we can create a FreeCalypso modem module
> > in the form factor copied from Huawei GTM900.
>
> While I praise your enthusiasm, my personal opinion on this is that it
> looks like it's rather futile.  If the GTM900-B can be obtained at
> incredibly low prices (one-digit USD figures), people who just want
> to play with Calypso based devices are unlikely going to invest
> a considerably larger (probably by two orders of magnitude) amount
> of money in a "clone" just for MCSI and tri-band RF.

Just to be clear, none of my FC modem module ideas (neither the present
GTM900-like FCM40 idea nor the longer-standing SIM900-like SMT module
idea) are intended for people who "just want to play" with a Calypso
device.  For someone who "just wants to play", our FCDEV3B (specifically
made for playing on a lab bench, not for any kind of end use application)
is the ideal device, and for those who are too cheap to pay the fair
price for an FCDEV3B, there are indeed dirt-cheap options like old
Motorola phones and the recently discovered GTM900-B.  I would argue
that Mot C1xx phones offer the absolute lowest barrier to entry, even
lower than GTM900-B: with a Mot C1xx phone the only extra needed item
is a cable (readily available from your Sysmocom web shop or from
UberWaves on ebay), whereas with GTM900-B one also needs a breakout or
interface board, and I have yet to find one which is both readily
available and decent enough for me to be able to recommend.

Instead all of the proposed FC modem module ideas are intended to
address a very different target audience: vertical market applications.
Over the years that I've been running this project, there have been a
few people approaching me about using Calypso modems to do various
wildly non-standard things, hacks on GSM which no standards-following
GSM MS implementation will ever do, but which become possible given a
freely hackable modem.  Some of these people fall into the "researcher"
category, meaning that the scope of their hack is limited to cobbling
something together in their own lab, without making a volume product
out of it - these are the same kind of people who typically inhabit
the OsmocomBB camp, and whether they use OBB or FC as their firmware
starting point, they will definitely use something dirt-cheap like a
Mot C1xx phone for their hw platform.

Yet there are other people who also seek to do wildly non-standard
hacks on GSM, but unlike the "researcher" types, these other folks are
making or seeking to make some kind of volume commercial product in
which a hacked GSM modem doing non-standard things is a required
component.  These people are the only kind who might ever be willing
to pay for newly made Calypso modems (about $250 apiece in 100 piece
minimum order quantity), and so these are the people I had in mind
each time I came up with a FreeCalypso modem module idea, both the
original SMT module idea and the more recent GTM900-like FCM40 idea.

As a more specific case, there is one particular group of people who
are doing non-standard hacks with GSM, and their particular application
involves voice rather than data - a kind of GSM voice gateway, but
doing some non-standard things on the GSM side which no standard off-
the-shelf modem can do.  For these kinds of applications bringing a
digital voice channel out of the modem is strongly preferable over
analog, but none of the existing historical Calypso-based modem modules
(BenQ M32, Huawei GTM900-B) bring out MCSI, Calypso's poorly documented
external digital voice interface.  The FCM40 idea was targeted at that
market.

> But by all means, don't let my negativity keep you from doing anything.

Right now it is only an idea proposal and nothing more.  I have spent
a total of about 1h 40min on it (creating the MCL and the netlist in
the freecalypso-schem repo as a proof of concept), and I currently
have no concrete plans of doing much more unless some vertical market
customer expresses interest.

I am, however, thinking of designing and building a simple modem
module test board that would provide power, dual UART and SIM socket
connections to a GTM900-style module (FPC interface).  This simple
MMTB would serve two purposes:

1) If someone like Harald/Sysmocom would be willing to put it up in a
web shop (the design files will be public domain, and I may be able to
provide a few assembled pieces on consignment), preferably together
with the GTM900-B modules themselves, then the GTM900-B + MMTB combo
can become an officially endorsed platform, a low-cost alternative to
our much more expensive FCDEV3B.

2) If I ever do build my proposed FCM40 at least as a proof of concept
(the cost of design creation and prototype fab+assembly run would be
relatively inexpensive, something I might be willing to do after my
big surgery), then the very same MMTB will serve as a production test
and validation fixture for those FCM40 prototype pieces.

> I personally just don't see the use case, sorry.  Interest in GSM is
> decreasing very rapidly anyways, as are networks/deployments.  And
> with plenty of inexpensive, old hardware around.

My own interest in GSM is very much two-pronged: there are things
which I am interested in for my own enjoyment, and there are things
which I would be happy to do for a commercial (vertical market)
customer *if they pay*.  There is fairly little intersection between
these two areas.  The vertical market situation has already been
covered above; as for my own personal interest, I am most interested
in a FreeCalypso Libre Dumbphone handset.

And yes, there is of course the other problem of GSM networks going
away.  We have previously talked about me possibly buying a sysmoBTS
unit from you so I can set up my own GSM network for myself; while
there is no way I can afford it right now when I'm running a financial
marathon trying to get my sex correction surgery done on schedule, I
am hoping to be able to afford it some time in late 2020, after my big
surgery.

Moving beyond GSM, I would be quite interested in working on UMTS aka
3G - I already have the necessary test equipment as in my CMU200
equipped with 3G options - but I am NOT willing to skip 3G and jump
directly to LTE.  And when it comes to 3G/UMTS, my focus would be on
the UE side, not network side, and on circuit-switched voice services
rather than data.  But because I don't have any leaked sources and
docs for any 2G+3G (or 3G-only) chipset that are anywhere close to
what we have for Calypso, the only way how I can imagine building a
Libre UMTS UE implementation would be SDR-based - but at that point we
are straying so far away from what I consider to be fun that I would
be willing to work on something like that *only* if someone were to
fully cover my surgery cost in return.

M~

From mychaela.falconia at gmail.com  Sun Sep 22 03:50:58 2019
From: mychaela.falconia at gmail.com (Mychaela Falconia)
Date: Sat, 21 Sep 2019 19:50:58 -0800
Subject: Firmware deblobbing stats
Message-ID: <CA+uuBqZAvQxNBWdzRGDb=Yie5XufbS6ULfKmSrNKQaeaRkt6AQ@mail.gmail.com>

Hello FreeCalypso community,

As a little fun exercise, I have just written a tool that allows us to
quantitatively measure exactly how much we have deblobbed our TI-based
modem firmware.  For years I've been saying that our starting point
(the TCS211 semi-src that has been salvaged from the ruins of former
Openmoko) was approximately half source, half blobs, and that our
current FreeCalypso modem fw is almost completely blob-free - but of
course such statements are general terms, lacking quantitative
substance.  The new blobstat tool finally gives us the actual numbers
which we haven't had until now.

Given a final build product that has been produced from a mixture of
source components and linkable binary objects, just how can one
quantify precisely what percentage is source and what percentage is
blobs?  The answer can be obtained from the map file produced by the
linker.  Just like the more popular ELF, TI's object format (a COFF
variant) is based around sections: linkable objects consist of
sections, and so does the final link output.  Every byte in the final
fw image belongs to some section, and each of these output sections
from the linker's perspective is made up of various input sections
(meaning sections taken from linkable objects), with a few bits added
by the linker itself (long Thumb call trampolines and some filler and
padding bytes).  The map file produced by the linker shows the
allocation of every byte in the final fw image: it lists all generated
output sections, shows what input sections each output section is made
of, and shows all linker-added fillers and trampolines.

My blobstat program reads and parses these map files, taking account
of every code section that went into that final link.  It also reads
another file (a classification spec) that indicates which linkable
*.lib files (or which individual objects within these libs) should be
counted in the src category, versus which ones should be counted in
the blob category.  One can also define any other classification
categories as desired.

Let's look at our starting point first.  There are no surviving map or
COFF files corresponding to moko11 or moko10, but there is a surviving
map file corresponding to moko10-beta1; we can use this map file
because there is no difference in the state of source vs blobs between
moko10-beta1 and moko11.  Analyzing this gsm_<blah>.map file from
moko10-beta1 with blobstat, we get the following numbers:

* The total number of bytes in the final fw image that came from
linkable code bits (as opposed to linker-generated fillers and
trampolines) is 0x2156FC.  For comparison, the total image size is
0x2255B4 - there is almost 64 KiB of dead space in there, filled with
padding.

* The portion of the bits which were either compiled from source by OM
or for which they had the exact corresponding source which they chose
not to touch is 0xD3D34 bytes, or about 40% of the total.

* The portion of the bits coming from linkable *.lib files for which
OM did not have corresponding source is 0x1419C8 of the total.

Thus my original assessment of OM's firmware being about half source,
half blobs was pretty close to the true numbers, which turn out to be
60% blobs, 40% source.

Now let's look at our current FreeCalypso production firmware: namely,
the 20190409 build of FC Magnetite hybrid for the fcdev3b target.  Here
the total number of non-padding, non-filler, non-trampoline bytes (the
actual code size) is 0x23F7A4, and guess what the blob percentage is...
The only parts of FC Magnetite hybrid fw which exist as blobs with no
exact corresponding source are Nucleus, the OSL and OSX glue layers of
GPF, and the TMS470 compiler's RTS library.  These blob bits add up to
a grand total of 0xA82C (43052) bytes, comprising about 1.8% of the
total fw code size.  Thus we have gone from 60% blobs, 40% source to
98% source, 1.8% blobs.

So what are we going to do with these last remaining 43052 bytes of
code which we currently use in the form of binary objects with no
corresponding source?  Out of this entire blob division, the one part
which currently stands as the last remaining bone in our figurative
throat (OSL and OSX bits of GPF) weighs 0x3A90 (14992) bytes: about
one third of the total blob division, or just 0.6% of the total fw
code size.  Needless to say, a blob that weighs a total of 14992 bytes
and comes in the form of COFF objects with full symbolic info (-g
style) is very easy to reverse-engineer and thoroughly understand.
There are no mysteries in this OSL/OSX glue code, it is very thoroughly
understood - at least by me.  Instead the problem is that I am not
able to turn this disassembly understanding into recompilable C code -
more precisely, I am not able to produce os_???.c code that can be fed
to TI's TMS470 compiler (the specific version used in the TCS211
program) and which would produce output exactly matching the original
blobs.

I have recently written an article explaining the situation with these
OSL/OSX components and where our Magnetite and Selenite firmwares
stand with respect to them:

https://www.freecalypso.org/hg/freecalypso-docs/file/tip/Firmware-deblobbing

Oh, and the new blobstat program resides in the freecalypso-reveng
repository:

https://www.freecalypso.org/hg/freecalypso-reveng/file/tip/blobstat

Hasta la Victoria, Siempre,
Mychaela aka The Mother