FCDEV3B hardware bug: sleep mode self-reboot

[Mychaela Falconia]

Hi DS,

Your idea of putting a 2nd scope channel on the UART Tx line in order
to correlate the voltage rail trace to what the modem was doing is
brilliant!  But it also shows that the noise seen on the V-DBB rail is
probably a false lead: the correlation with UART Tx activity clearly
shows that the noise appears when the new boot cycle messages are
emitted, i.e., after the erratic reboot itself has already happened,
and thus does not appear to be a triggering cause.

In your DS1Z_QuickPrint1.png capture the time window of interest is in
the approximately 1.5 s of "silence" on the UART Tx line before the
reboot: this is the time window during which the erratic self-reboot
bug has to be manifesting, yet nothing stands out visibly on the V-DBB
rail during this window.

I do have a few other interesting observations though, but I need to
explain some background first.  TCS211 fw configures L1 on boot to do
an ADC measurement run every 102 TDMA frames (about 470.77 ms) when in
"cell selection mode 0", i.e., the completely idle state when no radio
bring-up commands have been issued yet - these are the L1 ADC messages
we've all seen pouring continuously out of an idle modem.  On boot all
sleep modes are enabled, and L1 will actually put the modem into deep
sleep during each of those 470 ms or so windows between ADC wake-ups.

Your latest scope captures show what these ADC sleep-wake sequences
look like *after* the reboot in each case.  Notice how the V-DBB
voltage dips during the awake time and rises during sleep, and also
note the peculiar behaviour of the UART Tx line: it is logically high
throughout each of the inactivity windows between ADC wake-ups, but
the actual voltage rises in an RC fashion as the sleep progresses.  I
assume that we are seeing changes in the V-IO supply voltage, which
probably also dips during awake times and rises during sleep.

Now consider what happens when an AT+CFUN=1 command is issued.  When
no erratic self-reboot occurs, i.e., on our current boards with small
sleep disabled or in any sleep mode setting on non-affected pre-existing
Calypso targets, this command takes a few seconds to complete, i.e., a
few seconds pass before OK is returned.  With the SIM I am using (I
don't know if it is SIM-dependent or not), I see 9 or 10 ADC message
pairs in the rvinterf output in between the AT+CFUN=1 command and the
OK response.  Meanwhile L1 stops doing big sleep or deep sleep, which
is why you are not seeing the same behaviour (up/down steps on V-DBB
and the RC-style voltage climb on the UART Tx line) during this time
as after a reboot.

When I issue AT+CFUN=1 with small sleep (or all sleep modes) enabled,
I usually get 4 or 5 ADC outputs between the command and the reboot,
although it can be more: I just repeated the offending operation a few
times as a test, and one time I got 7 ADC outputs before the reboot,
i.e., it was getting close to the point where it would have returned
the OK response.  Thus if you are looking at a scope capture and you
see that the UART Tx activity associated with the ADC output suddenly
stops a few ADC intervals after the AT+CFUN=1 command, you know *that*
is the point where the Calypso suddenly capsized.  What I am trying to
capture (and compare between our ailing FCDEV3B and working Openmoko-
made units) is the system behaviour at various probe-able points in
the time window between the issuance of AT+CFUN=1 and the sudden
cessation of ADC UART output.

At this point further investigation will probably need to wait until I
acquire my own oscilloscope and can do my own probing, both on the
FCDEV3B as well as on an Openmoko-made GTA02 board for comparison.
Yesterday I did the experiment with powering up one of my bare GTA02
MBs (the one on which I also removed the WLAN daughterboard and the
easily removable top cover part of the metal shield) from a bench
supply, with alligator clips on the battery connector, using Om's NOR
U-Boot and the USB interface to it for turning on the Calypso and
connecting it to the headset jack, and it worked.  The next step in
that direction will be to reflash the modem on that GTA02 MB with our
own Magnetite fw that supports AT commands via fc-shell (a feature of
our own invention not present in Om's historical firmwares, and needed
in the present case because the internal UART is not accessible when
using only U-Boot on the AP), put a SIM card in the socket and see if
AT+CFUN=1 works in this arrangement.

> This is a DS1054Z. It has 1 GSa/s and up to 100 MHz on one channel
> (it is normally locked to 50 MHz, but there exists a free online generator
> that allows unlocking features). I do recommend though that you buy it
> new though instead of a used one (it costs ~$399 new), for instance from
> https://www.amazon.com/dp/B012938E76 or https://www.adafruit.com/product/2145

I like your recommendation, and I will probably buy one after I read
the documentation to make sure it can do what I have in mind.

My idea is to put one of the scope channels on the UART line that
carries commands to the Calypso (RxD from the modem's perspective) and
use it as the trigger, i.e., have the scope trigger when rvinterf
sends the binary packet with the AT+CFUN=1 command in it.  Have two
other channels on the UART Tx line so I can see what the modem is
doing, and on whatever actual signal I am trying to health-check, i.e.,
using 3 scope channels in total.  I hope that the scope has a capture
window that spans several seconds (your latest captures are 12 s if I
am reading them correctly, so hopefully no problem there), and can
start the capture window at some defined time before the trigger point.

But I already see that it will be a *huge* amount of work...

> No, AT%SLEEP=2 is required. Otherwise AT+CFUN=1 will always reboot.
> However I thought AT%SLEEP=0 would reenable sleep modes, which I realize
> now was a mistake.

The sleep modes are defined in l1_const.h:

#define NO_SLEEP              00   // ------ + ------ + ------
#define SMALL_SLEEP           01   // SMALL  + ------ + ------
#define BIG_SLEEP             02   // ------ +   BIG  + ------
#define DEEP_SLEEP            03   // ------ +   BIG  +  DEEP
#define ALL_SLEEP             04   // SMALL  +   BIG  +  DEEP

The boot-up default set in cst_pei.c is ALL_SLEEP (all sleep modes
enabled), corresponding to AT%SLEEP=4.

> > An fc-fsio upload to an otherwise idle modem produces a situation that
> > is similar to what happens on AT+CFUN=1: no Nucleus tasks are running,
> > thus the Nucleus scheduler falls into the idle loop which activates
> > small sleep, but the UART interrupts act similarly to the SIM interface
> > interrupts in the other case, producing a lot of short sleep-wake
> > sequences back to back in rapid succession.  But it is interesting to
> > note that the behaviour in this case is a dead freeze, rather than a
> > reboot.
>
> That's an interesting find! Perhaps a bug in TI's interrupt handling
> mechanism them.

Please remember that just like AT+CFUN=1, the fc-fsio upload operation
in question works flawlessly on *all* pre-existing Calypso targets,
and runs into trouble when sleep is enabled *only* on our own FCDEV3B.
Thus bugs in TI's firmware cannot be the culprit here, the problem is
clearly hardware rather than fw, and I have every reason to believe
that the actual problem is exactly the same in the SIM case and in the
UART case, even though the latter manifests as a dead freeze rather
than a reboot.

It is also worth noting that when I did the AT+CFUN=1 bombing operation
several times in a row as a test, one time it did produce a similar
dead freeze (rvinterf output stopped) instead of the usual reboot.

M~