FCDEV3B hardware bug: sleep mode self-reboot

[Mychaela Falconia]

Hi DS,

> I had no problem identifying the three caps, and indeed the voltage is
> at 1.5V which is expected. In the three cases, I have issued the AT+CFUN
> command while observing the output on the scope.

OK, so far, so good.

> There was no voltage drop,
> however there was some slight "noise", which is to say the voltage varied
> a little bit around 1.5V (at most +/- 0.1V). However this could be due to
> the power draw when the board was rebooting, as this noise appears much
> after the AT command. Please have a look at the oscilloscope screenshot
> that illustrates the situation:
> https://www.freecalypso.org/members/ds/NewFile1.png

Aha, we have at least *some* visible behaviour change on the power
rail: an increase in the noise relative to the perfect 1.5 V.  You are
of course correct that it is related to the power draw, but what we
don't know is whether this power draw-related noise only happens as a
consequence of the reboot, or if it happens as a result of the Calypso
going rapidly in and out of small sleep, and is a prelude to the
reboot.

At this point two questions are in order:

1: Which of the three probe points did your posted o'scope trace come
from: C214, C211 or C212?  Was the noise ripple exactly the same as
far as you could tell at all 3 points, or was it greater or lesser at
one of the points?

2: One experiment that might be able to shed some light on whether the
power supply noise ripple we are seeing is a consequence of the reboot
or a prelude to it and possibly its cause would be to issue a tgtreset
command through fc-shell instead of AT+CFUN=1.  This command directly
triggers a reboot through the watchdog, without the flurry of very
short sleep-wake sequences which is the prime suspect in this
investigation, so if you issue tgtreset while holding the scope probe
on the V-DBB probe points, we'll see what a reboot looks like by
itself, without the flurry of sleep-wake sequences.

Another very important experiment would be to probe the same power
rail on an Openmoko-made GTA02 board which does not exhibit the self-
reboot behaviour, and see if similar noise appears there or not.  As
it happens, I have two "bare" GTA02 motherboards (no case or display)
bought as scrap from GDC, and on one of them I have already removed
the shieldcan top cover over the modem section and the WLAN
daughterboard which gets in the way.  I am going to try powering it up
from my bench power supply with alligator clips on the battery
connector, to see if I can get a live NOR U-Boot shell this way
(either through USB or through the debug board connector), and then
enable and boot the Calypso from U-Boot.  If I can power up and boot
the Calypso modem on this bare GTA02 MB in this manner, I will have
the first part of the setup for probing the V-DBB net on a working
Openmoko-made unit.

The second required part would be for me to acquire my own oscilloscope
- as I wrote previously, I am already looking into it.  Would you mind
telling me what kind of oscilloscope you have used?  Looking on ebay,
I see a number of Rigol scopes with 500 MSa/s or even 1 GSa/s sampling
rates for around $400 or even less, which is definitely affordable, so
I wonder if I can get the same scope you have or maybe even better.

> I've noted that the radio part of the board does not seems turned on
> until after AT+COPS=0 is issued;

Correct: in the TCS211 version of the fw the AT+CFUN=1 command does
not send any requests to L1, thus the radio does not get turned on at
all until the AT+COPS=0 command.  OTOH, the TCS3.2 version of the G23M
PS we got from the LoCosto source which we use in our TCS2/TCS3 hybrid
firmwares (Magnetite hybrid and Citrine) does send a request to L1 to
do a preliminary scan (power measurement) across all channels upon
AT+CFUN=1, so the radio does get turned on - but only for Rx, not Tx.
The very first time a GSM MS turns on its transmitter is when sending
a RACH for a Location Update (registering to the network), and that
step can only happen after the MS has received, synced to and selected
a cell to try registering to.

> hence the possibility that the RF part
> is not (at least directly) responsible for this situation

Correct, I have every reason to believe that the RF part has nothing
whatsoever to do with the sleep mode self-reboot bug.

> (RF TX would
> likely be the most power hungry part of the board).

Yup, it's the RF PA.  But the PA draws its power from raw VBAT, not
from any of the regulators, and the VBAT supply to the PA is usually
routed separately from the VBAT supply to the chipset.  On BenQ's M32
modem there are separate VBAT pins for the two power consumers (3 pins
for the PA supply and just one for the chipset), on Openmoko's GTA02
only the VBAT supply for the chipset (Iota+Rita) goes through the
AP-controlled power switch while the PA has its own fat power trace
directly to the battery, and on our FCDEV3B the two VBAT legs join at
the JP2/JP3 jumpers, next to the big red tantalum cap (1000 uF).

> By the way, I've noticed something interesting:  if I wait until AT+CFUN=1
> receives ATI: OK, and then issue AT%SLEEP=0 then AT+COPS=0, it appears the
> board can successfully register without rebooting.

Wait a minute, are you saying that you are able to issue AT+CFUN=1
right away *without* preceding it with any sleep mode commands and get
an OK response without a reboot occurring right there at that step?
Or you are saying that you did AT%SLEEP=2 (big sleep only) before
AT+CFUN=1 and then AT%SLEEP=0 (disable all sleep modes) afterward?
Please clarify.

> If I'm not wrong, after
> AT+CFUN=1 the board starts talking to the (U)SIM card, which could perhaps
> be a culprit for the observed behaviour wrt/ sleeping.

Yes, the modem does a lot of message exchanges with the SIM upon
AT+CFUN=1, and it is my current working hypothesis that the sleep mode
self-reboot occurs when there is a flurry of back-to-back sleep-wake
sequences of very short duration in rapid succession, caused by the
structure of Nucleus tasks and interrupt service routines in the fw
wrt/ the SIM interface.

Another recent observation: last Friday I got the loudspeaker parts
from Digi-Key, and tried connecting a speaker to the FCDEV3B.  I will
write later in a separate post about issues with the speaker itself,
but I wanted to try playing some E1 melodies through it, so I started
by uploading the rich set of E1 melodies extracted from the TSM30
source:

ftp://ftp.freecalypso.org/pub/GSM/ringtone/tsm30-melody-e1.tar.gz

The actual upload operation was:

fc-fsio upload-subtree <host_source_directory> /mel

I got a perplexing surprise when the FCDEV3B would just freeze in the
middle of the large-ish fc-fsio upload operation.  I would reboot it
with the hardware RESET button, retry again: same effect.  Then I
disabled all sleep modes with AT%SLEEP=0 and tried the fc-fsio upload
again: this time it succeeded, like it has always worked reliably on
all pre-existing Calypso targets.

An fc-fsio upload to an otherwise idle modem produces a situation that
is similar to what happens on AT+CFUN=1: no Nucleus tasks are running,
thus the Nucleus scheduler falls into the idle loop which activates
small sleep, but the UART interrupts act similarly to the SIM interface
interrupts in the other case, producing a lot of short sleep-wake
sequences back to back in rapid succession.  But it is interesting to
note that the behaviour in this case is a dead freeze, rather than a
reboot.

To be continued when I get my own scope - so DS, please tell me what
scope you were using so I have a reference point for my search for a
scope for my home lab.

Hasta la Victoria, Siempre,
Mychaela aka The Mother