--- a/doc/TCH-bit-access	Wed Dec 28 10:05:46 2022 +0000
+++ b/doc/TCH-bit-access	Thu Dec 29 09:32:31 2022 +0000
@@ -2,17 +2,19 @@
 implements one nifty feature which is not used at all in standard phone or modem
 operation, but which can be used for all kinds of interesting hacks: the traffic
 channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink
-direction (to be fed to one of several speech decoders) can be read out of the
-DSP's API RAM in real time, and in the uplink direction the user can feed her
-own bits to the input of the GSM 05.03 channel encoder, effectively suppressing
-the output of the internal vocoder.
+direction (to be fed to the channel mode-appropriate speech decoder) can be read
+out of the DSP's API RAM in real time, and in the uplink direction the user can
+feed her own bits to the input of the GSM 05.03 channel encoder, effectively
+suppressing the output of the internal vocoder.
 
 The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel
-modes, corresponding to the FR and EFR codecs; it is not currently known if it
-can also be used in TCH/HS, data traffic or AMR modes.  I am aware though of an
-anecdotal report that someone tried to make this feature work with AMR, but was
-unsuccessful - hence we should be prepared for the possibility that the hack
-is not possible with AMR.
+modes, corresponding to FR1 and EFR codecs; it also appears to work for TCH/HS
+(HR1 codec), but we (FreeCalypso) haven't tested it because almost no one uses
+that infamous HR1 codec - the commercial GSM network in our part of the world
+gives you a full-rate channel if your phone does not support AMR.  It would be
+possible to implement HR1 in our own test GSM network, but the effort that would
+be required is difficult to justify.  Exploring TCH tap modes with AMR or CSD
+traffic channels is likewise a subject for further study.
 
 In order to make use of this TCH bit access feature, one needs 3 things:
 
@@ -28,10 +30,14 @@
 bits into and out of Calypso GSM devices running one of our firmwares in the
 form of an extension to TI's RVTMUX interface, i.e., we have defined a new
 RVTMUX channel for this TCH interface and defined the packet types and formats
-to be sent over the wire.  On the Calypso side this interface is implemented in
-FreeCalypso GSM firmwares (currently only in Citrine, but we can also make a
-special version of TCS211 with this feature added if the need arises), and on
-the host tools side there is support in rvinterf and fc-shell.
+to be sent over the wire.  On the Calypso side the special functionality in
+question was originally implemented in FC Citrine firmware in 2016 and then set
+aside for some years; when the right time came to resurrect this feature in late
+2022, it turned out that the original implementation from 2016 was slightly
+incorrect, and the new implementation in FC Tourmaline fw is slightly different.
+On the host tools side the RVTMUX-based TCH interface is supported in rvinterf
+and fc-shell; the new version as of fc-host-tools-r18 supports both 2016 and
+2022 versions of this over-the-wire interface.
 
 The TCH bit access mechanism in FreeCalypso has been designed with an objective
 of presenting to the user exactly what TI's DSP presents to us, i.e., standing
@@ -53,6 +59,11 @@
 these hexadecimal strings correctly, you (the user) need to understand the bit
 order and mapping used by TI's implementation of the GSM 05.03 channel encoder.
 
+As of late 2022, there is a new TCH-tap-modes article in our freecalypso-docs
+repository that covers in detail the format of TI's DSP buffers for TCH DL and
+UL bits, as well as all known information about TCH DL status words and bit
+flags.  But here is our original description from 2016:
+
 Recall from the GSM specs that the 260 bits which comprise one speech frame are
 not all treated equally, instead they are divided into 182 class 1 bits which
 are protected by a convolutional encoder and 78 class 2 bits which are
@@ -77,51 +88,7 @@
 * Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the
   standard GSM 05.03 order.
 
-Uplink testing
-==============
-
-The uplink sending mechanism can be exercised as follows:
-
-1. Record a speech sample in the GSM 06.10 codec format using any Unix/Linux
-   audio tool that can write the de facto standard libgsm format.  For example,
-   using SoX:
-
-   rec -c1 recording.gsm
-
-   SoX will write the recording in the GSM 06.10 libgsm format based on the
-   .gsm suffix at the end of the recording file name; the -c1 option is needed
-   to disable stereo, otherwise the recording will be slowed down 2x.
-
-2. Convert it from libgsm standard format into our ad hoc hex strings for
-   playing into the TCH uplink:
-
-   fc-fr2tch recording.gsm recording.tch-ul
-
-3. In fc-shell, when you are in an established voice call, issue this command:
-
-   tch play recording.tch-ul
-
-You should now hear the speech sample you recorded in step 1 above on the other
-end of the GSM call.  Please note, though, that for this example to work, the
-call must be connected in the FR codec mode, not EFR.  If you try it on an EFR
-call, the vocoder on the other end of the call will try to interpret your FR
-codec bits per EFR, resulting in garbage.  In order to make this procedure work
-properly with EFR, you would need to generate a speech sample in EFR format and
-then put it in the correct hex string form with the correct bit order for
-feeding to TI's GSM 05.03 channel encoder implementation, and at the present
-moment the tools to do this feat are lacking.
-
-The fc-fr2tch step above is new with the current version of FreeCalypso host
-tools.  My original implementation of the tch play command in fc-shell read
-libgsm GSM 06.10 speech files directly, but I changed it to read hex strings
-which correspond literally to what goes into TI's implementation of the 05.03
-channel encoder to make it more general:
-
-1) to support playing EFR speech bits into TCH/EFS channels;
-2) to support people who may want to pass totally non-speech data over traffic
-   channels in TFO mode - see below.
-
-Downlink testing
+TCH DL recording
 ================
 
 When you are in an established voice call in fc-shell, you can record the
@@ -140,99 +107,102 @@
 until you issue this tch record stop command, the output file is not closed and
 thus may not be written to the file system.
 
-The recording is written in an ASCII format that is similar but not identical
-to what the tch play command takes as its input.  Here is an example of what
-you might get from tch record:
-
-C214 281D 0063 81C008000008200046000000000000000000000000000007056608060B0A010B09
-C204 0540 003A 9391480D3051F4BD81DF5EB35069BEBC4AEDF756351C4C19689BB1CA4EA5D4F5F5
-C204 05A2 0047 65B80F9E690F7C8CE4ED80DEF69DA6436518AB99ABCE5815E6B562C5CE4EAC5DC5
-C204 04CA 0044 A4483744B04371ED0334ECB350AF28C639B7F095519EF0242D299B6405124F77A5
-C214 2544 0066 83800400000C1000E80000000000000000000000000000300674A07070F080D0B2
-C200 4E8A 0000 07071DF0F83B9521EE61CFF095AA8C0E560300F6A5573C31F3E00601ED4AAE7E2F
-C200 4C94 0000 077385ADE20450B2E410961D6C5B0A173ACF9E2D38D77C28CED8495D88AA4DE72E
-C200 4ABC 0000 077F0CF86004132DA5C0A6D5A4B0BD4B28159A07D8F4282DC6AAAB27503BC02701
-
-The example above is quoted from an actual recording made from a call to WWV at
-+1-303-499-7111, a time of day service.  This example shows 5 garbled frames
-followed by 3 good ones.
-
-In each line the first 3 space-separated 16-bit words are status words from the
-DSP, and the rest is a hex string of 66 digits giving the 260-bit frame with 4
-extra dummy bits at the boundary between the protected and unprotected portions.
+The recording is written in an ASCII line-based format with one line for every
+received TCH DL frame, but the exact format of each written line will depend on
+which firmware version is in use.  If you are running ancient Citrine firmware
+that emits its TCH DL output in the old format from 2016 (now known to be
+incomplete and thus unusable for proper decoding), fc-shell will likewise write
+its ASCII output in the old format, which won't be covered further as it is
+deprecated and not practically useful.  However, if you are running current
+FreeCalypso firmware with the resurrected (late 2022) version of the TCH tap
+feature, each TCH DL frame will be sent by the fw and received by fc-shell in
+the new over-the-wire format, and fc-shell will write the recording file in the
+new ASCII format documented in the TCH-tap-modes article in freecalypso-docs.
 
-Now here comes the unpleasant part: radio systems are inherently error-prone,
-hence if you are naively expecting every received downlink frame to be a valid
-speech frame that matches exactly what the other end transmitted, you will be
-disappointed.  The downlink frame stream will always consist of both valid and
-invalid frames, and in the standard processing chain inside TI's DSP the speech
-decoder block that receives these frames from the channel decoder will look at
-the status words written by the latter in order to decide what to do with each
-frame.  But unfortunately I don't know the details, as the DSP is a mostly
-undocumented black box.
+Once you have captured a TCH DL recording, what can you do with it?  If the
+recording came from an FR1 call, you will need to pass it through an Rx DTX
+handler for FR1 (see GSM 06.11, 06.12 and 06.31 specs) before you can pass it
+to a naive GSM 06.10 decoder such as classic Unix libgsm, and if the recording
+came from an EFR call, you will need to pass it to a proper EFR (not AMR!)
+decoder that includes the necessary EFR Rx DTX handler.  Neither of the two
+just-mentioned library pieces (neither the Rx DTX handler for FR1 nor a proper,
+not-same-as-AMR implementation of GSM EFR) could be found among the existing
+body of FOSS as of 2022, thus we (FreeCalypso and Themyscira Wireless)
+implemented our own.  Please look for our GSM codec libraries & utilities
+package, which is expected to reach its first official release some time in
+early 2023.
 
-Our FreeCalypso firmwares forward the downlink frame status words from the DSP
-to the external host and my current implementation of the tch record command
-writes them into the file because it is valuable information which should not
-be discarded, but unfortunately I don't know their meaning.  The first status
-word consists of bit flags, the last one seems to be some kind of error count
-(what kind of errors? I don't know), and the middle one is totally unknown.
+Inside our gsm-codec-lib package you will find gsmfr-dlcap-* and gsmefr-dlcap-*
+utilities that read TCH downlink capture files written by fc-shell tch record
+and perform various decoding operations - please refer to further documentation
+within that package.
+
+Please don't use the old fc-tch2fr utility - the function it performs is now
+known to be a bogo-transform, and it can't grok the new TCH DL recording format
+which you will get with current FreeCalypso fw.
 
-There is also an fc-tch2fr utility that does sort of an inverse of fc-fr2tch,
-except that fc-tch2fr expects files written by tch record which contain the
-status words at the beginning - the files written by fc-fr2tch and read by
-tch play don't have these status words; they just have 66 hex digits per line
-giving the bits to be fed to the GSM 05.03 block in the DSP.  The fc-tch2fr
-utility works by disregarding the status words; the result will be a .gsm file
-which you can play with SoX etc, but it will produce horrible sounds wherever
-some garbled frames were received on the call downlink.
+TCH UL play
+===========
+
+The uplink sending mechanism can be exercised as follows:
+
+1. If you are going to be in an FR1 call, prepare a speech sample in the GSM
+   06.10 codec format using any Unix/Linux audio tool that can write the de
+   facto standard libgsm format.  For example, using SoX:
+
+   rec -c1 recording.gsm
 
-Passing non-speech data over TCH
-================================
+   SoX will write the recording in the GSM 06.10 libgsm format based on the
+   .gsm suffix at the end of the recording file name; the -c1 option is needed
+   to disable stereo, otherwise the recording will be slowed down 2x.
+   Alternatively, you can use our new gsmfr-encode utility (gsm-codec-lib
+   package) to encode from WAV into GSM 06.10, or gsmfr-encode-r for raw BE
+   input instead of WAV.
+
+   OTOH, if you are going to be in an EFR call rather than FR1, you will need to
+   prepare a speech sample in the EFR codec format instead.  You will need to
+   use Themyscira gsmefr-encode or gsmefr-encode-r utilities, or convert from
+   AMR (MR122 mode only, no DTX) with our gsm-amr2efr utility.
 
-If you are incredibly lucky and you happen to live in a part of the world where
-the local GSM network implements TFO (tandem-free operation, see GSM 02.53),
-i.e., if your GSM network transparently passes codec frames from one end of the
-call to the other without transcoding back and forth in the middle, you might
-be able to send arbitrary non-speech data bits over this TCH connection.  If
-you are going to attempt such a feat, you are mostly on your own as the GSM
-network in my part of the world does not support TFO, but here are some general
-ideas:
+2. Convert your speech sample from libgsm standard format (FR1) or Themyscira
+   gsmx format (EFR) into our ad hoc hex strings for playing into a TCH uplink:
+
+   fc-fr2tch recording.gsm recording.tch-ul
 
-* Heed the difference between the 182 protected bits and the 78 unprotected
-  bits of each 260-bit frame.  The bits in the first class are less likely to
-  be corrupted by radio errors.  Also the first 50 out of these class 1 bits
-  are protected by a CRC, so if they get corrupted, there will be a flag bit
-  somewhere in the DSP status words - but I don't know which bit it is...
+   or
+
+   fc-efr2tch recording.gsmx recording.tch-ul
 
-* Given the lack of detailed information about the exact meaning of the DSP
-  status words, implement some error checking of your own, e.g., an in-band
-  CRC within your frame payload so you can check on the receiving end if you
-  got an uncorrupted frame from your data source or not.
+3. In fc-shell, when you are in an established voice call, issue this command:
+
+   tch play recording.tch-ul
 
-* Expect to lose some frames - even if your radio environment is pristine and
-  error-free, some frames will still be lost because their timeslots had to be
-  stolen by FACCH on either end of the call.  You will need to implement
-  acknowledgements, retransmissions and retries - the usual stuff.
+You should now hear the speech sample you recorded in step 1 above on the other
+end of the GSM call.  Needless to say, the TCH mode of the call (TCH/FS or
+TCH/EFS) needs to match the codec in which your to-be-played recording was
+prepared, otherwise the other end of the call will receive garbage!
 
-EFR differences
-===============
+Controlling the selection of speech codec for calls
+===================================================
 
-TCH/FS and TCH/EFS channel modes corresponding to the FR and EFR codecs,
-respectively, share the same channel encoder that takes 260-bit frames as input.
-An EFR speech frame is only 244 bits, but as the GSM 05.03 spec explains, each
-EFR frame gets expanded to 260 bits by adding some CRC and repetition bits.
-One would thus expect that the behaviour of the channel encoder block should be
-strictly identical between TCH/FS and TCH/EFS channel modes, but it appears that
-TI's implementation exhibits one difference in behaviour between the two modes.
+One very obvious shortcoming of the present facilities for voice TCH redirection
+is that we only support FR1 and EFR codecs, but not AMR.  However, most GSM
+networks prefer to use AMR when the MS supports it - and in regular operation
+with a speaker & mic user connection (as opposed to TCH tap modes), our current
+FreeCalypso firmwares do support AMR when running on Calypso C035 silicon with
+DSP ROM version 3606.  (DSP ROM version 3416 together with the respective patch
+version also appears to have working AMR, at least in light testing, although
+of course we do NOT recommend it for production use.)  Therefore, if you wish
+to play with EFR, you need to tell the network (via the Bearer Capabilities
+information element in CC messages) that your MS does not support AMR, and if
+you wish to play with FR1, you need to tell the network that your MS only
+supports FR1 and no others.
 
-It appears that TI's implementation of the GSM 05.03 channel encoder computes
-the EFR-specific CRC bits internally, such that bits [65,72] of each uplink
-frame fed to the DSP in the TCH/EFS mode need to be 0 and will be replaced by
-the DSP with CRC bits computed per the standard.  It also appears that their
-implementation of the channel decoder verifies this CRC, sets a status bit
-indicating whether the check succeeded or failed, and then zeroes these bits
-out, i.e., the original bits in these CRC bit positions are lost.  No problem
-if what you are passing over TCH/EFS is indeed EFR speech frames, but it will
-be a problem if you are passing non-speech data over your TCH and put something
-else in those bits which the spec allocates to CRC.
+The outstanding issue here is that we need to implement some mechanism in our
+FreeCalypso firmwares, probably a custom AT command, that will modify the
+Bearer Capabilities IE (artificially restrict the set of supported codecs) on a
+per-call basis.  Until we implement the necessary support, the only current way
+to create such codec-restricted operation is by writing a /pcm/MSCAP file into
+FFS with the desired bit mask of supported codecs - but this method is hugely
+inconvenient because this file is read only on fw boot, thus every modification
+requires a full reboot cycle.