# HG changeset patch # User Mychaela Falconia # Date 1672306351 0 # Node ID 3de3b34189be306633273c2fde23b471696c12c1 # Parent 94890123a74f04d1455ef6d127186a02c3b694fb doc/TCH-bit-access: update for newly resurrected version diff -r 94890123a74f -r 3de3b34189be doc/TCH-bit-access --- a/doc/TCH-bit-access Wed Dec 28 10:05:46 2022 +0000 +++ b/doc/TCH-bit-access Thu Dec 29 09:32:31 2022 +0000 @@ -2,17 +2,19 @@ implements one nifty feature which is not used at all in standard phone or modem operation, but which can be used for all kinds of interesting hacks: the traffic channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink -direction (to be fed to one of several speech decoders) can be read out of the -DSP's API RAM in real time, and in the uplink direction the user can feed her -own bits to the input of the GSM 05.03 channel encoder, effectively suppressing -the output of the internal vocoder. +direction (to be fed to the channel mode-appropriate speech decoder) can be read +out of the DSP's API RAM in real time, and in the uplink direction the user can +feed her own bits to the input of the GSM 05.03 channel encoder, effectively +suppressing the output of the internal vocoder. The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel -modes, corresponding to the FR and EFR codecs; it is not currently known if it -can also be used in TCH/HS, data traffic or AMR modes. I am aware though of an -anecdotal report that someone tried to make this feature work with AMR, but was -unsuccessful - hence we should be prepared for the possibility that the hack -is not possible with AMR. +modes, corresponding to FR1 and EFR codecs; it also appears to work for TCH/HS +(HR1 codec), but we (FreeCalypso) haven't tested it because almost no one uses +that infamous HR1 codec - the commercial GSM network in our part of the world +gives you a full-rate channel if your phone does not support AMR. It would be +possible to implement HR1 in our own test GSM network, but the effort that would +be required is difficult to justify. Exploring TCH tap modes with AMR or CSD +traffic channels is likewise a subject for further study. In order to make use of this TCH bit access feature, one needs 3 things: @@ -28,10 +30,14 @@ bits into and out of Calypso GSM devices running one of our firmwares in the form of an extension to TI's RVTMUX interface, i.e., we have defined a new RVTMUX channel for this TCH interface and defined the packet types and formats -to be sent over the wire. On the Calypso side this interface is implemented in -FreeCalypso GSM firmwares (currently only in Citrine, but we can also make a -special version of TCS211 with this feature added if the need arises), and on -the host tools side there is support in rvinterf and fc-shell. +to be sent over the wire. On the Calypso side the special functionality in +question was originally implemented in FC Citrine firmware in 2016 and then set +aside for some years; when the right time came to resurrect this feature in late +2022, it turned out that the original implementation from 2016 was slightly +incorrect, and the new implementation in FC Tourmaline fw is slightly different. +On the host tools side the RVTMUX-based TCH interface is supported in rvinterf +and fc-shell; the new version as of fc-host-tools-r18 supports both 2016 and +2022 versions of this over-the-wire interface. The TCH bit access mechanism in FreeCalypso has been designed with an objective of presenting to the user exactly what TI's DSP presents to us, i.e., standing @@ -53,6 +59,11 @@ these hexadecimal strings correctly, you (the user) need to understand the bit order and mapping used by TI's implementation of the GSM 05.03 channel encoder. +As of late 2022, there is a new TCH-tap-modes article in our freecalypso-docs +repository that covers in detail the format of TI's DSP buffers for TCH DL and +UL bits, as well as all known information about TCH DL status words and bit +flags. But here is our original description from 2016: + Recall from the GSM specs that the 260 bits which comprise one speech frame are not all treated equally, instead they are divided into 182 class 1 bits which are protected by a convolutional encoder and 78 class 2 bits which are @@ -77,51 +88,7 @@ * Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the standard GSM 05.03 order. -Uplink testing -============== - -The uplink sending mechanism can be exercised as follows: - -1. Record a speech sample in the GSM 06.10 codec format using any Unix/Linux - audio tool that can write the de facto standard libgsm format. For example, - using SoX: - - rec -c1 recording.gsm - - SoX will write the recording in the GSM 06.10 libgsm format based on the - .gsm suffix at the end of the recording file name; the -c1 option is needed - to disable stereo, otherwise the recording will be slowed down 2x. - -2. Convert it from libgsm standard format into our ad hoc hex strings for - playing into the TCH uplink: - - fc-fr2tch recording.gsm recording.tch-ul - -3. In fc-shell, when you are in an established voice call, issue this command: - - tch play recording.tch-ul - -You should now hear the speech sample you recorded in step 1 above on the other -end of the GSM call. Please note, though, that for this example to work, the -call must be connected in the FR codec mode, not EFR. If you try it on an EFR -call, the vocoder on the other end of the call will try to interpret your FR -codec bits per EFR, resulting in garbage. In order to make this procedure work -properly with EFR, you would need to generate a speech sample in EFR format and -then put it in the correct hex string form with the correct bit order for -feeding to TI's GSM 05.03 channel encoder implementation, and at the present -moment the tools to do this feat are lacking. - -The fc-fr2tch step above is new with the current version of FreeCalypso host -tools. My original implementation of the tch play command in fc-shell read -libgsm GSM 06.10 speech files directly, but I changed it to read hex strings -which correspond literally to what goes into TI's implementation of the 05.03 -channel encoder to make it more general: - -1) to support playing EFR speech bits into TCH/EFS channels; -2) to support people who may want to pass totally non-speech data over traffic - channels in TFO mode - see below. - -Downlink testing +TCH DL recording ================ When you are in an established voice call in fc-shell, you can record the @@ -140,99 +107,102 @@ until you issue this tch record stop command, the output file is not closed and thus may not be written to the file system. -The recording is written in an ASCII format that is similar but not identical -to what the tch play command takes as its input. Here is an example of what -you might get from tch record: - -C214 281D 0063 81C008000008200046000000000000000000000000000007056608060B0A010B09 -C204 0540 003A 9391480D3051F4BD81DF5EB35069BEBC4AEDF756351C4C19689BB1CA4EA5D4F5F5 -C204 05A2 0047 65B80F9E690F7C8CE4ED80DEF69DA6436518AB99ABCE5815E6B562C5CE4EAC5DC5 -C204 04CA 0044 A4483744B04371ED0334ECB350AF28C639B7F095519EF0242D299B6405124F77A5 -C214 2544 0066 83800400000C1000E80000000000000000000000000000300674A07070F080D0B2 -C200 4E8A 0000 07071DF0F83B9521EE61CFF095AA8C0E560300F6A5573C31F3E00601ED4AAE7E2F -C200 4C94 0000 077385ADE20450B2E410961D6C5B0A173ACF9E2D38D77C28CED8495D88AA4DE72E -C200 4ABC 0000 077F0CF86004132DA5C0A6D5A4B0BD4B28159A07D8F4282DC6AAAB27503BC02701 - -The example above is quoted from an actual recording made from a call to WWV at -+1-303-499-7111, a time of day service. This example shows 5 garbled frames -followed by 3 good ones. - -In each line the first 3 space-separated 16-bit words are status words from the -DSP, and the rest is a hex string of 66 digits giving the 260-bit frame with 4 -extra dummy bits at the boundary between the protected and unprotected portions. +The recording is written in an ASCII line-based format with one line for every +received TCH DL frame, but the exact format of each written line will depend on +which firmware version is in use. If you are running ancient Citrine firmware +that emits its TCH DL output in the old format from 2016 (now known to be +incomplete and thus unusable for proper decoding), fc-shell will likewise write +its ASCII output in the old format, which won't be covered further as it is +deprecated and not practically useful. However, if you are running current +FreeCalypso firmware with the resurrected (late 2022) version of the TCH tap +feature, each TCH DL frame will be sent by the fw and received by fc-shell in +the new over-the-wire format, and fc-shell will write the recording file in the +new ASCII format documented in the TCH-tap-modes article in freecalypso-docs. -Now here comes the unpleasant part: radio systems are inherently error-prone, -hence if you are naively expecting every received downlink frame to be a valid -speech frame that matches exactly what the other end transmitted, you will be -disappointed. The downlink frame stream will always consist of both valid and -invalid frames, and in the standard processing chain inside TI's DSP the speech -decoder block that receives these frames from the channel decoder will look at -the status words written by the latter in order to decide what to do with each -frame. But unfortunately I don't know the details, as the DSP is a mostly -undocumented black box. +Once you have captured a TCH DL recording, what can you do with it? If the +recording came from an FR1 call, you will need to pass it through an Rx DTX +handler for FR1 (see GSM 06.11, 06.12 and 06.31 specs) before you can pass it +to a naive GSM 06.10 decoder such as classic Unix libgsm, and if the recording +came from an EFR call, you will need to pass it to a proper EFR (not AMR!) +decoder that includes the necessary EFR Rx DTX handler. Neither of the two +just-mentioned library pieces (neither the Rx DTX handler for FR1 nor a proper, +not-same-as-AMR implementation of GSM EFR) could be found among the existing +body of FOSS as of 2022, thus we (FreeCalypso and Themyscira Wireless) +implemented our own. Please look for our GSM codec libraries & utilities +package, which is expected to reach its first official release some time in +early 2023. -Our FreeCalypso firmwares forward the downlink frame status words from the DSP -to the external host and my current implementation of the tch record command -writes them into the file because it is valuable information which should not -be discarded, but unfortunately I don't know their meaning. The first status -word consists of bit flags, the last one seems to be some kind of error count -(what kind of errors? I don't know), and the middle one is totally unknown. +Inside our gsm-codec-lib package you will find gsmfr-dlcap-* and gsmefr-dlcap-* +utilities that read TCH downlink capture files written by fc-shell tch record +and perform various decoding operations - please refer to further documentation +within that package. + +Please don't use the old fc-tch2fr utility - the function it performs is now +known to be a bogo-transform, and it can't grok the new TCH DL recording format +which you will get with current FreeCalypso fw. -There is also an fc-tch2fr utility that does sort of an inverse of fc-fr2tch, -except that fc-tch2fr expects files written by tch record which contain the -status words at the beginning - the files written by fc-fr2tch and read by -tch play don't have these status words; they just have 66 hex digits per line -giving the bits to be fed to the GSM 05.03 block in the DSP. The fc-tch2fr -utility works by disregarding the status words; the result will be a .gsm file -which you can play with SoX etc, but it will produce horrible sounds wherever -some garbled frames were received on the call downlink. +TCH UL play +=========== + +The uplink sending mechanism can be exercised as follows: + +1. If you are going to be in an FR1 call, prepare a speech sample in the GSM + 06.10 codec format using any Unix/Linux audio tool that can write the de + facto standard libgsm format. For example, using SoX: + + rec -c1 recording.gsm -Passing non-speech data over TCH -================================ + SoX will write the recording in the GSM 06.10 libgsm format based on the + .gsm suffix at the end of the recording file name; the -c1 option is needed + to disable stereo, otherwise the recording will be slowed down 2x. + Alternatively, you can use our new gsmfr-encode utility (gsm-codec-lib + package) to encode from WAV into GSM 06.10, or gsmfr-encode-r for raw BE + input instead of WAV. + + OTOH, if you are going to be in an EFR call rather than FR1, you will need to + prepare a speech sample in the EFR codec format instead. You will need to + use Themyscira gsmefr-encode or gsmefr-encode-r utilities, or convert from + AMR (MR122 mode only, no DTX) with our gsm-amr2efr utility. -If you are incredibly lucky and you happen to live in a part of the world where -the local GSM network implements TFO (tandem-free operation, see GSM 02.53), -i.e., if your GSM network transparently passes codec frames from one end of the -call to the other without transcoding back and forth in the middle, you might -be able to send arbitrary non-speech data bits over this TCH connection. If -you are going to attempt such a feat, you are mostly on your own as the GSM -network in my part of the world does not support TFO, but here are some general -ideas: +2. Convert your speech sample from libgsm standard format (FR1) or Themyscira + gsmx format (EFR) into our ad hoc hex strings for playing into a TCH uplink: + + fc-fr2tch recording.gsm recording.tch-ul -* Heed the difference between the 182 protected bits and the 78 unprotected - bits of each 260-bit frame. The bits in the first class are less likely to - be corrupted by radio errors. Also the first 50 out of these class 1 bits - are protected by a CRC, so if they get corrupted, there will be a flag bit - somewhere in the DSP status words - but I don't know which bit it is... + or + + fc-efr2tch recording.gsmx recording.tch-ul -* Given the lack of detailed information about the exact meaning of the DSP - status words, implement some error checking of your own, e.g., an in-band - CRC within your frame payload so you can check on the receiving end if you - got an uncorrupted frame from your data source or not. +3. In fc-shell, when you are in an established voice call, issue this command: + + tch play recording.tch-ul -* Expect to lose some frames - even if your radio environment is pristine and - error-free, some frames will still be lost because their timeslots had to be - stolen by FACCH on either end of the call. You will need to implement - acknowledgements, retransmissions and retries - the usual stuff. +You should now hear the speech sample you recorded in step 1 above on the other +end of the GSM call. Needless to say, the TCH mode of the call (TCH/FS or +TCH/EFS) needs to match the codec in which your to-be-played recording was +prepared, otherwise the other end of the call will receive garbage! -EFR differences -=============== +Controlling the selection of speech codec for calls +=================================================== -TCH/FS and TCH/EFS channel modes corresponding to the FR and EFR codecs, -respectively, share the same channel encoder that takes 260-bit frames as input. -An EFR speech frame is only 244 bits, but as the GSM 05.03 spec explains, each -EFR frame gets expanded to 260 bits by adding some CRC and repetition bits. -One would thus expect that the behaviour of the channel encoder block should be -strictly identical between TCH/FS and TCH/EFS channel modes, but it appears that -TI's implementation exhibits one difference in behaviour between the two modes. +One very obvious shortcoming of the present facilities for voice TCH redirection +is that we only support FR1 and EFR codecs, but not AMR. However, most GSM +networks prefer to use AMR when the MS supports it - and in regular operation +with a speaker & mic user connection (as opposed to TCH tap modes), our current +FreeCalypso firmwares do support AMR when running on Calypso C035 silicon with +DSP ROM version 3606. (DSP ROM version 3416 together with the respective patch +version also appears to have working AMR, at least in light testing, although +of course we do NOT recommend it for production use.) Therefore, if you wish +to play with EFR, you need to tell the network (via the Bearer Capabilities +information element in CC messages) that your MS does not support AMR, and if +you wish to play with FR1, you need to tell the network that your MS only +supports FR1 and no others. -It appears that TI's implementation of the GSM 05.03 channel encoder computes -the EFR-specific CRC bits internally, such that bits [65,72] of each uplink -frame fed to the DSP in the TCH/EFS mode need to be 0 and will be replaced by -the DSP with CRC bits computed per the standard. It also appears that their -implementation of the channel decoder verifies this CRC, sets a status bit -indicating whether the check succeeded or failed, and then zeroes these bits -out, i.e., the original bits in these CRC bit positions are lost. No problem -if what you are passing over TCH/EFS is indeed EFR speech frames, but it will -be a problem if you are passing non-speech data over your TCH and put something -else in those bits which the spec allocates to CRC. +The outstanding issue here is that we need to implement some mechanism in our +FreeCalypso firmwares, probably a custom AT command, that will modify the +Bearer Capabilities IE (artificially restrict the set of supported codecs) on a +per-call basis. Until we implement the necessary support, the only current way +to create such codec-restricted operation is by writing a /pcm/MSCAP file into +FFS with the desired bit mask of supported codecs - but this method is hugely +inconvenient because this file is read only on fw boot, thus every modification +requires a full reboot cycle.