FreeCalypso > hg > freecalypso-tools

comparison doc/TCH-bit-access @ 907:3de3b34189be

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc/TCH-bit-access: update for newly resurrected version
author Mychaela Falconia <falcon@freecalypso.org>
date Thu, 29 Dec 2022 09:32:31 +0000
parents 2748f257312b
children 8f7c50e1fa3b
comparison
equal deleted inserted replaced
906:94890123a74f 907:3de3b34189be
1 It has been discovered that the DSP ROM in the Calypso GSM baseband processor 1 It has been discovered that the DSP ROM in the Calypso GSM baseband processor
2 implements one nifty feature which is not used at all in standard phone or modem 2 implements one nifty feature which is not used at all in standard phone or modem
3 operation, but which can be used for all kinds of interesting hacks: the traffic 3 operation, but which can be used for all kinds of interesting hacks: the traffic
4 channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink 4 channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink
5 direction (to be fed to one of several speech decoders) can be read out of the 5 direction (to be fed to the channel mode-appropriate speech decoder) can be read
6 DSP's API RAM in real time, and in the uplink direction the user can feed her 6 out of the DSP's API RAM in real time, and in the uplink direction the user can
7 own bits to the input of the GSM 05.03 channel encoder, effectively suppressing 7 feed her own bits to the input of the GSM 05.03 channel encoder, effectively
8 the output of the internal vocoder. 8 suppressing the output of the internal vocoder.
9 9
10 The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel 10 The DSP mechanism in question is known to work in TCH/FS and TCH/EFS channel
11 modes, corresponding to the FR and EFR codecs; it is not currently known if it 11 modes, corresponding to FR1 and EFR codecs; it also appears to work for TCH/HS
12 can also be used in TCH/HS, data traffic or AMR modes. I am aware though of an 12 (HR1 codec), but we (FreeCalypso) haven't tested it because almost no one uses
13 anecdotal report that someone tried to make this feature work with AMR, but was 13 that infamous HR1 codec - the commercial GSM network in our part of the world
14 unsuccessful - hence we should be prepared for the possibility that the hack 14 gives you a full-rate channel if your phone does not support AMR. It would be
15 is not possible with AMR. 15 possible to implement HR1 in our own test GSM network, but the effort that would
16 be required is difficult to justify. Exploring TCH tap modes with AMR or CSD
17 traffic channels is likewise a subject for further study.
16 18
17 In order to make use of this TCH bit access feature, one needs 3 things: 19 In order to make use of this TCH bit access feature, one needs 3 things:
18 20
19 1) Firmware on the Calypso (the ARM part) that reads downlink bits from the DSP 21 1) Firmware on the Calypso (the ARM part) that reads downlink bits from the DSP
20 and writes uplink bits into it while doing everything else that a GSM fw must 22 and writes uplink bits into it while doing everything else that a GSM fw must
26 28
27 In the case of FreeCalypso, we have defined our own protocol for passing TCH 29 In the case of FreeCalypso, we have defined our own protocol for passing TCH
28 bits into and out of Calypso GSM devices running one of our firmwares in the 30 bits into and out of Calypso GSM devices running one of our firmwares in the
29 form of an extension to TI's RVTMUX interface, i.e., we have defined a new 31 form of an extension to TI's RVTMUX interface, i.e., we have defined a new
30 RVTMUX channel for this TCH interface and defined the packet types and formats 32 RVTMUX channel for this TCH interface and defined the packet types and formats
31 to be sent over the wire. On the Calypso side this interface is implemented in 33 to be sent over the wire. On the Calypso side the special functionality in
32 FreeCalypso GSM firmwares (currently only in Citrine, but we can also make a 34 question was originally implemented in FC Citrine firmware in 2016 and then set
33 special version of TCS211 with this feature added if the need arises), and on 35 aside for some years; when the right time came to resurrect this feature in late
34 the host tools side there is support in rvinterf and fc-shell. 36 2022, it turned out that the original implementation from 2016 was slightly
37 incorrect, and the new implementation in FC Tourmaline fw is slightly different.
38 On the host tools side the RVTMUX-based TCH interface is supported in rvinterf
39 and fc-shell; the new version as of fc-host-tools-r18 supports both 2016 and
40 2022 versions of this over-the-wire interface.
35 41
36 The TCH bit access mechanism in FreeCalypso has been designed with an objective 42 The TCH bit access mechanism in FreeCalypso has been designed with an objective
37 of presenting to the user exactly what TI's DSP presents to us, i.e., standing 43 of presenting to the user exactly what TI's DSP presents to us, i.e., standing
38 out of the way as much as possible. TI's DSP presents TCH downlink bits and 44 out of the way as much as possible. TI's DSP presents TCH downlink bits and
39 accepts TCH uplink bits in the form of an array of 16-bit words; the bit order 45 accepts TCH uplink bits in the form of an array of 16-bit words; the bit order
51 these hex digits correspond directly to the 33 bytes being read out of or 57 these hex digits correspond directly to the 33 bytes being read out of or
52 written into DSP API words. Therefore, in order to generate and/or interpret 58 written into DSP API words. Therefore, in order to generate and/or interpret
53 these hexadecimal strings correctly, you (the user) need to understand the bit 59 these hexadecimal strings correctly, you (the user) need to understand the bit
54 order and mapping used by TI's implementation of the GSM 05.03 channel encoder. 60 order and mapping used by TI's implementation of the GSM 05.03 channel encoder.
55 61
62 As of late 2022, there is a new TCH-tap-modes article in our freecalypso-docs
63 repository that covers in detail the format of TI's DSP buffers for TCH DL and
64 UL bits, as well as all known information about TCH DL status words and bit
65 flags. But here is our original description from 2016:
66
56 Recall from the GSM specs that the 260 bits which comprise one speech frame are 67 Recall from the GSM specs that the 260 bits which comprise one speech frame are
57 not all treated equally, instead they are divided into 182 class 1 bits which 68 not all treated equally, instead they are divided into 182 class 1 bits which
58 are protected by a convolutional encoder and 78 class 2 bits which are 69 are protected by a convolutional encoder and 78 class 2 bits which are
59 transmitted without any forward error correction. Furthermore, the first 50 of 70 transmitted without any forward error correction. Furthermore, the first 50 of
60 the class 1 bits are also protected by a CRC. The order in which the bits 71 the class 1 bits are also protected by a CRC. The order in which the bits
75 yourself; 86 yourself;
76 87
77 * Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the 88 * Bits 186 through 263 correspond to the 78 unprotected (class 2) bits in the
78 standard GSM 05.03 order. 89 standard GSM 05.03 order.
79 90
80 Uplink testing 91 TCH DL recording
81 ==============
82
83 The uplink sending mechanism can be exercised as follows:
84
85 1. Record a speech sample in the GSM 06.10 codec format using any Unix/Linux
86 audio tool that can write the de facto standard libgsm format. For example,
87 using SoX:
88
89 rec -c1 recording.gsm
90
91 SoX will write the recording in the GSM 06.10 libgsm format based on the
92 .gsm suffix at the end of the recording file name; the -c1 option is needed
93 to disable stereo, otherwise the recording will be slowed down 2x.
94
95 2. Convert it from libgsm standard format into our ad hoc hex strings for
96 playing into the TCH uplink:
97
98 fc-fr2tch recording.gsm recording.tch-ul
99
100 3. In fc-shell, when you are in an established voice call, issue this command:
101
102 tch play recording.tch-ul
103
104 You should now hear the speech sample you recorded in step 1 above on the other
105 end of the GSM call. Please note, though, that for this example to work, the
106 call must be connected in the FR codec mode, not EFR. If you try it on an EFR
107 call, the vocoder on the other end of the call will try to interpret your FR
108 codec bits per EFR, resulting in garbage. In order to make this procedure work
109 properly with EFR, you would need to generate a speech sample in EFR format and
110 then put it in the correct hex string form with the correct bit order for
111 feeding to TI's GSM 05.03 channel encoder implementation, and at the present
112 moment the tools to do this feat are lacking.
113
114 The fc-fr2tch step above is new with the current version of FreeCalypso host
115 tools. My original implementation of the tch play command in fc-shell read
116 libgsm GSM 06.10 speech files directly, but I changed it to read hex strings
117 which correspond literally to what goes into TI's implementation of the 05.03
118 channel encoder to make it more general:
119
120 1) to support playing EFR speech bits into TCH/EFS channels;
121 2) to support people who may want to pass totally non-speech data over traffic
122 channels in TFO mode - see below.
123
124 Downlink testing
125 ================ 92 ================
126 93
127 When you are in an established voice call in fc-shell, you can record the 94 When you are in an established voice call in fc-shell, you can record the
128 downlink TCH bits as follows: 95 downlink TCH bits as follows:
129 96
138 105
139 You can issue this stop command before or after the call is terminated, but 106 You can issue this stop command before or after the call is terminated, but
140 until you issue this tch record stop command, the output file is not closed and 107 until you issue this tch record stop command, the output file is not closed and
141 thus may not be written to the file system. 108 thus may not be written to the file system.
142 109
143 The recording is written in an ASCII format that is similar but not identical 110 The recording is written in an ASCII line-based format with one line for every
144 to what the tch play command takes as its input. Here is an example of what 111 received TCH DL frame, but the exact format of each written line will depend on
145 you might get from tch record: 112 which firmware version is in use. If you are running ancient Citrine firmware
146 113 that emits its TCH DL output in the old format from 2016 (now known to be
147 C214 281D 0063 81C008000008200046000000000000000000000000000007056608060B0A010B09 114 incomplete and thus unusable for proper decoding), fc-shell will likewise write
148 C204 0540 003A 9391480D3051F4BD81DF5EB35069BEBC4AEDF756351C4C19689BB1CA4EA5D4F5F5 115 its ASCII output in the old format, which won't be covered further as it is
149 C204 05A2 0047 65B80F9E690F7C8CE4ED80DEF69DA6436518AB99ABCE5815E6B562C5CE4EAC5DC5 116 deprecated and not practically useful. However, if you are running current
150 C204 04CA 0044 A4483744B04371ED0334ECB350AF28C639B7F095519EF0242D299B6405124F77A5 117 FreeCalypso firmware with the resurrected (late 2022) version of the TCH tap
151 C214 2544 0066 83800400000C1000E80000000000000000000000000000300674A07070F080D0B2 118 feature, each TCH DL frame will be sent by the fw and received by fc-shell in
152 C200 4E8A 0000 07071DF0F83B9521EE61CFF095AA8C0E560300F6A5573C31F3E00601ED4AAE7E2F 119 the new over-the-wire format, and fc-shell will write the recording file in the
153 C200 4C94 0000 077385ADE20450B2E410961D6C5B0A173ACF9E2D38D77C28CED8495D88AA4DE72E 120 new ASCII format documented in the TCH-tap-modes article in freecalypso-docs.
154 C200 4ABC 0000 077F0CF86004132DA5C0A6D5A4B0BD4B28159A07D8F4282DC6AAAB27503BC02701 121
155 122 Once you have captured a TCH DL recording, what can you do with it? If the
156 The example above is quoted from an actual recording made from a call to WWV at 123 recording came from an FR1 call, you will need to pass it through an Rx DTX
157 +1-303-499-7111, a time of day service. This example shows 5 garbled frames 124 handler for FR1 (see GSM 06.11, 06.12 and 06.31 specs) before you can pass it
158 followed by 3 good ones. 125 to a naive GSM 06.10 decoder such as classic Unix libgsm, and if the recording
159 126 came from an EFR call, you will need to pass it to a proper EFR (not AMR!)
160 In each line the first 3 space-separated 16-bit words are status words from the 127 decoder that includes the necessary EFR Rx DTX handler. Neither of the two
161 DSP, and the rest is a hex string of 66 digits giving the 260-bit frame with 4 128 just-mentioned library pieces (neither the Rx DTX handler for FR1 nor a proper,
162 extra dummy bits at the boundary between the protected and unprotected portions. 129 not-same-as-AMR implementation of GSM EFR) could be found among the existing
163 130 body of FOSS as of 2022, thus we (FreeCalypso and Themyscira Wireless)
164 Now here comes the unpleasant part: radio systems are inherently error-prone, 131 implemented our own. Please look for our GSM codec libraries & utilities
165 hence if you are naively expecting every received downlink frame to be a valid 132 package, which is expected to reach its first official release some time in
166 speech frame that matches exactly what the other end transmitted, you will be 133 early 2023.
167 disappointed. The downlink frame stream will always consist of both valid and 134
168 invalid frames, and in the standard processing chain inside TI's DSP the speech 135 Inside our gsm-codec-lib package you will find gsmfr-dlcap-* and gsmefr-dlcap-*
169 decoder block that receives these frames from the channel decoder will look at 136 utilities that read TCH downlink capture files written by fc-shell tch record
170 the status words written by the latter in order to decide what to do with each 137 and perform various decoding operations - please refer to further documentation
171 frame. But unfortunately I don't know the details, as the DSP is a mostly 138 within that package.
172 undocumented black box. 139
173 140 Please don't use the old fc-tch2fr utility - the function it performs is now
174 Our FreeCalypso firmwares forward the downlink frame status words from the DSP 141 known to be a bogo-transform, and it can't grok the new TCH DL recording format
175 to the external host and my current implementation of the tch record command 142 which you will get with current FreeCalypso fw.
176 writes them into the file because it is valuable information which should not 143
177 be discarded, but unfortunately I don't know their meaning. The first status 144 TCH UL play
178 word consists of bit flags, the last one seems to be some kind of error count 145 ===========
179 (what kind of errors? I don't know), and the middle one is totally unknown. 146
180 147 The uplink sending mechanism can be exercised as follows:
181 There is also an fc-tch2fr utility that does sort of an inverse of fc-fr2tch, 148
182 except that fc-tch2fr expects files written by tch record which contain the 149 1. If you are going to be in an FR1 call, prepare a speech sample in the GSM
183 status words at the beginning - the files written by fc-fr2tch and read by 150 06.10 codec format using any Unix/Linux audio tool that can write the de
184 tch play don't have these status words; they just have 66 hex digits per line 151 facto standard libgsm format. For example, using SoX:
185 giving the bits to be fed to the GSM 05.03 block in the DSP. The fc-tch2fr 152
186 utility works by disregarding the status words; the result will be a .gsm file 153 rec -c1 recording.gsm
187 which you can play with SoX etc, but it will produce horrible sounds wherever 154
188 some garbled frames were received on the call downlink. 155 SoX will write the recording in the GSM 06.10 libgsm format based on the
189 156 .gsm suffix at the end of the recording file name; the -c1 option is needed
190 Passing non-speech data over TCH 157 to disable stereo, otherwise the recording will be slowed down 2x.
191 ================================ 158 Alternatively, you can use our new gsmfr-encode utility (gsm-codec-lib
192 159 package) to encode from WAV into GSM 06.10, or gsmfr-encode-r for raw BE
193 If you are incredibly lucky and you happen to live in a part of the world where 160 input instead of WAV.
194 the local GSM network implements TFO (tandem-free operation, see GSM 02.53), 161
195 i.e., if your GSM network transparently passes codec frames from one end of the 162 OTOH, if you are going to be in an EFR call rather than FR1, you will need to
196 call to the other without transcoding back and forth in the middle, you might 163 prepare a speech sample in the EFR codec format instead. You will need to
197 be able to send arbitrary non-speech data bits over this TCH connection. If 164 use Themyscira gsmefr-encode or gsmefr-encode-r utilities, or convert from
198 you are going to attempt such a feat, you are mostly on your own as the GSM 165 AMR (MR122 mode only, no DTX) with our gsm-amr2efr utility.
199 network in my part of the world does not support TFO, but here are some general 166
200 ideas: 167 2. Convert your speech sample from libgsm standard format (FR1) or Themyscira
201 168 gsmx format (EFR) into our ad hoc hex strings for playing into a TCH uplink:
202 * Heed the difference between the 182 protected bits and the 78 unprotected 169
203 bits of each 260-bit frame. The bits in the first class are less likely to 170 fc-fr2tch recording.gsm recording.tch-ul
204 be corrupted by radio errors. Also the first 50 out of these class 1 bits 171
205 are protected by a CRC, so if they get corrupted, there will be a flag bit 172 or
206 somewhere in the DSP status words - but I don't know which bit it is... 173
207 174 fc-efr2tch recording.gsmx recording.tch-ul
208 * Given the lack of detailed information about the exact meaning of the DSP 175
209 status words, implement some error checking of your own, e.g., an in-band 176 3. In fc-shell, when you are in an established voice call, issue this command:
210 CRC within your frame payload so you can check on the receiving end if you 177
211 got an uncorrupted frame from your data source or not. 178 tch play recording.tch-ul
212 179
213 * Expect to lose some frames - even if your radio environment is pristine and 180 You should now hear the speech sample you recorded in step 1 above on the other
214 error-free, some frames will still be lost because their timeslots had to be 181 end of the GSM call. Needless to say, the TCH mode of the call (TCH/FS or
215 stolen by FACCH on either end of the call. You will need to implement 182 TCH/EFS) needs to match the codec in which your to-be-played recording was
216 acknowledgements, retransmissions and retries - the usual stuff. 183 prepared, otherwise the other end of the call will receive garbage!
217 184
218 EFR differences 185 Controlling the selection of speech codec for calls
219 =============== 186 ===================================================
220 187
221 TCH/FS and TCH/EFS channel modes corresponding to the FR and EFR codecs, 188 One very obvious shortcoming of the present facilities for voice TCH redirection
222 respectively, share the same channel encoder that takes 260-bit frames as input. 189 is that we only support FR1 and EFR codecs, but not AMR. However, most GSM
223 An EFR speech frame is only 244 bits, but as the GSM 05.03 spec explains, each 190 networks prefer to use AMR when the MS supports it - and in regular operation
224 EFR frame gets expanded to 260 bits by adding some CRC and repetition bits. 191 with a speaker & mic user connection (as opposed to TCH tap modes), our current
225 One would thus expect that the behaviour of the channel encoder block should be 192 FreeCalypso firmwares do support AMR when running on Calypso C035 silicon with
226 strictly identical between TCH/FS and TCH/EFS channel modes, but it appears that 193 DSP ROM version 3606. (DSP ROM version 3416 together with the respective patch
227 TI's implementation exhibits one difference in behaviour between the two modes. 194 version also appears to have working AMR, at least in light testing, although
228 195 of course we do NOT recommend it for production use.) Therefore, if you wish
229 It appears that TI's implementation of the GSM 05.03 channel encoder computes 196 to play with EFR, you need to tell the network (via the Bearer Capabilities
230 the EFR-specific CRC bits internally, such that bits [65,72] of each uplink 197 information element in CC messages) that your MS does not support AMR, and if
231 frame fed to the DSP in the TCH/EFS mode need to be 0 and will be replaced by 198 you wish to play with FR1, you need to tell the network that your MS only
232 the DSP with CRC bits computed per the standard. It also appears that their 199 supports FR1 and no others.
233 implementation of the channel decoder verifies this CRC, sets a status bit 200
234 indicating whether the check succeeded or failed, and then zeroes these bits 201 The outstanding issue here is that we need to implement some mechanism in our
235 out, i.e., the original bits in these CRC bit positions are lost. No problem 202 FreeCalypso firmwares, probably a custom AT command, that will modify the
236 if what you are passing over TCH/EFS is indeed EFR speech frames, but it will 203 Bearer Capabilities IE (artificially restrict the set of supported codecs) on a
237 be a problem if you are passing non-speech data over your TCH and put something 204 per-call basis. Until we implement the necessary support, the only current way
238 else in those bits which the spec allocates to CRC. 205 to create such codec-restricted operation is by writing a /pcm/MSCAP file into
206 FFS with the desired bit mask of supported codecs - but this method is hugely
207 inconvenient because this file is read only on fw boot, thus every modification
208 requires a full reboot cycle.