# HG changeset patch # User Mychaela Falconia # Date 1670797236 0 # Node ID 8eb0e7a394096b3c4848007ef99732bb2d9cdbdb # Parent 22601ae99434f1e0ce7b26a04c4aa4f8402f3e92 doc: document command line utilities diff -r 22601ae99434 -r 8eb0e7a39409 doc/AMR-EFR-conversion --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/AMR-EFR-conversion Sun Dec 11 22:20:36 2022 +0000 @@ -0,0 +1,25 @@ +We have two simple utilities that allow one to experiment with "dumb" bit- +shuffling conversion between AMR 12k2 and EFR codec formats, to explore +capabilities and limitations of this approach. + +gsm-amr2efr reads an AMR speech recording in RFC 4867 storage format (the common +.amr format) and converts it to EFR in gsmx format. The AMR input to this +utility must consists of MR122 frames only - no other AMR modes, no SID and no +NO_DATA gaps. The intent is that one can take a starting speech sample in WAV +format, encode it into AMR with amrnb-enc from opencore-amrnb (by default that +utility produces MR122 encoding without DTX), and then convert the AMR output to +EFR with gsm-amr2efr. One can then encode the same starting-point WAV speech +sample with gsmefr-encode (matching official EFR from ETSI) and compare the two +EFR outputs. When you do this experiment, you will see that the two EFR outputs +will be different (you can then analyze encoded speech parameter diffs with +gsmrec-dump), but each version can be fed to an EFR decoder, resulting in +OK-sounding speech. + +gsm-efr2amr performs the opposite conversion: it reads an EFR session recording +in gsmx format and converts it to AMR storage format. The input to gsm-efr2amr +is allowed to contain Themyscira BFI markers in addition to EFR frames; these +BFI markers will be turned into AMR NO_DATA frames. The same input can also +contain EFR SID frames - however, gsm-efr2amr will not detect them and won't +give them any special handling, instead they will be bit-reshuffled into MR122 +just like EFR speech frames. The result of such "dumb" conversion is invalid +AMR, and when you decode it with amrnb-dec, you will hear some strange noises. diff -r 22601ae99434 -r 8eb0e7a39409 doc/Calypso-TCH-downlink --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Calypso-TCH-downlink Sun Dec 11 22:20:36 2022 +0000 @@ -0,0 +1,44 @@ +It has been discovered that the DSP ROM in the Calypso GSM baseband processor +implements one nifty feature which is not used at all in standard phone or modem +operation, but which can be used for all kinds of interesting hacks: the traffic +channel (TCH) bits coming out of the GSM 05.03 channel decoder in the downlink +direction (to be fed to one of several speech decoders) can be read out of the +DSP's API RAM in real time, and in the uplink direction the user can feed her +own bits to the input of the GSM 05.03 channel encoder, effectively suppressing +the output of the internal vocoder. + +I (Mother Mychaela) previously played with this functionality back in 2016, and +it's been mostly shelved since then. This functionality is now getting renewed +attention in late 2022: now that we have a proper set of codec libraries (the +present package) and a proper understanding of Rx DTX handling requirements, we +can take another shot at decoding TCH downlink captures taken from Calypso GSM +MS. + +For reasons of project resource planning, the work to study Calypso DSP downlink +bits in EFR mode is currently postponed until after we build our long-overdue FC +Venus board, but in the case of FR1 codec, we have a TCH downlink recording that +was captured back in 2016, and by analyzing this recording, we can make the +following observations: + +* DSP status word 0 contains various flags; we already knew some of them, but + now we got additional knowledge: + + bit 2 is BFI + bits 4:3 give a 2-bit SID flag per GSM 06.31 section 6.1.1 + + Unfortunately I couldn't find TAF anywhere - but oh well. + +* DSP status word 1 appears to be garbage, DSP status word 2 appears to be some + kind of error count - no new knowledge here. + +* By heeding the newly-understood BFI flag in DSP status word 0, we can do a + much cleaner decoding of TCH downlink capture than we did back in 2016. + +The new gsmfr-cvt-dlcap utility in the present package replaces the old +fc-tch2fr utility from FC host tools - the latter should now be considered a +bogon. The output from gsmfr-cvt-dlcap is in our gsmx format, containing +Themyscira BFI markers in those frame positions where Calypso DSP indicated BFI +on TCH downlink, and it needs to be fed to gsmfr-decode from the present +package; our FR1 Rx DTX preprocessor will take care of SIDs and BFIs, and the +complete decoding chain should then closely replicate what must be happening +inside the original DSP. diff -r 22601ae99434 -r 8eb0e7a39409 doc/Codec-utils --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Codec-utils Sun Dec 11 22:20:36 2022 +0000 @@ -0,0 +1,39 @@ +Standalone command line utilities for FR and EFR codecs +======================================================= + +The pre-existing FOSS opencore-amr package includes amrnb-enc and amrnb-dec test +programs: the first reads linear PCM from a WAV file and emits AMR encoder +output in a .amr file (RFC 4867 AMR storage format), the second reads this .amr +format and emits AMR decoder output as WAV. Inspired by these simple test +programs, the present package offers equivalent command line utilities for GSM +FR and EFR codecs. Here they are: + +gsmfr-encode This utility reads linear PCM from a WAV file, runs libgsm + 06.10 encoder and writes the output in the classic .gsm format + (directly abutted FR codec frames of 33 bytes each). We don't + currently have a Tx-side DTX implementation (VAD etc) for GSM + FR, hence the output from gsmfr-encode will always consist of + good speech frames only. + +gsmfr-decode This utility reads our gsmx format (see Binary-file-format + article), which is a superset of the classic libgsm format. + The input to gsmfr-decode may be a pure .gsm recording as + produced by gsmfr-encode or toast from libgsm package, or it + can also contain SID frames and/or BFI markers. The processing + performed by gsmfr-decode begins with our FR1 Rx DTX handler + preprocessor, which will be an identity transform for pure .gsm + input but becomes important for real-world input containing SIDs + and BFIs, and is followed by gsm_decode() from libgsm. The + decoded output is written as WAV. + +gsmefr-encode This utility reads linear PCM from a WAV file, runs our EFR + encoder (Themyscira libgsmefr) and writes the output in our gsmx + format. There is an option to enable or disable DTX: -d enables + DTX, otherwise it is disabled. (This option mirrors amrnb-enc.) + +gsmefr-decode This utility reads our gsmx format (which must be EFR, not FR1) + and feeds all frames and BFIs to our EFR decoder. The decoded + output is written as WAV. + +WAV reading and writing functions used by the above utilities have been lifted +out of opencore-amrnb test code. diff -r 22601ae99434 -r 8eb0e7a39409 doc/RTP-analysis --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/RTP-analysis Sun Dec 11 22:20:36 2022 +0000 @@ -0,0 +1,20 @@ +The present package includes a utility named rtp-gsmfr-extr; this utility +extracts a single RTP stream in either FR1 or EFR codec format from a pcap file, +presumably captured with tcpdump on an IP network serving either an IP-based BTS +or a gateway from an E1-based BTS to RTP - the intent is to extract a GSM call +uplink that has been rendered into an RTP stream by a BTS. The RTP stream being +extracted must be fully continuous without any gaps, using Themyscira +RTP-BFI-extension BFI marker packets in those 20 ms windows where no good +traffic frame has been received. rtp-gsmfr-extr verifies continuity of the RTP +stream being extracted: any detected discontinuity (either a sequence number +jump indicating packet loss or a timestamp jump indicating an intentional gap +generated at the source) will be reported, and the extraction will stop there. + +To run rtp-gsmfr-extr, you need to have a pcap file (obviously), and you need to +identify the RTP stream to be extracted by either source or destination IP:port. +rtp-gsmfr-extr will look at every UDP packet that matches this src-IP-port or +dest-IP-port filter, and then check for valid RTP, verify the expected increment +in sequence and timestamp numbers, check for either FR1 or EFR payload (or a +Themyscira BFI marker for FR & EFR), and finally write the extracted frame +stream to a gsmx file. This gsmx output can then be analyzed with gsmrec-dump, +or decoded to playable WAV with gsmfr-decode or gsmefr-decode. diff -r 22601ae99434 -r 8eb0e7a39409 doc/Utils-overview --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Utils-overview Sun Dec 11 22:20:36 2022 +0000 @@ -0,0 +1,43 @@ +Here is the complete list of all command line utilities included in the present +package: + +gsm-amr2efr See AMR-EFR-conversion article. +gsm-efr2amr + +gsmefr-cod-parse See EFR-testing article. +gsmefr-dec-parse + +gsmefr-decode See Codec-utils article. +gsmefr-encode + +gsmefr-etsi-dec See EFR-testing article. +gsmefr-etsi-enc + +gsmefr-rec2etsi This program reads an EFR session recording in gsmx + format and converts it into ETSI decoder input (.dec) + format. It was written early in the project as a sanity + check, to verify that the format of TCH uplink from + sysmoBTS in EFR mode matches our expectations, before + embarking on the labor-intensive project to librify EFR + reference code from ETSI. + +gsmfr-cvt-dlcap See Calypso-TCH-downlink article. + +gsmfr-decode See Codec-utils article. +gsmfr-encode + +gsmfr-hand-test This program was written as part of a debug chase: + there seemed to be a design flaw in libgsmfrp, but the + actual bug turned out to be in the way we patched + osmo-bts-sysmo to produce a continuous RTP stream. + +gsmfr-max-out Another test program written as part of the same debug + chase. + +gsmfr-preproc This program reads a gsmx file in FR codec format, + passes it through our Rx DTX preprocessor (libgsmfrp), + and writes the preprocessor output to another .gsm file. + +gsmrec-dump See Binary-file-format article. + +rtp-gsmfr-extr See RTP-analysis article.