# HG changeset patch # User Mychaela Falconia # Date 1616804929 0 # Node ID ac527de89a0df2acdd1d4f24bb2b50afd536b18d # Parent 3a80bfa874962d9c60e5d1fe8a6a3c5b3a5d5cac pcm-sms-decode documentation diff -r 3a80bfa87496 -r ac527de89a0d CHANGES --- a/CHANGES Thu Mar 25 05:56:06 2021 +0000 +++ b/CHANGES Sat Mar 27 00:28:49 2021 +0000 @@ -9,12 +9,19 @@ error markers when the message body contains non-displayable characters, allowing the user to see the full message content, good or bad. +* pcm-sms-decode allows a new -s option to number decoded records as 1..N + instead of 0..N-1. + * fcup-smdump: if the output contains Phonebook-Name lines (if the AT command modem reported a phonebook name along with the message in the +CMGL response), these phonebook names are now emitted as quoted strings with the same possible backslash escapes as defined in the SIM-data-formats specification in the freecalypso-docs repository. +* Documentation: new doc/Nonfree-phones-user-data article explains how to use + fc-simint and pcm-sms-decode to retrieve user data (phonebooks and SMS) from + phones that have not been fully liberated yet. + Changes from fc-host-tools-r13 to fc-host-tools-r14: * fc-loadtool got new commands for reading the OTP protection register of diff -r 3a80bfa87496 -r ac527de89a0d doc/Nonfree-phones-user-data --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/Nonfree-phones-user-data Sat Mar 27 00:28:49 2021 +0000 @@ -0,0 +1,115 @@ +Using FreeCalypso tools to access user data on non-free phones +============================================================== + +If you are using a Motorola C1xx or Pirelli DP-L10 phone as your daily driver, +running its original proprietary fw for the lack of anything better, you can +still use FreeCalypso tools to get to some of your user data, namely, your +phonebook and your SMS store. You can even do the same with a non-Calypso +phone if you can move the SIM in and out of it. This article tells you how to +do such tricks. + +Phonebook howto +=============== + +Whenever you have to use a phone whose firmware is not fully liberated, you +should always store your phonebook contacts on the SIM, rather than "on the +phone". SIM phonebook storage format is strictly defined by standard specs, +leaving no room for proprietary firmwares to apply any wacky proprietary ideas +of their own - OTOH, "on the phone" storage really means storing your data in +proprietary fw data structures in only-Cthulhu-knows-what format. + +If you have your contacts stored in your SIM phonebook, you can manipulate the +latter with FreeCalypso tools: you can dump your SIM phonebook and save it on +your larger host computer, you can restore these backups back to the SIM, and +you can freely edit your SIM phonebook. If your phone is a supported Calypso +model, you can perform all of these manipulations without removing your SIM - +you only need to shut down the phone's regular fw for the duration of the +operation, just like you would do for flashing with fc-loadtool. The other way +is to remove your SIM from the phone and stick it into a standalone smart card +"reader" device - this path is totally independent of all phones. + +You will need to have both the present FC host tools package and the separate +FC SIM tools package installed on your host machine. See the instructions in +the SIM-manipulation article for how to run fc-simint for the purpose of +operating on a SIM card inside your Calypso phone. Once you have landed at the +simtool> prompt, see the doc/User-oriented-commands article in the fc-sim-tools +repository for instructions on how to enter your PIN (if your SIM requires one) +and how to manipulate your phonebooks. + +SMS storage howto +================= + +Motorola's firmware always stores received SMS on the SIM, whereas sent SMS are +stored in their FFS (flash file system, see Compal-FFS article) in their own +proprietary format. You can transfer your received SMS to your larger host +computer for longer-term archival as follows: + +1) Shut down the phone's regular fw and run fc-simint to operate on the SIM; + +2) Once you are at the simtool> prompt, enter your PIN if your SIM requires one, + and then save the SIM SMS store to a Unix host file with the save-sms-bin + command, as explained in the doc/User-oriented-commands article in the + fc-sim-tools repository. You can then issue an sms-erase-all command in the + same seesion to clean out the SIM SMS store. + +3) The files saved by the fc-simint/fc-simtool save-sms-bin command are binary, + but the message content can then be fully decoded to human-readable ASCII + (or ISO 8859-1 or UTF-8 if desired) with our pcm-sms-decode utility, + described later in this article. + +To retrieve your sent SMS from Compal's FFS, you will need to similarly shut +down the phone's regular fw and read out the FFS sectors with fc-loadtool. You +can then use our tiffs utility to parse the FFS structure, but the format of the +actual files used by Compal's proprietary fw implementation has not been studied +in detail by the Mother, thus you will need to use your own brain cells for +further decoding. + +If you use a Pirelli DP-L10 phone rather than Motorola C1xx, Pirelli's fw gives +you the option of storing received SMS either on the SIM or "on the phone", the +latter meaning Pirelli's FFS. Sent SMS are always stored in Pirelli's FFS. + +If you use SIM storage for received SMS on your Pirelli phone, then the +procedure for retrieving those SMS is the same as for Mot C1xx, as detailed +above. However, with this phone model, configuring the fw to store received SMS +in its FFS ("on the phone") is generally preferable: Pirelli's fw maintains a +file in its FFS (/pcm/SMS) whose binary format is exactly the same as the +standards-defined format for storing SMS on the SIM, thus reading out /pcm/SMS +with fc-fsio is just as good as reading out EF_SMS with fc-simint or fc-simtool, +but is much more convenient: fc-fsio operates on the phone while its regular fw +is running, whereas direct SIM manipulation with fc-simint requires shutting +down the phone's regular fw. Pirelli's fw also accesses its SMS store faster +when FFS storage is used, rather than SIM. + +Decoding received SMS binary files with pcm-sms-decode +====================================================== + +Whether you have retrieved your SMS store from a SIM with fc-simint/fc-simtool +save-sms-bin command or read out Pirelli's /pcm/SMS file with fc-fsio, the +binary format is exactly the same: a bundle of 176-byte records directly abutted +together, with each individual record having the standard format defined in GSM +TS 11.11 and its successor 3GPP TS 51.011. Looking at raw hex dumps won't be +very helpful, as the reversed nibble format used for phone numbers and SC +timestamps is not directly human-readable, and the message bodies will usually +be in the form of packed septets - hence a special decoding program is needed. + +Our pcm-sms-decode utility does the needed job of reading these binary files and +fully decoding them into human-readable ASCII, or ISO 8859-1 or UTF-8 if your +host system supports such extended character sets and you have received SMS +content that uses beyond-ASCII characters. pcm-sms-decode was added to FC host +tools suite in early 2020; it is based on and shares most of its code with our +earlier sms-pdu-decode utility from 2018. (The latter is meant for decoding SMS +PDUs retrieved from FreeCalypso AT command modems with fcup-smdump.) + +pcm-sms-decode uses the same backslash escapes in its output and accepts the +same -e, -u and -h options as sms-pdu-decode - please refer to the description +of sms-pdu-decode in the User-phone-tools article for the details. + +By default, pcm-sms-decode numbers the records being decoded as 0 to N-1, where +N is the total number of records in the binary file given to it (file size equal +to N*176 bytes). As of fc-host-tools-r15, newly added -s option causes it to +number these same records as 1 to N instead. The original 0-based record +numbering was implemented when the tool was used solely to decode /pcm/SMS from +Pirelli's fw (before the introduction of fc-simtool), but now that we have +fc-simtool and fc-simint, and given that the SIM interface protocol and SIM file +system definition use 1-based record numbers, we need to support the same +convention in pcm-sms-decode for consistency.