FreeCalypso > hg > freecalypso-tools

annotate doc/Nonfree-phones-user-data @ 928:65953c172f24

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
rvinterf/lowlevel: new hex dump format
author Mychaela Falconia <falcon@freecalypso.org>
date Tue, 23 May 2023 05:23:19 +0000
parents ac527de89a0d
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
808
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
1 Using FreeCalypso tools to access user data on non-free phones
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
2 ==============================================================
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
3
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
4 If you are using a Motorola C1xx or Pirelli DP-L10 phone as your daily driver,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
5 running its original proprietary fw for the lack of anything better, you can
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
6 still use FreeCalypso tools to get to some of your user data, namely, your
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
7 phonebook and your SMS store. You can even do the same with a non-Calypso
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
8 phone if you can move the SIM in and out of it. This article tells you how to
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
9 do such tricks.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
10
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
11 Phonebook howto
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
12 ===============
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
13
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
14 Whenever you have to use a phone whose firmware is not fully liberated, you
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
15 should always store your phonebook contacts on the SIM, rather than "on the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
16 phone". SIM phonebook storage format is strictly defined by standard specs,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
17 leaving no room for proprietary firmwares to apply any wacky proprietary ideas
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
18 of their own - OTOH, "on the phone" storage really means storing your data in
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
19 proprietary fw data structures in only-Cthulhu-knows-what format.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
20
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
21 If you have your contacts stored in your SIM phonebook, you can manipulate the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
22 latter with FreeCalypso tools: you can dump your SIM phonebook and save it on
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
23 your larger host computer, you can restore these backups back to the SIM, and
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
24 you can freely edit your SIM phonebook. If your phone is a supported Calypso
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
25 model, you can perform all of these manipulations without removing your SIM -
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
26 you only need to shut down the phone's regular fw for the duration of the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
27 operation, just like you would do for flashing with fc-loadtool. The other way
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
28 is to remove your SIM from the phone and stick it into a standalone smart card
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
29 "reader" device - this path is totally independent of all phones.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
30
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
31 You will need to have both the present FC host tools package and the separate
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
32 FC SIM tools package installed on your host machine. See the instructions in
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
33 the SIM-manipulation article for how to run fc-simint for the purpose of
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
34 operating on a SIM card inside your Calypso phone. Once you have landed at the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
35 simtool> prompt, see the doc/User-oriented-commands article in the fc-sim-tools
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
36 repository for instructions on how to enter your PIN (if your SIM requires one)
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
37 and how to manipulate your phonebooks.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
38
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
39 SMS storage howto
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
40 =================
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
41
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
42 Motorola's firmware always stores received SMS on the SIM, whereas sent SMS are
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
43 stored in their FFS (flash file system, see Compal-FFS article) in their own
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
44 proprietary format. You can transfer your received SMS to your larger host
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
45 computer for longer-term archival as follows:
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
46
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
47 1) Shut down the phone's regular fw and run fc-simint to operate on the SIM;
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
48
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
49 2) Once you are at the simtool> prompt, enter your PIN if your SIM requires one,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
50 and then save the SIM SMS store to a Unix host file with the save-sms-bin
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
51 command, as explained in the doc/User-oriented-commands article in the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
52 fc-sim-tools repository. You can then issue an sms-erase-all command in the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
53 same seesion to clean out the SIM SMS store.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
54
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
55 3) The files saved by the fc-simint/fc-simtool save-sms-bin command are binary,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
56 but the message content can then be fully decoded to human-readable ASCII
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
57 (or ISO 8859-1 or UTF-8 if desired) with our pcm-sms-decode utility,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
58 described later in this article.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
59
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
60 To retrieve your sent SMS from Compal's FFS, you will need to similarly shut
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
61 down the phone's regular fw and read out the FFS sectors with fc-loadtool. You
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
62 can then use our tiffs utility to parse the FFS structure, but the format of the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
63 actual files used by Compal's proprietary fw implementation has not been studied
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
64 in detail by the Mother, thus you will need to use your own brain cells for
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
65 further decoding.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
66
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
67 If you use a Pirelli DP-L10 phone rather than Motorola C1xx, Pirelli's fw gives
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
68 you the option of storing received SMS either on the SIM or "on the phone", the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
69 latter meaning Pirelli's FFS. Sent SMS are always stored in Pirelli's FFS.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
70
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
71 If you use SIM storage for received SMS on your Pirelli phone, then the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
72 procedure for retrieving those SMS is the same as for Mot C1xx, as detailed
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
73 above. However, with this phone model, configuring the fw to store received SMS
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
74 in its FFS ("on the phone") is generally preferable: Pirelli's fw maintains a
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
75 file in its FFS (/pcm/SMS) whose binary format is exactly the same as the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
76 standards-defined format for storing SMS on the SIM, thus reading out /pcm/SMS
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
77 with fc-fsio is just as good as reading out EF_SMS with fc-simint or fc-simtool,
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
78 but is much more convenient: fc-fsio operates on the phone while its regular fw
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
79 is running, whereas direct SIM manipulation with fc-simint requires shutting
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
80 down the phone's regular fw. Pirelli's fw also accesses its SMS store faster
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
81 when FFS storage is used, rather than SIM.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
82
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
83 Decoding received SMS binary files with pcm-sms-decode
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
84 ======================================================
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
85
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
86 Whether you have retrieved your SMS store from a SIM with fc-simint/fc-simtool
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
87 save-sms-bin command or read out Pirelli's /pcm/SMS file with fc-fsio, the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
88 binary format is exactly the same: a bundle of 176-byte records directly abutted
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
89 together, with each individual record having the standard format defined in GSM
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
90 TS 11.11 and its successor 3GPP TS 51.011. Looking at raw hex dumps won't be
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
91 very helpful, as the reversed nibble format used for phone numbers and SC
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
92 timestamps is not directly human-readable, and the message bodies will usually
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
93 be in the form of packed septets - hence a special decoding program is needed.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
94
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
95 Our pcm-sms-decode utility does the needed job of reading these binary files and
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
96 fully decoding them into human-readable ASCII, or ISO 8859-1 or UTF-8 if your
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
97 host system supports such extended character sets and you have received SMS
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
98 content that uses beyond-ASCII characters. pcm-sms-decode was added to FC host
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
99 tools suite in early 2020; it is based on and shares most of its code with our
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
100 earlier sms-pdu-decode utility from 2018. (The latter is meant for decoding SMS
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
101 PDUs retrieved from FreeCalypso AT command modems with fcup-smdump.)
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
102
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
103 pcm-sms-decode uses the same backslash escapes in its output and accepts the
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
104 same -e, -u and -h options as sms-pdu-decode - please refer to the description
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
105 of sms-pdu-decode in the User-phone-tools article for the details.
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
106
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
107 By default, pcm-sms-decode numbers the records being decoded as 0 to N-1, where
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
108 N is the total number of records in the binary file given to it (file size equal
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
109 to N*176 bytes). As of fc-host-tools-r15, newly added -s option causes it to
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
110 number these same records as 1 to N instead. The original 0-based record
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
111 numbering was implemented when the tool was used solely to decode /pcm/SMS from
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
112 Pirelli's fw (before the introduction of fc-simtool), but now that we have
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
113 fc-simtool and fc-simint, and given that the SIM interface protocol and SIM file
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
114 system definition use 1-based record numbers, we need to support the same
ac527de89a0d pcm-sms-decode documentation
Mychaela Falconia <falcon@freecalypso.org>
parents:
diff changeset
115 convention in pcm-sms-decode for consistency.