FreeCalypso > hg > freecalypso-tools

comparison doc/Flash-programming @ 676:b6b8307d195b

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
doc: new articles Binary-file-formats and Flash-programming
author Mychaela Falconia <falcon@freecalypso.org>
date Sun, 08 Mar 2020 22:15:57 +0000
parents
children 3a41d69e8104
comparison
equal deleted inserted replaced
675:8b1e86dcc3ac 676:b6b8307d195b
1 Our Calypso device flash programming tool fc-loadtool can be used in several
2 different paradigms; this article is an attempt to clarify the many available
3 modes of usage. You should also read the companion article Binary-file-formats
4 for further background.
5
6 Flashing firmware release images
7 ================================
8
9 In conventional forward engineering environments where you develop or maintain
10 firmware for hardware made by your own company (meaning no reverse eng, no
11 "illicit" aftermarket tinkering on hw made by some other company aeons ago),
12 you have a firmware build system that produces fw build images (some of which
13 may subsequently be blessed as releases), and you have a tool that flashes
14 these fw build images into your hardware, operating as efficiently as possible,
15 automated as much as possible, requiring minimal user action for the boring
16 repetitive task of flashing a new fw image every time you build one. And if
17 you become lucky enough to produce your hardware in volume, the same objectives
18 of maximal efficiency and automation carry over into the production line
19 environment as well.
20
21 In TI's environment the standardized format for firmware build images which are
22 then flashable into hardware targets was a variant of Motorola SREC written in
23 *.m0 files, a variant which we have named moko-style m0 after its most famous
24 user. The special quirk of this particular SREC variant is its peculiar byte
25 order. TI's firmware build system produces a *.m0 S-record image as its final
26 build product, and TI's official Calypso flash programming tool (FLUID) takes
27 these *.m0 files as its input.
28
29 Since the beginning of FreeCalypso we have had two ways of flashing TI-built
30 firmware images into suitable targets (initially OM GTA02 modem, then many
31 others including our own FCDEV3B):
32
33 1) Our fc-loadtool has had a flash program-m0 command from the beginning,
34 programming device flash with bits from an m0 file directly and natively.
35 However, prior to fc-host-tools-r12 this command was poorly supported: it
36 ran significantly slower than flash program-bin, had poorer progress
37 indication and did not perform CRC-32 verification at the end, which is an
38 important integrity check. Also this original flash program-m0 command (as
39 opposed to flash e-program-m0 added in fc-host-tools-r13) does not include a
40 built-in erase step, thus prior to fc-host-tools-r13 the user had to have
41 outside knowledge of how many sectors to erase first with a separate flash
42 erase command.
43
44 The new flash e-program-m0 command added in fc-host-tools-r13 is m0 image
45 flashing finally done right. It reads in the specified S-record image in
46 moko-style m0 format, builds a map of potentially discontiguous flash
47 regions into which the image deposits bits, erases the set of flash sectors
48 which need to be erased before programming these regions, then programs the
49 new image bits into flash, exactly like TI's own FLUID.
50
51 2) The alternative way is to first convert the *.m0 S-record image produced by
52 TI's hex470 post-linker tool into straight binary (*.bin) with a FreeCalypso
53 tool called mokosrec2bin, then program the binary fw image into flash with
54 fc-loadtool command flash program-bin. This method is the one we've been
55 using since 2017, and our FC Magnetite firmware build system is now set up
56 to produce not only fwimage.m0, but also fwimage.bin (it runs mokosrec2bin),
57 and it also generates an fc-loadtool command script (a text file named
58 flash-script) with two commands in it: a flash erase command with a
59 calculated sector address range and a flash program-bin command to program
60 the accompanying fwimage.bin image.
61
62 As of fc-host-tools-r13 both methods work equally well: if you have an official
63 FreeCalypso firmware release (containing fwimage.m0, fwimage.bin and
64 flash-script files) which you need to flash into a device such as our own
65 FCDEV3B or OM GTA02 (but *not* Mot C1xx!), you can execute either
66 'exec flash-script' or 'flash e-program-m0 fwimage.m0' at the loadtool> prompt,
67 and both ways will produce exactly the same result with equal performance and
68 reliability. And if you need a more special operation such as erasing the
69 entire flash (factory production lines) or erasing and reprogramming only a
70 certain part of the normally affected sector range, that's what custom command
71 scripting ability is for.
72
73 For the sake of symmetry, we also have a flash e-program-bin command that is a
74 binary image format counterpart to flash e-program-m0: it first erases the
75 sectors into which new bits will be programmed, then programs the new bits.
76 Thus a third equally good way to flash a new FreeCalypso fw release into a
77 target such as FCDEV3B or GTA02 is to execute
78 'flash e-program-bin 0 fwimage.bin' - but don't *ever* do it on a Mot C1xx
79 phone!
80
81 Flash backup and restore
82 ========================
83
84 A completely different paradigm takes place on alien targets such as Motorola
85 C1xx and Pirelli DP-L10, made by alien manufacturers, meaning not FreeCalypso,
86 not Openmoko and not TI. The most important flash operation on these alien
87 targets is making a flash dump; these dumps can then be used for forensics,
88 reverse engineering, or simply as a backup. When we subsequently write to
89 flash on these alien targets (after having saved a backup first), we are not
90 flashing an m0 fw image or a binary image made from one with mokosrec2bin,
91 instead the most common operations are:
92
93 * Flashing a backup image back into the same device it was originally made
94 from (flash restore);
95
96 * Changing a device from one firmware version to a different one by programming
97 its flash with firmware bits that were originally read out from some fw-donor
98 unit;
99
100 * Surgical manipulations such as erasing FFS sectors or rewriting one specific
101 part of the flash based on reverse-engineered understanding of its structure.
102
103 This different paradigm leads to a different mode of usage for fc-loadtool:
104 instead of needing a maximally-automated operation that flashes a firmware
105 release image with as little user thought involvement as possible, our flash
106 manipulations need to be of a more manual peek-n-poke manner. We provide a
107 flash dump2bin command for making and saving flash dumps first and foremost,
108 allowing any part of the flash to be dumped and saved selectively if desired,
109 including the second flash bank on the Pirelli DP-L10 and likewise on our own
110 FCDEV3B. When it comes to flash write operations, we provide a manual flash
111 erase command that allows (and requires) the operator to specify exactly which
112 sector range should be erased and a manual flash program-bin command that
113 allows any range of 16-bit words to be programmed at any flash address, with
114 the bits to be programmed coming from a binary file, either the whole file or
115 any specified subrange.
116
117 These manual flash erase and flash program-bin commands give full control to
118 the operator, allowing every possible flash manipulation which the hardware
119 itself allows, at the expense of requiring the operator to think about which
120 flash addresses, offsets and lengths need to be operated on, and either enter
121 long commands manually or write a command script.
122
123 Given our historical origins (long before we got to the point of producing our
124 own hardware, we started out by exploring the forbidden GSM realm of devices
125 made by alien manufacturers who were hostile to our cause), our original flash
126 manipulation support in fc-loadtool had been centered around the manual
127 peek-n-poke paradigm, with elementary flash erase and flash program-bin commands
128 as our main staple, and no thought had been given originally to producing
129 functionality that would work like FLUID or like our current flash e-program-m0
130 and e-program-bin commands. But all actively maintained software evolves, and
131 as our FreeCalypso family of projects has matured over the years, we now offer
132 richer functionality covering a wider range of use cases.
133
134 Binary vs. S-records
135 ====================
136
137 (Please read the companion article Binary-file-formats for background, then
138 come back here.)
139
140 If you are exploring and manipulating the flash content of a GSM device in an
141 aftermarket fashion, as opposed to flashing your own fw builds into your own hw
142 design produced by your own company like Openmoko did in the late 2000s and
143 like we do currently at FreeCalypso HQ, then binary is the generally preferred
144 format: you make dumps with flash dump2bin, and when you selectively program
145 these images back into devices, you use flash program-bin with the right offsets
146 and length, along with appropriate flash erase commands.
147
148 We also have flash dump2srec and flash program-srec commands in fc-loadtool,
149 they were implemented back in the founding stage of FreeCalypso in 2013 for the
150 sake of completeness and symmetry (it seemed right to support both binary and
151 S-record formats), but they never got any practical use: if you are making a
152 flash dump, you would normally want to examine it afterward, and any such
153 examination almost always needs a straight binary image, not S-records.
154 Furthermore, our flash program-bin command allows you to selectively program
155 just a particular portion of a binary image file into flash, at any arbitrary
156 flash address, but we don't have the same flexibility with flash program-srec -
157 the latter command is really just a sibling of program-m0 with the opposite
158 byte order.
159
160 Thus the short summary is as follows:
161
162 * If you are flashing an official firmware release image into your device, you
163 need to use flash e-program-bin or flash e-program-m0 depending on whether
164 the image is provided in *.bin or *.m0 format, or alternatively our older
165 flash program-bin or flash program-m0 commands preceded by a separate flash
166 erase command with the right sector range, possibly packaged in a supplied
167 fc-loadtool command script.
168
169 * If you are restoring a flash dump made with flash dump2bin or performing
170 aftermarket flash manipulations on Mot C1xx or Pirelli DP-L10 phones or other
171 such alien devices, you need to use binary-format-based flash manipulations
172 commands; the specific commands will depend on exactly what you are seeking
173 to do.
174
175 * flash program-srec and e-program-srec commands do not currently have a valid
176 use case.
177
178 Special considerations for Compal phones
179 ========================================
180
181 Motorola C1xx and Sony Ericsson J100 phones made by Compal have brickable flash:
182 the right kind of flash-resident bootloader must always be present at the
183 beginning of the flash, or else the phone is unrecoverably bricked. We have
184 special support in fc-loadtool for minimizing the bricking vulnerability window
185 when operating on these phones, but this special support requires user
186 cooperation, meaning that you must limit your flash manipulations on these
187 phones to a narrower subset:
188
189 * flash program-m0, program-srec, e-program-m0 and e-program-srec commands are
190 not appropriate for these brickable phones - do not use any of these commands
191 on these targets.
192
193 * Flash sector 0 must be manipulated only with the special
194 flash erase-program-boot command, not any of the regular erase or program
195 commands.
196
197 * Regular flash erase, flash program-bin and flash e-program-bin commands can
198 and should be used for the rest of the flash starting at offset 0x10000 - but
199 you still need to understand what you are doing.