# HG changeset patch # User Mychaela Falconia # Date 1520749920 0 # Node ID 3d45660f78f0ae420808de5ab38d183f01498097 # Parent b61b81d3cece515ba130eba9d250d55187e6ee1b doc/User-phone-tools article written diff -r b61b81d3cece -r 3d45660f78f0 doc/User-phone-tools --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/User-phone-tools Sun Mar 11 06:32:00 2018 +0000 @@ -0,0 +1,373 @@ +FreeCalypso User Phone Tools are a new software addition to the FreeCalypso +family. These tools are programs that run on a Unix host computer such as a +GNU/Linux PC or laptop and communicate with a FreeCalypso phone or modem via +the standard AT command interface, rather than any of the formerly proprietary +interfaces specific to TI's internal architecture. The following tools are +currently available: + +fcup-at Issues an arbitrary AT command given on the command line. + +fcup-settime Issues AT+CCLK command to the target to set its clock to the + host computer's notion of local time. + +fcup-smdump Retrieves a dump of SMS records (received, sent or stored + messages) from the FC device's SMS storage (currently SIM + storage; ME storage may be implemented in the future), + optionally deleting them from the severely space-limited + SIM/ME storage afterward. + +fcup-smsend* Tools for sending outgoing SMS from a host computer through a + FreeCalypso phone or modem and/or writing such outgoing SMS + into the FC device's SMS storage. + +fcup-smwrite Debug and development tool: writes arbitrary message records + into the FC device's SMS storage (currently SIM storage) in any + of the possible 4 states, with arbitrary incoming or outgoing + SMS PDU content. + +Because these tools communicate with the target via standards-defined AT +commands, in theory they ought to work with any AT-command-speaking 3GPP phone +or modem and not just our own FreeCalypso. However, experience has shown that +in the case of the common proprietary implementations, practice does not match +theory: when I (Mychaela) tried these same AT commands against a random +off-the-shelf proprietary modem (Huawei E303 USB stick modem for 3G), the +following problems were seen: + +* The essential AT+CMGL=4 command for retrieving the full set of SMS records + from SIM storage in PDU mode appears to be broken: all I got was a hang. + Its text mode counterpart AT+CMGL="ALL" produces incomplete output. + +* Qualcomm/Huawei's implementation of the AT command interface does not allow + AT+CSCS to be set to "HEX"; our fcup-smdump implementation uses this setting + so that the phonebook names returned along with SMS PDUs in the +CMGL + responses can be parsed reliably no matter what weird characters they might + contain. + +* Setting AT+CSCS to "8859-1" is not supported either; this setting is used by + our fcup-smsend and fcup-smsendmult tools when sending in text mode. + +* Sending outgoing SMS with fcup-smsend in PDU mode (which does not touch + AT+CSCS) works in that the message goes out, but the tool complains afterward + because the echo after the ^Z is different from what our tools expect. + +Because of these quirks, our FC User Phone Tools officially work only with our +own FreeCalypso phones and modems, and are not expected to work against various +proprietary implementations. Let us not forget that the broken and buggy nature +of the common proprietary implementations is the very reason why we need +FreeCalypso in the first place. + +Target interface options +======================== + +Our fcup-* tools can communicate with the AT-command-speaking target in one of +two ways: + +* The default is the standard AT command interface over a dedicated UART. As + of this writing, the only FreeCalypso device that provides a full-featured AT + command interface of this kind is our FCDEV3B modem, but the ultimate goal of + the project is to build our own end user phone handset (a Libre Dumbphone) + that will also provide a full-featured AT command interface on its USB port + via a built-in CP2102 or FT232R chip. + +* As a dirty hack, one can run FreeCalypso GSM fw on some alien hw targets, + currently Motorola C139 and Pirelli DP-L10. In this hacked-up configuration + there is no dedicated UART available for a standard AT command interface, but + there is a hack that allows a limited subset of AT commands to be passed over + the RVTMUX binary packet interface provided by the running FreeCalypso GSM fw. + Our fcup-* tools can work with such targets to a limited extent. + +The AT-over-RVTMUX mechanism was originally invented back in 2015 as a +development aid, and was never intended for production use or to support any +kind of end user functionality. One of its limitations is that the strings +that are sent to ATI via this interface are limited to 254 characters, whereas +sending or writing SMS in hex format requires longer strings. As a result, SMS +sending and writing functionality via fcup-smsend* is limited when a crippled +Motorola or Pirelli hw target is used instead of proper FreeCalypso hardware. + +All fcup-* tools take the following common command line options for selecting +the AT command target interface: + +-B baud Valid only when -p is also given; selects a different baud rate + than the default 115200 bps. + +-n Dry run debug mode with no target interface at all: the AT + commands which would otherwise be sent to the target are simply + printed on stdout. + +-p ttyport Names the serial port to be used to talk to the target. + +-R Use the AT-over-RVTMUX interface instead of the standard AT + command interface over a dedicated UART. + +-X program Use the specified external program as the AT target + communication back-end; read the source code for the details. + +-R and -p options interact as follows: + +Neither -R The standard dedicated AT command interface is used; +nor -p FC_GSM_DEVICE= environment variable needs to be set + to point to the serial port. + +-p only The standard dedicated AT command interface is used; + the serial port is named with the -p option. + +-R only AT-over-RVTMUX interface is used; the fcup-* tool connects + to an already running rvinterf process. + +-R and -p AT-over-RVTMUX interface is used; a new rvinterf process + is launched to talk RVTMUX on the specified serial port. + +Retrieving and decoding stored SMS +================================== + +As of this writing, our current FreeCalypso GSM firmware supports only SIM +storage for SMS, i.e., there is no working mechanism currently for storing SMS +records (received and sent messages) in the phone's or modem's own flash file +system. The capacity of this SIM SMS storage is determined by the SIM issuer, +but it is typically quite limited, on the order of 20 to 30 messages. + +The model adopted for FreeCalypso is that incoming (and possibly saved outgoing) +messages initially accumulate in the SIM storage as they come in, and then the +user periodically transfers them to her larger host computer, simultaneously +deleting them from the SIM storage to reclaim the limited space. The retrieval +of stored SMS from FreeCalypso GSM devices is accomplished with our fcup-smdump +utility; unlike SMS sending/writing, this operation works exactly the same +whether the FC GSM device offers a full-featured AT command interface or only +AT over RVTMUX. SMS retrieval is always done in PDU mode, and the output from +fcup-smdump contains raw SMS PDUs in the form of long hex strings. A separate +utility called sms-pdu-decode then does what its name says. + +The intended mode of usage is something like this: + +fcup-smdump -d >> long-term-sms-log + +The -d option to fcup-smdump tells it to delete the retrieved messages from the +SIM or future ME storage; this option should only be used when the output is +redirected into some kind of longer-term storage. In the above model the file +named long-term-sms-log becomes what its name says as new messages retrieved +from the FC GSM device get added to it; the format will look like this: + +Received message: +XXXXXX... + +Received message: +XXXXXX... + +Sent message: +XXXXXX... + +Stored unsent message: +XXXXXX... + +Received message: +XXXXXX... + +Each of the "XXXXXX..." lines will be a long hex string giving an SMS PDU. The +idea is that the complete record of all received and sent messages should be +stored on the user's big computer in raw PDU form, rather than decoded, and the +decoding utility sms-pdu-decode should be invoked by the user (with the message +log file as input) as needed for reading these messages. + +The message decoding utility sms-pdu-decode does its best to decode and show +everything without dropping any bits: in addition to the actual decoded message +characters and the From/To address (the "end user" content of the message), it +decodes and shows the SC address, the first octet, the MR octet for outgoing +messages, PID and DCS octets, the SC timestamp or the validity period fields, +and the UDH bytes if present. However, some bits can still be lost in the +decoding, which is why it is important to archive messages in the raw PDU form: + +* Padding bits used to round the From/To address and septet-based user data to + an octet boundary and to round any UDH to a septet boundary are not decoded. + +* If the user data portion of the message is 8-bit or compressed data (per the + DCS octet), it is shown as a raw hex dump, which is lossless, but if it is + GSM7 or UCS-2 text (GSM 03.38 character encodings), the characters are + converted to the user's character set (plain ASCII only by default) for + display, and some characters may not be displayable. + +Character sets and encodings +---------------------------- + +By default, sms-pdu-decode only emits 7-bit ASCII characters in its output; any +GSM7 or UCS-2 characters which fall outside of this plain ASCII repertoire are +displayed as the '?' error character and the presence of such decoding errors +is indicated in the Length: header. This conservative default behaviour can be +modified as follows: + +-e option extends the potential output character repertoire from 7-bit ASCII to +8-bit ISO 8859-1. Any 8859-1 high characters are emitted as single bytes, +i.e., are NOT encoded in UTF-8 - this option is intended for non-UTF-8 +environments. + +-u option extends the potential output character repertoire to the entire Basic +Multilingual Plane of Unicode, and changes the output encoding to UTF-8. + +-h option causes the user data portion of every message to be displayed as a +raw hex dump; in the case of GSM7-encoded messages, this hex dump shows the +unpacked septets. + +Composing and sending outgoing SMS +================================== + +When used with a FreeCalypso GSM device that offers the full AT command +interface (currently only the FCDEV3B modem), the primary SMS sending/writing +tool fcup-smsend offers the following capabilities: + +* Sending outgoing messages in either GSM7 or UCS-2 encoding; +* Sending either single or long (concatenated) SMS; +* Message body input in ASCII, ISO 8859-1 or UTF-8; +* Message body input either on the command line or on stdin; +* Any messages sent through this tool (single or concatenated) may be + multiline, i.e., may contain embedded newlines; +* Messages sent in GSM7 encoding can contain ASCII characters [\]^ and {|}~ + - the tool is smart enough to do the necessary escape encoding. + +The default and preferred AT command interface mode for sending/writing SMS is +PDU mode, which works great when the GSM device provides a proper AT command +interface. However, when a message of maximum or near-maximum length is being +submitted to the modem in PDU mode, the hex string that needs to be sent is +longer than what the crippled AT-over-RVTMUX mechanism can handle, thus if you +are using crippled Motorola or Pirelli hardware, you need to give the -t option +to fcup-smsend or fcup-sendmult, telling these tools to use text mode instead +of PDU mode on the AT command interface. In this text (-t) mode the following +restrictions apply: + +* Only single SMS can be sent, not concatenated; +* Only GSM7-encoded messages can be sent, not UCS-2; +* No multiline messages can be sent, i.e., no newlines in the message body; +* ASCII characters [\]^ and {|}~ won't be sent correctly - GSM 07.05 text mode + drops them. + +But if you have to use FreeCalypso on crippled hardware, the -t option does +allow you to send GSM7-encoded SMS of the full maximum length of 160 characters. +If you attempt to use PDU mode (no -t option) with an AT-over-RVTMUX back-end +(-R option), the send or write operation will fail if the generated PDU is +longer than 127 octets; the length of the generated PDU depends not only on the +message body length, but also on the length of the destination address. + +The invokation syntax is as follows: + +fcup-smsend [options] dest-addr [message] + +The destination address must be given on the command line; the address digits +may be optionally followed by a comma and an address type byte, either decimal +or hexadecimal with 0x prefix. The default address type is 0x91 if the number +begins with a '+' or 0x81 otherwise. If the message body is given on the +command line, it must be given as a single argument; if no message body argument +is given, the message body will be read from stdin. Any trailing newlines are +stripped before SMS encoding. + +The following options are supported, in addition to the common target interface +options listed earlier: + +-c Enables concatenated SMS. Concatenated SMS will be sent only + if the message body exceeds 160 GSM7 or 70 UCS-2 characters, + otherwise plain SMS will be sent whether -c is given or not - + but the -c option enables the possibility of sending + concatenated SMS. + +-C refno Enables concatenated SMS like -c, but also explicitly sets the + concatenated SMS reference number to be used. The number can + be either decimal or hexadecimal with 0x prefix. + +-q Concatenated SMS quiet mode. If -c is given without -q, the + tool prints a message on stdout indicating whether the message + was sent as single or concatenated, and in how many parts. + -q suppresses this additional output. + +-t Use text mode instead of PDU mode on the AT command interface. + This option is incompatible with -c and with -U, and introduces + other restrictions listed above. + +-u By default, if the message body input contains any 8-bit + characters, they are interpreted as ISO 8859-1. With -u they + are interpreted as UTF-8 instead. This option is only relevant + for GSM7 output encoding, and it is implemented by converting + the input first from UTF-8 to 8859-1, and then from 8859-1 to + GSM7 - thus all UTF-8 input characters must fall into the + 8859-1 repertoire, and it is not currently possible to send + GSM7-encoded messages containing the few Greek letters or the + Euro currency symbol allowed by GSM 03.38 encoding. + +-U Send message in UCS-2 encoding instead of GSM7. Any 8-bit + characters in the message body input are interpreted as UTF-8, + and the entire Basic Multilingual Plane of Unicode is allowed. + +-w By default the outgoing message is sent out on the GSM network + with the AT+CMGS command. With this -w option, the message is + first written into SIM or future ME SMS storage with AT+CMGW, + then sent out on the GSM network with AT+CMSS. + +-W Write only, not send: the message is written into storage with + AT+CMGW and no further action is taken. The modem's +CMGW: + responses with message storage indices are forwarded to stdout. + With this option the destination address argument can be a null + string or omitted altogether. + +Concatenated SMS reference numbers +---------------------------------- + +Every concatenated SMS transmission needs a reference number, and this number +needs to increment from one concatenated SMS to the next, to help message +recipients sort out which is which. If the reference number is not given +explicitly with -C, fcup-smsend creates (opens with O_RDWR|O_CREAT) a file +named .concat_sms_refno in the invoking user's $HOME directory; automatically +incrementing reference numbers are maintained in this file. The initial seed +is an XOR of all bytes of the current time returned by gettimeofday(2), +followed by simple linear incrementing; these reference numbers do not need to +be random in any kind of cryptographically secure sense. + +fcup-smsendmult +=============== + +As an alternative to sending concatenated SMS, one can use the fcup-smsendmult +utility to send several single (no UDH) messages in one batch. This utlity +supports both text and PDU modes (PDU mode is still the preferred default when +it can be used), and when PDU mode is used, it supports both GSM7 and UCS-2 +output encodings just like fcup-smsend. The messages to be sent are read from +stdin, and each input line produces a new message. + +The entire batch of messages can be sent to a single recipient, or each message +in the batch can have its own individual destination address. If the +destination address is given on the command line, each input line read from +stdin is just a message body; if no destination address is given on the command +line, each input line must have the following format: + + + +-t, -u, -U, -w and -W command line options are unchanged from fcup-smsend. + +If you have to use FreeCalypso on crippled hardware, fcup-smsendmult -t can be +a viable alternative to sending concatenated SMS, as each message in the batch +can be up to the maximum limit of 160 characters. + +fcup-smsendpdu +============== + +This utility sends out SMS PDUs that have been prepared externally; it only +works in PDU mode, which limits its usefulness to high-end FreeCalypso hardware +with a full AT command interface. The PDUs to be sent out are read from stdin, +one long hex string PDU per line; one can send either a single message or a +batch. Because the destination address and all content details are encoded in +the PDU, the tool does not care if the messages are going to the same recipient +or to different recipients, nor does it care if they constitute a concatenated +SMS transmission or not. -w and -W options work the same way as in fcup-smsend +and fcup-smsendmult. + +fcup-smwrite +============ + +This utility is a debug and development tool; it differs from fcup-smsendpdu in +the following ways: + +* fcup-smsendpdu can send messages out with AT+CMGS, write them into memory + with AT+CMGW, or do a write-then-send sequence (-w option) with AT+CMGW + followed by AT+CMSS. fcup-smwrite only issues AT+CMGW commands. + +* fcup-smwrite passes a second argument to AT+CMGW that sets the message state + to any of the possible 4 values; fcup-smsend* -W put them in the "stored + unsent" state. + +* The input to fcup-smsendpdu is just PDU hex strings; the input to + fcup-smwrite needs to have the same format as fcup-smdump output in order to + indicate what state each message should be written in.