Saturday, 15 April 2017

Embedded Product Design on Linux

An embedded product on Linux involves several stages. As an example, we use a Zigbee embedded product on Linux to discuss this.

The software components that make up an embedded product.

Dynamic UI: HTML, JQuery, Javascript, XML data, CGI process

Application and Libraries: Many applications, and third-party libraries

The figure above shows the interaction among the application processes, CGI app and Dynamic UI.

Linux Kernel and Drivers:
It maybe required to integrate new drivers into Linux kernel space, to enable the SoC on the main board.

It maybe required to modify boot loader if we upgrade to new generation of flash and DRAM IC.

Real-Life Embedded Systems problems and solutions:

  • Zigbee Carrier board integration:

Old carrier board is responding to Zigbee Ping command, and can find its IEEE address.
New carrier board is responding to Zigbee Ping command, but cannot find its IEEE address.

Reasons: Firmware is upgraded in New carrier board, so a upgraded SDK is required to work with the new firmware in New carrier board.

Debugging process: In Linux, open the serial port where the Zigbee module is connected, then set the firmware in carrier board to command mode, then send the Zigbee Ping command to the carrier board.

  • Board Graceful shutdown:

Power-off the board immediately without proper shutdown sequence will cause linux file system corruption.

Solution: Ask MSP430 to set an interrupt to MPC8315 when the power switch is pressed. An interrupt handler is added to linux kernel (as a linux driver module) to handle MSP430 interrupt. The driver module will create a kthread to listen to the interrupt. When interrupt arrives, a user space helper function is called to handler the proper shutdown sequence, such as carrying out the "init 0" command.

  • Move DRAM content forward:

We want to copy DRAM content.


void loader(void)
  long * ptr = loki_plus_dram;
  int i;
  for (i=0; ptr[i]!=0x1FFFF; i++)
    *((long *)ptr[i]) = ptr[i+1];

  • TI Zigbee chipset cc2530 UART 1 setup:

Unable to communicate to TI cc2530's UART 1 port.

In IAR Workbench, Project->Options, choose C/C++ compiler, set:

This is a must for the source code to call hal_uart_isr() function in hal_uart.c

  • Wifi camera lose connectivity to Wifi AP:

Wifi camera lost connection to Wifi AP after 1 day, meaning Wifi AP cannot detect the Wifi camera after 1 day.

In /etc/hostapd.conf, set wpa_strict_rekey = 0
or set wpa_group_rekey = 0 and wpa_ptk_rekey = 0 if maintaining wpa_strict_rekey = 1

  • Glucometer timeout issue:

The Bluetooth program on linux is not able to read all data from Glucometer, read function hang, need to power off/on Glucometer .

Use a function called setsockopt to set socket to non-blocking with timeout.

setsockopt(sk, SOL_SOCKET, SO_RCVTIMEO, (struct timeval *)&tv, sizeof(struct timeval));
ret = read(sk, buf, sizeof(buf));
if (ret == -1)
  read again

  • Add debug statement control:

debug statements are lost if we do not have serial port to monitor them

Define a debug macro

//DEBUG functions
#define LOG_OUTPUT stderr

#define LOG(level, ...) \
        { \
                fprintf(LOG_OUTPUT, level); \
                fprintf(LOG_OUTPUT, __VA_ARGS__); \
                fflush(LOG_OUTPUT); \

#define DEBUG(...)   LOG("Myglucohealth   ", __VA_ARGS__)

Actual usage:
DEBUG("Can't find adapter %s\n", adapter);

Normally the debug statements will be sent to console, but we can redirect them to anywhere with a easy command:
<process_command>  &> /mnt/storage/bluetooth/process_command.log  &
---> redirect both stdout and stderr to a file

  • TCP socket: too many open files:

A TCP client connected to TCP server every 60 seconds and close the connection after sending a message. After 6 hours, socket call failed, saying "socket: too many open files"

The TCP server must close the socket that is returned by accept(). And the TCP client should close the socket first, then the TCP server close it. This is due to the TIME_WAIT settings which will create waiting burden on the TCP server if the TCP server closes the socket first, and if the TCP server is a busy one.

Use netstat to check for open socket connections
Use "ls /proc/<pid>/fd" to check for open file descriptor
Turn off SO_LINGER, call it after open the socket, eg: setsockopt(sd, SOL_SOCKET, SO_LINGER,...);

  • Simultaneous Wifi and Ethernet transmission, and data lost

The problem occurs when there are simultaneous Wifi and Ethernet transmission to the DUT, and DUT does not receive data via the Wifi after a few hours. For this debugging, check the interrupt mask and interrupt enable register bits. On the general interrupt controller (GIC), the Wifi and ethernet bits are adjacent to each other. When data is received at the hardware interface, internet status bit is set. When inside the ISR, interrupt enable bit is disabled, and interrupt mask bit is masked out. When exiting ISR, interrupt enable bit is enabled, interrupt mask bit is cleared. The problem is due to a hardware error that ethernet interrupt bit will affect the Wifi interrupt bit. The software workaround is to enable the Wifi and Ethernet interrupt bits in the ISR when data is received.

Wednesday, 21 December 2016

Bluetooth Protocol and IEEE11073

Bluetooth Protocol Stack:

Bluetooth Stack Partitioning:

host controller - Bluetooth hardware
Bluetooth host - the PC
HCI - Host Controller Interface

Pairing is the process of establish the right to make a connection. Once the units have been paired and provided the paired unit is "trusted" then connecting becomes automatic. Otherwise you must respond each time an attempt is made to connect that it is ok. Keeping the unit in a connected state makes it easy and convenient to accept calls but you do pay for the higher battery drainage.

Pairing is also and act of exchanging of authentication key.

Inquiry Procedure:
The inquiry procedure enables a device to discover which devices are in range, and determine the addresses and clocks for the devices. The inquiry procedure involve a unit (the source)sending out inquiry packets (inquiry state) and then receiving the inquiry reply  .The unit that receives the inquiry packets (the destination),  will hopefully be in the inquiry scan state to receive the inquiry packets. The destination will then enter the inquiry response state and send an inquiry reply to the source. After the inquiry procedure has completed, a connection can be established using the paging procedure.

Paging Procedure:
With the paging procedure, an actual connection can be established. The paging procedure typically follows the inquiry procedure. Only the Bluetooth device address is required to set up a connection. Knowledge about the clock (clock estimate) will accelerate the setup procedure. A unit that establishes a connection will carry out a page procedure and will automatically be the master of the connection. The procedure occurs as follows:

1:   A device (the source) pages another device (the destination ) : Page state

2:  The destination receives the page : Page Scan state

3:   The destination sends a reply to the source. : Slave Response state: Step 1

4:   The source sends an FHS packet to the destination : Master Response state: Step 1

5:   The destination sends it's second reply to the source. : Slave Response state : Step 2

6:   The destination & source then switch to the source channel parameters  :Master Response state: Step 2 &  Slave Response state: Step 3

Bluetooth Protocol:

On Linux, I use Bluez, an open source implementation of Bluetooth protocol. Bluez will run a daemon called bluetoothd, kernel modules called rfcomm, bluetooth, bnep, etc. Bluez is used for receiving and sending the Bluetooth messages.

MCAP - Multi Channel Adaptation Protocol. It is used by HDP. It facilitates the creation of communication link for exchanging commands, and data links to transfer actual Medical Device data. MCAP guarantees reliable transmission of data.

SDP - Service Discovery Protocol. It is used by all Bluetooth profiles to discover available services on remote devices, so that connections over L2CAP can be established.

L2CAP - Logical Link Control and Adaption Protocol. It supports protocol multiplexing, packet segmentation and re-assembly, quality of service, re-transmission, flow control. It is only used for ACL link.

Bluetooth Profile:
It is an interface specification, resides on top of the Bluetooth core specification.
It specifies:
  • user interface formats
  • specific parts of Bluetooth protocol stack used by the profile
Advanced Audio Distribution Profile - A2DP. It defines how high quality audio can be streamed from one device to another over a Bluetooth connection. For example, music can be streamed from a mobile phone to a wireless headset.

Health Device Profile - HDP. It is designed to facilitate transmission and reception of Medical Device data. It also performs SDP behavior, to connect to remote HDP devices.

IEEE 11073 stack. It performs building, transmission, reception, parsing of IEEE PDU packets for the associated IEEE 11073 agent/manager. It links to HDP.

Serial Port Profile - SPP. It is used to simulate RS-232

Generic Access Profile - GAP. It describes the required features of all Bluetooth profiles, including inquiry, connection, authentication procedures.

Device ID Profile - DIP. It provides device specific information through use of the SDP. If vendor specific info is required, this profile provides specific way to acquire this info. It enables identification of the manufacturer, product id, product version. It is useful in allowing a PC to identify a connecting device and download appropriate drivers.

Bluetooth Power Class:
class 1: long range (100m)
class 3: long range (10m)
class 3: long range (10cm)

Bluetooth CoD:
class 0xSSDDdd (three bytes)

The default CoD (Class of Device) is 0x000100 which simply stands for "Computer".

The Bluetooth device class is a high-level description of the bluetooth device, composed of three bytes: the "Major Service Class" (byte "SS" above), the "Major Device Class" (byte "DD" above) and the "Minor Device Class" (byte "dd" above). These classes describe the high-level capabilities of the device, such as "Networking Device", "Computer", etc. This information is often used by clients who are looking for a certain type of service around them.

Where it becomes tricky is that another type of mechanism for service discovery exists: "SDP", as in "Service Discovery Protocol".

In practice, most Bluetooth clients scan their surroundings in two successive steps: they first look for all bluetooth devices around them and find out their "class". You can do this on Linux with the hcitool scan command. Then, they use SDP in order to check if a device in a given class offers the type of service that they want.

Bluetooth Master/Slave:
It is not possible to detrmine bluetooth master or slave programatically.The master/slave roles which a device assumes is invisible to the user(application).The same device can act as a master of one piconet and a slave in an (adjacent) piconet.What needs to be decided in your application is whether the device will act as a client or a server or both and what will be the services(profiles) that it supports.

Bluetooth FH:

All Bluetooth devices participating in a given piconet are synchronized to the frequency-hopping channel for the piconet. The frequency hopping sequence is determined by the master's device address and the phase of the hopping sequence (the frequency to hop at a specific time) is determined by the master's internal clock. Therefore, all slaves in a piconet must know the master's device address and must synchronize their clocks with the master's clock.

During the connection handshake, the slave receives one packet (FHS, Frequency Hop Synchronization) that contains information about the master's BD_ADDR and the clock, so the slave's internal clock can be synchronized.

Bluetooth State Machine:

Bluetooth device can move from one state to a new state.

Bluetooth VDP

Vena Dongle Protocol (VDP) is an application protocol between Bluecore module and host microprocessor.

Host micro without IEEE11073 stack
Host micro with IEEE11073 stack

The messages that are carried across VDP link:
1. HDP messages
2. SPP messages for accessing legacy devices
3, Bluetooth device discovery, device pairing control packets.

Bluetooth Link Management Protocol (LMP)

In the usage of Nonin Oximeter, we have a problem where Oximeter fails to transmit measurement data to Bluetooth module after using for some time.

The root cause is because the oximeter sends out LMP "not_accepted" in response to the LMP "au_rand" message from Bluetooth module. Here is the message trace in terms of LMP opcode. The "not_accepted" message is the 2nd last message. 

au_rand contains a random number that is encrypted by link key. When au_rand is not accepted, meaning the oximeter cannot compute the same number using the link key, meaning the oximeter link key database is corrupted

BT Module                Oximeter 
--------------              ------------------ 
                              not accepted 

In normal operation, Bluetooth module challenges oximeter by sending the random 128-bit value AU_RAND and expects from oximeter the response: 
K is link key, E is Bluetooth authentication function. The oximeter receives the AU_RAND and sends back SRES as response. 

When oximeter sends not_accepted after it receives au-rand, meaning the oximeter link key database is corrupted. 

The behavior in this case is up to the Bluetooth module requesting the authentication. It can demand a new pairing by sending LMP in_rand, or it can refuse the connection and sends LMP detach. So for the Bluetooth module, the Bluez stack simply detach the link. 

So we need to modify Bluez stack for solving this "Oximeter takes measurement but no data is transmitted" problem. 

Bluetooth Packet Format

This is a standard Bluetooth packet:

ACCESS CODEAccess code is used for synchronization, DC offset compensation and identification.
HEADERHeader contains link control (LC) information.
PAYLOADPayload carries voice and data fields of upper layers.

Payload Format

HEADER [8-16] BODY [Indicated
in header]

HEADERHeader is one or 2 bytes longer. Only data fields have a payload header.
BODYPayload body includes the user host information. The length os the body is indicated in the length field of the payload header.
CRC CODEThe 16-bit cyclic redundancy check code is generated by the CRC-CCITT polynomial 210041 (octal).


Asynchronous Connectionless Link. One of the two types of data links defined for the Bluetooth Systems, it is an asynchronous (packet-switched) connection between two devices created on the LMP level. This type of link is used primarily to transmit ACL packet data. The ACL link is a point-to-multipoint link between the master and all the slaves participating on the piconet. In the slots not reserved for the SCO links, the master can establish an ACL link on a per-slot basis to any slave, including the slave already engaged in an SCO link. Only a single ACL link can exist. For most ACL packets, packet retransmission is applied.


Synchronous Connection-Oriented. The SCO link is a symmetric point-to-point link between a master and a single slave in the piconet. The master maintains the SCO link by using reserved slots at regular intervals (circuit switched type). The SCO link mainly carries voice information. The master can support up to three simultaneous SCO links while slaves can support two or three SCO links. SCO packets are never retransmitted. SCO packets are used for 64 kB/s speech transmission.


Data - Medium Rate. An ACL link data packet type for medium rate data. DM1 packets carry information data only, contining a 16-bit CRC code and up to 18 info bytes. They are encoded using 2/3 FEC and the packet can cover up to a single time slot. DM3 packets are the same except they can cover up to 3 time slots, and can carry up to 123 info bytes. DM5 packets are the same again except they can cover up to 5 time slots and can hold up to 226 info bytes. See also Bluetooth.


Data-High Rate. An ACL link data packet type for high rate data. DH1 packets are similar to DM1 packets, except the info in the payload is not FEC encoded. This means the DH1 packet can carry up to 28 info bytes and covers a single time slot. The DH3 is the same except it can cover up to 3 time slots and contain up to 185 info bytes. The DH5 packet is the same again except it can cover up to 5 time slots and contains up to 341 info bytes See also Bluetooth packet types.

Bluetooth D-Bus commands:


dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.freedesktop.DBus.Introspectable.Introspect

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez org.freedesktop.DBus.Introspectable.Introspect

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/690/hci0 org.bluez.Adapter.RemoveDevice objpath:/org/bluez/690/hci0/dev_00_1C_05_00_4A_AD

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/1282/hci0 org.bluez.Adapter.ListDevices

dbus-send --system --type=method_call --print-reply --dest=org.bluez /  org.bluez.Manager.FindAdapter string:hci0

dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.bluez.Manager.ListAdapters

Antidote Healthd:

dbus-send --system --type=method_call --print-reply /com/signove/health org.freedesktop.DBus.Introspectable.Introspect

Bluetooth IEEE11073 

IEEE11073 is an Bluetooth application layer protocol, used by HDP devices. IEEE11073 specifies the formats and standards that HDP devices use to send messages. The standard itself is long winded and talking about it is time consuming. IEEE11073 is promoted by the Continua Alliance.

On Linux, I use Antidote, an open source implementation of IEEE11073. It is developed by Signove Technologies. It is written in C.

As an example, we can sniff an Bluetooth Nonin 9560BT Oximeter. Oximeter is set as Master. Bluetooth host is slave. If encrypted data that is captured is in error, we need to look at these LMP frames.


If these LMP frames are missing, we need to delete the oximeter device from Bluetooth database, then re-pair the device. It is because the link key is generated at local and remote devices during pairing stage. If devices are already paired, then link key generation won't be performed by the devices (link key is saved in devices already), and sniffer won't be able to capture the link key, and can't decrypt the bluetooth packets.

The IEEE11073 message exchange is shown below.

Link Management (LM):

The LM assists in the pairing procedure by communicating parameters and results between the local and remote Bluetooth devices. The calculations are done in the baseband. On the link manager level, the pairing procedure starts with the transmission of the PDU LMP_in_rand (containing the 128-bit random number
IN_RAND) from one of the units to the other. This PDU will trigger the generation of the initialization key for use in the protocol for creating the actual link key. If a unit key is to be used, the LMP unit key
command is sent in one direction only, with the secret unit key XORed with the initialization key as its parameter. The receiver can easily calculate the key from this. If a combination key is to be created, two contributions (one from each side) are needed. This is accomplished through the LMP_comb_key PDU. The argument of this PDU is also a random number, but generating this and deriving the link key from it is slightly more complicated. 

After these PDUs have been exchanged, both ends are able to compute the link key. As a final check of success of this procedure and the established link key, a mutual authentication event is performed. A 128-bit challenge is sent through the LMP_au_rand , and the 32-bit response uses the LMP sres PDU. If these authentications do not fail, both the local and the remote Host Controller notify their respective hosts.

Click on the image below to see the link key exchanges.

Tuesday, 15 November 2016

Zigbee Tutorial

IEEE802.15.4 defines the Physical and MAC layers, and ZigBee defines the network and application layers. Zigbee layers build on top of IEEE802.15.4.

Zigbee provides the features of low power consumption, low cost, small footprint, and mesh networking capabilities.

Two version: Zigbee 2006, Zigbee Pro (2007)
Zigbee Pro can scale up to thousands of devices.

Zigbee Architecture:

Application Profile is defined by Zigbee specification.
Eg. Home Automation (HA) and Smart Energy (SE).

Devices within an application profile communicate with each other by means of clusters, which may
be inputs to or outputs from the device.

Eg. In HA profile, cluster 0x0006 is the on/off cluster. Cluster 0x0702 is the zigbee power consumption report cluster. In HA profile, to use Zigbee on/off switch to control Zigbee power switch, we can bind the endpoint number of Zigbee on/off switch, with endpoint number Zigbee power switch, on the cluster 0x0006.

An endpoint defines a communication entity within a device. Zigbee Device Object (ZDO) is always endpoint zero. In all, 240 endpoints are available for use within ZigBee device, with endpoint zero dedicated to the ZigBee Device Object (ZDO), which provides control and management commands.

Joining a Zigbee network:

A ZigBee router or coordinator that wishes to allow other devices to join must issue a NLME-PERMIT-JOINING.request.  The joining device must issue a NLME-JOIN.request with the rejoin flag set to FALSE.

ZigBee Security:

It is based on the 128-bit AES algorithm.

Zigbee coordinator also functions as the trust center when security is enabled on the network.
  • Master key - used as an initial shared secret to generate Link key
  • Trust center master key - used to establish the Link key
  • Link Key - at the Application layer, two types: Trust center link key; application layer link key. Link key is used to encrypt the network key
  • Network Key - at the Network layer
Zigbee Concepts:

Three type of devices:
Zigbee end device, Zigbee router, Zigbee coordinator.

PAN ID - Personal Area Network Identifier. The Zigbee coordinator assigns a PAN ID to the network.

Binding - connecting endpoints
Device discovery - finding out other Zigbee devices, requires
  • IEEE address
  • NWK address
Service discovery - finding out services available on endpoints at the receiving device

IEEE802.15.4 Concepts:

IEE802.15.4 operating frequency is at 2.4GHz and 915MHz.
2.4GHz: channel 11-26
915MHz: channel 1-10

IEEE802.15.4 defines 4 types of MAC frame:
  • beacon frame
  • data frame
  • Ack frame
  • MAC command frame

IEEE802.15.4 data frame in action:

Zigbee Home Automation profile:

The Zigbee HA profile defines various device types that have various roles in the HA network. Device types such as Power Outlet(0x0009), On/Off Switch(0x0000), Combined Interface(0x0007), Pump(0x0303), Temperature sensor(0x0302), IAS devices; are frequently used.

Intruder Alarm Systems - IAS
  • IAS Zone(0x0402) - door sensor, smoke sensor
  • IAS Warning Device(0x0403) - siren
  • IAS ACE(0x0401) - remote control
  • IAS CIE(0x0400) - usually it is the Zigbee coordinator
To control zigbee devices. Firstly, zigbee devices must join the zigbee network. Then, Zigbee coordinator can send a data frame to the zigbee devices, to perform desired action. Zigbee devices can also report its status periodically to the zigbee coordinator.

On/Off switch can bind to power outlet, to control the power outlet directly. It is useful when the zigbee coordinator is down.

Zigbee Certification:

We can do testing in four modes:

Zigbee continuous TX
Zigbee continuous RX
Zigbee packet TX
Zigbee packet RX

For TX test, we can set TX channel, TX power, use range extender (an IC) to obtain higher TX power, send modulated or unmodulated signals.

To perform a Zigbee RX test, we need to setup a Zigbee end device and a Zigbee coordinator. The Zigbee end device can join the Zigbee coordinator, provided with the right key and PAN ID.

Wednesday, 14 October 2015

Device Tree in Linux

A device tree is the tree data structure that describes the physical devices in the system hardware. In Linux, the device tree source (DTS) files are located in arch/<xxx>/boot/dts

  •   dts for board level files
  •   dtsi for Soc level definitions

A device tree compiler (DTC) compiles the source into binary files. The DTC source code is in scripts/dtc.c.

The device tree blob (DTB) is produced by the DTC compiler. It is loaded by the bootloader and parsed by kernel at boot time.

Let's look at an real example where a platform driver is using two clocks.

In xxx_probe(struct platform_device *pdev)
    my_port->aaclk = clk_get(&pdev->dev, "aa");
    my_port->bbclk = clk_get(&pdev->dev, "bb");

The corresponding device tree has to reflect the two clocks.

uart0: serial@3000 {
compatible = "xxx,uart";
reg = <0x3000 0x100>;
interrupt-parent = <&gic>;
interrupts = <11 12>;
clocks = <&oscclk 0>,<&oscclk 1>;
clock-names = "aa","bb";

The two clock-names needs to have two matching clocks source.

Let's look at at an advanced example of device tree usage:

This example shows an oscillator clock being fed to pll1, pll2 and pll3. The hw3_clk can be derived from either pll2, pll3 or osc clock. 

To model the hardware using device tree. We declare the following entries.

osc: oscillator {
#clock-cells = <0>;
compatible = "fixed-clock";
clock-frequency = <20000000>;
clock-output-names = "osc20M";

pll2: pll2 {
 #clock-cells = <0>;
compatible = "abc123,pll2-clock";
clock-frequency = <23000000>;
clocks = <&osc>;
reg = <0x05 0x10>;

pll3: pll3 {
 #clock-cells = <0>;
compatible = "abc123,pll3-clock";
clock-frequency = <23000000>;
clocks = <&osc>;
reg = <0x15 0x10>;

hw3_clk: hw3_clk { 
    #clock-cells = <0>; 
    compatible = "abc123,hw3-clk"; 
    clocks = <&pll2>,<&pll3>; 
    clock-output-names = "hw3_clk"; 

In the source code, to register hw_clk3 as a mux, and show the parent relationship of both pll2 and pll3, we declare the below.

of_property_read_string(node, "clock-output-names", &clk_name); 
parent_name[0] = of_clk_get_parent_name(node, 0); 
parent_name[1] = of_clk_get_parent_name(node, 1); 

clk = clk_register_mux(NULL, clk_name, parent_name, 
      ARRAY_SIZE(parent_name), 0, 
      regs_base , offset_bit, one_bit, 0, NULL);

Sunday, 8 March 2015

More USB stuff

Useful USB debug knowledge:

You can test the USB mass storage gadget's behavior with a Linux host.

To send a Set-Config request with value 0, to de-configure the gadget

        echo 0 >/sys/bus/usb/devices/.../bConfigurationValue

To send a Set-Config request with value 1, to re-configure the gadget

        echo 1 >/sys/bus/usb/devices/.../bConfigurationValue

where the "..." part is replaced with the gadget's device path.

Set-Config has only two stages, Setup and Status stage; there is no Data stage.

Host                        Device
----- Setup Packet --->         |
----- Data0 Packet --->         |== Setup stage
<---- Ack Packet ------         |

----- In Packet ------>         |
<---- Data1 Packet ----         |== Status stage
----- Ack Packet ----->         |

The Data1 packet above contains no data, as below.

      PID   !PID  CRC

USB software layering:

Client SW:

It decides what transfers are to be made with the USB device function.

USB Driver (USBD):

The software interface between the USB System SW and the Client SW. USBD provides the device drivers that the operating systems use to access USB devices.

USB System SW:

It is usually supplied with the operating systems. It can be seemed to include USBD and HCD.

Host Controller Driver (HCD):

The software interface between Host Controller and USB System SW. This interface allows a USBD to support different Host Controllers. HCD is an abstraction of the Host Controller hardware.

Host Controller:

It manages USB protocol level interface. It ensures USB bus access rules defined by the protocol are obeyed, such as inter packet timings, timeouts.
USB Host and Device Partitioning
Control Transfer:

It is started with setup transaction from Host to Device. The Setup transaction is followed by zero or more Data transactions. A Status Transaction completes the control transfer.

Setup Transaction consists of:
setup packet
data0 packet - the request
Ack packet

Data transaction
In packet
data1 packet - the data that is returned by device
Ack packet

Status transaction
out packet
data1 packet - has no data
Ack packet

USB Device Request:

The USB host uses USB requests to get info from USB devices. These request are made using control transfer, from host to device default control pipe.The standard requests are:

  • Get Status
  • Clear Feature
  • Set Feature
  • Set Address
  • Get Descriptor
  • Set Descriptor
  • Get Configuration
  • Set Configuration
  • Get Interface
  • Set Interface
  • Synch Frame

An example of USB request sequence in USB Command Verifier test:

80 06 00 01 00 00 12 00  -  Get device descriptor

Data(12 01 00 02 00 00 00 40 25 05 A5 A4 33 03 01 02  00 01)  - Return device descriptor
80 06 00 02 00 00 09 00 – get config descriptor

Data(09 02 20 00 01 01 04 C0 01) – return config descriptor
00 09 01 00 00 00 00 00 – set configuration

0 byte data
80 06 00 02 00 00 09 00 – get config descriptor

09 02 20 00 01 01 04 C0 01 – return config desc
80 06 00 02 00 00 20 00 – get config desc

0 byte data
80 06 00 01 00 00 12 00 – get device desc

Data(12 01 00 02 00 00 00 40 25 05 A5 A4 33 03 01 02  00 01) – return device desc
80 06 00 06 00 00 0A 00 – get device qualifier

Data(0A 06 00 02 00 00 00 40 01 00) – return device qualifier
80 06 00 02 00 00 09 00 – get config desc

Data(09 02 20 00 01 01 04 C0 01) – return config desc
80 08 00 00 00 00 01 00 – get configuration

Data(01) – return

USB Descriptor:

USB devices report their attributes using descriptors. A descriptor is a defined data structure. Standard USB descriptors are:

  • Device descriptor
  • Device Qualifier descriptor
  • Configuration descriptor
  • Interface descriptor
  • Endpoint descriptor
  • String descriptor

Sunday, 4 January 2015

USB Mass Storage Device Enumeration, and usbmon

An usbmon text output dissection

Some quick commands to before running usbmon:

# mount -t debugfs none_debugs /sys/kernel/debug
# modprobe usbmon
# cat /proc/bus/usb/devices
/*to find out the usb devices number on the usb bus*/

# cat /sys/kernel/debug/usb/usbmon/2u > /tmp/mon.out
As an example, a control packet:

f4ae6f40 1217192721 S Co:2:018:0 s 21 ff 0000 0000 0000 0

f4ae6f40 - URB Tag

1217192721 - Timestamp in microseconds

S - Event Type, S is submission

Co:2:018:0 - "Address". It consists of four fields, separated by colons: URB type and direction, Bus number, Device address, and Endpoint number. 
Co is Control output.

s - URB Status. This is a letter, so 's' is Setup Tag of control packet. It can also be several numbers separated by colons.

21 ff 0000 0000 0000 - Setup packet, if present, consists of 5 words: one of each for bmRequestType, bRequest, wValue, wIndex, wLength

0 - Data Length. For submissions, this is the requested length. For callbacks,this is the actual length.

Now we will look at mass storage device enumeration within the USB gadget framework. The USB gadget framework is available in Linux and U-boot.

USB halt and wedge feature:

Two types of events may cause a USB halt condition.

1) The device did not receive a handshake packet from USB host after communication.
2) The device receive set halt request from the host

Set halt and set wedge are basically the same, with the difference that, the device will ignore clear halt request from the host if device is in set wedge condition.

USB mass storage gadget enumeration steps:

The USB code can be divided into two parts: Usb Mass storage gadget code and Usb device controller code.

Firstly, the host PC sends the setup data to get descriptors from Usb device. It is performed using control transfer.

So the Usb chip receives it and interrupt is triggered. In Usb device controller code,
myudc_isr(void) [the interrupt handler]

reads the IVECT register and gets 0x00 value. It passes control to gadget code
fsg_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)

The setup data is processed, the response is prepared. If there is data to send out to EP0, the device controller function
my_ep_queue(struct usb_ep *ep, struct usb_request *req, gfp_t gfp_flags)

is called to send out the data.

After device, configuration, interface, endpoint, string descriptors are received by the host PC, the host PC detects the Usb device as a mass storage device (that is for a Usb mass storage device).

Device Descriptors:
As an example, the device descriptor is dissected here.

First Byte : bLength (the size of the descriptors in bytes) 0x12
Second Byte: bDescriptor (device descriptor)                 0x1
3rd-4th Byte: bcdUSB (USB spec number)                   0x0002
5th Byte: bDeviceClass (class code)                              0x00
6th Byte: bDeviceSubClass (subclass code)                   0x00
7th Byte: bDeviceProtocol (protocol code)                   0x00
8th Byte: bMaxPacketSize (max. packet size)                0x40
9-10 Byte: idVendor (vendor ID)                                  0x2505
11-12 Byte: idProduct (product ID)                               0xA5A4
13-14 Byte: bcdDevice (device release number)             0x0103
15th byte: iManufacturer (index of manufacturer string desc)      0x01
16th byte: iProduct (index of product string desc)                      0x02
17th byte: iSerialNumber (index of serial num. string desc)         0x03
18th byte: bNumConfigurations (number of possible config.)      0x01

Next step, the host PC will send SCSI commands to Usb device. The SCSI command is wrapped in CBW and sent using bulk transfer.

 So the Usb chip receives it and interrupt is triggered. In Usb device controller code,
myudc_isr(void) [the interrupt handler]

reads the IVECT register and gets 0x28 value for EP1 OUT IRQ. The data is read from FIFO, and sent to gadget function
bulk_out_complete(struct usb_ep *ep, struct usb_request *req)

in gadget code. This function will wake up thread in get_next_command((struct fsg_dev *fsg) and the thread will process the SCSI command. The get_next_command() is waiting for a buffer, which must be the same as the buffer returned by interrupt handler.

Again, if there is data to be replied to host PC, the device controller function
my_ep_queue(struct usb_ep *ep,  struct usb_request *req, gfp_t gfp_flags)
is called to send data to EP1 IN using bulk transfer.

After that, for every SCSI command, the gadget code will call gadget function
send_status(struct fsg_dev *fsg)

to prepare the CSW, and CSW is sent to the host PC using device controller function
my_ep_queue(struct usb_ep *ep,  struct usb_request *req, gfp_t gfp_flags)

 via EP1 IN.

The Transfer Protocol:

The Bulk Only Transport (BOT) is defined by Usb mass storage specification. It defines how Usb host can use bulk transfer to send commands and receives response from Usb device.

In the first transfer, the host sends a command in a structure called a Command Block Wrapper (CBW). The CBW is 31 byte in length. Many CBWs are followed by a transfer that contains data sent to the host or device.
For example, the CBW structure of a SCSI Inquiry command:

dCBWSignature  0x55 0x53 0x42 0x43
dCBWDataTransferLength   0x24
bCBWCBLength             0x06 (The valid length of CBWCB in bytes)
Operation Code               0x12
enableVitalProductData    false
commandSupportData       0x0
page or opcode                  0x0
allocation length                0x24
control                              0x0

In the final transfer, the device returns status in a structure called a Command Status Wrapper (CSW).

CSW structure:

dCSWSignature    0x55 0x53 0x42 0x43
dCSWDataResidue    0x0
bCSWStatus               0x0

The SCSI Commands:

The SCSI commands used by the Usb mass storage device are from SCSI Primary Commands (SPC-2) and SCSI Block Commands (SBC-2).

Some of the commonly used commands, together with the command codes, are listed below.

SCSI Inquiry                0x12
SCSI Request Sense    0x3
SCSI Read Capacity    0x25
SCSI Read 10             0x28
SCSI Mode Sense 6    0x1A
SCSI Mode Select 6    0x15
SCSI Test Unit Ready  0x00

SD Card Partition Table:

The SD card will try to mimic a hard disk.

The MBR is located at offset 0x0, size is 512 bytes.

                      boot code (first 446 bytes)
                      partition #1  0x1BE (each partition is 16 bytes)
   MBR           partition #2
                       partition #3
                       partition #4
                       0xaa55   0x1FE  (signature)

Partition table info: (from offset 0x1BE)

0x00  0x80 if bootable, else 0x0
0x01  start of partition in CHS
0x04  type of partition
0x05  end of partition
0x08  relative offset to the partition in LBA (Partition LBA begin)
0x0C  size of the partition

For example, if the relative offset in LBA is 0x2000, it means the start of the boot sector is 0x2000.

At 0x2000, it is the FAT boot record (if formatted in FAT file system).

For example, FAT32 boot record:

0x00  3 bytes    0xeb0090       jump instruction
0x03  8 bytes    MSDOS5.0    OEM name
0x0B  2 bytes    0x0200           bytes per sector
0x0D  1 byte      0x08               sectors per cluster
0x0E   2 bytes     0x20               number of reserved sector (usually 0x20)
0x10   1 byte      0x2                 number of FATs
0x11   2 bytes                           N/A
0x13   2 bytes                            N/A
0x15   1 byte    0xF8                 media descriptors(F8 for hard disk)
0x16   2 bytes                            N/A
0x18   2 bytes                           sectors per track
0x1A  2 bytes                             number of heads
0x24   4 bytes                         sectors per FAT
0x2C   4 bytes                          root directory first cluster
0x5A  420 bytes                      bootstrap code
0x1fe  2 bytes    0x55aa           end of boot sector mark

Important info that are needed for accessing the FAT32 filesystem:

(unsigned long)fat_begin_lba = Partition_LBA_Begin + Number_of_Reserved_Sectors;
(unsigned long)cluster_begin_lba = Partition_LBA_Begin + Number_of_Reserved_Sectors + (Number_of_FATs * Sectors_Per_FAT);
(unsigned char)sectors_per_cluster = sectors_per_cluster;
(unsigned long)root_dir_first_cluster = root_directory_first_cluster;

Wednesday, 19 November 2014

[iOS App] 'NSInvalidArgumentException', reason : unrecognized selector sent to Instance

I am going through iOS App development recently. I use modal segue to move from parent ViewController to child ViewController, as shown in the picture below.

When i return from child ViewController to parent ViewController.  There is an error.

2014-11-20 09:00:48.229 Heritage[64395:60b] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[XYZContinentTableViewController topViewController]: unrecognized selector sent to instance 0x8c60050'

The error shows up when i have 3 level of ViewController, and specifically, from second level ViewController to root ViewController.

I found out the root cause. It is because the unwind segue

- (IBAction)unwindToRegion:(UIStoryboardSegue *)segue
{  }

is mixed up with prepareForSegue.

- (void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender
To fix the error, in prepareForSegue, add the segue identifier checking. Make sure prepareForSegue is only processing the forward segue (In our case, the UserShowSegueCountry).

- (void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender
    if ([[segue identifier] isEqualToString:@"UserShowSegueCountry"])
    UINavigationController *nav = segue.destinationViewController;
    XYZCountryTableViewController *vcToPushTo = (XYZCountryTableViewController *)nav.topViewController;


Wednesday, 15 October 2014

IP address conflict and its detection

We have an IP network, with devices getting their IP address from DHCP server. A problem could arise when a device is off the network, then reconnect to the network, and its IP address is the same as other device which has just joined the network. This could be due to DHCP server thinks that the DHCP lease has expired, but the device doesn't. This is a problem of IP address conflict.

To detect IP address conflict, we can use two methods

1) ARP probe: It is an ARP request. The sending device fills the source MAC address with the hardware address of its interface, and the source IP address must be all ZEROes. The destination MAC address should be all ZEROes. The destination IP address is the address being probe. If the sending device receives any ARP on the interface where probe is being performed, where the packet's source IP is the IP being probed for, then this IP is used by other host. So it is an IP conflict.

2) Gratuitous ARP: It could mean both gratuitous ARP request or gratuitous ARP reply. Gratuitous in this case means a request/reply is not required, but could be used in some cases. In a gratuitous ARP request,  the source and destination IP are both set to the IP of the device issuing the packet, and the destination MAC is the broadcast address ff:ff:ff:ff:ff:ff. Ordinarily, no reply packet will occur. A gratuitous ARP reply is a reply to which no request has been made. When a device receives an ARP request containing a source IP that matches its own, then it knows there is an IP conflict.

Tuesday, 2 September 2014

Java Serializer

In Computer Science, serialisation is a process of translating data structures or objects into a byte stream that can be stored (in a file or memory buffer), or can be transmitted across a network link, and the byte stream can be re-read and re-constructed into the same clone.  Serialisation is a way to prevent the byte ordering (endianness), memory layout problem.

In Java, the implement the serialisation. You can write a Java class to implement the Serializable interface.

class IsSerializable implements Serializable  
    private String plug = "Plugin to Java!";  
    public IsSerializable()  
            FileOutputStream out = new FileOutputStream("/mnt/sdcard/yes-out.txt");  
            ObjectOutputStream oos = new ObjectOutputStream(out);  
        catch (Exception e)  


protected void onCreate(Bundle savedInstanceState) {

        IsSerializable is = new IsSerializable();  

If you run this code, there is an error message: com.example.xxxxx.MainActivity.

The reason is that you can't get the whole class to be serialised.

If we replace oos.writeObject(this); with oos.writeObject(plug);  the problem is fixed.