Wednesday, 2 August 2017

Bitcoin core tutorial and source code walk through (Part 4)

1) Today i will talk about some basic concepts in cryptography and its applications in Bitcoin.

- one way mapping
- arbitrary length input to fixed length (smaller) output
- many to 1 mapping
- hashed message authentication code (HMAC )
- difficult to reverse

- 1 to 1 mapping
- reversible

private key --> public key --> bitcoin address
      ECC encryption         hashing
            K = G*k
G: generator function
k: private key
K: public key

Bitcoin network node with 4 functions
- wallet, miner, blockchain, network routing

Merkle root: binary hash tree used in a block, it is a summary of transactions in a block

Fork: a change in consensus rules

Colored coins:
- use bitcoin transaction to record extrinsic assets
- bitcoin is intrinsic to the blockchain
- avoid using a colored coin related key in a regular bitcoin wallet,
- colored coins should not be sent to address managed by regular wallets

- coinbase transaction is first transaction in each block
- coinbase in genesis block cannot be spent, and that coinbase transaction cannot be displayed

- pay to public key hash
- hide the public key until users are ready to spend their coins
- P2PK is pay to public key
- the first bitcoin transaction is P2PK

2) In this section, i will show you how to get the transaction

$ ./src/bitcoin-cli -regtest getblock 60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986
  "hash": "60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986",
  "confirmations": 511,
  "strippedsize": 226,
  "size": 226,
  "weight": 904,
  "height": 1,
  "version": 536870912,
  "versionHex": "20000000",
  "merkleroot": "2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34",
  "tx": [

The command getblock returns the txid of the block. The "tx" field contains the txid.

$ ./src/bitcoin-cli -regtest gettransaction 2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34
  "amount": 50.00000000,
  "confirmations": 511,
  "generated": true,
  "blockhash": "60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986",
  "blockindex": 0,
  "blocktime": 1499419297,
  "txid": "2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34",
  "walletconflicts": [
  "time": 1499419297,
  "timereceived": 1499419297,
  "bip125-replaceable": "no",
  "details": [
      "account": "",
      "address": "mhTqgiUaBFPgA2kPrKqDNVqh8xsrkBMkGu",
      "category": "generate",
      "amount": 50.00000000,
      "vout": 0
  "hex": "02000000010000000000000000000000000000000000000000000000000000000000000000ffffffff03510101ffffffff0200f2052a01000000232102910a020f84bca55765fdfc4ef9e788cd6ee137242338533543f6914979fb7ce5ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf900000000"

The command gettransaction returns the object about the given transaction.

3)  In this analysis, the source code that handles gettransaction RPC will be walked through.

For gettransaction rpc

UniValue gettransaction(const JSONRPCRequest& request)

The function firstly get a pointer to the CWallet object.
It creates a CWalletTx object, CAmount object, call WalletTxToJSON() to display the output in JSON format. The ListTransactions() will include the sent and received transaction.

Then EncodeHexTx()  encodes a string, and returns the UniValue to the RPC caller.

Sunday, 9 July 2017

Bitcoin core tutorial and source code walk through (Part 3)

1) The bitcoin-cli command can be used to get blockhash and get block For testing purpose, the regtest network is used.

bitcoin-cli -regtest getblockhash 0
(it returns the block hash)
bitcoin-cli -regtest getblock 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
(it takes block hash as parameter)

2) In this analysis, the source code that handles getblockhash and getblock RPC will be walked through.

For getblockhash rpc
define RegisterBlockchainRPCCommands(CRPCTable &t)

UniValue getblockhash(const JSONRPCRequest& request)

This function checks the block height, make sure it is not out of range. Then it calls CBlockIndex class GetBlockHash() function


GetBlockHash() simply returns the pointer to hash block. 

For getblock rpc
UniValue getblock(const JSONRPCRequest& request)
This function firstly check the verbosity. It checks mapBlockIndex is not out of range. The it calls ReadBlockFromDisk() and blockToJSON() to print out the block data in JSON format.

UniValue blockToJSON(const CBlock& block, const CBlockIndex* blockindex, bool txDetails)

It builds the JSON value pair using the CBlock and CBlockIndex.


bool ReadBlockFromDisk(CBlock& block, const CDiskBlockPos& pos, const Consensus::Params& consensusParams)
This function builds a CAutoFile object which contains a FILE *. The OpenBlockFile() is used to return the FILE *. Then, the FILE * is copied to CBlock.

FILE* OpenBlockFile(const CDiskBlockPos &pos, bool fReadOnly)

It is a misc function that calls OpenDiskFile().

static FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)

It uses fsbridge::fopen() to build FILE *, and returns FILE *. The fsbridge is Bitcoin core namespace.

mapBlockIndex is defined in validation.cpp as

BlockMap mapBlockIndex;
and extern BlockMap mapBlockIndex; in validation.h.

typedef std::unordered_map<uint256, CBlockIndex*, BlockHasher> BlockMap;

It is unordered_map in C++ std namespace .

Wednesday, 5 July 2017

Bitcoin core tutorial and source code walk through (Part 2)

1) The bitcoin-cli command can be used to send bitcoin to any recipient address. For testing purpose, the regtest network is used.

bitcoin-cli -regtest getaccountaddress ""  - get the address from default account
bitcoin-cli -regtest sendtoaddress mjSTBVs....... {amount} <comment>  - this returns transaction id

2) In this analysis, the source code that handles sendtoaddress RPC will be walked through.

In HTTPReq_JSONRPC(HTTPRequest *req, const std::string &)
call jreq.parse(valRequest)    - parse the request
call tableRPC.execute(jreq)  - execute the RPC comamnd
In CRPCTable::execute(const JSONRPCRequest &request)
initialise pcmd
 CRPCCommand *pcmd = tableRPC[request.strMethod]
The tableRPC array is filled in init.cpp
In AppInitParameterInteraction()
call RegisterAllCoreRPCCommands(tableRPC)
and RegisterWalletRPCCommands(tableRPC)
define the function RegisterAllCoreRPCCommands(CRPCTable &t), which calls
define RegisterBlockchainRPCCommands(CRPCTable &t)
define RegisterNetRPCCommands(CRPCTable &t)
define RegisterMiningRPCCommands(CRPCTable &t)
define RegisterRawTransactionRPCCommands(CRPCTable &t)

Last but not least, RegisterWalletRPCCommands is defined in
The RegisterWalletRPCCommands(CRPCTable &t) fill up the CRPCCommand array.

In the CRPCComamnd,  actor function pointer, sendtoaddress function is assigned to that field.

The function
UniValue sendtoaddress(const JSONRPCRequest & request)
Firstly, the address is gotten from parameter 0. Then, the amount is gotten from parameter 1. The it calls the SendMoney() function, and returns the hash value of the transaction.

The function
static void SendMoney(CWallet * const pwallet, const CTxDestination &address, CAmount value, bool fSubtractFeeFromAmount, CWalletTx & wtxNew)
Firstly, it gets the balance from the wallet, parse the bitcoin address, find out the recipient.
Then, it declares std::vector<CRecipient> vecSend
and append the recipient to the array vecSend
Then, it calls CWallet function CreateTransaction and CommitTransaction.

In the CreateTransaction(const std::vector<CRecipient>& vecSend, CWalletTx& wtxNew, CReserveKey& reservekey, CAmount& nFeeRet, int& nChangePosInOut, std::string& strFailReason, const CCoinControl* coinControl, bool sign)

Firstly, it checks the recipient amount , which must be >= 0. Then, it sets the lockTime to the next block to discourage fee sniping. The CTxOut txout object is constructed. The txout is appended to CMutableTransaction txNew vout vector. Then, it chooses the Bitcoin to use for transaction. If input value is greater than selected value, it creates a CTxOut and insert it to same CMutableTransaction vout at position nChangePosInOut.

After that, it creates CTxIn and appends the CTxIn to vin vector. It calculate the minimum fee needed. If the fee needed is less than min relay fee, the transaction is dropped. Then the min fee inclusive of necessary fee is calculated.

If transaction requires signature, a signature is produced (via ProduceSignature function). UpdateTransaction() is called to update with signature data.

Then, the CMutableTransaction txNew is embedded in CWalletTx wtxNew object. If wtxNew size is bigger than mempool's chain limit, transaction will be dropped.

In the bool CWallet::CommitTransaction(CWalletTx& wtxNew, CReserveKey& reservekey, CConnman* connman, CValidationState& state)

It calls AddToWallet(wtxNew) and coin.BindWallet(this) to notify old coins are spent.
It then broadcast transaction if wtxNew is accepted to mempool

bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFlushOnClose)

It creates the walletdb. It creates the std:pair object from mapWallet.insert(std::make_pair(hash, wtxIn)). From the std::pair object, it creates another CWalletTx wtx object. If it is new transaction, it gets the order position from walletdb, and adds the hash. If it is not new, the code merges the hash block. Finally, it writes to disk with walletdb.WriteTx(wtx).

mapWallet is declared in wallet/wallet.h as
std::map<uint256, CWalletTx> mapWallet;

Tuesday, 4 July 2017

Bitcoin core tutorial and source code walk through (Part 1)

Bitcoin core is the reference implementation of Bitcoin code.

1) Download Bitcoin core from github

2) Build it
./configure --enable_debug
make install

3) Setup the bitconf.conf. In the minimum, setup the rcpuser and rpcpassword field
bitconf.conf configuration file path:
OS                Default bitcoin datadir      Typical path to configuration file
Windows     %APPDATA%\Bitcoin\     C:\Users\username\AppData\Roaming\Bitcoin\bitcoin.conf
Linux           $HOME/.bitcoin/               /home/username/.bitcoin/bitcoin.conf
Mac OSX    $HOME/Library/Application Support/Bitcoin/             /Users/username/Library/Application Support/Bitcoin/bitcoin.conf

4) Useful commands
bitcoind -daemon  ; bitcoin-cli stop      - start and stop bitcoin daemon
bitcoind -printtoconsole                        - print debug message to console
bitcoind -daemon -proxy=   - run bitcoind w/o downloading new blocks
bitcoind -testnet    bitcoin-cli -testnet      - run in test network
bitcoind -regtest    bitcoin-cli -regtest     - run in regression test network
bitcoind -debug=rpc                              - set the debug level

bitcoin-cli getblockhash <number>
bitcoin-cli getblock <hash>
bitcoin-cli getblockcount
bitcoin-cli -regtest getblockchaininfo
bitcoin-cli -regtest generate 500      - mine 500 blocks
bitcoin-cli -regtest getbalance          - check rewards

5) Source code walk through is based on v0.14.99.0 (bitcoind -version)
main() calls AppInit()
AppInit() calls AppInitParameterInteraction()
then calls AppInitMain(threadGroup, scheduler) - main init function
AppInitParameterInteraction() - setup debug level, register RPC commands
            lock directory
            start lightweight task scheduler thread
            AppInitServers() -  start App servers, such as HTTP, RPC servers
            create zero message queue
            cache size calculations
            check genesis block
           CConnman::Start(scheduler, connOptions)
CConnman::Start(...)     - start threads to accept connections, process messages
call InitBinds(std::vector<CService>&, std::vector<CService>&) - bind to address and port number
InitBinds() call GetListenPort()
LoadBlockIndex(chainparams)        - load block index from db
GetBlockSubsidy(nHeight, consensusParams)       - get mining block rewards
CMerkleBlock constructor initialises CPartialMerkleTree
CPartialMerkleTree constructor calls TraverseAndBuild()
StartRPC()  - called by AppInitServers
StartHTTPServer()  - clled by AppInitServers
LogPrint          - debug log macro
LogPrintf         - debug log macro
LogAcceptCategory   - check the BCLog category and decide to print log or not
HTTPReq_JSONRPC(HTTPRequest* req, const std::string &)     - parse the rpc request that is sent from bitcoin-cli

Sunday, 2 July 2017

Thesis on Bitcoin

Here is my master thesis on Bitcoin. The topic is :


It is on google drive at the link below :
click at thesis link

Saturday, 15 April 2017

Embedded Product Design on Linux

An embedded product on Linux involves several stages. As an example, we use a Zigbee embedded product on Linux to discuss this.

The software components that make up an embedded product:

Dynamic UI: HTML, JQuery, Javascript, XML data, CGI process

Application and Libraries: Many applications, and third-party libraries

The figure above shows the interaction among the application processes, CGI app and Dynamic UI.

Linux Kernel and Drivers:
It maybe required to integrate new drivers into Linux kernel space, to enable the SoC on the main board.

It maybe required to modify boot loader if we upgrade to new generation of flash and DRAM IC.

Real-Life Embedded Systems problems and solutions:

  • Zigbee Carrier board integration:

Old carrier board is responding to Zigbee Ping command, and can find its IEEE address.
New carrier board is responding to Zigbee Ping command, but cannot find its IEEE address.

Reasons: Firmware is upgraded in New carrier board, so a upgraded SDK is required to work with the new firmware in New carrier board.

Debugging process: In Linux, open the serial port where the Zigbee module is connected, then set the firmware in carrier board to command mode, then send the Zigbee Ping command to the carrier board.

  • Board Graceful shutdown:

Power-off the board immediately without proper shutdown sequence will cause linux file system corruption.

Solution: Ask MSP430 to set an interrupt to MPC8315 when the power switch is pressed. An interrupt handler is added to linux kernel (as a linux driver module) to handle MSP430 interrupt. The driver module will create a kthread to listen to the interrupt. When interrupt arrives, a user space helper function is called (by using usermode helper API).

The userspace function handles the proper shutdown sequence, such as carrying out the "init 0" command.

  • Move DRAM content forward:

We want to copy DRAM content.


void loader(void)
  long * ptr = loki_plus_dram;
  int i;
  for (i=0; ptr[i]!=0x1FFFF; i++)
    *((long *)ptr[i]) = ptr[i+1];

  • TI Zigbee chipset cc2530 UART 1 setup:

Unable to communicate to TI cc2530's UART 1 port.

In IAR Workbench, Project->Options, choose C/C++ compiler, set:

This is a must for the source code to call hal_uart_isr() function in hal_uart.c

  • Wifi camera lose connectivity to Wifi AP:

Wifi camera lost connection to Wifi AP after 1 day, meaning Wifi AP cannot detect the Wifi camera after 1 day.

In /etc/hostapd.conf, set wpa_strict_rekey = 0
or set wpa_group_rekey = 0 and wpa_ptk_rekey = 0 if maintaining wpa_strict_rekey = 1

  • Glucometer timeout issue:

The Bluetooth program on linux is not able to read all data from Glucometer, read function hang, need to power off/on Glucometer .

Use a function called setsockopt to set socket to non-blocking with timeout.

setsockopt(sk, SOL_SOCKET, SO_RCVTIMEO, (struct timeval *)&tv, sizeof(struct timeval));
ret = read(sk, buf, sizeof(buf));
if (ret == -1)
  read again

  • Add debug statement control:

debug statements are lost if we do not have serial port to monitor them

Define a debug macro

//DEBUG functions
#define LOG_OUTPUT stderr

#define LOG(level, ...) \
                fprintf(LOG_OUTPUT, level); \
                fprintf(LOG_OUTPUT, __VA_ARGS__); \
                fflush(LOG_OUTPUT); \

#define DEBUG(...)   LOG("Myglucohealth   ", __VA_ARGS__)

Actual usage:
DEBUG("Can't find adapter %s\n", adapter);

Normally the debug statements will be sent to console, but we can redirect them to anywhere with a easy command:
<process_command>  &> /mnt/storage/bluetooth/process_command.log  &
---> redirect both stdout and stderr to a file

  • TCP socket: too many open files:

A TCP client connected to TCP server every 60 seconds and close the connection after sending a message. After 6 hours, socket call failed, saying "socket: too many open files"

The TCP server must close the socket that is returned by accept(). And the TCP client should close the socket first, then the TCP server close it. This is due to the TIME_WAIT settings which will create waiting burden on the TCP server if the TCP server closes the socket first, and if the TCP server is a busy one.

Use netstat to check for open socket connections
Use "ls /proc/<pid>/fd" to check for open file descriptor
Turn off SO_LINGER, call it after open the socket, eg: setsockopt(sd, SOL_SOCKET, SO_LINGER,...);

  • Simultaneous Wifi and Ethernet transmission, and data lost

When there are simultaneous Wifi and Ethernet transmission to the DUT, and DUT does not receive data via the Wifi after a few hours.

Check the interrupt mask and interrupt enable register bits. On the general interrupt controller (GIC), the Wifi and ethernet bits are adjacent to each other. When data is received at the hardware interface, internet status bit is set. When inside the ISR, interrupt enable bit is disabled, and interrupt mask bit is masked out. When exiting ISR, interrupt enable bit is enabled, interrupt mask bit is cleared. If ISR interrupt enable bits is not set, something is amiss.

The problem is due to a hardware error that ethernet interrupt bit will affect the Wifi interrupt bit. The software workaround is to enable the Wifi and Ethernet interrupt enable bits in the ISR when exiting ISR.

Wednesday, 21 December 2016

Bluetooth Protocol and IEEE11073

Bluetooth Protocol Stack:

Bluetooth Stack Partitioning:

host controller - Bluetooth hardware
Bluetooth host - the PC
HCI - Host Controller Interface

Pairing is the process of establish the right to make a connection. Once the units have been paired and provided the paired unit is "trusted" then connecting becomes automatic. Otherwise you must respond each time an attempt is made to connect that it is ok. Keeping the unit in a connected state makes it easy and convenient to accept calls but you do pay for the higher battery drainage.

Pairing is also and act of exchanging of authentication key.

Inquiry Procedure:
The inquiry procedure enables a device to discover which devices are in range, and determine the addresses and clocks for the devices. The inquiry procedure involve a unit (the source)sending out inquiry packets (inquiry state) and then receiving the inquiry reply  .The unit that receives the inquiry packets (the destination),  will hopefully be in the inquiry scan state to receive the inquiry packets. The destination will then enter the inquiry response state and send an inquiry reply to the source. After the inquiry procedure has completed, a connection can be established using the paging procedure.

Paging Procedure:
With the paging procedure, an actual connection can be established. The paging procedure typically follows the inquiry procedure. Only the Bluetooth device address is required to set up a connection. Knowledge about the clock (clock estimate) will accelerate the setup procedure. A unit that establishes a connection will carry out a page procedure and will automatically be the master of the connection. The procedure occurs as follows:

1:   A device (the source) pages another device (the destination ) : Page state

2:  The destination receives the page : Page Scan state

3:   The destination sends a reply to the source. : Slave Response state: Step 1

4:   The source sends an FHS packet to the destination : Master Response state: Step 1

5:   The destination sends it's second reply to the source. : Slave Response state : Step 2

6:   The destination & source then switch to the source channel parameters  :Master Response state: Step 2 &  Slave Response state: Step 3

Bluetooth Protocol:

On Linux, I use Bluez, an open source implementation of Bluetooth protocol. Bluez will run a daemon called bluetoothd, kernel modules called rfcomm, bluetooth, bnep, etc. Bluez is used for receiving and sending the Bluetooth messages.

MCAP - Multi Channel Adaptation Protocol. It is used by HDP. It facilitates the creation of communication link for exchanging commands, and data links to transfer actual Medical Device data. MCAP guarantees reliable transmission of data.

SDP - Service Discovery Protocol. It is used by all Bluetooth profiles to discover available services on remote devices, so that connections over L2CAP can be established.

L2CAP - Logical Link Control and Adaption Protocol. It supports protocol multiplexing, packet segmentation and re-assembly, quality of service, re-transmission, flow control. It is only used for ACL link.

Bluetooth Profile:
It is an interface specification, resides on top of the Bluetooth core specification.
It specifies:
  • user interface formats
  • specific parts of Bluetooth protocol stack used by the profile
Advanced Audio Distribution Profile - A2DP. It defines how high quality audio can be streamed from one device to another over a Bluetooth connection. For example, music can be streamed from a mobile phone to a wireless headset.

Health Device Profile - HDP. It is designed to facilitate transmission and reception of Medical Device data. It also performs SDP behavior, to connect to remote HDP devices.

IEEE 11073 stack. It performs building, transmission, reception, parsing of IEEE PDU packets for the associated IEEE 11073 agent/manager. It links to HDP.

Serial Port Profile - SPP. It is used to simulate RS-232

Generic Access Profile - GAP. It describes the required features of all Bluetooth profiles, including inquiry, connection, authentication procedures.

Device ID Profile - DIP. It provides device specific information through use of the SDP. If vendor specific info is required, this profile provides specific way to acquire this info. It enables identification of the manufacturer, product id, product version. It is useful in allowing a PC to identify a connecting device and download appropriate drivers.

Bluetooth Power Class:
class 1: long range (100m)
class 3: long range (10m)
class 3: long range (10cm)

Bluetooth CoD:
class 0xSSDDdd (three bytes)

The default CoD (Class of Device) is 0x000100 which simply stands for "Computer".

The Bluetooth device class is a high-level description of the bluetooth device, composed of three bytes: the "Major Service Class" (byte "SS" above), the "Major Device Class" (byte "DD" above) and the "Minor Device Class" (byte "dd" above). These classes describe the high-level capabilities of the device, such as "Networking Device", "Computer", etc. This information is often used by clients who are looking for a certain type of service around them.

Where it becomes tricky is that another type of mechanism for service discovery exists: "SDP", as in "Service Discovery Protocol".

In practice, most Bluetooth clients scan their surroundings in two successive steps: they first look for all bluetooth devices around them and find out their "class". You can do this on Linux with the hcitool scan command. Then, they use SDP in order to check if a device in a given class offers the type of service that they want.

Bluetooth Master/Slave:
It is not possible to detrmine bluetooth master or slave programatically.The master/slave roles which a device assumes is invisible to the user(application).The same device can act as a master of one piconet and a slave in an (adjacent) piconet.What needs to be decided in your application is whether the device will act as a client or a server or both and what will be the services(profiles) that it supports.

Bluetooth FH:

All Bluetooth devices participating in a given piconet are synchronized to the frequency-hopping channel for the piconet. The frequency hopping sequence is determined by the master's device address and the phase of the hopping sequence (the frequency to hop at a specific time) is determined by the master's internal clock. Therefore, all slaves in a piconet must know the master's device address and must synchronize their clocks with the master's clock.

During the connection handshake, the slave receives one packet (FHS, Frequency Hop Synchronization) that contains information about the master's BD_ADDR and the clock, so the slave's internal clock can be synchronized.

Bluetooth State Machine:

Bluetooth device can move from one state to a new state.

Bluetooth VDP

Vena Dongle Protocol (VDP) is an application protocol between Bluecore module and host microprocessor.

Host micro without IEEE11073 stack
Host micro with IEEE11073 stack

The messages that are carried across VDP link:
1. HDP messages
2. SPP messages for accessing legacy devices
3, Bluetooth device discovery, device pairing control packets.

Bluetooth Link Management Protocol (LMP)

In the usage of Nonin Oximeter, we have a problem where Oximeter fails to transmit measurement data to Bluetooth module after using for some time.

The root cause is because the oximeter sends out LMP "not_accepted" in response to the LMP "au_rand" message from Bluetooth module. Here is the message trace in terms of LMP opcode. The "not_accepted" message is the 2nd last message. 

au_rand contains a random number that is encrypted by link key. When au_rand is not accepted, meaning the oximeter cannot compute the same number using the link key, meaning the oximeter link key database is corrupted

BT Module                Oximeter 
--------------              ------------------ 
                              not accepted 

In normal operation, Bluetooth module challenges oximeter by sending the random 128-bit value AU_RAND and expects from oximeter the response: 
K is link key, E is Bluetooth authentication function. The oximeter receives the AU_RAND and sends back SRES as response. 

When oximeter sends not_accepted after it receives au-rand, meaning the oximeter link key database is corrupted. 

The behavior in this case is up to the Bluetooth module requesting the authentication. It can demand a new pairing by sending LMP in_rand, or it can refuse the connection and sends LMP detach. So for the Bluetooth module, the Bluez stack simply detach the link. 

So we need to modify Bluez stack for solving this "Oximeter takes measurement but no data is transmitted" problem. 

Bluetooth Packet Format

This is a standard Bluetooth packet:

ACCESS CODEAccess code is used for synchronization, DC offset compensation and identification.
HEADERHeader contains link control (LC) information.
PAYLOADPayload carries voice and data fields of upper layers.

Payload Format

HEADER [8-16] BODY [Indicated
in header]

HEADERHeader is one or 2 bytes longer. Only data fields have a payload header.
BODYPayload body includes the user host information. The length os the body is indicated in the length field of the payload header.
CRC CODEThe 16-bit cyclic redundancy check code is generated by the CRC-CCITT polynomial 210041 (octal).


Asynchronous Connectionless Link. One of the two types of data links defined for the Bluetooth Systems, it is an asynchronous (packet-switched) connection between two devices created on the LMP level. This type of link is used primarily to transmit ACL packet data. The ACL link is a point-to-multipoint link between the master and all the slaves participating on the piconet. In the slots not reserved for the SCO links, the master can establish an ACL link on a per-slot basis to any slave, including the slave already engaged in an SCO link. Only a single ACL link can exist. For most ACL packets, packet retransmission is applied.


Synchronous Connection-Oriented. The SCO link is a symmetric point-to-point link between a master and a single slave in the piconet. The master maintains the SCO link by using reserved slots at regular intervals (circuit switched type). The SCO link mainly carries voice information. The master can support up to three simultaneous SCO links while slaves can support two or three SCO links. SCO packets are never retransmitted. SCO packets are used for 64 kB/s speech transmission.


Data - Medium Rate. An ACL link data packet type for medium rate data. DM1 packets carry information data only, contining a 16-bit CRC code and up to 18 info bytes. They are encoded using 2/3 FEC and the packet can cover up to a single time slot. DM3 packets are the same except they can cover up to 3 time slots, and can carry up to 123 info bytes. DM5 packets are the same again except they can cover up to 5 time slots and can hold up to 226 info bytes. See also Bluetooth.


Data-High Rate. An ACL link data packet type for high rate data. DH1 packets are similar to DM1 packets, except the info in the payload is not FEC encoded. This means the DH1 packet can carry up to 28 info bytes and covers a single time slot. The DH3 is the same except it can cover up to 3 time slots and contain up to 185 info bytes. The DH5 packet is the same again except it can cover up to 5 time slots and contains up to 341 info bytes See also Bluetooth packet types.

Bluetooth D-Bus commands:


dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.freedesktop.DBus.Introspectable.Introspect

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez org.freedesktop.DBus.Introspectable.Introspect

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/690/hci0 org.bluez.Adapter.RemoveDevice objpath:/org/bluez/690/hci0/dev_00_1C_05_00_4A_AD

dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/1282/hci0 org.bluez.Adapter.ListDevices

dbus-send --system --type=method_call --print-reply --dest=org.bluez /  org.bluez.Manager.FindAdapter string:hci0

dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.bluez.Manager.ListAdapters

Antidote Healthd:

dbus-send --system --type=method_call --print-reply /com/signove/health org.freedesktop.DBus.Introspectable.Introspect

Bluetooth IEEE11073 

IEEE11073 is an Bluetooth application layer protocol, used by HDP devices. IEEE11073 specifies the formats and standards that HDP devices use to send messages. The standard itself is long winded and talking about it is time consuming. IEEE11073 is promoted by the Continua Alliance.

On Linux, I use Antidote, an open source implementation of IEEE11073. It is developed by Signove Technologies. It is written in C.

As an example, we can sniff an Bluetooth Nonin 9560BT Oximeter. Oximeter is set as Master. Bluetooth host is slave. If encrypted data that is captured is in error, we need to look at these LMP frames.


If these LMP frames are missing, we need to delete the oximeter device from Bluetooth database, then re-pair the device. It is because the link key is generated at local and remote devices during pairing stage. If devices are already paired, then link key generation won't be performed by the devices (link key is saved in devices already), and sniffer won't be able to capture the link key, and can't decrypt the bluetooth packets.

The IEEE11073 message exchange is shown below.

Link Management (LM):

The LM assists in the pairing procedure by communicating parameters and results between the local and remote Bluetooth devices. The calculations are done in the baseband. On the link manager level, the pairing procedure starts with the transmission of the PDU LMP_in_rand (containing the 128-bit random number
IN_RAND) from one of the units to the other. This PDU will trigger the generation of the initialization key for use in the protocol for creating the actual link key. If a unit key is to be used, the LMP unit key
command is sent in one direction only, with the secret unit key XORed with the initialization key as its parameter. The receiver can easily calculate the key from this. If a combination key is to be created, two contributions (one from each side) are needed. This is accomplished through the LMP_comb_key PDU. The argument of this PDU is also a random number, but generating this and deriving the link key from it is slightly more complicated. 

After these PDUs have been exchanged, both ends are able to compute the link key. As a final check of success of this procedure and the established link key, a mutual authentication event is performed. A 128-bit challenge is sent through the LMP_au_rand , and the 32-bit response uses the LMP sres PDU. If these authentications do not fail, both the local and the remote Host Controller notify their respective hosts.

Click on the image below to see the link key exchanges.

Tuesday, 15 November 2016

Zigbee Tutorial

IEEE802.15.4 defines the Physical and MAC layers, and ZigBee defines the network and application layers. Zigbee layers build on top of IEEE802.15.4.

Zigbee provides the features of low power consumption, low cost, small footprint, and mesh networking capabilities.

Two version: Zigbee 2006, Zigbee Pro (2007)
Zigbee Pro can scale up to thousands of devices.

Zigbee Architecture:

Application Profile is defined by Zigbee specification.
Eg. Home Automation (HA) and Smart Energy (SE).

Devices within an application profile communicate with each other by means of clusters, which may
be inputs to or outputs from the device.

Eg. In HA profile, cluster 0x0006 is the on/off cluster. Cluster 0x0702 is the zigbee power consumption report cluster. In HA profile, to use Zigbee on/off switch to control Zigbee power switch, we can bind the endpoint number of Zigbee on/off switch, with endpoint number Zigbee power switch, on the cluster 0x0006.

An endpoint defines a communication entity within a device. Zigbee Device Object (ZDO) is always endpoint zero. In all, 240 endpoints are available for use within ZigBee device, with endpoint zero dedicated to the ZigBee Device Object (ZDO), which provides control and management commands.

Joining a Zigbee network:

A ZigBee router or coordinator that wishes to allow other devices to join must issue a NLME-PERMIT-JOINING.request.  The joining device must issue a NLME-JOIN.request with the rejoin flag set to FALSE.

ZigBee Security:

It is based on the 128-bit AES algorithm.

Zigbee coordinator also functions as the trust center when security is enabled on the network.
  • Master key - used as an initial shared secret to generate Link key
  • Trust center master key - used to establish the Link key
  • Link Key - at the Application layer, two types: Trust center link key; application layer link key. Link key is used to encrypt the network key
  • Network Key - at the Network layer
Zigbee Concepts:

Three type of devices:
Zigbee end device, Zigbee router, Zigbee coordinator.

PAN ID - Personal Area Network Identifier. The Zigbee coordinator assigns a PAN ID to the network.

Binding - connecting endpoints
Device discovery - finding out other Zigbee devices, requires
  • IEEE address
  • NWK address
Service discovery - finding out services available on endpoints at the receiving device

IEEE802.15.4 Concepts:

IEE802.15.4 operating frequency is at 2.4GHz and 915MHz.
2.4GHz: channel 11-26
915MHz: channel 1-10

IEEE802.15.4 defines 4 types of MAC frame:
  • beacon frame
  • data frame
  • Ack frame
  • MAC command frame

IEEE802.15.4 data frame in action:

Zigbee Home Automation profile:

The Zigbee HA profile defines various device types that have various roles in the HA network. Device types such as Power Outlet(0x0009), On/Off Switch(0x0000), Combined Interface(0x0007), Pump(0x0303), Temperature sensor(0x0302), IAS devices; are frequently used.

Intruder Alarm Systems - IAS
  • IAS Zone(0x0402) - door sensor, smoke sensor
  • IAS Warning Device(0x0403) - siren
  • IAS ACE(0x0401) - remote control
  • IAS CIE(0x0400) - usually it is the Zigbee coordinator
To control zigbee devices. Firstly, zigbee devices must join the zigbee network. Then, Zigbee coordinator can send a data frame to the zigbee devices, to perform desired action. Zigbee devices can also report its status periodically to the zigbee coordinator.

On/Off switch can bind to power outlet, to control the power outlet directly. It is useful when the zigbee coordinator is down.

Zigbee Certification:

We can do testing in four modes:

Zigbee continuous TX
Zigbee continuous RX
Zigbee packet TX
Zigbee packet RX

For TX test, we can set TX channel, TX power, use range extender (an IC) to obtain higher TX power, send modulated or unmodulated signals.

To perform a Zigbee RX test, we need to setup a Zigbee end device and a Zigbee coordinator. The Zigbee end device can join the Zigbee coordinator, provided with the right key and PAN ID.

Wednesday, 14 October 2015

Device Tree in Linux

A device tree is the tree data structure that describes the physical devices in the system hardware. In Linux, the device tree source (DTS) files are located in arch/<xxx>/boot/dts

  •   dts for board level files
  •   dtsi for Soc level definitions

A device tree compiler (DTC) compiles the source into binary files. The DTC source code is in scripts/dtc.c.

The device tree blob (DTB) is produced by the DTC compiler. It is loaded by the bootloader and parsed by kernel at boot time.

Let's look at an real example where a platform driver is using two clocks.

In xxx_probe(struct platform_device *pdev)
    my_port->aaclk = clk_get(&pdev->dev, "aa");
    my_port->bbclk = clk_get(&pdev->dev, "bb");

The corresponding device tree has to reflect the two clocks.

uart0: serial@3000 {
compatible = "xxx,uart";
reg = <0x3000 0x100>;
interrupt-parent = <&gic>;
interrupts = <11 12>;
clocks = <&oscclk 0>,<&oscclk 1>;
clock-names = "aa","bb";

The two clock-names needs to have two matching clocks source.

Let's look at at an advanced example of device tree usage:

This example shows an oscillator clock being fed to pll1, pll2 and pll3. The hw3_clk can be derived from either pll2, pll3 or osc clock. 

To model the hardware using device tree. We declare the following entries.

osc: oscillator {
#clock-cells = <0>;
compatible = "fixed-clock";
clock-frequency = <20000000>;
clock-output-names = "osc20M";

pll2: pll2 {
 #clock-cells = <0>;
compatible = "abc123,pll2-clock";
clock-frequency = <23000000>;
clocks = <&osc>;
reg = <0x05 0x10>;

pll3: pll3 {
 #clock-cells = <0>;
compatible = "abc123,pll3-clock";
clock-frequency = <23000000>;
clocks = <&osc>;
reg = <0x15 0x10>;

hw3_clk: hw3_clk { 
    #clock-cells = <0>; 
    compatible = "abc123,hw3-clk"; 
    clocks = <&pll2>,<&pll3>; 
    clock-output-names = "hw3_clk"; 

In the source code, to register hw_clk3 as a mux, and show the parent relationship of both pll2 and pll3, we declare the below.

of_property_read_string(node, "clock-output-names", &clk_name); 
parent_name[0] = of_clk_get_parent_name(node, 0); 
parent_name[1] = of_clk_get_parent_name(node, 1); 

clk = clk_register_mux(NULL, clk_name, parent_name, 
      ARRAY_SIZE(parent_name), 0, 
      regs_base , offset_bit, one_bit, 0, NULL);

Sunday, 8 March 2015

More USB stuff

Useful USB debug knowledge:

You can test the USB mass storage gadget's behavior with a Linux host.

To send a Set-Config request with value 0, to de-configure the gadget

        echo 0 >/sys/bus/usb/devices/.../bConfigurationValue

To send a Set-Config request with value 1, to re-configure the gadget

        echo 1 >/sys/bus/usb/devices/.../bConfigurationValue

where the "..." part is replaced with the gadget's device path.

Set-Config has only two stages, Setup and Status stage; there is no Data stage.

Host                        Device
----- Setup Packet --->         |
----- Data0 Packet --->         |== Setup stage
<---- Ack Packet ------         |

----- In Packet ------>         |
<---- Data1 Packet ----         |== Status stage
----- Ack Packet ----->         |

The Data1 packet above contains no data, as below.

      PID   !PID  CRC

USB software layering:

Client SW:

It decides what transfers are to be made with the USB device function.

USB Driver (USBD):

The software interface between the USB System SW and the Client SW. USBD provides the device drivers that the operating systems use to access USB devices.

USB System SW:

It is usually supplied with the operating systems. It can be seemed to include USBD and HCD.

Host Controller Driver (HCD):

The software interface between Host Controller and USB System SW. This interface allows a USBD to support different Host Controllers. HCD is an abstraction of the Host Controller hardware.

Host Controller:

It manages USB protocol level interface. It ensures USB bus access rules defined by the protocol are obeyed, such as inter packet timings, timeouts.
USB Host and Device Partitioning
Control Transfer:

It is started with setup transaction from Host to Device. The Setup transaction is followed by zero or more Data transactions. A Status Transaction completes the control transfer.

Setup Transaction consists of:
setup packet
data0 packet - the request
Ack packet

Data transaction
In packet
data1 packet - the data that is returned by device
Ack packet

Status transaction
out packet
data1 packet - has no data
Ack packet

USB Device Request:

The USB host uses USB requests to get info from USB devices. These request are made using control transfer, from host to device default control pipe.The standard requests are:

  • Get Status
  • Clear Feature
  • Set Feature
  • Set Address
  • Get Descriptor
  • Set Descriptor
  • Get Configuration
  • Set Configuration
  • Get Interface
  • Set Interface
  • Synch Frame

An example of USB request sequence in USB Command Verifier test:

80 06 00 01 00 00 12 00  -  Get device descriptor

Data(12 01 00 02 00 00 00 40 25 05 A5 A4 33 03 01 02  00 01)  - Return device descriptor
80 06 00 02 00 00 09 00 – get config descriptor

Data(09 02 20 00 01 01 04 C0 01) – return config descriptor
00 09 01 00 00 00 00 00 – set configuration

0 byte data
80 06 00 02 00 00 09 00 – get config descriptor

09 02 20 00 01 01 04 C0 01 – return config desc
80 06 00 02 00 00 20 00 – get config desc

0 byte data
80 06 00 01 00 00 12 00 – get device desc

Data(12 01 00 02 00 00 00 40 25 05 A5 A4 33 03 01 02  00 01) – return device desc
80 06 00 06 00 00 0A 00 – get device qualifier

Data(0A 06 00 02 00 00 00 40 01 00) – return device qualifier
80 06 00 02 00 00 09 00 – get config desc

Data(09 02 20 00 01 01 04 C0 01) – return config desc
80 08 00 00 00 00 01 00 – get configuration

Data(01) – return

USB Descriptor:

USB devices report their attributes using descriptors. A descriptor is a defined data structure. Standard USB descriptors are:

  • Device descriptor
  • Device Qualifier descriptor
  • Configuration descriptor
  • Interface descriptor
  • Endpoint descriptor
  • String descriptor

Sunday, 4 January 2015

USB Mass Storage Device Enumeration, and usbmon

An usbmon text output dissection

Some quick commands to before running usbmon:

# mount -t debugfs none_debugs /sys/kernel/debug
# modprobe usbmon
# cat /proc/bus/usb/devices
/*to find out the usb devices number on the usb bus*/

# cat /sys/kernel/debug/usb/usbmon/2u > /tmp/mon.out
As an example, a control packet:

f4ae6f40 1217192721 S Co:2:018:0 s 21 ff 0000 0000 0000 0

f4ae6f40 - URB Tag

1217192721 - Timestamp in microseconds

S - Event Type, S is submission

Co:2:018:0 - "Address". It consists of four fields, separated by colons: URB type and direction, Bus number, Device address, and Endpoint number. 
Co is Control output.

s - URB Status. This is a letter, so 's' is Setup Tag of control packet. It can also be several numbers separated by colons.

21 ff 0000 0000 0000 - Setup packet, if present, consists of 5 words: one of each for bmRequestType, bRequest, wValue, wIndex, wLength

0 - Data Length. For submissions, this is the requested length. For callbacks,this is the actual length.

Now we will look at mass storage device enumeration within the USB gadget framework. The USB gadget framework is available in Linux and U-boot.

USB halt and wedge feature:

Two types of events may cause a USB halt condition.

1) The device did not receive a handshake packet from USB host after communication.
2) The device receive set halt request from the host

Set halt and set wedge are basically the same, with the difference that, the device will ignore clear halt request from the host if device is in set wedge condition.

USB mass storage gadget enumeration steps:

The USB code can be divided into two parts: Usb Mass storage gadget code and Usb device controller code.

Firstly, the host PC sends the setup data to get descriptors from Usb device. It is performed using control transfer.

So the Usb chip receives it and interrupt is triggered. In Usb device controller code,
myudc_isr(void) [the interrupt handler]

reads the IVECT register and gets 0x00 value. It passes control to gadget code
fsg_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)

The setup data is processed, the response is prepared. If there is data to send out to EP0, the device controller function
my_ep_queue(struct usb_ep *ep, struct usb_request *req, gfp_t gfp_flags)

is called to send out the data.

After device, configuration, interface, endpoint, string descriptors are received by the host PC, the host PC detects the Usb device as a mass storage device (that is for a Usb mass storage device).

Device Descriptors:
As an example, the device descriptor is dissected here.

First Byte : bLength (the size of the descriptors in bytes) 0x12
Second Byte: bDescriptor (device descriptor)                 0x1
3rd-4th Byte: bcdUSB (USB spec number)                   0x0002
5th Byte: bDeviceClass (class code)                              0x00
6th Byte: bDeviceSubClass (subclass code)                   0x00
7th Byte: bDeviceProtocol (protocol code)                   0x00
8th Byte: bMaxPacketSize (max. packet size)                0x40
9-10 Byte: idVendor (vendor ID)                                  0x2505
11-12 Byte: idProduct (product ID)                               0xA5A4
13-14 Byte: bcdDevice (device release number)             0x0103
15th byte: iManufacturer (index of manufacturer string desc)      0x01
16th byte: iProduct (index of product string desc)                      0x02
17th byte: iSerialNumber (index of serial num. string desc)         0x03
18th byte: bNumConfigurations (number of possible config.)      0x01

Next step, the host PC will send SCSI commands to Usb device. The SCSI command is wrapped in CBW and sent using bulk transfer.

 So the Usb chip receives it and interrupt is triggered. In Usb device controller code,
myudc_isr(void) [the interrupt handler]

reads the IVECT register and gets 0x28 value for EP1 OUT IRQ. The data is read from FIFO, and sent to gadget function
bulk_out_complete(struct usb_ep *ep, struct usb_request *req)

in gadget code. This function will wake up thread in get_next_command((struct fsg_dev *fsg) and the thread will process the SCSI command. The get_next_command() is waiting for a buffer, which must be the same as the buffer returned by interrupt handler.

Again, if there is data to be replied to host PC, the device controller function
my_ep_queue(struct usb_ep *ep,  struct usb_request *req, gfp_t gfp_flags)
is called to send data to EP1 IN using bulk transfer.

After that, for every SCSI command, the gadget code will call gadget function
send_status(struct fsg_dev *fsg)

to prepare the CSW, and CSW is sent to the host PC using device controller function
my_ep_queue(struct usb_ep *ep,  struct usb_request *req, gfp_t gfp_flags)

 via EP1 IN.

The Transfer Protocol:

The Bulk Only Transport (BOT) is defined by Usb mass storage specification. It defines how Usb host can use bulk transfer to send commands and receives response from Usb device.

In the first transfer, the host sends a command in a structure called a Command Block Wrapper (CBW). The CBW is 31 byte in length. Many CBWs are followed by a transfer that contains data sent to the host or device.
For example, the CBW structure of a SCSI Inquiry command:

dCBWSignature  0x55 0x53 0x42 0x43
dCBWDataTransferLength   0x24
bCBWCBLength             0x06 (The valid length of CBWCB in bytes)
Operation Code               0x12
enableVitalProductData    false
commandSupportData       0x0
page or opcode                  0x0
allocation length                0x24
control                              0x0

In the final transfer, the device returns status in a structure called a Command Status Wrapper (CSW).

CSW structure:

dCSWSignature    0x55 0x53 0x42 0x43
dCSWDataResidue    0x0
bCSWStatus               0x0

The SCSI Commands:

The SCSI commands used by the Usb mass storage device are from SCSI Primary Commands (SPC-2) and SCSI Block Commands (SBC-2).

Some of the commonly used commands, together with the command codes, are listed below.

SCSI Inquiry                0x12
SCSI Request Sense    0x3
SCSI Read Capacity    0x25
SCSI Read 10             0x28
SCSI Mode Sense 6    0x1A
SCSI Mode Select 6    0x15
SCSI Test Unit Ready  0x00

SD Card Partition Table:

The SD card will try to mimic a hard disk.

The MBR is located at offset 0x0, size is 512 bytes.

                      boot code (first 446 bytes)
                      partition #1  0x1BE (each partition is 16 bytes)
   MBR           partition #2
                       partition #3
                       partition #4
                       0xaa55   0x1FE  (signature)

Partition table info: (from offset 0x1BE)

0x00  0x80 if bootable, else 0x0
0x01  start of partition in CHS
0x04  type of partition
0x05  end of partition
0x08  relative offset to the partition in LBA (Partition LBA begin)
0x0C  size of the partition

For example, if the relative offset in LBA is 0x2000, it means the start of the boot sector is 0x2000.

At 0x2000, it is the FAT boot record (if formatted in FAT file system).

For example, FAT32 boot record:

0x00  3 bytes    0xeb0090       jump instruction
0x03  8 bytes    MSDOS5.0    OEM name
0x0B  2 bytes    0x0200           bytes per sector
0x0D  1 byte      0x08               sectors per cluster
0x0E   2 bytes     0x20               number of reserved sector (usually 0x20)
0x10   1 byte      0x2                 number of FATs
0x11   2 bytes                           N/A
0x13   2 bytes                            N/A
0x15   1 byte    0xF8                 media descriptors(F8 for hard disk)
0x16   2 bytes                            N/A
0x18   2 bytes                           sectors per track
0x1A  2 bytes                             number of heads
0x24   4 bytes                         sectors per FAT
0x2C   4 bytes                          root directory first cluster
0x5A  420 bytes                      bootstrap code
0x1fe  2 bytes    0x55aa           end of boot sector mark

Important info that are needed for accessing the FAT32 filesystem:

(unsigned long)fat_begin_lba = Partition_LBA_Begin + Number_of_Reserved_Sectors;
(unsigned long)cluster_begin_lba = Partition_LBA_Begin + Number_of_Reserved_Sectors + (Number_of_FATs * Sectors_Per_FAT);
(unsigned char)sectors_per_cluster = sectors_per_cluster;
(unsigned long)root_dir_first_cluster = root_directory_first_cluster;