Thursday, 5 October 2017

Linux Information

1. Linux Auto Login

Sometimes we may want to skip the linux login prompt, for example, a remotely-powered linux server in the basement of a building. Here is how we do it on RedHat linux 7.3.

  • modify /etc/inittabComment out:
    1:2345:respawn:/sbin/mingetty tty1
    Add:
    1:2345:/usr/bin/openvt -f -c 1 -w -- /bin/login -f root
    -c 1 for console 1
    -- for separator

  • modify ~/.bash_profileAdd:
    startx
    to start the X windows
  • modify ~/.xinitrcAdd:
    gnome-session
    or
    startkde

2. Linux Programming

Linux Command Line Tools
Some useful tools are listed here.
% nm <object-files>
list symbols from object files

% readelf -d <elf-files>
display dynamic info about ELF files

% ldd <executable-files>
list shared library dependencies

% ar crv <lib-files> <object-files>
create static library

***GNU Make***
Make commands is used extensively in Linux software development.

A) Make command line syntax:
The `-s' or `--silent' flag prevents all echoing, as if all commands are started with `@'.
eg. make -s zImage
The `-f' flag specifies a name of the makefile.
eg. make -f myMakefile
We can run make -n <target>, to tell make to print out what it should have done, without actually doing it.

B) The general Makefile syntax is:
target : dependency
 rules (or commands)
The command lines start with a tab character. Blank lines and lines of just comments may appear among the command lines; they are ignored. (But beware, an apparently "blank" line that begins with a tab is not blank! It is an empty command.)
To make a particular target, we need to pass its name to make as a parameter. Without a parameter, make will try to make the first target listed in the makefile.
Make invokes a shell for executing rules, and uses a new shell for each rule. To let all script commands appear on one logical line, we must add backslash at the end of each line. The @sign will tell make not to print out each command on standard output.

C) Makefile contents:
Makefiles contain five kinds of things: explicit rules, implicit rules, variable definitions, directives, and comments.

directive:
     include filenames...

When make processes an include directive, it suspends reading of the containing makefile and reads from each listed file in turn. When that is finished, make resumes reading the makefile in which the directive appears.
If you want make to simply ignore a makefile which does not exist and cannot be remade, with no error message, use the -include directive instead of include, like this:
     -include filenames...

D) Other Makefile stuff:
Targets that do not refer to files but are just actions are called phony targets.
objects = main.o kbd.o command.o display.o \
               insert.o search.o files.o utils.o

.PHONY : clean
     clean :
             -rm $(objects)

clean:
 -rm -f *.o
The rules of making the target "clean" comes without dependency. This mean that the target is always considered out of date, and its rule is always executed. Also notice that the command starts with "-". This tells make to ignore the result of the command, and so make clean will always succeed. The `-' is discarded before the command is passed to the shell for execution.
For example,

     clean:
             -rm -f *.o
A few special macros are defined in make. Here is the summary.
foo.o : foo.c
 gcc -c $(CFLAGS) %^ -o $@

$^ is the dependency (foo.c)
$@ is the target (foo.o)

foo.o : foo.c defs.h hack.h
 gcc -c $(CFLAGS) $< -o $@

$< is the first dependency (foo.c)

.c.a:
 gcc -c $<
 ar rv $@ $*.o

$* is the name of the target without the suffix
Make has a special syntax for dealing with library files. The syntax is lib(file.o). It means that the object file file.o is stored in the library file lib.a.
$(LIBRARY): $(LIBRARY)(db_api.o)
db_api.o: db_api.c db_api.h
For a project with multiple subdirectories, a main makefile in the main directory will invoke the sub-makefiles. The syntax is like this:
(cd subdirectory;$(MAKE))
or, equivalently, this:
$(MAKE) -C subdir
The brackets ensure that it is all processed by a single shell. Since a new shell is invoked for this, the program running the make doesn't execute the cd command. Only the shell invoked to carry out the rule is in a different directory.

E) Variable substitution
To substitute a variable's value, write a dollar sign followed by the name of the variable in parentheses or braces: either `$(foo)' or `${foo}' is a valid reference to the variable foo. This special significance of `$' is why you must write `$$' to have the effect of a single dollar sign in a file name or command.
The first flavor of variable is a Recursively expanded variable. Variables of this sort are defined by lines using `=' or by the define directive.
foo = $(bar)
bar = $(ugh)
ugh = Huh?

all:@echo $(foo)
It will echo `Huh?'. `$(foo)' expands to `$(bar)' which expands to `$(ugh)' which finally expands to `Huh?'.
To avoid all the problems and inconveniences of recursively expanded variables, there is another flavor: Simply expanded variables. Simply expanded variables are defined by lines using `:='
foo := $(bar)
bar := $(ugh)
ugh := Huh?

all:@echo $(foo)
This will echo nothing, as $(foo) is undefined here.
There is another way, called a Conditional variable assignment operator, because it only has an effect if the variable is not yet defined. This statement:
  FOO ?= bar
is exactly equivalent to this:
ifeq ($(origin FOO), undefined)
  FOO = bar
endif

F) String Substitution (in Makefile):
$(patsubst pattern,replacement,text)
Finds whitespace-separated words in text that match pattern and replaces them with replacement. Here pattern may contain a `%' which acts as a wildcard, matching any number of any characters within a word. If replacement also contains a `%', the `%' is replaced by the text that matched the `%' in pattern. `%' characters in patsubst function invocations can be quoted with preceding backslashes (`\'). Whitespace between words is folded into single space characters; leading and trailing whitespace is discarded.
For example,
$(patsubst %.c,%.o,x.c.c bar.c)
produces the value `x.c.o bar.o'.
Substitution references are a simpler way to get the effect of the patsubst function:
$(var:pattern=replacement)
is equivalent to
$(patsubst pattern,replacement,$(var))
The second shorthand simplifies one of the most common uses of patsubst: replacing the suffix at the end of file names.
$(var:suffix=replacement)
is equivalent to
$(patsubst %suffix,%replacement,$(var))
For example, you might have a list of object files:
objects = foo.o bar.o baz.o
To get the list of corresponding source files, you could simply write:
$(objects:.o=.c)
instead of using the general form:
$(patsubst %.o,%.c,$(objects))
Another example in real Makefile:
SUB = server client

%.build:
 (cd $(patsubst %.build, %, $@) && $(MAKE))

%.clean:
 (cd $(patsubst %.clean, %, $@) && $(MAKE) clean)

all:  $(patsubst %, %.build, $(SUB))
clean:  $(patsubst %, %.clean, $(SUB))

G) Conditional statement (in Makefile):
Three types: if-else-fi, ifeq-else-endif, ifdef-else-endif.
@if [ -z $(DESTDIR) ]; then \
    /sbin/depmod -ae ; \
elif [ -f $(SYSTEMMAP) ]; then \
    /sbin/depmod -ae -b $(DESTDIR) -F $(SYSTEMMAP) ; \
else \
    echo "Don't forget the target system."; \
fi
ifeq ($(strip $(foo)),)
  text-if-empty
else
  text-if-not-empty
endif
Don't specify $(foo) for variable referencing if used in ifdef statement
foo = $(bar)
ifdef foo
  frobozz = yes
else
  frobozz = no
endif


Using GNU GDB and gdbserver
GDB can be used to debug programs written in C and C++. GDB distribution contains gdbserver in one of its subdirectory. We run gdbserver on target platform in order to save space, and use GDB on host to connect to the gdbserver for debugging. For more information, click here.

Compiling and generating library images
The sequences of generating a *.a (static library)
a) gcc -c xxx.c
b) ar crv libxxx.a xxx.o

The sequences of generating a *.so (dynamic library)

gcc -c -fPIC xxx.c
gcc -shared -o libxxx.so xxx.o

Files, Pipes, Sockets
Everything is represented as a file under Linux. Even hardware devices are represented by files in Linux. The low-level system calls can be used to access the files, and similarly the hardware devices, pipes, sockets.
int open(const char *path, int flags);
- open system call returns a new file deescccriptor
size_t read(int fd, void * buf, size_t nbytes);
- read system call reads up to nbytes frrommm file associated with the file descriptor fd to buf
size_t write(int fd, const void * buf, size_t nbytes);
- write system call writes up to nbytes tooo the file referenced by the file descriptor fd from the buf
int close(int fd);
- close system call closes a file descriiptttor, so that it may be reused
int ioctl(int fd, int cmd, ...);
- ioctl system call provides an interfacce for controlling the behavior of devices
off_t lseek(int fd, long int offset, int whence);
- lseek system call sets the read/write pooointer of a file descriptor
The standard library and its header file stdio.h provide an interface to low-level system calls. The library provides functions to take care of the buffering of the devices. In standard library, the equivalent to a file descriptor is called a stream, and is implemented as FILE * .
FILE * fopen(const char *filename, const char modes);
- fopen library function returns a FILE * pointer
size_t fread(void * ptr, size_t size, size_t nitems, FILE * stream);
- fread library function reads to ptr frrommm the stream, for a record of size and a count of nitems
size_t fwrite(const void * ptr, size_t size, size_t nitems, FILE * stream);
- fwrite library function writes from thhe ptr to the stream, for a record of size and a count of nitems
int fclose(FILE * stream);
- fclose library function closes the speeciiified stream
int fseek(FILE * stream, long int offset, int whence);
- fseek library function sets the positiionnn in the stream for the next read or write on that stream.
Pipe is a primitive form of inter process communications. It allows the data flow from one process to go to another process. For shell commands, it is entered as :
cmd1 | cmd2
For high-level pipe function, they will operate on file stream.
FILE * popen(const char * command, const char * open_mode);
It allows a program to invoke another program as a new process, and either pass data or receive data from it.
int pclose(FILE * stream);
It closes the file stream associated with the pipe.
At the same time, a low-lovel pipe function provides a way of passing data between two programs, without the overhead of invoking a shell to interpret the requested command.
int pipe(int fd[2]);
It is passed an array of two integer file descriptors. Any data written to fd[1] can be read back from fd[0]. Low-level system calls, read and write, are used to access the data.
A named pipe exists in the file system as a special type of file, but behaves like the unnamed pipes we discussed earlier. It is called FIFO too.
int mkfifo(const char * filename, mode_t mode);
It creates a named pipe, using absolute pathname.
We can remove the FIFO by using the rm command, or from within a program by using the unlink function.
A FIFO exists as a named file, not as an open file descriptor. It must be opened before it can be read from or written to. The open and close system calls can be used for the purpose. The read and write system calls can be used to access the FIFO after it is opened.

Using GCC __func__ macro
GCC provides three magic variables which hold the name of the current function, as a string. The first of these is __func__, which is part of the C99 standard:
The identifier __func__ is implicitly declared by the translator as if, immediately following the opening brace of each function definition, the declaration
           static const char __func__[] = "function-name";
appeared, where function-name is the name of the lexically-enclosing function. This name is the unadorned name of the function.
__FUNCTION__ is another name for __func__. Older versions of GCC recognize only this name. However, it is not standardized. For maximum portability, we recommend you use __func__, but provide a fallback definition with the preprocessor:
     #if __STDC_VERSION__ < 199901L
     # if __GNUC__ >= 2
     #  define __func__ __FUNCTION__
     # else
     #  define __func__ ""
     # endif
     #endif
In C, __PRETTY_FUNCTION__ is yet another name for __func__. However, in C++, __PRETTY_FUNCTION__ contains the type signature of the function as well as its bare name, such as "void a::sub(int)".
Click here to see the examples: exam.c and exam.h file.

Debugging
There are 8 severity levels in linux kernel, defined in <linux/kernel.h>. DEFAULT_MESSAGE_LOGLEVEL is specified in kernel/printk.c, and applied to printk with no specified priority. If priority is less than console_loglevel, the message is displayed. If both klogd and syslogd are running on the system, kernel message are appended to /var/log/messages, independent of console_loglevel.
In <linux/kernel.h>, #define console_loglevel as DEFAULT_CONSOLE_LOGLEVEL. As printk writes message to a circular buffer, it then wakes up any process that is waiting for the message. If klogd is running, it retrieves kernel message and dispatch them to syslogd, which in turn check the settings in /etc/syslog.conf.

To use syslogd:



  • /etc/init.d/syslog -- script file
  • /etc/syslog.conf -- config file
  • create /var/log directory.

    An example syslog.conf config file:
    # cat syslog.conf
    *.*                      /var/log/messages
    

    AWK 
    It is an interpreted programming language for performing complex text processing tasks. It is also a simple text processing utility. It stands for the names of its authors: Aho, Weinberger & Kernighan.
    simple syntax:
    awk <search pattern> {<program actions>} <data filename>
    eg. awk '/gold/ {print $5,$6,$7,$8}' coins.txt
    full syntax:
    awk 'BEGIN   {<initialization>}
     <search pattern 1> {<program actions>}
     <search pattern 2> {<program actions>}
     ...
     END   {<final actions>}'
    
    eg. awk 'END {print NR, "conis"}' coins.txt
    
    NR - Awk's pre-defined varaibles, stands for Number of Records
    Awk regards each line of input data as composed of multiple fields, which are essentially words separated by blank spaces. A blank space is the default "field separator". To tell Awk to use another field separator, use -F.
    eg. awk -F\"  -- use " as the separator
    
    eg. awk -F\" '/REL/ {print $$2}' include/linux/version.h
    

    SED
    Stream editor
    sed -e s/<reg expr>/<replace text>/{flags}
    
    eg. sed -e '1,2 s/line/LINE/' test.txt
    eg. sed -e 's/cat/dog/g' test.txt
    
    regular expr
    ^ - match the beginning of the line
    $ - match the end of the line
    . - match any single character
    * - match arbitrary many occurrences of charater
    ? - match 0 or 1 instance of character
    

    Spinlock and Semaphore
  • Spinlocks are very small and fast, and can be used anywhere. If your task can't get the spinlock, your task keeps trying (spinning) until your task can.
  • Semaphore can have more than one holder at any time (the number decided at initialization time), although it is most commonly used as a single-holder lock (known as mutex). If your task can't get a semaphore, your task will put itself on the queue, and be woken up when the semaphore is released. This means the CPU will do something else while your task is waiting.
  • Semaphore is used for synchronization between user contexts. User Context means the kernel is executing on behalf of a particular process (ie. a system call or trap) or kernel thread. This is not to be confused with userspace. It can be interrupted by software or hardware interrupts.
  • Spinlock is used for synchronization between user context and interrupt context. spin_lock_bh() disables softirqs on that CPU, then grabs the lock. spin_lock_irq() is defined to disable interrupts on that cpu, then grab the lock.

    Exporting Symbols
  • In source file:
     #define EXPORT_SYMTAB
    
    and then after you write your function:
     EXPORT_SYMBOL(your_function_name);
    
  • In Makefile:

  •  export-objs := filename.o
    

    Networking commands
    Commands for device statistics:
    cat /proc/net/dev
      show device statistics
    cat /proc/net/snmp
      show snmp statistics
    
    
    Commands for routing table:
    route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.1
      add gateway 192.168.1.1 to routing table for network 192.168.2.0
    route del -net 192.168.2.0/24
      delete routing table entry
    route -n
      show routing table status
    

    Commands for ethernet:
    ifconfig eth0 txqueuelen 10000
      increase the transmit queue of the network interface
    echo 1 > /proc/sys/net/ipv4/ip_forward
      enable ip forwarding
    /sbin/ethtool -K eth0 tso off
      disable TCP segment offloading
    /bin/uname -r
      show kernel version
    
    netstat -p --tcp
      show network connection for tcp protocol

    Reset procedures for MIPS and ARM


  • MIPS IDT438
    sys_reboot (kernel/sys.c)
    --> machine_restart (arch/mips/kernel/reeseet.c)
        --> _machine_restart (assigned in arch/mips/rc32438/79EB438/setup.c)
           --> idt_reset (arch/mips/rc32438/79EB438/reset.c)
    
  • ARM IXP425

  • sys_reboot (kernel/sys.c)
    --> machine_restart (arch/arm/kernel/prooceess.c)
        --> arch_reset (include/asm-arm/arch-ixp425/system.h)
    

    3. Shell Script Programming

    A. Passing parameters to shell script
    As a simple example, i have a shell script which requires to take in one parameter. I name it "cp_script".
    #!/bin/sh
    echo "Parameter " $1
    
    Then i call the shell script on the command line and pass it a parameter:
    $./cp_script madwifi
    Parameter madwifi
    
    That's it, using $1 in shell script to take in the command line parameter.

    B. If-else-fi in shell script:
    [ expr ] is used to see if an expression is true.
    if [ -f $dir/$file ] || [ -f $dir/$newfile ]; then
        echo "Either this filename [$file] exists"
        echo "Or this filename [$newfile] exists"
    elif [ -d $dir ]; then
        echo "This dirname [$dir] exists"
    else
        echo "Neither [$dir] or [$file or $newfile] exist"
    fi
    

    C. Variable declaration:
    We use something as:
    #!/bin/sh
    
    TC="/sbin/tc"
    
    $TC qdisc ls
    $TC qdisc del dev eth0 root
    $TC qdisc add dev eth0 root pfifo_fast
    
    
    
    
    D. The getopt command
    The getopt command is used to parse the command line parameters. It is made up of two parts: options and non-option parameters.
    -o specifies the short option, -n is the name of the program, -- is the start of non-option parameters
    #! /bin/bash
    
    echo "param" $@
    TEMP=`getopt -o nr -n oudanhodou -- "$@"`
    echo "temp" $TEMP
    unset TEMP
    
    
    Execution:
    $ ./oudanhodou -r
    param -r
    temp -r --
    
    $ ./oudanhodou -r fresh
    param -r fresh
    temp -r -- 'fresh'
    

    4. Linux Setup

    A. Linux network setup on Redhat 9
    /etc/hosts
    127.0.0.1  localhost
    
    /etc/host.conf
    order host,bind
    
    /etc/resolv.conf
    nameserver  192.168.4.254
    
    It is the IP of DNS server.
    /etc/services
    
    It contains port allocation information.

    B. Change ip address
    /etc/sysconfig/network-scripts/ifcfg-eth0
    IPADDR=192.168.4.239
    DHCP_HOSTNAME=192.168.4.251
    
    /sbin/ifup eth0
    
    C. Use dhcp client
    execute redhat-config-network
    
    then execute /sbin/dhclient
    
    settings in /etc/dhclient-eth0.conf
    
    For Dhcp server and client settings on embedded system, click on DHCP info.

    D. Update linux kernel
    1. create floppy boot disk
    ls /lib/modules
     2.4.2-2
    
    mkbootdisk --device /dev/fd0 2.4.2-2
    
    2. clean old kernel config
    make mrproper
    
    3. change whatever settings you want
    make menuconfig
    
    4. update the change
    make dep
    
    5. make it
    make bzImage
    
    6. make the modules
    make modules
    
    7. install the newly made modules
    make modules_install
    
    8. update the boot loader config file with new kernel information
    /etc/lilo.conf
    
    9. read from new boot loader config file and store into boot sector
    /sbin/lilo -v
    
    
    E. lilo information
    "/sbin/lilo -v" read the /etc/lilo.conf file, to determine what to write to MBR.
    lilo.conf example
    boot=/dev/hda #install lilo in the first harddisk
    map=/boot/map
    install=/boot/boot.b #specify new boot sector file
    prompt
    timeout=50 #wait for 5 sec
    lba32    #describe harddisk geometry
    default=linux
    image=/boot/vmlinus-2.4.20-8 #specify linux kernel
     label=linux
     root=/dev/hda1  #specify root partition
    image=/boot/vmlinus
     label=failsafe
     root=/dev/hda1
     initrd=/boot/initrd.img
    other=/dev/hda2
     label=windows
     table=/dev/hda
    
    
    F. Make ramdisk
    mkdir ramdisk
    cd ramdisk
    dd if=/dev/zero of=initrd bs=1k count=8192
     bs is the block size
    /sbin/mke2fs -vFm0 initrd 8192
    mkdir mnt
    sudo mount -o loop initrd mnt
    cd mnt
    cp -a .....
    cd ..
    sudo umount initrd
    
    To do "mount -o loop" on target platform:
    1. kernel got to have loopback device support enabled
    2. On file system, create /dev/loop file
    3. kernel got to enable ext2 support
    
    target bootloader configured with:
    bootparm1=/root=/dev/ram init=/linuxrc rw
    
    
    G. Starting X windows
    startx > log 2>&1
    gcc -Wl,-v 2>&1 | grep "GNU ld"
    

    H. Sudo List
    In /etc/sudoers file, we can add in a list of users to specify their permissions.
    User_Alias   SW_STAFF = julian, hunk, yeosv
    # User privilege specification
    root   ALL=(ALL) ALL
    SW_STAFF  ALL=NOPASSWD: ALL
    newcomer ALL=(ALL) NOPASSWD: ALL
    
    
    I. Kernel Booting Sequences
    This is the kernel booting sequences for ARM architecture processor. This information may apply to other architecture with minor modification.
    (arch/arm/boot/compressed/head.S)
     -> setup the stack
     -> call decompressed_kernel (arch/arm/boot/compressed/misc.c)
     -> jump to decompressed code
    
    (arch/arm/kernel/head.S)
    _stext   -> start_kernel (init/main.c)     [ task 0, idle task ]
       -> setup_arch (arch/arm/kernel/setup.c)
       -> trap_init
       -> init_IRQ
       -> sched_init
       -> softirq_init
       -> time_init
       -> console_init
       -> init_modules
       -> kmem_cache_init
       -> calibrate_delay
       -> mem_init
       -> kmem_cache_sizes_init
       -> fork_init
       -> proc_caches_init
       -> vfs_caches_init
       -> ...
       -> rest_init             [ launch init kernel thread ]
        -> init (init/main.c)     [ the "init" kernel thread ]
         -> do_basic_setup
         -> prepare_namespace
         -> launch /sbin/init
    
    include/asm-arm/arch-ixp425/memory.h
     contains PAGE_OFFSET and PHYS_OFFSET
    
    arch/arm/boot/Makefile
     contains ZTEXTADDR and ZRELADDR

    Saturday, 26 August 2017

    Bitcoin core tutorial and source code walk through (Part 5)

    1) In part 5, we will look at Bitcoin raw transaction. Raw transaction gives you full manual control of the Bitcoin transaction, with the possibility of making errors and losing bitcoins permanently.

    The bitcoin-cli command can be used to create raw transaction to send to any recipient address. There is decode raw transaction command to display the raw transaction that we have created. After that, we can use send raw transaction command to actually spend the bitcoin.

    For testing purpose, the regtest network is used.


    Firstly, we use listunspent command to find out the UTXO (the txid) in the wallet
    ./src/bitcoin-cli -regtest listunspent 

    Then, we use getaccountaddress to find out the default address (the address) 

    ./src/bitcoin-cli -regtest getaccountaddress ""

    Then, we use createrawtransaction command to construct the raw transaction
    ./src/bitcoin-cli -regtest createrawtransaction "[{\"txid\":\"2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34\",\"vout\":0}]" "{\"mjQgnYbydSCupJzkQS9HDBoP1CGcZZQMjt\":0.01}

    The txid is from the listunspent command and address is from the getaccountaddress command. The vout of 0 is the vector output associated with the txid. The 0.01 is the amount to be paid to the address.

    Then, we sign the raw transaction
    ./src/bitcoin-cli -regtest signrawtransaction 020000000134fbcf80f43d49bc103a2a52332b94476684f742c16a87880f0160de0694a42c0000000000ffffffff0140420f00000000001976a9142ab10fb15ad20af203ee529a973cdbb1a0c2e68688ac00000000
    {
      "hex": "020000000134fbcf80f43d49bc103a2a52332b94476684f742c16a87880f0160de0694a42c0000000049483045022100b5928c21d2ea2436e669948e6eb4036a0d1de9b70210d3b103d475c9cf5b7b0b02200ac3a7fd690b16607ba833bef0e15dd84a21f0b609692727af7838dd7ac21f9d01ffffffff0140420f00000000001976a9142ab10fb15ad20af203ee529a973cdbb1a0c2e68688ac00000000",
      "complete": true

    }

    Finally, we spend the bitcoins.
    ./src/bitcoin-cli -regtest sendrawtransaction 020000000134fbcf80f43d49bc103a2a52332b94476684f742c16a87880f0160de0694a42c0000000049483045022100b5928c21d2ea2436e669948e6eb4036a0d1de9b70210d3b103d475c9cf5b7b0b02200ac3a7fd690b16607ba833bef0e15dd84a21f0b609692727af7838dd7ac21f9d01ffffffff0140420f00000000001976a9142ab10fb15ad20af203ee529a973cdbb1a0c2e68688ac00000000
    error code: -26
    error message:
    256: absurdly-high-fee

    See here, if the ( input - output ) = transaction fee is too big, bitcoin core will reject to send the transaction.

    2) In this analysis, the source code that handles raw transaction RPC will be walked through.
    rpc/rawtransaction.cpp
    UniValue createrawtransaction(const JSONRPCRequest& request)
    This function creates an object of CMutableTransaction rawTx. It then sets the rawTx.nLockTime from parameter 2. It parses txid and vout, and push CTxIn object into rawTx. The function checks for output data or address. If address is used, it sets CScript scriptPubKey with the address. The the CTxOut is set with the amount and scriptPubKey. Lastly, it encodes rawTx with the function EncodeHexTx(rawTx) and returns the hex.

    UniValue decoderawtransaction(const JSONRPCRequest& request)
    The function creates an object of CMutableTransaction mtx, call DecodeHexTx() into mtx. It then calls TxToUniv() and returns the UniValue result.

    UniValue sendrawtransaction(const JSONRPCRequest& request)
    The function creates an object of CMutableTransaction mtx, decode the parameter with DecodeHexTx(). It creates CTransactionRef object , with the data of decoded mtx. It checks the mempool, the connection manager. If connection manager (g_connman) is available, it creates CInv object and push the inventory into CNode * pnode. It returns the transaction hash (txid).

    UniValue signrawtransaction(const JSONRPCRequest& request)
    The function creates std::vector<unsigned char> txData from the raw transaction hex data, and then CDataStream is made from the txData.
    Next, the std::vector<CMutableTransaction> txVariants is made from the CDataStream objects. Then, a new CMutableTransaction mergedTx is created from txVariants.
    Then, then CCoinsView and CCoinsViewCache are created to fetch previous inputs. 
    If the optional key is passed as parameters, it is stored in CKey object.
    It previous txout is passed as parameters, a COutPoint object and Coin are created to add to CCoinsView object.
    It also checks if redeem script is given or not in the parameters.
    Then, CKeyStore object is added. 
    In the signing part, a SignatureData sigdata is created. ProduceSignature() is called to sign and CombineSignature() is called to merge other signatures. The UpdateTransaction() is called to update the mergedTx object. Finally, the UniValue result is filled with hex and complete field and returned.

    script/sign.cpp
    bool ProduceSignature(const BaseSignatureCreator& creator, const CScript& fromPubKey, SignatureData& sigdata)
    This function takes the CScript, SignatureData, and BaseSignatureCreator, and calls SignStep() to update boolean variable named "solved". Depending the type of TX_SCRIPTHASH or TX_WITNESS_V0_KEYHASH or TX_WITNESS_V0_SCRIPTHASH, it processes and updates the sigdata. The return value is a logical "and" of VerifyScript() and the bool variable "solved".


    static bool SignStep(const BaseSignatureCreator& creator, const CScript& scriptPubKey, std::vector<valtype>& ret, txnouttype& whichTypeRet, SigVersion sigversion)
    The function consists of a big switch-case conditions. Depending on the key type, different actions are taken to sign the public key. The key type is decided using Solver().

    script/standard.cpp
    bool Solver(const CScript& scriptPubKey, txnouttype& typeRet, std::vector<std::vector<unsigned char> >& vSolutionsRet)
    This function firstly setup a std:multimap templates of txnouttype and CScript. If transaction is pay to script hash, or witness keyhash, or witness script hash, or null data; set the transaction type and return. If not, the function enters a for loop. In the for loop, it setup two CScript iterator, compare the iterators. If there is a match, it returns the transaction type if it is not TX_MULTISIG. Otherwise, the function set type to TX_NONSTANDARD and returns false.

    Wednesday, 2 August 2017

    Bitcoin core tutorial and source code walk through (Part 4)

    1) Today i will talk about some basic concepts in cryptography and its applications in Bitcoin.

    Hashing
    - deterministic
    - arbitrary length input to fixed length output
    - could be many to 1 mapping (collision)
    - difficult to reverse (one way mapping)
    - hashed message authentication code (HMAC), integrity and authenticity of the message

    Encryption
    - 1 to 1 mapping
    - reversible

    private key --> public key --> bitcoin address
          ECC encryption         hashing
                K = G*k
    where
    G: generator function
    k: private key
    K: public key

    Bitcoin network node with 4 functions
    - wallet, miner, blockchain, network routing

    Merkle root: binary hash tree used in a block, it is a summary of transactions in a block

    Fork: a change in consensus rules

    Colored coins:
    - use bitcoin transaction to record extrinsic assets
    - bitcoin is intrinsic to the blockchain
    - avoid using a colored coin related key in a regular bitcoin wallet,
    - colored coins should not be sent to address managed by regular wallets

    Coinbase
    - coinbase transaction is first transaction in each block
    - coinbase in genesis block cannot be spent, and that coinbase transaction cannot be displayed

    P2PKH
    - pay to public key hash
    - hide the public key until users are ready to spend their coins
    - P2PK is pay to public key
    - the first bitcoin transaction is P2PK

    2) In this section, i will show you how to get the details of bitcoin transaction

    $ ./src/bitcoin-cli -regtest getblock    60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986
    {
      "hash": "60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986",
      "confirmations": 511,
      "strippedsize": 226,
      "size": 226,
      "weight": 904,
      "height": 1,
      "version": 536870912,
      "versionHex": "20000000",
      "merkleroot": "2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34",
      "tx": [
        "2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34"
      ],
    ......
    }

    The command getblock returns the txid of the block. The "tx" field contains the txid.

    $ ./src/bitcoin-cli -regtest gettransaction  2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34
    {
      "amount": 50.00000000,
      "confirmations": 511,
      "generated": true,
      "blockhash": "60fba3ca57ca71dfcc3361abf3b2b3c199e6f4a0b1c1d4e7d8ac12cb9a7de986",
      "blockindex": 0,
      "blocktime": 1499419297,
      "txid": "2ca49406de60010f88876ac142f7846647942b33522a3a10bc493df480cffb34",
      "walletconflicts": [
      ],
      "time": 1499419297,
      "timereceived": 1499419297,
      "bip125-replaceable": "no",
      "details": [
        {
          "account": "",
          "address": "mhTqgiUaBFPgA2kPrKqDNVqh8xsrkBMkGu",
          "category": "generate",
          "amount": 50.00000000,
          "vout": 0
        }
      ],
      "hex": "02000000010000000000000000000000000000000000000000000000000000000000000000ffffffff03510101ffffffff0200f2052a01000000232102910a020f84bca55765fdfc4ef9e788cd6ee137242338533543f6914979fb7ce5ac0000000000000000266a24aa21a9ede2f61c3f71d1defd3fa999dfa36953755c690689799962b48bebd836974e8cf900000000"
    }

    The command gettransaction returns the object about the given transaction.

    3)  In this analysis, the source code that handles gettransaction RPC will be walked through.

    For gettransaction rpc

    wallet/rpcwallet.cpp
    UniValue gettransaction(const JSONRPCRequest& request)

    The function firstly get a pointer to the CWallet object.
    It creates a CWalletTx object, CAmount object, calls WalletTxToJSON() to display the output in JSON format. The ListTransactions() will include the sent and received transaction. Then, it calls EncodeHexTx().

    core_write.cpp
    The EncodeHexTx(const CTransaction& tx, const int serialFlags)  encodes a string, and returns the UniValue to the RPC caller.

    Sunday, 9 July 2017

    Bitcoin core tutorial and source code walk through (Part 3)

    1) The bitcoin-cli command can be used to get blockhash and get block information. For testing purpose, the regtest network is used.

    bitcoin-cli -regtest getblockhash 0
    (it returns the block hash)
    bitcoin-cli -regtest getblock 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
    (it takes block hash as parameter)

    2) In this analysis, the source code that handles getblockhash and getblock RPC will be walked through.


    For getblockhash rpc
    rpc/blockchain.cpp
    define RegisterBlockchainRPCCommands(CRPCTable &t)

    UniValue getblockhash(const JSONRPCRequest& request)

    This function checks the block height, make sure it is not out of range. Then it calls CBlockIndex class GetBlockHash() function

    chain.h

    GetBlockHash() simply returns the pointer to hash block. 

    For getblock rpc
    rpc/blockchain.cpp
    UniValue getblock(const JSONRPCRequest& request)
    This function firstly check the verbosity. It checks mapBlockIndex is not out of range. The it calls ReadBlockFromDisk() and blockToJSON() to print out the block data in JSON format.

    UniValue blockToJSON(const CBlock& block, const CBlockIndex* blockindex, bool txDetails)

    It builds the JSON value pair using the CBlock and CBlockIndex.

    validation.cpp

    bool ReadBlockFromDisk(CBlock& block, const CDiskBlockPos& pos, const Consensus::Params& consensusParams)
    This function builds a CAutoFile object which contains a FILE *. The OpenBlockFile() is used to return the FILE *. Then, the FILE * is copied to CBlock.

    FILE* OpenBlockFile(const CDiskBlockPos &pos, bool fReadOnly)

    It is a misc function that calls OpenDiskFile().

    static FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)

    It uses fsbridge::fopen() to build FILE *, and returns FILE *. The fsbridge is Bitcoin core namespace.

    mapBlockIndex is defined in validation.cpp as

    BlockMap mapBlockIndex;
    and extern BlockMap mapBlockIndex; in validation.h.

    typedef std::unordered_map<uint256, CBlockIndex*, BlockHasher> BlockMap;

    It is unordered_map in C++ std namespace .


    Wednesday, 5 July 2017

    Bitcoin core tutorial and source code walk through (Part 2)

    1) The bitcoin-cli command can be used to send bitcoin to any recipient address. For testing purpose, the regtest network is used.

    bitcoin-cli -regtest getaccountaddress ""  - get the address from default account
    bitcoin-cli -regtest sendtoaddress mjSTBVs....... {amount} <comment>  - this returns transaction id

    2) In this analysis, the source code that handles sendtoaddress RPC will be walked through.

    httprpc.cpp 
    In HTTPReq_JSONRPC(HTTPRequest *req, const std::string &)
    call jreq.parse(valRequest)    - parse the request
    call tableRPC.execute(jreq)  - execute the RPC comamnd
    rpc/server.cpp
    In CRPCTable::execute(const JSONRPCRequest &request)
    initialise pcmd
     CRPCCommand *pcmd = tableRPC[request.strMethod]
    The tableRPC array is filled in init.cpp
    init.cpp
    In AppInitParameterInteraction()
    call RegisterAllCoreRPCCommands(tableRPC)
    and RegisterWalletRPCCommands(tableRPC)
    rpc/register.h
    define the function RegisterAllCoreRPCCommands(CRPCTable &t), which calls
    RegisterBlockchainRPCCommands(t)
    RegisterMiscRPCCommands(t)
    RegisterMiningRPCCommands(t)
    RegisterRawTransactionRPCCommands(t)
    rpc/blockchain.cpp
    define RegisterBlockchainRPCCommands(CRPCTable &t)
    rpc/net.cpp
    define RegisterNetRPCCommands(CRPCTable &t)
    rpc/mining.cpp
    define RegisterMiningRPCCommands(CRPCTable &t)
    rpc/rawtransaction.cpp
    define RegisterRawTransactionRPCCommands(CRPCTable &t)

    Last but not least, RegisterWalletRPCCommands is defined in
    wallet/rpcwallet.cpp
    The RegisterWalletRPCCommands(CRPCTable &t) fill up the CRPCCommand array.

    In the CRPCComamnd,  actor function pointer, sendtoaddress function is assigned to that field.

    The function
    UniValue sendtoaddress(const JSONRPCRequest & request)
    Firstly, the address is gotten from parameter 0. Then, the amount is gotten from parameter 1. The it calls the SendMoney() function, and returns the hash value of the transaction.

    The function
    static void SendMoney(CWallet * const pwallet, const CTxDestination &address, CAmount value, bool fSubtractFeeFromAmount, CWalletTx & wtxNew)
    Firstly, it gets the balance from the wallet, parse the bitcoin address, find out the recipient.
    Then, it declares std::vector<CRecipient> vecSend
    and append the recipient to the array vecSend
    Then, it calls CWallet function CreateTransaction and CommitTransaction.

    wallet/wallet.cpp
    In the CreateTransaction(const std::vector<CRecipient>& vecSend, CWalletTx& wtxNew, CReserveKey& reservekey, CAmount& nFeeRet, int& nChangePosInOut, std::string& strFailReason, const CCoinControl* coinControl, bool sign)

    Firstly, it checks the recipient amount , which must be >= 0. Then, it sets the lockTime to the next block to discourage fee sniping. The CTxOut txout object is constructed. The txout is appended to CMutableTransaction txNew vout vector. Then, it chooses the Bitcoin to use for transaction. If input value is greater than selected value, it creates a CTxOut and insert it to same CMutableTransaction vout at position nChangePosInOut.

    After that, it creates CTxIn and appends the CTxIn to vin vector. It calculate the minimum fee needed. If the fee needed is less than min relay fee, the transaction is dropped. Then the min fee inclusive of necessary fee is calculated.

    If transaction requires signature, a signature is produced (via ProduceSignature function). UpdateTransaction() is called to update with signature data.

    Then, the CMutableTransaction txNew is embedded in CWalletTx wtxNew object. If wtxNew size is bigger than mempool's chain limit, transaction will be dropped.

    In the bool CWallet::CommitTransaction(CWalletTx& wtxNew, CReserveKey& reservekey, CConnman* connman, CValidationState& state)

    It calls AddToWallet(wtxNew) and coin.BindWallet(this) to notify old coins are spent.
    It then broadcast transaction if wtxNew is accepted to mempool

    bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFlushOnClose)

    It creates the walletdb. It creates the std:pair object from mapWallet.insert(std::make_pair(hash, wtxIn)). From the std::pair object, it creates another CWalletTx wtx object. If it is new transaction, it gets the order position from walletdb, and adds the hash. If it is not new, the code merges the hash block. Finally, it writes to disk with walletdb.WriteTx(wtx).

    mapWallet is declared in wallet/wallet.h as
    std::map<uint256, CWalletTx> mapWallet;


    Tuesday, 4 July 2017

    Bitcoin core tutorial and source code walk through (Part 1)

    Bitcoin core is the reference implementation of Bitcoin code.

    1) Download Bitcoin core from github
    https://github.com/bitcoin/bitcoin

    2) Build it
    /autogen.sh
    ./configure --enable_debug
    make
    make install

    3) Setup the bitconf.conf. In the minimum, setup the rcpuser and rpcpassword field
    bitconf.conf configuration file path:
    OS                Default bitcoin datadir      Typical path to configuration file
    Windows     %APPDATA%\Bitcoin\     C:\Users\username\AppData\Roaming\Bitcoin\bitcoin.conf
    Linux           $HOME/.bitcoin/               /home/username/.bitcoin/bitcoin.conf
    Mac OSX    $HOME/Library/Application Support/Bitcoin/             /Users/username/Library/Application Support/Bitcoin/bitcoin.conf

    4) Useful commands
    bitcoind -daemon  ; bitcoin-cli stop      - start and stop bitcoin daemon
    bitcoind -printtoconsole                        - print debug message to console
    bitcoind -daemon -proxy=192.168.1.1:80   - run bitcoind w/o downloading new blocks
    bitcoind -testnet    bitcoin-cli -testnet      - run in test network
    bitcoind -regtest    bitcoin-cli -regtest     - run in regression test network
    bitcoind -debug=rpc                              - set the debug level

    bitcoin-cli getblockhash <number>
    bitcoin-cli getblock <hash>
    bitcoin-cli getblockcount
    bitcoin-cli -regtest getblockchaininfo
    bitcoin-cli -regtest generate 500      - mine 500 blocks
    bitcoin-cli -regtest getbalance          - check rewards

    5) Source code walk through is based on v0.14.99.0 (bitcoind -version)
    bitcoind.cpp  
    main() calls AppInit()
    AppInit() calls AppInitParameterInteraction()
    then calls AppInitMain(threadGroup, scheduler) - main init function
    init.cpp
    AppInitParameterInteraction() - setup debug level, register RPC commands
    AppInitMain()
                lock directory
                start lightweight task scheduler thread
                AppInitServers() -  start App servers, such as HTTP, RPC servers
                RegisterNodeSignals(GetNodeSignals())
                create zero message queue
                cache size calculations
                LoadBlockIndex(chainparams)
                check genesis block
                InitBlockIndex(chainparams)
                InitLoadWallet()
                GetConsensus()
               CConnman::Start(scheduler, connOptions)
               wallet->postInitProcess(scheduler)
    net.cpp
    CConnman::Start(...)     - start threads to accept connections, process messages
    call InitBinds(std::vector<CService>&, std::vector<CService>&) - bind to address and port number
    InitBinds() call GetListenPort()
    validation.cpp
    LoadBlockIndex(chainparams)        - load block index from db
    GetBlockSubsidy(nHeight, consensusParams)       - get mining block rewards
    merkleblock.cpp
    CMerkleBlock constructor initialises CPartialMerkleTree
    CPartialMerkleTree constructor calls TraverseAndBuild()
    net_processing.cpp
    ProcessMessages->ProcessMessage->ProcessGetData->CMerkleBlock
    net.h
    RegisterNodeSignals->CNodeSignals.ProcessMessages
    init.cpp 
    RegisterNodeSignals
    rpc/server.cpp
    StartRPC()  - called by AppInitServers
    httpserver.cpp
    StartHTTPServer()  - clled by AppInitServers
    util.cpp
    LogPrintStr
    util.h
    LogPrint          - debug log macro
    LogPrintf         - debug log macro
    LogAcceptCategory   - check the BCLog category and decide to print log or not
    httprpc.cpp
    HTTPReq_JSONRPC(HTTPRequest* req, const std::string &)     - parse the rpc request that is sent from bitcoin-cli

    Sunday, 2 July 2017

    Thesis on Bitcoin

    Here is my master thesis on Bitcoin. The topic is :

    THE RISK PERCEPTIONS AND TECHNOLOGY ADOPTIONS OF E­-COMMERCE USERS
    TOWARDS DIGITAL CURRENCY

    It is on google drive at the link below :
    click at thesis link

    Saturday, 15 April 2017

    Embedded Product Design on Linux

    An embedded product on Linux involves several stages. As an example, we use a Zigbee embedded product on Linux to discuss this.

    The software components that make up an embedded product:

    Dynamic UI: HTML, JQuery, Javascript, XML data, CGI process

    Application and Libraries: Many applications, and third-party libraries


    The figure above shows the interaction among the application processes, CGI app and Dynamic UI.

    Linux Kernel and Drivers:
    It maybe required to integrate new drivers into Linux kernel space, to enable the SoC on the main board.

    Bootloader:
    It maybe required to modify boot loader if we upgrade to new generation of flash and DRAM IC.


    Real-Life Embedded Systems problems and solutions:


    • Zigbee Carrier board integration:

    Problem:
    Old carrier board is responding to Zigbee Ping command, and can find its IEEE address.
    New carrier board is responding to Zigbee Ping command, but cannot find its IEEE address.

    Reasons: Firmware is upgraded in New carrier board, so a upgraded SDK is required to work with the new firmware in New carrier board.

    Debugging process: In Linux, open the serial port where the Zigbee module is connected, then set the firmware in carrier board to command mode, then send the Zigbee Ping command to the carrier board.

    • Board Graceful shutdown:

    Problem:
    Power-off the board immediately without proper shutdown sequence will cause linux file system corruption.

    Solution: Ask MSP430 to set an interrupt to MPC8315 when the power switch is pressed. An interrupt handler is added to linux kernel (as a linux driver module) to handle MSP430 interrupt. The driver module will create a kthread to listen to the interrupt. When interrupt arrives, a user space helper function is called (by using usermode helper API).

    The userspace function handles the proper shutdown sequence, such as carrying out the "init 0" command.

    • Move DRAM content forward:

    Problem:
    We want to copy DRAM content.

    Solution:

    void loader(void)
    {
      long * ptr = loki_plus_dram;
      int i;
      for (i=0; ptr[i]!=0x1FFFF; i++)
      {
        *((long *)ptr[i]) = ptr[i+1];
      }
      return;
    }

    • TI Zigbee chipset cc2530 UART 1 setup:

    Problem:
    Unable to communicate to TI cc2530's UART 1 port.

    Solution:
    In IAR Workbench, Project->Options, choose C/C++ compiler, set:
    HAL_UART_DMA = 0
    HAL_UART_ISR=2

    This is a must for the source code to call hal_uart_isr() function in hal_uart.c

    • Wifi camera lose connectivity to Wifi AP:

    Problem:
    Wifi camera lost connection to Wifi AP after 1 day, meaning Wifi AP cannot detect the Wifi camera after 1 day.

    Solution:
    In /etc/hostapd.conf, set wpa_strict_rekey = 0
    or set wpa_group_rekey = 0 and wpa_ptk_rekey = 0 if maintaining wpa_strict_rekey = 1

    • Glucometer timeout issue:

    Problem:
    The Bluetooth program on linux is not able to read all data from Glucometer, read function hang, need to power off/on Glucometer .

    Solution:
    Use a function called setsockopt to set socket to non-blocking with timeout.

    setsockopt(sk, SOL_SOCKET, SO_RCVTIMEO, (struct timeval *)&tv, sizeof(struct timeval));
    ret = read(sk, buf, sizeof(buf));
    if (ret == -1)
    {
      timeout!!
      read again
    }

    • Add debug statement control:

    Problem:
    debug statements are lost if we do not have serial port to monitor them

    Solution:
    Define a debug macro


    //DEBUG functions
    #define LOG_OUTPUT stderr

    #define LOG(level, ...) \
            {
                    fprintf(LOG_OUTPUT, level); \
                    fprintf(LOG_OUTPUT, __VA_ARGS__); \
                    fflush(LOG_OUTPUT); \
            }

    #define DEBUG(...)   LOG("Myglucohealth   ", __VA_ARGS__)

    Actual usage:
    DEBUG("Can't find adapter %s\n", adapter);

    Normally the debug statements will be sent to console, but we can redirect them to anywhere with a easy command:
    <process_command>  &> /mnt/storage/bluetooth/process_command.log  &
    ---> redirect both stdout and stderr to a file

    • TCP socket: too many open files:

    Problem:
    A TCP client connected to TCP server every 60 seconds and close the connection after sending a message. After 6 hours, socket call failed, saying "socket: too many open files"

    Solution:
    The TCP server must close the socket that is returned by accept(). And the TCP client should close the socket first, then the TCP server close it. This is due to the TIME_WAIT settings which will create waiting burden on the TCP server if the TCP server closes the socket first, and if the TCP server is a busy one.

    Tips:
    Use netstat to check for open socket connections
    Use "ls /proc/<pid>/fd" to check for open file descriptor
    Turn off SO_LINGER, call it after open the socket, eg: setsockopt(sd, SOL_SOCKET, SO_LINGER,...);

    • Simultaneous Wifi and Ethernet transmission, and data lost

    Problem:
    When there are simultaneous Wifi and Ethernet transmission to the DUT, and DUT does not receive data via the Wifi after a few hours.

    Debugging:
    Check the interrupt mask and interrupt enable register bits. On the general interrupt controller (GIC), the Wifi and ethernet bits are adjacent to each other. When data is received at the hardware interface, internet status bit is set. When inside the ISR, interrupt enable bit is disabled, and interrupt mask bit is masked out. When exiting ISR, interrupt enable bit is enabled, interrupt mask bit is cleared. If ISR interrupt enable bits is not set, something is amiss.

    Solution:
    The problem is due to a hardware error that ethernet interrupt bit will affect the Wifi interrupt bit. The software workaround is to enable the Wifi and Ethernet interrupt enable bits in the ISR when exiting ISR.

    Wednesday, 21 December 2016

    Bluetooth Protocol and IEEE11073

    Bluetooth Protocol Stack:

    Bluetooth Stack Partitioning:

    host controller - Bluetooth hardware
    Bluetooth host - the PC
    HCI - Host Controller Interface

    Pairing:
    Pairing is the process of establish the right to make a connection. Once the units have been paired and provided the paired unit is "trusted" then connecting becomes automatic. Otherwise you must respond each time an attempt is made to connect that it is ok. Keeping the unit in a connected state makes it easy and convenient to accept calls but you do pay for the higher battery drainage.

    Pairing is also and act of exchanging of authentication key.

    Inquiry Procedure:
    The inquiry procedure enables a device to discover which devices are in range, and determine the addresses and clocks for the devices. The inquiry procedure involve a unit (the source)sending out inquiry packets (inquiry state) and then receiving the inquiry reply  .The unit that receives the inquiry packets (the destination),  will hopefully be in the inquiry scan state to receive the inquiry packets. The destination will then enter the inquiry response state and send an inquiry reply to the source. After the inquiry procedure has completed, a connection can be established using the paging procedure.

    Paging Procedure:
    With the paging procedure, an actual connection can be established. The paging procedure typically follows the inquiry procedure. Only the Bluetooth device address is required to set up a connection. Knowledge about the clock (clock estimate) will accelerate the setup procedure. A unit that establishes a connection will carry out a page procedure and will automatically be the master of the connection. The procedure occurs as follows:

    1:   A device (the source) pages another device (the destination ) : Page state

    2:  The destination receives the page : Page Scan state

    3:   The destination sends a reply to the source. : Slave Response state: Step 1

    4:   The source sends an FHS packet to the destination : Master Response state: Step 1

    5:   The destination sends it's second reply to the source. : Slave Response state : Step 2

    6:   The destination & source then switch to the source channel parameters  :Master Response state: Step 2 &  Slave Response state: Step 3

    Bluetooth Protocol:

    On Linux, I use Bluez, an open source implementation of Bluetooth protocol. Bluez will run a daemon called bluetoothd, kernel modules called rfcomm, bluetooth, bnep, etc. Bluez is used for receiving and sending the Bluetooth messages.

    MCAP - Multi Channel Adaptation Protocol. It is used by HDP. It facilitates the creation of communication link for exchanging commands, and data links to transfer actual Medical Device data. MCAP guarantees reliable transmission of data.

    SDP - Service Discovery Protocol. It is used by all Bluetooth profiles to discover available services on remote devices, so that connections over L2CAP can be established.

    L2CAP - Logical Link Control and Adaption Protocol. It supports protocol multiplexing, packet segmentation and re-assembly, quality of service, re-transmission, flow control. It is only used for ACL link.


    Bluetooth Profile:
    It is an interface specification, resides on top of the Bluetooth core specification.
    It specifies:
    • user interface formats
    • specific parts of Bluetooth protocol stack used by the profile
    Eg.
    Advanced Audio Distribution Profile - A2DP. It defines how high quality audio can be streamed from one device to another over a Bluetooth connection. For example, music can be streamed from a mobile phone to a wireless headset.

    Health Device Profile - HDP. It is designed to facilitate transmission and reception of Medical Device data. It also performs SDP behavior, to connect to remote HDP devices.

    IEEE 11073 stack. It performs building, transmission, reception, parsing of IEEE PDU packets for the associated IEEE 11073 agent/manager. It links to HDP.

    Serial Port Profile - SPP. It is used to simulate RS-232

    Generic Access Profile - GAP. It describes the required features of all Bluetooth profiles, including inquiry, connection, authentication procedures.

    Device ID Profile - DIP. It provides device specific information through use of the SDP. If vendor specific info is required, this profile provides specific way to acquire this info. It enables identification of the manufacturer, product id, product version. It is useful in allowing a PC to identify a connecting device and download appropriate drivers.

    Bluetooth Power Class:
    class 1: long range (100m)
    class 3: long range (10m)
    class 3: long range (10cm)


    Bluetooth CoD:
    class 0xSSDDdd (three bytes)

    The default CoD (Class of Device) is 0x000100 which simply stands for "Computer".

    The Bluetooth device class is a high-level description of the bluetooth device, composed of three bytes: the "Major Service Class" (byte "SS" above), the "Major Device Class" (byte "DD" above) and the "Minor Device Class" (byte "dd" above). These classes describe the high-level capabilities of the device, such as "Networking Device", "Computer", etc. This information is often used by clients who are looking for a certain type of service around them.

    Where it becomes tricky is that another type of mechanism for service discovery exists: "SDP", as in "Service Discovery Protocol".

    In practice, most Bluetooth clients scan their surroundings in two successive steps: they first look for all bluetooth devices around them and find out their "class". You can do this on Linux with the hcitool scan command. Then, they use SDP in order to check if a device in a given class offers the type of service that they want.

    Bluetooth Master/Slave:
    It is not possible to detrmine bluetooth master or slave programatically.The master/slave roles which a device assumes is invisible to the user(application).The same device can act as a master of one piconet and a slave in an (adjacent) piconet.What needs to be decided in your application is whether the device will act as a client or a server or both and what will be the services(profiles) that it supports.

    Bluetooth FH:

    All Bluetooth devices participating in a given piconet are synchronized to the frequency-hopping channel for the piconet. The frequency hopping sequence is determined by the master's device address and the phase of the hopping sequence (the frequency to hop at a specific time) is determined by the master's internal clock. Therefore, all slaves in a piconet must know the master's device address and must synchronize their clocks with the master's clock.

    During the connection handshake, the slave receives one packet (FHS, Frequency Hop Synchronization) that contains information about the master's BD_ADDR and the clock, so the slave's internal clock can be synchronized.

    Bluetooth State Machine:

    Bluetooth device can move from one state to a new state.

    Bluetooth VDP

    Vena Dongle Protocol (VDP) is an application protocol between Bluecore module and host microprocessor.

    Host micro without IEEE11073 stack
    Host micro with IEEE11073 stack

    The messages that are carried across VDP link:
    1. HDP messages
    2. SPP messages for accessing legacy devices
    3, Bluetooth device discovery, device pairing control packets.

    Bluetooth Link Management Protocol (LMP)

    In the usage of Nonin Oximeter, we have a problem where Oximeter fails to transmit measurement data to Bluetooth module after using for some time.

    The root cause is because the oximeter sends out LMP "not_accepted" in response to the LMP "au_rand" message from Bluetooth module. Here is the message trace in terms of LMP opcode. The "not_accepted" message is the 2nd last message. 

    au_rand contains a random number that is encrypted by link key. When au_rand is not accepted, meaning the oximeter cannot compute the same number using the link key, meaning the oximeter link key database is corrupted

    BT Module                Oximeter 
    --------------              ------------------ 
                                   host_connection_req 
                                   version_req 
                                   version_res 
                                   features_req 
                                   features_res 
                                   slot_offset 
                                   switch_req 
                                   accepted 
    accepted 
                                   feature_req 
    SET_AFH 
    setup_complete 
    feature_res 
    channel_classification 
                                  setup_complete 
                                  timing_accuracy_req 
    max_slot 
    max_slot_req 
                                  channel_classification 
    timing_accuracy_res 
                                  accepted 
                                  max_slot_req 
    feature_req 
                                  feature_res 
    accepted 
    au_rand 
                                  not accepted 
                                  detach

    In normal operation, Bluetooth module challenges oximeter by sending the random 128-bit value AU_RAND and expects from oximeter the response: 
       SRES = E(K, AU_RAND, BD_ADDR_OXIMETER) 
    K is link key, E is Bluetooth authentication function. The oximeter receives the AU_RAND and sends back SRES as response. 

    When oximeter sends not_accepted after it receives au-rand, meaning the oximeter link key database is corrupted. 

    The behavior in this case is up to the Bluetooth module requesting the authentication. It can demand a new pairing by sending LMP in_rand, or it can refuse the connection and sends LMP detach. So for the Bluetooth module, the Bluez stack simply detach the link. 

    So we need to modify Bluez stack for solving this "Oximeter takes measurement but no data is transmitted" problem. 

    Bluetooth Packet Format

    This is a standard Bluetooth packet:
    ACCESS CODE [72] HEADER [54] PAYLOAD [0-2745]

    ACCESS CODEAccess code is used for synchronization, DC offset compensation and identification.
    HEADERHeader contains link control (LC) information.
    PAYLOADPayload carries voice and data fields of upper layers.

    Payload Format

    HEADER [8-16] BODY [Indicated
    in header]
    CRC CODE [16]

    HEADERHeader is one or 2 bytes longer. Only data fields have a payload header.
    BODYPayload body includes the user host information. The length os the body is indicated in the length field of the payload header.
    CRC CODEThe 16-bit cyclic redundancy check code is generated by the CRC-CCITT polynomial 210041 (octal).

    ACL

    Asynchronous Connectionless Link. One of the two types of data links defined for the Bluetooth Systems, it is an asynchronous (packet-switched) connection between two devices created on the LMP level. This type of link is used primarily to transmit ACL packet data. The ACL link is a point-to-multipoint link between the master and all the slaves participating on the piconet. In the slots not reserved for the SCO links, the master can establish an ACL link on a per-slot basis to any slave, including the slave already engaged in an SCO link. Only a single ACL link can exist. For most ACL packets, packet retransmission is applied.

    SCO

    Synchronous Connection-Oriented. The SCO link is a symmetric point-to-point link between a master and a single slave in the piconet. The master maintains the SCO link by using reserved slots at regular intervals (circuit switched type). The SCO link mainly carries voice information. The master can support up to three simultaneous SCO links while slaves can support two or three SCO links. SCO packets are never retransmitted. SCO packets are used for 64 kB/s speech transmission.

    DM

    Data - Medium Rate. An ACL link data packet type for medium rate data. DM1 packets carry information data only, contining a 16-bit CRC code and up to 18 info bytes. They are encoded using 2/3 FEC and the packet can cover up to a single time slot. DM3 packets are the same except they can cover up to 3 time slots, and can carry up to 123 info bytes. DM5 packets are the same again except they can cover up to 5 time slots and can hold up to 226 info bytes. See also Bluetooth.

    DH

    Data-High Rate. An ACL link data packet type for high rate data. DH1 packets are similar to DM1 packets, except the info in the payload is not FEC encoded. This means the DH1 packet can carry up to 28 info bytes and covers a single time slot. The DH3 is the same except it can cover up to 3 time slots and contain up to 185 info bytes. The DH5 packet is the same again except it can cover up to 5 time slots and contains up to 341 info bytes See also Bluetooth packet types.

    Bluetooth D-Bus commands:

    Bluez:

    dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.freedesktop.DBus.Introspectable.Introspect

    dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez org.freedesktop.DBus.Introspectable.Introspect

    dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/690/hci0 org.bluez.Adapter.RemoveDevice objpath:/org/bluez/690/hci0/dev_00_1C_05_00_4A_AD

    dbus-send --system --type=method_call --print-reply --dest=org.bluez /org/bluez/1282/hci0 org.bluez.Adapter.ListDevices

    dbus-send --system --type=method_call --print-reply --dest=org.bluez /  org.bluez.Manager.FindAdapter string:hci0


    dbus-send --system --type=method_call --print-reply --dest=org.bluez / org.bluez.Manager.ListAdapters

    Antidote Healthd:

    dbus-send --system --type=method_call --print-reply --dest=com.signove.health /com/signove/health org.freedesktop.DBus.Introspectable.Introspect

    Bluetooth IEEE11073 

    IEEE11073 is an Bluetooth application layer protocol, used by HDP devices. IEEE11073 specifies the formats and standards that HDP devices use to send messages. The standard itself is long winded and talking about it is time consuming. IEEE11073 is promoted by the Continua Alliance.

    On Linux, I use Antidote, an open source implementation of IEEE11073. It is developed by Signove Technologies. It is written in C.

    As an example, we can sniff an Bluetooth Nonin 9560BT Oximeter. Oximeter is set as Master. Bluetooth host is slave. If encrypted data that is captured is in error, we need to look at these LMP frames.

    LMP_in_rand
    LMP_accepted
    LMP_comb_key

    If these LMP frames are missing, we need to delete the oximeter device from Bluetooth database, then re-pair the device. It is because the link key is generated at local and remote devices during pairing stage. If devices are already paired, then link key generation won't be performed by the devices (link key is saved in devices already), and sniffer won't be able to capture the link key, and can't decrypt the bluetooth packets.

    The IEEE11073 message exchange is shown below.


    Link Management (LM):


    The LM assists in the pairing procedure by communicating parameters and results between the local and remote Bluetooth devices. The calculations are done in the baseband. On the link manager level, the pairing procedure starts with the transmission of the PDU LMP_in_rand (containing the 128-bit random number
    IN_RAND) from one of the units to the other. This PDU will trigger the generation of the initialization key for use in the protocol for creating the actual link key. If a unit key is to be used, the LMP unit key
    command is sent in one direction only, with the secret unit key XORed with the initialization key as its parameter. The receiver can easily calculate the key from this. If a combination key is to be created, two contributions (one from each side) are needed. This is accomplished through the LMP_comb_key PDU. The argument of this PDU is also a random number, but generating this and deriving the link key from it is slightly more complicated. 

    After these PDUs have been exchanged, both ends are able to compute the link key. As a final check of success of this procedure and the established link key, a mutual authentication event is performed. A 128-bit challenge is sent through the LMP_au_rand , and the 32-bit response uses the LMP sres PDU. If these authentications do not fail, both the local and the remote Host Controller notify their respective hosts.


    Click on the image below to see the link key exchanges.