Build a Robot From A Power Wheelchair (1/2)

I have been working on a robotic project for a while. In this project, we build a robotic wheelchair controlled by head motions.

IMG_2245

It starts from a commercial powered wheelchair (Titan Front Wheel Drive Power Chair).

We took over the control signals, added a number of sensors and wrote some computer vision softwares. It is a lot of fun. At this time point, we can drive it with head motions, a little slow but very easy to control. A short video is posted.

Robotic Wheelchair
[KGVID width=”640″ height=”360″]http://personal.stevens.edu/~hli18/video/robot.mp4[/KGVID]

The most challenging part so far, to our surprise, is how to take over the driving signals. In fact, once the powered wheelchair can be controlled with digital signals, it is somehow a robot.

We can find quite a few posts discussing how to hack the joysticks with Arduino. If you have a VR2 joystick, it would be much easier. This post (http://myrobotnstuff.blogspot.com.au/2012/07/circuit-for-wheelchair-robot.html) explains the process well. Basically, you can reproduce the digital signals triggered by the joystick with the Arduino.

But unfortunately, the joystick with the Titan power chair is with a different design and it is much more difficult to hack.
It is Dynamic Shark controller. If your remote (joystick) looks like this one , you probably have this kind of joystick.

The difficulty in reproducing the control signals generated by this kind of joystick is that it directly outputs 25V differential signals from the remote to the power module, which is a big black box under the seat. The signals are encoded and there is probably no way to get the encoding protocol.

We did quite a few rounds of trial and error and finally figured out a plan after burning out a joystick…

IMG_1587

This kind of joystick works in a special way. At the end of the handle of joystick, there is with a small coil. 4 small coils are on the board under it. Because of the electromagnetic induction, the coil on the handle of the joystick produces different signals at different positions in the 4 small coils under it.

Roughly speaking, the (analog) signals produced by the 4 small coils are then processed by an on-board micro-controller to produce the driving signal. Because the signals produced by the micro-controller are encoded in an unknown protocol, we choose to get in by reproducing the outputs from the 4 small coils.

If you take away all 4 small coils (DO NOT do that…), the board looks like this

A

After some processing circuits, finally the outputs of the 4 coils can be found as two signals passed the two resistances in the red circuits. If you use an oscilloscope to check the signals, you will find that they are sine waves!

With this observation, the plan is clear. Soldering two wires at the right end of these two resistances (be careful!), we need to use an Arduino to reproduce these two sine waves to drive the wheelchair.

One of the two sine waves is for the X-direction and the other one is controlling the driving in the Y-direction.
The phase the sine wave indicates the sign (forward in X-direction or backward in X-direction) and the amplitude indicates the speed. The two sine waves should be sort-of aligned, which means the phase difference should be either 0 or 180.

With an Arduino, we will produce these sine waves signals to the two wires.

Convert the saved checkpoint of cuda-convenet to the proto of Caffe

*Credits of the scripts go to jamt9000 on github, thanks for the helpful work!

You are a deep learning consumer, you have a saved check point you trained with the cuda-convenet. You want to use Caffe to evaluate the trained neural networks on new data possiblely due to the absent of GPU. Hence you need to convert the saved checkpoint to the format compatible to Caffe.

1. check out this version of caffe: jamt9000/caffe:convert-cuda-convnet or my copy.

$git clone https://github.com/jamt9000/caffe/tree/convert-cuda-convnet convert-cuda-convnet

2. compile it following the Caffe installation instruction

## Misc. Setup 
$make all
$make pycaffe

3. convert the check-point with the following codesnippet

import sys
sys.path.append('./python/')
sys.path.append('./tools/extra/')

from convert_net import *

if (len(sys.argv) < 2):
    print "Usage:- path-to-saved-convnet"
    sys.exit(-1)

arg_convnet = sys.argv[1]
output_prototxt = arg_convnet + "-prototxt"
output_proto = arg_convnet + "-proto"

netpt = cudaconv_to_prototxt(arg_convnet)
fh=open(output_prototxt, 'w')
fh.write(netpt)
fh.close()
print "Wrote {0}".format(output_prototxt)

netpb = cudaconv_to_proto(arg_convnet)
fh=open(output_proto, 'wb')
fh.write(netpb.SerializeToString())
fh.close()
print "Wrote {0}".format(output_proto)
$python convert-convnet2caffe.py saved-convnet-checkpoint

4. Now you should find two files: saved-convnet-checkpoint-proto and saved-convnet-checkpoint-prototxt, they are in an older format, you need some tools provided by Caffe to upgrade them.

5. checkout and compile the up-to-date (or any more recent stable) version of Caffe

6. Under build/tools/ you will find tools named upgrade_net_proto_text.bin and upgrade_net_proto_binary.bin, use them to upgrade the saved-convnet-checkpoint-proto and saved-convnet-checkpoint-prototxt accordingly.

7. Almost done, you may need to manually fix the prototext file a little bit for example, add the input layer following any example proto text files provided by the Caffe.

Good luck!

[OpenCV] detectMultiScale: output detection score

OpenCV provides quite decent implementation of the Viola-Jones Face detector.

A quick example looks like this (OpenCV 2.4.5 tested):

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            cascade.detectMultiScale(img, objs, scale_factor, min_neighbors);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}
g++ -std=c++0x -I/usr/local/include `pkg-config --libs opencv` main.cc -o main

The detection results are as shown below:
result

For more serious user, it would be nice to have a detection result for each detected face.
The OpenCV provides a overloaded function designed for this usage which is lack of detailed documentation:

vector reject_levels;
vector level_weights;
cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors);

The reject_levels and level_weights will keep being empty until you write it like this (The whole file):

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            vector reject_levels;
            vector level_weights;
            cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors, 0, Size(), Size(), true);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
                putText(img_color, std::to_string(level_weights[n]),
                        Point(objs[n].x, objs[n].y), 1, 1, Scalar(0,0,255));
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}

However, this will give you a large number of detected rectangles:
result-org

This is because OpenCV skips the step of filtering out the overlapped small rectangles. I have no idea whether this is by design. But output likes this would not be helpful at least in my own case.

So we would need to make our own changes in the OpenCV's source code.
There are different ways to design detection score, such as
"In the OpenCV implementation, stage_sum is computed and compared against the i stage_threshold for each stage to accept/reject a candidate window. We define the detection score for a candidate window as K*stage_when_rejected + stage_sum_for_stage_when_rejected. If a window is accepted by the cascade, we just K*last_stage + stage_sum_for_last_stage. Choosing K as a large value e.g., 1000, we ensure that windows rejected at stage i have higher score than those rejected at stage i-1." from http://vis-www.cs.umass.edu/fddb/faq.html

Actually, I found a straightforward design of detection score works well in my own work. In the last stage of the face detector in OpenCV, detection rectangles are grouped into clustered to eliminated small overlapped rectangles while keeping the most potential rectangles. The number of final detected faces is at most same as the number of clusters. So we can simply use the number of rectangles grouped into the cluster as the detection score of the associated final rectangle, which may not be accurate but could work.

To make this change, in OpenCV-2.4.5, find the file modules/objdetect/src/cascadedetect.cpp (line 200)

// modules/objdetect/src/cascadedetect.cpp (line 200)
// int n1 = levelWeights ? rejectLevels[i] : rweights[i]; //< comment out this line
int n1 = rweights[i]; //< the change

We then modify the file main.cc accordingly:

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            vector reject_levels;
            vector level_weights;
            cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors, 0, Size(), Size(), true);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
                putText(img_color, std::to_string(reject_levels[n]),
                        Point(objs[n].x, objs[n].y), 1, 1, Scalar(0,0,255));
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}

And we can have the detection scores like this:
result-final

Linux Swap文件

想象一下,两个实验进程跑了两天,还有一天就跑完了,这个时候你发现如果再跑一会儿内存就要爆了…怎么办? (好惊险的感觉 XD)

好吧,其实用到的只是很基本的操作系统知识,不过还真难得用到一回。

程序面对的都是虚拟内存。64位的操作系统下,虚拟内存非常大,但是实际物理内存相对而言小得多。所以,操作系统对内存分页 (就是分成一块一块的,每一块儿叫做一页) 物理内存一旦满了,把暂时不需要的页写到硬盘里。过了一会儿程序又要访问被写到硬盘里的那部分内容,操作系统就在物理内存中选一个页 (怎么选很讲究的),把硬盘里的那个给换回来。程序不停的运行,操作系统就换来换去…

所以,上面我们遇到的情形就可以解决了。把其中一个进程挂起 (suspend),Linux下可以用

Ctrl+Z

,然后这部分内存就是暂时不用的了。这个时候用

$top

查看内存使用情况,可以看到一个CPU占用率为0的进程占用的内存越来越少,另一个越来越多。这样就行了,等一个进程跑完,再用

$fg

命令把挂起的进程调到前台就可以了。

但是等等。虚拟内存具体是在哪里呢?数据终究是写在内存/硬盘上的,Linux下被换到硬盘上的内存在Swap分区 (交换分区) 里。安装系统的时候需要格式化一个分区为Swap格式,就是这个分区。

$swapon -s

可以查看交换分区的大小。

糟糕!刚才那个被挂起的进程占用了24G的内存,但是现在看到我的交换分区只用12G,怎么办?一旦交换分区和内存都满了,会发生神马事情,我也没有体验过,估计应该是系统卡死或者卡而不死吧。

所以,应该赶紧增加交换分区的大小才是。可是如果你和我一样,很悲催的没有Root权限 (Root权限貌似是必须的…),而且也根本没有多余的分区可以挂载了,怎么办?

可以用Swap文件 (点题) ! 就是把一个文件用做swap分区,Linux下什么都是文件,分区应该也是吧。要增加系统可用的虚拟内存,当然这要求你硬盘剩余空间够大…

$dd if=/dev/zero of=~/swapfile bs=1024 count=41943040

会在HOME下创建一个40G的文件”~/swapfile”,命令要执行一会儿,需要写一段时间硬盘,执行完了会显示写硬盘的速度,可以用来做测速的。
然后告诉系统用这个文件做交换文件

mkswap ~/swapfile

就没问题了。
(实际上不行,还需要这个命令, –!)

sudo swapon ~/swapfile

其实最一开始那个情形下,如果两个进程继续跑下去,操作系统仍然会把一部分内容换出来的。只是大量换页操作会让程序执行的时间更长,而且如果交换空间不够大,系统最终仍然可能会被卡死。

感叹一下,一个实验要跑三天的同学伤不起啊..

Draw ROC Curve

A piece of fairly simple Matlab script to draw the ROC Curve from an array of scores and an array of labels.

function [Tps, Fps] = ROC(scores, labels)
 
%% Sort Labels and Scores by Scores
sl = [scores; labels];
[d1 d2] = sort(sl(1,:));
 
sorted_sl = sl(:,d2);
s_scores = sorted_sl(1,:);
s_labels = round(sorted_sl(2,:));
 
%% Constants
counts = histc(s_labels, unique(s_labels));
 
Tps = zeros(1, size(s_labels,2) + 1);
Fps = zeros(1,  size(s_labels,2) + 1);
 
negCount = counts(1);
posCount = counts(2);
 
%% Shift threshold to find the ROC
for thresIdx = 1:(size(s_scores,2)+1)
 
    % for each Threshold Index
    tpCount = 0;
    fpCount = 0;
 
    for i = [1:size(s_scores,2)]
 
        if (i >= thresIdx)           % We think it is positive
            if (s_labels(i) == 1)   % Right!
                tpCount = tpCount + 1;
            else                    % Wrong!
                fpCount = fpCount + 1;
            end
        end
 
    end
 
    Tps(thresIdx) = tpCount/posCount;
    Fps(thresIdx) = fpCount/negCount;
 
end
 
%% Draw the Curve

% Sort [Tps;Fps]
x = Tps;
y = Fps;

% Interplotion to draw spline line
count = 100;
dist = (x(1) - x(size(x,2)))/100;
xx = [x(1):-dist:x(size(x,2))];

% In order to get the interpolations, we remove all the unique numbers
[d1 d2] = unique(x);
uni_x = x(1,d2);
uni_y = y(1,d2);
yy = spline(uni_x,uni_y,xx);

% No value should exceed 1
yy = min(yy, 1);

plot(x,y,'x',xx,yy);

Hope it helps.


Some improvements were added.

For a sample input:

>> scores = rand(1,20)*100

scores =

  Columns 1 through 7

   43.8744   38.1558   76.5517   79.5200   18.6873   48.9764   44.5586

  Columns 8 through 14

   64.6313   70.9365   75.4687   27.6025   67.9703   65.5098   16.2612

  Columns 15 through 20

   11.8998   49.8364   95.9744   34.0386   58.5268   22.3812

>> labels = round(rand(1,20))

labels =

  Columns 1 through 12

     1     0     1     1     1     1     1     0     0     0     1     0

  Columns 13 through 20

     1     0     1     0     0     0     1     0

>> ROC(scores,labels);

Gives an output like:
ROC

Fork it on Github: DrawROC on Github

[iOS]How to save and load a custom object?

Here we go.

How to write a custom object which can be archived to a text file?

Usually, we use NSKeyedArchiver to serialize an object and write it to a file.
Correspondingly, NSKeyedUnarchiver is used to get the object from the file.

The NSKeyedArchiver’s interface is like this:

+ (BOOL)archiveRootObject:(id)rootObject toFile:(NSString *)path;

NSKeyedArchiver is a NSCoder. The rootObject should conform to the protocol NSCoding.

@protocol NSCoding

- (void)encodeWithCoder:(NSCoder *)aCoder;
- (id)initWithCoder:(NSCoder *)aDecoder;

@end

For example, now we have a customObject:

@interface CustomObject : NSObject  {
    NSString*   mStringValue;
    int         mIntValue;
    BOOL        mBOOLValue;
}

@property (non-atomic, retain) NSString *stringValue;
@property (nonatomic, assign) int       intValue;
@property (nonatomic, assign) BOOL      boolValue;

@end

It conforms to NSCoding, so we need to implement those two methods in the .m file

#define kEncodeKeyStringValue   @"kEncodeKeyStringValue" 
#define kEncodeKeyIntValue      @"kEncodeKeyIntValue" 
#define kEncodeKeyBOOLValue     @"kEncodeKeyBOOLValue" 

#pragma mark - NSCoding
- (void)encodeWithCoder:(NSCoder *)aCoder {
    [aCoder encodeObject:self.stringValue   forKey:kEncodeKeyStringValue];
    [aCoder encodeInt:self.intValue         forKey:kEncodeKeyIntValue];
    [aCoder encodeBool:self.boolValue       forKey:kEncodeKeyBOOLValue];
}

- (id)initWithCoder:(NSCoder *)aDecoder {
    if ((self = [super init]))
    {
        self.stringValue = [aDecoder decodeObjectForKey:kEncodeKeyStringValue];
        self.intValue    = [aDecoder decodeIntForKey:kEncodeKeyIntValue];
        self.boolValue   = [aDecoder decodeBoolForKey:kEncodeKeyBOOLValue];
    }
    return self;
}

Now, we can save and load a customObject like:

- (IBAction)saveObjectPlain:(id)sender {

    printf("==========================================\n");
    printf("saveObjectPlain===========================\n");
    printf("==========================================\n");

    //< Create and Save the Object
    {
        CustomObject *obj = [[[CustomObject alloc] init] autorelease];
        obj.stringValue = @"The String Value";
        obj.intValue    = 12345;
        obj.boolValue   = YES;
        [NSKeyedArchiver archiveRootObject:obj toFile:[self tempFilePath]];
        printf("Save: \n %s \n", [[obj description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    //< Load and Print the Object
    {
        CustomObject *obj = [NSKeyedUnarchiver unarchiveObjectWithFile:[self tempFilePath]];
        printf("Load: \n %s \n", [[obj description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    printf("==========================================\n");
}

Quite easy, right?

How about to save/load an array of our CustomObjects?
Since NSArray conforms to protocol NSCoding, it is quite straightforward.

- (IBAction)saveObjectsInArray:(id)sender {

    printf("==========================================\n");
    printf("saveObjectsInArray========================\n");
    printf("==========================================\n");
    
    //< Create Two Keys and Save the Object
    {
        CustomObject *obj1 = [[[CustomObject alloc] init] autorelease];
        obj1.stringValue = @"The String Value 1";
        obj1.intValue    = 12345;
        obj1.boolValue   = YES;
        
        CustomObject *obj2 = [[[CustomObject alloc] init] autorelease];
        obj2.stringValue = @"The String Value 2";
        obj2.intValue    = 54321;
        obj2.boolValue   = NO;
        
        NSArray *array = [NSArray arrayWithObjects:obj1, obj2, nil];
        [NSKeyedArchiver archiveRootObject:array toFile:[self tempFilePath]];

        printf("Save: \n %s \n ", [[array description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    //< Load and Print the Object
    {
        NSArray *array = [NSKeyedUnarchiver unarchiveObjectWithFile:[self tempFilePath]];
        printf("Load: \n %s \n ", [[array description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    printf("==========================================\n");
}

So, if there is a member variable of type NSArray in our CustomObject,
we need to make sure the array contains only objects conform to NSCoding.
Otherwise, an exception of unrecognized selector [.. encodeWithCoder:] will be raised.

One step further, how to save/load a NSDictionary has a CustomObject as the key and another CustomObject as the Object.

This is actually another problem. We need to make sure CustomObject can be used as a key in the NSDictionary.
According to the documentation, NSDictionary requires its key to conform to the protocol NSCopying.

@protocol NSCopying

- (id)copyWithZone:(NSZone *)zone;

@end

The key object needs to be copied to a specified piece of memory described by NSZone.
The implementation is like:

#pragma mark - NSCopying
- (id)copyWithZone:(NSZone *)zone {
    CustomObject *copy = [[CustomObject allocWithZone:zone] init];
    copy.stringValue = [[self.stringValue copy] autorelease];
    copy.intValue = self.intValue;
    copy.boolValue = self.boolValue;
    return copy;
}

We use allocWithZone to allocate a CustomObject on the indicated piece of memory.
Then copy member variables to the new instance.

The save/load procedure is like:

- (IBAction)saveObjectAsKey:(id)sender {

    printf("==========================================\n");
    printf("saveObjectAsKey===========================\n");
    printf("==========================================\n");
    
    //< Create Two Keys and Save the Object
    {
        CustomObject *keyObj = [[[CustomObject alloc] init] autorelease];
        keyObj.stringValue = @"The String Value Key";
        keyObj.intValue    = 12345;
        keyObj.boolValue   = YES;

        CustomObject *valObj = [[[CustomObject alloc] init] autorelease];
        valObj.stringValue = @"The String Value Value";
        valObj.intValue    = 54321;
        valObj.boolValue   = NO;
        
        NSDictionary *dict = [NSDictionary dictionaryWithObject:valObj forKey:keyObj];
        [NSKeyedArchiver archiveRootObject:dict toFile:[self tempFilePath]];

        printf("Save: \n %s \n", [[dict description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    //< Load and Print the Object
    {
        NSDictionary *dict = [NSKeyedUnarchiver unarchiveObjectWithFile:[self tempFilePath]];
        printf("Load: \n %s \n", [[dict description] cStringUsingEncoding:NSUTF8StringEncoding]);
    }
    
    printf("==========================================\n");
}

The whole demo can be found here:
Demo CustomObject NSCoding

Hope it helps! :]

----------------------------------------------------
When we were learning new things, we searched for answers in Google, from Stackoverflow and followed well-written tutorials. When other one faces problems we met and solved, we are glad to answer their questions, post tutorial for the related topics. This is called the spirit of Community. I love it.

[Demo]OmniGridView

OmniGridView Code (GitHub)

OmniGridView

This is a half-finished grid view for iOS.
Cause Apple’s UITableView only allows us to add and reuse vertical cells, we need to write our own
scroll view when we want to have horizontal cells.

HOW TO USE

Drag the OmniGridView Folder to your project. Use the OmniGridView like any other UIView.
Assign a delegate to OmniGridView and implement the OmniGridViewDelegate.

Please refer to the OmniGridTestViewController.m to see the information you need to provide.

这是一个半成品的Grid View。 因为Apple的UITableView只提供垂直滚动的表格,我们可能会需要水平滚动的表格。
这里提供一个全方向滚动包含单元格重用机制的Grid View。

如何使用

把OmniGridView文件夹加入你的工程。就像使用UIView一样使用OmniGridView。
设置OminiGridView的delegate实现OmniGridViewDelegate。

请参考OmniGridTestViewController.m的实现.

#pragma mark OmniGridViewDelegate

@implementation OmniGridTestViewController (Private)

- (OmniGridCell *)gridCellAt:(OmniGridLoc *)gridLoc inGridView:(OmniGridView *)gridView {
    OmniGridCell *gridCell = [gridView dequeueReusableGridCell];
    if (!gridCell)
    {
        gridCell = [[OmniGridCell alloc] init];
        gridCell.layer.borderColor = [UIColor blackColor].CGColor;
        gridCell.layer.borderWidth = 1.0f;
        gridCell.textLabel.textAlignment = UITextAlignmentCenter;
    }
    
    gridCell.textLabel.text = [NSString stringWithFormat:@"(%d,%d)", gridLoc.row, gridLoc.col];
    
    return gridCell;
}

- (float)gridCellHeightInGridView:(OmniGridView *)gridView {
    return 200.0f;
}

- (float)gridCellWidthInGridView:(OmniGridView *)gridView {
    return 200.0f;
}

- (int)numberOfGridCellsInRow:(int)row inGridView:(OmniGridView *)gridView {
    return 12;
}

- (int)numberOfRowsInGridView:(OmniGridView *)gridView {
    return 12;
}

@end