in Article, Blog Posts, Computer Vision (机器视觉), Solution

[OpenCV] detectMultiScale: output detection score

OpenCV provides quite decent implementation of the Viola-Jones Face detector.

A quick example looks like this (OpenCV 2.4.5 tested):

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            cascade.detectMultiScale(img, objs, scale_factor, min_neighbors);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}
g++ -std=c++0x -I/usr/local/include `pkg-config --libs opencv` main.cc -o main

The detection results are as shown below:
result

For more serious user, it would be nice to have a detection result for each detected face.
The OpenCV provides a overloaded function designed for this usage which is lack of detailed documentation:

vector reject_levels;
vector level_weights;
cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors);

The reject_levels and level_weights will keep being empty until you write it like this (The whole file):

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            vector reject_levels;
            vector level_weights;
            cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors, 0, Size(), Size(), true);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
                putText(img_color, std::to_string(level_weights[n]),
                        Point(objs[n].x, objs[n].y), 1, 1, Scalar(0,0,255));
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}

However, this will give you a large number of detected rectangles:
result-org

This is because OpenCV skips the step of filtering out the overlapped small rectangles. I have no idea whether this is by design. But output likes this would not be helpful at least in my own case.

So we would need to make our own changes in the OpenCV's source code.
There are different ways to design detection score, such as
"In the OpenCV implementation, stage_sum is computed and compared against the i stage_threshold for each stage to accept/reject a candidate window. We define the detection score for a candidate window as K*stage_when_rejected + stage_sum_for_stage_when_rejected. If a window is accepted by the cascade, we just K*last_stage + stage_sum_for_last_stage. Choosing K as a large value e.g., 1000, we ensure that windows rejected at stage i have higher score than those rejected at stage i-1." from http://vis-www.cs.umass.edu/fddb/faq.html

Actually, I found a straightforward design of detection score works well in my own work. In the last stage of the face detector in OpenCV, detection rectangles are grouped into clustered to eliminated small overlapped rectangles while keeping the most potential rectangles. The number of final detected faces is at most same as the number of clusters. So we can simply use the number of rectangles grouped into the cluster as the detection score of the associated final rectangle, which may not be accurate but could work.

To make this change, in OpenCV-2.4.5, find the file modules/objdetect/src/cascadedetect.cpp (line 200)

// modules/objdetect/src/cascadedetect.cpp (line 200)
// int n1 = levelWeights ? rejectLevels[i] : rweights[i]; //< comment out this line
int n1 = rweights[i]; //< the change

We then modify the file main.cc accordingly:

// File: main.cc
#include 

using namespace cv;

int main(int argc, char **argv) {

    CascadeClassifier cascade;
    const float scale_factor(1.2f);
    const int min_neighbors(3);

    if (cascade.load("./lbpcascade_frontalface.xml")) {

        for (int i = 1; i < argc; i++) {

            Mat img = imread(argv[i], CV_LOAD_IMAGE_GRAYSCALE);
            equalizeHist(img, img);
            vector objs;
            vector reject_levels;
            vector level_weights;
            cascade.detectMultiScale(img, objs, reject_levels, level_weights, scale_factor, min_neighbors, 0, Size(), Size(), true);

            Mat img_color = imread(argv[i], CV_LOAD_IMAGE_COLOR);
            for (int n = 0; n < objs.size(); n++) {
                rectangle(img_color, objs[n], Scalar(255,0,0), 8);
                putText(img_color, std::to_string(reject_levels[n]),
                        Point(objs[n].x, objs[n].y), 1, 1, Scalar(0,0,255));
            }
            imshow("VJ Face Detector", img_color);
            waitKey(0);
        }
    }

    return 0;
}

And we can have the detection scores like this:
result-final

Write a Comment

Comment

17 Comments

  1. Hi Haoxiang, should this be possible by using the below functions without a hack in Opencv. I am using OPencv 3.1.0:

    std::vector faceDetected;
    std::vector numFacesCollated;
    faceDetector.detectMultiScale(gray, faceDetected, numFacesCollated, scaleFactor, minNeighbors, flags, minFaceSize);
    std::vector numDetections;
    cv::groupRectangles(faceDetected, numDetections, 1, GROUP_EPS);

    for (int i = 0; i < faceDetected.size(); i++) {
    cv::Rect rect = faceDetected.at(i);
    cv::rectangle(frame, rect, cv::Scalar(0, 0, 255), 1);
    cv::putText(frame, std::to_string(numDetections[i]),
    cv::Point(rect.x, rect.y), 1, 1, cv::Scalar(0, 0, 255));
    }

    • These are my global variables:

      double scaleFactor = 1.05;
      int minNeighbors = 2;
      int flags = 0;
      cv::Size minFaceSize(30, 30);
      double GROUP_EPS = 0.2;

      • Hi Saurabh, I don’t know if they changed anything in OpenCV 3.1. Your code looks good to me.

  2. 我看了FDDB给出的关于viola-jones的ROC曲线,总感觉用他们的计算方式把V&J的效果弄得太差了。实际上,opencv2.4有一套自己的roc计算方法。巧的是,他们计算分数的方式和你在文中所说的不谋而合。https://docs.google.com/document/d/14r34Pd51lKZNlfJIQVRS_3kbow1OcJBKV7wTRRAW5Vg/preview 这个手册中的第七页有详细的介绍。

    • Hi Andy,
      Though I can’t understand you post. Languages – wise, I think we are facing the same issue. I am also trying to follow the FDDB protocol and getting the Opencv VJ ROC curve. It seems like using detectmultiscale has an overload function detectmultiscale3 which was design to do so but still not clear how and of possible at all as rejectLevels are set to 1 when min Neighborhood are zero.
      Do you have some insight on that issue?
      Thanks,
      ZRi

      • Hi ZR,

        I did’t evaluate my work on FDDB dataset, so I am afraid I cannot tell you how to implement the FDDB evaluation for OpenCV VJ. But there are two information I can share with you. First, in my work, I used detectMultiScale() setting groupThreshold to 0, then used groupRectangles() setting groupThreshold to 1 and got output boxes’ weights(neighborhood rectangles). Took weights as score, I drew a ROC curve. Second, if you want post OpenCV VJ ROC curve in your paper work or something, I think the best way is to copy the OpenCV VJ ROC data on FDDB official website, http://vis-www.cs.umass.edu/fddb/rocCurves/ViolaJonesScore_n0_DiscROC.txt

        Good luck,

        Andy

  3. Hi Haoxiang,
    Are you familiar with a way of getting the confidence score for each rectangle in the case of zero min neighbors. I want to use it for the ROC curve as described in FDDB protocol.
    Currently rejectLevels are set to 1 in the case of zero min neighbors / threshold and weights are available though not clear what they are.

    Thanks,
    ZR

    • I think in this case, you can only rely on how well the feature matches the detection window. Not sure if there is an easy way to do that. You may want to use softcascade. I did a quick search, it seems OpenCV now has it. Softcsscade should give confidence score for every detection window.

  4. 您好,我按照您给的方法对opencv源代码进行了改动可是却并没有得到像您那样的结果,请问您有什么建议么?我目前是想用FDDB对opencv人脸检测的算法进行评估,可是一直弄不明白detection score应该如何输出。
    谢谢~

      • 是的~我也不是很明白为什么,我用的版本是opencv2.4.10应该不会有很大区别的,可是并不能做到像您后面所给出的那样的结果,还是会有很多矩形,同时影响检测结果,2.4.5版本我下载不到了,您那里有的话可以发给我试一试么~太感谢了~我还是比较想把FDDB的结果做出来~谢谢您~

  5. Hey ZR,
    great tutorial! I am using OpenCV 3.1 and when I compile the first version with out the scores everything looks fine. But when I use the function which takes the int and double vector, I only one face, which is a wrong detection.
    Did you tried it also with OpenCV3.1?

Webmentions

  • How to get detection score from OpenCV cascade classifier | chooru.code 2016/07/07

    […] You can also hack the code a bit to get other types of scores with each detected location as demonstrated in this post. […]