Structure-measure: A New Way to Evaluate Foreground Maps

 

Abstract

Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the filed of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several widely-used measures such as Area Under the Curve (AUC), Average Precision (AP) and the recently proposed Fωβ(Fbw) have been utilized to evaluate the similarity between a non-binary saliency map (SM) and a ground-truth (GT) map. These measures are based on pixel-wise errors and often ignore the structural similarities. Behavioral vision studies, however, have shown that the human visual system is highly sensitive to structures in scenes. Here, we propose a novel, efficient, and easy to calculate measure known an structural similarity measure (Structure-measure) to evaluate non-binary foreground maps. Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a SM and a GT map. We demonstrate superiority of our measure over existing ones using 5 meta-measures on 5 benchmark datasets.

 

Publication

Deng-Ping FanMing-Ming ChengYunLiu, TaoLi,  Ali Borji

Structure-measure: A new way to evaluate foreground maps IEEE ICCV, 2017  

( Spotlight, Accept rate: 2.61% [56/2143] )

[Project Page][bib][official version][中文版pdf][ICCV slide][ICCV Poster]

[Valse ppt][Valse2018中文版海报][Video][Matlab code][C++ code]

 

Motivation

  • Region perspectives: Although it is difficult to describe the object structure of a foreground map, we notice that the entire structure of an object can be well illustrated by combining structures of constituent object-parts (regions).

 

  • Object perspectives: In the high-quality SMs, the foreground region of the maps contrast sharply with the background regions and these regions usually have approximately uniform distributions.

 

Current Evaluation

  • Current evaluation measures (AP, AUC, Fbw) are based on pixel-wise manner and consider each pixel as independent. Hence, they all ignore the structure of the foreground maps, thus result the same score.

 

S-measure FrameWork

 

Evaluate examples

 

(a) Image                (b) GT          (c) state-of-art map  (d) generic map

Meta-measure 2: Generic vs. state-of-the-art. A evaluation measure should give the FM which generated by the state-of-the-art method (c) a higher score than the generic map (d) that do not consider the content of the image. Unfortunately, all of the current evaluation measure give the map in (d) a higher score than (c). Only our measure correctly ranks the state-of-the-art result higher.

Quantitative Evaluation

 

 

  • Table 1. Quantitative comparison with current measures on 3 meta-Measures. The best result is highlighted in blue. MM:meta-Measure.

3018total visits,45visits today

6 thoughts on “Structure-measure: A New Way to Evaluate Foreground Maps

    • April 16, 2018 at 7:39 am
      Permalink

      Thanks

      Reply
  • May 16, 2018 at 2:24 pm
    Permalink

    您好!我想问一个简单的问题,foreground maps是什么?是不是就是显著图?百度了好久都没找到有效的答案,麻烦老师解惑!

    Reply
    • May 28, 2018 at 1:39 pm
      Permalink

      foreground maps指的是前景图,显著图可以是前景图中的一种情况,显著性物体检测可以直接使用这个公式进行评价。

      Reply
  • July 19, 2018 at 2:21 pm
    Permalink

    以后会更新python版本吗?

    Reply
    • July 25, 2018 at 9:26 am
      Permalink

      过段时间有空我更新一个吧,这个方法很简单,你可以根据matlab版本自己写一个。

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *