目錄:

• 1. 圖片預(yù)處理
• 2. RPN網(wǎng)絡(luò)預(yù)測(cè)
• 3. RPN to ROIs
• 4. Classifier 網(wǎng)絡(luò)預(yù)測(cè)
• 5. Classifier網(wǎng)絡(luò)輸出對(duì) ROIs過(guò)濾與修正
• 6. NMS (非最大值抑制)
• 7. 坐標(biāo)轉(zhuǎn)換為原始圖片維度

一: 輸入輸出

輸入:

• cls: RPN網(wǎng)絡(luò)的輸出, shape為 [1, 37, 50, 49]
• reg: RPN網(wǎng)絡(luò)的輸出, shape為 [1, 37, 50, 196]

輸出:

• R: ROIs, shape為 [300, 4]

二: 流程

• 遍歷所有的anchor, 7個(gè)anchorSize, 7個(gè)anchorRatio, 共 49個(gè)anchor
• 計(jì)算每個(gè)anchor在每個(gè)feature map pixel上的矩形 (x, y, w, h)
• 使用RPN網(wǎng)絡(luò)的reg輸出來(lái)修正每個(gè)anchor的矩形
• 限制anchor矩形范圍值
• 非最大值抑制，找出300個(gè)ROIs

三: code by code

RPN的輸出除以std_scaling = 4
原因是: 訓(xùn)練時(shí)最終計(jì)算出regr后，乘了個(gè)std_scaling

regr.divi(std_scaling);

獲取行與列的size.

int rows = (int)cls.shape()[1];
int cols = (int)cls.shape()[2];

每個(gè)anchor(共49個(gè)) 經(jīng)過(guò)RPN 的輸出 regr 修正后的ROIs

A.shape = [4, 37, 50, 49]
INDArray A = Nd4j.zeros(4, cls.shape()[1], cls.shape()[2], cls.shape()[3]);

遍歷每個(gè)anchor
一共49個(gè)layer.

int curr_layer = 0;
 
for (int i = 0; i < anchor_sizes.length; i++)
{
    for (int j = 0; j < anchor_ratios.length; j++)
    {
        int anchor_size = anchor_sizes[i];
        int[] anchor_ratio = anchor_ratios[j];

計(jì)算出每個(gè)anchor在feature map 維度上的寬和高.
anchor_x: anchor 的寬
anchor_y: anchor 的高
rpn_stride: 16 是VGG16模型 輸入圖片size 到 feature map size 的縮小比例.

int anchor_x = (anchor_size * anchor_ratio[0]) / rpn_stride;
int anchor_y = (anchor_size * anchor_ratio[1]) / rpn_stride;

獲取當(dāng)前anchor在RPN輸出regr上的回歸值.
current_regr 是當(dāng)前anchor的回歸值, shape = [37, 50, 4], 并將其permute成 [4, 37, 50]

INDArray current_regr = regr.get(NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.interval(4 * curr_layer, 4 * curr_layer + 4));
current_regr = current_regr.permute(2, 0, 1);

構(gòu)建MeshGrid, 用來(lái)生成anchor在每個(gè)pixel上的坐標(biāo).
X: 是x坐標(biāo)
0, 1, 2, ....., 47, 48, 49
共37列
Y: 是y坐標(biāo)
0,
1,
2,
.....
34,
35,
36
共50行.

INDArray[] meshgrid = Nd4j.meshgrid(Nd4j.arange(cols).castTo(DataType.INT), Nd4j.arange(rows).castTo(DataType.INT));
INDArray X = meshgrid[0];
INDArray Y = meshgrid[1];

計(jì)算出當(dāng)前anchor 在feature map 上的矩形 [x, y, w, h]

X.subi(anchor_x / 2);
Y.subi(anchor_y / 2);

// 設(shè)置每個(gè)anchor在所有像素點(diǎn)上的x, y, w, h

A.put(new INDArrayIndex[]{NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, X);
A.put(new INDArrayIndex[]{NDArrayIndex.point(1), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, Y);
A.put(new INDArrayIndex[]{NDArrayIndex.point(2), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, anchor_x);
A.put(new INDArrayIndex[]{NDArrayIndex.point(3), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, anchor_y);

使用RPN 的輸出regr 來(lái)修正每個(gè)anchor對(duì)應(yīng)的矩形.
該邏輯是與訓(xùn)練時(shí)生成RPN網(wǎng)絡(luò)標(biāo)注數(shù)據(jù)執(zhí)行了相反的操作.

INDArray applyRegr = apply_regr_np(A.get(NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)), current_regr);
A.put(new INDArrayIndex[]{NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, applyRegr);

將坐標(biāo)從[x, y, w, h] 轉(zhuǎn)換為 [x1, y1, x2, y2]
限制的anchor矩形坐標(biāo)
x1 >= 0
y1 >= 0
x2 <= cols - 1
y2 <= rows - 1

INDArrayIndex[] x_indices = new INDArrayIndex[]{NDArrayIndex.point(0), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] y_indices = new INDArrayIndex[]{NDArrayIndex.point(1), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] w_indices = new INDArrayIndex[]{NDArrayIndex.point(2), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArrayIndex[] h_indices = new INDArrayIndex[]{NDArrayIndex.point(3), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)};
INDArray x = A.get(x_indices[0], x_indices[1], x_indices[2], x_indices[3]);
INDArray y = A.get(y_indices[0], y_indices[1], y_indices[2], y_indices[3]);
INDArray w = A.get(w_indices[0], w_indices[1], w_indices[2], w_indices[3]);
INDArray h = A.get(h_indices[0], h_indices[1], h_indices[2], h_indices[3]);

限制寬度和高度最小值為1.
格式為: x, y, w, h

A.put(w_indices, Transforms.max(w, 1.0));
A.put(h_indices, Transforms.max(h, 1.0));

轉(zhuǎn)換為 x1, y1, x2, y2

A.put(w_indices, x.add(w));
A.put(h_indices, y.add(h));

將 [x1, y1], [x2, y2] 約束在 [0, cols - 1], [0, rows - 1] 范圍內(nèi).
w_indice/h_indice 其實(shí)表示的是 x2, y1 的 indices.

A.put(x_indices, Transforms.max(x, 0.0));
A.put(y_indices, Transforms.max(y, 0.0));
A.put(w_indices, Transforms.min(w, cols - 1));
A.put(h_indices, Transforms.min(h, rows - 1));
INDArray applyRegr = apply_regr_np(A.get(NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)), current_regr);
A.put(new INDArrayIndex[]{NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.all(), NDArrayIndex.point(curr_layer)}, applyRegr);

將 坐標(biāo)的shape (4, 37, 50, 49) 轉(zhuǎn)換為 (90650, 4)
將 概率的shape (1, 37, 50, 49) 轉(zhuǎn)換為 (90650,)

INDArray all_boxes = A.permute(0, 3, 1, 2).reshape(4, -1).permute(1, 0);
INDArray all_probs = cls.permute(0, 3, 1, 2).reshape(-1);

刪除坐標(biāo)中 （x1 > x2 || y1 > y2）的無(wú)效坐標(biāo).

INDArray x1 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(0));
INDArray y1 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(1));
INDArray x2 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(2));
INDArray y2 = all_boxes.get(NDArrayIndex.all(), NDArrayIndex.point(3));
 
INDArrayIndex validIdx = validIndex_restrain_x1x2y1y2(x1, x2, y1, y2);
 
INDArray valid_box = all_boxes.get(validIdx, NDArrayIndex.all());
INDArray valid_probs = all_probs.get(validIdx);

執(zhí)行NMS(非最大值抑制)
overlap_thresh 為 0.7
maxBoxes 為 300

return non_max_suppression_fast(valid_box, valid_probs, 0.7f, 300);