Computer Vision Coding Example: Swift Metal

CMSR ML Studio generated program source codes are very easy to integrate into your applocations. This page will show how to code to use Computer Vision functions. Depending on modeling types, calling parameters are different.

Model Code Generation and Compilation

From Xcode, create a new metal file "CNNForward.metal" and copy the content of the downloaded "CNNForward.metal" file. And Create "MetalSwiftModel.swift" and copy the content of the downloaded "MetalSwiftModel.swift" file.

From CMSR ML Studio, perform the followings to generate model codes;

1. Open your computer vision model.
2. From "Modeling" menu, select "Export model as program code".
3. Select "Language" as "GPU: Swift/Metal" and press the "Copy" button. This will generate swift model program to your system clipboard. Create a swift file into your Xcode project and copy clipboard content and save, say, "CMSRModel.swift".

Copy model parameter file into locations where your main program can access. And modify access paths in your main program. To create model parameter file, from "Modeling" menu, select "Export model as file (MyDataSay, BI Server, CNN, ...etc.)".

CNN/FCN Classification Example

The following code shows how CNN/FCN classification is called. Note that CMSRModel is the model class that CMSR Studio generates.

import Swift
import Foundation

let model: CMSRModel = CMSRModel();

let imagefilepath : String = "/Users/cho/cnnimages.rgb";
let modelfilepath : String = "/Users/cho/modelfile.cnn";

let height : Int = 64;
let width : Int = 64;
var IMAGE: [[[Int]]] = Array(repeating: Array(repeating: Array(repeating: 0, count: 3), count: width), count: height)

var outclass: Int;

let blackandwhite: Bool  = false;
let r0g1b2 : Bool = true;
let maxOutCount = 4;
var outClassIndex: [Int] = [0, 0, 0, 0, 0, 0]
var outClassProbability: [Float] = [0, 0, 0, 0, 0, 0]
  
// initialize resources;
let _ = model.initializeGpu()
let _ = model.initializeModel(modelfilepath);

// the followings can be repeated as many time as needed;
model.fillRgbArray(imagefilepath, &IMAGE, height, width); // can get from camera.
outclass =  model.evaluate(
           maxOutCount,
           &outClassIndex,
           &outClassProbability,
           blackandwhite,
           r0g1b2,
           &IMAGE
       );
for i in 0..<maxOutCount {
   print(" \(i), \(outClassIndex[i]), \(outClassProbability[i])");
}

M-CNN Multi-value Output Regression Example

The following code shows how M-CNN multivalue output regression is called. Note that CMSRModel is the model class that CMSR Studio generates.

import Swift
import Foundation

let model: CMSRModel = CMSRModel();

let imagefilepath : String = "/Users/cho/mcnnimages.rgb";
let modelfilepath : String = "/Users/cho/modelfile.mcnn";

let height : Int = 70;
let width : Int = 70;
var IMAGE: [[[Int]]] = Array(repeating: Array(repeating: Array(repeating: 0, count: 3), count: width), count: height)

var outclass: Int;

let blackandwhite: Bool  = false;
let r0g1b2 : Bool = true;
var outValues: [Float] = [0, 0, 0, 0, 0, 0]

// initialize resources;  
let _ = model.initializeGpu();
let _ = model.initializeModel(modelfilepath);

// the followings can be repeated as many times as needed;
model.fillRgbArray(imagefilepath, &IMAGE, height, width); // can get from camera.
model.evaluate(
                &outValues,
                blackandwhite,
                r0g1b2,
                &IMAGE
            );
for i in 0..<2 {
    print(" \(i) \(outValues[i])");
}

OD-CNN Object Detection Example

The following code shows how OD-CNN object detection is called. Note that CMSRModel is the model class that CMSR Studio generates. The X/Y coordinates are the center locations of detected object bounding boxes. Output width and height are relative size where whole size is 1.0.

import Swift
import Foundation

let model: CMSRModel = CMSRModel();

let imagefilepath: String = "/Users/cho/odcnnimages.rgb";
let modelfilepath: String = "/Users/cho/modelfile.odcnn";

let height: Int = 245;
let width: Int  = 245;
var IMAGE: [[[Int]]] = Array(repeating: Array(repeating: Array(repeating: 0, count: 3), count: width), count: height)

var outclass: Int;
let areasizefilter: Int = 1;

let blackandwhite: Bool  = false;
let r0g1b2 : Bool = true;

let maxOutCount: Int = 5;
var outClassIndex: [Int] = [0, 0, 0, 0, 0, 0]
var outClassProbability: [Float] = [0, 0, 0, 0, 0, 0]
var outX: [Float] = [0, 0, 0, 0, 0, 0]
var outY: [Float] = [0, 0, 0, 0, 0, 0]
var outWidth: [Float] = [0, 0, 0, 0, 0, 0]
var outHeight: [Float] = [0, 0, 0, 0, 0, 0]
  
// initialize resources;
let _ = model.initializeGpu();
let _ = model.initializeModel(modelfilepath);

// the followings can be repeated as many time as needed;
model.fillRgbArray(imagefilepath, &IMAGE, height, width); // can get from camera.
outclass =  model.evaluate(
        maxOutCount,
       &outClassIndex,
       &outClassProbability,
       &outX,
       &outY,
       &outWidth,
       &outHeight,
       blackandwhite,
       r0g1b2,
       &IMAGE
       // areasizefilter
       );
for i in 0..<outclass {
    print(" \(i), \(outClassIndex[i]), \(outClassProbability[i]), \(outX[i]), \(outY[i]), \(outWidth[i]), \(outHeight[i])");
}

T-CNN Similarity Regression/Face Recognition Example

The following code shows how T-CNN similarity regression is called. Note that CMSRModel is the model class that CMSR Studio generates.

import Swift
import Foundation

let model: CMSRModel = CMSRModel();

let imagefilepath1: String = "/Users/cho/tcnnimages1.rgb";
let imagefilepath2: String = "/Users/cho/tcnnimages2.rgb";
let modelfilepath: String = "/Users/cho/modelfile.tcnn";

let height: Int = 70;
let width: Int  = 70;

var IMAGE1: [[[Int]]] = Array(repeating: Array(repeating: Array(repeating: 0, count: 3), count: width), count: height)
var IMAGE2: [[[Int]]] = Array(repeating: Array(repeating: Array(repeating: 0, count: 3), count: width), count: height)

let blackandwhite: Bool = false;
let r0g1b2 : Bool = true;
let outvalue: Float;

// initialize resources;
let _ = model.initializeGpu();
let _ = model.initializeModel(modelfilepath);

// the followings can be repeated as many times as needed;
model.fillRgbArray(imagefilepath1, &IMAGE1, height, width); // can get from camera.
model.fillRgbArray(imagefilepath2, &IMAGE2, height, width); // can get from camera.
outvalue =  model.evaluate(
           blackandwhite,
           r0g1b2,
           &IMAGE1,
           &IMAGE2
       );
print("Out value: \(outvalue)");