Video Effects SDK - iOS Swift Package

Add real-time AI video enhancement that makes video meeting experience more effective and comfortable to your application in a few hours.

This repository contains the Swift Package version of Objective-C iOS xcframeworks versions of Video Effects SDK that you can integrate into your project/product. This package and xcframework can be included in Swift iOS project.

Also, there is the Sample Xcode project with Video Effects SDK integration, so you can just build and run it to see SDK in action.

Obtaining Effects SDK License

To receive a Effects SDK license please fill in the contact form on effectssdk.com website.

Technical Details

• SDK available for iOS 13 and newer.
• Frames preprocessing/postprocessing could be run on CPU or GPU.
• ML inference could be run only on CPU.
• 4 Virtual Background presets (Quality, Balanced, Speed and Lightning). See pipeline configuration.

Features

• Virtual backgrounds (put image as a background) - implemented
• Background blur - implemented
• Beautification/Touch up my appearance - implemented
• Auto framing/Smart Zoom - implemented
• Auto color correction - implemented
• Color grading - implemented

Usage details

The entrypoint of the SDK is the instance of SDKFactory. Using an SDKFactory instance you will be able to prepare frames for processing and configure the pipeline of processing (enable transparency, blur, replace background etc).

Usage

Add package dependency into your project. See Adding package dependencies to your app

Import Video Effects SDK into your code.

import TSVB

Preparation:

• Create an instance of SDKFactory.
• Authorize the instance of SDKFactory by using auth(customerID:) method.
• Create an instance of FrameFactory by using newFrameFactory() method of SDKFactory.
• Create an instance of Pipeline by using newPipeline() method of SDKFactory.
• Enable background blur using enableBlurBackground(power:) method or background replacement using enableReplaceBackground(controller:) method of Pipeline.
• When the background replacement is enabled you can pass image which will be used as a background: ReplacementController.background

Frame processing:

• Put your frame to Frame using newFrame(format:data:bytesPerLine:width:height:makeCopy:) method of FrameFactory.
• Process it through process(frame:error:) method of Pipeline.
• If your frame is CVPixelBuffer, you just can use process(pixelBuffer:metalCompatible:error:) method of Pipeline.

Performance tip: Prefer process(pixelBuffer:metalCompatible:error:) method and pass metal compatible CVPixelBuffer for better performance when Metal pipeline is active.

Use separate Pipeline instances per video stream.

init()
{
    super.init()
    
    let sdkFactory = SDKFactory()
    let authResult = try await sdkFactory.auth(customerID: "CUSTOMER_ID")
    if (authResult.status == .active) { 
        frameFactory = sdkFactory.newFrameFactory();
        pipeline = sdkFactory.newPipeline();
        pipeline.enableReplaceBackground(&backgroundController);
    }
    
    return self;
}

Class Reference

SDKFactory

func auth(customerID:String) async throws -> AuthResult

Performs authorization of the instance. SDKFactory can not be used until it's authorized. Method performs https request to obtain license for customerID. Required internet connection. Returns AuthResult

Parameters:

• customerID:String - Your unique customer id. See Obtaining Effects SDK License

Example:

let authResult = try await sdkFactory.auth(customerID:"CUSTOMER_ID")
if (authResult.status == .active) {
    startVideoProcessing()
}

"CUSTOMER_ID" should be replaced by your id.

func auth(key:String) -> AuthResult

Parameters:

• key:String - Unique client's secret key. DO NOT reveal it. See Obtaining Effects SDK License

Offline authorization with a secret key. Returns AuthResult

Authorizes an instance of SDKFactory similar to auth(customerID:), but performs license verification without web requests.
Internet connection is not required.

Usage example

let result = sdkFactory.auth(key: "Your_secret_key")
guard (result.status == .active) else {
    handleAuthorizationFailure(result: result)
    return
}

sdkFrameFactory = factory.newFrameFactory()
sdkPipeline = factory.newPipeline()

func newFrameFactory() -> FrameFactory?

Creates new instance of FrameFactory.

func newPipeline() -> Pipeline?

Creates new instance of Pipeline.

AuthResult

var status:AuthStatus { get }

Holds result of authorization. See AuthStatus.

enum FrameFormat

• rba32 - RGBA format with 8 bits per channel (32 bits per pixel).
• bgra32 - BGRA format with 8 bits per channel (32 bits per pixel).

enum Lock - Determines access to locked memory

• read
• write
• readWrite

FrameFactory

func newFrame(format:FrameFormat, data:UnsafeMutableRawPointer, bytesPerLine:Uint32, width:Uint32, height:Uint32, makeCopy:Bool) -> Frame?

Creates Frame from raw RGBA or BGRA data.

Parameters:

• format:(FrameFormat) - format of raw data.
• data:UnsafeMutableRawPointer - pointer at raw data.
• bytesPerLine:Uint32 - number of bytes per line of frame.
• width:Uint32 - number of pixels in horizontal direction.
• height:Uint32 - number of pixels in vertical direction.
• makeCopy:Bool - if set to true - data will be copied, otherwise Frame will keep the pointer to data (DON'T release the data while it's processing).

func image(withContentOfFile:String) -> Frame?;

Frame

var width:Uint32 { get }

Returns number of pixels in horizontal direction.

var height:Uint32 { get }

Returns number of pixels in vertical direction.

var format:FrameFormat { get }

Returns format of frame.

func lock:(_ lock: Lock) -> LockedFrameData?

Parameters:

• lock:Lock - Determines access to the memory. See Lock Gets access to memory of the frame. Returns LockedFrameData protocol which provides ability to get pointers to internal data of Frame (DON’T use the Frame until LockedFrameData wasn’t released).

func toCVPixelBuffer() -> CVPixelBuffer?;

Converts internal storage to CVPixelBuffer and returns that reference if successful, otherwise returns nil. If internal storage is CVPixelBuffer already then returns that reference. Frame must not be used after toCVPixelBuffer() called. When Metal pipeline is active the result of process(_ frame:error:) and process(pixelBuffer:metalCompatible:error:) is metal compatible.

LockedFrameData

Keeps access to the data inside Frame and returns pointers to that data. If it was obtained with lock(.write) or lock(.readWrite): , then the changes will be applied after LockedFrameData will be released. If it was obtained with lock(.read) then data must not be changed.

func dataPointer(ofPlanar:Int32) -> UnsafeMutableRawPointer?

Returns pointer to planar data. If the Frame created by newFrame(format:data:bytesPerLine:width:height:makeCopy:) where makeCopy was false then returns same pointer that was passed Parameters:

• ofPlanar:Int32 - depends on frame format. For FrameFormat.rgba32 or FrameFormat.bgra32 always should be used 0. For FrameFormat.nv12 - 0 returns pointer to Y component, 1 returns pointer to UV component.

func bytesPerLine(ofPlanar:Int32) -> Uint32;

Returns number of bytes per line. Parameters:

• ofPlanar:Int32 - see dataPointer(ofPlanar:).

PipelineErrorCode - error codes

• ok - success
• invalidArgument - one or more arguments are incorrect.
• noFeaturesEnabled - processing pipeline is not configured.
• engineInitializationError - can’t initialize OpenVINO, the hardware/software is not supported.
• resourceAllocationError - not enough memory, disc space etc.

Pipeline

Configuration of effects and frame processor. Use separate instances for each video stream.

func setConfiguration(_ configuration: PipelineConfiguration) -> PipelineError

Parameters:

• configuration: PipelineConfiguration Configures pipeline, determines what to use for image processing (see PipelineConfiguration). This method is optional.

func copyConfiguration() -> PipelineConfiguration?

Returns a copy of current configuration of pipeline. Can be used to get an instance of PipelineConfiguration.

func copyDefaultConfiguration() -> PipelineConfiguration?

Returns a copy of the default configuration. Can be used to get an instance of PipelineConfiguration.

func enableBlurBackground(power: Float) -> PipelineError

Enables background blur. Parameters:

• power:Float - power of blur from 0 to 1.

func disableBlurBackground()

Disables background blur.

func enableReplaceBackground(_ controller:UnsafeMutablePointer<ReplacementController?>?) -> PipelineError

Enables background replacement, default background is transparent. The custom image for the background could be set using the property ReplacementController.background. Parameters:

• controller:UnsafeMutablePointer<ReplacementController?>? - Pointer at variable to store an instance of ReplacementController. Can be nil. Example

var controller : ReplacementController? = nil
pipeline.enableReplaceBackground(&controller)

func disableReplaceBackground()

Disables background replacement.

-func enableDenoiseBackground() -> PipelineError

Enables video denoising. By default, denoises the background only; to denoise the foreground, set denoiseWithFace to YES.

-func disableDenoiseBackground()

Disables denoising.

var denoiseLevel: Float { get set };

Power of denoising: higher number = more visible effect. Value from 0 to 1.

var denoiseWithFace: Bool { get set }

If YES, the pipeline denoises the background and foreground of the video. Otherwise, background only. Default is NO.

func enabledBeautification() -> PipelineError

Enables face beautification.

func disableBeautification()

Disables face beautification.

var beautificationLevel: Float { get set }

Could be from 0 to 1. Higher number -> more visible effect of beautification.

func enabledColorCorrection() -> PipelineError

Enables color correction. Improves colors with the help of ML. Disables another enabled color correction effect. Note: Preparation starts asynchronously after a frame process, the effect may be delayed.

func enabledColorCorrection(withReference:Frame) -> PipelineError

Enables color grading. Generates a color palette from reference and apply it to the video. Parameters:

• withReference:Frame - The reference to generate a color palette. If enabled, generates a new color palette with referenceFrame. Disables another enabled color correction effect.

func enableColorCorrection(withLutFile:String) -> PipelineError

Enables color filtering with a Lookup Table (Lut). Parameters:

• withLutFile:String - path to .cube file. Supports only 3D Lut with maximum size 256 (256x256x256). If enabled, switches Lut. Disables another enabled color correction effect.

func disableColorCorrection()

Disables color correction.

func enabledSmartZoom() -> PipelineError

Enables smart zoom.

Smart Zoom crops around the face.

func disableSmartZoom()

Disables smart zoom.

var smartZoomLevel: Float { get set }

could be from 0 to 1. Defines how much area should be filled by a face. Higher number -> more area.

func enableSharpening() -> PipelineError

Enables sharpening effect. Sharpening makes the video look better by enhancing its clarity. It reduces blurriness in the video.

func disableSharpening()

Disables sharpening effect.

var sharpeningPower: Float { get set }

Current power of the sharpening effect. Power could be from 0 to 1. Higher number -> sharper result.

func process(_ frame: Frame, error: UnsafeMutablePointer<PipelineError>?) -> Frame?;

Returns processed frame the same format with input (with all effects applied). In case of error, returns nil. Parameters:

• frame: Frame - frame for processing.
• error: UnsafeMutablePointer? - nil or pointer at variable to store error.

func process(pixelBuffer: CVPixelBuffer, metalCompatible: bool, error: UnsafeMutablePointer<PipelineError>?) -> Frame?

Same as process:error: but expects CVPixelBuffer as an argument.

Parameters:

• pixelBuffer: CVPixelBuffer - CVPixelBuffer for processing.
• metalCompatible: bool - Set true if CVPixelBuffer is compatible with metal.
• error: UnsafeMutablePointer? - nil or pointer at variable to store error.

Recommended to use metal compatible CVPixelBuffer. The argument has not effect if CPU pipeline. If you does not know pixelBuffer is metal compatible or not then set metalCompatible to false. CVPixelBuffer received within AVCaptureVideoDataOutputSampleBufferDelegate.captureOutput(_:didOutput:from:) from AVCaptureSession is metal compatible.

For manual creating, to create metal compatible CVPixelBuffer, set kCVPixelBufferMetalCompatibilityKey attribute.

Supported formats:

• kCVPixelFormatType_32BGRA
• kCVPixelFormatType_32RGBA
• kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
• kCVPixelFormatType_420YpCbCr8BiPlanarFullRange

ReplacementController

var background: Frame? { get set }

Holds custom image for background replacement. If nil then processing replaces background by transparency. To reset the background set nil.

PipelineConfiguration

var backend : Backend { get set }

Determines pipeline that performs image processing.

var segmentationPreset: SegmentationPreset { get set }

Set the segmentation mode. Segmentation mode allow to choose combination of quality and speed of segmentation. Balanced mode is enabled by default.

var isSegmentationOnNeuralEngineEnabled: bool { get set }

Determines Apple Neural Engine acceleration is enabled. Has effect on system with Apple Neural Engine support. If host system does not support Neural Engine then the setting is ignored. Supported on Mac with M1 chips and newer. And iPhones of 2018 and newer.

enum Backend

• CPU - CPU-based pipeline.
• GPU - GPU-based pipeline.

enum SegmentationPreset

• quality - Quality is preferred.
• balanced - Balanced quality and speed.
• speed - Speed is preferred.
• lightning - Speed is prioritized.

enum AuthStatus

• active - Authorization is succeeded and the license is active. SDK can be used to enhance your video.
• inactive - Authorization is failed because the license is deactivated or no such license.
• expired - Authorization is failed because the license is expired. Contact us to update it.