What's New / Spatial, 3D & RealityKit

What's new in ImmersiveMediaSupport

+15 NewiOS · macOS · visionOS

ImmersiveMediaSupport provides API for authoring and playback control of immersive media, including camera setup, shot framing, and metadata for spatial and 3D content. It includes presentation commands and a preview renderer.

15 new APIs, no deprecations or removals. Two new types: struct Overrides and struct ShotFlipCommand. ShotFlipCommand pairs with PresentationCommand.shotFlip and adds an isShotFlipped flag and a process method. SetCameraCommand gains a cameraOverrides property, an overrides property, and a new init. Also added: metadataItem, metadataTrack, and ImmersivePreviewRenderer.

New

15
var

cameraOverrides

NewiOSmacOS
public var cameraOverrides: SetCameraCommand.Overrides? { get }

The current camera params which would override static metadata to use when rendering playback for the PTS specified in the last call to the update method.

extension

ImmersivePreviewRenderer

NewmacOSvisionOS
extension ImmersivePreviewRenderer : Sendable
Declaration
extension ImmersivePreviewRenderer : Sendable {
}
extension

ImmersivePreviewRenderer.Eye

NewmacOSvisionOS
extension ImmersivePreviewRenderer.Eye : Equatable
Declaration
extension ImmersivePreviewRenderer.Eye : Equatable {
}
extension

ImmersivePreviewRenderer.Eye

NewmacOSvisionOS
extension ImmersivePreviewRenderer.Eye : Hashable
Declaration
extension ImmersivePreviewRenderer.Eye : Hashable {
}
var

isShotFlipped

NewiOSmacOS
public var isShotFlipped: Bool { get }

A Boolean value that indicates whether to vertically flip the video frame for the PTS specified in the last call to the update method.

func

metadataItem

NewiOSmacOS
public func metadataItem(for time: CMTime, segmentDuration: CMTime? = nil) throws -> AVMetadataItem?

Retrieves the metadata item that starts at the specified presentation timestamp. Should run on the same thread with PresentationDescriptorReader.

Parameters

time
The presentation timestamp of the desired metadata.
segmentDuration
Optional duration for splitting metadata items into segments.

ReturnsThe AVMetadataItem at the given timestamp, or nil if none exists.

func

metadataTrack

NewiOSmacOS
public func metadataTrack(segmentDuration: CMTime? = nil, end: CMTime? = nil) throws -> [AVMetadataItem]

Retrieves all metadata items, optionally segmented and filtered by end time. Should run on the same thread with PresentationDescriptorReader.

Parameters

segmentDuration
Optional duration for splitting metadata items into segments.
end
Optional end time to limit the range of returned metadata items. Required when the one of metadata item has no defined end time and segmentDuration is valid.

ReturnsAn array of AVMetadataItem objects.

func

process

NewmacOS
public func process(_ frame: ImmersiveVideoFrame, for eye: ImmersivePreviewRenderer.Eye = .left, rotation: Rotation3D = .identity, fieldOfView: Float = 90, presentationDescriptor: PresentationDescriptor, renderTexture: any MTLTexture) async

Performs the rendering of an immersive video frame.

This method is available on macOS and provides manual control over camera pose and projection parameters. It updates the camera pose based on the provided rotation, configures the projection matrix using the specified field of view, and processes presentation commands to determine the active camera and rendering parameters.

The rendering operation is asynchronous and completes when the frame has been submitted to the GPU. Access commandBuffer after this method returns to retrieve the command buffer for synchronization.

Parameters

frame
The ImmersiveVideoFrame to render.
eye
The eye of the user.
rotation
The rotation to apply to the ImmersiveVideoFrame.
fieldOfView
The field of view to be used when processing frame.
presentationDescriptor
The presentation descriptor for the commands to process on the frame.
renderTexture
The texture to render to.
func

process

NewvisionOS
public func process(_ frame: ImmersiveVideoFrame, presentationDescriptor: PresentationDescriptor, drawable: LayerRenderer.Drawable) async

Performs the rendering of an immersive video frame.

This method is available on visionOS and integrates with the CompositorServices framework to provide head-tracked stereoscopic rendering. It automatically processes both left and right eye views, applies device anchor transforms for head tracking.

The rendering operation is asynchronous and completes when the frame has been submitted to the GPU. Access commandBuffer after this method returns to retrieve the command buffer for synchronization.

Parameters

frame
The ImmersiveVideoFrame to render.
presentationDescriptor
The presentation descriptor for the commands to process on the frame.
drawable
The drawable to render to.
struct

ShotFlipCommand

NewiOSmacOS
public struct ShotFlipCommand

A command type to flip the video frames vertically during playback for the duration of the command.

Declaration
public struct ShotFlipCommand {

    /// A unique command ID for the immersive media file.
    public var id: Int { get }

    /// The time this command starts during playback.
    public var time: CMTime

    /// The duration of the command.
    public var duration: CMTime

    /// The offset from the start time of this command.
    public var offset: CMTime?

    /// Creates a command instance for a certain time, duration and offset.
    /// - Parameters:
    ///   - id: The unique ID of this command.
    ///   - time: The time for this command to start during playback.
    ///   - duration: The duration of this command during playback.
    ///   - offset: The offset from the start time of this command.
    public init(id: Int, time: CMTime, duration: CMTime, offset: CMTime? = nil)

    public init(from decoder: any Decoder) throws

    public func encode(to encoder: any Encoder) throws
}
extension

ShotFlipCommand

NewiOSmacOS
extension ShotFlipCommand : Sendable
Declaration
extension ShotFlipCommand : Sendable {
}
case

PresentationCommand.shotFlip

NewiOSmacOS
case shotFlip(ShotFlipCommand)

A value that represents a command that mirrors a whole frame vertically for the duration of the command.

init

SetCameraCommand.init

NewiOSmacOS
public init(id: Int, time: CMTime, cameraID: String, overrides: SetCameraCommand.Overrides?)

Creates a command with a specific ID, cameraID, start time and override parameters.

Parameters

id
The unique ID of this command.
time
The time for this command to start during playback.
cameraID
The cameraID associated with this command.
overrides
The override parameters for the camera
var

SetCameraCommand.overrides

NewiOSmacOS
public var overrides: SetCameraCommand.Overrides?

The override parameters for the camera.

struct

SetCameraCommand.Overrides

NewiOSmacOS
public struct Overrides : Codable, Sendable

Override camera/venue params

Declaration
public struct Overrides : Codable, Sendable {

    /// Camera rotation to override the rotation in ``ImmersiveCamera/pose``.
    ///
    /// This rotation is applied uniformly to both eyes of a stereo frame as
    /// an image reorientation. It does not remap eye assignment, reproject parallax, or account
    /// for the stereo baseline encoded in the source video.
    ///
    /// For stereo content, only rotations that keep the stereo baseline horizontal in viewer
    /// space produce comfortable fusion:
    /// - **Yaw** (rotation about the viewer-up axis) — safe at any magnitude, though yaw angles
    ///   approaching ±180° invert the parallax sign (depth appears reversed).
    /// - **Pitch** (rotation about the camera's local right / baseline axis) — safe at any
    ///   magnitude; the baseline is the rotation axis and remains invariant.
    /// - **Roll** (rotation about the camera-forward axis) — rotates the baseline out of
    ///   horizontal, creating vertical disparity between eyes. Small roll angles cause eye
    ///   strain; larger angles (≥ a few degrees) prevent fusion entirely.
    /// - Any mixed rotation that decomposes to a non-zero roll component exhibits the same
    ///   failure proportional to the roll magnitude.
    ///
    /// For mono content there are no stereo constraints and the full rotation space is valid.
    public var rotation: Rotation3DFloat?

    /// Inline lens definition (ILPD) data to override lens calibration
    public var lensDefinitionData: ImmersiveCameraLensDefinition?

    /// Dynamic mask data to override camera mask
    public var maskData: ImmersiveDynamicMask?

    public init(rotation: Rotation3DFloat? = nil, lensDefinitionData: ImmersiveCameraLensDefinition? = nil, maskData: ImmersiveDynamicMask? = nil)

    /// Encodes this value into the given encoder.
    ///
    /// If the value fails to encode anything, `encoder` will encode an empty
    /// keyed container in its place.
    ///
    /// This function throws an error if any values are invalid for the given
    /// encoder's format.
    ///
    /// - Parameter encoder: The encoder to write data to.
    public func encode(to encoder: any Encoder) throws

    /// Creates a new instance by decoding from the given decoder.
    ///
    /// This initializer throws an error if reading from the decoder fails, or
    /// if the data read is corrupted or otherwise invalid.
    ///
    /// - Parameter decoder: The decoder to read data from.
    public init(from decoder: any Decoder) throws
}

No APIs match your filter.

← More in Spatial, 3D & RealityKit