Skip to main content

Orientation

Camera Sensor Orientation​

A Camera sensor is configured to deliver buffers in a specific size, for example 4096x2160. To avoid re-allocating such large buffers every time the phone rotates, the Camera pipeline will always deliver buffers in it's native sensor orientation (see sensorOrientation) and frames need to be rotated to appear up-right.

Since actually rotating pixels in such large buffers is really expensive and causes an unnecessary performance overhead, VisionCamera applies rotations through flags or transforms:

Photo & Video Output​

Photos and videos will be captured in a potentially "wrong" orientation, and VisionCamera will write an EXIF flag to the photo/video file with the correct presentation rotation.

info

This is handled automatically, and it's behaviour can be controlled via the outputOrientation property.

Preview View​

The Preview output will stream in a potentially "wrong" orientation, but uses view transforms (rotate + translate matrix) to properly display the Camera stream "up-right".

info

This will always happen automatically according to the screen's rotation.

Frame Processor Output​

Frame Processors will stream frames in a potentially "wrong" orientation, and the client is responsible for properly interpreting the Frame data.

info

This needs to be handled manually, see Frame.orientation. For example, in MLKit just pass the Frame's orientation to the detect(...) method.

Implementation​

VisionCamera supports three ways to implement orientation:

  • Camera UI (preview view) is locked, but the buttons can rotate to the desired photo/video output orientation (recommended)
  • Camera UI (preview view) also rotates alongside with the photo/video output orientation
  • Both Camera UI (preview view) and photo/video output orientation are locked to a specific orientation

The outputOrientation prop​

The orientation in which photos and videos are captured can be adjusted via the outputOrientation property:

<Camera {...props} outputOrientation="device" />

  • "device": With the output orientation set to device (the default), photos and videos will be captured in the phone's physical orientation, even if the screen-rotation is locked. This means, even though the preview view and other views don't rotate to landscape, holding the phone in landscape mode will still capture landscape photos. This is the same behaviour as in the iOS stock Camera.

  • "preview": Similar to device, the preview orientation mode will follow the phone's physical orientation, allowing the user to capture landscape photos and videos - but will always respect screen-rotation locks. This means that the user is not able to capture landscape photos or videos if the preview view and other views stay in portrait mode (e.g. if the screen-lock is on).

  • "portrait", "portrait-upside-down", "landscape-left", "landscape-right": All captured photos and videos will be locked to the given orientation mode.

Listen to orientation changes​

Whenever the output orientation changes, the onOutputOrientationChanged event will be called with the new output orientation. This is a good point to rotate the buttons to the desired output orientation now.

The onPreviewOrientationChanged event will be called whenever the preview orientation changes, which might be unrelated to the output orientation. Depending on the device's natural orientation (e.g. iPads being landscape by default), you should rotate all buttons on the UI relative to the preview orientation.

As a helper method, VisionCamera fires the onUIRotationChanged event whenever the target UI rotation changes. You can directly apply this rotation to any UI elements such as buttons to rotate them to the correct orientation:

const [uiRotation, setUiRotation] = useState(0)
const uiStyle: ViewStyle = {
  transform: [{ rotate: `${uiRotation}deg` }]
}

return (
  <View>
    <Camera {...props} onUIRotationChanged={setUiRotation} />
    <FlipCameraButton style={uiStyle} />
  </View>
)
tip

For a smoother user experience, you should animate changes to the UI rotation. Use a library like react-native-reanimated to smoothly animate the rotate style.

The Frame's orientation​

In a Frame Processor, frames are streamed in their native sensor orientation. This means even if the phone is rotated from portrait to landscape, the Frame's width and height stay the same.

The Frame's orientation represents it's orientation relative to the current target orientation.

For example, if the phone is held in portrait mode and the Frame's orientation is landscape-right, it is 90° rotated and needs to be counter-rotated by -90° to appear "up-right". Instead of actually rotating pixels in the buffers, frame processor plugins just need to interpret the frame as being rotated.

MLKit handles this via a orientation property on the MLImage/VisionImage object:

public override func callback(_ frame: Frame, withArguments _: [AnyHashable: Any]?) -> Any? {
  let mlImage = MLImage(sampleBuffer: frame.buffer)
  mlImage.orientation = frame.orientation
  // ...

Orientation in Skia Frame Processors​

A Skia Frame Processor applies orientation via rotation and translation. This means the coordinate system stays the same, but output will be rotated accordingly. For a landscape-left frame, (0,0) will not be top left, but rather top right.


🚀 Next section: Exposure​