Stabilize camera images on Android SDK (Kotlin/Java)

ARCore now supports Electronic Image Stabilization (EIS), which helps produce a smooth camera preview. EIS achieves stabilization by observing phone movement using gyro and applying compensation homography mesh within the boundaries of camera texture that counters the minor shakes. EIS is only supported in the device's portrait orientation. All orientations will be supported in the 1.39.0 release of ARCore.

Query for EIS support and enable EIS

To enable EIS, configure your session to use ImageStabilizationMode.EIS. If the device doesn't support the EIS feature, this will cause an exception to be thrown from ARCore.

if (!session.isImageStabilizationModeSupported(Config.ImageStabilizationMode.EIS)) {
  return;
}
Config config = session.getConfig();
config.setImageStabilizationMode(Config.ImageStabilizationMode.EIS);
session.configure(config);

if (!session.isImageStabilizationModeSupported(Config.ImageStabilizationMode.EIS)) return
session.configure(
  session.config.apply { imageStabilizationMode = Config.ImageStabilizationMode.EIS }
)

Transform coordinates

When EIS is on, the renderer needs to use the modified device coordinates and matching texture coordinates that incorporate the EIS compensation when rendering the camera background. To get the EIS compensated coordinates, use Frame.transformCoordinates3d(), using OPENGL_NORMALIZED_DEVICE_COORDINATES as input and EIS_NORMALIZED_DEVICE_COORDINATES as output to get 3D device coordinates and EIS_TEXTURE_NORMALIZED as output to get 3D texture coordinates. For now, the only supported input coordinate type for Frame.transformCoordinates3d() is OPENGL_NORMALIZED_DEVICE_COORDINATES.

final FloatBuffer cameraTexCoords =
    ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_3D)
        .order(ByteOrder.nativeOrder())
        .asFloatBuffer();

final FloatBuffer screenCoords =
    ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_3D)
        .order(ByteOrder.nativeOrder())
        .asFloatBuffer();

final FloatBuffer NDC_QUAD_COORDS_BUFFER =
    ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_2D)
        .order(ByteOrder.nativeOrder())
        .asFloatBuffer()
        .put(
            new float[] {
              /*0:*/ -1f, -1f, /*1:*/ +1f, -1f, /*2:*/ -1f, +1f, /*3:*/ +1f, +1f,
            });

final VertexBuffer screenCoordsVertexBuffer =
    new VertexBuffer(render, /* numberOfEntriesPerVertex= */ 3, null);
final VertexBuffer cameraTexCoordsVertexBuffer =
    new VertexBuffer(render, /* numberOfEntriesPerVertex= */ 3, null);

NDC_QUAD_COORDS_BUFFER.rewind();
frame.transformCoordinates3d(
    Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES,
    NDC_QUAD_COORDS_BUFFER,
    Coordinates3d.EIS_NORMALIZED_DEVICE_COORDINATES,
    screenCoords);
screenCoordsVertexBuffer.set(screenCoords);

NDC_QUAD_COORDS_BUFFER.rewind();
frame.transformCoordinates3d(
    Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES,
    NDC_QUAD_COORDS_BUFFER,
    Coordinates3d.EIS_TEXTURE_NORMALIZED,
    cameraTexCoords);
cameraTexCoordsVertexBuffer.set(cameraTexCoords);

val COORDS_BUFFER_SIZE_2D = 2 * 4 * Float.SIZE_BYTES
val COORDS_BUFFER_SIZE_3D = 3 * 4 * Float.SIZE_BYTES
val cameraTexCoords =
  ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_3D)
    .order(ByteOrder.nativeOrder())
    .asFloatBuffer()
val screenCoords =
  ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_3D)
    .order(ByteOrder.nativeOrder())
    .asFloatBuffer()
val cameraTexCoordsVertexBuffer = VertexBuffer(render, /* numberOfEntriesPerVertex= */ 3, null)
val screenCoordsVertexBuffer = VertexBuffer(render, /* numberOfEntriesPerVertex= */ 3, null)
val NDC_QUAD_COORDS_BUFFER =
  ByteBuffer.allocateDirect(COORDS_BUFFER_SIZE_2D)
    .order(ByteOrder.nativeOrder())
    .asFloatBuffer()
    .apply {
      put(
        floatArrayOf(
          /* 0: */
          -1f,
          -1f,
          /* 1: */
          +1f,
          -1f,
          /* 2: */
          -1f,
          +1f,
          /* 3: */
          +1f,
          +1f
        )
      )
    }
NDC_QUAD_COORDS_BUFFER.rewind()
frame.transformCoordinates3d(
  Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES,
  NDC_QUAD_COORDS_BUFFER,
  Coordinates3d.EIS_NORMALIZED_DEVICE_COORDINATES,
  screenCoords
)
screenCoordsVertexBuffer.set(screenCoords)

NDC_QUAD_COORDS_BUFFER.rewind()
frame.transformCoordinates3d(
  Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES,
  NDC_QUAD_COORDS_BUFFER,
  Coordinates3d.EIS_TEXTURE_NORMALIZED,
  cameraTexCoords
)
cameraTexCoordsVertexBuffer.set(cameraTexCoords)

When EIS is off, the output 3D coordinates are equivalent to their 2D counterparts, with z values set to produce no change.

Modify shaders

The 3D coordinates calculated about should be passed to background rendering shaders. The vertex buffers are now 3D with EIS:

layout(location = 0) in vec4 a_Position;
layout(location = 1) in vec3 a_CameraTexCoord;
out vec3 v_CameraTexCoord;
void main() {
  gl_Position = a_Position;
  v_CameraTexCoord = a_CameraTexCoord;
}

Additionally, the fragment shader needs to apply perspective correction:

precision mediump float;
uniform samplerExternalOES u_CameraColorTexture;
in vec3 v_CameraTexCoord;
layout(location = 0) out vec4 o_FragColor;
void main() {
  vec3 tc = (v_CameraTexCoord / v_CameraTexCoord.z);
  o_FragColor = texture(u_CameraColorTexture, tc.xy);
}

See the hello_eis_kotlin sample app for more details.