在 Android 上使用 ML Kit 偵測姿勢

ML Kit 提供兩個針對姿勢偵測所最佳化的 SDK。

SDK 名稱pose-detectionpose-detection-accurate
導入作業在建構期間,程式碼和資產會以靜態方式連結至您的應用程式。在建構期間,程式碼和資產會以靜態方式連結至您的應用程式。
對應用程式大小的影響 (包括程式碼和素材資源)約 10.1 MB約 13.3 MB
成效Pixel 3 XL:約 30 FPSPixel 3XL:使用 CPU 約 23FPS,使用 GPU 可達約 30 FPS

立即試用

事前準備

  1. 在專案層級的 build.gradle 檔案中,請務必在 buildscriptallprojects 區段中納入 Google 的 Maven 存放區。
  2. 將 ML Kit Android 程式庫的依附元件新增至模組的應用程式層級的 Gradle 檔案,通常為 app/build.gradle

    dependencies {
      // If you want to use the base sdk
      implementation 'com.google.mlkit:pose-detection:18.0.0-beta5'
      // If you want to use the accurate sdk
      implementation 'com.google.mlkit:pose-detection-accurate:18.0.0-beta5'
    }
    

1. 建立「PoseDetector」的執行個體

PoseDetector 種付款方式

如要偵測圖片中的姿勢,請先建立 PoseDetector 的執行個體,並 指定偵測工具設定

偵測模式

PoseDetector 會在兩種偵測模式下運作。請務必選擇符合的選項 所需用途

STREAM_MODE (預設)
姿勢偵測器會先偵測到 然後執行姿勢偵測在後續影格中 除非人員符合,否則系統不會執行人偵測步驟 模糊錯誤,或系統不再以高可信度偵測到錯誤。姿勢偵測器會 嘗試追蹤最有名的使用者,並逐一傳回他們的姿勢 推論這麼做可減少延遲時間和順暢偵測。以下模式使用時機: 或需要偵測影片串流中的姿勢。
SINGLE_IMAGE_MODE
姿勢偵測器會偵測人然後跑步姿勢 偵測。每張圖片都會執行人為偵測步驟,因此延遲 值較高,且無人追蹤。使用姿勢時使用這個模式 偵測靜態圖像或不需要追蹤的位置

硬體設定

PoseDetector 支援多種硬體設定,可進行最佳化調整 成效:

  • CPU:僅使用 CPU 執行偵測工具
  • CPU_GPU:同時使用 CPU 和 GPU 執行偵測工具

建構偵測工具選項時,您可以使用 API setPreferredHardwareConfigs:控制硬體選取。根據預設 所有硬體設定都會設為首選

ML Kit 會採用每項設定的可用性、穩定性、正確性和延遲時間 並從首選設定中挑選最適合的設定如果沒有 適用偏好的設定,系統會自動使用 CPU 設定 並設為備用委刊項ML Kit 會依據 不會造成阻力 使用者初次執行偵測工具時,會使用 CPU。在所有 準備作業完成後,下列執行作業將使用最合適的設定。

setPreferredHardwareConfigs 的使用範例:

  • 為了讓 ML Kit 挑選最佳設定,請勿呼叫這個 API。
  • 如果不想啟用任何加速功能,請僅傳入 CPU
  • 如果您想使用 GPU 卸載 CPU (即使 GPU 速度可能較慢),請改為 在 CPU_GPU 之內。

指定姿勢偵測器選項:

Kotlin

// Base pose detector with streaming frames, when depending on the pose-detection sdk
val options = PoseDetectorOptions.Builder()
    .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
    .build()

// Accurate pose detector on static images, when depending on the pose-detection-accurate sdk
val options = AccuratePoseDetectorOptions.Builder()
    .setDetectorMode(AccuratePoseDetectorOptions.SINGLE_IMAGE_MODE)
    .build()

Java

// Base pose detector with streaming frames, when depending on the pose-detection sdk
PoseDetectorOptions options =
   new PoseDetectorOptions.Builder()
       .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
       .build();

// Accurate pose detector on static images, when depending on the pose-detection-accurate sdk
AccuratePoseDetectorOptions options =
   new AccuratePoseDetectorOptions.Builder()
       .setDetectorMode(AccuratePoseDetectorOptions.SINGLE_IMAGE_MODE)
       .build();

最後,建立 PoseDetector 的執行個體。傳送您指定的選項:

Kotlin

val poseDetector = PoseDetection.getClient(options)

Java

PoseDetector poseDetector = PoseDetection.getClient(options);

2. 準備輸入圖片

如要偵測圖片中的姿勢,請建立 InputImage 物件 從 Bitmapmedia.ImageByteBuffer、位元組陣列或 裝置。然後,將 InputImage 物件傳遞至 PoseDetector

如要使用姿勢偵測功能,圖片尺寸應至少為 480x360 像素。如要即時偵測姿勢,請擷取影格 達到這個最低解析度將有助於縮短延遲時間

您可以建立InputImage 不同來源的 ANR 物件,說明如下。

使用 media.Image

如要建立InputImage 物件,例如從 media.Image 物件擷取圖片 裝置的相機,請傳遞 media.Image 物件和映像檔的 旋轉為 InputImage.fromMediaImage()

如果您使用 CameraX 程式庫、OnImageCapturedListenerImageAnalysis.Analyzer 類別會計算旋轉值 不必確保憑證管理是否適當 因為 Google Cloud 會為您管理安全性

Kotlin

private class YourImageAnalyzer : ImageAnalysis.Analyzer {

    override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null) {
            val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
            // Pass image to an ML Kit Vision API
            // ...
        }
    }
}

Java

private class YourAnalyzer implements ImageAnalysis.Analyzer {

    @Override
    public void analyze(ImageProxy imageProxy) {
        Image mediaImage = imageProxy.getImage();
        if (mediaImage != null) {
          InputImage image =
                InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
          // Pass image to an ML Kit Vision API
          // ...
        }
    }
}

如果您沒有使用相機程式庫提供圖片的旋轉角度, 可根據裝置的旋轉角度和相機方向來計算 感應器:

Kotlin

private val ORIENTATIONS = SparseIntArray()

init {
    ORIENTATIONS.append(Surface.ROTATION_0, 0)
    ORIENTATIONS.append(Surface.ROTATION_90, 90)
    ORIENTATIONS.append(Surface.ROTATION_180, 180)
    ORIENTATIONS.append(Surface.ROTATION_270, 270)
}

/**
 * Get the angle by which an image must be rotated given the device's current
 * orientation.
 */
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
@Throws(CameraAccessException::class)
private fun getRotationCompensation(cameraId: String, activity: Activity, isFrontFacing: Boolean): Int {
    // Get the device's current rotation relative to its "native" orientation.
    // Then, from the ORIENTATIONS table, look up the angle the image must be
    // rotated to compensate for the device's rotation.
    val deviceRotation = activity.windowManager.defaultDisplay.rotation
    var rotationCompensation = ORIENTATIONS.get(deviceRotation)

    // Get the device's sensor orientation.
    val cameraManager = activity.getSystemService(CAMERA_SERVICE) as CameraManager
    val sensorOrientation = cameraManager
            .getCameraCharacteristics(cameraId)
            .get(CameraCharacteristics.SENSOR_ORIENTATION)!!

    if (isFrontFacing) {
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360
    } else { // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360
    }
    return rotationCompensation
}

Java

private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
static {
    ORIENTATIONS.append(Surface.ROTATION_0, 0);
    ORIENTATIONS.append(Surface.ROTATION_90, 90);
    ORIENTATIONS.append(Surface.ROTATION_180, 180);
    ORIENTATIONS.append(Surface.ROTATION_270, 270);
}

/**
 * Get the angle by which an image must be rotated given the device's current
 * orientation.
 */
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
private int getRotationCompensation(String cameraId, Activity activity, boolean isFrontFacing)
        throws CameraAccessException {
    // Get the device's current rotation relative to its "native" orientation.
    // Then, from the ORIENTATIONS table, look up the angle the image must be
    // rotated to compensate for the device's rotation.
    int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
    int rotationCompensation = ORIENTATIONS.get(deviceRotation);

    // Get the device's sensor orientation.
    CameraManager cameraManager = (CameraManager) activity.getSystemService(CAMERA_SERVICE);
    int sensorOrientation = cameraManager
            .getCameraCharacteristics(cameraId)
            .get(CameraCharacteristics.SENSOR_ORIENTATION);

    if (isFrontFacing) {
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
    } else { // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
    }
    return rotationCompensation;
}

然後,請傳遞 media.Image 物件和 將度數值旋轉為 InputImage.fromMediaImage()

Kotlin

val image = InputImage.fromMediaImage(mediaImage, rotation)

Java

InputImage image = InputImage.fromMediaImage(mediaImage, rotation);

使用檔案 URI

如要建立InputImage 物件,將應用程式結構定義與檔案 URI 傳遞至 InputImage.fromFilePath()。如果您要 使用 ACTION_GET_CONTENT 意圖提示使用者選取 取自圖片庫應用程式中的圖片。

Kotlin

val image: InputImage
try {
    image = InputImage.fromFilePath(context, uri)
} catch (e: IOException) {
    e.printStackTrace()
}

Java

InputImage image;
try {
    image = InputImage.fromFilePath(context, uri);
} catch (IOException e) {
    e.printStackTrace();
}

使用 ByteBufferByteArray

如要建立InputImage ByteBufferByteArray 的物件,請先計算圖片 與先前 media.Image 輸入中所述的旋轉角度相同。 接著,使用緩衝區或陣列建立 InputImage 物件,以及 高度、寬度、顏色編碼格式以及旋轉角度:

Kotlin

val image = InputImage.fromByteBuffer(
        byteBuffer,
        /* image width */ 480,
        /* image height */ 360,
        rotationDegrees,
        InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
)
// Or:
val image = InputImage.fromByteArray(
        byteArray,
        /* image width */ 480,
        /* image height */ 360,
        rotationDegrees,
        InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
)

Java

InputImage image = InputImage.fromByteBuffer(byteBuffer,
        /* image width */ 480,
        /* image height */ 360,
        rotationDegrees,
        InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
);
// Or:
InputImage image = InputImage.fromByteArray(
        byteArray,
        /* image width */480,
        /* image height */360,
        rotation,
        InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12
);

使用 Bitmap

如要建立InputImage 物件中,Bitmap 物件,請做出以下宣告:

Kotlin

val image = InputImage.fromBitmap(bitmap, 0)

Java

InputImage image = InputImage.fromBitmap(bitmap, rotationDegree);

圖像以 Bitmap 物件和旋轉角度表示。

3. 處理圖片

將準備好的 InputImage 物件傳遞至 PoseDetectorprocess 方法。

Kotlin

Task<Pose> result = poseDetector.process(image)
       .addOnSuccessListener { results ->
           // Task completed successfully
           // ...
       }
       .addOnFailureListener { e ->
           // Task failed with an exception
           // ...
       }

Java

Task<Pose> result =
        poseDetector.process(image)
                .addOnSuccessListener(
                        new OnSuccessListener<Pose>() {
                            @Override
                            public void onSuccess(Pose pose) {
                                // Task completed successfully
                                // ...
                            }
                        })
                .addOnFailureListener(
                        new OnFailureListener() {
                            @Override
                            public void onFailure(@NonNull Exception e) {
                                // Task failed with an exception
                                // ...
                            }
                        });

4. 取得偵測到的姿勢相關資訊

如果系統在圖片中偵測到人物,姿勢偵測 API 會傳回 Pose 具有 33 PoseLandmark 秒的物件。

如果人物出現在圖像中,模型會指派該人物 外框 外面缺少的地標座標,會將其低 InFrameConfidence 值。

如果在 Pose 畫面中未偵測到任何人 物件不包含 PoseLandmark

Kotlin

// Get all PoseLandmarks. If no person was detected, the list will be empty
val allPoseLandmarks = pose.getAllPoseLandmarks()

// Or get specific PoseLandmarks individually. These will all be null if no person
// was detected
val leftShoulder = pose.getPoseLandmark(PoseLandmark.LEFT_SHOULDER)
val rightShoulder = pose.getPoseLandmark(PoseLandmark.RIGHT_SHOULDER)
val leftElbow = pose.getPoseLandmark(PoseLandmark.LEFT_ELBOW)
val rightElbow = pose.getPoseLandmark(PoseLandmark.RIGHT_ELBOW)
val leftWrist = pose.getPoseLandmark(PoseLandmark.LEFT_WRIST)
val rightWrist = pose.getPoseLandmark(PoseLandmark.RIGHT_WRIST)
val leftHip = pose.getPoseLandmark(PoseLandmark.LEFT_HIP)
val rightHip = pose.getPoseLandmark(PoseLandmark.RIGHT_HIP)
val leftKnee = pose.getPoseLandmark(PoseLandmark.LEFT_KNEE)
val rightKnee = pose.getPoseLandmark(PoseLandmark.RIGHT_KNEE)
val leftAnkle = pose.getPoseLandmark(PoseLandmark.LEFT_ANKLE)
val rightAnkle = pose.getPoseLandmark(PoseLandmark.RIGHT_ANKLE)
val leftPinky = pose.getPoseLandmark(PoseLandmark.LEFT_PINKY)
val rightPinky = pose.getPoseLandmark(PoseLandmark.RIGHT_PINKY)
val leftIndex = pose.getPoseLandmark(PoseLandmark.LEFT_INDEX)
val rightIndex = pose.getPoseLandmark(PoseLandmark.RIGHT_INDEX)
val leftThumb = pose.getPoseLandmark(PoseLandmark.LEFT_THUMB)
val rightThumb = pose.getPoseLandmark(PoseLandmark.RIGHT_THUMB)
val leftHeel = pose.getPoseLandmark(PoseLandmark.LEFT_HEEL)
val rightHeel = pose.getPoseLandmark(PoseLandmark.RIGHT_HEEL)
val leftFootIndex = pose.getPoseLandmark(PoseLandmark.LEFT_FOOT_INDEX)
val rightFootIndex = pose.getPoseLandmark(PoseLandmark.RIGHT_FOOT_INDEX)
val nose = pose.getPoseLandmark(PoseLandmark.NOSE)
val leftEyeInner = pose.getPoseLandmark(PoseLandmark.LEFT_EYE_INNER)
val leftEye = pose.getPoseLandmark(PoseLandmark.LEFT_EYE)
val leftEyeOuter = pose.getPoseLandmark(PoseLandmark.LEFT_EYE_OUTER)
val rightEyeInner = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE_INNER)
val rightEye = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE)
val rightEyeOuter = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE_OUTER)
val leftEar = pose.getPoseLandmark(PoseLandmark.LEFT_EAR)
val rightEar = pose.getPoseLandmark(PoseLandmark.RIGHT_EAR)
val leftMouth = pose.getPoseLandmark(PoseLandmark.LEFT_MOUTH)
val rightMouth = pose.getPoseLandmark(PoseLandmark.RIGHT_MOUTH)

Java

// Get all PoseLandmarks. If no person was detected, the list will be empty
List<PoseLandmark> allPoseLandmarks = pose.getAllPoseLandmarks();

// Or get specific PoseLandmarks individually. These will all be null if no person
// was detected
PoseLandmark leftShoulder = pose.getPoseLandmark(PoseLandmark.LEFT_SHOULDER);
PoseLandmark rightShoulder = pose.getPoseLandmark(PoseLandmark.RIGHT_SHOULDER);
PoseLandmark leftElbow = pose.getPoseLandmark(PoseLandmark.LEFT_ELBOW);
PoseLandmark rightElbow = pose.getPoseLandmark(PoseLandmark.RIGHT_ELBOW);
PoseLandmark leftWrist = pose.getPoseLandmark(PoseLandmark.LEFT_WRIST);
PoseLandmark rightWrist = pose.getPoseLandmark(PoseLandmark.RIGHT_WRIST);
PoseLandmark leftHip = pose.getPoseLandmark(PoseLandmark.LEFT_HIP);
PoseLandmark rightHip = pose.getPoseLandmark(PoseLandmark.RIGHT_HIP);
PoseLandmark leftKnee = pose.getPoseLandmark(PoseLandmark.LEFT_KNEE);
PoseLandmark rightKnee = pose.getPoseLandmark(PoseLandmark.RIGHT_KNEE);
PoseLandmark leftAnkle = pose.getPoseLandmark(PoseLandmark.LEFT_ANKLE);
PoseLandmark rightAnkle = pose.getPoseLandmark(PoseLandmark.RIGHT_ANKLE);
PoseLandmark leftPinky = pose.getPoseLandmark(PoseLandmark.LEFT_PINKY);
PoseLandmark rightPinky = pose.getPoseLandmark(PoseLandmark.RIGHT_PINKY);
PoseLandmark leftIndex = pose.getPoseLandmark(PoseLandmark.LEFT_INDEX);
PoseLandmark rightIndex = pose.getPoseLandmark(PoseLandmark.RIGHT_INDEX);
PoseLandmark leftThumb = pose.getPoseLandmark(PoseLandmark.LEFT_THUMB);
PoseLandmark rightThumb = pose.getPoseLandmark(PoseLandmark.RIGHT_THUMB);
PoseLandmark leftHeel = pose.getPoseLandmark(PoseLandmark.LEFT_HEEL);
PoseLandmark rightHeel = pose.getPoseLandmark(PoseLandmark.RIGHT_HEEL);
PoseLandmark leftFootIndex = pose.getPoseLandmark(PoseLandmark.LEFT_FOOT_INDEX);
PoseLandmark rightFootIndex = pose.getPoseLandmark(PoseLandmark.RIGHT_FOOT_INDEX);
PoseLandmark nose = pose.getPoseLandmark(PoseLandmark.NOSE);
PoseLandmark leftEyeInner = pose.getPoseLandmark(PoseLandmark.LEFT_EYE_INNER);
PoseLandmark leftEye = pose.getPoseLandmark(PoseLandmark.LEFT_EYE);
PoseLandmark leftEyeOuter = pose.getPoseLandmark(PoseLandmark.LEFT_EYE_OUTER);
PoseLandmark rightEyeInner = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE_INNER);
PoseLandmark rightEye = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE);
PoseLandmark rightEyeOuter = pose.getPoseLandmark(PoseLandmark.RIGHT_EYE_OUTER);
PoseLandmark leftEar = pose.getPoseLandmark(PoseLandmark.LEFT_EAR);
PoseLandmark rightEar = pose.getPoseLandmark(PoseLandmark.RIGHT_EAR);
PoseLandmark leftMouth = pose.getPoseLandmark(PoseLandmark.LEFT_MOUTH);
PoseLandmark rightMouth = pose.getPoseLandmark(PoseLandmark.RIGHT_MOUTH);

提升成效的訣竅

結果的品質取決於輸入圖片的品質:

  • 為了讓 ML Kit 準確偵測姿勢,圖像中的人物 足夠的像素資料來呈現出來為獲得最佳成效,主旨應該 至少 256x256 像素
  • 在即時應用程式中偵測到姿勢時,建議您也考量 輸入圖片的整體尺寸可處理較小的圖片 因此,為了縮短延遲時間,擷取解析度較低的圖片 務必遵循上述解決要求,並確保拍攝主體 盡量增加圖片張量
  • 圖像對焦品質不佳也可能會影響準確度。如果沒有可接受的結果 請要求使用者重新擷取圖片

如要在即時應用程式中使用姿勢偵測功能,請遵循下列準則,以達到最佳影格速率:

  • 使用基本姿勢偵測 SDK 和 STREAM_MODE
  • 建議以較低的解析度拍攝圖片。不過,也請留意這個 API 的圖片尺寸規定。
  • 如果您使用 Cameracamera2 API、 限制對偵測工具的呼叫如果影片有新影片 影格掉落時,表示影格是否可用。詳情請參閱 VisionProcessorBase 類別的範例。
  • 如果您是使用 CameraX API, 請務必將背壓策略設為預設值 ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST。 這麼做可保證系統一次只會傳送一張圖片進行分析。如果圖片較多 會在分析器忙碌時產生,這些作業會自動遭到捨棄,不會排入佇列 廣告放送。以呼叫方式關閉要分析的圖片後 ImageProxy.close(),最新一張圖片才會放送。
  • 如果使用偵測工具的輸出內容將圖像重疊 先從 ML Kit 取得結果,然後算繪圖片 並疊加單一步驟這會轉譯至顯示介面 每個輸入影格只能建立一次詳情請參閱 CameraSourcePreview 如需範例,請前往快速入門導覽課程範例應用程式中的 GraphicOverlay 類別。
  • 如果你使用 Camera2 API, ImageFormat.YUV_420_888 格式。如果使用舊版 Camera API,請以 ImageFormat.NV21 格式。

後續步驟