Real-Time Image Analysis with CameraX

By Michael Samuel Naeem · July 14, 2024 · 5 min read

The core of any camera feature that "understands" what it sees — a document scanner, a live translator, a barcode reader — is a CameraX ImageAnalysis use case delivering frames to your code in real time. The whole game is keeping up: the camera produces frames faster than most analysis can process them, so the difference between a smooth feature and a laggy one is how you handle backpressure and buffer lifecycle. Get those two right and everything else is detail.

I've built several live-camera features on CameraX. The mistakes are always the same, and they're always about frame flow, not the analysis itself.

The three use cases, bound together

CameraX gives you Preview, ImageCapture, and ImageAnalysis. For a live feature you bind Preview (what the user sees) and ImageAnalysis (what your code sees) to the same lifecycle:

val preview = Preview.Builder().build().also {
    it.setSurfaceProvider(previewView.surfaceProvider)
}

val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
    .setTargetResolution(Size(1280, 720))
    .build()
    .also {
        it.setAnalyzer(cameraExecutor) { proxy -> analyze(proxy) }
    }

cameraProvider.bindToLifecycle(
    lifecycleOwner, CameraSelector.DEFAULT_BACK_CAMERA, preview, analysis
)

Bind to a LifecycleOwner and CameraX starts and stops the camera with the lifecycle automatically — no manual open/release, which removes a whole category of camera-leak bugs.

Backpressure is the whole ballgame

The camera pushes ~30 frames a second. If your analyzer takes 60ms per frame, you can only process ~16 of them. What happens to the other 14? That's the backpressure strategy:

• STRATEGY_KEEP_ONLY_LATEST (use this): CameraX keeps only the newest frame and drops the rest. Your analyzer always works on fresh data and never builds a backlog. Latency stays flat.
• STRATEGY_BLOCK_PRODUCER: CameraX blocks until you finish, so you see every frame but the camera stalls and the preview stutters. Only correct when you genuinely must process every frame (rare — usually offline capture, not live).

For anything user-facing, KEEP_ONLY_LATEST. Chasing "process every frame" in a live viewfinder is how you get a feature that feels broken.

Close the ImageProxy or everything freezes

Each frame arrives as an ImageProxy backed by one buffer from a small pool. Hold onto it and you starve the pool; the analyzer simply stops getting called and the feature "freezes" with no crash and no log. The fix is disciplined closing:

private fun analyze(proxy: ImageProxy) {
    try {
        val rotation = proxy.imageInfo.rotationDegrees
        // ... run detection on proxy ...
    } finally {
        proxy.close()   // non-negotiable, even on exception
    }
}

If you hand the frame to an async consumer (like ML Kit's process() which returns a Task), close the proxy in the task's completion listener, not synchronously — closing while ML Kit still reads the buffer corrupts the frame. This ordering bug is the single most common CameraX + ML Kit defect I see.

Rotation and format

Two details that silently ruin accuracy:

• Rotation. The buffer is not oriented the way the user holds the phone. proxy.imageInfo.rotationDegrees tells you how to rotate for upright analysis. ML Kit's InputImage.fromMediaImage(image, rotationDegrees) takes it directly — pass it or your detector reads a sideways world and misses everything.
• Format. Default output is YUV_420_888, which is what the sensor produces and what ML Kit consumes natively — no conversion. Only request RGBA_8888 if your own code needs RGB, and know CameraX pays a conversion cost to give it to you.

Throttle when you don't need every frame

Many features don't need 30fps analysis. A document-edge detector at 5fps feels instant and saves battery and heat — and phones throttle the camera when they get hot, so less work can mean a more stable frame rate. A simple time gate:

private var lastRun = 0L
private val minIntervalMs = 200L   // ~5 fps

private fun analyze(proxy: ImageProxy) {
    val now = SystemClock.elapsedRealtime()
    if (now - lastRun < minIntervalMs) { proxy.close(); return }
    lastRun = now
    try { /* detect */ } finally { proxy.close() }
}

Feeding fewer, well-chosen frames to something like on-device ML Kit vision usually beats hammering it at full frame rate.

Keep heavy work off the main thread

Pass a dedicated single-thread executor to setAnalyzer. The analysis runs there, off the UI thread, so even a slow detector never janks the preview. When you have a result to draw (a bounding box overlay), post only that small result back to the main thread — never the frame. Marshaling whole frames to the UI thread is a classic self-inflicted jank source.

What I'd take away

A real-time CameraX pipeline lives or dies on frame flow, not cleverness. Use STRATEGY_KEEP_ONLY_LATEST so you always analyze fresh frames, close every ImageProxy (in a finally, or after your async consumer finishes) or the feed freezes, honor rotationDegrees so your detector sees an upright world, keep the default YUV format when feeding ML Kit, throttle to the frame rate your feature actually needs, and run analysis on a dedicated executor. Nail those and you get a live camera feature that stays smooth, cool, and accurate.

Common production mistakes

Teams get camerax image analysis wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Shipping camerax image analysis on Android fails quietly when you test only on flagship devices, skip process-death scenarios, or assume minSdk behavior matches latest API docs. Emulator-only validation misses OEM-specific battery optimizations and background execution limits.

Resources

• ImageAnalysis (CameraX)
• CameraX architecture
• CameraX preview
• ML Kit with CameraX
• ImageProxy reference