How to correct the perspective of trading cards in an iOS app

Photos of flat items such as trading cards, book covers, documents or packaging are rarely taken perfectly straight on. The result is a skewed rectangle on a cluttered background, which looks poor in a listing and is hard to compare with other items.

This tutorial walks through Card Straight, an open-source demo iPhone app that photographs a trading card, uses the Remove Background API to find the card's edges, straightens it on-device, and optionally produces a blurred, relit background with the Image Editing API.

The full source code is available on GitHub: Photoroom/api-demo-apps → iOS Perspective Correction App (MIT license).

What the app produces

From a single photo, the app creates two kinds of results:

• Transparent: a tight, perspective-corrected cutout of the card, exported as a PNG with transparency.

• Background blur + relight: the card straightened in place, with the original background kept but blurred and the lighting harmonized, exported as a JPEG.

(original photo)	Transparent finish	Background blur + relight finish

The interesting part of this app is the division of labour: Photoroom does the hard perception work (where is the card in the image) while the phone does the geometry it is uniquely placed to do: warping the pixels to a rectangle, and using its own camera calibration to work out the card's true proportions.

The API is called once per photo for the transparent finish, and once more only if the user asks for the blurred finish.

Running the app

You need Xcode, an iPhone running iOS 17 or later (the Simulator works for everything except the camera), and a Photoroom API key. If you don't have one yet, here are the steps to create yours.

1. Clone the repository and open iOS Perspective Correction App/ios-perspective-correction.xcodeproj in Xcode.

2. Select the ios-perspective-correction scheme and an iPhone destination. On a physical device, pick your development team under Signing & Capabilities.

3. Build and run. Tap the gear icon and paste your API key: it is stored in the device Keychain and never bundled in the app or logged.

4. Tap Take a photo and point the camera at a single, fully visible card on a contrasting background. Automatic capture fires once the card is stable and sharp; you can also use the shutter, Choose from library, or Try a sample card (nine bundled photos that work in the Simulator).

5. Review the result, adjust Proportions or Rotate if needed, pick the finish in Settings → Finish, then save or share.

Use Sandbox mode while you explore: prepend sandbox_ to your API key and calls are free (results are watermarked).

How it works

The pipeline has four stages. Two of them call the Photoroom API; the other two run entirely on the device.

Step 1: Capture and normalize the photo

CameraCapture.swift does more than take a picture: it configures the session so that each frame comes with a trustworthy camera intrinsic matrix (focal length and optical centre, in pixels). It picks the rear physical wide-angle camera, prefers the 4K preset and falls back to 1080p, locks the zoom factor to 1, turns off video stabilization (which crops and shifts the frame), enables geometric distortion correction where supported, and turns on isCameraIntrinsicMatrixDeliveryEnabled on the connection. The captured still is the exact video sample buffer that passed the auto-capture checks, read together with its kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix attachment, rather than a separate full-resolution photo whose intrinsics could differ.

session.sessionPreset = session.canSetSessionPreset(.hd4K3840x2160) ? .hd4K3840x2160 : .hd1920x1080
device.videoZoomFactor = 1
if connection.isVideoStabilizationSupported { connection.preferredVideoStabilizationMode = .off }
if connection.isCameraIntrinsicMatrixDeliverySupported { connection.isCameraIntrinsicMatrixDeliveryEnabled = true }
 
// Later, for the captured frame:
let intrinsics = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) as? Data

Orientation is handled with AVCaptureDevice.RotationCoordinator: sensor pixels are left unrotated, the closest cardinal EXIF orientation is attached to the JPEG, and the same orientation is stored alongside the calibration. This matters in Step 3, where the app has to undo orientation and resizing before it can project detected corners into calibrated camera coordinates; otherwise it would pair a portrait, cropped image with a landscape, full-frame matrix.

Before anything is sent to Photoroom, CardProcessor.swift applies the EXIF orientation and downsizes the photo to at most 3,000 pixels on its longest edge:

let original = CGImageSourceCreateThumbnailAtIndex(source, 0, [
    kCGImageSourceCreateThumbnailFromImageAlways: true,
    kCGImageSourceCreateThumbnailWithTransform: true,   // bake in EXIF rotation
    kCGImageSourceThumbnailMaxPixelSize: 3000
] as CFDictionary)

Both steps matter for API integrations in general: the Remove Background API doesn't carry EXIF metadata into its result, so an un-rotated upload comes back rotated (see Troubleshooting), and a smaller upload means a faster round trip with no visible loss for this use case.

The shutter can also fire on its own. An on-device Vision rectangle detector checks preview frames at about 5 Hz; once a whole card is visible, the app waits 0.6 s for aiming, gives autofocus up to 1.2 s to settle, then requires two sufficiently sharp frames (Laplacian variance on a 256×256 patch of the card's interior, threshold 80) 0.15–0.5 s apart before capturing. None of these preview frames are sent to Photoroom, only the single captured frame is, so automatic capture costs nothing in API calls. Manual capture, library import and the bundled samples remain available; the README documents the heuristics in full.

Step 2: Get a segmentation mask from Photoroom

Rather than asking for the cutout, the app asks the Remove Background API for the segmentation mask by setting channels=alpha (see Download the segmentation mask). The equivalent curl call is:

curl --request POST \
  --url https://sdk.photoroom.com/v1/segment \
  --header 'x-api-key: YOUR_API_KEY' \
  --form image_file=@/path/to/card.jpg \
  --form channels=alpha \
  --output mask.png

In Swift, Processing/PhotoroomClient.swift builds the same multipart request by hand with URLSession, with no third-party dependency:

var request = URLRequest(url: URL(string: "https://sdk.photoroom.com/v1/segment")!)
request.httpMethod = "POST"
request.timeoutInterval = 90
request.setValue(apiKey, forHTTPHeaderField: "x-api-key")
request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
request.setValue("image/png", forHTTPHeaderField: "Accept")

var body = Data()
body.append("--\(boundary)\r\nContent-Disposition: form-data; name=\"image_file\"; filename=\"card.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n")
body.append(imageData)
body.append("\r\n--\(boundary)\r\nContent-Disposition: form-data; name=\"channels\"\r\n\r\nalpha\r\n")
body.append("--\(boundary)--\r\n")
request.httpBody = body

Why the mask rather than the cutout? The app needs the mask twice: once as a binary image to find the card's corners, and once at full resolution to make the background transparent. Requesting the grayscale mask gives it both from a single, smaller response.

The mask is then used in two ways in CardProcessor.finish:

• It is downscaled to at most 1,600 pixels and thresholded into a binary image for the geometry step below.

• It is composited onto the original photo as its alpha channel (CIBlendWithAlphaMask), which removes the background, including around the card's rounded corners, before the perspective warp.

Step 3: Find the corners and straighten the card on-device

Processing/CardGeometry.swift turns the binary mask into four corners without any machine learning, using only classic image geometry:

1. Flood-fill the background from the image border, so that enclosed holes inside the card don't affect its outline.

2. Keep the largest connected foreground component (the card, or a graded slab if that is what was segmented).

3. Compute its convex hull and simplify it to four significant vertices.

4. Assign the boundary pixels to the four sides and fit a line to each with two passes of robust least squares, which trims rounded corners and stray pixels.

5. Intersect the four lines to get the corners, order them, and reject detections whose confidence or fill ratio is too low.

With the corners known, Core Image's CIPerspectiveCorrection filter warps the transparent cutout into a rectangle.

Recovering the card's true proportions from the camera

A perspective warp needs to know the target rectangle's width-to-height ratio, and the photo alone cannot tell you: a square seen at an angle and a tall rectangle seen straight on can produce the same four corners. This is where the camera intrinsics from Step 1 come in.

Processing/CameraCalibration.swift first maps the four corners found on the (rotated, resized) mask back to the unrotated sensor frame, then applies the inverse intrinsic matrix to turn each corner into a viewing ray. For a flat rectangle, the first two columns of the homography from the unit square to those rays are the two physical side vectors, up to a common scale, so the ratio of their lengths is the card's real width/height ratio. The estimate is discarded when the calibration is invalid, when the recovered sides are not close to perpendicular (orthogonality above 0.12), when the ratio falls outside 0.2–5, or when perturbing the corners by a few pixels changes the ratio by more than 6%, which indicates the corners are too imprecise to trust.

guard let base = solve(corners), (0.2...5).contains(base.ratio), base.orthogonality < 0.12 else { return nil }
// ... re-solve with slightly perturbed corners ...
guard uncertainty < 0.06 else { return nil }
return AspectRatioEstimate(widthOverHeight: base.ratio, relativeUncertainty: uncertainty)

This is why the Proportions control behaves differently depending on where the photo came from. Photos taken in the app have intrinsics, so proportions default to Automatic. Photos imported from the library have none, so the app shows apparent proportions as a preview and waits for the user to pick the Standard card preset (2.5 : 3.5) or enter Custom dimensions before exporting. In both cases the estimate concerns proportions only, not physical size in millimeters. Changing proportions or rotating the result re-renders locally from the retained cutout and mask and never triggers another API call.

Finally, an on-device text-recognition pass (Vision) compares the four possible orientations and rotates the card upright when printed text clearly favors one of them.

Step 4 (optional): Blur and relight the background with two Image Editing API calls

If the user selects the Background blur finish, the app warps the whole photo (not just the cutout) so that the card becomes a rectangle while its surroundings are kept, then sends this image to the Image Editing API twice.

The first call uses AI Expand to fill the transparent borders that the warp leaves around the photo:

curl --request POST \
  --url https://image-api.photoroom.com/v2/edit \
  --header 'x-api-key: YOUR_API_KEY' \
  --form imageFile=@perspective-corrected-card.png \
  --form removeBackground=false \
  --form referenceBox=originalImage \
  --form expand.mode=ai.auto \
  --output expanded.png

The second call takes the expanded image and applies Background Blur and AI Relight, exporting a JPEG:

curl --request POST \
  --url https://image-api.photoroom.com/v2/edit \
  --header 'x-api-key: YOUR_API_KEY' \
  --form imageFile=@expanded.png \
  --form removeBackground=false \
  --form referenceBox=originalImage \
  --form background.blur.mode=gaussian \
  --form background.blur.radius=0.012 \
  --form lighting.mode=ai.preserve-hue-and-saturation \
  --form export.format=jpeg \
  --output result.jpg

A few details worth noting in these two calls:

• removeBackground=false keeps the original background so it can be expanded, then blurred. Photoroom still detects the subject internally, which is how the blur is applied to the background only.

• referenceBox=originalImage tells the API to treat the whole canvas, not the cropped subject, as the frame of reference, so the framing of the input image is preserved exactly.

• lighting.mode=ai.preserve-hue-and-saturation improves the lighting while keeping the card's colours faithful, which matters for collectibles. Turning Relight off in the app simply omits this field.

In PhotoroomClient.swift, the two calls are chained, with a cancellation check between them so that switching finish mid-way doesn't waste a request:

func blurredFinish(image: Data, apiKey: String, relight: Bool = true) async throws -> Data {
    let expanded = try await send(Self.finishRequest(image: image, apiKey: apiKey, step: .expand))
    try Task.checkCancellation()
    return try await send(Self.finishRequest(image: expanded, apiKey: apiKey, step: .blur, relight: relight))
}

After the JPEG comes back, the app crops it to the card's rectified bounds with 5% padding, so the final image is framed the same way as the "Original" it is compared against.

Handling errors

The client maps HTTP status codes to user-facing messages, which is a good pattern to reuse in any mobile integration:

static func responseError(status: Int) -> PhotoroomError {
    switch status {
    case 401, 403: .invalidKey     // missing or rejected API key
    case 402:      .credits        // no API credits left
    case 429:      .rateLimited    // default limit is 60 images per minute
    default:       .server(status)
    }
}

Each case produces a recoverable error in the UI (a Retry button, or a prompt to check the key in Settings), and an in-flight request is cancelled if the user starts a new capture. Timeouts are set generously (90 s for segmentation, 180 s for the AI edits), because AI features on large images can take several seconds.

Adapting the app to your use case

Although the demo is built around trading cards, nothing in the API calls is specific to them. The same pipeline works for any flat rectangular item: comics, vinyl sleeves, book covers, boxed products, ID cards or documents. Here's how to adapt it:

• Different proportions: the aspect-ratio acceptance range in CardGeometry.swift is 1:1 to 5:1, and the Standard card preset (2.5 : 3.5) is just a default. Add presets for your item's dimensions.

• A different finish: the second Image Editing API call is where the look is decided. Replace the blur with a solid background.color, an AI Background generated from a background.prompt, or add an AI Shadow with shadow.mode. Use the Try it now! button on each feature page to find the parameters, then paste them into finishRequest.

• Server-side processing: if you'd rather not ship API keys in a mobile app, move PhotoroomClient behind your own backend and keep the on-device geometry as is. The API calls are plain multipart HTTP requests and translate directly to any language; see Code Samples.

• Batch processing: the Scripts/GenerateReadmeExamples.swift script in the repository runs the exact same processing code on macOS from the command line, which is a convenient starting point for processing a folder of photos.

Cost per photo: the transparent finish uses 1 Remove Background API call; the blurred finish adds 2 Image Editing API calls. Local adjustments (proportions, rotation, orientation) are free.

Going further

• Browse the source: Photoroom/api-demo-apps

• The repository also contains a Web demo app and a general iOS demo app.

• Related tutorials: How to improve images for second-hand item marketplaces, How to create sticker images.