Hi all,

We have a solution for those who have this issue in the future. 

-

We used OpenCV to do the software ISP and configured media-ctl + v4l2-ctl to dial gains and configure image format.

Below, we will provide manual instructions for setting up ISP for imx219 and an SD image that is preconfigured. 

Still, if anyone sees this and knows of a more efficient / better ISP software solution for this board, please reply below.

-

OS used: tisdk-debian-trixie-am62xxsip-evm-10.01.10.04.wic.xz  

uEnv.txt contains: name_overlays=ti/k3-am625-sk-m2-cc3351.dtbo ti/k3-am62x-sk-csi2-imx219.dtbo 

-

For media-ctl + v4l2-ctl commands below, you will need to adjust the py script to match the resolution for 480p or 1080p. 

Pick and run either resolution set up below:

========================================================================

480p set up

# Set formats on each pad to 640x480 SRGGB8

media-ctl -d /dev/media0 --set-v4l2 '"imx219 4-0010":0 [fmt:SRGGB8_1X8/640x480]'

-

# Bridge sink → match sensor

media-ctl -d /dev/media0 --set-v4l2 '"cdns_csi2rx.30101000.csi-bridge":0 [fmt:SRGGB8_1X8/640x480]'

-

# TI CSI-RX sink → match bridge

media-ctl -d /dev/media0 --set-v4l2 '"30102000.ticsi2rx":0 [fmt:SRGGB8_1X8/640x480]'

-

# Set the capture node format (matches your script's FOURCC=RGGB)

v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=RGGB

-

# Digital and Analog gains

v4l2-ctl -d /dev/v4l-subdev2 --set-ctrl analogue_gain=160,digital_gain=512

========================================================================

1080p set up

# Sensor → 1920x1080, SRGGB8

media-ctl -d /dev/media0 --set-v4l2 '"imx219 4-0010":0 [fmt:SRGGB8_1X8/1920x1080]'

-

# Bridge sink → match sensor

media-ctl -d /dev/media0 --set-v4l2 '"cdns_csi2rx.30101000.csi-bridge":0 [fmt:SRGGB8_1X8/1920x1080]'

-

# TI CSI-RX sink → match bridge

media-ctl -d /dev/media0 --set-v4l2 '"30102000.ticsi2rx":0 [fmt:SRGGB8_1X8/1920x1080]'

-

# Video node format (the DMA context)

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RGGB

-

# analog and digital gain

v4l2-ctl -d /dev/v4l-subdev2 --set-ctrl analogue_gain=160,digital_gain=512

========================================================================

========================================================================

# verify changes have been made with 

• media-ctl -d /dev/media0 -p | sed -n '/imx219/,+10p;/csi-bridge/,+10p;/ticsi2rx/,+10p'
• v4l2-ctl -d /dev/video0 --get-fmt-video

-

Then, you need to run the Python script attached below and modify it to fit your video output preferences.

In our case, we are running a Flask server, which will host and make the stream available to a Linux-based laptop that connects to the am62sip’s DRP USB-C port. 

The SD image will have this configuration set up.

-

Python script:

import socket, sys, time, cv2, numpy as np

# ---- I/O ----
WIDTH, HEIGHT, FPS = 640, 480, 30
HOST, PORT = "127.0.0.1", 9000
BOUNDARY = "ThisRandomString"   # must match boundary in Flask server 

# ---- ISP tweaks  ----
B_GAIN, G_GAIN, R_GAIN = 1.40, 1.00, 1.40   # per-channel white balance
SAT = 1.40                                  # saturation scale (HSV)
GAMMA = 1.20                                
JPEG_QUALITY = 70

# Profiling
PRINT_EVERY = 30    # print averages every N frames
def now(): return time.perf_counter()

# BGGR worked best
# other options: RGGB/GRBG/GBRG
BAYER2BGR = cv2.COLOR_BayerBG2BGR

# ---- open the camera as raw Bayer8 ----
cap = cv2.VideoCapture("/dev/video0", cv2.CAP_V4L2)
if not cap.isOpened():
    sys.exit("Can't open /dev/video0")

cap.set(cv2.CAP_PROP_FRAME_WIDTH,  WIDTH)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, HEIGHT)
cap.set(cv2.CAP_PROP_FPS,          FPS)
cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"RGGB"))  # demosaic as BGGR
cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)                           # avoid libv4l auto-convert

# ---- TCP server (Flask connects here) ----
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind((HOST, PORT))
srv.listen(1)
print(f"[producer] Waiting for Flask on {HOST}:{PORT} ...")

# ---- precompute gamma LUT ----
x = np.arange(256, dtype=np.float32) / 255.0
lut_gamma = (np.clip(x ** (1.0 / max(GAMMA, 1e-6)), 0, 1) * 255.0 + 0.5).astype(np.uint8)

# ---- gain LUTs (build once) ----
def gain_lut(g):
    t = (np.arange(256, dtype=np.float32) * float(g)).clip(0, 255)
    return t.astype(np.uint8)

lut_b_gain = gain_lut(B_GAIN)
lut_g_gain = gain_lut(G_GAIN)
lut_r_gain = gain_lut(R_GAIN)
lut_sat_gain = gain_lut(SAT)

# TurboJPEG init (falls back to OpenCV if unavailable)
USE_TURBOJPEG = True
try:
    from turbojpeg import TurboJPEG, TJPF_BGR, TJSAMP_420
    _tj = TurboJPEG()
    _tj_fmt = TJPF_BGR
    _tj_sub = TJSAMP_420   # 4:2:0 = fastest/common
    _have_tj = True
    print("[jpeg] Using TurboJPEG")
except Exception:
    _have_tj = False
    print("[jpeg] TurboJPEG not available; using OpenCV")


# ---- ISP (profiled) ----
def isp_bayer8_to_bgr_profiled(bayer8):
    """Return (bgr, t_dem, t_wb, t_sat, t_gamma) in seconds."""
    t0 = now()
    bgr = cv2.cvtColor(bayer8, BAYER2BGR)           # demosaic
    t1 = now()

    # WB via LUTs (no float32, no split/merge churn)
    b, g, r = cv2.split(bgr)
    b = cv2.LUT(b, lut_b_gain)
    g = cv2.LUT(g, lut_g_gain)
    r = cv2.LUT(r, lut_r_gain)
    bgr = cv2.merge([b, g, r])
    t2 = now()

    # Saturation in HSV
    hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
    hsv[..., 1] = cv2.LUT(hsv[..., 1], lut_sat_gain)
    bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
    t3 = now()

    bgr = cv2.LUT(bgr, lut_gamma)                   # gamma
    t4 = now()
    return bgr, (t1 - t0), (t2 - t1), (t3 - t2), (t4 - t3)


def to_jpeg_bytes(bgr):
    bgr = np.ascontiguousarray(bgr)

    # Use TurboJPEG if present
    if _have_tj and USE_TURBOJPEG:
        # Try known kwarg names across versions
        # TODO: just find the version we use
        try:
            return _tj.encode(
                bgr, quality=int(JPEG_QUALITY),
                pixel_format=_tj_fmt,
                jpeg_subsample=_tj_sub      # newer pyTurboJPEG
            )
        except TypeError:
            try:
                return _tj.encode(
                    bgr, quality=int(JPEG_QUALITY),
                    pixel_format=_tj_fmt,
                    sampling_factor=_tj_sub  # older pyTurboJPEG
                )
            except TypeError:
                # Fall back to library default subsampling (usually 4:2:0)
                return _tj.encode(
                    bgr, quality=int(JPEG_QUALITY),
                    pixel_format=_tj_fmt
                )

    # OpenCV fallback 
    params = [
        int(cv2.IMWRITE_JPEG_QUALITY), int(JPEG_QUALITY),
        int(cv2.IMWRITE_JPEG_OPTIMIZE), 0,
        int(cv2.IMWRITE_JPEG_PROGRESSIVE), 0,
    ]
    ok, buf = cv2.imencode(".jpg", bgr, params)
    if not ok:
        raise RuntimeError("cv2.imencode('.jpg') failed")
    return buf.tobytes()



# ---- simple rolling stats ----
# TODO: check if stat collection causes lag or breaks anything if left on too long
stats = {
    "read":0, "norm":0, "isp_dem":0, "isp_wb":0, "isp_sat":0, "isp_gam":0,
    "yuyv":0, "bgr3":0, "gray":0, "jpeg":0, "send":0, "total":0, "count":0
}
last_print = now()     
frames_since = 0       

def print_and_reset_stats():
    global last_print, frames_since
    c = max(1, stats["count"])
    def ms(x): return f"{(x/c)*1000:.1f}"
    elapsed = now() - last_print
    fps = frames_since / max(1e-9, elapsed)

    line = (
        f"[avg {c} frames | {elapsed:.2f}s | producer_fps={fps:.2f}] "
        f"read={ms(stats['read'])} norm={ms(stats['norm'])}  "
        f"ISP dem={ms(stats['isp_dem'])} wb={ms(stats['isp_wb'])} sat={ms(stats['isp_sat'])} gam={ms(stats['isp_gam'])}  "
        f"YUYV={ms(stats['yuyv'])} BGR3={ms(stats['bgr3'])} gray={ms(stats['gray'])}  "
        f"jpeg={ms(stats['jpeg'])} send={ms(stats['send'])} total={ms(stats['total'])}"
    )
    print(line)

    for k in stats: stats[k] = 0
    last_print = now()
    frames_since = 0

while True:
    conn, addr = srv.accept()
    print(f"[producer] Flask connected from {addr}, streaming...")
    try:
        while True:
            t_total0 = now()

            t0 = now()
            ok, frame = cap.read()                # camera read
            t1 = now()
            if not ok:
                print("[producer] camera read failed")
                break
            stats["read"] += (t1 - t0)

            # normalize odd shapes (some libv4l builds flatten to 1 x (W*H))
            t2 = now()
            if frame.ndim == 2 and frame.shape == (1, WIDTH*HEIGHT):
                frame = frame.reshape(HEIGHT, WIDTH)
            elif frame.ndim == 1 and frame.size == WIDTH*HEIGHT:
                frame = frame.reshape(HEIGHT, WIDTH)
            t3 = now()
            stats["norm"] += (t3 - t2)

            # path select + processing
            if frame.ndim == 2 and frame.shape == (HEIGHT, WIDTH):
                # Bayer8 → ISP breakdown
                bgr, t_dem, t_wb, t_sat, t_gam = isp_bayer8_to_bgr_profiled(frame)
                stats["isp_dem"] += t_dem
                stats["isp_wb"]  += t_wb
                stats["isp_sat"] += t_sat
                stats["isp_gam"] += t_gam
            elif frame.ndim == 3 and frame.shape[2] == 2:
                # YUYV fallback
                ty0 = now()
                bgr = cv2.cvtColor(frame, cv2.COLOR_YUV2BGR_YUY2)
                bgr = cv2.LUT(bgr, lut_gamma)
                ty1 = now()
                stats["yuyv"] += (ty1 - ty0)
            elif frame.ndim == 3 and frame.shape[2] == 3:
                # already BGR/RGB-ish
                tb0 = now()
                bgr = cv2.LUT(frame, lut_gamma)
                tb1 = now()
                stats["bgr3"] += (tb1 - tb0)
            else:
                # last resort: make it gray then BGR
                tg0 = now()
                gray = frame.reshape(-1).astype(np.uint8)[:WIDTH*HEIGHT].reshape(HEIGHT, WIDTH)
                bgr  = cv2.cvtColor(gray, cv2.COLOR_GRAY2BGR)
                bgr  = cv2.LUT(bgr, lut_gamma)
                tg1 = now()
                stats["gray"] += (tg1 - tg0)

            # JPEG encode
            tj0 = now()
            jpg = to_jpeg_bytes(bgr)
            tj1 = now()
            stats["jpeg"] += (tj1 - tj0)

            # send one MJPEG part
            ts0 = now()
            header = (f"--{BOUNDARY}\r\n"
                      f"Content-Type: image/jpeg\r\n"
                      f"Content-Length: {len(jpg)}\r\n\r\n").encode("ascii")
            try:
                conn.sendall(header); conn.sendall(jpg); conn.sendall(b"\r\n")
            except (BrokenPipeError, ConnectionResetError):
                print("[producer] client disconnected")
                break
            ts1 = now()
            stats["send"] += (ts1 - ts0)

            t_total1 = now()
            stats["total"] += (t_total1 - t_total0)
            stats["count"] += 1
            frames_since += 1     # <-- increment for wall-clock FPS

            if stats["count"] >= PRINT_EVERY:
                print_and_reset_stats()

    finally:
        conn.close()
        print("[producer] Waiting for Flask to reconnect...")
        if stats["count"] > 0:
            print_and_reset_stats()
            

SD image: 

(Link will be updated later today)

-

root dir contains:

ISP_pipeline.py  __pycache__  faster_flask.py  setup_imx219.sh  venv

IPS_pipeline.py is the OpenCV ISP implementation. Runs 480p at 30fps with adequate white balancing. 

faster_flask.py contains a Flask server that hosts video for connected Linux-based laptops (Tested on Ubuntu 22.04, MacOS, and ChromeOS)

setup_imx219.sh contains setup scripts for media-ctl and v4l2-ctl 

-

How to use the provided SD image.

Connect power to AM62SIP, then connect the DRP USB-C port to Linux based laptop.

30 seconds after the AM62SIP board boots, a new Ethernet connection will connect on the laptop

On the laptop, open Google Chrome and go to 192.168.7.1:8080
The video stream will begin

If the video stream does not start >1 minute after AM625 boot, refresh the Chrome page.

-

Pipeline description and documentation:

-

End-to-End CSI Camera → On-Device ISP → MJPEG over HTTP

This document describes the complete data path and configuration used to stream frames from the IMX219 CSI camera to a web endpoint for display. It covers:

1. media graph configuration (pad formats and capture node)
2. the Python “producer” that performs ISP and emits MJPEG over TCP
3. the Flask relay served by Gunicorn that exposes the MJPEG stream over HTTP
4. how a client displays the stream
5. verification, monitoring, and common levers

Big Picture

IMX219 sensor (SRGGB8) 
   └─> Cadence CSI-2 Bridge 
         └─> TI CSI-RX
               └─> /dev/video0 (RGGB, 640x480, Bayer8)
                     └─> Python Producer (OpenCV ISP → TurboJPEG → MJPEG over TCP:9000)
                           └─> Flask Relay @ 127.0.0.1:8080 (gunicorn)
                                 └─> Browser hits http://<board>:8080/ and renders MJPEG

1) Media graph configuration (480p, SRGGB8)

The Linux media pipeline must be configured so the sensor, CSI bridge, and receiver agree on format and resolution. The capture node /dev/video0 is then set to a matching format.

# Sensor → 640x480, SRGGB8 (8-bit raw Bayer)
media-ctl -d /dev/media0 --set-v4l2 '"imx219 4-0010":0 [fmt:SRGGB8_1X8/640x480]'

# Bridge sink → match sensor
media-ctl -d /dev/media0 --set-v4l2 '"cdns_csi2rx.30101000.csi-bridge":0 [fmt:SRGGB8_1X8/640x480]'

# TI CSI-RX sink → match bridge
media-ctl -d /dev/media0 --set-v4l2 '"30102000.ticsi2rx":0 [fmt:SRGGB8_1X8/640x480]'

# Capture node (DMA context) → geometry + Bayer FOURCC tag (RGGB)
v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=RGGB

# Sensor gain controls
v4l2-ctl -d /dev/v4l-subdev2 --set-ctrl analogue_gain=160,digital_gain=512

Why:

• The three media-ctl calls set the format at each subdevice pad so the MIPI CSI2 frames traverse the graph without conversion.
• The v4l2-ctl call binds the capture node to the same geometry and a Bayer FOURCC. The producer script reads from this node in “raw Bayer 8-bit” mode.
• Gain controls are adjusted at the sensor subdevice.

Verification:

media-ctl -d /dev/media0 -p | sed -n '/imx219/,+10p;/csi-bridge/,+10p;/ticsi2rx/,+10p'
v4l2-ctl -d /dev/video0 --get-fmt-video

2) Producer application (OpenCV ISP → MJPEG over TCP)

File: stream_bayer_v4l2_to_flask_fix_profiled.py

Purpose: Read Bayer8 from /dev/video0, run a minimal ISP, encode as JPEG, and publish an MJPEG byte stream over a plain TCP socket. A downstream HTTP server relays it unchanged.

2.1 Device and IO parameters

WIDTH, HEIGHT, FPS = 640, 480, 30
HOST, PORT = "127.0.0.1", 9000
BOUNDARY = "ThisRandomString"

• WIDTH/HEIGHT must match the configured capture node.
• FPS is requested on the V4L2 device; actual rate is governed by sensor timing.
• BOUNDARY is the multipart marker used in the MJPEG stream. It must match the Flask relay’s boundary.

2.2 Open the capture node in Bayer mode

cap = cv2.VideoCapture("/dev/video0", cv2.CAP_V4L2)
cap.set(cv2.CAP_PROP_FRAME_WIDTH,  WIDTH)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, HEIGHT)
cap.set(cv2.CAP_PROP_FPS,          FPS)
cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"RGGB"))
cap.set(cv2.CAP_PROP_CONVERT_RGB,  0)

• CONVERT_RGB=0 disables libv4l color conversion; frames arrive as single-channel Bayer8.
• The kernel surfaces a Bayer FOURCC tag; the producer performs all demosaic and color work in user space.

2.3 ISP stages (minimal, CPU-only)

B_GAIN, G_GAIN, R_GAIN = 1.40, 1.00, 1.40
SAT, GAMMA = 1.40, 1.20

Processing per frame:

1. Demosaic: cv2.cvtColor(bayer8, cv2.COLOR_BayerBG2BGR)
   • Converts raw mosaiced Bayer8 to 3-channel BGR.
   • The constant COLOR_BayerBG2BGR selects the BGGR layout used by this pipeline.
2. White balance (WB): channel-wise LUTs (cv2.LUT)
   • Three precomputed 8-bit LUTs scale each channel to adjust color temperature neutrally and cheaply.
3. Saturation: HSV domain scaling
   • Convert BGR→HSV, scale S, convert back to BGR. Provides predictable saturation control.
4. Gamma: one 8-bit LUT
   • Approximates an sRGB-like transfer function to lift shadows and compress highlights.

Each step’s timing is accumulated and reported periodically for monitoring.

2.4 JPEG encoding (TurboJPEG preferred)

# TurboJPEG (preferred) with subsampling 4:2:0
from turbojpeg import TurboJPEG, TJPF_BGR, TJSAMP_420
JPEG_QUALITY = 70

• TurboJPEG provides lower latency and CPU load than OpenCV’s encoder.
• If TurboJPEG is not available, the code falls back to cv2.imencode(".jpg", ...).

2.5 MJPEG framing over TCP

The producer runs a small TCP server that waits for one client (the relay). For each encoded JPEG:

--<BOUNDARY>\\r\\n
Content-Type: image/jpeg\\r\\n
Content-Length: <N>\\r\\n
\\r\\n
<jpeg bytes>\\r\\n

• The relay reads these bytes and forwards them unchanged as an HTTP response body of type multipart/x-mixed-replace.

• The producer prints periodic timing lines:

  [avg 30 frames | 4.61s | producer_fps=6.51] read=21.4 ... jpeg=42.0 ... total=153.6
  

Use these values to track where time is spent.

3) HTTP relay (Flask app served by Gunicorn)

Files:

• faster_flask.py (Flask app)
• Run command:

gunicorn -w 1 -k gthread --threads 2 --timeout 0 -b 0.0.0.0:8080 faster_flask:app

3.1 Behavior

The relay connects to the producer’s TCP socket, reads chunks, and yields them directly to clients. No parsing, buffering or re-framing is performed in the relay.

@app.route('/')
def stream_mjpeg():
    def generate():
        with socket.create_connection(("127.0.0.1", 9000)) as s:
            s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
            s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1 << 20)  # 1 MB
            CHUNK = 65536
            while True:
                chunk = s.recv(CHUNK)
                if not chunk:
                    break
                yield chunk

    return Response(
        generate(),
        mimetype='multipart/x-mixed-replace; boundary=ThisRandomString',
        direct_passthrough=True
    )

• Gunicorn provides a production WSGI server with worker/thread control and proper socket handling.
• direct_passthrough=True avoids unnecessary buffering in Flask/Werkzeug.
• TCP_NODELAY minimizes latency by disabling Nagle’s algorithm between producer and relay.
• The BOUNDARY string matches the producer.

---

4) Displaying the stream

Any HTTP client that understands MJPEG can render the stream.

• Web browser: open http://<board-ip>:8080/
• ffplay: ffplay -fflags nobuffer -flags low_delay -framedrop -rtsp_flags prefer_tcp http://<board-ip>:8080/
• mpv: mpv --no-cache --profile=low-latency http://<board-ip>:8080/

The content type is multipart/x-mixed-replace; boundary=ThisRandomString. Each part is a complete JPEG image.

---

5) Data path summary

IMX219 sensor (SRGGB8 640x480)
   → cdns_csi2rx bridge (same bus code/geometry)
   → TI CSI-RX (same)
   → /dev/video0 (640x480, FOURCC=RGGB)
   → Python producer (OpenCV):
       - demosaic (BGGR → BGR)
       - WB (per-channel LUT)
       - saturation (HSV scale)
       - gamma (LUT)
       - JPEG encode (TurboJPEG or OpenCV)
   → TCP MJPEG with boundary "ThisRandomString" on 127.0.0.1:9000
   → Flask app relayed by Gunicorn at <http://0.0.0.0:8080/>
   → Browser/Player renders MJPEG

6) Configuration levers

• Resolution:

  Set all three subdevice pads and the capture node as shown in section 1. Update WIDTH/HEIGHT in the producer script.

• Frame rate:

  The capture request sets CAP_PROP_FPS. Effective FPS is determined by sensor timings (exposure, blanking, link frequency). Adjust vertical_blanking/exposure on the sensor subdevice as needed to respect timing budgets.

• Color processing:

  • WB gains: B_GAIN/G_GAIN/R_GAIN.
  • Saturation: SAT.
  • Gamma: GAMMA.
  • Demosaic pattern constant: BAYER2BGR = cv2.COLOR_BayerBG2BGR. This constant must match the physical mosaic orientation for correct colors.

• JPEG:

  JPEG_QUALITY controls compression. TurboJPEG with 4:2:0 subsampling provides the best speed/size trade-off on CPU.

• Network/relay:

  Chunk size and socket options in faster_flask.py are set for low latency. Gunicorn is configured with one worker and two threads to keep scheduling overhead minimal.

---

7) Monitoring and troubleshooting

• Producer logs:
  • “Waiting for Flask on 127.0.0.1:9000” indicates the relay is not yet connected.
  • “camera read failed” indicates the capture node is not delivering frames. Verify media graph formats and /dev/video0 geometry.
• Verify media graph anytime:

media-ctl -d /dev/media0 -p | sed -n '/imx219/,+10p;/csi-bridge/,+10p;/ticsi2rx/,+10p'
v4l2-ctl -d /dev/video0 --get-fmt-video

• Gunicorn/Flask logs:

  Confirm HTTP requests reach /: and that the connection stays open. Intermittent disconnects point to the upstream TCP source or client network instability.

---

8) Autostart on boot (cron)

1. Setup script to configure the media graph (as documented previously)
2. Start the Gunicorn relay in the Python virtualenv
3. Start the producer pipeline in the same virtualenv

@reboot sleep 2 && /usr/local/bin/setup_imx219_640x480.sh >>/var/log/imx219-setup.log 2>&1
@reboot sleep 10 && /bin/bash -lc 'source /root/venv/bin/activate && cd /root/test_deploy/feature_tests/opencv_image_pipeline && exec gunicorn -w 1 -k gthread --threads 2 --timeout 0 -b 0.0.0.0:8080 faster_flask:app' >>/var/log/faster_flask.log 2>&1
@reboot sleep 15 && /bin/bash -lc "source /root/venv/bin/activate && cd /root/test_deploy/feature_tests/opencv_image_pipeline && exec python ISP_pipeline.py" >>/var/log/isp_pipeline.log 2>&1

Glossary

Sensor source pad (IMX219)

The sensor’s output pad (imx219 4-0010: pad 0). It drives MIPI D-PHY (1 clock + N data lanes) and emits CSI-2 packets carrying raw Bayer pixels.

• Role: Produce the image stream on the wire (e.g., RAW8 SRGGB8_1X8 at 640×480).

• What must match: The mediabus code and resolution configured on this pad must be mirrored on the downstream bridge sink.

• How it’s set:

  media-ctl -d /dev/media0 --set-v4l2 '"imx219 4-0010":0 [fmt:SRGGB8_1X8/640x480]'

---

Bridge sink

The sink pad of the CSI-2 bridge subdevice (cdns_csi2rx…csi-bridge: pad 0). It is the front-end that speaks D-PHY + CSI-2 at the electrical/protocol level.

• Role: Terminate D-PHY lanes, deskew and align them, parse CSI-2 packet headers (short/long), check ECC/CRC, and expose a clean, byte-aligned stream with explicit frame/line boundaries to the receiver.

• What must match: Mediabus code and resolution must equal the sensor’s source pad.

• How it’s set:

  media-ctl -d /dev/media0 --set-v4l2 '"cdns_csi2rx.30101000.csi-bridge":0 [fmt:SRGGB8_1X8/640x480]'

---

TI CSI-RX sink

The sink pad (pad 0) of the TI CSI-2 receiver subdevice (30102000.ticsi2rx). This is the receiver’s input to the SoC image pipeline.

• Role: Consume the bridge’s parsed stream, unpack pixel payloads as needed (e.g., RAW10 bit-unpacking), and feed one or more DMA contexts (video nodes).

• What must match: Mediabus code and resolution must equal the bridge’s source / sensor’s output.

• How it’s set:

  media-ctl -d /dev/media0 --set-v4l2 '"30102000.ticsi2rx":0 [fmt:SRGGB8_1X8/640x480]'

---

Capture node (DMA context) → geometry + Bayer FOURCC tag (RGGB)

A V4L2 video node (e.g., /dev/video0) bound to one receiver context. It is the userspace-visible endpoint that DMAs frames into system memory.

• Role: Allocate buffers, drive DMA, and expose frames to applications.

• Geometry: Width/height written per frame by the DMA engine.

• FOURCC tag: The userspace pixel format identifier. For raw Bayer in this pipeline, use RGGB to indicate single-plane 8-bit Bayer with RGGB mosaic; applications then demosaic in software.

• How it’s set:

  v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=RGGB

---

FOURCC

A four-character code used by V4L2 to name pixel formats (e.g., RGGB, YUYV, MJPG). It describes how bytes are laid out in memory and what color interpretation to apply. In this pipeline, RGGB tells userspace the buffer is 8-bit raw Bayer with an RGGB arrangement.

---

media-ctl

A userspace utility for the Media Controller API. It configures subdevices and links in a complex camera graph:

• What it sets:
  • Pad formats (mediabus code, width, height) on sensor/bridge/receiver pads.
  • Active links between entities (usually already enabled by the kernel for simple graphs).
• Why it’s needed: V4L2 video nodes see only the endpoint DMA format. The upstream path (sensor → bridge → CSI-RX) must also agree on mediabus format and geometry; media-ctl programs those.
• Persistence: settings are kept until reboot or reconfigure.
• Typical sequence: set the sensor source pad, then the bridge sink/source, then the CSI-RX sink. Finally, configure the video node with v4l2-ctl.
• Example:

media-ctl -d /dev/media0 \\
  --set-v4l2 '"imx219 4-0010":0 [fmt:SRGGB8_1X8/640x480]'
media-ctl -d /dev/media0 \\
  --set-v4l2 '"cdns_csi2rx.30101000.csi-bridge":0 [fmt:SRGGB8_1X8/640x480]'
media-ctl -d /dev/media0 \\
  --set-v4l2 '"30102000.ticsi2rx":0 [fmt:SRGGB8_1X8/640x480]'

Multipart marker (boundary)

In an HTTP multipart/x-mixed-replace response (MJPEG), the boundary is the delimiter string that separates consecutive parts (frames). Each part starts with --<boundary> and contains per-part headers (e.g., Content-Type: image/jpeg, Content-Length:) followed by the raw image bytes. The producer and consumer must agree on the same boundary string.

---

libv4l

A userspace library that wraps V4L2 and can perform automatic pixel format conversions, scaling, and controls emulation. For raw pipelines, this is typically disabled (CAP_PROP_CONVERT_RGB=0) to avoid hidden conversions and to ensure low latency and predictable formats.

---

Demosaic (debayer)

The process of reconstructing a full-color RGB image from the single-channel Bayer mosaic. Algorithms range from simple bilinear interpolation to edge-aware and frequency-domain methods.

• Bayer patterns: the 2×2 tile that repeats across the sensor. Common orders:
  • RGGB: top row R G; bottom row G B
  • BGGR: top row B G; bottom row G R
  • GRBG and GBRG: green leading with swapped R/B positions
• In this pipeline: the mediabus code is SRGGB8_1X8 (sensor mosaic). After capture, OpenCV’s cv2.cvtColor(bayer8, cv2.COLOR_BayerBG2BGR) is selected because empirical testing showed BGGR provided the correct orientation for the captured buffer (node → userspace). The correct conversion constant depends on how the driver presents the mosaic at the node; if colors are swapped, choose the matching COLOR_Bayer*2BGR variant.

---

White balance

A per-channel gain correction to neutralize color casts from illumination or sensor response. Implemented here by applying three 8-bit LUTs (B, G, R) so that each channel is scaled independently without per-pixel floating-point work.

---

LUT (Look-Up Table)

A precomputed array used to transform pixel values efficiently. For 8-bit images, a LUT is length-256. Applying a LUT (cv2.LUT) replaces each pixel with LUT[pixel]. LUTs are used here for per-channel white balance and for gamma correction.

---

HSV domain scaling (saturation)

A color adjustment performed by converting BGR → HSV, scaling the S (saturation) channel by a factor, clipping to 0–255, then converting back to BGR. This increases or decreases color intensity while preserving hue and lightness approximately. It is more visually stable than scaling channels directly in RGB.

---

Gamma

A non-linear mapping that compresses or expands mid-tones for perceptual display.

• Why: human vision is non-linear; applying a gamma curve can make images appear more natural when displayed or encoded.
• Form: for 8-bit, build a LUT L[i] = round(255 * (i/255)^(1/γ)). Values γ > 1 brighten shadows and mid-tones; γ < 1 darkens them.
• In this pipeline: a gamma LUT is applied after demosaic/WB/saturation to shape tonal response before JPEG encoding. The chosen value (~1.2) approximates a mild sRGB-ish lift without heavy computation.
• Order: WB → (optional) saturation → gamma is a common sequence; applying gamma last ensures chroma operations operate on a near-linear domain.

---

TurboJPEG

A Python wrapper around libjpeg-turbo, a SIMD-accelerated JPEG codec. It offers significantly faster encode/decode than baseline libjpeg/OpenCV, especially on ARM with NEON. In this pipeline it encodes BGR frames with quality and subsampling controls (typically 4:2:0) to reduce CPU time per frame.

---

multipart/x-mixed-replace

An HTTP response type used for server-push streams, such as MJPEG. The server keeps the connection open and sends a sequence of parts, each a complete image with headers, separated by the boundary marker. Browsers and simple clients render the most recent part, enabling live preview without WebSockets or chunked JSON.

---

Gunicorn

A production WSGI HTTP server for Python applications. Compared to Flask’s built-in dev server, Gunicorn provides better concurrency, timeouts, worker models, and robustness. In this pipeline it runs the Flask relay with a threaded worker, proxying bytes from the local TCP producer to HTTP clients with minimal overhead.

---

Media graph (a.k.a. media topology)

The directed graph of entities and links that represent the camera pipeline in the kernel: Sensor → CSI-2 Bridge → CSI-RX → Video node(s). Each entity has pads (SINK/SOURCE) with formats. The graph is queried and configured via the Media Controller API (media-ctl -p, --set-v4l2).

---

Multipart marker vs. frame integrity

The boundary itself does not “split” images; it simply delimits them in the HTTP stream. Frame tearing typically comes from producer under-run/over-run, client rendering cadence, or network/CPU stalls, not from the boundary mechanism. Ensuring each part includes a correct Content-Length and sending complete JPEGs atomically maintains integrity.

---

Frame interval / FPS (subdevice vs. node)

Some sensors support setting frame intervals at the subdevice level (--set-subdev-fps). If unsupported, effective FPS is governed by exposure/blanking and by the downstream consumer’s ability to keep up. The node’s CAP_PROP_FPS is a request; the actual cadence depends on the upstream timing.

---

10-bit vs 8-bit sensor modes

IMX219 supports 8-bit and 10-bit bayer on the media bus (SRGGB8_1X8 and SRGGB10_1X10). Ten-bit modes increase precision but may cost bandwidth and CPU in userspace (if unpacked). This pipeline uses 8-bit to minimize bandwidth and processing cost.

---

BGGR / RGGB / GRBG / GBRG (pattern notes)

These are the four standard 2×2 Bayer tile orders. The correct OpenCV conversion constant must match how the captured buffer’s mosaic appears at the node:

• cv2.COLOR_BayerBG2BGR for BGGR,

• cv2.COLOR_BayerRG2BGR for RGGB,

• cv2.COLOR_BayerGR2BGR for GRBG,

• cv2.COLOR_BayerGB2BGR for GBRG.

  A quick test is to point at a red object; if it appears blue, swap between RG and BG variants.