Low FPS on video stream

[gmijo47]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Question

Hi there,
I'm working on some realtime object detecion and my fps is always 0.3 FPS (both on GPU T4, and CPU), i started form time_in_zone example. On start it spikes, for single second GPU-40FPS, CPU 10 FPS, and as time goes it goes down to 0.3FPS

requirments.txt
supervision>=0.20.0 ultralytics==8.2.22 opencv-python==4.8.0.76 inference==0.9.17

my app.py

from utils.configHandler import ConfigHandler
from inference import InferencePipeline
from inference.core.interfaces.camera.entities import VideoFrame
from ultralytics import YOLO
from utils.functions import find_in_list

import cv2
import supervision as sv
from typing import List

COLORS = sv.ColorPalette.from_hex(["#E6194B", "#3CB44B", "#FFE119", "#3C76D1"])
COLOR_ANNOTATOR = sv.ColorAnnotator(color=COLORS)
LABEL_ANNOTATOR = sv.LabelAnnotator(
    color=COLORS, text_color=sv.Color.from_hex("#000000")
)

class DataProcessor:
    def __init__(self, classes: List[int], config):
        self.classes = classes
        self.config = config
        self.tracker = sv.ByteTrack(minimum_matching_threshold=0.8)
        self.fps_monitor = sv.FPSMonitor()

    def on_prediction(self, detections: sv.Detections, frame: VideoFrame) -> None:
        self.fps_monitor.tick()
        fps = self.fps_monitor.fps

        detections = detections[find_in_list(detections.class_id, self.classes)]
        detections = self.tracker.update_with_detections(detections)

        annotated_frame = frame.image.copy()
        match fps:
            case fps if fps <= 5.0:
                bg_color = "#D20103"
            case fps if fps <= 12.5:
                bg_color = "#FFDE59"
            case _:
                bg_color = "#7DDA58"
        annotated_frame = sv.draw_text(
            scene=annotated_frame,
            text=f"{fps:.2f}",
            text_anchor=sv.Point(40, 30),
            background_color=sv.Color.from_hex(bg_color),
            text_color=sv.Color.from_hex("#000000"),
        )

        annotated_frame = COLOR_ANNOTATOR.annotate(
            scene=annotated_frame,
            detections=detections,
        )
        labels = [
            f"#{tracker_id}"
            for tracker_id in detections.tracker_id
        ]
        annotated_frame = LABEL_ANNOTATOR.annotate(
            scene=annotated_frame,
            detections=detections,
            labels=labels,
        )              
        print("fps" + str(fps))
        if self.config.debug:
            cv2.imshow("Processed Video", annotated_frame)
            cv2.waitKey(1)

def main() -> None:
    config = ConfigHandler()

    model = YOLO(config.weights)

    def inference_callback(frame: VideoFrame) -> sv.Detections:
        results = model(frame.image, verbose=config.model_verbose, conf=config.confidence, device=config.device)[0]
        return sv.Detections.from_ultralytics(results).with_nms(threshold=config.iou)

    processor = DataProcessor(classes=config.classes, config=config)

    pipeline = InferencePipeline.init_with_custom_logic(
        video_reference=config.rtsp_url,
        on_video_frame=inference_callback,
        on_prediction=processor.on_prediction,
    )

    pipeline.start()

    try:
        pipeline.join()
    except KeyboardInterrupt:
        pipeline.terminate()

if __name__ == "__main__":
    main()

Thanks in advance.

Additional

No response

[SkalskiP]

Hi @gmijo47 👋🏻 Let me convert this question into a discussion first.

[SkalskiP]

Could you run pip list on your machine and paste the result here.

[gmijo47]

@SkalskiP just to ping this, i don't think this is hardware issue

[LinasKo]

Hi @gmijo47 👋

What model are you using? Can you share the config values you get from ConfigHandler?

[gmijo47]

Model is yolo v8 nano, trained on custom data set, i will provide values from config within hour

[gmijo47]

Thanks for waiting here is config.json

{
    "weights": "models\\SUMInSight_v3.1.pt",
    "device": "cpu",
    "confidence_threshold": 0.3,
    "iou_threshold": 0.7,
    "classes": [0],
    "model_verbose": false,
    "debug": true 
}

also here is configHandler, basic class which retrieves values form config.json

import os
import json

class ConfigHandler:
    def __init__(self, config_file='config.json'):
        self.config_file = config_file
        self.config = self.load_config()

    def load_config(self):
        with open(self.config_file) as f:
            return json.load(f)

    @property
    def rtsp_url(self):
        return os.getenv('RSTP_CONN')

    @property
    def weights(self):
        return self.config['weights']

    @property
    def device(self):
        return self.config['device']

    @property
    def confidence(self):
        return self.config['confidence_threshold']

    @property
    def iou(self):
        return self.config['iou_threshold']

    @property
    def classes(self):
        return self.config['classes']
    
    @property
    def debug(self):
        return self.config['debug']
    
    @property
    def model_verbose(self):
        return self.config['model_verbose']

[gmijo47]

@LinasKo sorry for the ping, can you look thought this, I'm still unable to locate source of problem, Thanks in advance

[LinasKo]

Confirming the issue.
When running a local RTSP, passing it to InferencePipeline which applies a local Ultralytics or Inference model, the pipeline slows down to a crawl, even when a small model such as yolov8n is used.

Here are the reproduction steps:

Dependencies:

pip install supervision inference ultralytics

Running a small RTSP stream on a Mac camera:

ffmpeg -f avfoundation -framerate 30 -video_size 640x480 -i "0:none" -preset ultrafast -tune zerolatency -f mpegts udp://127.0.0.1:9090

Code to view and annotate (no extra files needed):

import numpy as np
from inference import InferencePipeline, get_model
from inference.core.interfaces.camera.entities import VideoFrame
from ultralytics import YOLO

import cv2
import supervision as sv
from typing import Any, List

COLORS = sv.ColorPalette.from_hex(["#E6194B", "#3CB44B", "#FFE119", "#3C76D1"])
COLOR_ANNOTATOR = sv.ColorAnnotator(color=COLORS)
LABEL_ANNOTATOR = sv.LabelAnnotator(
    color=COLORS, text_color=sv.Color.from_hex("#000000")
)

def draw_fps(frame: np.ndarray, fps: float) -> np.ndarray:
    match fps:
        case fps if fps <= 5.0:
            bg_color = "#D20103"
        case fps if fps <= 12.5:
            bg_color = "#FFDE59"
        case _:
            bg_color = "#7DDA58"
    frame = sv.draw_text(
        scene=frame,
        text=f"{fps:.2f}",
        text_anchor=sv.Point(40, 30),
        background_color=sv.Color.from_hex(bg_color),
        text_color=sv.Color.from_hex("#000000"),
    )
    return frame

def draw_detections(frame: np.ndarray, detections: sv.Detections) -> np.ndarray:
    frame = COLOR_ANNOTATOR.annotate(
        scene=frame,
        detections=detections,
    )
    labels = [
        f"#{class_id}"
        for class_id in detections.class_id
    ]
    frame = LABEL_ANNOTATOR.annotate(
        scene=frame,
        detections=detections,
        labels=labels,
    )
    return frame

class DataProcessor:
    def __init__(self):
        self.fps_monitor = sv.FPSMonitor()

    def on_prediction(self, detections_response: Any, frame: VideoFrame) -> None:
        self.fps_monitor.tick()
        fps = self.fps_monitor.fps

        detections = sv.Detections(
            xyxy=np.array([detections_response[0]]),
            confidence=np.array([detections_response[2]]),
            class_id=np.array([detections_response[3]]),
        )

        annotated_frame = frame.image.copy()
        annotated_frame = draw_fps(annotated_frame, fps)
        annotated_frame = draw_detections(annotated_frame, detections)

        cv2.imshow("Processed Video", annotated_frame)
        cv2.waitKey(1)

def main() -> None:
    # # Option 1: Ultralytics
    # model = YOLO("yolov8n")

    # def inference_callback(frame: VideoFrame) -> sv.Detections:
    #     frame = frame[0]
    #     results = model(frame.image, device="cpu", verbose=False)[0]
    #     detections = sv.Detections.from_ultralytics(results)
    #     return detections

    # Option 2: Inference
    model = get_model("yolov8n-640")

    def inference_callback(frame: VideoFrame) -> sv.Detections:
        frame = frame[0]
        results = model.infer(frame.image)[0]
        detections = sv.Detections.from_inference(results)
        return detections

    processor = DataProcessor()

    pipeline = InferencePipeline.init_with_custom_logic(
        video_reference="udp://127.0.0.1:9090",
        on_video_frame=inference_callback,
        on_prediction=processor.on_prediction,
    )

    pipeline.start()

    try:
        pipeline.join()
    except KeyboardInterrupt:
        pipeline.terminate()

if __name__ == "__main__":
    main()

My guess would be that InferencePipeline can't keep up but doesn't drop any requests.
Was it designed to support running models locally rather than on our servers?

[gmijo47]

Yeah i guess it doesn't drop any frames, i even got further and limited RTSP stream to 5 frames per second, and even on 1, but it can't keep up as you said.
It's is designed to run locally, and system is designed to be running independently of other services

[PawelPeczek-Roboflow]

ok, I may have found the cause of strange behaviour, please verify on your end - if that's the reason I will provide explanation

 pipeline = InferencePipeline.init_with_custom_logic(
        video_reference="udp://127.0.0.1:9090",
        on_video_frame=inference_callback,
        on_prediction=processor.on_prediction,
        source_buffer_filling_strategy=BufferFillingStrategy.DROP_OLDEST, <---
        source_buffer_consumption_strategy=BufferConsumptionStrategy.EAGER, <---
    )

[gmijo47]

Confirming, FPS is stable now, that was a problem, please provide explanation, so i can mark it as answer. Big thanks for your dedication and help.

[LinasKo]

Here's the gist: some cameras emit info about the FPS they produce. InferencePipeline uses that info when managing its processing queue. However, we found cases where the reported FPS is incorrect - it can even say, for a brief moment, that it's 10x of what it actually is, which wrecks the adaptive queue algorithm.

By setting the values Pawel suggested, you use a simpler algorithm that doesn't need this info from the camera, thereby being resistant to FPS fluctuations.

[PawelPeczek-Roboflow]

Yeah, we've implemented "adaptive" strategies of frames consumption (which based on declared FPS predicts if device is able to consume all frames of incoming stream) to address the following scenarios, which happened a lot for people in the past:

• consumption of hi-resolution streams at low-end devices (say 4K@60fps at Jetson Nano - then device is not able to decode all frames of on-line stream and perform inference due to hardware limitations - if not adaptive strategy forcing skipping frames decoding - then after short period of time, stream consumption is terminated with strange error of opencv backend)
• consumption of multiple streams which exceeds hardware capacity of device - the same happens when u are not able to keep up with decoding of all frames

The problem arises when stream is announced with wrong FPS value. At MacBooks that we tested - camera usually announce 30fps, but usually the pace of the stream is ~24 fps which is a problem causing the InferencePipeline to freeze from time to time. Sometimes, however, we've witnessed such absurd values of FPS as 200.

Now, the adaptive strategies are faulty in those scenarios and I expect u are not the only one who over time will experience that issue. We have couple of ideas on how to address it, will be reporting solutions in release notes when we fix the problem.