Skip to content

Releases: roboflow/inference

v0.9.8

29 Dec 19:37
@paulguerrie paulguerrie
v0.9.8
29693ed
Compare
Choose a tag to compare
v0.9.8

What's Changed

Highlights

Grounding DINO

Support for a new core model, Grounding DINO has been added. Grounding DINO is a zero-shot object detection model that you can use to identify objects in images and videos using arbitrary text prompts.

Inference SDK For Core Models

You can now use the Inference SDK with core models (like CLIP). No more complicated request and payload formatting. See the docs here.

Built In Jupyter Notebook

Roboflow Inference Server containers now include a built in Jupyter notebook for development and testing. This notebook can be accessed via the inference server landing page. To use it, go to localhost:9001 in your browser after starting an inference server. Then select "Jump Into An Inference Enabled Notebook". This will open a new tab with a Jupyterlab session, preloaded with example notebooks and all of the inference dependancies.

New Contributors

Full Changelog: v0.9.7...v0.9.8

Contributors

yeldarby, bigbitbus, and 3 other contributors
Assets 2
Loading

v0.9.7

20 Dec 21:13
@paulguerrie paulguerrie
v0.9.7
f64d58c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.7

What's Changed

Highlights

Stream Management API (Enterprise)

The stream management api is designed to cater to users requiring the execution of inference to generate predictions using Roboflow object-detection models, particularly when dealing with online video streams. It enhances the functionalities of the familiar inference.Stream() and InferencePipeline() interfaces, as found in the open-source version of the library, by introducing a sophisticated management layer. The inclusion of additional capabilities empowers users to remotely manage the state of inference pipelines through the HTTP management interface integrated into this package. More info.

Model Aliases

Some common public models now have convenient aliases! The with this release, the COCO base weights for YOLOv8 models can be accessed with user friendly model IDs like yolov8n-640. See all available model aliases here.

Other Improvements

  • Improved inference CLI commands
  • Unified batching APIs so that all model types can accept batch requests
  • Speed improvements for HTTP interface

New Contributors

Full Changelog: v0.9.6...v0.9.7

Contributors

josephofiowa, probicheaux, and 3 other contributors
Assets 2
Loading

v0.9.7rc2 - Test release for fix with CLI run problem

18 Dec 15:21
@PawelPeczek-Roboflow PawelPeczek-Roboflow
v0.9.7.rc2
49816f9
This commit was signed with the committer’s verified signature.
PawelPeczek-Roboflow Paweł Pęczek
GPG key ID: A20D4B3B43DF723D
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.7rc2 - Test release for fix with CLI run problem Pre-release
Pre-release 
v0.9.7.rc2

Fix makefile, such that onnx runtime is installed
Assets 2
Loading

v0.9.7rc1 - Test release for fix with CLI run problem

18 Dec 15:10
@PawelPeczek-Roboflow PawelPeczek-Roboflow
v0.9.7.rc1
0de3ce2
This commit was signed with the committer’s verified signature.
PawelPeczek-Roboflow Paweł Pęczek
GPG key ID: A20D4B3B43DF723D
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.7rc1 - Test release for fix with CLI run problem Pre-release
Pre-release 
v0.9.7.rc1

Fix problem with device request not being list
Assets 2
Loading

v0.9.6

13 Dec 18:12
@paulguerrie paulguerrie
v0.9.6
21ce7de
Compare
Choose a tag to compare
v0.9.6

What's Changed

Highlights

CogVLM

Inference server users can now run CogVLM for a fully self hosted, multimodal LLM. See the example here.

Slim Docker Images

For use cases that do not need Core Model functionality (e.g. CLIP), there are -slim docker images available which include fewer dependancies and are much smaller.

  • roboflow/roboflow-inference-server-cpu-slim
  • roboflow/roboflow-inference-server-gpu-slim

Breaking Changes

Infer API Update

The infer() method of Roboflow models now returns an InferenceResponse object instead of raw model output. This means that using models in application logic should feel similar to using models via the HTTP interface. In practice, programs that used the following pattern

...
model = get_roboflow_model(...)
results = model.infer(...)
results = model.make_response(...)
...

should be updated to

...
model = get_roboflow_model(...)
results = model.infer(...)
...

New Contributors

Full Changelog: v0.9.5...v0.9.6

Contributors

sberan, ShingoMatsuura, and 5 other contributors
Assets 2
Loading

v0.9.5

05 Dec 16:07
@paulguerrie paulguerrie
v0.9.5
8b4e413
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.5

0.9.5

Features, Fixes, and Improvements

Full Changelog: v0.9.3...v0.9.5.rc2

New inference.Stream interface

We are excited to introduce the upgraded version of our stream interface: InferencePipeline. Additionally, the WebcamStream class has evolved into a more versatile VideoSource.

This new abstraction is not only faster and more stable but also provides more granular control over the entire inference process.

Can I still use inference.Stream?

Absolutely! The old components remain unchanged for now. However, be aware that this abstraction is slated for deprecation over time. We encourage you to explore the new InferencePipeline interface and take advantage of its benefits.

What has been improved?

  • Performance: Experience A significant boost in throughput, up to 5 times, and improved latency for online inference on video streams using the YOLOv8n model.
  • Stability: InferencePipeline can now automatically re-establish a connection for online video streams if a connection is lost.
  • Prediction Sinks: Introducing prediction sinks, simplifying the utilization of predictions without the need for custom code.
  • Control Over Inference Process: InferencePipeline intelligently adapts to the type of video source, whether a file or stream. Video files are processed frame by frame, while online streams prioritize real-time processing, dropping non-real-time frames.
  • Observability: Gain insights into the processing state through events exposed by InferencePipeline. Reference implementations letting you to monitor processing are also available.

How to Migrate to the new Inference Stream interface?

You need to change a few lines of code to migrate to using the new Inference stream interface.

Below is an example that shows the old interface:

import inference

def on_prediction(predictions, image):
    pass

inference.Stream(
    source="webcam", # or "rstp://0.0.0.0:8000/password" for RTSP stream, or "file.mp4" for video
    model="rock-paper-scissors-sxsw/11", # from Universe
    output_channel_order="BGR",
    use_main_thread=True, # for opencv display
    on_prediction=on_prediction, 
)

Here is the same code expressed in the new interface:

from inference.core.interfaces.stream.inference_pipeline import InferencePipeline
from inference.core.interfaces.stream.sinks import render_boxes

pipeline = InferencePipeline.init(
    model_id="rock-paper-scissors-sxsw/11",
    video_reference=0,
    on_prediction=render_boxes,
)
pipeline.start()
pipeline.join()

Note the slight change in the on_prediction handler, from:

def on_prediction(predictions: dict, image: np.ndarray) -> None:
    pass

Into:

from inference.core.interfaces.camera.entities import VideoFrame

def on_prediction(predictions: dict, video_frame: VideoFrame) -> None:
    pass

Want to know more?

Here are useful references:

Parallel Robofolow Inference server

The Roboflow Inference Server supports concurrent processing. This version of the server accepts and processes requests asynchronously, running the web server, preprocessing, auto batching, inference, and post processing all in separate threads to increase server FPS throughput. Separate requests to the same model will be batched on the fly as allowed by $MAX_BATCH_SIZE, and then response handling will occurr independently. Images are passed via Python's SharedMemory module to maximize throughput.

These changes result in as much as a 76% speedup on one measured workload.

Note

Currently, only Object Detection, Instance Segmentation, and Classification models are supported by this module. Core models are not enabled.

Important

We require a Roboflow Enterprise License to use this in production. See inference/enterpise/LICENSE.txt for details.

How To Use Concurrent Processing

You can build the server using ./inference/enterprise/parallel/build.sh and run it using ./inference/enterprise/parallel/run.sh

We provide a container at Docker Hub that you can pull using docker pull roboflow/roboflow-inference-server-gpu-parallel:latest. If you are pulling a pinned tag, be sure to change the $TAG variable in run.sh.

This is a drop in replacement for the old server, so you can send requests using the same API calls you were using previously.

Performance

We measure and report performance across a variety of different task types by selecting random models found on Roboflow Universe.

Methodology

The following metrics are taken on a machine with eight cores and one gpu. The FPS metrics reflect best out of three trials. The column labeled 0.9.5.parallel reflects the latest concurrent FPS metrics. Instance segmentation metrics are calculated using "mask_decode_mode": "fast" in the request body. Requests are posted concurrently with a parallelism of 1000.

Results

Workspace Model Model Type split 0.9.5.rc FPS 0.9.5.parallel FPS
senior-design-project-j9gpp nbafootage/3 object-detection train 30.2 fps 44.03 fps
niklas-bommersbach-jyjff dart-scorer/8 object-detection train 26.6 fps 47.0 fps
geonu water-08xpr/1 instance-segmentation valid 4.7 fps 6.1 fps
university-of-bradford detecting-drusen_1/2 instance-segmentation train 6.2 fps 7.2 fps
fy-project-y9ecd cataract-detection-viwsu/2 classification train 48.5 fps 65.4 fps
hesunyu playing-cards-ir0wr/1 classification train 44.6 fps 57.7 fps

Contributors

sberan, yeldarby, and 7 other contributors
Assets 2
Loading

v0.9.4

27 Oct 15:51
@paulguerrie paulguerrie
v0.9.4
2a4d7d0
Compare
Choose a tag to compare
v0.9.4

Summary

This release includes new logic to validate models on load. This mitigates an issue seen when the model artifacts are corrupted during download.

Assets 2
Loading

v0.9.3

24 Oct 18:55
@paulguerrie paulguerrie
v0.9.3
a352d66
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.3

Summary
This release includes:

  • DocTR for detecting and recognizing text
  • Updates to our stream interface
  • Some bug fixes and other maintenance
Assets 2
Loading

v0.9.2

13 Oct 20:56
@paulguerrie paulguerrie
v0.9.2
22da70a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.2

Summary

  • Bugfix parsing base64 image string when source is browser (was adding unnecessary prefix)
  • Validate that equal or fewer than MAX_BATCH_SIZE images are being passed to object detection inference
  • Default MAX_BATCH_SIZE to infinity
  • Add batch regression tests
  • Add CLI to readme
  • Add generic stream object
  • Add preprocess/predict/postprocess to Clip to match base interface
  • Readme updates
  • Landing page
Assets 2
Loading

v0.9.1

09 Oct 22:04
@paulguerrie paulguerrie
v0.9.1
3ee92f4
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.9.1

Summary

This release includes a new stream interface, making it easy to run latency optimized inference and run custom callbacks (documentation coming soon)

Assets 2
Loading