python: migrate applications to use IoT3 Core SDK #149

ymorin-orange · 2024-08-14T13:51:38Z

WIP: needs #154 to be merged first

Features:

Fix IoT3 Core SDK
- Fixes python/iot3: connecting to a UNIX socket raises an exception #155
- Fixes python/iot3: subscribing to all topics at once raises an exception #156
Use IoT3 Core SDK
- use the MQTT wrapper
- implement telemetry

Prepare an OpenTelemetry collector on localhost:

$ docker container run \
    --rm \
    -p 16686:16686 \
    -p 4318:4318 \
    jaegertracing/all-in-one:1.58

then open a browser on the Jaegger UI:

http://localhost:16686/

Prepare an MQTT broker on localhost (adapt based on your distribution):
```
$ sudo systemctl start mosquitto.service
```
and then start an MQTT client to dump the messages on the broker:
```
$ mosquitto_sub -V 5 -t '#' -F '%j' |jq .
```
Start a gpsd daemon
1. create a fake NEMA log, e.g. in file NMEA.log, e.g. the sample available
  on wikipedia
2. start a gpsd daemon sending fake data:
```
$ TMPDIR=/tmp gpsfake -q -P 2947 -u -n -c 0.1 NMEA.log
```

Test IoT3 Core SDK:

if your python is older than 3.11, install an environment with python 3.11
(you'll need to have a development environment with gcc and at least libffi-dev
and libssl-dev installed; adapt to your distribution):

$ wget https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tar.xz
$ tar xJf Python-3.11.9.tar.xz
$ cd Python-3.11.9
$ ./configure --prefix "${HOME}/tmp/python/3.11" --with-openssl=/usr
$ make install
$ export PATH="${HOME}/tmp/python/3.11/bin:${PATH}"

create a /venv/ in a writable location, and activate it:

$ python3.11 -m venv ~/tmp/its
$ . ~/tmp/its/bin/activate

install the ITS components:

(venv) cd python
(venv) pip3 --disable-pip-version-check install \
        --no-binary ':all:' \
        ./iot3 \
        ./its-interqueuemanager \
        ./its-quadkeys \
        ./its-status \
        ./its-vehicle

Note: its-info is not modified, so tested is not needed, so
it's not istalled.

test the its-status component:

create a configuration file, e.g. /tmp/its-status.cfg, with this content:

[generic]
id = test-status
frequency = 1.0

[stdout]
enabled = false

[mqtt]
enabled = true
host = localhost
port = 1883
username = USERNAME
password = PASSWORD
client_id = its-status
topic = status/system

[system]
data-dir = /

[gnss]
host = localhost
port = 2947
persistence = 5.0

Note: adapt USERNAME and PASSWORD with the appropriate credentials,
and adapt host and port is you are usign a remote broker and/or it
is listening on a non-standard port.

start its-status with that configuration file:
```
(venv) $ its-status -c /tmp/its-status.cfg
```

test the its-vehicle component:

create a configuration file, e.g. /tmp/its-vehicle.cfg, with this content:

[general]
instance-id = test-me
type = CAM
report-freq = 0.5
topic-pub-prefix = PROJECT/inQueue/SUFFIX/
topic-sub-prefix = PROJECT/outQueue/SUFFIX/
depth = 22
messages = cam cpm denm
depth-sub-cam = 18
depth-sub-cpm = 18
depth-sub-denm = 15
speed-thresholds = 8.33 19.44 27.78

[telemetry]
endpoint = http://localhost:4318

[broker.main]
host = localhost
port = 1883
username = USERNAME
password = PASSWORD

[gpsd]
port = 2947
persistence = 10.0

start its-vehicle with that configuration file:
```
(venv) $ its-status -c /tmp/its-vehicle.cfg
```

test the its-iqm component:

create a configuration file, e.g. /tmp/its-iqm.cfg, with this content:

[general]
instance-id = ora_geo_1234
prefix = PROJECT
suffix = SUFFIX

[telemetry]
endpoint = http://localhost:4318

[local]
host = localhost
port = 1883

start its-iqm with that configuration file:
```
(venv) $ its-iqm -c /tmp/its-iqm.cfg
```

Look up traces in the Jaeger UI
Stop the ITS components with Ctrl-C on each

Expected results

Pre-requisites are all met.

The Jaegger UI is displayed in your browser
The Mosqiutto server and clients are running
The gpsd daemon is running

The ITS components behave as expected;

the contaienr starts
the venv is activated
the ITS components are installed

the its-status component emits status messages, and the verbose dump
from mosquitto_sub would look like:

{
  "tst": "2024-09-20T08:01:04.047514Z+0200",
  "topic": "status/system",
  "qos": 0,
  "retain": 0,
  "payloadlen": 836,
  "payload": {
    "time_sources": null,
    "gnss": {
      "software": "gpsd 3.22",
      "model": "NMEA0183",
      "mode": 3
    },
    "cellular": [],
    "version": "1.2.0",
    "type": "status",
    "id": "test-status",
    "system": {
      "type": "obu",
      "hardware": "Unknown",
      "os_release": {
        "PRETTY_NAME": "Ubuntu 22.04.5 LTS",
        "NAME": "Ubuntu",
        "VERSION_ID": "22.04",
        "VERSION": "22.04.5 LTS (Jammy Jellyfish)",
        "VERSION_CODENAME": "jammy",
        "ID": "ubuntu",
        "ID_LIKE": "debian",
        "HOME_URL": "https://www.ubuntu.com/",
        "SUPPORT_URL": "https://help.ubuntu.com/",
        "BUG_REPORT_URL": "https://bugs.launchpad.net/ubuntu/",
        "PRIVACY_POLICY_URL": "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy",
        "UBUNTU_CODENAME": "jammy"
      },
      "cpu_load": [
        0.650390625,
        0.8369140625,
        0.9921875
      ],
      "memory": [
        67095187456,
        53034876928
      ],
      "storage": [
        981132795904,
        506785951744
      ]
    },
    "timestamp": 1726812064.0466053
  }
}

the its-vehicle component emits status messages, and the verbose dump
from mosquitto_sub would look like (notice the traceparent user
property):

{
  "tst": "2024-09-20T08:05:52.970202Z+0200",
  "topic": "PROJECT/inQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/2/1/0/3/1/0/3/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-057264776dc53a957fbb14530684cc45-47096d87822883fa-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812352969,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 47411,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436189667,
          "longitude": 15019833,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 3007,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}

the its-iqm component emits status messages, and the verbose dump from
mosquitto_sub would look like (notice the traceparent user property,
and the different topics: inQueue, outQueue, and interQuweue):

{
  "tst": "2024-09-20T08:08:49.388984Z+0200",
  "topic": "PROJECT/inQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-e6386928b78d3f87f403ef63ce71c33e-4c1e46c2e528c427-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}
{
  "tst": "2024-09-20T08:08:49.390629Z+0200",
  "topic": "PROJECT/outQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-f431be1d1d66f07b110507d58185a3fd-4851c79cb4026051-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}
{
  "tst": "2024-09-20T08:08:49.390795Z+0200",
  "topic": "PROJECT/interQueue/SUFFIX/cam/test-me/1/2/0/2/2/2/0/3/0/2/2/3/0/2/0/2/2/3/0/0/2/3",
  "qos": 0,
  "retain": 0,
  "payloadlen": 641,
  "properties": {
    "user-properties": {
      "traceparent": "00-f431be1d1d66f07b110507d58185a3fd-2a7a8db15dec456d-00"
    }
  },
  "payload": {
    "type": "cam",
    "origin": "self",
    "version": "1.1.3",
    "source_uuid": "test-me",
    "timestamp": 1726812529387,
    "message": {
      "protocol_version": 1,
      "station_id": 7603010,
      "generation_delta_time": 27284,
      "basic_container": {
        "station_type": 5,
        "reference_position": {
          "latitude": 436209667,
          "longitude": 14956333,
          "altitude": 0
        },
        "confidence": {
          "position_confidence_ellipse": {
            "semi_major_confidence": 10,
            "semi_minor_confidence": 50,
            "semi_major_orientation": 1
          },
          "altitude": 1
        }
      },
      "high_frequency_container": {
        "heading": 2729,
        "speed": 998,
        "longitudinal_acceleration": 161,
        "drive_direction": 0,
        "vehicle_length": 40,
        "vehicle_width": 20,
        "confidence": {
          "heading": 2,
          "speed": 3,
          "vehicle_length": 0
        }
      }
    }
  }
}

The Jaegger UI displays traces for the its-vehicle and its-interqueuemanager
services, with appropriate links between traces.
The ITS components all stop

Although the unix socket transport does not require a TCP port, paho-mqtt expexts that a valid TCP port be specified when connecting. Technically, specifying a port is not necessary in that case, so we could in theory change our call to connect_async() to not pass a port at al. However, that would make the code a bit more complex. So, to simplify things, just use a valid TCP port, i.e. one that is strictly greater than 0. We could have decided to use the default port, 1883, but that would be a bit misleading. Instead, we just use 1, which is odd enough that it will at least raise an eyebrow to an unsuspecting reviewer. Signed-off-by: Yann E. MORIN <[email protected]>

The unsubscribe() method expects keyword arguments, not positional ones. Signed-off-by: Yann E. MORIN <[email protected]>

The vareious ITS clients (its-vehicle, its-iqm...) currently use the APIs of paho.mqtt 1.6.1, while the newly-introduce IoT3 Core SDK uses those of 2.1.0. In paho.mqtt 2.0.x, there was a breaking API change, so a client written against paho.mqtt 1.6.1 (or earlier) could not be used as-is with paho.mqtt 2.0.x. but in paho.mqtt 2.1.0, there was another API change that made it possible to use clients written against 1.6.1, without any modification, while also alowing clients written againt the 2.0.0 API as well. So we can accept anything from 1.6.1 onward, except for versions 2.0.x. Signed-off-by: Yann E. MORIN <[email protected]>

Signed-off-by: Yann E. MORIN <[email protected]>

ymorin-orange requested review from tigroo and nbuffon August 14, 2024 13:52

ymorin-orange mentioned this pull request Aug 14, 2024

python: implement IoT3 Core SDK #144

Merged

ymorin-orange force-pushed the yem/its-iot3 branch 2 times, most recently from 3135f4d to 865cce9 Compare September 9, 2024 13:51

tigroo removed their request for review September 11, 2024 07:49

tigroo added enhancement New feature or request Python Python code labels Sep 11, 2024

tigroo added this to the Sprint 2 milestone Sep 11, 2024

ymorin-orange added 17 commits September 11, 2024 13:35

python/iot3/core: fix unsubscribing to all MQTT topics

1a6236a

The unsubscribe() method expects keyword arguments, not positional ones. Signed-off-by: Yann E. MORIN <[email protected]>

python/status: use iot3.core.mqtt

a18dd54

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: no one ever join()s the authority threads

cdcfbdb

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: use iot3.core.mqtt for queue-manager

0d09fa9

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: use iot3.core.mqtt for MQTT authority client

d9016c3

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: drop custom MQTT client

f444f21

Signed-off-by: Yann E. MORIN <[email protected]>

python/iqm: add telemetry

f472826

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop tracking from ITS message

d71d5de

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: don't instanciate dummy custom tracker

f53d1b8

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop custom, dummy tracker

fede7cb

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: don't use abbrev() which is going to disapear

c262b0b

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: only start() clients once all have been created

e6f8a02

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: insert trampoline on message reception

e09a884

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: use iot3.core.mqtt

16e196f

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: drop our custom MQTT Client

b72feba

Signed-off-by: Yann E. MORIN <[email protected]>

python/vehicle: add telemetry

102170e

Signed-off-by: Yann E. MORIN <[email protected]>

ymorin-orange force-pushed the yem/its-iot3 branch from 865cce9 to 102170e Compare September 11, 2024 13:49

ymorin-orange self-assigned this Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python: migrate applications to use IoT3 Core SDK #149

python: migrate applications to use IoT3 Core SDK #149

ymorin-orange commented Aug 14, 2024 •

edited

Loading

python: migrate applications to use IoT3 Core SDK #149

Are you sure you want to change the base?

python: migrate applications to use IoT3 Core SDK #149

Conversation

ymorin-orange commented Aug 14, 2024 • edited Loading

ymorin-orange commented Aug 14, 2024 •

edited

Loading