[Bug] Dependency using a deprecated and removed module (pkg_resources)

[brokensound77]

Describe the Bug

The dependency marshmallow_jsonschema depends on pkg_resources, which has been deprecated and removed as of python 3.12. This is especially problematic since we enforce py312 in this repo.

An issue exists in the upstream library to update, but no progress has been made. The last update was over a year ago, so unsure if it is being maintained.

Per https://docs.python.org/3/whatsnew/3.12.html:

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

pkg_resources was moved to within setuptools, so the simplest fix would be to add setuptools as a required dependency

Although it is included in the Makefile to install setuptools as a prepatory step, this resolves the issue when make is unavailable or when this is being installed and imported as a dependency.

Full error (occurring on execution not install):

import detection_rules

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/__init__.py", line 13, in <module>
    from . import (  # noqa: E402
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/devtools.py", line 32, in <module>
    from . import attack, rule_loader, utils
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/rule_loader.py", line 20, in <module>
    from .mappings import RtaMappings
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/mappings.py", line 12, in <module>
    from .rule import TOMLRule
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/rule.py", line 33, in <module>
    from .mixins import MarshmallowDataclassMixin, StackCompatMixin
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/detection_rules/mixins.py", line 15, in <module>
    import marshmallow_jsonschema
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jib/PycharmProjects/detection-rules-fork/venv312test/lib/python3.12/site-packages/marshmallow_jsonschema/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution
  File "/Applications/PyCharm.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'pkg_resources'

---

IINM, it is also the actual fix to the distutils problem outline in troubleshooting.md

---

The best fix for both of these would be to remove our dependency on the deprecated libraries, but that would require upstream lib changes.

From the pkg_resources docs:

Use of pkg_resources is deprecated in favor of importlib.resources, importlib.metadata and their backports (importlib_resources, importlib_metadata). Some useful APIs are also provided by packaging (e.g. requirements and version parsing). Users should refrain from new usage of pkg_resources and should work to port to importlib-based solutions.

To Reproduce

1. pip install .
2. import detection_rules

Expected Behavior

No response

Screenshots

No response

Desktop - OS

None

Desktop - Version

No response

Additional Context

No response

[shashank-elastic]

The error is reproducible on pip install .

Created a new venv

detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  [email protected] 
❯ python3 -m venv .venv_new    
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  [email protected] 
❯ source .venv_new/bin/activate
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv_new) on ☁️  [email protected] 
❯ 

pip install

❯ pip install .
Looking in indexes: https://pypi.org/simple, https://shashank.suryanarayana%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing /Users/shashankks/elastic_workspace/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/jk/t_tlgnwx4w998xqw3_kjzyx00000gn/T/pip-install-s6kwdf_x/detection-rules-kql_23861e9674b34fbcbe7eb5703d2e834c
Successfully built detection_rules detection-rules-kibana detection-rules-kql
Installing collected packages: pytoml, lark-parser, jsl, XlsxWriter, urllib3, typing-extensions, typeguard, toml, semver, rpds-py, PyYAML, packaging, mypy-extensions, marko, idna, eql, Click, charset-normalizer, certifi, attrs, typing-inspect, requests, referencing, marshmallow, elastic-transport, detection-rules-kql, marshmallow-union, marshmallow-jsonschema, marshmallow-dataclass, jsonschema-specifications, elasticsearch, jsonschema, detection-rules-kibana, detection_rules
Successfully installed Click-8.1.7 PyYAML-6.0.2 XlsxWriter-3.2.0 attrs-24.2.0 certifi-2024.8.30 charset-normalizer-3.3.2 detection-rules-kibana-0.4.0 detection-rules-kql-0.1.7 detection_rules-0.1.0 elastic-transport-8.15.0 elasticsearch-8.12.1 eql-0.9.19 idna-3.10 jsl-0.2.4 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 lark-parser-0.12.0 marko-2.0.3 marshmallow-3.21.3 marshmallow-dataclass-8.6.1 marshmallow-jsonschema-0.13.0 marshmallow-union-0.1.15.post1 mypy-extensions-1.0.0 packaging-24.1 pytoml-0.1.21 referencing-0.35.1 requests-2.31.0 rpds-py-0.20.0 semver-3.0.2 toml-0.10.2 typeguard-3.0.2 typing-extensions-4.10.0 typing-inspect-0.9.0 urllib3-2.2.3
(.venv) 

Error

    import marshmallow_jsonschema
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/Users/shashankks/elastic_workspace/detection-rules/.venv_new/lib/python3.12/site-packages/marshmallow_jsonschema/__init__.py", line 1, in <module>
    from pkg_resources import get_distribution
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pkg_resources'
>>> 

Also noticed we don't notice this error on pip install ".[dev]"

Created a new venv

❯ python3 -m venv .venv_new1   
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 on ☁️  [email protected] 
❯ source .venv_new1/bin/activate
(.venv) 
detection-rules on  main [$?] is 📦 v0.1.0 via 🐍 v3.12.5 (.venv_new1) on ☁️  [email protected] 
❯ 

pip install dev

❯ pip install ".[dev]"
Looking in indexes: https://pypi.org/simple, https://shashank.suryanarayana%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing /Users/shashankks/elastic_workspace/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/jk/t_tlgnwx4w998xqw3_kjzyx00000gn/T/pip-install-8kydmjrf/detection-rules-kql_fb529215442b43a8b907f5de29561a96
Successfully installed Click-8.1.7 Deprecated-1.2.14 PyGithub-2.2.0 PyYAML-6.0.2 XlsxWriter-3.2.0 attrs-24.2.0 certifi-2024.8.30 cffi-1.17.1 cfgv-3.4.0 charset-normalizer-3.3.2 cryptography-43.0.1 detection-rules-kibana-0.4.0 detection-rules-kql-0.1.7 detection_rules-0.1.0 distlib-0.3.8 elastic-transport-8.15.0 elasticsearch-8.12.1 eql-0.9.19 filelock-3.16.1 flake8-7.0.0 identify-2.6.1 idna-3.10 iniconfig-2.0.0 jsl-0.2.4 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 lark-parser-0.12.0 marko-2.0.3 marshmallow-3.21.3 marshmallow-dataclass-8.6.1 marshmallow-jsonschema-0.13.0 marshmallow-union-0.1.15.post1 mccabe-0.7.0 mypy-extensions-1.0.0 nodeenv-1.8.0 packaging-24.1 pep8-naming-0.13.0 platformdirs-4.3.6 pluggy-1.5.0 pre-commit-3.6.2 pycodestyle-2.11.1 pycparser-2.22 pyflakes-3.2.0 pyjwt-2.9.0 pynacl-1.5.0 pytest-8.3.3 pytoml-0.1.21 referencing-0.35.1 requests-2.31.0 rpds-py-0.20.0 semver-3.0.2 setuptools-75.1.0 toml-0.10.2 typeguard-3.0.2 typing-extensions-4.10.0 typing-inspect-0.9.0 urllib3-2.2.3 virtualenv-20.26.5 wrapt-1.16.0
(.venv) 

Note setuptools-75.1.0 being installed as part of dev

No Error

# code object from '/Users/shashankks/elastic_workspace/detection-rules/detection_rules/__pycache__/ml.cpython-312.pyc'
import 'detection_rules.ml' # <_frozen_importlib_external.SourceFileLoader object at 0x10e348bf0>
import 'detection_rules' # <_frozen_importlib_external.SourceFileLoader object at 0x1005bf350>
>>> 

We can freeze setuptools==75.1.0 as a dependency non dev installs for pip @Mikaayenson

But if we want to remove our dependancy on pkg_resources, this would take some more cycles to discuss do we wanna do that , and how do we wanna do that.