"Applying Contextual Bandits for Recommendations with Tensorflow and TF-Agents" lab (exercise_movielens_notebook.ipynb) miserably fails with a dependency carnage

[MrCsabaToth]

This is part of the Prof. Machine Learning Engineer learning Path, and within that the Reinforcement Learning module. The lab: https://www.cloudskillsboost.google/course_sessions/2920313/labs/325116

So the lab instructs to instantiate a Workbook (Anaconda w Jupyterlab) with TF 2.6 (which is a lowest selectable TF version I think). The lab code itself want to install TF 2.4 though, which fails because that's incompatible with some other packages (?). Right away it starts with some spurious rm -rfs from the python3 venv's package internals.

ERROR: Could not find a version that satisfies the requirement tensorflow==2.4 (from versions: 2.5.0, 2.5.1, 2.5.2, 2.5.3, 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.7.0rc0, 2.7.0rc1, 2.7.0, 2.7.1, 2.7.2, 2.7.3, 2.7.4, 2.8.0rc0, 2.8.0rc1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0, 2.10.1, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0, 2.11.1, 2.12.0rc0, 2.12.0rc1, 2.12.0, 2.12.1, 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.13.1, 2.14.0rc0, 2.14.0rc1, 2.14.0)
ERROR: No matching distribution found for tensorflow==2.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.6.5 requires gast==0.4.0, but you have gast 0.3.3 which is incompatible.

Continuing the large import block fails complaining that it needs TF>=2.13. (Current is 2.6)

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[1], line 7
      4 from absl import flags
      6 import tensorflow as tf  # pylint: disable=g-explicit-tensorflow-version-import
----> 7 from tf_agents.bandits.agents import dropout_thompson_sampling_agent as dropout_ts_agent
      8 from tf_agents.bandits.agents import lin_ucb_agent
      9 from tf_agents.bandits.agents import linear_thompson_sampling_agent as lin_ts_agent

File /opt/conda/lib/python3.9/site-packages/tf_agents/__init__.py:70
     66 _ensure_tf_install()
     68 import sys as _sys
---> 70 from tf_agents import agents
     71 from tf_agents import bandits
     72 from tf_agents import distributions

File /opt/conda/lib/python3.9/site-packages/tf_agents/agents/__init__.py:17
      1 # coding=utf-8
      2 # Copyright 2020 The TF-Agents Authors.
      3 #
   (...)
     13 # See the License for the specific language governing permissions and
     14 # limitations under the License.
     16 """Module importing all agents."""
---> 17 from tf_agents.agents import data_converter
     18 from tf_agents.agents import tf_agent
     19 # TODO(b/130564501): Do not import classes directly, only expose modules.

File /opt/conda/lib/python3.9/site-packages/tf_agents/agents/data_converter.py:27
     23 import typing
     25 import tensorflow as tf
---> 27 from tf_agents.trajectories import policy_step
     28 from tf_agents.trajectories import time_step as ts
     29 from tf_agents.trajectories import trajectory

File /opt/conda/lib/python3.9/site-packages/tf_agents/trajectories/__init__.py:18
      1 # coding=utf-8
      2 # Copyright 2020 The TF-Agents Authors.
      3 #
   (...)
     13 # See the License for the specific language governing permissions and
     14 # limitations under the License.
     16 """Trajectories module."""
---> 18 from tf_agents.trajectories import policy_step
     19 from tf_agents.trajectories import time_step
     20 from tf_agents.trajectories import trajectory

File /opt/conda/lib/python3.9/site-packages/tf_agents/trajectories/policy_step.py:25
     23 import collections
     24 from typing import Any, Mapping, Text, NamedTuple, Optional, Union
---> 25 from tf_agents.typing import types
     28 ActionType = Union[types.NestedSpecTensorOrArray, types.NestedDistribution]
     31 class PolicyStep(
     32     NamedTuple(
     33         'PolicyStep',
   (...)
     36          ('info', types.NestedSpecTensorOrArray)
     37         ])):

File /opt/conda/lib/python3.9/site-packages/tf_agents/typing/types.py:29
     27 import numpy as np
     28 import tensorflow as tf
---> 29 import tensorflow_probability as tfp
     31 if sys.version_info < (3, 7):
     32   ForwardRef = typing._ForwardRef  # pylint: disable=protected-access

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/__init__.py:20
     15 """Tools for probabilistic reasoning in TensorFlow."""
     17 # Contributors to the `python/` dir should not alter this file; instead update
     18 # `python/__init__.py` as necessary.
---> 20 from tensorflow_probability import substrates
     21 # from tensorflow_probability.google import staging  # DisableOnExport
     22 # from tensorflow_probability.google import tfp_google  # DisableOnExport
     23 from tensorflow_probability.python import *  # pylint: disable=wildcard-import

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/substrates/__init__.py:17
      1 # Copyright 2019 The TensorFlow Probability Authors.
      2 #
      3 # Licensed under the Apache License, Version 2.0 (the "License");
   (...)
     13 # limitations under the License.
     14 # ============================================================================
     15 """TensorFlow Probability alternative substrates."""
---> 17 from tensorflow_probability.python.internal import all_util
     18 from tensorflow_probability.python.internal import lazy_loader  # pylint: disable=g-direct-tensorflow-import
     21 jax = lazy_loader.LazyLoader(
     22     'jax', globals(),
     23     'tensorflow_probability.substrates.jax')

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/python/__init__.py:138
    135 if _tf_loaded():
    136   # Non-lazy load of packages that register with tensorflow or keras.
    137   for pkg_name in _maybe_nonlazy_load:
--> 138     dir(globals()[pkg_name])  # Forces loading the package from its lazy loader.
    141 all_util.remove_undocumented(__name__, _lazy_load + _maybe_nonlazy_load)

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/python/internal/lazy_loader.py:57, in LazyLoader.__dir__(self)
     56 def __dir__(self):
---> 57   module = self._load()
     58   return dir(module)

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/python/internal/lazy_loader.py:37, in LazyLoader._load(self)
     35 """Load the module and insert it into the parent's globals."""
     36 if callable(self._on_first_access):
---> 37   self._on_first_access()
     38   self._on_first_access = None
     39 # Import the target module and insert it into the parent's namespace

File /opt/conda/lib/python3.9/site-packages/tensorflow_probability/python/__init__.py:59, in _validate_tf_environment(package)
     55 #   required_tensorflow_version = '1.15'  # Needed internally -- DisableOnExport
     57   if (distutils.version.LooseVersion(tf.__version__) <
     58       distutils.version.LooseVersion(required_tensorflow_version)):
---> 59     raise ImportError(
     60         'This version of TensorFlow Probability requires TensorFlow '
     61         'version >= {required}; Detected an installation of version {present}. '
     62         'Please upgrade TensorFlow to proceed.'.format(
     63             required=required_tensorflow_version,
     64             present=tf.__version__))
     66   if (package == 'mcmc' and
     67       tf.config.experimental.tensor_float_32_execution_enabled()):
     68     # Must import here, because symbols get pruned to __all__.
     69     import warnings

ImportError: This version of TensorFlow Probability requires TensorFlow version >= 2.13; Detected an installation of version 2.6.5. Please upgrade TensorFlow to proceed.

After I perform a pip install --user tensorflow==2.16, unsurprisingly it still fails:

2023-10-03 01:38:19.732250: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/opt/conda/lib/python3.9/site-packages/tf_agents/typing/types.py:77: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.
  Bool = Union[bool, np.bool, Tensor, Array]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 7
      4 from absl import flags
      6 import tensorflow as tf  # pylint: disable=g-explicit-tensorflow-version-import
----> 7 from tf_agents.bandits.agents import dropout_thompson_sampling_agent as dropout_ts_agent
      8 from tf_agents.bandits.agents import lin_ucb_agent
      9 from tf_agents.bandits.agents import linear_thompson_sampling_agent as lin_ts_agent

File /opt/conda/lib/python3.9/site-packages/tf_agents/__init__.py:70
     66 _ensure_tf_install()
     68 import sys as _sys
---> 70 from tf_agents import agents
     71 from tf_agents import bandits
     72 from tf_agents import distributions

File /opt/conda/lib/python3.9/site-packages/tf_agents/agents/__init__.py:17
      1 # coding=utf-8
      2 # Copyright 2020 The TF-Agents Authors.
      3 #
   (...)
     13 # See the License for the specific language governing permissions and
     14 # limitations under the License.
     16 """Module importing all agents."""
---> 17 from tf_agents.agents import data_converter
     18 from tf_agents.agents import tf_agent
     19 # TODO(b/130564501): Do not import classes directly, only expose modules.

File /opt/conda/lib/python3.9/site-packages/tf_agents/agents/data_converter.py:27
     23 import typing
     25 import tensorflow as tf
---> 27 from tf_agents.trajectories import policy_step
     28 from tf_agents.trajectories import time_step as ts
     29 from tf_agents.trajectories import trajectory

File /opt/conda/lib/python3.9/site-packages/tf_agents/trajectories/__init__.py:18
      1 # coding=utf-8
      2 # Copyright 2020 The TF-Agents Authors.
      3 #
   (...)
     13 # See the License for the specific language governing permissions and
     14 # limitations under the License.
     16 """Trajectories module."""
---> 18 from tf_agents.trajectories import policy_step
     19 from tf_agents.trajectories import time_step
     20 from tf_agents.trajectories import trajectory

File /opt/conda/lib/python3.9/site-packages/tf_agents/trajectories/policy_step.py:25
     23 import collections
     24 from typing import Any, Mapping, Text, NamedTuple, Optional, Union
---> 25 from tf_agents.typing import types
     28 ActionType = Union[types.NestedSpecTensorOrArray, types.NestedDistribution]
     31 class PolicyStep(
     32     NamedTuple(
     33         'PolicyStep',
   (...)
     36          ('info', types.NestedSpecTensorOrArray)
     37         ])):

File /opt/conda/lib/python3.9/site-packages/tf_agents/typing/types.py:77
     74 NestedSpecTensorOrArray = Union[NestedSpec, NestedTensor, NestedArray]
     76 Int = Union[int, np.int16, np.int32, np.int64, Tensor, Array]
---> 77 Bool = Union[bool, np.bool, Tensor, Array]
     79 Float = Union[float, np.float16, np.float32, np.float64, Tensor, Array]
     80 FloatOrReturningFloat = Union[Float, Callable[[], Float]]

File /opt/conda/lib/python3.9/site-packages/numpy/__init__.py:305, in __getattr__(attr)
    300     warnings.warn(
    301         f"In the future `np.{attr}` will be defined as the "
    302         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
    304 if attr in __former_attrs__:
--> 305     raise AttributeError(__former_attrs__[attr])
    307 # Importing Tester requires importing all of UnitTest which is not a
    308 # cheap import Since it is mainly used in test suits, we lazy import it
    309 # here to save on the order of 10 ms of import time for most users
    310 #
    311 # The previous way Tester was imported also had a side effect of adding
    312 # the full `numpy.testing` namespace
    313 if attr == 'testing':

AttributeError: module 'numpy' has no attribute 'bool'.
`np.bool` was a deprecated alias for the builtin `bool`. To avoid this error in existing code, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

I tried the solution notebook as well just to be sure, same thing.
So in a nutshell: the notebook fails at the beginning.