jammy: _openssl.abi3.so: undefined symbol: FIPS_mode

This might be related to the fact that the charm is built on 20.04 but then runs on 22.04.

I ran into the same problem here, I tried to build on 22.04 (current
master branch), but charmcraft 1.5/stable does not support built
on 22.04 and the current charmcraft.yaml file does not work for
a newer version of charmcraft.

This is a snippet from my bundle where I use jammy series.
```yaml
octavia-diskimage-retrofit:
    charm: ch:octavia-diskimage-retrofit
    channel: yoga/edge
    options:
      amp-image-tag: 'octavia-amphora'
      retrofit-series: jammy
```

This issue needs further fixing in order for the charm to work on Jammy. At present the charm pins cryptography<3.4 in order to avoid the Rust build time dependency.

Actually the package in Ubuntu does something similar but also contains patches to allow cryptography 3.4.x to work with OpenSSL 3.x [0].

In a charms wheelhouse we do not have the luxury or desire to carry patches for Python packages, so I think what we need to do is unpin cryptography and just add the required build dependencies.

Given reactive charms are already compiling C source dependencies avoiding dependencies written in other languages feel artificial to me.

We of course want to stop building source packages on install altogether, but that is a longer canvas to bleach.

0: https://git.launchpad.net/ubuntu/+source/python-cryptography/tree/debian/patches/openssl3?h=ubuntu/jammy

Additionally, at least charmcraft 1.7 is needed for a build on 22.04.

Reviewed: https://review.opendev.org/c/openstack/charm-octavia-diskimage-retrofit/+/852747
Committed: https://opendev.org/openstack/charm-octavia-diskimage-retrofit/commit/2fb0cf125028b3e164741afcdbff466038de134a
Submitter: "Zuul (22348)"
Branch: master

commit 2fb0cf125028b3e164741afcdbff466038de134a
Author: Frode Nordahl <email address hidden>
Date: Wed Aug 10 16:44:55 2022 +0200

    Build separately for each supported series and use binary builds

    Charms for OpenStack Yoga supports both Ubuntu Focal and Jammy
    which means Python 3.8 and Python 3.10. Managing dependencies
    across those two versions is non-trivial and we need to build
    the charm on the series the charm is supposed to support.

    Switch to using a binary build which allows pip's dependency
    resolution to work.

    Update charm to consume the 1.0/stable track for the snap.

    Bundles:
    - Drop the renaming of the charm artifact and reference the
      series specific charm artifact in the bundles instead.
    - Fix gss mirror list, jammy retrofit-series and use 8.0/stable
      for MySQL.

    Closes-Bug: #1981334
    Closes-Bug: #1970653
    Closes-Bug: #1975491
    Change-Id: I8c924038ee1c5ff258c41a44ad14ebc86a107b1b

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia-diskimage-retrofit/+/860979

Reviewed: https://review.opendev.org/c/openstack/charm-octavia-diskimage-retrofit/+/860979
Committed: https://opendev.org/openstack/charm-octavia-diskimage-retrofit/commit/5fd698d29757df7603691078fff80cb38edb9cb6
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 5fd698d29757df7603691078fff80cb38edb9cb6
Author: Frode Nordahl <email address hidden>
Date: Wed Aug 10 16:44:55 2022 +0200

    Build separately for each supported series and use binary builds

    Charms for OpenStack Yoga supports both Ubuntu Focal and Jammy
    which means Python 3.8 and Python 3.10. Managing dependencies
    across those two versions is non-trivial and we need to build
    the charm on the series the charm is supposed to support.

    Switch to using a binary build which allows pip's dependency
    resolution to work.

    Update charm to consume the 1.0/stable track for the snap.

    Bundles:
    - Drop the renaming of the charm artifact and reference the
      series specific charm artifact in the bundles instead.
    - Fix gss mirror list, jammy retrofit-series and use 8.0/stable
      for MySQL.

    Closes-Bug: #1981334
    Closes-Bug: #1970653
    Closes-Bug: #1975491
    Change-Id: I8c924038ee1c5ff258c41a44ad14ebc86a107b1b
    (cherry picked from commit 2fb0cf125028b3e164741afcdbff466038de134a)

I'm hitting the same issue with ironic-conductor charm on jammy. Here's the result of running juju action `set-temp-url-secret`:

```
$ juju run-action ironic-conductor/0 set-temp-url-secret --wait
unit-ironic-conductor-0:
  UnitId: ironic-conductor/0
  id: "40"
  message: '/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so:
    undefined symbol: FIPS_mode'
  results: {}
  status: failed
  timing:
    completed: 2022-12-01 15:23:43 +0000 UTC
    enqueued: 2022-12-01 15:23:41 +0000 UTC
    started: 2022-12-01 15:23:41 +0000 UTC
```

This appears to affect charm-ironic-conductor for releases prior to zed.

we might need to convert ironic-conductor to a "binary charm".

> I'm hitting the same issue with ironic-conductor charm on jammy.

is this jammy-yoga?

Looking at this recent CI run[0] of jammy-yoga, allegedly the set_temp_url_secret() setup function ran successfully:

2023-02-18 03:32:42.452818 | focal-medium | Configure zaza.openstack.charm_tests.ironic.setup.set_temp_url_secret:
2023-02-18 03:32:42.452826 | focal-medium | Start: 1676689848.988125
2023-02-18 03:32:42.452834 | focal-medium | Finish: 1676689852.602591
2023-02-18 03:32:42.452842 | focal-medium | Elapsed Time: 3.6144659519195557
2023-02-18 03:32:42.452850 | focal-medium | PCT Of Run Time: 1

Although the juju status suggests something different:

ironic-conductor/0* blocked idle 12 172.16.0.163 invalid enabled-deploy-interfaces config, run set-temp-url-secret action on leader to enable direct deploy method

Since the deployment succeeded there is no juju-crashdump available to inspection, this will need to be reproduced locally (the false positive of the action execution and the action failure reported in comment #9 )

[0] https://review.opendev.org/c/openstack/charm-ironic-conductor/+/872438/1#message-e0a1db58a3c6f20205d10ea32eca229b1e91b765

I'm hitting the same issue installing keystone-saml-mellon from yoga/stable on jammy. Subscribing field-critical because it is blocking customer deployment.

```
$ juju debug-log -i keystone-saml-mellon/4
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.migrationminion migration phase is now: NONE
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.logger logger worker started
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.upgrader no waiter, upgrader is done
unit-keystone-saml-mellon-4: 22:58:30 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.uniter unit "keystone-saml-mellon/4" started
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.uniter resuming charm install
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/keystone-saml-mellon-31 from API server
unit-keystone-saml-mellon-4: 22:58:31 INFO juju.worker.uniter hooks are retried true
unit-keystone-saml-mellon-4: 22:58:31 INFO juju.worker.uniter.storage initial storage attachments ready
unit-keystone-saml-mellon-4: 22:58:31 INFO juju.worker.uniter found queued "install" hook
unit-keystone-saml-mellon-4: 22:58:30 INFO juju Starting unit workers for "keystone-saml-mellon/4"
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.apicaller [79829f] "unit-keystone-saml-mellon-4" successfully connected to "10.77.0.53:17070"
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.apicaller [79829f] password changed for "unit-keystone-saml-mellon-4"
unit-keystone-saml-mellon-4: 22:58:30 INFO juju.worker.apicaller [79829f] "unit-keystone-saml-mellon-4" successfully connected to "10.77.0.53:17070"
unit-keystone-saml-mellon-4: 23:03:20 INFO unit.keystone-saml-mellon/4.juju-log Reactive main running for hook install
unit-keystone-saml-mellon-4: 23:03:20 INFO unit.keystone-saml-mellon/4.juju-log Initializing Leadership Layer (is leader)
unit-keystone-saml-mellon-4: 23:03:20 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: reactive/layer_openstack.py:15:default_install
unit-keystone-saml-mellon-4: 23:03:20 INFO unit.keystone-saml-mellon/4.juju-log Installing ['libapache2-mod-auth-mellon'] with options: ['--option=Dpkg::Options::=--force-confold']
unit-keystone-saml-mellon-4: 23:03:29 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: reactive/keystone_saml_mellon_handlers.py:29:keystone_departed
unit-keystone-saml-mellon-4: 23:03:29 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: reactive/keystone_saml_mellon_handlers.py:69:assess_status
unit-keystone-saml-mellon-4: 23:03:29 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
unit-keystone-saml-mellon-4: 23:03:29 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: hooks/relations/keystone-fid-service-provider/provides.py:66:departed:keystone-fid-service-provider
unit-keystone-saml-mellon-4: 23:03:29 INFO unit.keystone-saml-mellon/4.juju-log Invoking reactive handler: hooks/relations/juju-info/requires.py:24:broken:container
unit-key...

field-critical is for both ironic-conductor and keystone-saml-mellon. We have both of these applications in a customer deployment at the moment.

It looks like charm-ironic-conductor and charm-keystone-saml-mellon both build with --binary-wheels in their stable/zed branches so this might just need one more cherry-pick to stable/yoga for now.

As a work-around I think you should be able to use zed charm tracks with openstack-origin set to yoga.

Corey, thank you for looking into it!

I have configured `channel: zed/stable` and left openstack-origin unchanged (it defaults to `distro` and I'm deploying on jammy, so it should mean yoga) for both keystone-saml-mellon and ironic-conductor.

For keystone-saml-mellon it did the job. I was able to successfully deploy the charm.

But the workaround does not work for ironic-conductor. When I run `set-temp-url-secret` action, it fails, and I get the following in the debug-log:

```
$ juju debug-log -i ironic-conductor/0
[...]
unit-ironic-conductor-0: 23:37:02 ERROR unit.ironic-conductor/0.juju-log action "set-temp-url-secret" failed: "local variable 'keystone_session' referenced before assignment" "Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 112, in main
    action(args)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 60, in set_temp_url_secret
    os_cli = api_utils.OSClients(keystone_session)
UnboundLocalError: local variable 'keystone_session' referenced before assignment
```

@phausman, Can you share the show-action-output from the failed action? There should be a log with "Failed to create keystone session ... " that may have more information.

@corey.bryant unfortunately it doesn't show more that this:

```
$ juju show-action-output 194 -m openstack
UnitId: ironic-conductor/2
id: "194"
message: local variable 'keystone_session' referenced before assignment
results: {}
status: failed
timing:
  completed: 2023-04-28 13:50:03 +0000 UTC
  enqueued: 2023-04-28 13:50:02 +0000 UTC
  started: 2023-04-28 13:50:02 +0000 UTC
```

In a debug session with Natalia we were able to find a little more details:

Failed to create keystone session ("'PathDistribution' object has no attribute '_normalized_name'")

https://github.com/pypa/setuptools/issues/3452 which says to update to importlib_metadata 4.3 or later.

The version of importlib_metadata in the wheelhouse.txt is 2.1.3.

I'm moving this new issue reported in comments #18-#21 to a new bug at: https://bugs.launchpad.net/charm-ironic-conductor/+bug/2018018

I've tried cherry-picking the commit from stable/zed to build with binary wheels as Corey mentioned previously as the fix. It's in Gerrit at https://review.opendev.org/c/openstack/charm-ironic-conductor/+/887116. Tests pass, but it will need review, of course.

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/910293

Reviewed: https://review.opendev.org/c/openstack/charm-keystone-saml-mellon/+/910293
Committed: https://opendev.org/openstack/charm-keystone-saml-mellon/commit/dd01dd71f90edfe283adba1c780a1f6fd1e8d853
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit dd01dd71f90edfe283adba1c780a1f6fd1e8d853
Author: Trent Lloyd <email address hidden>
Date: Fri Mar 15 02:28:19 2024 +0000

    Build separately for focal and jammy

    cryptography pinned to the last release without a rust dependency (<4.3)
    does not support openssl v3 and thus Jammy, as a result the following
    error occurs when deploying on Jammy-Yoga but not Focal-Yoga:
    ImportError: /var/lib/juju/agents/unit-keystone-saml-mellon-4/.venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: undefined symbol: FIPS_mode

    Update to the latest version of cryptography to resolve this and
    backport various build changes to support this from 1167cc613316 (Add
    Kinetic and Zed support):
    - Unpin cryptography
    - Install the required rust compiler at build time,
    - Switch the charm to build binary wheels
    - Build separate binary releases for Focal (20.04) and Jammy (22.04).
    - Switch to charmcraft_channel 2.0/stable for 22.04 support
    - Move interface/layer env vars to charmcraft.yaml
    - Pin tox to < 4.0.0
    - Re-generate a new build.lock with latest versions

    Closes-Bug: #1975491
    Change-Id: I4666c4a079ba34f9c3e310c3f4ead3ae8476c007

Though the patch for charm-keystone-saml-mellon was merged, a new build hasn't appeared in the charm store.

Trying to build locally now I get an error related to trying to match the LXD version. Guessing this is why for the charm recipe but the recipe and build results don't seem to be public. Will follow-up.

~/src/charms/keystone-saml-mellon-yoga$ tox -e build
ROOT: will run in automatically provisioned tox, host /usr/bin/python3 is missing [requires (has)]: pip<20.3 (22.0.2), virtualenv<20.0 (20.24.7), setuptools<50.0.0 (59.6.0), tox<4.0.0 (4.11.4)
ROOT: install_deps> python -I -m pip install 'pip<20.3' 'setuptools<50.0.0' 'tox<4.0.0' 'tox>=3.18.0' 'virtualenv<20.0'
ROOT: provision> .tox/.tox/bin/python -m tox -e build
build create: /home/lathiat/src/charms/keystone-saml-mellon-yoga/.tox/build
build run-test-pre: PYTHONHASHSEED='0'
build run-test: commands[0] | charmcraft clean
Cleaned project 'keystone-saml-mellon'.
build run-test: commands[1] | charmcraft -v pack
Starting charmcraft version 2.0.0
Logging execution to '/home/lathiat/snap/charmcraft/common/cache/charmcraft/log/charmcraft-20240429-081853.944073.log'
Packing the charm. /snap/charmcraft/1033/lib/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 5.21.1 LTS is an invalid version and will not be supported in a future release
  warnings.warn(

LXD '5.21.1 LTS' does not meet the minimum required version '4.0'.
Visit https://snapcraft.io/lxd for instructions on how to install the LXD snap for your distribution.

The problem is that the recipe is set for charmcraft 1.5/stable, but osci.yaml is configured with 2.0/stable; hence it built and tested okay in zOSCI, but won't build on launchpad for the final push to the charmhub. (Almost?) all the yoga charms are built using 1.5/stable; any reason this was 2.0/stable? There are some changes in the variables between 1.5 and 2.0:

024-04-29 11:18:02.595 execute action charm:Action(part_name='charm', step=Step.STAGE, action_type=ActionType.RUN, reason=None) /bin/bash: line 26: CRAFT_STAGE: unbound variable

As noted in the commit message, 1.5 doesn't support building on 22.04.

With a single binary release, that was OK because we built on 20.04 and ran on both. But when we switch to separate binary builds for 20.04 + 22.04 you have to build 22.04 on 22.04 and thus you need at least 2.0.

So every yoga charm we switch to binary builds (which is increasingly more of them), they will need to be switch to build with 2.0+

> As noted in the commit message, 1.5 doesn't support building on 22.04.

Ah, I'd forgotten that little trip-hazard! Thanks for the reminder. I'll update the recipes and then schedule the rebuilds and all should be good.

All the keystone-saml-mellon charms are now uploaded apart from arm64 which is taking a while to build.