Fuel VMware DVS plugin

Neutron failed with message: KeyError: 'vmware_dvs

Bug #1586072 reported by Igor Zinovik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel VMware DVS plugin
Invalid
High
Igor Gajsin
Fuel VMware DVS plugin 3.0.0

Bug Description

Problem description:
Originally I was trying to reproduce problem described in
https://bugs.launchpad.net/fuel/+bug/1566171
but did not managed to reproduce it.

Nevertheless test on CI fails:
http://jenkins-tpi.bud.mirantis.net:8080/job/9.0.vcenter.vcenter_custom_hostname/32/console

Controller fails to finish openstack_network/networks.pp deployment task:
2016-05-26 10:51:54 +0000 Scope(Class[Openstack_tasks::Openstack_network::Networks]) (notice): MODULAR: openstack_network/networks.pp
2016-05-26 10:51:54 +0000 Puppet (notice): Compiled catalog for controller-59031.test.domain.local in environment production in 0.21 seconds
2016-05-26 10:51:56 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>". Retrying for 9 sec.
2016-05-26 10:51:59 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>". Retrying for 6 sec.
2016-05-26 10:52:01 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>". Retrying for 4 sec.
2016-05-26 10:52:04 +0000 Puppet::Type::Neutron_network::ProviderNeutron (notice): Unable to complete neutron request due to non-fatal error: "Execution of '/usr/bin/neutron net-list --format=csv --column=id --quote=none' returned 1: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>". Retrying for 1 sec.
2016-05-26 10:52:07 +0000 Puppet (err): Could not prefetch neutron_network provider 'neutron': Can't retrieve net-list because Neutron or Keystone API is not available.

/var/log/neutron/server.log contains this:
2016-05-26 10:51:25.784 15456 DEBUG oslo_concurrency.lockutils [-] Lock "manager" released by "neutron.manager._create_instance" :: held 0.220s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi [-] Couldn't lookup app: neutron
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi Traceback (most recent call last):
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return loadobj(APP, uri, name=name, **kw)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return context.create()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return self.object_type.invoke(self)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi **context.local_conf)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi val = callable(*args, **kw)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi app = loader.get_app(app_name, global_conf=global_conf)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi name=name, global_conf=global_conf).create()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return self.object_type.invoke(self)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi **context.local_conf)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi val = callable(*args, **kw)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/auth.py", line 71, in pipeline_factory
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi app = loader.get_app(pipeline[-1])
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi name=name, global_conf=global_conf).create()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return self.object_type.invoke(self)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return fix_call(context.object, context.global_conf, **context.local_conf)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi val = callable(*args, **kw)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 71, in factory
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return cls(**local_config)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 75, in __init__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi plugin = manager.NeutronManager.get_plugin()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 233, in get_plugin
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return weakref.proxy(cls.get_instance().plugin)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 227, in get_instance
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi cls._create_instance()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return f(*args, **kwargs)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 213, in _create_instance
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi cls._instance = cls()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 116, in __init__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi plugin_provider)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 149, in _get_plugin_instance
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return plugin_class()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/quota/resource_registry.py", line 121, in wrapper
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return f(*args, **kwargs)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 154, in __init__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi self.mechanism_manager = managers.MechanismManager()
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 311, in __init__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi name_order=True)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/stevedore/named.py", line 56, in __init__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi self._init_plugins(extensions)
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/stevedore/named.py", line 112, in _init_plugins
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi self.extensions = [self[n] for n in self._names]
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 284, in __getitem__
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi return self._extensions_by_name[name]
2016-05-26 10:51:25.785 15456 ERROR oslo.service.wsgi KeyError: 'vmware_dvs'

Scenario:

  1. Install fuel-plugin-vmware-dvs
  2. Add 1 controller
  3. Add 1 compute-vmware node
  4. Deploy changes

Reproducibility:
seems like a floating problem.
When I tried to investigating the environment neutron-server was running fine.

Actual result:
  Deployment fails.

Expected result:
  Cluster gets successfully deployed.

Igor Gajsin (igajsin)
Changed in fuel-plugin-vmware-dvs:
milestone: none → 3.0.0
assignee: nobody → Igor Gajsin (igajsin)
Sergey Shevorakov (sshevorakov)
tags: added: acceptance-fail
Andrian Noga (anoga)
Changed in fuel-plugin-vmware-dvs:
importance: Undecided → High
status: New → Triaged
Igor Gajsin (igajsin)
Changed in fuel-plugin-vmware-dvs:
status: Triaged → Invalid
Revision history for this message
Igor Gajsin (igajsin) wrote :

There is no bug.

The 2 tasks should finish its works to configure neutron-server correctly: server_config and fuel-plugin-vmware-dvs after them.

Server_config launches the neutron-server in incorrect state and it fails.

fuel-plugin-vmware-dvs adds config and launches it again.

Finally neutron-server works.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.