[ffe] drbd doesn't start: "No response from the DRBD driver! Is the module loaded?" but module is loaded

Please add your configuration file so I can reproduce this.

Thanks
chuck

Hello Chuck!
I attached the configuration. On Hardy I used an similar configuration with no problem.

Thanks.

Can you try the version in my ppa when it is built?

Thanks
chuck

Yes, I will try it tomorrow.
Thank you.

Ok, I tested your package "drbd8-utils_8.2.6-0ubuntu1~ppa1_amd64.deb", but the behavior is the same. I get the same error messages as previous.
In addition:
cat /proc/drbd
version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by phil@fat-tyre, 2008-05-30 12:59:17

Is there an incompatibility between drbd and Kernel 2.6.27?

On the Machine where I test intrepid is Kernel 2.6.27-3-server running. It is an amd64 Machine with 16 Cores (4 Quad-Opterons) and 64GB RAM.
Does drbd run on i386? I do not have the possibility to test that.

In addition:
In /var/log/messages I noticed a segfault from drbdadm:
Sep 19 08:13:09 corvus kernel: [ 450.353857] drbd: initialised. Version: 8.2.6 (api:88/proto:86-88)
Sep 19 08:13:09 corvus kernel: [ 450.353866] drbd: GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by phil@fat-tyre, 2008-05-30 12:59:17
Sep 19 08:13:09 corvus kernel: [ 450.353868] drbd: registered as block device major 147
Sep 19 08:13:09 corvus kernel: [ 450.353872] drbd: minor_table @ 0xffff881023527100
Sep 19 08:23:07 corvus kernel: [ 1048.432302] drbdadm[8658]: segfault at 30 ip 00000000004067f2 sp 00007fffee806ef0 error 4 in drbdadm[400000+16000]

On the drbd mailing list I got a hint to a bug report for OpenSuSE 11:
http://lists.linbit.com/pipermail/drbd-user/2008-September/010298.html
May be it is the same problem.

Can you try the updated version in my ppa?

Thanks
chuck

Hi!
there are no compiled packages for ppa2 Version? Is something wrong with the build process?
regards,
Heiko

Errr.... drbd module in kernel is 8.2.6, and drbd8-utils is 8.0.12.

The drbd8-utils in Chucks ppa are at Version 8.2.6. -> https://launchpad.net/~zulcss/+archive

But even Version ppa2 has the same behavior here as the Versions before.

With Kernel 2.6.27-3-server and 2.6.27-4-server and drbd8-utils_8.2.6-0ubuntu1~ppa2_amd64.deb
I get also the message "No response from the DRBD driver! Is the module loaded?"
lsmod shows that the module is loaded and and in /var/log/messages I can see
Sep 24 08:29:14 corvus kernel: [ 327.655157] drbd: initialised. Version: 8.2.6 (api:88/proto:86-88)
Sep 24 08:29:14 corvus kernel: [ 327.655169] drbd: GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by phil@fat-tyre, 2008-05-30 12:59:17
Sep 24 08:29:14 corvus kernel: [ 327.655171] drbd: registered as block device major 147
Sep 24 08:29:14 corvus kernel: [ 327.655173] drbd: minor_table @ 0xffff880422d62a00

...only I got no segfault at this time (as I described in the post from 2008-09-19)

This could be result of missing 'cn' module (kernelspace <-> userspace connector) in Intrepid's kernel, which was fixed in latest Tim's commit (Sep 26 - bb41f6529b9d2ba72abca16a9ad5c8279d903bde). I'll test new kenel source just to confirm.

Nope... That didn't solve the issue :/

Heiko, could you test my test packages?

Kernel (amd64):
http://www.grad.hr/~ivoks/drbd/linux-image-2.6.27-4-server_2.6.27-4.6_amd64.deb

Drbd8-utils (amd64):
http://www.grad.hr/~ivoks/drbd/drbd8-utils_8.2.7~rc1-0ivoks1_amd64.deb

If you need i386 packages, let me know.

Thanks

amd64 packages are fine.

It seems to not solve the issue: Only the timeout seems to be at a higher value?!

I get also the "no response from drbd driver" message.

Here the output from the different Commands:

root@corvus:~# cat /proc/drbd
version: 8.2.7~rc1 (api:88/proto:86-88)
GIT-hash: b00832ffcf713b24f7547423c3d55e1f192d01d3 build by ivoks@indigo, 2008-09-29 20:28:51

root@corvus:~# drbdadm attach r0
No response from the DRBD driver! Is the module loaded?
Bad: fgets returned NULL while waiting for data: Interrupted system call

root@corvus:~# /etc/init.d/drbd start
Starting DRBD resources: [ d(r0) No response from the DRBD driver! Is the module loaded?
s(r0) No response from the DRBD driver! Is the module loaded?
n(r0) No response from the DRBD driver! Is the module loaded?
].
..........
***************************************************************
 DRBD's startup script waits for the peer node(s) to appear.
 - In case this node was already a degraded cluster before the
   reboot the timeout is 120 seconds. [degr-wfc-timeout]
 - If the peer was available before the reboot the timeout will
   expire after 0 seconds. [wfc-timeout]
   (These values are for resource 'r0'; 0 sec -> wait forever)
 To abort waiting enter 'yes' [ 119]:No response from the DRBD driver! Is the module loaded?
yes [ 679]:

root@corvus:~#

And in /var/log/messages:
Oct 1 12:20:07 corvus kernel: [ 4283.482632] drbd: module cleanup done.
Oct 1 12:21:51 corvus kernel: [ 4386.640204] drbd: initialised. Version: 8.2.7~rc1 (api:88/proto:86-88)
Oct 1 12:21:51 corvus kernel: [ 4386.640212] drbd: GIT-hash: b00832ffcf713b24f7547423c3d55e1f192d01d3 build by ivoks@indigo, 2008-09-29 20:28:51
Oct 1 12:21:51 corvus kernel: [ 4386.640214] drbd: registered as block device major 147
Oct 1 12:21:51 corvus kernel: [ 4386.640217] drbd: minor_table @ 0xffff8808238f2500

On Wed, 01 Oct 2008 12:27:45 -0000
Heiko <email address hidden> wrote:

> It seems to not solve the issue: Only the timeout seems to be at a
> higher value?!

OK. I'll prepare new packages during the day.

New packages are pointless. This is a bug in Ubuntu's kernel, and it isn't related to drbd patch. Same patch works on vanilla kernel, while on Ubuntu's it doesn't.

Hi Ante,

Just curious which version of the upstream vanilla kernel you tested? 2.6.27-rc8?

On Thu, 02 Oct 2008 21:29:23 -0000
Leann Ogasawara <email address hidden> wrote:

> Just curious which version of the upstream vanilla kernel you tested?
> 2.6.27-rc8?

No, -rc7 - the one which is the base for Ubuntu kernel.

I have solution. I'm just testing it and building amd64 kernel, so Heiko could test it too.

Heiko, could you install linux-image-2.6.27-5-server from repository and this package:

http://www.grad.hr/~ivoks/drbd/drbd8-utils_8.2.6-2_amd64.deb

Thank you.

Or you could use:

deb http://ppa.launchpad.net/ivoks/ubuntu intrepid main

Ok, that seems to look good:

With Kernel 2.6.27-5-server and your drbd8-utils_8.2.6-2_amd64 package I get in /proc/drbd:

version: 8.2.6 (api:88/proto:86-88)
GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by phil@fat-tyre, 2008-05-30 12:59:17
 0: cs:WFConnection st:Secondary/Unknown ds:Inconsistent/DUnknown C r---
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 oos:15728124

And in /var/log/messages:
Oct 4 13:19:28 corvus kernel: [ 1537.034552] drbd: initialised. Version: 8.2.6 (api:88/proto:86-88)
Oct 4 13:19:28 corvus kernel: [ 1537.034564] drbd: GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by phil@fat-ty
re, 2008-05-30 12:59:17
Oct 4 13:19:28 corvus kernel: [ 1537.034566] drbd: registered as block device major 147
Oct 4 13:19:28 corvus kernel: [ 1537.034567] drbd: minor_table @ 0xffff880423486600
Oct 4 13:19:28 corvus kernel: [ 1537.043390] drbd0: disk( Diskless -> Attaching )
Oct 4 13:19:28 corvus kernel: [ 1537.043406] drbd0: Starting worker thread (from cqueue [325])
Oct 4 13:19:28 corvus kernel: [ 1537.045234] drbd0: No usable activity log found.
Oct 4 13:19:28 corvus kernel: [ 1537.045244] drbd0: Backing device's merge_bvec_fn() = ffffffffa0059c80
Oct 4 13:19:28 corvus kernel: [ 1537.045246] drbd0: max_segment_size ( = BIO size ) = 4096
Oct 4 13:19:28 corvus kernel: [ 1537.045253] drbd0: drbd_bm_resize called with capacity == 31456248
Oct 4 13:19:28 corvus kernel: [ 1537.045464] drbd0: resync bitmap: bits=3932031 words=61438
Oct 4 13:19:28 corvus kernel: [ 1537.045470] drbd0: size = 15 GB (15728124 KB)
Oct 4 13:19:28 corvus kernel: [ 1537.045475] drbd0: Writing the whole bitmap, size changed
Oct 4 13:19:28 corvus kernel: [ 1537.046340] drbd0: writing of bitmap took 0 jiffies
Oct 4 13:19:28 corvus kernel: [ 1537.046347] drbd0: 15 GB (3932031 bits) marked out-of-sync by on disk bit-map.
Oct 4 13:19:28 corvus kernel: [ 1537.047134] drbd0: reading of bitmap took 0 jiffies
Oct 4 13:19:28 corvus kernel: [ 1537.047522] drbd0: recounting of set bits took
 additional 0 jiffies
Oct 4 13:19:28 corvus kernel: [ 1537.047525] drbd0: 15 GB (3932031 bits) marked out-of-sync by on disk bit-map.
Oct 4 13:19:28 corvus kernel: [ 1537.047531] drbd0: disk( Attaching -> Inconsistent )
Oct 4 13:19:28 corvus kernel: [ 1537.047543] drbd0: Writing meta data super block now.
Oct 4 13:19:28 corvus kernel: [ 1537.047548] drbd0: Barriers not supported on meta data device - disabling
Oct 4 13:19:28 corvus kernel: [ 1537.050432] drbd0: conn( StandAlone -> Unconnected )
Oct 4 13:19:28 corvus kernel: [ 1537.050458] drbd0: Starting receiver thread (from drbd0_worker [8676])
Oct 4 13:19:28 corvus kernel: [ 1537.050488] drbd0: receiver (re)started
Oct 4 13:19:28 corvus kernel: [ 1537.050499] drbd0: conn( Unconnected -> WFConnection )

This looks as it should.

At the moment I've no time to do further tests. I have first to reinstall the second drbd-machine.
I think for the moment that the issue is solved with this packages.

Thanks @all

Hi,

I would like to upload a new version of drbd to the archive.

As it stands right now since there is a userland and kernel module mismatch, drbd is non-functional in hardy. As well has there is a bug in drbd which prevents the userland tools in 8.2.6 to communicate with the kernel module as well. This is fixed by ante as well.

If you have any questions please let me know.

Thanks
chuck

It's functional in Hardy, but isn't in Intrepid :D

As requested here is the attached diffsat. The testing that has been done is that it built correctly and there is no known regressions. It also has installed correctly. If you have any questions please feel free to contact me.

Regards
chuck

Requested changelog

FFe granted, please upload.

So, I can now confirm that all is running fine with uptodate Intrepid drbd and amd64-server-kernel packages.
Thank you all.

Marking Fix Released:

drbd8 (2:8.2.6-2ubuntu1) intrepid; urgency=low

  * Merge from debian experimental, remaining changes:
    - Update maintainer according to spec.
    - Made the lsb init script a bit more compliant.
    - Drop the kernel modules since it is apart of linux-ubuntu-modules.
  * Change CN_IDX_DRBD to 0x6 in drbd/linux/drbd.h

 -- Ante Karamatic <email address hidden> Sat, 04 Oct 2008 11:33:52 +0200

Difficulty to boot when processing "block device major 147" ....