Fiber Channel Array - Progress + New Problem

[Eric]

Hi folks. I got a QLogic 2200 dual-channel Fiber Channel PCI card and Debian now sees it and the Sun A5000 FC disk array I spoke of before. That's the good news.

The bad news is that Linux is only seeing 7 of the 14 drives, or one channel of the array. Does anyone have any insight into this? Is there a limit to the number of drives that Linux will see? I am not sure how to proceed. Thanks. Eric

[fastvfr]

When you check the total disk size it ia only half of what is installed?

Have you had that array up and running before this? What filesystem is it formatted with?

And, finally, what Debian kernel are you using?

[Eric]

Originally posted by fastvfr
When you check the total disk size it ia only half of what is installed?

Have you had that array up and running before this? What filesystem is it formatted with?

And, finally, what Debian kernel are you using?

It isn't a RAID array. Each disk is supposed to show up as a seperate drive. There are 14 drives in the array, but only seven of them show up with drive letters in Linux. I plan on setting them up in a software RAID5 array, once I get the other seven drives to show. Eric

[Sinister Angel]

Originally posted by Eric
It isn't a RAID array. Each disk is supposed to show up as a seperate drive. There are 14 drives in the array, but only seven of them show up with drive letters in Linux. I plan on setting them up in a software RAID5 array, once I get the other seven drives to show. Eric

What do you mean "drive letters"?

[Eric]

Originally posted by Sinister Angel
What do you mean "drive letters"?

The server has one SCSI SCA drive installed, which is designated 'SDA'. The seven FC drives that are showing up are designated 'SDB' through 'SDH'. There should also be drives 'SDI' through 'SDO', but they are not showing up. Eric

[Dandapani]

The number 7 looks suspicously like the number of available scsi id's on a single ended scsi chain... not that this helps, but it might trigger someone's thoughts.

[0100010]

There is not a drive limit of 7 drives in Linux, it seems it is more like the driver is not properly seeing the 2nd fiber channel on the controller card.

Maybe an initialization sequence issue? http://lists.us.dell.com/pipermail/linux-poweredge/2004-November/041043.html

[Eric]

This is starting to look like a Linux issue. Before, I had one SCSI drive installed & the FC array. The one SCSI drive and seven of the FC drives would come up. Today, I replaced the SCSI drive with an IDE drive and now eight FC drives come up: Seven on the front channel and one on the back. The OS is only seeing 8 SCSI (SDA - SDH) drives. In the /dev directory, there are drive files for SDA1-11 through SDH1-11. The drive letters stop there and so do the drives that will come up.

So, how do I add more disk drives to a Linux system? Any ideas? Thanks again. Eric

[greenlead]

Have you tried the Debian forums? They know more than I do about it, since the only Debian install I've used was AGNULA's DeMuDi.

[0100010]

(It's late, I'm thinking "out loud").

What kernel is being used?

You are saying in /dev/sd* - you only see sda through sdh? Maybe use makedev to add the others?

When you list the drives (# ls /dev/sd*) you only see 8?

Maybe this? http://groups-beta.google.com/group/linux.debian.user/browse_thread/thread/24217250f6d37af7/0565204b2864883c?q=add+scsi+drives+debian&rnum=6#0565204b2864883c

EDIT : I think this is more along your line : http://groups-beta.google.com/group/comp.os.linux.hardware/browse_thread/thread/3167c59c50fbf830/f0f6bf7962c17656?q=fiber+channel+scsi+drives+debian&rnum=4#f0f6bf7962c17656

EDIT2 : Link to driver : http://www.feral.com/bk.html

EDIT3 : If these links are useless, I apologize... I'm going to sleep now.... ;f

[Eric]

Here is another piece of the puzzle. When I rebuild a kernel and try to run LILO with the Sun array connected to the server via the QLogic QLA2200 card, I get the following fatal error message:

Warning: '/proc/partitions' does not match '/dev' directory structure.
Name change: '/dev/sdq' -> '/tmp/dev.0'
Warning: '/dev' directory structure is incomplete; device (65, 0) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 1) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 16) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 17) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 32) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 33) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 48) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 49) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 64) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 65) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 80) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 81) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 96) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 112) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 128) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 144) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 160) is missing.
Warning: '/dev' directory structure is incomplete; device (65, 176) is missing.
Fatal: open /dev/sdo: No such file or directory



Also if I run cat /proc/scsi/scsi, I can see all fourteen FC drives. This is driving me batty. Eric

[frefoo]

Originally posted by Eric
Hi folks. I got a QLogic 2200 dual-channel Fiber Channel PCI card and Debian now sees it and the Sun A5000 FC disk array I spoke of before. That's the good news.

The bad news is that Linux is only seeing 7 of the 14 drives, or one channel of the array. Does anyone have any insight into this? Is there a limit to the number of drives that Linux will see? I am not sure how to proceed. Thanks. Eric

Eric,

Check the BIOS of the Q-logic card on boot (ALT-Q)

Make sure the max luns (something like that) is higher then 7

Here (http://download.qlogic.com/drivers/24032/QLogic_Linux10.pdf) is the EMC pdf that may help you configure it correctly.

[begin edit]

To answer your question on the number of luns a qla2202 will see is 255 drives per target id.

[end edit]

[frefoo]

Originally posted by Eric
Also if I run cat /proc/scsi/scsi, I can see all fourteen FC drives. This is driving me batty. Eric

I just noticed the above.

If I understand things correctly you have 7 luns on one port and the other 7 luns on the other port?

If that is correct make sure the target ID's are different on the Array.

In the EMC world the target ID is set by the Array, unless you configure the host for persistent binding. You may have the target ID an lun # the same on both ports which would casue the above, (assuming the qla2202 card uses the same controller # per prot) which I dont know since our servers use 2 qla2x00 cards for HA reasons (the qla2x00 is a single port card the qla2x02 is the dual port card).

So on the array..

Make sure each path is a different target ID. Or look into setting up persistent binding.

[frefoo]

Sorry all for this third post...

As a test pull one cable from the hba and boot the server.

If you see the 7 luns then shutdown the server and swap the cables....(pull the active one and plug in the disconnected one using the previous pulled cable).

Boot the server again and if you see 7 luns then the conflict is between the 2 hba ports.

Try to create test file systems on both channels.

Plug both cables in and see if you can mount the file systems.

When you reboot a third time.

[Eric]

I appreciate the advise and I will check into the LUN's. The thing is though, when I had a SCSI drive hooked up in addition to the FC array, the server saw the 1 SCSI drive and 7 FC drives. When I removed the SCSI drive and replaced it with an IDE drive, the server saw the IDE drive and <b>8</b> FC drives, seven on one channel and the 8th on the second channel. Since the server can see drives on both arrays, it makes me wonder if the problem isn't some sort of configuration issue on the server. Eric

[frefoo]

Originally posted by Eric
When I removed the SCSI drive and replaced it with an IDE drive, the server saw the IDE drive and <b>8</b> FC drives, seven on one channel and the 8th on the second channel. Since the server can see drives on both arrays, it makes me wonder if the problem isn't some sort of configuration issue on the server. Eric

I also agree with you that it is a configuration issue on the server. The server will see all of the drives (IDE or SCSI) if they are using seperate controller cards (aka IDE or SCSI).

The problem is when the controller cards are the same or better yet the path to disks are the same. You may have a conflict with the disks because the paths are the same.


[begin edit]

let me use Solaris as an example....

if I have a path to a disk say c2t15d0s2

c2 is set by Solaris (path_to_inst)

t15 is set by the array unless you use persistent binding (possible conflict if you dont use persistent binding)

d0 is the lun number set also by the array (also possible conflict).

So if I have 2 paths to a disk both of the paths are unable to have the same path c2t15d0s2.

If I change one of those settings controller,target,lun then it is a different disk to the OS and the OS is able to access them.

[end edit]