Plugin implementation status of Pool.status

[Gris Ge]

Hi Tony,

About the Pool.STATUS_OK we discussed:

* Nstor is always Pool.STATUS_UNKNOWN
* Targetd is always Pool.STATUS_UNKNOWN
* HP is Pool.STATUS_OTHER or STATUS_OK
* SMI-S:

No standard or vendor documentation mentioned possible
combination of OperationalStatus values.
In SNIA Lab, Got HP 3par and Fujitsu DX80S2 only set
DEGRADED value(I cannot tell the reason why they degraded).
The plugin code only handle OK, Degraded, Error in vendor
provided combinations.
I got no hardware to test about grow, reconstruction and etc
right now.

* User cases and plugin implantation:

* RAID degraded, no rebuild.
* ONTAP -- Pool.STATUS_DEGRADE
* MegaRAID -- Pool.STATUS_DEGRADE

* RAID degraded by one disk failure and rebuilding.
* ONTAP -- Pool.STATUS_RECONSTRUCTING
* MegaRAID -- Pool.STATUS_RECONSTRUCTING

* RAID failed (like: two disk failure in RAID 5)
* ONTAP -- Pool.STATUS_ERROR
* MegaRAID -- Pool.STATUS_ERROR

* Verifying RAID integrity.
* ONTAP -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'aggr-mediascrub-list-info' command
* MegaRAID -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'/c0/v1 show cc' command

* Expected -- Pool.STATUS_VERIFYING

* Growing RAID by adding more disks.
* ONTAP -- Pool.STATUS_GROW
* MegaRAID -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'/c0/v1 show migrate' command.

* Creating pool.
* ONTAP -- Pool.STATUS_INITIALIZING
* MegaRAID -- Need test. No idea.

* Pool disabled.
* ONTAP -- Pool.STATUS_STOPPED
* MegaRAID -- Not allowed.

============

Summery:

1. Only SMI-S might(never seen) have multiple status.

2. For degraded rebuilding:
It could express as
Pool.STATUS_RECONSTRUCTING|Pool.STATUS_DEGRADE
But reconstructing always mean degraded, so no need to
have multiple status.

3. For RAID grow:
It could be:
Pool.STATUS_OK | Pool.STATUS_GROW
Online grow
Pool.STATUS_STOPPED | Pool.STATUS_GROW
Offline grow
But we could introduced Pool.STATUS_OFFLINE_GROW also
to keep status as single status.
No existing plugins support offline grow.

So base on above information, I would suggest:

* Keep Pool.status as bit map, but documented that only single status
value will be used right now. Change SMI-S plugin to reflect this.
(Patch is coming soon).

* Pool.STATUS_OK -- everything is OK.

* Pool.STATUS_DEGRADE -- data accessible but lose redundancy.

* Pool.STATUS_ERROR -- data lose or data unavailable.

* Pool.STATUS_RECONSTRUCTING -- data accessible but lose redundancy
and rebuilding data.

* Pool.STATUS_STOPPED -- admin or some task disabled data access.

* Pool.STATUS_INITIALIZING -- pool creating, data not accessible.

* Pool.STATUS_GROW -- pool is expanding, data is accessible.
Performance might be impacted(status_info could indicate whether
performance impacted or not).

* Pool.STATUS_OTHER -- vendor expansion, check status_info.

* Pool.STATUS_VERIFYING -- Running data integrity check. Performance
is impacted. Data is still accessible.

Any comments?

[Tony Asleson]

Post by Gris Ge
Hi Tony,
* Nstor is always Pool.STATUS_UNKNOWN
* Targetd is always Pool.STATUS_UNKNOWN
* HP is Pool.STATUS_OTHER or STATUS_OK
No standard or vendor documentation mentioned possible
combination of OperationalStatus values.

SMI-S 1.6.1r5, Block book, page 95 section 5.2.1

Primary Subsidiary Desc.
Status status

(2) OK The pool is operational
(2) OK (19)Relocating The pool is operational, but
relocation
(3) Degraded Pool is operational, but lower QOS
(3) Degraded (19) Relocating Operational, lower QOS
(6) Error Storage pool is in error
(15) Dormant Non-operational
(15) Dormant (19) Relocating Non-operational & relocating

So yes the SMI-S standard has provision for primary and secondary status.

In my opinion the above SMI-S should be mapped to the library as:

STATUS_OK
STATUS_OK | STATUS_RECONSTRUCTING
STATUS_OK | STATUS_DEGRADED
STATUS_OK | STATUS_DEGRADED | STATUS_RECONSTRUCTING
STATUS_ERROR
STATUS_ERROR | STATUS_STOPPED
STATUS_ERROR | STATUS_STOPPED | STATUS_RECONSTRUCTING

Post by Gris Ge
In SNIA Lab, Got HP 3par and Fujitsu DX80S2 only set
DEGRADED value(I cannot tell the reason why they degraded).
The plugin code only handle OK, Degraded, Error in vendor
provided combinations.
I got no hardware to test about grow, reconstruction and etc
right now.
* RAID degraded, no rebuild.
* ONTAP -- Pool.STATUS_DEGRADE
* MegaRAID -- Pool.STATUS_DEGRADE
* RAID degraded by one disk failure and rebuilding.
* ONTAP -- Pool.STATUS_RECONSTRUCTING
* MegaRAID -- Pool.STATUS_RECONSTRUCTING
* RAID failed (like: two disk failure in RAID 5)
* ONTAP -- Pool.STATUS_ERROR
* MegaRAID -- Pool.STATUS_ERROR
* Verifying RAID integrity.
* ONTAP -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'aggr-mediascrub-list-info' command
* MegaRAID -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'/c0/v1 show cc' command
* Expected -- Pool.STATUS_VERIFYING
* Growing RAID by adding more disks.
* ONTAP -- Pool.STATUS_GROW
* MegaRAID -- Pool.STATUS_OK
Currently plugin has no idea about it. Need to use
'/c0/v1 show migrate' command.
* Creating pool.
* ONTAP -- Pool.STATUS_INITIALIZING
* MegaRAID -- Need test. No idea.
* Pool disabled.
* ONTAP -- Pool.STATUS_STOPPED
* MegaRAID -- Not allowed.
============
1. Only SMI-S might(never seen) have multiple status.
It could express as
Pool.STATUS_RECONSTRUCTING|Pool.STATUS_DEGRADE
But reconstructing always mean degraded, so no need to
have multiple status.
Pool.STATUS_OK | Pool.STATUS_GROW
Online grow
Pool.STATUS_STOPPED | Pool.STATUS_GROW
Offline grow
But we could introduced Pool.STATUS_OFFLINE_GROW also
to keep status as single status.
No existing plugins support offline grow.
* Keep Pool.status as bit map, but documented that only single status
value will be used right now. Change SMI-S plugin to reflect this.
(Patch is coming soon).

To maintain backwards compatibility all clients would need to treat the
return value as a bitmap. Any other use would cause the client to break
in the future when multiple bits are set. At this point I believe we
should document better and correct the plug-ins to match the documentation.

Post by Gris Ge
* Pool.STATUS_OK -- everything is OK.
* Pool.STATUS_DEGRADE -- data accessible but lose redundancy.

I would also advocate that if STATUS_DEGRADED is set that data is at
risk if another failure occurs.

Pool.STATUS_OK | Pool.STATUS_DEGRADED

Post by Gris Ge
* Pool.STATUS_ERROR -- data lose or data unavailable.
* Pool.STATUS_RECONSTRUCTING -- data accessible but lose redundancy
and rebuilding data.

They have all the disks for the RAID, but are rebuilding, Data at risk
if they lose another disk during this time.

Pool.STATUS_OK | Pool.STATUS_DEGRADED | Pool.STATUS_RECONSTRUCTING

Post by Gris Ge
* Pool.STATUS_STOPPED -- admin or some task disabled data access.

Pool.STATUS_ERROR | Pool.STATUS_STOPPED

Post by Gris Ge
* Pool.STATUS_INITIALIZING -- pool creating, data not accessible.

Pool.STATUS_ERROR | Pool.STATUS_INITIALIZING

Post by Gris Ge
* Pool.STATUS_GROW -- pool is expanding, data is accessible.
Performance might be impacted(status_info could indicate whether
performance impacted or not).

Pool.STATUS_OK | Pool.STATUS_GROWING

Post by Gris Ge
* Pool.STATUS_OTHER -- vendor expansion, check status_info.

Pool.STATUS_OK | Pool.STATUS_OTHER

Post by Gris Ge
* Pool.STATUS_VERIFYING -- Running data integrity check. Performance
is impacted. Data is still accessible.

Pool.STATUS_OK | Pool.STATUS_VERIFYING


Hypothetical time line sequence for a failed disk which is fixed

STATUS_OK
...
STATUS_OK | STATUS_DEGRADED
...
STATUS_OK | STATUS_DEGRADED | STATUS_RECONSTRUCTING
...
STATUS_OK

Post by Gris Ge
Any comments?

My original design/thought was that if the data was available for r/w
operations and data integrity was ensured that STATUS_OK would be set.
If no additional bits were set then the array believes it is operating
optimally, additional bits indicate additional information which may
effect performance or redundancy or tell more about why the pool is not
accessible.

Client code in my opinion should be something like:

if status == Pool.STATUS_OK:
print("No worries, get outside\n")
else:
if status & Pool.STATUS_OK:
print('Warning... something is sub optimal")
else:
print('We are offline, investigate why')

We should document what are incompatible combinations eg.
STATUS_OK cannot be set with STATUS_ERROR or STATUS_STOPPED

Regards,
Tony

[Gris Ge]

Post by Tony Asleson

Post by Gris Ge
Hi Tony,
* Nstor is always Pool.STATUS_UNKNOWN
* Targetd is always Pool.STATUS_UNKNOWN
* HP is Pool.STATUS_OTHER or STATUS_OK
No standard or vendor documentation mentioned possible
combination of OperationalStatus values.

SMI-S 1.6.1r5, Block book, page 95 section 5.2.1
Primary Subsidiary Desc.
Status status
(2) OK The pool is operational
(2) OK (19)Relocating The pool is operational, but
relocation
(3) Degraded Pool is operational, but lower QOS
(3) Degraded (19) Relocating Operational, lower QOS
(6) Error Storage pool is in error
(15) Dormant Non-operational
(15) Dormant (19) Relocating Non-operational & relocating
So yes the SMI-S standard has provision for primary and secondary status.
STATUS_OK
STATUS_OK | STATUS_RECONSTRUCTING
STATUS_OK | STATUS_DEGRADED
STATUS_OK | STATUS_DEGRADED | STATUS_RECONSTRUCTING
STATUS_ERROR
STATUS_ERROR | STATUS_STOPPED
STATUS_ERROR | STATUS_STOPPED | STATUS_RECONSTRUCTING

In section 29.1.3 Storage Pool Relocation:
In this profile, storage pool relocation includes: 1) relocating
storage pool onto a new group of extents; 2) merging storage pools.

It seems (19)Relocating is not RAID reconstructing.
Do you see any vendor use that value to indicate reconstructing?

In DMTF definition, "Dormant" indicates that the element
is inactive or quiesced. I think it means STATUS_STOPPED.
I'd better remove that one, and add it back until I see real
implementation.

Post by Tony Asleson
Hypothetical time line sequence for a failed disk which is fixed
STATUS_OK
...
STATUS_OK | STATUS_DEGRADED

Equal to STATUS_DEGRADED as every degraded is still accessible.

Post by Tony Asleson
...
STATUS_OK | STATUS_DEGRADED | STATUS_RECONSTRUCTING

Equal to STATUS_RECONSTRUCTING as every rebuild is degraded and
accessible.

Post by Tony Asleson
...
STATUS_OK

Post by Gris Ge
Any comments?

My original design/thought was that if the data was available for
r/w operations and data integrity was ensured that STATUS_OK would
be set. If no additional bits were set then the array believes it
is operating optimally, additional bits indicate additional
information which may effect performance or redundancy or tell more
about why the pool is not accessible.

So your suggest:
pros):
1. Leave enough space for future expansion.
cros):
1. No plugin implemented multi-bits status.
2. All multi-bits status could be use single bit status
like I mentioned in 'Summery' section of previous email.
3. User have to check multiple bits to make sure
everything is OK. Every time we add an additional
bit, user code has to add one more check.
For some user, data accessible is not enough to be OK.

So this questions lies to:
Should we design API for future expansion of imaginary products?
# the imaginary products includes SMI-S.

I would say no, once we got real user case where current design
cannot hold, we can bump to next major version.

But I will leave the decision to you. I will change all plugins to
honor that.

Post by Tony Asleson
print("No worries, get outside\n")
print('Warning... something is sub optimal")
print('We are offline, investigate why')
We should document what are incompatible combinations eg.
STATUS_OK cannot be set with STATUS_ERROR or STATUS_STOPPED

I prefer documenting all combinations and their meanings if we
finally choose multi-bits design.

Post by Tony Asleson
Regards,
Tony

Thank you to the valuable inputs.

Best regards.

[Tony Asleson]

Post by Gris Ge
It seems (19)Relocating is not RAID reconstructing.
Do you see any vendor use that value to indicate reconstructing?

No, but we have gotten very little from vendors on health status,
especially with regards to SMI-S.

Post by Gris Ge
In DMTF definition, "Dormant" indicates that the element
is inactive or quiesced. I think it means STATUS_STOPPED.
I'd better remove that one, and add it back until I see real
implementation.

We can add, but we cannot remove constants.

Post by Gris Ge
1. Leave enough space for future expansion.
1. No plugin implemented multi-bits status.

We could easily add multi-bit to our existing plug-ins, there is nothing
special required from the array except a mechanism to indicate something
we can interpret as a status and for us to interpret and set as documented.

Post by Gris Ge
2. All multi-bits status could be use single bit status
like I mentioned in 'Summery' section of previous email.

Yes, I'm not disagreeing with that.

Post by Gris Ge
3. User have to check multiple bits to make sure
everything is OK. Every time we add an additional
bit, user code has to add one more check.

Please elaborate on how adding a new enumerated error type doesn't
require the client to be modified to interpret the value?

Post by Gris Ge
For some user, data accessible is not enough to be OK.

You advocate that adding a bit is more problematic than adding an
enumerated value. To me this isn't an error code where the operations
was successful, else it is in error with the error reason, it's a
status. If we add a new enumerated status code, lets call it
STATUS_CONFRAZZLED, is that bad or good? What does an existing client
do if a newer plug-in returns it? I could see a client written with a
switch statement and this falling into the default which would likely be
report at highest error level.

I was thinking that if we used the STATUS_OK or STATUS_ERROR bit and sub
qualified it with other bits, it would at least convey to any client
that things are generally OK or NOT.

Ultimately in this case we can go either way. If others don't see the
benefit of having the multiple bits we can document them as being unique
error values while retaining the existing numerical values. Current
plug-ins aren't using mult-bit. If we do this we should also change the
documentation on all the other status fields too, to make it consistent.

Regards,
Tony

[Gris Ge]

Post by Tony Asleson

Post by Gris Ge
3. User have to check multiple bits to make sure
everything is OK. Every time we add an additional
bit, user code has to add one more check.

Please elaborate on how adding a new enumerated error type doesn't
require the client to be modified to interpret the value?

Single bit status:

STATUS_OK means everything is OK. Hence user can tell whether
everything is OK using old code when new status added.

Multiple bits status:

STATUS_OK means I/O is OK. Hence user can tell whether I/O
is OK using old code when new status added.

In storaged lsm module, I use single property 'is_OK' to indicate
whether user should check with storage admin.

Post by Tony Asleson

Post by Gris Ge
For some user, data accessible is not enough to be OK.

You advocate that adding a bit is more problematic than adding an
enumerated value. To me this isn't an error code where the
operations was successful, else it is in error with the error
reason, it's a status. If we add a new enumerated status code, lets
call it STATUS_CONFRAZZLED, is that bad or good? What does an
existing client do if a newer plug-in returns it? I could see a
client written with a switch statement and this falling into the
default which would likely be report at highest error level.

Make sense. Some state might does mean good or bad, they could
come with STATUS_OK or STATUS_ERROR simultaneous.

Post by Tony Asleson
I was thinking that if we used the STATUS_OK or STATUS_ERROR bit and
sub qualified it with other bits, it would at least convey to any
client that things are generally OK or NOT.
Ultimately in this case we can go either way. If others don't see
the benefit of having the multiple bits we can document them as
being unique error values while retaining the existing numerical
values. Current plug-ins aren't using mult-bit. If we do this we
should also change the documentation on all the other status fields
too, to make it consistent.

For lsm.System.status and lsm.Disk.status, they still need
bitmap status as exiting plugin implemented multiple status,
for example:
Disk.STATUS_SPARE | Disk.STATUS_PREDICTIVE_FAILURE
Disk.STATUS_ERROR | Disk.STATUS_STOPPED

To be consistent, let's keep using the bitmap in pool.

Let me update document and plugins to reflect that.

Post by Tony Asleson
Regards,
Tony