[Libstoragemgmt-devel] REST API demo: C + glibc socket + libmicrohttpd + json-c + uriparser

[Gris Ge]

Hey Guys,

I have drafted out a REST API demo for LSM, attached.
It could be count as my first C program is academy work does not count.

Any criticism or comments are well appreciated.

Thanks in advance.
Best regards.

========================

Compile:
yum install libmicrohttpd-devel json-c-devel uriparser-devel -y [1]
gcc lsm_rest.c -lmicrohttpd -ljson-c -luriparser -o lsm_restd

Run:
LSM_UDS_PATH=/tmp/lsm/ipc ./lsm_restd
# Change LSM_UDS_PATH to meet your dev environment.
# If you are using my lsmenv script, you can call:
# lsmenv sim ./lsm_restd

Query:
* curl http://localhost:8888/v0.1/systems?uri=sim://
* curl http://localhost:8888/v0.1/pools?uri=sim://
* curl http://localhost:8888/v0.1/volumes?uri=ontap://***@host&pass=pass

TODO:
* Need better handling of URI and password, http auth?
* Https support.
* Support /etc/lsm_restd.conf like:
[general]
interface = eth0, lo
port = 8888
ipv6 = yes
* Use the REST compliant way of handling errors. Like http 404 and etc.
* Use POST for create and modify.
* Use DELETE for delete.

Issues:
* Will timeout if you call 100 thread of curl at the same time.
* No idea why recv() buffer cannot be 4096.

Good to have:
* With config support, the alarm support could be config as:
[ontap_01]
uri=ontap://***@host
pass=pass # we can steal the way of wpa_passphrase for
# password saving.
alarm =yes
alarm_level = INFO
alarm_syslog = yes
alarm_mail = yes
alarm_mailto = ***@localhost
alarm_exec = notify-send "%level -- %subject" "%content"

[1] For RHEL 6, they are in EPEL6, EPEL6 report it has uriparser, but
actually does not exist:
https://admin.fedoraproject.org/pkgdb/acls/name/uriparser
For RHEL 7, they are in optional repo.
For fedora, all in official repo.

[Gris Ge]

Post by Gris Ge
Hey Guys,
I have drafted out a REST API demo for LSM, attached.
It could be count as my first C program is academy work does not count.
Any criticism or comments are well appreciated.

I forgot to mention: the socket part is almost copied from Tony's C++
code.

And using pure C code could make it possible to merge lsm_rest.c code
into lsmd.c.

Thanks.

[Tony Asleson]

Post by Gris Ge
Hey Guys,
I have drafted out a REST API demo for LSM, attached.
It could be count as my first C program is academy work does not count.

Demos are good, thanks!

Post by Gris Ge
Any criticism or comments are well appreciated.

I see that you aren't using the libStorageMgmt C library. As you are
outputting json from the RPC result, this seems reasonable I guess.

I have written zero RESTful services, but I think one of the big things
to figure out and get correct is how the API should follow the rest
design pattern and in particular how resources are represented.

For example:
GET /system --> returns all the systems
GET /system_id/volume --> returns all the volumes for this system
GET /system_id/volume/id --> returns information about specific volume
POST /system_id/volume?name=SomeDisk&size=1T --> creates volume
DELETE /system_id/volume/id --> deletes a volume
UPDATE /system_id/volume/id/resize?size=1024G \
--> update the size of the volume

Not sure if what I just threw out is even remotely correct, but figuring
this out will definitely effect how the code is constructed.

[Gris Ge]

Post by Tony Asleson
I see that you aren't using the libStorageMgmt C library. As you are
outputting json from the RPC result, this seems reasonable I guess.

Correct. I didn't use the libStorageMgmt C library, but use the socket
directly which save us from JSON -> API -> JSON conversion.

Post by Tony Asleson
POST /system_id/volume?name=SomeDisk&size=1T --> creates volume
Not sure if what I just threw out is even remotely correct, but figuring
this out will definitely effect how the code is constructed.

Looks good to me, not sure the POST part about "?name=SomeDisk&size=1T".
I am assuming the POST should use JSON encoded http body.

Once we decided how this REST go, we can keep a REST_API.README file in
source code tree maintained.

Post by Tony Asleson
v01_get(const char* resource_name, int fd, int *error_no);
Ideally it would be great if we could just parse the URL and call into
the plug-in with as little additional logic as possible. That way as
the API evolves the REST server wouldn't really need 1 to 1 changes when
the API gets additional functionality. For example we should be able to
add a new action on a class and not even need to touch the REST daemon.

Good point, will investigate on that. It seems libmicrohttpd
MHD_get_connection_values() might help on query parameters.

Post by Tony Asleson
The project is currently using functionality in libxml for uri parsing.
We could utilize that instead of introducing another library and that
one is available on EL6 too.

Will switch to that tool for URI parsing.

Post by Tony Asleson
I think it would be good to make a default value like we do in the
library with an option of overriding it as needed with the env. variable.

Make sense.

Post by Tony Asleson

Post by Gris Ge
* Need better handling of URI and password, http auth?

Maybe oauth or something similar?

Never code oauth before. Will look into it.

Post by Tony Asleson

Post by Gris Ge
* Will timeout if you call 100 thread of curl at the same time.

As some arrays cannot handle concurrent management calls, I believe we
should have locking where we serialize calls to the *same* array.

We can return "429 Too Many Requests" HTTP error to client in that case.
Any example array does not allow simultaneously access?

Post by Tony Asleson

Post by Gris Ge
* No idea why recv() buffer cannot be 4096.

I changed mine to 4096 and I haven't seen any issues. What error(s)
were you getting?

The recv() will get no data and become non-blocking (return with 0
immediately). It toke my whole day to investigate this out.
I tried on Fedora 17 and RHEL 7 beta, the same issue. Will double check
it.

Post by Tony Asleson
Can you elaborate on the alarm support? Are you thinking that we should
have the REST daemon poll for status and then notify on errors?

I am planning to merge the REST code into lsmd daemon.
The lsmd daemon will run with config like /etc/lsmd.conf.
The lsmd will server three part of works:
1. Setup sockets (current work).
2. REST API httpd server. (Can be disabled).
3. Alarm support. (Can be disabled).

The alarm support is using one of these two ways for query status
changing:
1. Register a trap in plugin. (SMI-S can, but seems too complicated).
2. Regularly query the requested/configed resources.

With config file supported in lsmd for this, we can allow user to define
how they been notified and what they want to hear.

Post by Tony Asleson
I sent an email about an idea to monitor indications/alerts etc. I got
some feedback from Tom about things to monitor, but no feedback on the
design I pitched. Please take a look at that too.

If I read them correctly, it can summarized as:
1. What to monitor.
ThinP full, disk fail, battery out, pool full, overheat, and etc.
2. How to monitor:
A. SCSI Unit Attention for RAID card. Not LSM's job.
B. Some tool regularly query their interested resource via LSM API.

Like I said above, I am thinking to let lsmd do the repeat query job.
For notification way, we can allow user to define the call-out exec
in lsmd config file.

Post by Tony Asleson
I attached a diff of my changes when just playing around with it today.

If this demo looks good to you, how about we add 'src/lsmd_w_rest.c'[1]
in current git tree where could improve it or even delete it if not good
enough?

[1] Since lsmd codes seems stable now, I am about to copy its code into
lsmd_w_rest.c and try to code out features mentioned above.

Post by Tony Asleson
Regards,
-Tony

Thanks.
Best regards.

[Tony Asleson]

Post by Gris Ge

Post by Tony Asleson
Not sure if what I just threw out is even remotely correct, but figuring
this out will definitely effect how the code is constructed.

Looks good to me, not sure the POST part about "?name=SomeDisk&size=1T".
I am assuming the POST should use JSON encoded http body.

Sure

Post by Gris Ge
Any example array does not allow simultaneously access?

Concurrent management calls to ontap will cause filer to start throwing
errors. You can easily do this with a couple command lines running
against the array doing non-related operations. I hope that we can have
2 client connected so that we can poll for status while another client
is making configuration changes. Otherwise, we have to interweave
requests which makes things much harder to coordinate between an
arbitrary client and a polling daemon.

Post by Gris Ge

Post by Tony Asleson

Post by Gris Ge
* No idea why recv() buffer cannot be 4096.

I changed mine to 4096 and I haven't seen any issues. What error(s)
were you getting?

The recv() will get no data and become non-blocking (return with 0
immediately). It toke my whole day to investigate this out.
I tried on Fedora 17 and RHEL 7 beta, the same issue. Will double check
it.

FYI- I was using F20. I retrieved 1000k volumes in one call with no
errors that I saw, but the client didn't validate the json either.

Post by Gris Ge

Post by Tony Asleson
Can you elaborate on the alarm support? Are you thinking that we should
have the REST daemon poll for status and then notify on errors?

I am planning to merge the REST code into lsmd daemon.

I would prefer to keep lsmd separate for the following reasons:

- The lsmd daemon is a forking daemon. It forks and the client process
execs the plug-in. Thus the client inherits everything the parent has
and has to clean up open file descriptors etc. The parent periodically
has to retrieve the exit status of the children so that the child
process which can be very short lived gets cleaned up quickly and not
left as a zombie. If we some how end up forking the process that is
running the web processing, things get pretty interesting to cleanup and
handle. Since libmicrohttpd supports select, poll, pthread & thread
pools to handle different threading models, I don't see how we can
accommodate any of these with the select loop in lsmd. So we need a
separate process to run the web server in, so for all practical purposes
they will be separate, thus no reason to be tied together.
- We can package the functionality separately
- Doesn't require the user to edit a config file, just install package
with secure defaults and use
- If the REST/lsmd/alerter process crashes it can be started separately
- Any process that is polling for status/alerts will need to be in a
separate process too. It will most likely need to be threaded so that
we can speed things up by checking a number of different arrays at the
same time and converging the results in one shared memory place.

Post by Gris Ge
The alarm support is using one of these two ways for query status
1. Register a trap in plugin. (SMI-S can, but seems too complicated).

I believe that if we want to support monitoring for a large number of
smi-s providers it would be good to try and support this. The plug-in
would be used to setup the indication and send the information to the
smi-s provider where to send the indications i.e. a web server.

Post by Gris Ge
2. Regularly query the requested/configed resources.
With config file supported in lsmd for this, we can allow user to define
how they been notified and what they want to hear.

Post by Tony Asleson
I sent an email about an idea to monitor indications/alerts etc. I got
some feedback from Tom about things to monitor, but no feedback on the
design I pitched. Please take a look at that too.

1. What to monitor.
ThinP full, disk fail, battery out, pool full, overheat, and etc.
A. SCSI Unit Attention for RAID card. Not LSM's job.

With Ewan's udev UA support, we may very well be interested in these
events and possibly providing a way to alert users and possibly
executing code in response to them.

Post by Gris Ge
B. Some tool regularly query their interested resource via LSM API.

Your missing the part where I separated the retrieval of the data from
the distribution. Utilizing dbus with a publish & subscribe model
allows an arbitrary user the ability to subscribe to these events. We
can provide a client that does email/tweet/desktop notification etc.,
but by utilizing the dbus transport other applications can get the
event(s) too and make decisions (in code) based on them.

Post by Gris Ge
If this demo looks good to you, how about we add 'src/lsmd_w_rest.c'[1]
in current git tree where could improve it or even delete it if not good
enough?

We can check it into src/ so that others can submit updates and we can
track its progress. The original file name was OK with me especially as
I believe it should be a separate daemon.

Post by Gris Ge
[1] Since lsmd codes seems stable now, I am about to copy its code into
lsmd_w_rest.c and try to code out features mentioned above.

The code you are coping isn't used by lsmd. lsmd has *no* knowledge of
the RPC transport & protocol etc. It simply knows that someone is
trying to access a unix domain socket and thus forks and execs the
appropriate plug-in based on which file was opened passing the file
descriptor to the plug-in (much like inetd). Once the client is talking
to the plug-in, the daemon is not in the picture. All communication is
directly from client to plug-in via socket. You can stop the daemon and
the client can continue to use the plug-in as long as the client don't
close down the connection.

The code you are copying is used and shared by C library API clients and
the C library plug-in API. We should certainly share this code instead
of copying it by using an internal shared library or static library. As
some of it lives in C++ files we can convert to straight C and place in
a separate file(s).

Regards,
Tony