vely:Example: hash

Example: hash

You will create your own hash server in this example. A REST API will enable end-user to add key/data pairs, query and delete them. The data is in memory-only; a more involved example could be constructed to persist the data in some form. This is an extremely fast, single-process hash server.

In a nutshell: web browser; Apache; REST API; Unix sockets; 3 source files, 59 lines of code.

Setup prerequisites

Install Vely - you can use standard packaging tools such as apt, dnf, pacman or zypper.

Because it is used in this example, you will need to install Apache as a web server.

After installing Vely, turn on syntax highlighting in vim if you're using it:

vv -m

Get the source code

The source code is a part of Vely installation. It is a good idea to create a separate source code directory for each application (and you can name it whatever you like). In this case, unpacking the source code will do that for you:

tar xvf $(vv -o)/examples/hash.tar.gz
cd hash

Setup application

The very first step is to create an application. The application will be named "hash", but you can name it anything (if you do that, change it everywhere). It's simple to do with vf:

sudo vf -i -u $(whoami) hash

This will create a new application home (which is "/var/lib/vv/hash") and do the application setup for you. Mostly that means create various subdirectories in the home folder, and assign them privileges. In this case only current user (or the result of "whoami" Linux command) will own those directories with 0700 privileges; it means a secure setup.

Build application

Use vv utility to make the application:

vv -q

Start your application server

To start the application server for your web application use vf FastCGI process manager. The application server will use a Unix socket to communicate with the web server (i.e. a reverse-proxy):

vf -w 1 hash

This will start 1 daemon process to serve the incoming requests.

See vf for more options to help you achieve best performance.

If you want to stop your application server:

vf -m quit hash

Setup web server

This shows how to connect your application listening on a Unix socket (started with vf) to Apache web server.

- Step 1:
To setup Apache as a reverse proxy and connect your application to it, you need to enable FastCGI proxy support, which generally means "proxy" and "proxy_fcgi" modules - this is done only once:

For Debian (like Ubuntu) and OpenSUSE systems you need to enable proxy and proxy_fcgi modules:
```
sudo a2enmod proxy
sudo a2enmod proxy_fcgi
```
For Fedora systems (or others like Archlinux) enable proxy and proxy_fcgi modules by adding (or uncommenting) LoadModule directives in the Apache configuration file - the default location of this file on Linux depends on the distribution. For Fedora (such as RedHat), Archlinux:
```
sudo vi /etc/httpd/conf/httpd.conf
```
For OpenSUSE:
```
sudo vi /etc/apache2/httpd.conf
```
Add this to the end of the file:
```
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
```

- Step 2:
Edit the Apache configuration file:

For Debian (such as Ubuntu):
```
sudo vi /etc/apache2/apache2.conf
```
for Fedora (such as RedHat), Archlinux:
```
sudo vi /etc/httpd/conf/httpd.conf
```
and for OpenSUSE:
```
sudo vi /etc/apache2/httpd.conf
```

Add this to the end of file ("/hash" is the application path (see request_URL) and "hash" is your application name):

ProxyPass "/hash" unix:///var/lib/vv/hash/sock/sock|fcgi://localhost/hash

- Step 3:
Finally, restart Apache. On Debian systems (like Ubuntu) or OpenSUSE:

sudo systemctl restart apache2

On Fedora systems (like RedHat) and Arch Linux:

sudo systemctl restart httpd

Note: you must not have any other URL resource that starts with "/hash" (such as for example "/hash.html" or "/hash_something" etc.) as the web server will attempt to pass them as a reverse proxy request, and they will likely not work. If you need to, you can change the application path to be different from "/hash", see request_URL.

Access application server from the browser

Use the following URL(s) to access your application server from a client like browser (see request_URL). Use actual IP or web address instead of 127.0.0.1 if different.

Note: if your server is on the Internet and it has a firewall, you may need to allow HTTP traffic - see ufw, firewall-cmd etc.

- Test hash server with a bash script
To test the hash server, you can use "test_hash" bash script (see source code):

./test_hash

This will insert 1000 key/data value pairs, query them, delete and then query again. The result is:

Keys added
Keys queried
Keys deleted
Keys queried
Hash server test successful

- Use REST API from command line
Here is the REST API for your application.

Substitute the loopback "localhost" with the IP or web address of your server.

Note that in these REST calls, the application path is "/hash", and the request name is "/server", followed by URL payload (see request_URL).

The request method used is based on REST methodology, i.e. POST to create resource, GET to obtain, DELETE to delete it, etc. This is how you can use the API:

Here is an URL payload to add a key/data pair; use the "add" operation (i.e. "add" is the value for "op" parameter, written as path segment "/op/add"), and (as an example) specify "key_1" for a key (written as path segment "/key/key_1") and "data_1" for data (written as path segment "/data/data_1"). The entire URL is:
```
curl -X POST  "http://localhost/hash/server/op/add/key/key_1/data/data_1"
```
The result:
```
Added [key_1]
```
To query the hash server, use the "query" operation (i.e. "query" is the value for "op" parameter, written as path segment "/op/query"), and (as an example) specify "key_1" for a key (written as path segment "/key/key_1"):
```
curl -X GET "http://localhost/hash/server/op/query/key/key_1"
```
The result:
```
Value [data_1]
```
To delete a key/data pair, use the "delete" operation (i.e. "delete" is the value for "op" parameter, written as path segment "/op/delete"), and (as an example) specify "key_1" for a key (written as path segment "/key/key_1"):
```
curl -X DELETE "http://localhost/hash/server/op/delete/key/key_1"
```
The result:
```
Deleted [data_1]
```

Files

You are now done with the example! What follows are the source files in this project so you can examine how it works:

Create hash server (_startup.vely)

Hash itself (i.e. the fast-access structure used to store and search data based on a key) is created on process startup in a startup_handler, which in this case allocates new-hash using unmanaged memory (see memory_handling). It means this hash will remain available across all the requests a process serves, which is exactly what a hash server needs.

#include "vely.h"

void _startup() {

    // Create memory available for the life of process
    manage-memory off
    new-hash define h size 1024
    manage-memory on

    // Register hash as process-wide data
    set-app process-data h

}

Hash server (server.vely)

This is the hash server. Because the data kept in hash needs to exist beyond a single request, we use unmanaged memory (see memory_handling). This way hash data stays allocated and available for the life of the server process.

The first step is to obtain the global_process_data via get-app statement. This global process data is the pointer to the hash data, which has been set in set-app in the startup_handler (_startup.vely).

Next, you get the input parameters (see input-param); in this case "op" (operation requested), "key" (value of a key) and "data" (value of data). Then, depending on the operation requested, the data is added, deleted and retrieved (i.e. queried). Note that because input parameter values are automatically freed when the request is finished, they are copied before being stored in the hash.

#include "vely.h"

void server() {
    out-header default

    // Unmanaged memory, i.e. not released at the end of the request
    manage-memory off

    // Get hash allocated in _startup.vely
    vely_hash *h;
    get-app process-data to h

    // Get input parameters
    input-param op
    input-param key
    input-param data

    if (!strcmp (op, "add")) {
        // Add data to hash, make a copy as input params are request-scoped
        copy-string key to define c_key
        copy-string data to define c_data
        write-hash h key c_key value c_data
        @Added [<<p-out key>>]
    }
    else if (!strcmp (op, "delete")) {
        // Delete data and obtain the value deleted
        read-hash h key (key) value define val old-key define okey delete status define st
        if (st == VV_ERR_EXIST) {
            @Not found [<<p-out key>>]
        } else {
            // If found, then delete key and value
            @Deleted [<<p-out val>>]
            delete-mem val
            delete-mem okey
        }
    }
    else if (!strcmp (op, "query")) {
        // Query hash based on key value
        read-hash h key (key) value define val status define st
        if (st == VV_ERR_EXIST) {
            @Not found, queried [<<p-out key>>]
        } else {
            @Value [<<p-out val>>]
        }
    }
}

bash testing script (test_hash)

Test hash server by adding 1000 key/data pairs, querying to make sure the data is correct, deleting all of them and then querying to make sure they were all deleted.

#Restart hash server for this test
vf -m restart hash

#Add 1000 key/data pairs. Key value is 0,1,2... and data values are "data_0", "data_1", "data_2" etc.
for i in {1..1000}; do
    if [ "$(curl -s "http://localhost/hash/server/op/add/key/$i/data/data_$i")" != "Added [$i]" ]; then
        echo "Error adding key [$i]"
        exit -1
    fi
done

echo "Keys added"

#Query all 1000 keys and check that values retrieved are the correct ones.
for i in {1..1000}; do
    if [ "$(curl -s "http://localhost/hash/server/op/query/key/$i")" != "Value [data_$i]" ]; then
        echo "Error querying key [$i]"
        exit -1
    fi
done

echo "Keys queried"

#Delete all 1000 keys
ERR="0"
for i in {1..1000}; do
    if [ "$(curl -s "http://localhost/hash/server/op/delete/key/$i")" != "Deleted [data_$i]" ]; then
        echo "Error deleting key [$i]"
        exit -1
    fi
done

echo "Keys deleted"

#Query all 1000 keys and check that values retrieved are the correct ones.
for i in {1..1000}; do
    if [ "$(curl -s "http://localhost/hash/server/op/query/key/$i")" != "Not found, queried [$i]" ]; then
        echo "Error querying key [$i] after deletion."
        exit -1
    fi
done

echo "Keys queried"

echo "Hash server test successful"