Vely logo Empower C
install  tutorials  examples
documentation  license  about

12.1.0 released on Sep 19, 2022

call-web



PURPOSE:


Get content of URL resource (call a web address).

SYNTAX:


call-web <URL> \
    response [ define ] <result> \
    [ response-code [ define ] <response code> ] \
    [ status [ define ] <status> ] \
    [ headers [ define ] <headers> ] \
    [ request-headers custom <header name>=<header value> [ , ... ] ] \
    [ post [ fields <field name>=<field value> [ , ... ] ] [ files <file name>=<file location> [ , ... ] ] ] \
    [ error [ define ] <error> ] \
    [ cert <certificate> | no-cert ] \
    [ cookie-jar <cookie jar> ] \
    [ timeout <timeout> ]


DESCRIPTION:


With call-web, you can get the content of any accessible URL resource, for example web page, image, PDF document, XML document etc. It allows you to programmatically download URL's content, including the header. For instance, you might want to obtain (i.e. download) the source code of a web page and its HTTP headers. You can then save such downloaded items into files, analyze them, or do anything else.

<URL> is the resource locator, for example "https://somewebpage.com" or if you are downloading an image (for instance) it could be "https://webpage.com/image.jpg". Anything you can access from a client (such as web browser), you can also obtain programmatically. You can specify any URL parameters, for example "https://somewebpage.com?par1=val1".

Response and headers


The result is obtained via "response" clause into variable <result>, and the length (in bytes) of such response is obtained via "status" clause in <status> variable. The response code (such as 200 for "OK", 404 for "Not Found" etc.) is available via optional "response-code" clause in number variable <response code>; the default value is 0 if response code is unavailable (due to error for instance).

The optional "headers" clause allows for retrieval of response headers (such as HTTP headers) in <headers> variable.

Each of <result>, <response code>, <headers> and <status> can be created if they don't exist with the corresponding "define" clause.

Status


In case of error, <status> is negative, and has value of VV_ERR_FAILED (typically indicating system issue, such as lack of memory, library or system issue or local permissions), VV_ERR_WEB_CALL (error in accessing URL or obtaining data) - otherwise <status> is the length in bytes of the response. Optionally, you can obtain the error message (if any) via "error" clause in <error> variable (which can also be created with "define" clause if it doesn't exist). Error is an empty string ("") if there is no error.

Timeout


If "timeout" clause is specified, call-web will timeout if operation has not completed within <timeout> seconds. If this clause is not specified, the default timeout is 120 seconds. If timeout occurs, <status> will be VV_ERR_WEB_CALL and <error> will indicate timeout. Timeout cannot be negative nor greater than 3600 seconds.

HTTPS and certificates


You can call any valid URL that uses protocol supported by the underlying library (cURL). If you're using https protocol (or any other that requires a SSL/TSL certificate), you can either use the local CA (certificate authority) issued, specify the location of a certificate with "cert" clause, or if you do not want it checked, use "no-cert". By default, the locally installed certificates are used; if the URL you are visiting is not trusted via those certificates, and you still want to visit it, use "no-cert", and if you do have a no-CA (i.e. self-signed certificate) for that URL, use "cert" to provide it as a file name (either a full path or a name relative to current working directory, see vv).

Cookies


If you'd like to obtain cookies (for example to maintain session or examine their values), use "cookie-jar" clause. <cookie jar> specifies the location of a file holding cookies. Cookies are read from this file (which can be empty or non-existent to begin with) before making a call-web and any changes to cookies are reflected in this file after the call. This way, multiple calls to the same server maintain cookies the same way browser would do. Make sure the same <cookie jar> file is not used across different application spaces (meaning it should be under the application home directory (see vv), which is the most likely method of implementation anyway).

Binary result


The result of call-web (which is <result>) can be a text value or a binary value. If it is a binary value (for example if getting "JPG", "PNG", "PDF" or other documents), then <status> is the number of bytes in a buffer that holds the value. If it is a string value (such as if downloading "HTML" document as a text), then <status> is the string length.

Request headers


You can specify custom request headers with "request-headers" clause, using "custom" subclause with a list of <header name>=<header value> pairs separated by a comma. For example, here custom header "Vely-header" has value of "Some_ID", and "Another-Header" a value of "New_ID":
call-web "http://website.com/app_name?req=reqname&act=get_file" response resp response-code rc status len request-headers custom "Vely-header"="Some_ID", "Another-Header"="New_ID"

On the receiving side you can get any such custom header by using the "HTTP_" convention, where custom headers are present as web environment variables, capitalized with use of underscore for dashes and prefixed with "HTTP_", for example variable "hvh0" would have a value of "Some_ID" and variable "hvh1" a value of "New_ID":
get-sys web-environment "HTTP_VELY_HEADER" to define hvh0
get-sys web-environment "HTTP_ANOTHER_HEADER" to define hvh1


POST, sending files


In order to POST, for instance to send files, use "post" clause. You can specify fields with "fields" subclause in the form of <field name>=<field value> pairs separated by a comma. For instance, here two fields are set (field "req" with value "web" and field "act" with value "setcookie"):
call-web "http://website.com" response resp response-code rc status len post fields "req"="web","act"="setcookie"

To include files, use "files" subclause in the form of <file name>=<file location> separated by commas. For example, here "file1" is the file name sent to the client (which can be anything), and local file "uploadtest" is the file whose contents is sent; and "file23" is the file name sent to the client (which can be anything),  and "fileup4" is the actual local file read and sent to the client. In this case files are in the application home directory (see how_vely_works), but in general you can specify a relative or absolute path:
call-web "http://website.com" response resp response-code rc status len post files "file1"="uploadtest", "file23"="fileup4"

You can specify both "files" and "fields" POST fields, for instance (along with getting error text and status):
call-web "http://website.com" response resp response-code rc status len post fields "req"="web","act"="setcookie" files "file1"="uploadtest", "file23"="fileup4" status define st error err

You can specify a maximum of 1000 POST files and fields.

EXAMPLES:


Get the web page and print it out:
call-web "https://website.com/page.html" response define resp
p-out resp

Get the "JPG" image from the web and save it to a file "pic.jpg":
call-web "https://website.com/images/someimg.jpg" status define wlen response define resp
write-file "pic.jpg" from resp length wlen


SEE ALSO:


Web ( call-web   out-header   send-file   silent-header  )  SEE ALL (documentation)



Copyright (c) 2022 DaSoftver LLC. Vely is a trademark of Dasoftver LLC. The software and information herein are provided "AS IS" and without any warranties or guarantees of any kind. Icons copyright PaweĊ‚ Kuna licensed under MIT. This web page is licensed under CC-BY-SA-4.0.