Emu

Documentation Status Travis Build Codacy Code Checks GitHub license Join the chat at https://gitter.im/bird-house/birdhouse
Emu (the bird)
Emus are curious birds who are known to follow and watch other animals and humans. Emus do not sleep continuously at night but in several short stints sitting down. [..]. (Wikipedia).

Emu is a Python package with some test proccess for Web Processing Services (WPS). Currently it is using the PyWPS 4.x server.

Full documentation is available on ReadTheDocs or in the docs directory.

Installation

Install from Conda

Ananconda Install Anaconda Build Anaconda Version Anaconda Downloads

Install the emu Conda package:

$ conda install -c birdhouse -c conda-forge emu
$ emu --help

Install from GitHub

Check out code from the Emu GitHub repo and start the installation:

$ git clone https://github.com/bird-house/emu.git
$ cd emu
$ conda env create -f environment.yml
$ source activate emu
$ python setup.py develop

… or do it the lazy way

The previous installation instructions assume you have Anaconda installed. We provide also a Makefile to run this installation without additional steps:

$ git clone https://github.com/bird-house/emu.git
$ cd emu
$ make clean    # cleans up a previous Conda environment
$ make install  # installs Conda if necessary and runs the above installation steps

Start Emu PyWPS service

After successful installation you can start the service using the emu command-line.

$ emu start --help # show help
$ emu start       # start service with default configuration

OR

$ emu start --daemon # start service as daemon
loading configuration
forked process id: 42

The deployed WPS service is by default available on:

http://localhost:5000/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Note

Remember the process ID (PID) so you can stop the service with kill PID.

You can find which process uses a given port using the following command (here for port 5000):

$ netstat -nlp | grep :5000

Check the log files for errors:

$ tail -f  pywps.log

… or do it the lazy way

You can also use the Makefile to start and stop the service:

$ make start
$ make status
$ tail -f pywps.log
$ make stop

Run Emu as Docker container

You can also run Emu as a Docker container, see the Tutorial.

Use Ansible to deploy Emu on your System

Use the Ansible playbook for PyWPS to deploy Emu on your system.

Configuration

Command-line options

You can overwrite the default PyWPS configuration by using command-line options. See the Emu help which options are available:

$ emu start --help
--hostname HOSTNAME        hostname in PyWPS configuration.
--port PORT                port in PyWPS configuration.

Start service with different hostname and port:

$ emu start --hostname localhost --port 5001

Use a custom configuration file

You can overwrite the default PyWPS configuration by providing your own PyWPS configuration file (just modifiy the options you want to change). Use one of the existing sample-*.cfg files as example and copy them to etc/custom.cfg.

For example change the hostname (demo.org) and logging level:

$ cd emu
$ vim etc/custom.cfg
$ cat etc/custom.cfg
[server]
url = http://demo.org:5000/wps
outputurl = http://demo.org:5000/outputs

[logging]
level = DEBUG

Start the service with your custom configuration:

# start the service with this configuration
$ emu start -c etc/custom.cfg

Developer Guide

Building the docs

First install dependencies for the documentation:

$ make bootstrap_dev
$ make docs

Running tests

Run tests using pytest.

First activate the emu Conda environment and install pytest.

$ cd emu
$ source activate emu
$ conda install pytest flake8  # if not already installed

Run quick tests (skip slow and online):

$ pytest -m 'not slow and not online'"

Run all tests:

$ pytest

Check pep8:

$ flake8

Run tests the lazy way

Do the same as above using the Makefile.

$ make test
$ make testall
$ make pep8

Bump a new version

Make a new version of Emu in the following steps:

  • Make sure everything is commit to GitHub.
  • Update CHANGES.rst with the next version.
  • Dry Run: bumpversion --dry-run --verbose --new-version 0.8.1 patch
  • Do it: bumpversion --new-version 0.8.1 patch
  • … or: bumpversion --new-version 0.9.0 minor
  • Push it: git push
  • Push tag: git push --tags

See the bumpversion documentation for details.

Tutorials

Tutorial: using Docker

Emu WPS is available as docker image. You can download the docker image from DockerHub or build it from the provided Dockerfile.

Start the container with the following command:

$ docker run -d -p 5000:5000 --name=emu birdhouse/emu

It is using the port 5000 for the PyWPS service and also to access the WPS outputs.

You can map the container port also to another port on your machine, for example: -p 8094:5000 (your machine port=8094, container port=5000).

Check the docker logs:

$ docker logs emu

Show running docker containers:

$ docker ps

Run a GetCapabilites WPS request:

Run DescribeProcess WPS request for Hello:

Execute Hello process with you user name:

Install Birdy WPS command line tool from Conda:

$ conda install -c birdhouse birdhouse-birdy

Use Birdy to access Emu WPS service:

$ export WPS_SERVICE=http://localhost:5000/wps
$ birdy -h
$ birdy hello -h
$ birdy hello --name Pingu

Stop and remove docker container:

$ docker stop emu

Using docker-compose

Use docker-compose (you need a recent version > 1.7) to start the container:

$ git clone https://github.com/bird-house/emu.git
$ cd emu
$ docker-compose up -d
$ docker-compose logs emu

Execute tail command in the running container to see the logs:

$ docker ps   # get the container name
NAMES
emu_emu_1
$ docker exec -it emu_emu_1 tail -f /opt/wps/pywps.log

You can customize the docker-compose.yml file. See the docker-compose documentation.

Stop the container with:

$ docker-compose down

Build image using docker-compose

You can build locally a new docker image from the Dockerfile by running docker-compose:

$ docker-compose build

Tutorial: using postgres database

You can use a postgres database for PyWPS, the default is sqlite. PyWPS is using SQLAlchemy, see the PYWPS documentation for details.

First run the Emu default installation:

$ git clone https://github.com/bird-house/emu.git
$ cd emu
$ make clean install

The default installation is using sqlite. We now need a postgres database. If you don’t have one yet you can use a postgres docker container.

$ docker pull postgres
$ docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres

The postgres database is now available on default port 5432.

SQLAlchemy needs the psycopg2 postgres adapter. You can install it via Conda into the emu environment.

$ conda install -n emu psycopg2

The SQLAlchemy connection string for this database is:

# postgresql+psycopg2://user:password@host:port/dbname
postgresql+psycopg2://postgres:postgres@localhost:5432/postgres

Configure this connection string in etc/custom.cfg, logging section, database option:

$ vim etc/custom.cfg
[logging]
level = INFO
database = postgresql+psycopg2://postgres:postgres@localhost:5432/postgres

Start the emu service:

$ emu start -c etc/custom.cfg

Your Emu WPS service should be available at the following URL:

$ firefox http://localhost:5000/wps?request=GetCapabilities&service=WPS

Processes

Say Hello

class emu.processes.wps_say_hello.SayHello[source]

hello Say Hello (v1.5)

Just says a friendly Hello. Returns a literal string output with Hello plus the inputed name.

Parameters:name (string) – Please enter your name.
Returns:output – A friendly Hello from us.
Return type:string

References

Sleep

class emu.processes.wps_sleep.Sleep[source]

sleep Sleep Process (v1.0)

Testing a long running process, in the sleep. This process will sleep for a given delay or 10 seconds if not a valid value.

Parameters:delay (float) – Delay between every update
Returns:sleep_output – Sleep Output
Return type:string

References

Wordcounter

class emu.processes.wps_wordcounter.WordCounter[source]

wordcounter Word Counter (v1.0)

Counts words in a given text.

Parameters:text (text/plain) – URL pointing to a text document, for example “Alice in Wonderland”: http://www.gutenberg.org/cache/epub/19033/pg19033.txt
Returns:output – Word counter result
Return type:application/json

References

Counts ocurrences of all words in a document.

Chomsky

class emu.processes.wps_chomsky.Chomsky[source]

chomsky Chomsky text generator (v1.0)

Generates a random chomsky text

Parameters:times (integer) – Generates a random chomsky text.
Returns:output – Chomsky text
Return type:text/plain

Generates a random chomsky text: http://code.activestate.com/recipes/440546-chomsky-random-text-generator/

CHOMSKY is an aid to writing linguistic papers in the style of the great master. It is based on selected phrases taken from actual books and articles written by Noam Chomsky. Upon request, it assembles the phrases in the elegant stylistic patterns that Chomsky is noted for. To generate n sentences of linguistic wisdom, type:

  • (CHOMSKY n) – for example
  • (CHOMSKY 5) generates half a screen of linguistic truth.

NCMeta

class emu.processes.wps_ncmeta.NCMeta[source]

ncmeta Return NetCDF Metadata (v4)

Return metadata from a netCDF dataset, either on file or an OpenDAP service.

Parameters:
Returns:

output – Metadata
Return type:

text/plain

References

Returns metadata of a NetCDF file or OpenDAP resource.