Tutorial on how to create a simple Flask REST API for OTRS

Flask image

The goal of this document is to show you how to create a simple Flask REST API for OTRS. To avoid having to setup OTRS or dealing with the connections to the database instead of learning and working with the code, I have created a small SQLite database with a few columns from the OTRS database. This allows us to test the REST API development process without dealing with other things such as setting up databases etc.

The code is shared on fa-code-branch fa-2x green GitHub

1. API design

First we need to decide what our users want and how we will provide this information.

  • Enter part of a customer name and have a list returned with possible matches
  • Get the details on a customer

As frontend, we will use a single page html. As back-end we will use Flask to get the jobdone. In the end, you would need to secure the api and check for credentials. This is not going to be part of this tutorial but maybe for another one.

1.1. Frontend

We'll use HTML/CSS3/Javascript (Jquery) We will need an input field to enter the customer name. This data will be sent automatically to the back-end if the users enters at least 3 characters. No point in stressing the back-end to search for all clients that start with a letter. Off course, this could be different if there would be a customer with only a 1 character name. Change the code accordingly.

The partial name is sent via ajax to the server. If at least 1 customer is found a list is returend with all the customers with a matching name.

1.2. Backend

As backend, we will use Flask. Data will be transfered in the JSON format. Very early and simplified api design might look something like this.

GET /customers/

Purpose: Return the list of customers with a name matching what was entered

IN:
    partial customername

RETURN:
    JSON
    [
      "Company One",
      "Company Two"
    ]
GET /customers/info/

Purpose: Return the information of the customer

IN:
    customername

RETURN:
    JSON
    {
      "change_by": 0,
      "change_time": null,
      "city": "Las Vegas",
      "comments": null,
      "country": "USA",
      "create_by": 0,
      "create_time": null,
      "customer_id": "A Company One",
      "name": "A Company One",
      "street": "A Company One Street",
      "url": null,
      "valid_id": 0,
      "zip": "45678"
    }

After this very rough and simple api design, we go on the creating the project. Off we go!

2. Create the project

Some useful info on organizing your code can be found here Organizing your project Depending on the size of your project, the way your structure your project can be quite different. Even if your project is small, it helps to factor in, if possible, if the project is going to keep on growing. If it does, start with an appropriate layout from the get go to avoid headaches later.

We create a project directory, with some subdirectories in it:

├── api
│   ├── static
│   │   ├── css
│   │   │   ├── flask_api_client.css
│   │   │   └── flask_api_main.css
│   │   ├── docs
│   │   │   └── html
│   │   │       ├── code.html
│   │   │       └── index.html
│   │   ├── img
│   │   │   ├── favicon.ico
│   │   │   └── logo.png
│   │   └── js
│   ├── templates
│   │   ├── 400.html
│   │   ├── 403.html
│   │   ├── 404.html
│   │   ├── 500.html
│   │   ├── base.html
│   │   └── index.html
│   ├── views
│   │   ├── customers
│   │   │   └── index.py
│   │   └── home
│   │       └── index.py
│   └── blueprints.py
├── client
│   └── index.html
├── config
│   └── default.py
├── data
│   ├── sqlite_partial_otrs_db.sql
│   └── temp.db
├── docs
│   ├── build
│   │   └── html
│   │       ├── code.html
│   │       └── index.html
│   ├── source
│   │   ├── code.rst
│   │   ├── conf.py
│   │   └── index.rst
│   └── Makefile
├── instance
│   └── config.py
├── logs
│   └── flask_api_otrs.log
├── tests
│   └── test_api.py
├── manage.py
├── README
├── LICENSE
└── requirements.txt

Create the virtual environment:

virtualenv venv
or
python -m "venv" venv

Now activate the environment:

source venv/bin/activate

Note

You could also use Autoenv. This allows you to create .env files with the activate command in it, while also having a place to put your export xyz statements in. These enviroment variables control how the app is started, what url the database is on and so on.

Install Flask and any other dependencies. If you downloaded the code, you could install the necessary tidbits in your virtualenv via the requirements file:

(venv)$ pip install -r requirements.txt

We will use Git as codecontrol. Creata a .gitignore files as we want some control over what files we add to the project:

vi .gitignore
# Lines starting with '#' are considered comments.
*.pyc
*.py~
*.wsgi~
*.db
build
dist
media
*egg*
*.log
*logs*
/logs/
*.rst~
*~
*debug.conf.py
*dev.py
tmp
*/main/migrations/*
*_local*
*.pid
venv
instance/config.py

Next, initialize the project:

git init .
git status

Now we're setup to start developing. I won't go over every detail in the code, that's what the code is for, just some of the main points.

3. Configuration

There are several ways on how to setup configuration for your project. I added a bit of all so yo show you how this can be done.

3.1. instance/config.py

A first method is to put the config parameters in config.py file in a directory or the root. This file contains the public configuration info. If you want to move the main public config to its own directory instead of in the root directory:

mkdir config
mv config.py config/default.py
touch config/__init__.py

For the private configuration info, we create a directory "instance" and put a config.py file in there containing all the private info.

For now, these files will only play a small role as we only set a DEBUG var, maybe a mailaddress and some info to access the SQLite database.

We first create a config/default.py file with a Config class that can be inherited for further customization. The instance/config.py file is used to override some of these settings.

Note

The default config file (whether config/default.py or config.py in the root folder) is added to source control. The local config file is not. This allows you to put sensitive data in the local file and not have it show up somewhere else.

The config/default.py file (or config.py as per your taste):

 1 import os
 2 
 3 class Config:
 4     """Basic Flask settings."""
 5     DEBUG=False
 6     SECRET_KEY = os.environ.get('SECRET_KEY') or 'myspecialsecretkey'
 7 
 8     """ OTRS Settings """
 9     DATABASE_NAME = 'otrs'
10     DATABASE_USER = 'otrs'
11     DATABASE_PASSWORD = os.environ.get('DATABASE_PASSWORD') or ""
12     DATABASE_HOST = os.environ.get('DATABASE_HOST') or ""
13     DATABASE_PORT = 5432
14 
15 
16 class DevelopmentConfig(Config):
17     DEBUG = True
18     DEBUG_TB_INTERCEPT_REDIRECTS = False
19     """Database path."""
20     BASEDIR = os.path.abspath(os.path.join(os.path.dirname(__file__),os.pardir))
21     DATABASE = os.path.join(BASEDIR,"data","temp.db")
22 
23 ...

As you can see, we even try to load some of the settings from the environment. More on that further in the document.

The configuration will help us avoid having debug mode on in production or using SQLite instead of PostgreSQL, or it will specify using SQLite for testing. To locally override these settings, use instance/config.py:

1 #DATABASE_HOST =  ""
2 #DATABASE_NAME = ""
3 #DATABASE_USER = ""
4 #DATABASE_PASSWORD = ""
5 #DATABASE_PORT = 5432
6 
7 """Flask settings."""
8 #SECRET_KEY = ""
9 ...

To create the app, we create a factory function. At app creation time, we load the configuration:

1 def create_app(config_name="default"):
2     ...
3     app = Flask(__name__, instance_relative_config=True)
4 
5     # Load the default configuration
6     app.config.from_object(config[config_name])
7 
8     # Load the configuration from the instance folder if one exists
9     app.config.from_pyfile('config.py', silent=True)

3.2. environment

Another way of loading a configuration is to specify some variables in the environment, typically done on Linux via the export command. These environmental vars can be accessed in the code with os.getenviron.get:

os.environ.get('DATABASE_PASSWORD')

This would search for an environmental variable DATABASE_PASSWORD. Often there is a bash script for instance start.sh that exports these variables before starting the application.

In both case, leave these files out of your version control since they (may) contain private information:

  1. instance/config.py
  2. start.sh (or whatever the script is called where you put the export statements)

Use .gitignore to do this:

vi .gitignore
...
instance/

or

vi .gitignore
...
start.sh

The advantage of using the instance directory is that the sensitive data can easily be imported in the tests. If using environment vars, it seems logical to also have to create a test script to set these environmental variables. However, the test database login info and so on might just not be that confidential or important. Then you could include this in the standard config.

A more thorough read up can be found here:

3.3. Start the app

To start the app with a start.sh script:

export DATABASE_PASSWORD='my-database-password'
export DATABASE_HOST="my-database-server"
export FLASK_CONFIG="development" # or test, production, ...

python manage.py

In this example, we have chosen not to do this and work with a manage.py script. This script creates the application, sets up the logger and finally starts the application. For convenience, the registered routes are also printed.

 1 ...
 2 app = create_app(os.getenv('FLASK_CONFIG') or 'default')
 3 
 4 ...
 5 def setup_logging(log_basedir="logs"):
 6     ...
 7 
 8 def list_routes(app):
 9     ...
10 
11 
12 if __name__=="__main__":
13     setup_logging()
14     ...
15     for route in list_routes(app):
16         log.info("Route {}".format(route))
17     app.run()

4. Api back-end functions

For our API, we need 2 functions. The functions return data in the JSON data format.

4.1. Search customers

This functions accepts a partial customer name and searches the customer database.

The client code is available in client/index.html. It has a form where you can enter the customer name:

1 <form id="customer_form" action="https://localhost:5000/customers/info" title="" method="get">
2     <input id="cust-search-field" type="text" placeholder="Type a customer company name..." pattern="[a-zA-Z0-9-_ ]{3,50}" autofocus required>
3     <input type="submit" id="submitButton" name="submitButton" value="Submit">
4 </form>

There is some javascript as well. First we have a function that triggers on keyup. It lauches an ajax request to the back-end function and receives a list of customers. The JQuery autocomplete functionality is used and only triggers after a minimum of 3 characters have been entered.

 1 <script>
 2 /* Autocomplete for search field */
 3 jQuery(function() {
 4     $("#cust-search-field").on('keyup', function(){
 5         var value = $(this).val();
 6         $.ajax({
 7             url: "https://localhost:5000/customers/",
 8             data: {
 9               'search': value
10             },
11             dataType: 'json',
12             success: function (data) {
13                 searchResult(data)
14             },
15             error: function () {
16                 console.log('error');
17             }
18         });
19     });
20   });
21 function searchResult(data) {
22     var $searchBox = $("#cust-search-field");
23     console.log(data);
24     /*list = data.list;*/
25     $searchBox.autocomplete({
26         source: data,
27         minLength: 3
28     });
29 }

The back-end function captures the search parameter we defined in the keyup Javascript function. Next it calls a helper function to execute an SQL search and return the jsonified list.

 1 @bp_customer.route("/")
 2 def get_customers():
 3     """Used to return a list of customers filtered on the search parameter
 4     Gets the search value from the request. You can manually add the search
 5     part by including '?search=<searchterm>' after the /customers/ link.
 6 
 7     Returns:
 8         Returns a 'Response' object containing the list of customers.
 9 
10     """
11     search = request.args.get('search')
12     log.debug("Searching '{}'".format(search))
13     customer = _get_customers(search)
14     if customer is not None and len(customer) > 0:
15         cust = [ cust_t[0] for cust_t in customer]
16         log.debug(cust)
17     else:
18         cust = None
19     resp = jsonify(cust)
20     resp.headers.add('Access-Control-Allow-Origin', '*')
21     return resp

4.2. Get customer info

This functions accepts a (partial customer) name, searches the customer database on info on that customer and returns it.

The client code is available in client/index.html. We use a div to display the results.

1 <h3>Company data</h3>
2 <div id="customer_info">
3 </div>

Since we allow partial customer names to be sent, this could result in no customers found or even multiple customers found. In such cases, we return None from back-end functions. To deal with these values, we have a hasValue helper function. We register a handler when the submit button is pressed to retrieve the result of the search.

 1 /* Helper function */
 2 function hasValue(data) {
 3     return (data !== undefined) && (data !== null) && (data !== "");
 4 }
 5 /* Submit customer search */
 6 /* Attach a submit handler to the form */
 7 $("#customer_form").submit(function(event) {
 8 
 9   /* stop form from submitting normally */
10   event.preventDefault();
11 
12   /* get the action attribute from the <form action=""> element */
13   var $form = $( this ),
14       url = $form.attr( 'action' );
15 
16   /* Send the data using post with element id name and name2*/
17   var posting = $.get( url, { search: $('#cust-search-field').val() });
18 
19   /* Alerts the results */
20   posting.done(function( data ) {
21         console.log('Receiving customer info');
22 
23         /* Pretty print some of the fields */
24         if (hasValue(data) ) {
25             var out = "";
26 
27             out = out.concat("<ul class='results'>");
28             out = out.concat("<li><span>Customer ID: </span>", data["customer_id"], "<li/>");
29             out = out.concat("<li><span>Street: </span>", data["street"], "<li/>");
30             out = out.concat("<li><span>City: </span>", data["zip"], ", ", data["city"], "<li/>");
31             out = out.concat("<li><span>Country: </span>", data["country"], "<li/>");
32             out = out.concat("</ul>");
33 
34             document.getElementById("customer_info").innerHTML = out;
35         }
36         else {
37             document.getElementById("customer_info").innerHTML = "<span style='font-weight: bold;'>No result found: no customer data found or more than one possible client found.</span>";
38         }
39   });
40 });

The back-end function captures the search parameter as with the first function. Next it calls a helper function to execute an SQL search and return the jsonified customer information.

 1 @bp_customer.route("/info")
 2 def get_customer_info():
 3     """Get the info of a customer. Expects a complete customer name
 4     Gets the search value from the request. You can manually add the search
 5     part by including '?search=<searchterm>' after the /customers/info/ link.
 6 
 7     Returns:
 8         Returns a 'Response' object containing a dictionary with the company information
 9 
10     """
11     customer = request.args.get('search')
12     log.debug("Searching '{}'".format(customer))
13     cust_info = _get_customer_info(customer)
14 
15     if cust_info is not None and len(cust_info) == 1:
16         log.debug("Customer data found")
17         cust_d = dict(zip(customer_fields, *cust_info))
18     else:
19         log.debug("No customer data found or more than one possible client.")
20         cust_d = None
21 
22     resp = jsonify(cust_d)
23     resp.headers.add('Access-Control-Allow-Origin', '*')
24     return resp

I've added some other functionality to the back-end (docs), api test, error handling. Have a look at the code to explore further.

The back-end

Client

The front-end

Client

5. Database access

For simplicity, the database used is SQLite. In a production environment this could be PostgreSQL or another DB. If you want change this, you should use SQLAlchemy as an ORM which makes interfacing with databases easier.

5.1. Use a production database

For instance if you would have an OTRS instance using a PostgreSQL database, you could interface with the database by changing the code in api/views/customers/index.py:

  • import psycopg2

  • edit execute_sql(sql)

    • uncomment this block of code:

      CON_PARAM = { 'dbname': current_app.config['DATABASE_NAME'],
      'user': current_app.config['DATABASE_USER'],
      'password': current_app.config['DATABASE_PASSWORD'],
      'host': current_app.config['DATABASE_HOST'],
      'port': current_app.config['DATABASE_PORT']
      
    • comment the sqlite3.connect line and uncomment the psycopg2 line:

      conn = psycopg2.connect(**CON_PARAM)
      #conn = sqlite3.connect(current_app.config['DATABASE'])
      
  • create a local file to put your settings in:

    $ vi instance/config.py
    

    These are possible settings:

    DATABASE_NAME - dbname: the database name
    DATABASE_USER - user: user name used to authenticate
    DATABASE_PASSWORD - password: password used to authenticate
    DATABASE_HOST - host: database host address
    DATABASE_PORT - port: connection port number
    SECRET_KEY= ''
    

6. Testing

Only a couple of example tests are provided to test basic testing. To be more complete when you design and develop an API, you could/should use coverage tests too. Testing the GUI is another possible suite of tests. To run the tests:

python -m unittest discover

7. Documentation

Documentation is done via Sphinx and the rst format. The documents are prebuilt for your convenience, something that might not be done in a production setting since you can just generate the docs your self.

To automatically document the code, the Sphinx autodoc extension is used. The source of the document is /docs/source/code.rst. We try to use the Google style comments as much as possible.

To build the documentation, from the command line:

$ source venv/bin/activate
$ make html

The output files are placed in /docs/build/html. Open the index.html or the code.html files in a browser.

As said before, for convenience the docs are available from the menu in the back-end application. In a production setting, you would use your webserver to set up a location to serve the docs and not have this served up by the Flask app.

8. Publish

The code is published in GitHub. In the future I might add more information on authentication and the use of SQLAlchemy.