RESTful web services using Python, Flask, Docker and MongoDB
In this article I’ll be looking at creating a basic application that uses the following technologies: Python, Flask, Docker and MongoDB. This sample application should give you a very good idea of how all these technologies work and how they fit together. The sample project can be found here: https://github.com/DewaldOosthuizen/python_rest_tutorial.git
Before we dive into developing the application, it is important to understand the technologies being used as well as the API we are trying to create. Looking at the technologies first, we need to understand what they do and how they do it. The key to mastering these technologies is to go through the documentation. I will include the links to the documentation of each technology, however, I will not be discussing them in detail in this article. You can and definitely should read up on them in your own time.
2. Design our API
Before we start to develop an application, we need to understand what it is that we will be developing. It’s easy to get confused along the way and miss something important that your API should have catered for.
Our application will be a very basic one. So let’s list all the functions of our API:
- Allow the user to retrieve a “Hello World” message without any authentication on it.
- Allow users to register
- Allow users to retrieve all their saved messages once they are authenticated
- Allow users to save a new message if they are authenticated
When I speak about authentication, I simply mean a user name and password for the purpose of this article. So, how do we go about designing an API? We start by drawing a table format mapping the layout.
|Resource||Parameters||Method||Status Codes||Success Return|
|/hello||-||GET||200 – OK||Message of “Hello World”|
200 – OK
301 – Invalid Username
|Message of “Registration Successful”|
200 – OK
301 – Invalid Username
302 – Invalid Password
|List of all messages belonging to the user|
200 – OK
301 – Invalid Username
302 – Invalid Password
303 – Invalid Message
|Message of “Message has been saved successfully”|
The above table format shows the API design we will be following in this article.
- The resource column shows under which URL our functionalities will be accessible.
- The parameters define what the service will require to perform its computation and return the desired results as indicated in the status code column.
- We will be using the POST method, because we are sending information into the services. The /hello service will be a GET as it does not require any parameters and it will be a basic hello world example.
3. Project Structure
First you will need to use an editor/IDE of your choice. I will be using Visual Studio Code. You can download it at: https://code.visualstudio.com/download
Another good one to use is Atom. You can download it at https://atom.io/.
We will start by creating a folder called Project. Inside this folder we will create two (2) new folders. One will be called db and the other one web. The folder db will contain all database related code and the web folder will contain all restful service code. Inside the db folder we will create a new file called Dockerfile, without any file extension to it. We will also create a Dockerfile for the web folder. Also in the web folder, we will create a requirements.txt file and an app.py file. Then navigate back to the root of the Project folder and create a docker-compose.yml file. This will leave you with a structure similar to the image below:
Be careful with the highlighted docker-compose.yml file as it is not part of the web folder - it is at the root of the Project folder. Now that we have our project structure we can start to develop and set-up our application.
4. Setting up our docker compose file
What is docker compose? We use docker compose to allow us to build multiple docker files at the same time. This helps us to setup an entire ecosystem for our application and run it all as a whole, rather than run multiple docker files separately. What does a docker compose file look like? Have a look at the image below:
This is all that we will include in our docker compose file. This is a ‘.yml’ file so it follows the YAML standards. This means it is rather strict on formatting so be sure to indent everything exactly as it is indented in the image.
At the top of the file we specified “version: ‘3’”. This tells docker compose which version we want to use for our compose file. You can read more about it here: https://docs.docker.com/compose/compose-file/compose-versioning/.
After the version we specify our services. First we have our web application and it is referenced as web. We then have the build, ports and links parameters underneath it.
The build specifies where the docker file that needs to be built will be. Remember we created a Dockerfile inside the web folder so we will build this service under “./web”.
The ports we map to will be what port your application will run on. By default Flask will map to port 5000 and that is why we specify “5000:5000” as the port we wish to use and expose for our application.
Links is a list of services that our application depends on and we depend on the my_db service as seen in the file.
After the web service we specify the my_db service and where it can be built from. This is also the my_db service that is referenced under the links section in the web service. Although we have a Dockerfile at “./db” to build for our service, we could have also used the green commented out section shown in the image. This would allow us to use a docker image for Mongo directly instead of using a Dockerfile to build the database.
5. Creating our Docker files
We created a Dockerfile in the db folder and specified inside our docker compose file that we would like to build our database with the Dockerfile we created. So inside the Dockerfile we add the following:
Yes, it truly is that simple. This basically tells docker that we would like to use the pre-build Mongo container and we would like to use version 3.6.4 of Mongo. This is the official Mongo image for docker. There are also other non-official images available.
Now we need to create a container that our python application will run in. We created a Dockerfile in the web folder and we told docker compose where to build it from, so let’s populate it with some data.
FROM python:3First, pull a python image for our docker environment. We pull a version of python 3 as specified. Again this is a pre-build image that is already set up to use python 3.
We then create our work directory at usr/src/app. It is considered good practice to create a work directory.
COPY requirements.txt ./
Then we copy the requirements.txt file from our application into the work directory.
RUN python3 -m pip install --user --no-cache-dir -r requirements.txt
Run an install command to install all defined required packages that we will need for this application.
COPY . .
Once all the requirements are installed we will copy the entire project to the work directory.
CMD ["python", "app.py"]
After all the above, we run our application by telling docker we want to run the file app.py with the python command.
6. Defining all the application required packages
Inside the requirements.txt file we created under the web folder, we can now add the following python packages that our application will depend on:
Flask and flask_restful is what we will use to build. They are restful services using python code.
pymongo is used to allow python to interact with our Mongo database
bcrypt is used to encrypt user passwords before storing them inside our database.
All these specified packages will be installed inside our web Dockerfile with the following command
RUN python3 -m pip install --user --no-cache-dir -r requirements.txt
7. Creating our restful services
Now that we did our entire app setup we can go ahead and create our services. We will create our application in the app.py file we created under the web folder. First we will import all the packages we will require for our application.
Flask, flask_restful, pymongo and bcrypt are all packages that we installed from our requirements.txt file. Next we will initialize our application:
Line 6 and 7, we initialize our Flask app by passing __name__ to the Flask constructor and we then create the API by passing the app to the Api constructor. This basically created a rest service that we can add functionality to as we will see later on.
From line 9 to 11 we connect to our database instance. Line 9 we give our database details for the Mongo client we want to connect to. Notice that we used the database name my_db in our connection. This is because we named our database my_db in our docker-compose.yml file. If you named yours something else then you will have to use that name. The port 27017 is the standard port for Mongo. On line 10 we connect to a database on our Mongo instance called projectDB. If it does not exist pymongo will create it for us. Next, we tell pymongo to reference a user’s collection inside our projectDB. This is where we will store all our information.
Now that we have initialized our application and our Mongo database we can start to create our resources, but before we do that we are going to create some helper functions. These functions are used to reduce code duplication and we can re-use the functions, should we need to.
The userExist function on line 18 is used to check if a user name already exists in our database - we then return True or False based on the result. You will see that we use our users’ collection that we created on line 11 to do our lookup query. We then check if the count of all the records found are equal to 0 or not. This will be used to verify and check valid user names when a user registers.
The verifyUser function will be used to check if a user is authenticated and allowed to use our API. We will encrypt our user passwords with bycrypt upon registration and we will then store the encrypted password in our database. Every time a user tries to use one of our resources we will call the verifyUser function to check if the usersname exists and if the password matches with the one we stored in our database. On line 33 we compare the password we store in the database with the one the user has entered when trying to use our resource. The checkpw function from bcrypt will automatically encrypt the user’s password before comparing it with the encrypted one we have in our database. We will then return True or False based on the validity of the username and password.
The getUserMessages retrieves all the messages that belong to a user. We will use this function inside our resources whenever we want to retrieve a user’s messages.
These are the only three helper functions we will be using. We can now start with our resources. Now we have to refer back to the design we created for our API. We will follow that design.
Our first resource will be the “/hello” resource where we will simply display the text “Hello World!”. This is similar to the standard example found on Flaks website that was referenced in the beginning of this article.
To create a resource we start by creating a class and we then pass the Resource as the constructor of the class. This Resource is the one we imported from flask_restful. Because the Hello resource does not take any arguments, we can make it use a GET method. We do that on line 52. This allows this class to act as a GET resource and we can then tell the API under which URL this class can be found. We will get to this later on.
The second resource we will be creating is the “/register” resource. Since we will be sending in a username and a password to register on the API we will be using a POST method. We then get the data the user has sent to the API from the request by using request.get_json() and saving the result in an object called data. This request object is the one we imported from flask. We can then access these arguments passed by the user as seen on line 62 and 63. We call the data object and then access the values inside as follows: data[“username”]. Once we get the username and password from the user’s data we can check if the username exists in our database as seen on line 66. If the name exists we can return a status code of 301 and a message of Invalid username. Since we are returning a JSON object as a response we will need to use the jsonify function before returning our object.
Once the username is valid and has passed the userExist check we will then encrypt the password and generate a salt to encrypt it with. We use bcrypt for the encryption. The default gensalt function of bcrypt has an encryption length of 12. We will assume the user always enters a password and won’t perform checks to see if it is blank or invalid. If you are creating a production ready service, you will definitely have to perform more checks here.
- Check if the username is valid. If it is supposed to be an email address, you will check the format as well.
- Check if the user has entered anything for a password.
- If the password has to be at a certain level of security you will also check things like password length, use of symbols and case sensitive text.
All these checks will then return a related status code and message back to the user so that he/she knows what is wrong and how to fix it. But to save us some time we will not be looking at all these options. We will stick to the bare minimum requirements.
Once we encrypted the password we will add the user to our database. You can see this on line 77. We again use our user’s collection and insert a new record/document containing the username, encrypted password and an empty list of messages. After inserting the user’s details we return a successful result with a status code of 200 and a corresponding message of “Registration successful”. Again the result is a JSON object and therefore we use jsonify when returning the result. Now we have a registration resource that allow the users to add themselves to our database so that they can use our API.
Now that our users have a way to register and use our API, they will need a way to retrieve the messages that they store on our API. We will be using the helper function getUserMessages that we created to retrieve all the messages for us.
On line 108 we verify our user using our helper function verifyUser and then return a 302 status code with a message of Invalid password. If the user is verified successfully, we get all the messages that belong to the user and then add them in a success response and jsonify the result – this is on line 118.
The resource starting point, line 129 up to line 155, should look very similar to what we have done on the retrieve resource. The only difference is we also get the message from the requested JSON, then on line 157 we check if the message is not blank or null. If the message is empty we return a status code of 303 and the message “Please supply a valid message”.
If all these checks have passed we then get all the messages that belong to the user and add the message we got from the request into the list. We add into the list by using the append function on line 168. On line 170 we update the user’s data with the new list of messages and thereafter we return a status code of 200 along with the “Message has been saved successfully” message.
Now that we have all our resources we need a way to access them. We do this by adding them to the API along with the path they can be accessed by.
From line 187 up to line 190 we add all four of our resources to our API and we give them a path. To add the resources we use the add_resource function which takes two arguments. The first is the class name we gave our resource and the second is the path the resource can be accessed by.
Once we added all our resources we run our application on line 193. This is very important as this starts the application and we can tell it where to run. We will run our application on host 0.0.0.0 which is a localhost and we will enable the debug logging by passing True as the debug value. You should not do this in production servers as it could be a security risk, only do this while testing.
8. Starting our service
Now that we have completed our service, we can go ahead and start our docker container. Open your terminal and navigate inside the root folder of the project where we can run the following command:
sudo docker-compose build – This will use our docker compose file to build the docker files which we have created in the web folder and in the db folder. We can now run our next command.
sudo docker-compose up – When we have finished building the containers we can start them up using this command.
Once your container is running you should see something like:
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) in your terminal. This means that our application is running on localhost at port 5000.
Open another terminal and type in sudo docker ps. This will list all our active running containers. You should see two of them. You will be able to identify them by the ports that they run on.
9. Testing our API
Now that our container is running we can open our browser and type in localhost:5000/hello as a URL. We should see the following:
We have just accessed the first resource that we built. The Hello resource which is located at the “/hello” path.
The next step is to test the rest of our API. We will use a tool called Postman to test our API. You can download it here: https://www.getpostman.com/downloads/.
If you prefer a different tool to test your API with, feel free to look at the following alternatives:
Now that we have installed postman we can create a REST call as indicated in the image below:
Because we are sending data to the server, we will have to use a POST method. We call the register resource at localhost:5000/register as this is where our service is running. When selecting the body tab in postman
we can then select the raw body type and pass in the JSON object which contains the username and password as the body of the POST method. After clicking on the send button you should see the below result:
This is the result we were looking for. You can go ahead and try to call this service again with the same details and you should get an Invalid username error because that’s the way we designed our API.
Once we have registered our user we can go ahead and save a message for our user.
Our saving functionality will be under the “/save” path. We will require a username, password and a message to save. We then call our service from localhost:5000/save. Again this will be a post method because we are sending data to the service. Once we hit the send button you should see the below results:
This is the correct message, which means we succeeded in saving our message in the database. Go ahead and save a few different messages by changing the message text and hitting the send button again. You can also try to enter an invalid username and an invalid password to recreate our failed results and status codes.
Since we now have a user record and a message for our user, we can go and retrieve our messages.
Our “/retrieve” rest service call will bring back the messages for the authenticated user. All we need for this service is a username and password. Since we’re sending data to the service again we will use a POST method. When we hit the send button we should see the messages of our user with a status code of 200.
That wraps up the functionality of our API. This gives you a basic idea of how these technologies can work together to create something great. You can play around with this sample project and add functionality to it to get a better understanding of how everything works or to evolve it into a more secure and complex API.
I hope that you found this information valuable and that it helps you understand the application of each one of these technologies and how they communicate with one another.