Using the Peewee ORM in practice

As Discussed in the post Using an ORM Tool for SQLite , we used the Peewee ORM to act as an intermediate layer between our SQLite database and our Python code.

Model.py

When starting off with SQLite you would create,access and edit your database as shown in the SQLite Database post. With Peewee this process is made a lot easier. A ‘model.py’ file was created. The code from within this file is shown and explained below;


from peewee import * # Imports all the libraries from the Peewee

db = SqliteDatabase('UPVSite.db')# Creates a database(db) called UPVSite.db
''' Standard convention would be to have a Plural as the DB name and singular for the class name(s)'''<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>

class Turbine_Weather(Model):
      time = DateTimeField()
      incoming_data = TextField()

class Meta:
      database = db

The above code is the start of the ‘model.py‘ script. Apart from what is already commented within the code, it is creating a Model type class named ‘Turbine_Weather‘ that contains the characteristics  time and incoming_data, that are declared as a DateTime field and Text field respectively. DateTimeField() and TextField() are two out of a long list of built in field type specifiers within Peewee, a full list can be found at Peewee Field Types . The class named Meta is declaring that the Turbine_Weather Model will have Peewee’s database characteristics which will be important for use with Flask and Flask-Admin later on.


class SensorAccess(object):

# Initialises access to the database
def __init__(self): ''' Self stops confusion between SQLite syntax that is similar to Python syntax '''
    db.connect() # Connects to database
    db.create_tables([Turbine_Weather], safe=True) # Checks if table is made

# Function to add data readings to db table
def add_reading(self,time, incoming_data):
    Turbine_Weather.create(time = time, incoming_data = incoming_data )

# Function to return data correlating to a specific time passed to in
def get_(self, time ):
    data_at_time = Turbine_Weather.get(Turbine_Weather.time == time)
    return data_at_time 

# This is a function to return all rows of data
def get_readings(self):
    return Turbine_Weather.select()

# Function to return the latest data readings up to a certain limit in
# descending order
def get_recent_readings (self , limit = 30):
    return Turbine_Weather.select() \
                      .order_by(Turbine_Weather.time.desc()) \
                      .limit(limit)

# Closes the connection to the database
def close(self):
    db.close()

The above code is the final part of the ‘model.py‘ script. Here an Object type class is being made named ‘SensorAccess’. This class contains all functions that will relate to the sensor and related databases, although in this case we only have one db associated with the sensor object. All database query’s are made as functions e.g.  get_recent_readings(self, limit = 30). 

SerialRead.py

In our case to use Peewee to add the data coming in through the antennas connected to the serial port of the PI a second .py script was needed, we named it as ‘SerialRead.py‘. This script is very similar to the code that was implemented in the Saving Data to a .CSV post. It updates on that by using the functionality within the ‘model.py‘ to add the incoming data sent from the STM32L4 chip to the UPVSite database.


import sqlite3
import time
import datetime
import serial
import string

import model 

print ("Starting... ")

ser_in = serial.Serial(port = '/dev/ttyS0', # ttyS0 is for the RPi 3
                    baudrate = 19200,
                    parity = serial.PARITY_NONE,
                    stopbits = serial.STOPBITS_ONE,
                    bytesize = serial.EIGHTBITS,
                    timeout = 100)
print ("Connecting... ")
data = model.SensorAccess()

try:
    while True :            

        si = ser_in.readline().strip() # si = Serial in<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
        read_time = datetime.datetime.now()#reads in current time into read_time 

        print(si)
        data.add_reading(read_time, si)# Adds the current time and incoming data

        time.sleep(300)# Sleeps for 5 minutes before updating its readings 

In the code above the data is read in as usual.  The major difference is the ‘data.add_reading(read_time, si)’ that uses the ‘add_reading(self, time)’ function within our ‘model.py‘ to add read in data to the database.

SQLite Database

The previous post regarding CSV is useful for local monitoring but in order for it be accessed online the data must be added to an SQLite database. SQL (Structured Query Language) is a programming language which can be understood by a database, it is similar to a CSV in which it stores data in columns and rows but can be accessed more easily by a server. The package sqlite3 must be installed before use.

SQL_2.png
SQL Basics

The above screen capture is showing basic commands of sqlite3 command line interface. To create a .db file (Database file), the following command is used:

sqlite3 fileName.db

Luckily, SQL is very easy to read and completing tasks that are normally complex in other languages are far easier in SQL due to it being designed with server maintenance in mind, searching the database for a keyword is far faster than say a for loop iterating through the entirety of the data. All commands end with a semicolon, a basic command:

CREATE TABLE – the name of the table follows this command followed by the column details e.g. CREATE TABLE NewTable (Foo REAL, Bar INTEGER);

The data types are also very easy to interpret with int being INTEGER and float being REAL. Contrary to most languages the data type succeeds the variable name which may be confusing.

Information on SQLite

Although it makes no difference to the function of the code, it is common practice to include the SQL commands/data types (e.g. CREATE, REAL) as upper case, this way when the SQL is included in a Python script it is easily distinguishable. In our server setup, SQL and Python will be used alongside each other as well as the code being shared between two.


import sqlite3
import time
import serial
import string

print ("Starting... ")
serialData = serial.Serial(port = '/dev/ttyS0', # ttyS0 is for the RPi 3
                           baudrate = 19200,
                           parity = serial.PARITY_NONE,
                           stopbits = serial.STOPBITS_ONE,
                           bytesize = serial.EIGHTBITS,
                           timeout = 100)

my_Db = sqlite3.connect('SQL_DB.db')
cursor_Obj = my_Db.cursor()
'''
when selecting stuff in a query, multiple rows are returned so the cursor
remembers where in the query you are
'''

# Storing data from port to database
while True:
  windSpeed, date = t
  reading_time = int(time.time())
  windSpeed = serialData # serial variable read from COM port
  print('Date: {0} windSpeed = {1:0.2f}%'format(date, windSpeed))
  cursor_Obj.execute('INSERT INTO Table1 VALUES (?, ?)',
                    (reading_time, '{0} Wind Speed: ', windSpeed)# finish
  cursor_Obj.commit() # must commit after each
  time.sleep(1) # reads once a second<span 				

The above code has similar aims to the previous CSV file but with the inclusion of SQL code instead of simply sending the data to a spreadsheet. The use of SQL allows the data to easily managed and transferred. For our project, we are using a particular engine of SQL and that's SQLite, an open source engine which is the most used out of all.

To view the SQLite database in GUI form we used the SQL Studio, available for Windows and Mac for free and is far easier to use than the command line, it also shows the data in a far clearer view.

SQL Studio

We added Date along with the two types of information successfully transferred wirelessly from the L4 chip (the MCU we’re using).

 

 

Saving Data to a .CSV

A text file is useful for basic data storage and formatting but unfortunately cannot organise the data very well. An alternative to a .txt is CSV file (Comma Separated Values), this allows for the ease of entry of a text file but with the ability to have the data in rows and columns rather than in disarray in normal text.

CSV is similar to a Microsoft Excel file but it can be used in more than just MS Office products making it useful for Linux. A CSV file can be viewed in LibreOffice Calc, a spreadsheet-based program installed by default with Raspbian (Raspberry Pi OS). When data is sent from the Python script it’s commas determine the row and column of where the data is stored.

To store the data with a timestamp the Python datetime module must be used. This allows the Pi to access the current date when storing the data from the serial port. In the code below, a column was made called date and one row was added which contained the current time and date.

import datetime
import csv

time1 = datetime.datetime.now()
time1.strftime("%A %d %b %y %H") # strftime: date formatting

with open('MyCSV.csv', 'w', newline='') as file: # with is a better way to access a file
 write = csv.writer(file)
 write.writerow(['Date'])
 write.writerow([time1])

The screenshot below from LibreOffice shows the result.

CSV Screenshot
Date CSV

The Python script was updated to allow for serial input as well as adding additional columns for future data input such as wind speed and voltage generated by the turbine.

import datetime
import csv
import serial
import string

print ("Starting... ")
serialData = serial.Serial(port = '/dev/ttyS0', # ttyS0 is for the RPi 3
                           baudrate = 19200,
                           parity = serial.PARITY_NONE,
                           stopbits = serial.STOPBITS_ONE,
                           bytesize = serial.EIGHTBITS,
                           timeout = 100)
with open('MyCSV.csv', 'w') as file: # with is a better way to access a file
  write = csv.writer(file)
  write.writerow([' ', 'UPV', 'A. Bailey', 'J. Harding', 'P. Malone'])
  write.writerow([' ']) # adds a blank row
  write.writerow([' ', 'Date', 'Wind Direction', 'Wind Speed', 'Voltage', 'Current', 'Power'])
  write.writerow([' ']) # adds a blank row

while True:
  data = serialData.readline().strip() # Storing incoming data from serial channel to variable "data" and stripping string
  print(data) # displays in the terminal of Pi
  time1 = datetime.datetime.now()
  time1.strftime("%A %d %b %y %H") # strftime: date formatting
  write.writerow([' ', time1, data]) # leaves blank a column

The screen capture below shows all data required to be stored by the turbine in its appropriate columns.

CSV Full
Full Data CSV

Saving Serial Data to a .txt

Serial communication has been established between the L4 and the Pi, next is to store the sent data. To do this a knowledge of basic file input/output is needed in addition to the previous PySerial script in the earlier post. The built-in Python function, open(), is used to do this.

More Information: File Handling in Python

import datetime
import serial
import string

print ("Starting... ")
## Serial setup
ser = serial.Serial(port = '/dev/ttyS0', 
 baudrate = 19200,
 parity = serial.PARITY_NONE,
 stopbits = serial.STOPBITS_ONE,
 bytesize = serial.EIGHTBITS,
 timeout = 100)

print ("Connected")
x = open("/home/pi/New.txt", "a") # gives directory, "a" is for appending

while True: 
 readData = ser.readline().strip() # Storing incoming data in "readData" and stripping string 
 x.write(readData) # writes data to variable x, to file New.txt
 x.close()

In the previous post, there were problems with viewing the strings as whole words, the wind direction was being printed as long column of characters which wasn’t very legible so a new function had to be used in PySerial. when reading in the serial data read() was originally used but readline() is a better alternative as it prints the entire transmission.

In the serial set up, a parameter was added called timeout. Timeout determines how long the port reads data, to make it easy the timeout was made to be 100s seconds.

More Information: PySerial Documentation

Unfortunately, the other Telit transmitter was being used for other parts of the project so a direct serial connection was made between a laptop via USB. To use a Windows machine to connect, serial connection software must be installed. We used Termite as it’s easy to install and use.

Termite Available Here: Termite Download

 

Wireless Coms with Pi

This month we began to make real progress with the Pi, Alex and Phil transmitted the wind direction using a Telit RF transmitter connected to the Cortex L4’s Tx (Transmitting) pin. An identical Telit was on the other side of the lab connected to the Pi’s UART Rx (Receiving) pin.

 

Image result for stm32l4 mcu clone
L4 Chip

 

To connect the Rx on the Pi, UART had to enabled in the Pi configuration due to it being disabled by default. To change this the following commands were used:

$ cd /boot/  # changes directory to boot

$ ls  # this shows the contents of the boot directory

$ sudo nano config.txt # uses nano to edit config.txt (nano is a text editor in Linux)

At the end of the file UART was enabled: UART = 1, the Pi was restarted to apply changes.

UART: Universal Asynchronous Receiver-Transmitter, part of the computer used to handle asynchronous serial communications.

To wire the Pi, the Tx wire from the Telit was connected to the Rx of the Pi and the Tx of the Pi was connected to Rx of the Telit. Ground was also connected as well as the 3.3V to power the Telit from the Pi.

Pi Uart.png
Pi UART Pinout

 

To display the serial data on the Pi, pyserial needed to be installed. This followed the same procedure as with other programs. To set up the serial port the following code was used in a Python script.

import serial
ser = serial.Serial( # using the serial library
  port =' /dev/ttyS0', # setting the port as com port 0
  baudrate = 9600, # = transfer rate, must match the L4 setting
  parity = serial.PARITY_NONE, # must match the L4 setting
  stopbits = serial.STOPBITS_ONE, # must match the L4
  bytesize = serial.EIGHTBITS, # usually 7/8, match the L4
  timeout = 5) # determines how long the port is open 

print("Connected ") # to tell the user that it's connected

try:
  while True: # infinite loop
    if not ser.in_waiting(): # if the serial port is waiting
      data = ser.read() # variable made equal to what is read
      print data # prints out Rx data in single characters 

finally:
  ser.close() #cleanup

The transfer was a success, printing the wind direction as the magnet was moved across the array of sensors on the other end of the transmission. Unfortunately, bytes were being transferred meaning only single characters were received.

The next objective is to receive the data as a string which would be easier to work with and could pass the data to a text or HTML file.

Pi Basics

First order of business was to learn about GPIO and how to light LEDs, detect input, wiring, etc. It started with lighting an LED from a Python script, then detecting a state change of a push button wired to a GPIO pin.

Luckily, the Pi has a great deal of documentation and tutorials online explaining the operation so getting started was far easier than with the CubeMX. Taking input to Pi was done using Python. To get started with the Pi, a few packages needed to be installed. To install programs on the Pi, the Terminal had to be used along with its commands.

Typical commands:

To Install: $ sudo apt-get install Program

To Update: $ sudo apt-get update  To Apply Update: $ sudo apt-get upgrade

This was needed to be done for both Python and GPIO, to access the pins as an input the following code was used, a pull-down resistor was used in the wiring, therefore, it must be specified in the code. To wire the Pi, jumper cables are recommended as they are easy to switch between pins.

Image result for pi 3 pinout
Pi 3 Pin Layout

The above diagram shows the pin layout for the Raspberry Pi 3. This is very helpful for general use and I would recommend setting this as the desktop background for the Pi, this makes for easy wiring when using the GPIO pins.

import RPi.GPIO as GPIO # imports GPIO calling it GPIO

GPIO.setmode(GPIO.BCM) # setting up pins
GPIO.setup(4, GPIO.IN, pull_up_down=GPIO.PUD_DOWN) # pin4 as input using pull down
Simple Button Circuit
Input Button Press Test

The above diagram shows the button connected to a breadboard and a resistor to prevent damaging the GPIO pin.

import RPi.GPIO as GPIO  # GPIO library
GPIO.setmode(GPIO.BOARD) # set up BOARD pin numbering
GPIO.setup(10, GPIO.IN)  # set GPIO10 as input (button)
GPIO.setup(12, GPIO.OUT) # set GPIO12 as output (LED)

try:
  while True:
    if GPIO.input(10):    # if port 10 == 1
      print ("HIGH")
      GPIO.output(12, 1)  # tuns LED on
    else:
      print ("LOW")
      GPIO.output(12, 0)  # tuns LED off  

finally: # this block will run no matter how the try block exits
  GPIO.cleanup()          # resets pins

A script was then written to detect a state change from an input pin which then lit an LED from a GPIO pin. Luckily, Python’s syntax is very easy to use which allows tasks to be done far faster than could be done with lower level programming languages like C.