Time series are a pivotal component of data analysis. This series goes through how to handle time series visualization and forecasting in Python 3. 

Time Series Visualization and Forecasting

### Introduction

Time-series analysis belongs to a branch of Statistics that involves the study of ordered, often temporal data. When relevantly applied, time-series analysis can reveal unexpected trends, extract helpful statistics, and even forecast trends ahead into the future. For these reasons, it is applied across many fields including economics, weather forecasting, and capacity planning, to name a few.

In this tutorial, we will introduce some common techniques used in time-series analysis and walk through the iterative steps required to manipulate, visualize time-series data.

## Prerequisites

This guide will cover how to do time-series analysis on either a local desktop or a remote server. Working with large datasets can be memory intensive, so in either case, the computer will need at least **2GB of memory** to perform some of the calculations in this guide.

For this tutorial, we’ll be using **Jupyter Notebook** to work with the data. If you do not have it already, you should follow our [tutorial to install and set up Jupyter Notebook for Python 3](https://www.digitalocean.com/community/tutorials/how-to-set-up-jupyter-notebook-for-python-3).


## Step 1 — Installing Packages
We will leverage the `pandas` library, which offers a lot of flexibility when manipulating data, and the `statsmodels` library, which allows us to perform statistical computing in Python. Used together, these two libraries extend Python to offer greater functionality and significantly increase our analytical toolkit.

Like with other Python packages, we can install `pandas` and `statsmodels` with `pip`. First, let’s move into our local programming environment or server-based programming environment:

```custom_prefix(sammy@ubuntu:~$)
cd <^>environments<^>
```

```custom_prefix(sammy@ubuntu:~/environments$)
. <^>my_env<^>/bin/activate
```

From here, let’s create a new directory for our project. We will call it `<^>timeseries<^>` and then move into the directory. If you call the project a different name, be sure to substitute your name for `<^>timeseries<^>` throughout the guide

```custom_prefix((my_env)\ssammy@ubuntu:~/environments$)
mkdir <^>timeseries<^>
cd <^>timeseries<^>
```

We can now install `pandas`, `statsmodels`, and the data plotting package [`matplotlib`](https://www.digitalocean.com/community/tutorials/how-to-plot-data-in-python-3-using-matplotlib#step-1-—-importing-matplotlib). Their dependencies will also be installed:

 ```custom_prefix((my_env)\ssammy@ubuntu:~/environments/timeseries$)
pip install pandas statsmodels matplotlib
 ```

At this point, we're now set up to start working with `pandas` and `statsmodels`.

## Step 2 — Loading Time-series Data

To begin working with our data, we will start up Jupyter Notebook:

```custom_prefix((my_env)\ssammy@ubuntu:~/environments/Timeseries$)
jupyter notebook
```

To create a new notebook file, select **New** > **Python 3** from the top right pull-down menu:

![Create a new Python 3 notebook](https://assets.digitalocean.com/articles/eng_python/JupyterNotebookPy3/jupyter-notebook-new.png)

This will open a notebook which allows us to load the required libraries (notice the standard shorthands used to  reference `pandas`, `matplotlib` and `statsmodels`). At the top of our notebook, we should write the following:

```python
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

After each code block in this tutorial, you should type `ALT + ENTER` to run the code and move into a new code block within your notebook.

Conveniently, `statsmodels` comes with built-in datasets, so we can load a time-series dataset straight into memory.

We'll be working with a dataset called "Atmospheric CO2 from Continuous Air Samples at Mauna Loa Observatory, Hawaii, U.S.A.," which collected CO2 samples from March 1958 to December 2001. We can bring in this data as follows:

```python
data = sm.datasets.co2.load_pandas()
co2 = data.data
```

Let's check what the first 5 lines of our time-series data look like:

```python
print(co2.head(5))
```

```
[secondary_label Output]
              co2
1958-03-29  316.1
1958-04-05  317.3
1958-04-12  317.6
1958-04-19  317.5
1958-04-26  316.4
```

With our packages imported and the CO2 dataset ready to go, we can move on to indexing our data.

## Step 3 — Indexing with Time-series Data

You may have noticed that the dates have been set as the index of our `pandas` DataFrame. When working with time-series data in Python we should ensure that dates are used as an index, so make sure to always check for that, which we can do by running the following:

```python
co2.index
```

```
[secondary_label Output]
DatetimeIndex(['1958-03-29', '1958-04-05', '1958-04-12', '1958-04-19',
               '1958-04-26', '1958-05-03', '1958-05-10', '1958-05-17',
               '1958-05-24', '1958-05-31',
               ...
               '2001-10-27', '2001-11-03', '2001-11-10', '2001-11-17',
               '2001-11-24', '2001-12-01', '2001-12-08', '2001-12-15',
               '2001-12-22', '2001-12-29'],
              dtype='datetime64[ns]', length=2284, freq='W-SAT')
```

The `dtype=datetime[ns]` field confirms that our index is made of date stamp objects, while `length=2284` and `freq='W-SAT'` tells us that we have 2,284 weekly date stamps starting on Saturdays.

Weekly data can be tricky to work with, so let's use the monthly averages of our time-series instead. This can be obtained by using the convenient `resample` function, which allows us to group the time-series into buckets (1 month), apply a [function](https://www.digitalocean.com/community/tutorials/how-to-define-functions-in-python-3) on each group (mean), and combine the result (one row per group).

```python
y = co2['co2'].resample('MS').mean()
```

Here, the term `MS` means that we group the data in buckets by months and ensures that we are using the start of each month as the timestamp:

```python
y.head(5)
```

```
[secondary_label Output]
1958-03-01    316.100
1958-04-01    317.200
1958-05-01    317.120
1958-06-01    315.800
1958-07-01    315.625
Freq: MS, Name: co2, dtype: float64
```

An interesting feature of `pandas` is its ability to handle date stamp indices, which allow us to quickly slice our data. For example, we can slice our dataset to only retrieve data points that come after the year `1990`:

```python
y['1990':]
```

```
[secondary_label Output]
1990-01-01    353.650
1990-02-01    354.650
               ...   
2001-11-01    369.375
2001-12-01    371.020
Freq: MS, Name: co2, dtype: float64
```

Or, we can slice our dataset to only retrieve data points between October `1995` and October `1996`:

```python
y['1995-10-01':'1996-10-01']
```

```
[secondary_label Output]
1995-10-01    357.850
1995-11-01    359.475
1995-12-01    360.700
1996-01-01    362.025
1996-02-01    363.175
1996-03-01    364.060
1996-04-01    364.700
1996-05-01    365.325
1996-06-01    364.880
1996-07-01    363.475
1996-08-01    361.320
1996-09-01    359.400
1996-10-01    359.625
Freq: MS, Name: co2, dtype: float64
```

With our data properly indexed for working with temporal data, we can move onto handling values that may be missing.

## Step 4 — Handling Missing Values in Time-series Data

Real world data tends be messy. As we can see from the plot, it is not uncommon for time-series data to contain missing values. The simplest way to check for those is either by directly plotting the data or by using the command below that will reveal missing data in ouput:

```python
y.isnull().sum()
```

```
[secondary_label Output]
5
```

This output tells us that there are 5 months with missing values in our time series.

Generally, we should "fill in" missing values if they are not too numerous so that we don’t have gaps in the data. We can do this in `pandas` using the [`fillna()` command](https://www.digitalocean.com/community/tutorials/an-introduction-to-the-pandas-package-and-its-data-structures-in-python-3#handling-missing-values). For simplicity, we can fill in missing values with the closest non-null value in our time series, although it is important to note that a rolling mean would sometimes be preferable.

```python
y = y.fillna(y.bfill())
```

With missing values filled in, we can once again check to see whether any null values exist to make sure that our operation worked:

```python
y.isnull().sum()
```

```
[secondary_label Output]
0
```

After performing these operations, we see that we have successfully filled in all missing values in our time series.

## Step 5 — Visualizing Time-series Data

When working with time-series data, a lot can be revealed through visualizing it. A few things to look out for are:

- **seasonality**: *does the data display a clear periodic pattern?*
- **trend**: *does the data follow a consistent upwards or downward slope?*
- **noise**: *are there any outlier points or missing values that are not consistent with the rest of the data?*

We can use the `pandas` wrapper around the `matplotlib` API to display a plot of our dataset:

```python
y.plot(figsize=(15, 6))
plt.show()
```

![Timeseries Visualization Figure 1](https://assets.digitalocean.com/articles/eng_python/Timeseries/part_1_fig_1.png)


Some distinguishable patterns appear when we plot the data. The time-series has an obvious seasonality pattern, as well as an overall increasing trend. We can also visualize our data using a method called time-series decomposition. As its name suggests, time series decomposition allows us to decompose our time series into three distinct components: trend, seasonality, and noise.

Fortunately, `statsmodels` provides the convenient `seasonal_decompose` function to perform seasonal decomposition out of the box. If you are interested in learning more, the reference for its original implementation can be found in the following paper, "[STL: A Seasonal-Trend Decomposition Procedure Based on Loess](http://www.wessa.net/download/stl.pdf)."

The script below shows how to perform time-series seasonal decomposition in Python. By default, `seasonal_decompose` returns a figure of relatively small size, so the first two lines of this code chunk ensure that the output figure is large enough for us to visualize.

```python
from pylab import rcParams
rcParams['figure.figsize'] = 11, 9

decomposition = sm.tsa.seasonal_decompose(y, model='additive')
fig = decomposition.plot()
plt.show()
```

![Timeseries Seasonal-Trend Decomposition Visualization Figure 2](https://assets.digitalocean.com/articles/eng_python/Timeseries/part_1_fig_2.png)

Using time-series decomposition makes it easier to quickly identify a changing mean or variation in the data. The plot above clearly shows the upwards trend of our data, along with its yearly seasonality. These can be used to understand the _structure_ of our time-series. The intuition behind time-series decomposition is important, as many forecasting methods build upon this concept of structured decomposition to produce forecasts.

## Conclusion
If you've followed along with this guide, you now have experience visualizing and manipulating time-series data in Python.

To further improve your skill set, you can load in another dataset and repeat all the steps in this tutorial. For example, you may wish to read a CSV file using the `pandas` library or use the `sunspots` dataset that comes pre-loaded with the `statsmodels` library: `data = sm.datasets.sunspots.load_pandas().data`.


In this tutorial, we will introduce some common techniques used in time-series analysis and walk through the iterative steps required to manipulate and visualize time-series data.

A Guide to Time Series Visualization with Python 3

In this tutorial, we will produce reliable forecasts of time series. We will begin by introducing and discussing the concepts of autocorrelation, stationarity, and seasonality, and proceed to  apply one of the most commonly used method for time-series forecasting, known as ARIMA.


A Guide to Time Series Forecasting with ARIMA in Python 3

### Introduction

In previous tutorials, we showed [how to visualize and manipulate time series data](https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-visualization-with-python-3), and [how to leverage the ARIMA method to produce forecasts from time series data](https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-arima-in-python-3). We noted how the correct parametrization of ARIMA models could be a complicated manual process that required a certain amount of  time.

Other statistical programming languages such a `R` provide [automated ways](https://www.rdocumentation.org/packages/forecast/versions/7.3/topics/auto.arima) to solve this issue, but those have yet to be officially ported over to Python. Fortunately, the Core Data Science team at Facebook recently published a new method called [`Prophet`](https://facebookincubator.github.io/prophet/), which enables data analysts and developers alike to perform forecasting at scale in Python 3.

### Prerequisites

This guide will cover how to do time series analysis on either a local desktop or a remote server. Working with large datasets can be memory intensive, so in either case, the computer will need at least **2GB of memory** to perform some of the calculations in this guide.

For this tutorial, we’ll be using **Jupyter Notebook** to work with the data. If you do not have it already, you should follow our [tutorial to install and set up Jupyter Notebook for Python 3](https://www.digitalocean.com/community/tutorials/how-to-set-up-jupyter-notebook-for-python-3).

## Step 1 — Pull Dataset and Install Packages

To set up our environment for time series forecasting with Prophet, let’s first move into our local programming environment or server-based programming environment:

```custom_prefix(sammy@ubuntu:~$)
cd <^>environments<^>
```

```custom_prefix(sammy@ubuntu:~/environments$)
. <^>my_env<^>/bin/activate
```

From here, let’s create a new directory for our project. We will call it `<^>timeseries<^>` and then move into the directory. If you call the project a different name, be sure to substitute your name for `<^>timeseries<^>` throughout the guide:

```custom_prefix((my_env)\ssammy@ubuntu:~/environments$)
mkdir <^>timeseries<^>
cd <^>timeseries<^>
```

We'll be working with the Box and Jenkins (1976) [Airline Passengers dataset](https://github.com/js2126/Air-Passengers), which contains time series data on the monthly number of airline passengers between 1949 and 1960. You can save the data by using the `curl` command with the `-O` flag to write output to a file and download the CSV:

```
curl -O https://assets.digitalocean.com/articles/eng_python/prophet/AirPassengers.csv
```

This tutorial will require the `pandas`, `matplotlib`, `numpy`, `cython` and `fbprophet` libraries. Like most other Python packages, we can install the `pandas`, `numpy`, `cython` and `matplotlib` libraries with pip:

```
pip install pandas matplotlib numpy cython
```

In order to compute its forecasts, the `fbprophet` library relies on the `STAN`  programming language, named in honor of the mathematician [Stanislaw Ulam](https://en.wikipedia.org/wiki/Stanislaw_Ulam). Before installing `fbprophet`, we therefore need to make sure that the `pystan` Python wrapper to `STAN` is installed:

```
pip install pystan
```

Once this is done we can install Prophet by using pip:

```
pip install fbprophet
```

Now that we are all set up, we can start working with the installed packages.


## Step 2 — Import Packages and Load Data

To begin working with our data, we will start up Jupyter Notebook:

```custom_prefix((my_env)\ssammy@ubuntu:~/environments/timeseries$)
jupyter notebook
```

To create a new notebook file, select **New** > **Python 3** from the top right pull-down menu:

![Create a new Python 3 notebook](https://assets.digitalocean.com/articles/eng_python/JupyterNotebookPy3/jupyter-notebook-new.png)

This will open a notebook which allows us to load the required libraries.


As is best practice, start by importing the libraries you will need at the top of your notebook (notice the standard shorthands used to  reference `pandas`, `matplotlib` and `statsmodels`):

```python
%matplotlib inline
import pandas as pd
from fbprophet import Prophet

import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
```

Notice how we have also defined the fivethirtyeight [`matplotlib` style](https://tonysyu.github.io/raw_content/matplotlib-style-gallery/gallery.html) for our plots.

After each code block in this tutorial, you should type `ALT + ENTER` to run the code and move into a new code block within your notebook.

Let's start by reading in our time series data. We can load the CSV file and print out the first 5 lines with the following commands:

```python
df = pd.read_csv('AirPassengers.csv')

df.head(5)
```

![DataFrame](https://assets.digitalocean.com/articles/eng_python/prophet/fig-1.png)

Our [DataFrame](https://www.digitalocean.com/community/tutorials/an-introduction-to-the-pandas-package-and-its-data-structures-in-python-3#dataframes) clearly contains a `Month` and `AirPassengers` column. The Prophet library expects as input a DataFrame with one column containing the time information, and another column containing the metric that we wish to forecast. Importantly, the time column is expected to be of the `datetime` type, so let's check the type of our columns:

```python
df.dtypes
```

```
[secondary_label Output]
Month            object
AirPassengers     int64
dtype: object
```

Because the `Month` column is not of the `datetime` type, we'll need to convert it:

```python
df['Month'] = pd.DatetimeIndex(df['Month'])
df.dtypes
```

```
[secondary_label Output]
Month            datetime64[ns]
AirPassengers             int64
dtype: object
```

We now see that our `Month` column is of the correct `datetime` type.

Prophet also imposes the strict condition that the input columns be named `ds` (the time column) and `y` (the metric column), so let's rename the columns in our DataFrame:

```python
df = df.rename(columns={'Month': 'ds',
                        'AirPassengers': 'y'})

df.head(5)
```

![DataFrame](https://assets.digitalocean.com/articles/eng_python/prophet/fig-2.png)

It is good practice to visualize the data we are going to be working with, so let's plot our time series:

```python
ax = df.set_index('ds').plot(figsize=(12, 8))
ax.set_ylabel('Monthly Number of Airline Passengers')
ax.set_xlabel('Date')

plt.show()
```

![Time Series Plot](https://assets.digitalocean.com/articles/eng_python/prophet/fig-3a.png)


With our data now prepared, we are ready to use the Prophet library to produce forecasts of our time series.

## Step 3 — Time Series Forecasting with Prophet

In this section, we will describe how to use the Prophet library to predict future values of our time series. The authors of Prophet have abstracted away many of the inherent complexities of time series forecasting and made it more intuitive for analysts and developers alike to work with time series data.

To begin, we must instantiate a new Prophet object. Prophet enables us to specify a number of arguments. For example, we can specify the desired range of our uncertainty interval by setting the `interval_width` parameter.

```python
# set the uncertainty interval to 95% (the Prophet default is 80%)
my_model = Prophet(interval_width=0.95)
```

Now that our Prophet model has been initialized, we can call its `fit` method with our DataFrame as input. The model fitting should take no longer than a few seconds.

```python
my_model.fit(df)
```

You should receive output similar to this:

```
[secondary_label Output]
<fbprophet.forecaster.Prophet at 0x110204080>
```

In order to obtain forecasts of our time series, we must provide Prophet with a new DataFrame containing a `ds` column that holds the dates for which we want predictions. Conveniently, we do not have to concern ourselves with manually creating this DataFrame, as Prophet provides the `make_future_dataframe` helper function:

```python
future_dates = my_model.make_future_dataframe(periods=36, freq='MS')
future_dates.tail()
```

![DataFrame](https://assets.digitalocean.com/articles/eng_python/prophet/fig-4.png)

In the code chunk above, we instructed Prophet to generate 36 datestamps in the future.

When working with Prophet, it is important to consider the frequency of our time series. Because we are working with monthly data, we clearly specified the desired frequency of the timestamps (in this case, `MS` is the start of the month). Therefore, the `make_future_dataframe` generated 36 monthly timestamps for us. In other words, we are looking to predict future values of our time series 3 years into the future.

The DataFrame of future dates is then used as input to the `predict` method of our fitted model.

```python
forecast = my_model.predict(future_dates)
forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].tail()
```

![Predict Model](https://assets.digitalocean.com/articles/eng_python/prophet/fig-5.png)

Prophet returns a large DataFrame with many interesting columns, but we subset our output to the columns most relevant to forecasting, which are:

- `ds`: the datestamp of the forecasted value
- `yhat`: the forecasted value of our metric (in Statistics, [`yhat`](http://www.chegg.com/homework-help/definitions/predicted-value-y-hat-31) is a notation traditionally used to represent the predicted values of a value `y`)
- `yhat_lower`: the lower bound of our forecasts
- `yhat_upper`: the upper bound of our forecasts

A variation in values from the output presented above is to be expected as Prophet relies on Markov chain Monte Carlo (MCMC) methods to generate its forecasts. MCMC is a stochastic process, so values will be slightly different each time. 

Prophet also provides a convenient function to quickly plot the results of our forecasts:

```python
my_model.plot(forecast,
              uncertainty=True)
```

![Forecast Plot](https://assets.digitalocean.com/articles/eng_python/prophet/fig-6.png)

Prophet plots the observed values of our time series (the black dots), the forecasted values (blue line) and the uncertainty intervals of our forecasts (the blue shaded regions).

One other particularly strong feature of Prophet is its ability to return the components of our forecasts. This can help reveal how daily, weekly and yearly patterns of the time series contribute to the overall forecasted values:

```python
my_model.plot_components(forecast)
```

![Components of Forecasts Plots](https://assets.digitalocean.com/articles/eng_python/prophet/fig-7.png)

The plot above provides interesting insights. The first plot shows that the monthly volume of airline passengers has been linearly increasing over time. The second plot highlights the fact that the weekly count of passengers peaks towards the end of the week and on Saturday, while the third plot shows that the most traffic occurs during the holiday months of July and August.

## Conclusion

In this tutorial, we described how to use the Prophet library to perform time series forecasting in Python. We have been using out-of-the box parameters, but Prophet enables us to specify many more arguments. In particular, Prophet provides the functionality to bring your own knowledge about time series to the table. 

Here are a few additional things you could try:

- Assess the effect of holidays by including your prior knowledge on holiday months (for example, we know that the month of December is a holiday month). The official documentation on [modeling holidays](https://facebookincubator.github.io/prophet/docs/holiday_effects.html) will be helpful.
- Change the range of your uncertainty intervals, or forecast further into the future.


For more practice, you could also try to load another time series dataset to produce your own forecasts. Overall, Prophet offers a number of compelling features, including the opportunity to tailor the forecasting model to the requirements of the user.




This tutorial shows how to produce time series forecasts using the Prophet library in Python 3.

A Guide to Time Series Forecasting with Prophet in Python 3

Sign up for Infrastructure as a Newsletter.

Working on improving health and education, reducing inequality, and spurring economic growth? DigitalOcean would like to help.

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

You get paid; we donate to tech nonprofits.

DigitalOcean makes it simple to launch in the cloud and scale up as you grow – whether you’re running one virtual machine or ten thousand.

Community

Company

Contact

Products

Solutions

DigitalOcean Kubernetes: new control plane is faster and free, enable HA for 99.95% uptime SLA