Python Histograms, Box Plots, & Distributions

Starting here? This lesson is part of a full-length tutorial in using Python for Data Analysis. Check out the beginning.

Goals of this lesson

In this lesson, you'll learn how to:

• Discern the distribution of a dataset
• Describe the shape of the data with basic statistics
• Make histograms
• Compare distributions with histograms
• Make box plots

Using flight data, you'll learn how to better compare trends among airlines, adjusting your analysis based on how many flights an airline flies. By the end, you'll know which airlines and airports are more or less reliable—and maybe even make it to Thanksgiving on time this year!

Loading data into Mode Python notebooks

Mode is an analytics platform that brings together a SQL editor, Python notebook, and data visualization builder. Throughout this tutorial, you can use Mode for free to practice writing and running Python code.

For this lesson, you'll be using records of United States domestic flights from the US Department of Transportation. To access the data, you’ll need to use a bit of SQL. Here’s how:

1. Log into Mode or create an account.
2. Navigate to this report and click Duplicate. This will take you to the SQL Query Editor, with a query and results pre-populated.
3. Click Python Notebook under Notebook in the left navigation panel. This will open a new notebook, with the results of the query loaded in as a dataframe.
4. The first input cell is automatically populated with datasets[0].head(n=5). Run this code so you can see the first five rows of the dataset.

datasets[0] is a list object. Nested inside this list is a DataFrame containing the results generated by the SQL query you wrote. To learn more about how to access SQL queries in Mode Python Notebooks, read this documentation.

Now you’re all ready to go.

Getting oriented with the data

This lesson uses data about US flight delays. For a data dictionary with more information, click here.

After following the steps above, go to your notebook and import NumPy and Pandas, then assign your DataFrame to the data variable so it's easy to keep track of:

import pandas as pd
import numpy as np
    

data = datasets[0] # assign SQL query results to the data variable
data = data.fillna(np.nan)
    

data.head() # our flight data, including delay minute counts by type, and total delay upon arrival
    

Distributions

Flying can be a frustrating and long endeavor in the U.S., especially around holidays and hurricane season when flights are delayed hours, sometimes days. As consumers, we're generally aware of those trends, but what about the day-to-day trends of air travel delays? Which airlines are doing better at getting you there on time, and which airports are the worst to fly out of?

One of the best ways to answer questions like these is to look at the distributions of the relevant variables. You can think of the distribution of a dataset or variable as a list of possible values, and some indication as to how frequently each value occurs. For a quick refresher on distributions, check out this lesson.

Before you look at the distributions of delays across airlines, start by exploring which airlines have the most delays.

First, build the series indicating whether or not flights are delayed, just as you did in the previous lesson:

data['delayed'] = data['arr_delay'].apply(lambda x: x > 0) #from previous lesson
    

Now count the number of delayed flights for each airline. Since you're only after one value for each airline, you don't have to use .groupby(), .size(), and .unstack() as in the previous lesson. Instead, just filter the dataset, the count up the rows for each carries using .value_counts():

delayed_flights = data[data['delayed'] == True] #filter to only rows where delayer == True
delayed_flights['unique_carrier'].value_counts() #count the number of rows for each carrier
    

WN    21150
EV    11371
OO    10804
AA     9841
DL     9803
UA     8624
MQ     8060
US     6353
B6     4401
NK     2133
AS     2104
F9     1848
HA     1354
VX      781
Name: unique_carrier, dtype: int64
    

It might be a good idea to look at the proportion of each airline's flights that were delayed, rather than just the total number of each airlines delayed flights.

ABC of Analysis: Always Be Curious: A great analyst will always be skeptical and curious. When you reach a point in your analysis where you might have an answer, go a little further. Consider what might be affecting your results and what might support a counterargument. The best analysis will be presented as a very refined version of the broader investigation. Always be curious!

Proportion of flights delayed

To calculate the proportion of flights that were delayed, complete these 4 steps:

1. Group by carrier and delayed

Group flights by unique_carrier and delayed, get the count with .size() (as you did in the previous lesson, use .unstack() to return a DataFrame:

data.groupby(['unique_carrier','delayed']).size().unstack()
    

Since you're going to use this for further analysis, it will be helpful to use .reset_index() to clean up the index. While you're at it, assign the new DataFrame to a variable.

delayed_by_carrier = data.groupby(['unique_carrier','delayed']).size().unstack().reset_index()
delayed_by_carrier[:5]
    

2. Create total flight count column

Create a simple derived column that sums delayed and on-time flights for each carrier. If you want a refreshed on derived columns, return to this lesson:

delayed_by_carrier['flights_count'] = (delayed_by_carrier[False] + delayed_by_carrier[True])
delayed_by_carrier[:5]
    

3. Calculate the proportion of total flights delayed

Divide the number of delayed flights by the total number of flights to get the proportion of delayed flights. Store this as another derived column:

delayed_by_carrier['proportion_delayed'] = delayed_by_carrier[True] / delayed_by_carrier['flights_count']
delayed_by_carrier[:4]
    

4. Sort by the proportion of flights delayed

Finally, output the whole table, sorted by proportion of flights delayed by carrier:

delayed_by_carrier.sort_values('proportion_delayed', ascending=False)
    

This paints a very different picture - Southwest is squarely in the middle of the pack when it comes to the percentage of flights delayed.

Basic statistics: mean, median, percentiles

But what about the length of time passengers are delayed on each airline? It's plausible that the proportion of flight delays may be lower for some airlines, but the length of those delays may be much longer. Passengers care more about longer flight delays than shorter ones, and they might choose a different airline given a record of long delays. For passengers, which airlines seem more reliable? Or in more measured terms, How many minutes are flights delayed on average, for each airline?

Mean

The mean, or the average, gives you a general idea of how many minutes flights were delayed for each airline. .pivot_table() calculates the mean of the aggregated values by default. You can pivot on the column unique_carrier to see the mean delay time aggregated by airline:

data.pivot_table(columns='unique_carrier', values='arr_delay').sort_values(ascending=False)
    

unique_carrier
MQ    35.627406
F9    28.836953
NK    22.779670
OO    19.031663
EV    18.358520
UA    16.094772
AA    15.616299
B6    13.576129
WN    11.273536
US     7.671557
HA     6.458937
DL     4.118949
VX     3.833908
AS     1.731951
Name: arr_delay, dtype: float64
    

Note that because .pivot_table calculated mean by default, the above is effectively the same as if you were to explicitly pass the argument like this (and it produces the exact same results):

data.pivot_table(columns='unique_carrier', values='arr_delay', aggfunc=np.mean).sort_values(ascending=False)
    

unique_carrier
MQ    35.627406
F9    28.836953
NK    22.779670
OO    19.031663
EV    18.358520
UA    16.094772
AA    15.616299
B6    13.576129
WN    11.273536
US     7.671557
HA     6.458937
DL     4.118949
VX     3.833908
AS     1.731951
Name: arr_delay, dtype: float64
    

Basic statistics with .describe()

.describe() is a handy function when you’re working with numeric columns. You can use .describe() to see a number of basic statistics about the column, such as the mean, min, max, and standard deviation. This can give you a quick overview of the shape of the data.

Before using describe(), select the arr_delay series for all Southwest flights:

southwest = data[data['unique_carrier'] == 'WN']['arr_delay']
    

62030    18.0
62031    11.0
62032     9.0
62033    44.0
62034    42.0
Name: arr_delay, dtype: float64
    

In case that was confusing, here's what just happened, evaluated inside-out, then left to right:

1. data['unique_carrier'] == 'WN'] creates a boolean index that returns True for rows that represent Southwest flights
2. Wrapping that in data[...] applies the boolean index to the DataFrame data.
3. ['arr_delay'] reduces the columns to just the ['arr_delay'] column (and the index).

You can now run .describe() on this new object you've created to get basic statistics:

southwest.describe()
    

count    42020.000000
mean        11.273536
std         36.438970
min        -55.000000
25%         -9.000000
50%          1.000000
75%         19.000000
max        535.000000
Name: arr_delay, dtype: float64
    

This is a feature that really sets python apart from SQL or Excel. It would take a lot of work to get this information in either of those tools, but here it is as easy as adding the .describe() method.

Here's a quick breakdown of the above as it relates to this particular dataset:

• count: there are 42,020 rows in the dataset, which is filtered to only show Southwest (WN).
• mean: the average delay.
• std: the standard deviation. More on this below.
• min: the shortest delay in the dataset. In this case, the flight was very early.
• 25%: the 25th percentile. 25% of delays were lower than -9.00.
• 50%: the 50th percentile, or the median. 50% of delays were lower than 1.00.
• 75%: the 75th percentile. 75% of delays were lower than 19.00.
• max: the longest delay in the dataset: 535.00.

data[data['origin'] == 'ORD']['arr_delay'].describe()
      

count    9207.000000
mean       38.073205
std        63.261760
min       -49.000000
25%        -3.000000
50%        17.000000
75%        57.000000
max       751.000000
Name: arr_delay, dtype: float64
      

Histograms

If all of Southwest's flights are delayed five minutes, but American Airlines' flights are sometimes delayed a full day, you might say that the uncertainty of American Airline's delays are higher. It's also possible that when the flights aren't late, they're early, and it’s arguable that that's a point in favor of the airline. There’s a lot of nuance to this problem that would be best explored visually.

You can visually represent the distribution of flight delays using a histogram. Histograms allow you to bucket the values into bins, or fixed value ranges, and count how many values fall in that bin.

Let's look at a small example first. Say you have two bins:

A = [0:10]
B = [10:20]

which represent fixed ranges of 0 to 10 and 10 to 20, respectively.

And we have the data:

[1, 2, 5, 9, 12, 20]

Therefore, our bins contain:

A = [1, 2, 5, 9]
B = [12, 20]

So the histogram would look like this:

pd.Series([1, 2, 5, 9, 12, 20]).plot(kind='hist', bins=[0,10,20]) # `bins` defines the start and end points of bins
    

Something very important to note is that histograms are not bar charts. In a bar chart, the height of the bar represents a numerical value (such as number of delayed flights), but each bar itself represents a category—something that cannot be counted, averaged, or summed (like airline).

In a histogram, the height of the bars represents some numerical value, just like a bar chart. The bars themselves, however, cannot be categorical—each bar is a group defined by a quantitative variable (like delay time for a flight).

One way to quickly tell the difference is that histograms do not have space between the bars. Because they represent values on a continuous spectrum, a space between bars would have a specific meaning: it would indicate a bin that is empty. Bar charts can have spaces between bars. Because they show categories, the order and spacing is not as important.

Check out this blog post by Flowing Data for a great explanation of histograms.

numpy .arange()

Say you’re interested in analyzing length of delays and you want to put these lengths into bins that represent every 10 minute period. You can use the numpy method .arange() to create a list of numbers that define those bins. The bins of ten minute intervals will range from 50 minutes early (-50) to 200 minutes late (200). The first bin will hold a count of flights that arrived between 50 and 40 minutes early, then 40 and 30 minutes, and so on.

bin_values = np.arange(start=-50, stop=200, step=10)
print bin_values
    

[-50 -40 -30 -20 -10   0  10  20  30  40  50  60  70  80  90 100 110 120
 130 140 150 160 170 180 190]
    

Let's plot the distribution of Southwest flight delays as a histogram using the bin_values variable created above:

wn_carrier = data[data['unique_carrier'] == 'WN']
wn_carrier['arr_delay'].hist(bins=bin_values, figsize=[14,6])
    

Great! You can see that the vast majority of Southwest flights weren't more than 30 minutes late. Though the airline has many flights, the majority of them aren't late enough to make you regret going on vacation.

virgin_bin_values = np.arange(start=-50, stop=200, step=5)
virgin_carrier = data[data['unique_carrier'] == 'VX']
virgin_carrier['arr_delay'].hist(bins=virgin_bin_values, figsize=[14,6])
      

virgin_carrier.head()
      

filter_criterion = (virgin_carrier['arr_delay'] > 20) & (virgin_carrier['arr_delay'] < 25)
virgin_late_20 = virgin_carrier[filter_criterion]
print virgin_late_20['origin'].shape
print virgin_late_20['origin'].value_counts()
      

(48,)
SFO    13
LAX    12
DAL     4
EWR     4
JFK     4
IAD     3
ORD     2
FLL     2
LAS     1
LGA     1
DCA     1
BOS     1
Name: origin, dtype: int64
      

filter_criterion = (virgin_carrier['arr_delay'] > 15) & (virgin_carrier['arr_delay'] < 20)
virgin_late_30 = virgin_carrier[filter_criterion]
print virgin_late_30['origin'].shape
print virgin_late_30['origin'].value_counts()
      

(43,)
LAX    15
SFO    12
JFK     4
BOS     3
DAL     3
LGA     2
DCA     1
EWR     1
ORD     1
SAN     1
Name: origin, dtype: int64
      

Comparing distributions with histograms

Seeing one distribution is helpful to give us a shape of the data, but how about two?

Compare the distribution of two airlines with a similar number of total flights, US Airways and Envoy Air:

bin_values = np.arange(start=-50, stop=200, step=10)
us_mq_airlines_index = data['unique_carrier'].isin(['US','MQ']) # create index of flights from those airlines
us_mq_airlines = data[us_mq_airlines_index] # select rows
group_carriers = us_mq_airlines.groupby('unique_carrier')['arr_delay'] # group values by carrier, select minutes delayed
group_carriers.plot(kind='hist', bins=bin_values, figsize=[12,6], alpha=.4, legend=True) # alpha for transparency
    

unique_carrier
MQ    Axes(0.125,0.125;0.775x0.775)
US    Axes(0.125,0.125;0.775x0.775)
Name: arr_delay, dtype: object
      
    

The two distributions look similar, but not the same (the third color is where they overlap). You can use .describe() to see key statistics about the carriers:

group_carriers.describe()
    

unique_carrier       
MQ              count    11275.000000
                mean        35.627406
                std         58.444090
                min        -51.000000
                25%         -2.000000
                50%         17.000000
                75%         53.000000
                max        788.000000
US              count    13972.000000
                mean         7.671557
                std         34.672795
                min        -59.000000
                25%        -10.000000
                50%         -1.000000
                75%         13.000000
                max        621.000000
dtype: float64
    

Looks like delays on Envoy Airlines are more distributed than delays on US Airways, meaning that the values are more spread out.

Standard deviation

One of the measures you see above is std, standard deviation, which describes how flight delays are dispersed. In comparing the histograms, you can see that US Airways' delays are most concentrated between -20 to 20 minutes, while Envoy Air's flight delays are more distributed from 0 to 200 minutes. You can say that Envoy Air's delays are more dispersed than US Airways' delays, meaning that for a given flight on either airline, you would be less certain about the length of the delay for the Envoy flight.

The standard deviation measure is based on the statistical concept of normal distribution, or a common expected shape of distribution among various types of data. The value for standard deviation defines a range above and below the mean for which a certain percentage of the data lie. You can see in this visualization that, for a normal distribution:

• 34.1% of records fall between the mean and one standard deviation higher.
• 34.1% of records fall between the mean and one standard deviation lower.

So you might say that 68.2% of data falls within one standard deviation of the mean.

It's easiest to understand this by taking a look at just the standard deviations from the output above:

print 'Envoy Air', group_carriers.describe()['MQ']['std'], 'minutes from the mean', group_carriers.describe()['MQ']['mean']
print 'US Airways', group_carriers.describe()['US']['std'], 'minutes from the mean', group_carriers.describe()['US']['mean']
    

Envoy Air 58.4440903966 minutes from the mean 35.627405765
US Airways 34.6727946615 minutes from the mean 7.67155740052
    

You can see that Envoy Air has a mean of about 35 minutes and a standard deviation of about 58 minutes. This means that 62.8% of the Envoy Air flights are between 23 minutes early (35 - 58 = -23) and 93 minutes late (35 + 58 = 93).

By contrast, US Airways has a lower mean, indicating that its flights, on average, are less delayed. US Airways also has a lower standard deviation, meaning delays are typically closer to the mean. More specifically, 62.8% of US Airways flights fall between 27 minutes early (7 - 34 = -27) and 41 minutes late (7 + 34 = 41).

Based on this information, you might say that US Airways delays have lower dispersion than Envoy Air.

Where not to vacation in the middle of January

Let's look at another very real dimension of this data—flight delays segmented by airport. Airports suffer from various challenges, such as weather, runway design, and congestion that impact on-time flight departures, so let's find out how they stack up. We don't want to get stuck in the mountains over New Years (or do we?).

To make things easier, you can look at the 20 highest volume airports by origin:

hi_volume = data['origin'].value_counts()[:20]
hi_volume
    

ATL    12678
ORD    10046
DFW     9854
DEN     7441
LAX     7434
IAH     5762
PHX     5610
SFO     5551
LAS     4902
MCO     4318
LGA     4075
DTW     4048
CLT     3959
MSP     3789
EWR     3754
SLC     3740
BOS     3738
SEA     3639
JFK     3609
FLL     3052
Name: origin, dtype: int64
    

hi_volume_airports_names = hi_volume.index.tolist()
print hi_volume_airports_names
    

['ATL', 'ORD', 'DFW', 'DEN', 'LAX', 'IAH', 'PHX', 'SFO', 'LAS', 'MCO', 'LGA', 'DTW', 'CLT', 'MSP', 'EWR', 'SLC', 'BOS', 'SEA', 'JFK', 'FLL']
    

Filtering a list of values with .isin()

To create a boolean index where you’re looking for values matching anything in a list, you must use .isin() with the desired list of matches.

You can create a boolean index filtering for the records that originated ('origin') in the 20 highest volume airports.

hi_volume_airports = data[data['origin'].isin(hi_volume_airports_names)]
hi_volume_airports.head()
    

For comparison, the above use of .isin() in Python is equivalent to the following SQL query:

SELECT * FROM tutorial.flights
  WHERE origin IN ('ATL', 'ORD', 'DFW', 'DEN', 'LAX', 'IAH', 'PHX', 'SFO', 'LAS', 'MCO', 'LGA', 'DTW', 'CLT', 'MSP', 'EWR', 'SLC', 'BOS', 'SEA', 'JFK', 'FLL');

Box plots

Now that you have a DataFrame of flights originating from high volume airports, you might ask: where did the longest flight delays originate in January 2015?

You can create a pivot table that pivots the flight date on the airport, where the values are the mean of the flight delays for that day.

hi_volume_airports_pivots = hi_volume_airports.pivot_table(index='flight_date', columns='origin', values='arr_delay')
hi_volume_airports_pivots
    

You can use .describe() to see the mean and dispersion for each airport:

hi_volume_airports_pivots.describe()
    

It's tough to compare airports simply by looking at a big table of numbers. This will be easier if you group the records for each airport and overlay them, as you did with Envoy Air and US Airways:

airport_bins = numpy.arange(-10,100,3)
hi_volume_airports_pivots.plot(kind='hist', bins=airport_bins, figsize=[12,6], alpha=.4, legend=True)
    

Well, that's also tough to read. The method of overlaying distributions has limits, at least when you want to compare many distributions at once. Luckily, there's a one-dimensional way of visualizing the shape of distributions called a box plot.

Box plots are composed of the same key measures of dispersion that you get when you run .describe(), allowing it to be displayed in one dimension and easily comparable with other distributions. The components of box plots are:

— Information Dashboard Design, Stephen Few

Try using box plots to compare the day-to-day distribution of delays at each airport:

hi_volume_airports_pivots = hi_volume_airports.pivot_table(index='flight_date', columns='origin', values='arr_delay')
hi_volume_airports_pivots.plot(kind='box', figsize=[16,8])
    

As you can see, it's visually simpler to compare many distributions with box plots. Airports like JFK had significant dispersion of delays, while LGA was evenly distributed around the most frequent average delay. ORD, however, was almost twice as delayed all the time, compared to every other high volume airport. Expect snow delays in Chicago in January!

Leading into this analysis, we posed a few key questions. To answer the last, Which airports are the worst to fly out of?, you can now say that you will (almost certainly) be delayed if you are flying out of Chicago in January, based on 2015 data. If you can help it, avoid connecting flights in Chicago.

hi_volume_airports.pivot_table(index='flight_date', columns='unique_carrier')['arr_delay'].plot(kind='box')
      

Lesson summary

As you've seen in this lesson, placing data in contrast enables you to better understand it. While it’s clear that all airlines and airports suffered from some delay, you can use statistics to quickly draw out abnormal trends and occurrences across the data. The deviation of data from a trend is often clearly revealed in visualization, which allows yous to visually identify unusual events, and dig deeper.

In this lesson, you learned how to:

• Discern the distribution of a dataset
• Describe the shape of the data with basic statistics
• Make histograms
• Compare distributions with histograms
• Make box plots