Drilling Detection with Machine Learning Part 2: Segmentation Starter Kit

Geospatial Analyst Brendan Jarrell explains, step-by-step, how to develop a machine learning model to detect oil and gas well pads from satellite imagery.

[This is the second post in a 3-part blog series describing SkyTruth’s effort to automate the detection of oil and gas well pads around the world using machine learning. This tool will allow local communities, conservationists, researchers, policymakers and journalists to see for themselves the growth of drilling in the areas they care about. This is a central part of SkyTruth’s work: to share our expertise with others so that anyone can help protect the planet, their communities, and the places they care about. You can read the first post in the series here. All of the code that will be covered in this post can be found here. Our training dataset is also available here.]

SkyTruth Intern Sasha Bylsma explained in our first post in this series how we create training data for a machine learning workflow that will be used to detect oil and gas well pads around the world. In this post, I’m going to explain how we apply a machine learning model to satellite imagery, explaining all the tools we use and steps we take to make this happen, so that anyone can create similar models on their own.

Once we have created a robust set of training data, we want to feed a satellite image into the machine learning model and have the model scan the image in search of well pads. We then look to the model to tell us where the well pads are located and give us the predicted boundary of each of the well pads. This is known as segmentation, as shown in Figure 1. 

Figure 1: An example of our current work on well pad segmentation. The original image is seen on the left; what the ML model predicts as a well pad can be seen on the right. Notice that the algorithm is not only returning the drilling site’s location, but also its predicted boundaries.

We want the model to identify well pad locations because of the crucial context that location data provides. For example, location can tell us if there is a high density of drilling in the area, helping nearby communities track increasing threats to their health. It can also calculate the total area of disturbed land in the area of interest, helping researchers, advocates and others determine how severely wildlife habitat or other land characteristics are diminished.  

In the past, SkyTruth did this work manually, with an analyst or volunteer viewing individual images to search for well pads and laboriously drawing their boundaries. Projects like FrackFinder, for example, may have taken staff and volunteers weeks to complete. Now, with the help of a machine learning model, we can come in on a Monday morning, let the model do its thing, and have that same dataset compiled and placed on a map in an hour or two. The benefits of leveraging this capability are obvious: we can scan thousands of images quickly and consistently, increasing the likelihood of finding well pads and areas with high levels of drilling.

Formatting the data

So how do we do this? The first thing we need to do is get our data into a format that will be acceptable for the machine learning model. We decided that we would use the TensorFlow API as our framework for approaching this task. TensorFlow is an open-source (i.e. “free-to-use”) software package that was developed by Google to give users access to a powerful math library specifically designed for machine learning. We exported data from Google Earth Engine in the TFRecord format; TFRecords are convenient packages for exporting information from Earth Engine for later use in TensorFlow. In our code under the section labeled “Get Training, Validation Data ready for UNET,” we see that there are a few steps we must fulfill to extract the TFRecords from their zipped up packages and into a usable format (see Figure 2). 

# Bands included in our input Feature Collection and S2 imagery.

bands = ['R','G','B']
label = 'Label'
featureNames = bands + [label]
# Convert band names into tf.Features.

cols = [
         tf.io.FixedLenFeature(shape=[256,256],dtype=tf.float32) for band in featureNames
       ]

"""Pass these new tensors into a dictionary, used to describe pieces of the input dataset."""
featsDict = dict(zip(featureNames,cols))

Figure 2:  Preprocessing code

Second, we create Tensorflow representations of the information we are interested in drawing out of each of our examples from the Google Earth Engine workflow (see the first post in this series for more explanation on how we made these samples). Each of the samples has a Red, Green, and Blue channel associated with it, as well as a mask band, called “label” in our code. As such, we create Tensorflow representations for each of these different channels that data will be plugged into. Think of the representations we create for each channel name as sorting bins; when a TFRecord is unpacked, the corresponding channel values from the record will be placed into the bin that represents it. After loading in all of our TFRecords, we push them into a TFRecord Dataset. A TFRecord Dataset is a dataset which is populated by several TFRecords. We then apply a few functions to the TFRecord Dataset that make the records interpretable by the model later on.

Validation dataset

Once the dataset is loaded in, we split the dataset into two. This is an important part of machine learning, where we set aside a small amount of the whole dataset. When the model is being trained on the larger portion of the dataset, known as the training data, it will not see this smaller subset, which we call the validation set. As its name suggests, the model uses this smaller fraction of information to perform a sanity check of sorts. It’s asking itself, “Okay, I think that a well pad looks like this. Am I close to the mark, or am I way off?” All of this is put in place to help the model learn the minute details and intricacies of the data we’ve provided it. Typically, we will reserve 15-30% of our total dataset for the validation set. The code necessary for splitting the dataset is shown in Figure 3 below.

# Get the full size of the dataset.
full_size = len(list(data))
print(f'Full size of the dataset: {full_size}','\n')

# Define a split for the dataset.
train_pct = 0.8
batch_size = 16
split = int(full_size * train_pct)

# Split it up.
training = data.take(split)
evaluation = data.skip(split)

# Get the data ready for training.
training = training.shuffle(split).batch(batch_size).repeat()
evaluation = evaluation.batch(batch_size)

# Define the steps taken per epoch for both training and evaluation.
TRAIN_STEPS = math.ceil(split / batch_size)
EVAL_STEPS = math.ceil((full_size - split)  / batch_size)

print(f'Number of training steps: {TRAIN_STEPS}')
print(f'Number of evaluation steps: {EVAL_STEPS}')

Figure 3: Validation split code snippet

Implementation in U-Net

Now it’s time for the fun stuff! We’re finally ready to begin setting up the model that we will be using for our segmentation task. We will be leveraging a model called a U-Net for our learning. Our implementation of the U-Net in TensorFlow follows a very similar structure to the one seen in the example here. In a nutshell, here is what’s happening in our U-Net code:

1.) The machine learning model is expecting a 256 pixel by 256 pixel by 3 band input. This is the reason why we exported our image samples in this manner from Earth Engine. Also, by chopping up the images into patches, we reduce the amount of information that needs to be stored in temporary memory at any given point. This allows our code to run without crashing.

2.) The computer scans the input through a set of encoders. An encoder’s job is to learn every little detail of the thing we’re instructing it to learn. So in our case, we want it to learn all of the intricacies that define a well pad in satellite imagery. We want it to learn that well pads are typically squares or rectangles, have well defined edges, and may or may not be in close proximity to other well pads. As the number of encoders increases further down the “U” shape of the model, it is learning and retaining more of these features that make well pads unique.

3.) As the computer creates these pixel-by-pixel classifications sliding down the “U,” it sacrifices the spatial information that the input once held. That is to say, the image no longer appears as a bunch of well pads scattered across a landscape. It appears more so as a big stack of cards. All of the pixels in the original image are now classified with their newly minted predictions (i.e. “I am a well pad” or “I am not a well pad”), but they don’t have any clue where in the world they belong. The task of the upper slope of the “U” is to stitch the spatial information onto the classified predictions generated by our model. In this light, the upward slope of the “U” is made up of filters known as decoders. The cool thing about the U-Net is that as we go further up the “U”, it will grab the spatial pattern associated with the same location on the downward slope of the U-Net. In short, the model gives its best shot at taking these classified predictions and making them back into an image. To see a visual representation of the U-Net model, refer to Figure 4 below.

Figure 4: A graphic representing the U-Net architecture, courtesy of Ronneberger, et al.

At the end of the trip through the model, we are left with an output image from the model. This image is the model’s best guess at whether or not what we’ve fed it shows well pads or not. Of course, the model’s best guess will not be absolute for each and every pixel in the image. Given what it has learned about well pads, (how they’re shaped, what color palette usually describes a well pad, etc.), the model returns values on a spectrum from 0 to 1. Wherever the values land in between these two numbers can be called the model’s confidence in its prediction. So for example, forested areas in the image would ideally show a confidence value near zero; conversely, drilling sites picked up in the image would have confidence values close to one. Ambiguous features in the image, like parking lots or agricultural fields, might have a value somewhere in the middle of zero and one. Depending on how well the model did when compared to the mask associated with the three band input, it will be reprimanded for mistakes or errors it makes using what’s known as a loss function. To read more about loss functions and how they can be used, be sure to check out this helpful blog. Now that we have the model set up, we are ready to gear up for training!

Data augmentation

Before we start to train, we define a function which serves the purpose of tweaking the inputs slightly every time they are seen by the model. This is a process known as data augmentation. The reason why we make these small changes is because we don’t have a large dataset. If we give the model a small dataset without making these tweaks, each time the model sees the image, it will essentially memorize the images as opposed to learning the characteristics of a well pad. It’s a pretty neat trick, because we can make a small dataset seem way larger than it actually is simply by mirroring the image on the y-axis or by rotating the image 90 degrees, for example. Our augmentation workflow is shown in Figure 5.

# Augmentation function to pass to Callback class.
def augment(image, mask):
 rand = np.random.randint(100)
  if rand < 25:
   image = tf.image.flip_left_right(image)
   mask = tf.image.flip_left_right(mask)

 elif rand >= 25 and rand < 50:
   image = tf.image.rot90(image)
   mask = tf.image.rot90(mask)

 elif rand >= 50 and rand < 75:
   image = tf.image.flip_up_down(image)
   mask = tf.image.flip_up_down(mask)

 else:
   pass

 return (image, mask)

# Callback for data augmentation.
class aug(tf.keras.callbacks.Callback):
 def on_training_batch_begin(self, batch, logs = None):
   batch.map(augment, num_parallel_calls = 5)
   batch.shuffle(10)

Figure 5: Augmentation function and checkpoints cell

Fitting the model to the dataset

Now it’s time to put this model to the test! We do this in a TensorFlow call known as .fit(). As the name suggests, it is going to “fit” the model to our input dataset. Let’s go ahead and take a look at the code from Figure 6, shown below. 

history = UNet.fit(
     x = training,
     epochs = model_epochs,
     steps_per_epoch = TRAIN_STEPS,
     validation_data = evaluation,
     validation_steps = EVAL_STEPS,
     callbacks = [aug(),cp,csv])

Figure 6: Fitting the model to the input dataset

It’s important to conceptually understand what each of the values passed into this function call represents. We start with the variable “x”: this expects us to pass in our training dataset, which was created earlier. The next argument is called epochs. Epochs describe how many times the model will see the entire dataset during the fitting process. This is somewhat of an arbitrary number, as some models can learn the desired information more quickly, thus requiring less training. Conversely, training a model for too long can become redundant or potentially lead to overfitting. Overfitting is when a model learns to memorize the images it’s trained on, but it doesn’t learn to generalize. Think of overfitting like memorizing a review sheet the night before a test; you memorize what is covered in the review, but any minor changes in the way questions are asked on the actual test could trip you up. For this reason, it is generally up to the user to determine how many epochs are deemed necessary based on the application. 

The next argument, steps_per_epoch (also validation_steps) describes how many batches of data should be taken from our training and validation sets respectively through each epoch. Batches are small chunks of the dataset; it is useful to divide up the dataset into batches to make the training process more computationally efficient. One would typically want to go through the whole dataset every epoch, so it’s best to set the steps as such. Validation_data is where we would specify the data we set aside during training to validate our model’s predictions. Remember, that data will not be seen by the model during its training cycle. The last argument is called callbacks. This is where we pass in the augmentation function. This function is instructed by our callback to run at the beginning of each new batch, therefore constantly changing the data during training. We also optionally pass in other callbacks which might be useful for later reference to our training session. Such callbacks might export the loss and metrics to our Google Drive in a comma-separated values format or might save checkpoints throughout the model, keeping track of which training epoch produces the lowest loss. There are many other pre-packaged callbacks which can be used; a full list of these callbacks can be found here. Now that we have all of that covered, it’s time to start learning! By running this code, we begin the training process and will continue until the model has finished running through all of the epochs we specified.

Once that has finished, we save the model and plot its metrics and its loss, as shown in Figure 7. Based upon how these plots look, we can tell how well we did during our training.

Figure 7: An example chart, showing plotted metrics (top) and loss (bottom). Metrics are used to evaluate the performance of our model, while loss is directly used during training to optimize the model. As such, a good model will have a greatly reduced loss by the time we reach the end of training.

And voila! You have made it through the second installment in our series. The next entry will cover post-processing steps of our machine learning workflow. Questions we will answer include:

– How do we make predictions on an image we’ve never seen before?

– How do we take a large image and chop it into smaller, more manageable pieces? 

– How do we take some new predictions and make them into polygons?

Stay tuned for our next entry, brought to you by Dr. Ry Covington, SkyTruth’s Technical Program Director. In case you missed it, be sure to check out the first post in this series. Happy skytruthing!