SkyTruth Board Member Darshan Karwat: Reimagining Engineering

Darshan Karwat wants to put social and environmental justice at the heart of engineering.

Darshan Karwat thinks big. Really big. Like as big as the cosmos. Or like reinventing an entire profession. So he asks a lot of questions. Questions that challenge conventional thinking. Questions that many people have never even thought about.

Darshan is an Assistant Professor in the School for the Future of Innovation in Society, College of Global Futures at Arizona State University (ASU).  He’s also a SkyTruth board member.  At ASU, Darshan runs re-Engineered, which his website describes as “…an experiment in an interdisciplinary engineering profession that embeds social justice, peace, and ecological holism at the very heart of technical design and decision making.”

When I asked him what this means, Darshan provided an example: He said he believes that most engineers choose their profession to make a positive difference in the world. But when they leave school, job opportunities are limited, often focusing on work that makes the most money such as weapons systems or drilling and mining for big corporations. This can cause  some engineers to leave the profession and others to just take these jobs regardless, because they studied engineering and need to make a living. They might not admit that somewhere along the way they lost their passion for their work. 

“That’s not to say that some amazing work isn’t happening,” he says. But I do feel that as engineers we might be telling ourselves the work we do is making the world a better place. When it comes down to it, is that actually true?  Is it? Are we having the beneficial impact we claim that we’re having?”

Darshan notes that the world we live in has largely been created by, designed by, and built by engineers. Therefore, the world that we desire — one that is just for all people, environmentally sustainable, and peaceful — is going to have to be built as well, and engineers should be at the heart of it.

Darshan conducting combustion chemistry experiments at the University of Michigan. Photo by Scott Wagnon.

How can engineering promote these things? One way, according to Darshan, is to stop doing work that promotes weapons production and technological development that leads to violent conflict and horrible pollution.

Another way is to consider design. Darshan wants to include people who traditionally have been excluded from the design process.

I think we should ask during the engineering design process: In what ways does this design actually promote social justice and not injustice?” he says. For example, instead of designing phones or electric cars for people with a lot of money to spend, what if we design them for people of modest means?  What about electric vehicles for people who can’t afford the latest Tesla? “To what extent do we care that people in inner city Detroit have access to electric cars?” Darshan asks. “If that’s a goal, it influences how we design cars.”

Then there’s space. Darshan earned his Ph.D. in aerospace engineering and environmental ethics from the University of Michigan and has been fascinated by rockets, space, and stars since he was two years old. He is still fascinated and hopes one day to do more in aerospace engineering.  Amidst the current discussions about going to Mars he asks: “Who is that for? I do believe that what goes on beyond the earthly realms ought to be for all of us. It’s something that inspires all of us. If so, then in what ways are we designing space systems or policy for everyone rather than just as a playground for the wealthy? 

He notes that we haven’t really asked people how space matters to them, but cites the Hubble telescope as an example of aerospace engineering that likely benefited everyone. According to NASA, in the 30 years since its launch, Hubble has made more than 1.4 million observations, sent back stunning images of stars, galaxies, and other astronomical objects, and deepened humanity’s understanding of the cosmos.  In contrast, Darshan notes that many visions for the future of space travel rely on privatizing space, not on the ways in which space is for all of our imaginations. 

So how does an aerospace engineer who dreamed of space as a boy growing up in Mumbai, India lead a movement to reshape an entire profession? Darshan says he was sensitized to environmental issues early on because of the poor air quality in Mumbai. As an undergraduate at the University of Michigan he became an environmental student activist. And he became aware that a lot of what we know about what’s happening on earth comes from satellites. For his senior project, he helped design an earth imaging orbitor.

When the time came for graduate school he chose a project related to climate change; namely, the combustion chemistry of biofuels (an alternative to fossil fuels). Then, a class on the history of science and technology triggered his thinking about the values people bring to science and engineering. And so his doctoral dissertation also looked critically at engineers in the biofuel and aviation industries and their motivation, asking engineers basic questions such as:  Why are you doing this?  And how are you doing this? His work explored how engineers view climate change: It is just another  technological problem to solve? Or is it something completely new and different?

After completing his Ph.D., Darshan moved  to Washington, D.C., as an American Association for the Advancement of Science (AAAS) Fellow. There, he worked on low-cost air sensors and climate resilience at the Environmental Protection Agency and on ocean wave energy devices at the Department of Energy before heading to ASU.

Darshan characterizes his team’s work as aspirational and nascent (he started at ASU in 2017). He established Project Confluence to work with communities and ask key questions about the challenges they face, particularly as they relate to environment, climate, and energy. Right now, his small team (a postdoctoral fellow, a graduate student, and an undergraduate) is also working collaboratively with a community group on understanding the intersection of air pollution, energy justice, and social justice in historically marginalized parts of Phoenix. Through geospatial analysis, his team is taking data that is generally inaccessible to the public, analyzing who is being burdened with environmental harms, and exploring what community assets might be mobilized to address those harms. He’s given several presentations to community stakeholders taking his team’s research directly to the people who asked for it. 

Darshan talking about the role of engineering in society during the opening of re:Engineered’s public art installation called “When Mental Walls Lead to Physical Walls.” Photo by Tanya Harrison.

His lab is also asking broader questions to understand the potential for scaling up community-based engineering. That is, how can more engineers actually do community-based, environmental and social justice work?  Darshan notes that existing programs helping engineers find this kind of work are too small. 

“Why aren’t hundreds or thousands of engineers across the US doing this kind of work?” he asks. “When you look at lead pollution or air quality around industrial facilities, fracking across Pennsylvania and elsewhere, the scale of the energy transitions required in rural and urban communities in the US…that’s the scale of the challenge, and that points to the number of engineers needing to engage. And how do we design a financially viable system of engineering and science to serve a community in Appalachia or Lubbock, Texas, or elsewhere where there’s opportunity for this?”

To that end, his team has been surveying engineers. Darshan thinks they’ve identified five primary reasons why scientists and engineers aren’t doing environmental and social justice work. Time and money are part of it. But they’ve discovered that engineers often don’t feel equipped to do this kind of work, even when they want to. They see it as outside their skill set or comfort zone. Or, they look at the world in a totally different way from the communities they would serve, making it difficult to collaborate. His team has a paper coming out soon in the journal Earth’s Future elaborating on these findings.

In addition, Darshan is a coordinator of the Constellation Prize to recognize engineers and their collaborators working for environmental protection, social justice, human rights, and peace.  (Nominations close October 31, 2020.)

Lastly, Darshan leads SkyTruth’s Opportunities Committee to help our board think creatively about how SkyTruth can engage different kinds of partners, ask different kinds of questions, and push us in new directions. One of the things that excites him about working with SkyTruth is learning how space-based data informs our ability to protect the Earth.  “Seeing that unfold is super inspiring,” he says. “Super super inspiring.  What SkyTruth is doing is really cool and needs to happen more, needs to be supported more.” 

At SkyTruth, we’re thrilled to have Darshan as part of our brain trust. We believe we are contributing to his vision by providing engineers and others the opportunity to create a greener, healthier, more just world, and innovating new approaches to using satellite data to accomplish this. Our internships give advanced undergraduate and graduate students real world professional experience and energize them to make the world a better place. But as Darshan notes, plenty of other opportunities exist —or can exist —for engineers to do good in the world. They just have to think big.

And ask a lot of questions.

SkyTruth’s Project Inambari Wins at Artisanal Mining Challenge for Rainforest Conservation

SkyTruth and partners at Wake Forest University and the Center for Amazonian Scientific Innovation win funding for a public map platform to track mining in the Amazon.

In southeastern Peru the eastern slopes of the Andes have been known as a gold producing region going back to the time of Incas. There, wild torrential rivers drop down in narrow gorges from the high Andes before feeding into the broad meandering tributaries of the Amazon. For millennia these waters have carried tiny flecks of gold which have slowly accumulated in river banks and soil.

This gold now places at risk one of our planet’s most remote and biodiverse rainforests. In the past few decades thousands of migrants have come to this once isolated region hoping to escape poverty with work in informal gold mining operations which have spread out from the muddy banks of once pristine rivers into the dense tropical forest. While gold has brought prosperity to some, the consequences for local communities have been severe. They include large scale devastation of natural habitats and contamination from the mercury used in extracting gold. This contamination now poisons fish and people who have long depended on the river for sustenance.

Drone footage of the La Pampa mining area in Madre de Dios, Peru from June 2019 following Peru’s Operation Mercury against illegal mining. Video courtesy of CINCIA.

On September 30, SkyTruth, along with its partners at Wake Forest University and the Center for Amazonian Scientific Innovation (CINCIA) were selected as a winner at the Artisanal Mining Grand Challenge for our proposal to develop a public map that tracks mining in the Amazon using satellite radar imagery. We call this effort Project Inambari after one of the rivers in the region that has been particularly devastated by illegal and unregulated gold mining.

My interest in a monitoring project in this area began in 2017, when I had a chance to travel to the city of Puerto Maldonado in Peru’s southern jungle and see for myself the devastating consequences of illegal mining. A new highway connecting Peru’s highlands and Pacific coast to Brazil had opened up the area to a flood of migrants looking for better pay in areas made newly accessible for mining and logging. Environmental devastation, crime, and forced labor attracted global attention to the area — even a visit from Pope Francis in January 2018. During my visit I was able to connect with researchers at CINCIA who were beginning to tackle the daunting question of how to restore a landscape stripped of topsoil and littered with toxic pools. 

At SkyTruth, we approached this challenge by exploring how radar satellite imagery could detect illegal mining activities in vast, remote, and often cloudy rainforest areas. SkyTruth has worked extensively with satellite radar for two marine applications: mapping offshore infrastructure and detecting slicks from vessels dumping oily bilge at sea. I was intrigued when I found a demonstration of automated land cover classification using the same satellite radar source, Sentinel-1 from the European Space Agency. I showed this to SkyTruth Geospatial Engineer Christian Thomas. Christian leads SkyTruth’s effort to map the expansion of mountaintop removal mining in Appalachia. In the Amazon, radar has the advantage of penetrating the frequent clouds that cover the forest during the rainy season and make it difficult to monitor using optical imagery. Christian was able to get promising results using the same classification methods to show the expansion of mining in the Madre de Dios region of southern Peru, where the Inambari River meets the Madre de Dios River. 

Then, in January 2020 we learned from one of our colleagues at the Wildlife Conservation Society about the Artisanal Mining Grand Challenge, an opportunity to win support for innovative technology projects addressing artisanal mining. We decided to enter our proposal for a satellite radar-based detection map. 


Watch a video of SkyTruth’s Bjorn Bergman explaining the inspiration for Project Inambari and what we hope the project can accomplish.

For the initial March 2020 ASM Grand Challenge deadline we prepared a written application detailing how our public monitoring map proposal could protect biodiversity, water quality, and human security in addition to outlining our plans for technical development, adoption, and financing. Much of our inspiration comes from the impact we have seen from the Global Fishing Watch map, which offers near real-time tracking of fishing activity around the world. And so at the core of our proposal we wanted to have a free public map showing mining activity. Software developer Dan Cogswell has produced the Project Inambari map, based on SkyTruth’s Alerts platform. Our map prototype allows you to play a timeline of detected mining, view and compare activity from one month to another, and overlay changes with satellite imagery. Future work will allow users to receive email alerts when new mining activity is detected in their areas of interest.

We were excited to learn on June 4 that, out of 90 teams, we had been selected to move on to the semi-finalist round. Each of the 26 semi-finalist teams produced a ten-minute pitch video discussing their proposal, potential users, and the impact we wanted to have. You can watch our team explain how Project Inambari will monitor for mining in SkyTruth’s pitch video below.

 

Then, on August 17 we were notified that we were one of 11 project teams that would compete for funding in the final round of the Artisanal Mining Challenge. 

One of the most interesting aspects of the competition process has been viewing proposals from many different groups — including university researchers, nongovernmental organizations, and entrepreneurs — to tackle the diverse environmental and social issues associated with artisanal mining around the world. The innovations included fascinating recovery strategies like phytoremediation (using plants to pull heavy metals out of polluted soils) and proposals for in-depth supply chain monitoring. We were particularly interested to see the work put forward by researchers at Wake Forest University and CINCIA (based in Madre de Dios, Peru) for a database of known mining sites to use as training data for a machine learning model. In addition, CINCIA is leading efforts to find ways to recover lands devastated by mining and to monitor and evaluate these areas with drone imagery. It was clear that their proposals would complement the ideas that we had for developing a public map showing mining activity, so we decided to join forces in the finalist round.

You can see Christian and Jorge Caballero, a GIS analyst at CINCIA, discuss some new developments with the mining detection model in the video below, and the potential to integrate CINCIA’s high resolution drone imagery for training and validation.

The video also briefly discusses the development of an email alert system where authorities or local communities can receive notifications when our model picks up new mining activity in an area of interest. The map already shows some interesting changes, such as how mining activity shifted to a legal mining corridor and to more remote areas such as the Pariamunu River following the Peruvian government’s Operation Mercury, which ramped up government action against illegal mining.  

Finally, our team gathered virtually at the Conservation X Labs Innovation Summit on September 30 for a final round of live questions from the Artisanal Mining Grand Challenge judges. The next morning we had the honor of hearing from Jane Goodall. She summed up the difficult quandary posed by artisanal mining, stressing both the great threat posed by mining to ecosystems and human livelihoods but also her realization that any attempts at habitat conservation would fail unless local people could find sustainable alternatives for making a living. All the teams then waited anxiously as prizes were given out. Project Inambari was chosen as one of five winners with a $118,000 prize. 

With this support from Conservation X Labs and our new collaboration with researchers at Wake Forest University and CINCIA, we are excited to kick off the next stage of development of the Project Inambari map. It’s been inspiring to be part of the challenge competition along with so many great teams and a real honor to have Project Inambari selected as a winner. 

 

Matthew Ibarra’s Final Project Was His Favorite

As a SkyTruth Intern, Matthew Ibarra learned new skills and helped protect Native lands.

As I finish up my internship at SkyTruth, I can honestly say that the experience has been everything I imagined it would be and more. My time here was a perfect amalgamation of what I love: namely, an organization that applies technology and gathers and analyzes data to protect the environment. 

When I started my internship at SkyTruth I was unsure of what to expect. I remember the first day I drove into the small town of Shepherdstown, West Virginia. I was worried. For the first time in my life I was working with like-minded individuals with special talents and skills far above my own. I thought that I would have to perform as well as my colleagues right off the bat. However, my fears quickly melted away upon meeting Brendan Jarrell, SkyTruth’s Geospatial Analyst and father to all us interns. Brendan assured me that I would be focusing on my own personal interests, developing practical skills, and applying them to the various projects happening at SkyTruth. Within my first week I became familiar with all the programs I needed for my internship, namely Google Earth Engine and QGIS. Both are programs that are critical in geospatial analysis that were completely new to me, despite having taken Geographic Information System (GIS) courses at West Virginia University. Interning at SkyTruth opened my eyes to the new possibilities of environmental monitoring and I was excited to get started.

My very first day I became familiar with the various satellites that orbit the Earth and provide the imagery that SkyTruth uses on a daily basis. The Landsat and Sentinel satellite missions provide imagery available for free to the public, allowing interns like myself to create maps and interactive data to track activity on Earth. My first task as an intern was to monitor Southeast Asian waters for bilge dumps — oily slicks of wastewater dumped in the ocean by ships. I used Google Earth Engine to access the necessary imagery easily. Then I used QGIS to create the various maps that we post on our Facebook page and blog posts. I found my first bilge dump on February 7, 2020. It was a 35 kilometer slick (almost 22 miles long) off the coast of Malaysia. 

Often, we can identify the likely polluter using Automatic Identification System (AIS) to track vessels at sea. Most vessels constantly send out a radio signal to broadcast their route. When those signal points align with a bilge dump it suggests that the ship is the likely source for that bilge slick. However, not all ships will transmit their signal at all times, and there have even been instances of ships spoofing their signal to different locations. For my first slick I was unable to match a ship’s AIS broadcast to the trail of the bilge dump, but I was able to do so several  times after that. We can’t know for certain who caused this slick, but imagery helps us paint a picture of potential suspects. My first slick pales in comparison to the many slicks I found in later months: later, I captured a few slicks that were over 100 kilometers (more than 62 miles) in length. I was also able to link a ship’s AIS broadcast to the trail of the slick. You can read more about slicks in the South East Asia region in my April 15 blog post here.

 

An example of likely bilge dumping from a vessel identified using Sentinel satellite imagery

Following my introduction to our bilge dumping detection work, I was thrilled to be assigned my first solo project for SkyTruth — updating SkyTruth’s FrackFinder. FrackFinder is an ongoing project at SkyTruth. It aims to keep track of the active oil and natural gas well pads in states such as West Virginia. Drilling permit data is often misleading; sites that are permitted to be drilled may not actually be drilled for several years. In the past, our FrackFinder app was hosted in Carto. Carto is a cloud-based mapping platform that provides limited GIS tools for analysis. I was tasked with giving the application an overhaul and bringing it into Google Earth Engine, a much more powerful and accessible program. 

Learning to code for Earth Engine was challenging for me. I had only one computer science course in college, and that was nearly three years ago. So I was surprised that my first project would revolve around coding. Initially, I was overwhelmed and I struggled to find a place to start. As time went on I slowly became more comfortable with spending large amounts of time solving tiny problems. Brendan was incredibly helpful and patient with teaching me everything I would need to know to be successful. He always made time for me and assisted me with my code numerous times. My finished app is far from perfect but I am proud of the work that I accomplished and I hope that it brings attention to the changing landscape of West Virginia caused by oil and natural gas drilling using hydraulic fracturing (fracking). 

 

The FrackTracker app for West Virginia

My second and final project was creating a visualization about the land surrounding Chaco Culture National Historical Park in New Mexico. Much like the update to the FrackFinder App, it involved the changing landscape surrounding the park due to the increase in fracking. I was tasked with creating a series of still images, an embeddable GIF which shows an animation of the rapid increase in drilling, and an app on Earth Engine that allows the user to zoom in and visually inspect each individual well surrounding the park. In the final months of my internship, I became comfortable using the programs that were foreign to me initially. I created a series of 19 images using QGIS from the years 2000-2018. You can see the collection of images for each year here. SkyTruth’s Geospatial Engineer Christian Thomas assisted me in creating the GIF. 

This project was special to me because I was able to help activists who are advocating for the passage of the Chaco Cultural Heritage Area Protection Act, legislation passed by the U.S. House of Representatives that would effectively create a 10-mile buffer zone surrounding the park and ensure the protection of the area for Native Americans and local communities for generations to come. The Senate has not yet passed the act. When I started my internship at SkyTruth I never would have believed that I would be advocating for protection of Native lands. I always believed issues like these were too big for one person to tackle, but if there’s anything I learned at SkyTruth is that only one person can create real change.

The growth of oil and gas wells within a 15-mile radius of Chaco Culture National Historical Park from 2000 – 2018

After interning at SkyTruth for the past eight months I am happy to say that I feel I have made a difference in the world. I accomplished so much that I thought would be impossible for me initially. I used to think oil slicks were tragedies that happened infrequently, limited to a few times a decade. I was shocked to learn that oily wastewater gets dumped into the ocean so frequently that I was able to capture more than  80 bilge dumps in my eight months at SkyTruth. 

In addition, one of my greatest passions is sustainable energy. I was thrilled to be an advocate for clean energy by showcasing the dangers of an ever-expanding oil and natural gas industry. West Virginia has been my home for the past five years during my time at West Virginia University and I was happy to be able to bring to light one of the growing concerns of the state through the 2018 FrackFinder update. Finally, I was able to advocate for the protection of Native lands through the most meaningful project to me — the Chaco Culture National Historical Park visualizations. It felt incredible fighting for something that was much bigger than myself. As I leave SkyTruth, I will miss contributing to the world in my own way.

SkyTruth has always been more than just a place to intern at for me. I have made unforgettable connections with my colleagues despite the various challenges that we all have to face every day, such as the ongoing COVID-19 pandemic. Never once did I feel that I was alone in my work. I always knew there were people supporting me and encouraging me in my projects even when I was working remotely. I will never forget Christian’s tour of Shepherdstown on my first day or Brendan’s talks about the best Star Wars movie. I cannot thank each of them enough for the patience and kindness they showed me in my short time with them. Everyone at SkyTruth has contributed to my success in some way. I will miss everyone, but I’ll carry my new skills and experiences with me for the rest of my life.   

Drilling Detection with Machine Learning Part 3: Making and mapping predictions

SkyTruth Technical Program Director Ry Covington, PhD explains challenges to generating meaningful predictions from the machine learning model, and outlines solutions.

[This is the final post in a 3-part blog series describing SkyTruth’s effort to automate the detection of oil and gas well pads around the world using machine learning. We hope that this series – and the resources we’ve provided throughout – will help other developers and scientists working on conservation issues to learn from our experience and build their own exciting tools.  You can read the two previous posts in the series here and here.]

SkyTruth Intern Sasha Bylsma and Geospatial Analyst Brendan Jarrell explained how we create training data and implement a machine learning model to detect oil and gas well pads. So, now that we’ve got a trained model, we just need to run it on a few satellite images and put the predictions on a map.  Seems easy enough…  

We started with some Sentinel-2 imagery collected over the Neuquén Basin in central Argentina. This is one of the most heavily drilled areas in Argentina, and we’ve used the Google Earth Engine platform to export a few Sentinel-2 images that we could work with.  

Screenshot of the Neuquén basin in Argentina. 

The images come out of Earth Engine as GeoTIFFs – a pretty standard file format for overhead imagery. We’ve used some Earth Engine magic to reduce the file size of each image so that they’re easier to handle, but they’re still a bit big for the model. The model expects simple, small patches of images: 256 pixels high, 256 pixels wide, and three channels (e.g., Red, Green, Blue) deep. Our Sentinel-2 GeoTIFFs are about 11,000 pixels high by 11,000 pixels wide, so that left us with a few things to figure out:

  • First, the model is expecting small, simple patches of images – no frills, just pixel values. That means that we have to take the geographic information that’s imbedded in the original GeoTIFF and set aside.  So, how do we do that? 
  • Second, how can we evenly slice up the full image into the small patches that the model is expecting?
  • Third, for every small image patch that the model sees, it returns a small, single channel prediction image with values between zero and one. The closer a pixel is to one, the more likely it is that pixel belongs to a drilling site.  But once the model makes predictions on all of the small images, how do we put them back together in the right order to get the original image’s size and shape?
  • Lastly, how do we take the prediction image and convert it into polygons that we can overlay on a map?

These were all things that we’d never done before, so it’s taken us quite a bit of trial and error to figure out how to make things work. In fact, we’re still working on them – we’ve got a workflow in place, but we’re always trying to refine and improve it. For now, let’s just look at what we’ve got working. 

Step 1. Converting our GeoTIFF to a NumPy array

We started off with a pretty small subset of a Sentinel-2 image that we could experiment with. It’s 1,634 pixels high, 1,932 pixels wide, and 3 channels deep. In developer’s parlance, its shape is: (1634, 1932, 3). The image is of Mari Menuco Lake in Argentina. There are a few dozen drilling sites along the southern portion of the lake that seemed like an ideal place to test out our workflow. Once we had everything working like expected, we’d run the larger Sentinel-2 images through.  

First, we used the GDAL Python API to load our image and collect its (a) geotransform and (b) projection. So, what are these two things? Well, basically, the geotransform is the formula that GDAL uses to go from pixel space (think rows and columns) to geographic space (think x [longitude] and y [latitude]), and the projection is just the coordinate reference system of the image. After we had those two pieces of information set aside for later, we pushed all of the image bands into an NumPy array. 

 

# Get geographic information.
projection = tiff.GetProjection()           

# Set spatial reference.
spatial_ref = osr.SpatialReference()     # Create empty spatial reference.
spatial_ref.ImportFromWkt(projection)    # Read the "projection" string.

# Collect all the bands in the .tif image.
bands = [tiff.GetRasterBand(band+1) for band in range(tiff.RasterCount)]

# Read each band as an array.
arrays = [band.ReadAsArray() for band in bands] 

# Combine into a single image. 
image = np.array(arrays)

# Format as (height, width, channels).
image = image.transpose(1,2,0)

GDAL reads and writes images differently than NumPy, so the last thing we did was transpose the axes to put our image in the format that we needed: height, width, channels.   

Step 2. Slicing our NumPy array and running predictions 

The next bit was tricky for us to figure out. We needed to take our image – 1634 by 1932 by 3 (1634, 1932, 3) – and slice it into even squares of (256, 256, 3). Our first problem: neither 1634 nor 1932 divides by 256 evenly, so we needed to figure out how to make the image patches overlap just enough to get a clean division.  

Our second problem: we also needed to keep track of where each patch lived in the larger image so that we could put the predictions back together in the right order later. We ended up giving each patch an ID and collecting the coordinates of their top-left pixel (their minimum x and minimum y). We pushed that information into a pandas dataframe – basically, a 2-D matrix of rows and columns – that we could set aside to rebuild our prediction image later.  

Many thanks to CosmiQ Works and all of the participants in the SpaceNet Challenges; the code snippets and GitHub repos that they’ve made available were immensely helpful for us as we tried to figure out how to implement this step.

 

# Set some variables.
patchSize = 256
overlap = 0.2
height, width, bands = image.shape
imgs, positions = [], []
columns = ['xmin', 'ymin']

# Work through the image and bin it up.
for x in range(0, width-1, int(patchSize*(1-overlap))):    
   for y in range(0, height-1, int(patchSize*(1-overlap))):
      
       # Get top-left pixel.
       xmin = min(x, width-patchSize)
       ymin = min(y, height-patchSize)

       # Get image patch.
       patch = image[ymin:ymin+patchSize, xmin:xmin+patchSize]

       # Set position.
       pos = [xmin, ymin]

       # Add to array.
       imgs.append(patch)
       positions.append(pos)

# Convert to NumPy array.
imageArray = np.array(imgs) / 255.0

# Create position datataframe.
df = pd.DataFrame(positions, columns=columns)
df.index = np.arange(len(positions))

Once we had the new array of patches – 80 patches of 256 by 256 by 3 – it was easy to run the model and generate some predictions.

# And, go. Don't forget the batch dimension.
predictions = model.predict(imageArray, batch_size=20, steps=4)

Step 3. Rebuilding our image

The model returns an array of predictions – (80, 256, 256, 1). The prediction values range from zero to one. So, a pixel value of .82 means that the model is 82% confident that pixel belongs to a drilling site.  

Side by side comparison of an image and its prediction.

We used the pandas dataframe that we made earlier to put all of these predictions back together in the right order and get the original image’s size and shape. The dataframe was where we recorded the ID and top-left pixel (their minimum x and minimum y) of each patch. First, we created an empty image that is the same size and shape as the original. Next, we went through the dataframe, took out each patch, and added it to the new empty image in the right spot (its top-left pixel).  

 

# Create numpy zeros of appropriate shape.
empty_img = np.zeros((height, width, 1))

# Create another zero array to record where pixels get overlaid.
overlays = np.zeros((height, width, 1))

# Iterate through patches.
for index, item in positions.iterrows():

   # Grab values for each row / patch.
   [xmin, ymin] = item

   # Grab the right patch.
   slice = predictions[index]
  
   x0, x1 = xmin, xmin + patchSize
   y0, y1 = ymin, ymin + patchSize

   # Add img_slice to empty_img.
   empty_img[y0:y1, x0:x1] += slice

   # Update count of overlapping pixels.
   overlays[y0:y1, x0:x1] += np.ones((patchSize, patchSize, 1))            

# Normalize the image to get our values between 0 and 1. 
rebuilt_img = np.divide(empty_img, overlay_count)

Most of our team are visual thinkers, so the easiest way for us to imagine rebuilding the image is like covering a blank sheet of white paper in pink sticky-notes, and then smoothing them all down to get a new, blank sheet of pink paper.   

Step 4. Converting our predictions to polygons

After rebuilding our prediction array to be the same size and shape as the original satellite image, we used the GDAL Python API to convert it into polygons that could go on a map. To try and clean things up a bit, we started by selecting only those pixels where the model was more than 70% confident they belonged to a drilling site. We set anything under that threshold to zero. This just helped us to eliminate some noise and clean up the predictions a bit. With that done, we used GDAL to convert the cleaned up prediction image into polygons and reattach the spatial information that we set aside at the beginning (i.e., the geotransform and the projection).  

  

# Band to use.
sourceBand = output.GetRasterBand(1)

# Set up shapefile.
shpDrv = ogr.GetDriverByName("ESRI Shapefile")                                
outFile = shpDrv.CreateDataSource("/gdrive/My Drive/detections.shp")      
layer = outFile.CreateLayer("detections", srs=spatial_ref)                    

# Add field.
idField = ogr.FieldDefn("id", ogr.OFTInteger)               
layer.CreateField(idField)

# And, convert. 
gdal.Polygonize(sourceBand, sourceBand, layer, 0, [], None)

And at this point we had our shapefile. From there, it was easy to upload that shapefile as an asset into our Earth Engine account and have a look at our predictions over satellite imagery. We did a bit of clean up and editing to make sure that all of the polygons look good — what’s sometimes called “human-in-the-loop” modeling — but, for the most part, we were all done.

Screenshot of the polygons in EE.

Lastly, we did a quick assessment to see how well our from-scratch workflow functioned. In the image above, red points are drilling sites that we got correct (true positives), green points are drilling sites that we missed (false negatives), and blue points are places where the model thought there was a drilling site when there wasn’t (false positives). Here are the numbers: 

Total number of validated ground truth points: 239
True Positives: 107
False Positives: 50
False Negatives: 82
Precision: 0.6815286624203821
Recall: 0.5661375661375662
F1-score: 0.6184971098265896

Precision, recall, and F1-score are all just metrics for understanding how a model performs. Your spam folder offers a good example. Imagine that your email’s spam model makes 30 predictions. Precision would measure the percentage of emails flagged as spam that it correctly classified as spam. Recall would measure the percentage of actual spam emails that it correctly classified as spam. Often, these two things are in tension – if the model is more aggressive (i.e., lowers the threshold for an email to be classified as spam) the recall will go up since they’d be capturing more actual spam emails. But the precision would go down, because they’re also capturing more emails overall, and it’s pretty likely that many of those won’t be spam. The F1-score builds off of precision and recall, and it’s probably easiest to think of it as a measure of overall accuracy. In the context of the drilling site work, our precision and recall numbers mean that 68% of the things we’re predicting as drilling sites actually are drilling sites, but we’re only picking up 56% of the drilling sites that are actually out there in the world. 

We hope that this series has helped to demystify the machine learning workflow for others. Figuring out how to build a machine learning data processing pipeline from scratch was an exciting challenge for us. We’re encouraged by our progress so far, but as the metrics above indicate, there’s still lots of room for improvement. We will keep at it because this technology stack — integrating remote sensing with cloud computing and machine learning — is at the heart of our Conservation Vision initiative to automate the analysis of imagery, to solve challenging conservation problems. 

So please stay tuned for updates in the future.  And don’t hesitate to send us a note if you have questions about what we’ve done or ideas about how we can improve things going forward.  We welcome the collaboration! 

 

Drilling Detection with Machine Learning Part 2: Segmentation Starter Kit

Geospatial Analyst Brendan Jarrell explains, step-by-step, how to develop a machine learning model to detect oil and gas well pads from satellite imagery.

[This is the second post in a 3-part blog series describing SkyTruth’s effort to automate the detection of oil and gas well pads around the world using machine learning. This tool will allow local communities, conservationists, researchers, policymakers and journalists to see for themselves the growth of drilling in the areas they care about. This is a central part of SkyTruth’s work: to share our expertise with others so that anyone can help protect the planet, their communities, and the places they care about. You can read the first post in the series here. All of the code that will be covered in this post can be found here. Our training dataset is also available here.]

SkyTruth Intern Sasha Bylsma explained in our first post in this series how we create training data for a machine learning workflow that will be used to detect oil and gas well pads around the world. In this post, I’m going to explain how we apply a machine learning model to satellite imagery, explaining all the tools we use and steps we take to make this happen, so that anyone can create similar models on their own.

Once we have created a robust set of training data, we want to feed a satellite image into the machine learning model and have the model scan the image in search of well pads. We then look to the model to tell us where the well pads are located and give us the predicted boundary of each of the well pads. This is known as segmentation, as shown in Figure 1. 

Figure 1: An example of our current work on well pad segmentation. The original image is seen on the left; what the ML model predicts as a well pad can be seen on the right. Notice that the algorithm is not only returning the drilling site’s location, but also its predicted boundaries.

We want the model to identify well pad locations because of the crucial context that location data provides. For example, location can tell us if there is a high density of drilling in the area, helping nearby communities track increasing threats to their health. It can also calculate the total area of disturbed land in the area of interest, helping researchers, advocates and others determine how severely wildlife habitat or other land characteristics are diminished.  

In the past, SkyTruth did this work manually, with an analyst or volunteer viewing individual images to search for well pads and laboriously drawing their boundaries. Projects like FrackFinder, for example, may have taken staff and volunteers weeks to complete. Now, with the help of a machine learning model, we can come in on a Monday morning, let the model do its thing, and have that same dataset compiled and placed on a map in an hour or two. The benefits of leveraging this capability are obvious: we can scan thousands of images quickly and consistently, increasing the likelihood of finding well pads and areas with high levels of drilling.

Formatting the data

So how do we do this? The first thing we need to do is get our data into a format that will be acceptable for the machine learning model. We decided that we would use the TensorFlow API as our framework for approaching this task. TensorFlow is an open-source (i.e. “free-to-use”) software package that was developed by Google to give users access to a powerful math library specifically designed for machine learning. We exported data from Google Earth Engine in the TFRecord format; TFRecords are convenient packages for exporting information from Earth Engine for later use in TensorFlow. In our code under the section labeled “Get Training, Validation Data ready for UNET,” we see that there are a few steps we must fulfill to extract the TFRecords from their zipped up packages and into a usable format (see Figure 2). 

# Bands included in our input Feature Collection and S2 imagery.

bands = ['R','G','B']
label = 'Label'
featureNames = bands + [label]
# Convert band names into tf.Features.

cols = [
         tf.io.FixedLenFeature(shape=[256,256],dtype=tf.float32) for band in featureNames
       ]

"""Pass these new tensors into a dictionary, used to describe pieces of the input dataset."""
featsDict = dict(zip(featureNames,cols))

Figure 2:  Preprocessing code

Second, we create Tensorflow representations of the information we are interested in drawing out of each of our examples from the Google Earth Engine workflow (see the first post in this series for more explanation on how we made these samples). Each of the samples has a Red, Green, and Blue channel associated with it, as well as a mask band, called “label” in our code. As such, we create Tensorflow representations for each of these different channels that data will be plugged into. Think of the representations we create for each channel name as sorting bins; when a TFRecord is unpacked, the corresponding channel values from the record will be placed into the bin that represents it. After loading in all of our TFRecords, we push them into a TFRecord Dataset. A TFRecord Dataset is a dataset which is populated by several TFRecords. We then apply a few functions to the TFRecord Dataset that make the records interpretable by the model later on.

Validation dataset

Once the dataset is loaded in, we split the dataset into two. This is an important part of machine learning, where we set aside a small amount of the whole dataset. When the model is being trained on the larger portion of the dataset, known as the training data, it will not see this smaller subset, which we call the validation set. As its name suggests, the model uses this smaller fraction of information to perform a sanity check of sorts. It’s asking itself, “Okay, I think that a well pad looks like this. Am I close to the mark, or am I way off?” All of this is put in place to help the model learn the minute details and intricacies of the data we’ve provided it. Typically, we will reserve 15-30% of our total dataset for the validation set. The code necessary for splitting the dataset is shown in Figure 3 below.

# Get the full size of the dataset.
full_size = len(list(data))
print(f'Full size of the dataset: {full_size}','\n')

# Define a split for the dataset.
train_pct = 0.8
batch_size = 16
split = int(full_size * train_pct)

# Split it up.
training = data.take(split)
evaluation = data.skip(split)

# Get the data ready for training.
training = training.shuffle(split).batch(batch_size).repeat()
evaluation = evaluation.batch(batch_size)

# Define the steps taken per epoch for both training and evaluation.
TRAIN_STEPS = math.ceil(split / batch_size)
EVAL_STEPS = math.ceil((full_size - split)  / batch_size)

print(f'Number of training steps: {TRAIN_STEPS}')
print(f'Number of evaluation steps: {EVAL_STEPS}')

Figure 3: Validation split code snippet

Implementation in U-Net

Now it’s time for the fun stuff! We’re finally ready to begin setting up the model that we will be using for our segmentation task. We will be leveraging a model called a U-Net for our learning. Our implementation of the U-Net in TensorFlow follows a very similar structure to the one seen in the example here. In a nutshell, here is what’s happening in our U-Net code:

1.) The machine learning model is expecting a 256 pixel by 256 pixel by 3 band input. This is the reason why we exported our image samples in this manner from Earth Engine. Also, by chopping up the images into patches, we reduce the amount of information that needs to be stored in temporary memory at any given point. This allows our code to run without crashing.

2.) The computer scans the input through a set of encoders. An encoder’s job is to learn every little detail of the thing we’re instructing it to learn. So in our case, we want it to learn all of the intricacies that define a well pad in satellite imagery. We want it to learn that well pads are typically squares or rectangles, have well defined edges, and may or may not be in close proximity to other well pads. As the number of encoders increases further down the “U” shape of the model, it is learning and retaining more of these features that make well pads unique.

3.) As the computer creates these pixel-by-pixel classifications sliding down the “U,” it sacrifices the spatial information that the input once held. That is to say, the image no longer appears as a bunch of well pads scattered across a landscape. It appears more so as a big stack of cards. All of the pixels in the original image are now classified with their newly minted predictions (i.e. “I am a well pad” or “I am not a well pad”), but they don’t have any clue where in the world they belong. The task of the upper slope of the “U” is to stitch the spatial information onto the classified predictions generated by our model. In this light, the upward slope of the “U” is made up of filters known as decoders. The cool thing about the U-Net is that as we go further up the “U”, it will grab the spatial pattern associated with the same location on the downward slope of the U-Net. In short, the model gives its best shot at taking these classified predictions and making them back into an image. To see a visual representation of the U-Net model, refer to Figure 4 below.

Figure 4: A graphic representing the U-Net architecture, courtesy of Ronneberger, et al.

At the end of the trip through the model, we are left with an output image from the model. This image is the model’s best guess at whether or not what we’ve fed it shows well pads or not. Of course, the model’s best guess will not be absolute for each and every pixel in the image. Given what it has learned about well pads, (how they’re shaped, what color palette usually describes a well pad, etc.), the model returns values on a spectrum from 0 to 1. Wherever the values land in between these two numbers can be called the model’s confidence in its prediction. So for example, forested areas in the image would ideally show a confidence value near zero; conversely, drilling sites picked up in the image would have confidence values close to one. Ambiguous features in the image, like parking lots or agricultural fields, might have a value somewhere in the middle of zero and one. Depending on how well the model did when compared to the mask associated with the three band input, it will be reprimanded for mistakes or errors it makes using what’s known as a loss function. To read more about loss functions and how they can be used, be sure to check out this helpful blog. Now that we have the model set up, we are ready to gear up for training!

Data augmentation

Before we start to train, we define a function which serves the purpose of tweaking the inputs slightly every time they are seen by the model. This is a process known as data augmentation. The reason why we make these small changes is because we don’t have a large dataset. If we give the model a small dataset without making these tweaks, each time the model sees the image, it will essentially memorize the images as opposed to learning the characteristics of a well pad. It’s a pretty neat trick, because we can make a small dataset seem way larger than it actually is simply by mirroring the image on the y-axis or by rotating the image 90 degrees, for example. Our augmentation workflow is shown in Figure 5.

# Augmentation function to pass to Callback class.
def augment(image, mask):
 rand = np.random.randint(100)
  if rand < 25:
   image = tf.image.flip_left_right(image)
   mask = tf.image.flip_left_right(mask)

 elif rand >= 25 and rand < 50:
   image = tf.image.rot90(image)
   mask = tf.image.rot90(mask)

 elif rand >= 50 and rand < 75:
   image = tf.image.flip_up_down(image)
   mask = tf.image.flip_up_down(mask)

 else:
   pass

 return (image, mask)

# Callback for data augmentation.
class aug(tf.keras.callbacks.Callback):
 def on_training_batch_begin(self, batch, logs = None):
   batch.map(augment, num_parallel_calls = 5)
   batch.shuffle(10)

Figure 5: Augmentation function and checkpoints cell

Fitting the model to the dataset

Now it’s time to put this model to the test! We do this in a TensorFlow call known as .fit(). As the name suggests, it is going to “fit” the model to our input dataset. Let’s go ahead and take a look at the code from Figure 6, shown below. 

history = UNet.fit(
     x = training,
     epochs = model_epochs,
     steps_per_epoch = TRAIN_STEPS,
     validation_data = evaluation,
     validation_steps = EVAL_STEPS,
     callbacks = [aug(),cp,csv])

Figure 6: Fitting the model to the input dataset

It’s important to conceptually understand what each of the values passed into this function call represents. We start with the variable “x”: this expects us to pass in our training dataset, which was created earlier. The next argument is called epochs. Epochs describe how many times the model will see the entire dataset during the fitting process. This is somewhat of an arbitrary number, as some models can learn the desired information more quickly, thus requiring less training. Conversely, training a model for too long can become redundant or potentially lead to overfitting. Overfitting is when a model learns to memorize the images it’s trained on, but it doesn’t learn to generalize. Think of overfitting like memorizing a review sheet the night before a test; you memorize what is covered in the review, but any minor changes in the way questions are asked on the actual test could trip you up. For this reason, it is generally up to the user to determine how many epochs are deemed necessary based on the application. 

The next argument, steps_per_epoch (also validation_steps) describes how many batches of data should be taken from our training and validation sets respectively through each epoch. Batches are small chunks of the dataset; it is useful to divide up the dataset into batches to make the training process more computationally efficient. One would typically want to go through the whole dataset every epoch, so it’s best to set the steps as such. Validation_data is where we would specify the data we set aside during training to validate our model’s predictions. Remember, that data will not be seen by the model during its training cycle. The last argument is called callbacks. This is where we pass in the augmentation function. This function is instructed by our callback to run at the beginning of each new batch, therefore constantly changing the data during training. We also optionally pass in other callbacks which might be useful for later reference to our training session. Such callbacks might export the loss and metrics to our Google Drive in a comma-separated values format or might save checkpoints throughout the model, keeping track of which training epoch produces the lowest loss. There are many other pre-packaged callbacks which can be used; a full list of these callbacks can be found here. Now that we have all of that covered, it’s time to start learning! By running this code, we begin the training process and will continue until the model has finished running through all of the epochs we specified.

Once that has finished, we save the model and plot its metrics and its loss, as shown in Figure 7. Based upon how these plots look, we can tell how well we did during our training.

Figure 7: An example chart, showing plotted metrics (top) and loss (bottom). Metrics are used to evaluate the performance of our model, while loss is directly used during training to optimize the model. As such, a good model will have a greatly reduced loss by the time we reach the end of training.

And voila! You have made it through the second installment in our series. The next entry will cover post-processing steps of our machine learning workflow. Questions we will answer include:

– How do we make predictions on an image we’ve never seen before?

– How do we take a large image and chop it into smaller, more manageable pieces? 

– How do we take some new predictions and make them into polygons?

Stay tuned for our next entry, brought to you by Dr. Ry Covington, SkyTruth’s Technical Program Director. In case you missed it, be sure to check out the first post in this series. Happy skytruthing!