Profile Earth Engine computation

The Earth Engine API provides tools for profiling the performance of your computations but they are not always the easiest to use to get the number you are looking for. The geetools library supercharge the original profiler to make any computation evaluation the easiest possible.

Set up environment

Install all the requireed libs if necessary. and perform the import satements upstream.

# uncomment if installation of libs is necessary
# !pip install earthengine-api geetools

import ee
import geetools
import pandas as pd

# uncomment if authetication to GEE is needed
# ee.Authenticate()
# ee.Intialize(project="<your_project>")

Example data

The following examples rely on a ee.FeatureCollection composed of three ecoregion features that define regions by which to reduce image data. The Image data are PRISM climate normals, where bands describe climate variables per month; e.g., July precipitation or January mean temperature.

ecoregions = (
    ee.FeatureCollection("projects/google/charts_feature_example")
    .select(["label", "value","warm"])
)

normClim = ee.ImageCollection('OREGONSTATE/PRISM/Norm91m').toBands()

default profiler

The default profiler from Earth Engine can be called as a context manager, it will print at the end of the cell the extensive description of your computation.

with ee.profilePrinting():
    normClim.geetools.byBands(
        regions = ecoregions,
        reducer = "mean",
        scale = 500,
        regionId = "label",
        bands = [f"{i:02d}_tmean" for i in range(1,13)],
    ).getInfo()

 EECU·s PeakMem Count  Description
  0.527     59k     6  Algorithm Image.reduceRegions
  0.293    570k    86  Loading assets: projects/google/charts_feature_example
  0.248    404k   951  (plumbing)
  0.160    212k   252  Loading assets: OREGONSTATE/PRISM/Norm91m/(...)
  0.052    1.1M    86  no description available
  0.009     328    72  Reprojecting pixels from GEOGCS["GCS_North_American_1983",DATUM["North_American_Datum_1983",SPHEROID[...] to GEOGCS["GCS_North_American_1983",DATUM["North_American_Datum_1983",SPHEROID[...]
  0.002    3.3k    13  Algorithm Collection.reduceColumns with reducer Reducer.toList
  0.002    9.6k    15  Algorithm ImageCollection.toBands
  0.002    3.9k    15  Algorithm Image.select
  0.001    5.7k    15  Algorithm Image.rename
  0.001    3.1k    14  Algorithm ReduceRegions.AggregationContainer
  0.000     424     3  Listing collection
  0.000    113k     3  Computing image mask from geometry
   -       121k    50  Loading assets: OREGONSTATE/PRISM/Norm91m
   -        90k    26  Algorithm Collection.reduceColumns
   -        45k    19  Algorithm List.map
   -        28k    14  Algorithm Dictionary.fromLists
   -        23k    40  Algorithm AggregateFeatureCollection.array
   -        10k    15  Algorithm Collection.loadTable
   -       9.9k    15  Algorithm ImageCollection.load
   -       8.7k     4  Algorithm Projection
   -       8.5k     1  Algorithm (user-defined function)
   -       7.2k     4  Algorithm ReduceRegions.ReduceRegionsEnumerator
   -       5.4k    15  Algorithm Collection.map
   -       3.3k    14  Algorithm Feature.select
   -       3.3k    10  Algorithm If
   -       3.2k    10  Algorithm Number.eq
   -       3.1k     4  Algorithm Reducer.forEach
   -       3.1k     4  Algorithm String.compareTo
   -       3.0k    37  Algorithm String
   -       3.0k    10  Algorithm ObjectType
   -       2.8k    20  Loading assets: OREGONSTATE/PRISM
   -       2.8k    20  Loading assets: projects/google
   -       1.8k     1  Algorithm Number.format
   -        624     5  Algorithm Reducer.mean
   -        392     7  Expression evaluation
   -        264    72  Algorithm Image.load computing pixels

This result is extremely useful but cannot be further explored in the notebook.

geetools profiler

The geetools profiler is a context manager object that fill a dictionary member (profile) with the content of the string profile. This dictionary can be transformed into a table easily.

# example with a simple function
with ee.geetools.Profiler() as p:
    ee.Number(3.14).add(0.00159).getInfo()
p.profile

{'EECU-s': [0.001, None],
 'PeakMem': [5040, 3100],
 'Count': [3, 3],
 'Description': ['(plumbing)', 'Algorithm']}

With a bigger method we can valorized the results as a pandas dataframe and extract key informations.

with ee.geetools.Profiler() as p:
    normClim.geetools.byBands(
        regions = ecoregions,
        reducer = "mean",
        scale = 500,
        regionId = "label",
        bands = [f"{i:02d}_tmean" for i in range(1,13)],
    ).getInfo()
df = pd.DataFrame(p.profile)
df.head()

	EECU-s	PeakMem	Count	Description
0	0.477	59000	6	Algorithm
1	0.259	569000	86	Loading
2	0.218	404000	951	(plumbing)
3	0.137	212000	252	Loading
4	0.040	1000000	86	no

# total EECU cost of the computation
float(df["EECU-s"].sum())

1.1429999999999998