Profile Earth Engine computation#
The Earth Engine API provides tools for profiling the performance of your computations but they are not always the easiest to use to get the number you are looking for. The geetools library supercharge the original profiler to make any computation evaluation the easiest possible.
Set up environment#
Install all the requireed libs if necessary. and perform the import satements upstream.
# uncomment if installation of libs is necessary
# !pip install earthengine-api geetools
import ee
import geetools
import pandas as pd
# uncomment if authetication to GEE is needed
# ee.Authenticate()
# ee.Intialize(project="<your_project>")
Example data#
The following examples rely on a ee.FeatureCollection composed of three ecoregion features that define regions by which to reduce image data. The Image data are PRISM climate normals, where bands describe climate variables per month; e.g., July precipitation or January mean temperature.
ecoregions = (
ee.FeatureCollection("projects/google/charts_feature_example")
.select(["label", "value","warm"])
)
normClim = ee.ImageCollection('OREGONSTATE/PRISM/Norm91m').toBands()
default profiler#
The default profiler from Earth Engine can be called as a context manager, it will print at the end of the cell the extensive description of your computation.
with ee.profilePrinting():
normClim.geetools.byBands(
regions = ecoregions,
reducer = "mean",
scale = 500,
regionId = "label",
bands = [f"{i:02d}_tmean" for i in range(1,13)],
).getInfo()
EECU·s PeakMem Count Description
0.341 59k 6 Algorithm Image.reduceRegions
0.224 570k 87 Loading assets: projects/google/charts_feature_example
0.168 354k 831 (plumbing)
0.027 722k 86 no description available
0.011 205k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/02@1662731486284455
0.011 209k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/06@1662731651724226
0.011 202k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/07@1662732032798195
0.011 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/12@1662731114457874
0.011 199k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/11@1662731554688435
0.011 199k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/09@1662730604988590
0.010 211k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/05@1662731334196830
0.010 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/01@1662731626359925
0.010 210k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/03@1662731338127317
0.010 212k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/04@1662730567169297
0.010 200k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/08@1662731245955723
0.009 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/10@1662731228874571
0.005 328 72 Reprojecting pixels from GEOGCS["GCS_North_American_1983",DATUM["North_American_Datum_1983",SPHEROID[...] to GEOGCS["GCS_North_American_1983",DATUM["North_American_Datum_1983",SPHEROID[...]
0.002 9.4k 15 Algorithm ImageCollection.toBands
0.001 5.8k 15 Algorithm Image.rename
0.001 3.9k 15 Algorithm Image.select
0.001 3.3k 13 Algorithm Collection.reduceColumns with reducer Reducer.toList
0.001 3.0k 14 Algorithm ReduceRegions.AggregationContainer
0.000 61k 51 Loading assets: OREGONSTATE/PRISM/Norm91m
0.000 424 3 Listing collection
0.000 113k 3 Computing image mask from geometry
- 90k 26 Algorithm Collection.reduceColumns
- 45k 19 Algorithm List.map
- 31k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/01
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/12
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/11
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/10
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/09
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/08
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/07
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/06
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/05
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/04
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/03
- 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/02
- 23k 40 Algorithm AggregateFeatureCollection.array
- 10k 15 Algorithm Collection.loadTable
- 9.9k 15 Algorithm ImageCollection.load
- 8.7k 4 Algorithm Projection
- 8.4k 1 Algorithm (user-defined function)
- 7.0k 4 Algorithm ReduceRegions.ReduceRegionsEnumerator
- 5.1k 15 Algorithm Collection.map
- 3.6k 14 Algorithm Dictionary.fromLists
- 3.3k 14 Algorithm Feature.select
- 3.3k 10 Algorithm If
- 3.2k 4 Algorithm String.compareTo
- 3.2k 4 Algorithm Reducer.forEach
- 3.1k 10 Algorithm Number.eq
- 3.0k 37 Algorithm String
- 3.0k 10 Algorithm ObjectType
- 2.8k 20 Loading assets: OREGONSTATE/PRISM
- 2.8k 20 Loading assets: projects/google
- 1.8k 1 Algorithm Number.format
- 576 5 Algorithm Reducer.mean
- 408 7 Expression evaluation
- 264 72 Algorithm Image.load computing pixels
This result is extremely useful but cannot be further explored in the notebook.
geetools profiler#
The geetools profiler is a context manager object that fill a dictionary member (profile) with the content of the string profile. This dictionary can be transformed into a table easily.
# example with a simple function
with ee.geetools.Profiler() as p:
ee.Number(3.14).add(0.00159).getInfo()
p.profile
{'EECU-s': [0.0, None],
'PeakMem': [4640, 3000],
'Count': [3, 3],
'Description': ['(plumbing)', 'Algorithm']}
With a bigger method we can valorized the results as a pandas dataframe and extract key informations.
with ee.geetools.Profiler() as p:
normClim.geetools.byBands(
regions = ecoregions,
reducer = "mean",
scale = 500,
regionId = "label",
bands = [f"{i:02d}_tmean" for i in range(1,13)],
).getInfo()
df = pd.DataFrame(p.profile)
df.head()
| EECU-s | PeakMem | Count | Description | |
|---|---|---|---|---|
| 0 | 0.297 | 59000 | 6 | Algorithm |
| 1 | 0.203 | 569000 | 87 | Loading |
| 2 | 0.158 | 372000 | 831 | (plumbing) |
| 3 | 0.028 | 716000 | 86 | no |
| 4 | 0.011 | 198000 | 11 | Loading |
# total EECU cost of the computation
float(df["EECU-s"].sum())
0.8099999999999999