Profile Earth Engine computation#
The Earth Engine API provides tools for profiling the performance of your computations but they are not always the easiest to use to get the number you are looking for. The geetools library supercharge the original profiler to make any computation evaluation the easiest possible.
Set up environment#
Install all the requireed libs if necessary. and perform the import satements upstream.
# uncomment if installation of libs is necessary
# !pip install earthengine-api geetools
import ee
import geetools
import pandas as pd
# uncomment if authetication to GEE is needed
# ee.Authenticate()
# ee.Intialize(project="<your_project>")
Example data#
The following examples rely on a ee.FeatureCollection composed of three ecoregion features that define regions by which to reduce image data. The Image data are PRISM climate normals, where bands describe climate variables per month; e.g., July precipitation or January mean temperature.
ecoregions = (
ee.FeatureCollection("projects/google/charts_feature_example")
.select(["label", "value","warm"])
)
normClim = ee.ImageCollection('OREGONSTATE/PRISM/Norm91m').toBands()
default profiler#
The default profiler from Earth Engine can be called as a context manager, it will print at the end of the cell the extensive description of your computation.
with ee.profilePrinting():
normClim.geetools.byBands(
regions = ecoregions,
reducer = "mean",
scale = 500,
regionId = "label",
bands = [f"{i:02d}_tmean" for i in range(1,13)],
).getInfo()
---------------------------------------------------------------------------
HttpError Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/data.py:408, in _execute_cloud_call(call, num_retries)
407 try:
--> 408 return call.execute(num_retries=num_retries)
409 except googleapiclient.errors.HttpError as e:
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/googleapiclient/_helpers.py:130, in positional.<locals>.positional_decorator.<locals>.positional_wrapper(*args, **kwargs)
129 logger.warning(message)
--> 130 return wrapped(*args, **kwargs)
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/googleapiclient/http.py:938, in HttpRequest.execute(self, http, num_retries)
937 if resp.status >= 300:
--> 938 raise HttpError(resp, content, uri=self.uri)
939 return self.postproc(resp, content)
HttpError: <HttpError 400 when requesting https://earthengine.googleapis.com/v1/projects/ee-geetools/value:compute?prettyPrint=false&alt=json returned "Profile.getProfiles: Profile not found.". Details: "Profile.getProfiles: Profile not found.">
During handling of the above exception, another exception occurred:
EEException Traceback (most recent call last)
Cell In[6], line 1
----> 1 with ee.profilePrinting():
2 normClim.geetools.byBands(
3 regions = ecoregions,
4 reducer = "mean",
(...)
7 bands = [f"{i:02d}_tmean" for i in range(1,13)],
8 ).getInfo()
File ~/.asdf/installs/python/3.10.15/lib/python3.10/contextlib.py:142, in _GeneratorContextManager.__exit__(self, typ, value, traceback)
140 if typ is None:
141 try:
--> 142 next(self.gen)
143 except StopIteration:
144 return False
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/_helpers.py:146, in profilePrinting(destination)
144 except ee_exception.EEException as exception:
145 if i == attempts - 1:
--> 146 raise exception
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/_helpers.py:141, in profilePrinting(destination)
139 for i in range(_PROFILE_RETRIES):
140 try:
--> 141 profile_text = get_profiles(ids=profile_ids).getInfo()
142 destination.write(profile_text)
143 break
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/computedobject.py:107, in ComputedObject.getInfo(self)
101 def getInfo(self) -> Optional[Any]:
102 """Fetch and return information about this object.
103
104 Returns:
105 The object can evaluate to anything.
106 """
--> 107 return data.computeValue(self)
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/data.py:1128, in computeValue(obj)
1125 body = {'expression': serializer.encode(obj, for_cloud_api=True)}
1126 _maybe_populate_workload_tag(body)
-> 1128 return _execute_cloud_call(
1129 _get_cloud_projects()
1130 .value()
1131 .compute(body=body, project=_get_projects_path(), prettyPrint=False)
1132 )['result']
File ~/checkouts/readthedocs.org/user_builds/geetools/envs/v1.12.0/lib/python3.10/site-packages/ee/data.py:410, in _execute_cloud_call(call, num_retries)
408 return call.execute(num_retries=num_retries)
409 except googleapiclient.errors.HttpError as e:
--> 410 raise _translate_cloud_exception(e)
EEException: Profile.getProfiles: Profile not found.
This result is extremely useful but cannot be further explored in the notebook.
geetools profiler#
The geetools profiler is a context manager object that fill a dictionary member (profile) with the content of the string profile. This dictionary can be transformed into a table easily.
# example with a simple function
with ee.geetools.Profiler() as p:
ee.Number(3.14).add(0.00159).getInfo()
p.profile
With a bigger method we can valorized the results as a pandas dataframe and extract key informations.
with ee.geetools.Profiler() as p:
normClim.geetools.byBands(
regions = ecoregions,
reducer = "mean",
scale = 500,
regionId = "label",
bands = [f"{i:02d}_tmean" for i in range(1,13)],
).getInfo()
df = pd.DataFrame(p.profile)
df.head()
# total EECU cost of the computation
float(df["EECU-s"].sum())