geonode.harvesting.tasks

Attributes

logger

Functions

harvesting_scheduler(self)	Check whether any of the configured harvesters needs to be run or not.
harvesting_dispatcher(self, harvesting_session_id)	Perform harvesting asynchronously.
harvest_resources(self, harvestable_resource_ids, ...)	Harvest a list of remote resources that all belong to the same harvester.
_harvest_resource(self, harvestable_resource_id, ...)	Harvest a single resource from the input harvestable resource id
_finish_harvesting(self, harvesting_session_id)
_handle_harvesting_error(self, task_id, *args, **kwargs)	Ensure harvester status is set back to ready in case of an error.
check_harvester_available(self, harvester_id)
update_harvestable_resources(self, refresh_session_id)
_update_harvestable_resources_batch(self, ...)
_finish_harvestable_resources_update(self, ...)
_handle_harvestable_resources_update_error(self, ...)
_delete_stale_harvestable_resources(harvester)	Delete harvestable resources that haven't been found on the current refresh.
finish_asynchronous_session(→ None)	Finish the asynchronous session and also reset the harvester status.
update_asynchronous_session(→ None)

Module Contents

geonode.harvesting.tasks.logger

geonode.harvesting.tasks.harvesting_scheduler(self)

Check whether any of the configured harvesters needs to be run or not.

This function is called periodically by celery beat. It is configured to run every HARVESTER_SCHEDULER_FREQUENCY_MINUTES minutes. This can be configured in the GeoNode settings. The default value is 0.5, which means that this function is called every thirty seconds.

geonode.harvesting.tasks.harvesting_dispatcher(self, harvesting_session_id: int)

Perform harvesting asynchronously.

This function kicks-off a harvesting session for the input harvester id. It defines an async workflow that results in the harvestable resources being checked and harvested onto the local GeoNode.

The implementation briefly consists of:

• start a harvesting session

• determine which of the known harvestable resources have to be harvested

• schedule each harvestable resource to be harvested asynchronously

• when all resources have been harvested, finish the harvesting session

Internally, the code uses celery’s chord workflow primitive. This is used to allow the harvesting of individual resources to be done in parallel (as much as the celery worker config allows for) and still have a final synchronization step, once all resources are harvested, that finalizes the harvesting session.

geonode.harvesting.tasks.harvest_resources(self, harvestable_resource_ids: List[int], harvesting_session_id: int)

Harvest a list of remote resources that all belong to the same harvester.

geonode.harvesting.tasks._harvest_resource(self, harvestable_resource_id: int, harvesting_session_id: int)

Harvest a single resource from the input harvestable resource id

geonode.harvesting.tasks._finish_harvesting(self, harvesting_session_id: int)

geonode.harvesting.tasks._handle_harvesting_error(self, task_id, *args, **kwargs)

Ensure harvester status is set back to ready in case of an error.

geonode.harvesting.tasks.check_harvester_available(self, harvester_id: int)

geonode.harvesting.tasks.update_harvestable_resources(self, refresh_session_id: int)

geonode.harvesting.tasks._update_harvestable_resources_batch(self, refresh_session_id: int, page: int, page_size: int)

geonode.harvesting.tasks._finish_harvestable_resources_update(self, refresh_session_id: int)

geonode.harvesting.tasks._handle_harvestable_resources_update_error(self, task_id, *args, **kwargs)

geonode.harvesting.tasks._delete_stale_harvestable_resources(harvester: geonode.harvesting.models.Harvester)

Delete harvestable resources that haven’t been found on the current refresh.

If a harvestable resource was not refreshed since the last check, it this means it has not been found this time around - maybe it is not available anymore, or maybe the user just changed the harvester parameters. Either way, the harvestable resource is no longer relevant, and therefore can be deleted.

When a harvestable resource is deleted, it may leave a corresponding GeoNode resource orphan. The corresponding GeoNode resource is the one that had been created locally as a result of harvesting information of the HarvestableResource on the remote server by the harvester worker. Depending on the value of the corresponding models.Harvester.delete_orphan_resources_automatically, the GeoNode resource may also be deleted as a result of running this method.

geonode.harvesting.tasks.finish_asynchronous_session(session_id: int, final_status: str, final_details: str | None = None, additional_processed_records: int | None = None) → None

Finish the asynchronous session and also reset the harvester status.

geonode.harvesting.tasks.update_asynchronous_session(session_id: int, total_records_to_process: int | None = None, additional_processed_records: int | None = None, additional_details: str | None = None) → None