geonode.harvesting.harvesters.base
Attributes
Exceptions
Common base class for all non-exit exceptions. |
Classes
Base class for harvesters. |
Functions
|
Download a resource file and store it using GeoNode's storage_manager. |
|
|
|
Module Contents
- exception geonode.harvesting.harvesters.base.HarvestingException[source]
Bases:
ExceptionCommon base class for all non-exit exceptions.
- class geonode.harvesting.harvesters.base.HarvestedResourceInfo[source]
- resource_descriptor: geonode.harvesting.resourcedescriptor.RecordDescription[source]
- class geonode.harvesting.harvesters.base.BaseHarvesterWorker(remote_url: str, harvester_id: int)[source]
Bases:
abc.ABCBase class for harvesters.
This provides two relevant things:
An interface that all custom GeoNode harvesting classes must implement;
Default implementation for common functionality.
- property allows_copying_resources: bool[source]
- Abstractmethod:
Whether copying remote resources is implemented by this worker
- classmethod from_django_record(harvester: Harvester)[source]
- Abstractmethod:
Return a new instance of the worker from the django harvester
- abstract get_num_available_resources() int[source]
Return the number of available resources on the remote service.
If there is a problem retrieving the number of available resource, this method shall raise HarvestingException.
- abstract list_resources(offset: int | None = 0) List[BriefRemoteResource][source]
Return a list of resources from the remote service.
If there is a problem listing resource, this method shall raise HarvestingException.
- abstract check_availability(timeout_seconds: int | None = 5) bool[source]
Check whether the remote service is online
- abstract get_geonode_resource_type(remote_resource_type: str) geonode.base.models.ResourceBase[source]
Return the GeoNode type that should be created from the remote resource type
- abstract get_resource(harvestable_resource: HarvestableResource) HarvestedResourceInfo | None[source]
Harvest a single resource from the remote service.
The return value is an instance of HarvestedResourceInfo. It stores an instance of RecordDescription and additionally whatever type is required by child classes to be able to customize resource creation/update on the local GeoNode. Note that the default implementation of update_geonode_resource() only needs the RecordDescription. The possibility to return additional information exists solely for extensibility purposes and can be left as None in the simple cases.
- classmethod get_extra_config_schema() Dict | None[source]
Return a jsonschema schema to be used to validate models.Harvester objects
- finalize_resource_update(geonode_resource: geonode.base.models.ResourceBase, harvested_info: HarvestedResourceInfo, harvestable_resource: HarvestableResource) geonode.base.models.ResourceBase[source]
Perform additional actions just after having created/updated a local GeoNode resource.
This method can be used to further manipulate the relevant GeoNode Resource that is being created/updated in the context of a harvesting operation. It is typically called from within base.BaseHarvesterWorker.update_geonode_resource as the last step, after having already acted upon the GeoNode resource. The default implementation does nothing.
- finalize_harvestable_resource_deletion(harvestable_resource: HarvestableResource) bool[source]
Perform additional actions just before deleting a harvestable resource.
This method is typically called from within models.HarvestableResource.delete(), just before deleting the actual harvestable resource. It can be useful for child classes that customize resource creation in order to also customize the deletion of harvestable resources. The default implementation does nothing.
- should_copy_resource(harvestable_resource: HarvestableResource) bool[source]
Return True if the worker is able to copy the remote resource.
The base implementation just returns False. Subclasses must re-implement this method if they support copying remote resources onto the local GeoNode.
- copy_resource(harvestable_resource: HarvestableResource, harvested_resource_info: HarvestedResourceInfo) pathlib.Path | None[source]
Copy a remote resource’s data to the local GeoNode.
The base implementation provides a generic copy using GeoNode’s storage_manager. Subclasses may need to re-implement this method if they require specialized behavior.
- get_geonode_resource_defaults(harvested_info: HarvestedResourceInfo, harvestable_resource: HarvestableResource) Dict[source]
Extract default values to be used by resource manager when updating a resource
- update_geonode_resource(harvested_info: HarvestedResourceInfo, harvestable_resource: HarvestableResource)[source]
Create or update a local GeoNode resource with the input harvested information.
If the underlying harvestable resource already exists as a local GeoNode resource, then it is updated. Otherwise it is created locally.
If something goes wrong with the update of the geonode resource this method should raise a RuntimeError. This will be caught by the harvesting task and handled appropriately.
- _update_existing_geonode_resource(geonode_resource: geonode.base.models.ResourceBase, defaults: Dict)[source]
- geonode.harvesting.harvesters.base.download_resource_file(url: str, target_name: str) pathlib.Path[source]
Download a resource file and store it using GeoNode’s storage_manager.
Downloads use the django UploadedFile helper classes. Depending on the size of the remote resource, we may download it into an in-memory buffer or store it on a temporary location on disk. After having downloaded the file, we use storage_manager to save it in the appropriate location.
- geonode.harvesting.harvesters.base._get_file_name(resource_info: HarvestedResourceInfo) str | None[source]
- geonode.harvesting.harvesters.base._consolidate_resource_keywords(resource_descriptor: geonode.harvesting.resourcedescriptor.RecordDescription, geonode_resource, harvester_id: int) List[str][source]