Check data exists before initial load

Data Services
1

Hi,

I have a data service which has default values as JSON file in the dataobjects folder. The service has some custom validation to check if combination of three fields is always unique.

Whenever the service restarts it tries to load all the default values to the database, as the data already exists in Mongo database the validation fails and the service crashes.

Is there a way I can clear data before loading it again? If yes any pointers to which class->method in data object service this change needs to go please?

Thanks
Shravan

2

Do your DataObjects defined in the JSON have id fields?

3

no here is the sample JSON. The validation I have is to have combination of Name, Category and Theme should be unique.

{
“Category”: “RAG”,
“Description”: “RAG Red”,
“Name”: “Red”,
“Value”: “#FF0000”,
“Theme”: “Light”,
“createdByUsername”: “AnonymousSystemUser”,
“createdDate”: 1615231370462,
“displayLabel”: “Red”,
“edition”: 1,
“id”: “301abf68-b3f6-48d7-a107-f2353aa4bf0e”,
“lastModifiedByUsername”: “AnonymousSystemUser”,
“lastModifiedDate”: 1615231370462,
“qualifiedId”: “/301abf68-b3f6-48d7-a107-f2353aa4bf0e”,
“qualifiedType”: “/Colour”,
“rootId”: “301abf68-b3f6-48d7-a107-f2353aa4bf0e”,
“rootType”: “Colour”,
“type”: “Colour”,
“version”: 1,
“superTypes”: [],
“serviceName”: “ColourServiceDataObjectService”,
“dataObjectAuthorizationToken”: {
“canUserEdit”: true,
“canUserDelete”: true,
“nonViewableFields”: [],
“nonEditableFields”: []
},
“typeInfo”: {},
“leafVersion”: “3.1”
}

4

Okay, since they already have ids, you can check if a DataObject with that ID exists before upserting.

In a generated Data Service, you can do this:

var dos = this.dataService.getDataObjectService();
var existingDOs = dos.load(
    List.of("<REPLACE_WITH_ID_YOU_WANT_TO_CHECK>"),
    Context.makeSystemContext()
);
if (existingDOs.size() == 0) {
    dos.addRoots(dataObjects, Context.makeSystemContext());
}
5

Oh wait, I may have misunderstood. Is that the JSON payload of a DataObject that has already been persisted by LEAF, or is that a sample DataObject that you have in your local JSON file?

6

This is the object from the JSON file in the dataobjects folder which is inserted on service load

7

Okay, then my solution given above should work :+1:

8

In my generated service I have this method on startup which is writing data. How do I access ids from the file? is this the right place to put something in?

void importDefaultData() {
this.dataService.upsertResources(ResourceLocations.INSTANCE, Context.makeSystemContext());
}

9

So in order to do this, you will have to import the DataObjects a different way. The dataService.upsertResources is sort of doing two things at once: it’s importing them from the file into memory and then upserting them via the dataService. You must remove that line and import the data a different way.

If you’re using LEAF 3.2.3, you can replace that code with the following:

Collection<DataObject> dataObjects = new ResourceLoader()
    .loadDataObjects(
        new ResourceConfigSource(<YOUR_APPLICATION>.class, "/dataobjects")
    );

In older versions, you can use:

Collection<DataObject> dataObjects = this.resourceLoader.loadArbitraryJsonArrayElements(
    new TypeReference<Collection<DataObject>>() {},
    new ResourceConfigSource(<YOUR_APPLICATION>.class, "/dataobjects")
);

Kudos to @mark.a.alvaro for the above snippet ^.

Then process the DataObjects the way I described:

var dos = this.dataService.getDataObjectService();
var existingDOs = dos.load(
   dataObjects.stream()
        .map(dObj -> dObj.getId())
        .collect(Collectors.toList()),
    Context.makeSystemContext()
);
if (existingDOs.size() == 0) {
    dos.addRoots(dataObjects, Context.makeSystemContext());
}

Since you have to remove the dataServices.upsertResources, you will most likely need to manually upsert the remaining beans that it was taking care of for you, as well:

Collection<BeanMetadata> beans = new ResourceLoader()
           .loadBeanMetadata(
              new ResourceConfigSource(<YOUR_APPLICATION>.class, "/beanmetadata"));

Collection<LookupList> lookupLists = new ResourceLoader()
           .loadLookupLists(
              new ResourceConfigSource(<YOUR_APPLICATION>.class, "/lookuplists"));


Collection<MetadataTemplate> metadataTemplates = new ResourceLoader()
           .loadMetadataTemplates(
              new ResourceConfigSource(<YOUR_APPLICATION>.class, "/metadatatemplates"));

this.dataService.getMetadataService()
    .upsertBeanMetadata(beans, Context.makeSystemContext());
this.dataService.getLookupListService()
    .upsert(lookupLists, Context.makeSystemContext());
this.dataService.getMetadataService()
    .upsertTemplates(metadataTemplates, Context.makeSystemContext());
1 Like
10

Thanks Jacob, managed to get this working with your code :slight_smile:

2 Likes