Utility to generate a random DataObject

Metadata
1

Greetings,

I want to be able to be able to randomly generate DataObjects. For example say I have the following data object definition:

<type type="Location" edition="1" displayName="Location" preferredDisplayPropName="nameAddress">

    <StringField            propName="name"
                            shortName="Name"
                            required="true"
                            columnViewable="false" />

    <StringField            propName="address"
                            shortName="Address"
                            columnViewable="false"
                            stringUiComponentType="TEXTAREA" />
                            
    <StringField            propName="address2"
                            shortName="Address 2"
                            columnViewable="false"
                            stringUiComponentType="TEXTAREA" />

    <StringField            propName="city"
                            shortName="City"
                            columnViewable="false"
                            stringUiComponentType="TEXTAREA" />


    <DynamicLookupListField  propName="state"
                             shortName="State"
                             parentPropName="region" >
        <subLists>
            <subList subListName="Region 01" parentListValue="Region 01"/>
            <subList subListName="Region 02" parentListValue="Region 02"/>
            <subList subListName="Region 03" parentListValue="Region 03"/>
            <subList subListName="Region 04" parentListValue="Region 04"/>
            <subList subListName="Region 05" parentListValue="Region 05"/>
            <subList subListName="Region 06" parentListValue="Region 06"/>
            <subList subListName="Region 07" parentListValue="Region 07"/>
            <subList subListName="Region 08" parentListValue="Region 08"/>
            <subList subListName="Region 09" parentListValue="Region 09"/>
            <subList subListName="Region 10" parentListValue="Region 10"/>
        </subLists>
    </DynamicLookupListField>
                          
    <StringField            propName="zip"
                            shortName="Zip Code"
                            columnViewable="false"
                            stringUiComponentType="TEXTAREA" />

    <StringField            propName="singleLineAddress"
                            shortName="Single Line"
                            required="true"
                            columnViewable="false"
                            stringUiComponentType="TEXTAREA" />

    <LookupListField
            propName="region"
            helpText="Region"
            listName="LocationRegions"
            required="false" />

    <CompositeField         propName="nameAddress"
                            shortName="Name Address"
                            required="true"
                            columnViewable="false"
                            userEditable="false"
                            delimiter=" - " >
        <propNamesToInclude>
            <propName>name</propName>
            <propName>singleLineAddress</propName>
        </propNamesToInclude>
    </CompositeField>

    <BooleanField           propName="verified"
                            required="true"
                            columnViewable="false"
                            defaultValue="false"/>

</type>

I want to be able to generate 10000 of these when the service starts up or potentially in a K8s job. This is primarily for load/performance testing.

2

You can programmatically generate a DataObject with

DataObject randomDob = DataObject.newBuilder()
                         .withType("Location")
                         .with("name", "MyLocation")
                         .with("address", "117 Random Address")
                         .with("address2", "")
                         .with("city", "MyCity")
                         .build()

Given that you want to do 10,000 of these, you can run a loop for 10,000 iterations and for every iteration put identical or random values in the name, address, address2, and city props.

for (int i = 0; i < 10,000; i++) {
    DataObject randomDob = DataObject.newBuilder()
                            .withType("Location")
                            .with("name", "MyLocation")
                            .with("address", "117 Random Address")
                            .with("address2", "")
                            .with("city", "MyCity")
                            .build()
}
3

If you have it on the Leidos Application store, you can also use Postman to create test Data Objects and take advantage of Postman’s Dynamic Variables feature to generate a bunch of data. Note though that this route requires you to hit the REST endpoints for the data service, but it’s an option and can be useful for load testing as well.

4

@howard.m.standley and I got into a slack call with @anthony.s.mercer to get him setup with the leaf-loader project to generate DataObjects.

For posterity, I will go over the steps that we helped Anthony with.

First, clone the leaf-loader project from BitBucket.

Next, you’ll need poetry to install the project’s dependencies, it can be installed with the following:
curl -sSL https://install.python-poetry.org | python3 -.

You can verify the installation was successful by running poetry --version. If that does not work, run the following command to ensure that the poetry binary is installed on the $PATH:
export PATH=$PATH:~/Library/Application\ Support/pypoetry/venv/bin/

Now that poetry has been installed, the project’s dependencies need to be installed. To do so, navigate to where the repo was cloned to and run poetry install. After poetry has installed the dependencies, it also will setup a virtual environment for us to use. To activate the virtual environment run:
source .venv/bin/activate.

After activating the environment, verify the leaf-loader cli was properly setup by running: leaf-loader in your terminal, you should see a cli help message like this:

Usage: leaf-loader [OPTIONS] COMMAND [ARGS]...

Options:
  -c, --count INTEGER       number of data objects to generate
  -s, --batch-size INTEGER  batch size of data objects to generate
  -m, --module TEXT         Name of a module that contains Beans
  -b, --bean-name TEXT      The name of a Bean to load and generate from a
                            given module
  --help                    Show this message and exit.

Commands:
  rest
  write

Mike and I created an example file that utilizes the leaf-loader’s modules to create beans that will generate DataObjects with the same fields as the bean posted in the question. Below is the example file:

from leaf_loader.metadata import Bean, Field, string, faker, boolean, Alternates

REGION = ["Region1", "Region2", "Region3"]

STATES = {
    "Region1": ["Md", "Va", "WV"],
    "Region2": ["Ca", "Or", "Oh"],
    "Region3": ["Nm", "Tx", "Ka"]
}

ADDRESS = Bean("Address", fields={
    Field("address", generator=faker.street_address, ratio=Field.REQUIRED, mutator=lambda prev: faker.street_address()),
    Field("city", generator=faker.city, ratio=Field.REQUIRED),
    Field("zip", generator=faker.postcode, ratio=Field.REQUIRED),
})

LAST_NAME = faker.last_name
FIRST_NAME = Alternates(generators=[faker.first_name_male, faker.first_name_female])


def name():
    return f'{FIRST_NAME()} {LAST_NAME()}'


def address():
    addr = ADDRESS()
    region = faker.random_element(REGION)
    st = faker.random_element(STATES[region])

    stateful = {
        "region": region,
        "state": st,
        'singleLineAddress': f'{addr["address"]} {addr["city"]}, {st} {addr["zip"]}'
    }
    addr.update(stateful)
    return addr


Location = Bean(bean_type="Location", fields={
    string.Constant("type", "Location"),
    Field(name="name", generator=name, ratio=Field.REQUIRED),
    address,
    boolean.Random("verified", ratio=Field.REQUIRED)
})

It would be a bit difficult to give a good explanation of how the leaf-loader works over a discourse thread, but the important parts are Beans and Fields. Bean is a class that accepts the type that the bean represents, and fields which is a list of Field objects. These Field objects are somewhat equivalent to a Python dictionary, where the first argument (“name”) is the key, and the second argument (generator) is a Callable that returns some value (lambdas and functions are both examples of Callables in Python). Field objects take other arguments but the two I just mentioned are required.

To use the cli to generate the Bean from the example file above, paste the example code in leaf-loader/src/leaf_loader/loader/example.py (example.py does not exist and will need to be created). The following command can be used to generate 10 (-c) DataObjects in total, 2 at a time (-s), with the Bean we created in the leaf_loader.loader.example module (-m), with the specific bean we want to generate called Location (-b) and write those generated objects to stdout:
leaf-loader -c 10 -s 2 -m leaf_loader.loader.example -b Location write stdout

The leaf-loader’s README contains a few other example commands but will need to be modified to fit your use case.

If you’d like a more in-depth explanation of what the leaf-loader has to offer, or are having trouble using the leaf-loader’s modules or the leaf-loader cli, you can reach out to Mike or myself.