Chapter 10. Understanding the Data Object Model (DOM)

[Note]

DOC features can be extended further depending on the project requirements. For more details, refer to Chapter Extending the Application Features.

The Data Object Model (DOM) is a set of classes that is used to handle scenario data in memory when writing complex processing. DOC provides internal mechanisms to map this data model to the Data Service relational database and most extension points related to data processing will receive the data they need in the form of a ready-to-use generated class called Collector. The DOM is available for the Java and the Python programming language.

A collector provides the following functions:

  • Creation and deletion of an instance of a given entity,

  • Access to all instances of a given entity,

  • Access to a specific instance using its business key, and

  • Read/Save its content into/from a snapshot text file.

DOM classes are generated from the JDL Business Data Model description and can be found in the gene-model-dom and gene-model-dom-python libraries of the generated application projects.

DOC assumes that all scenario data processing code uses a collector as input and provides a collector as output.

Typical examples include:

  • Model Checking

    • A collector containing the scenario data to be checked is received as input of the checker.

    • The checker code uses the collector API to browse through the different instances and to detect potential issues.

    • When problems are detected, the checker creates GeneIssue instances in the provided collector.

    • The collector in then saved back into the scenario data.

  • Optimization Engine Computation

    • A collector containing the scenario data to be processed is received as input of the engine.

    • The engine code uses the collector API to set up the optimization engine constraints and objectives.

    • The solution is extracted from the engine and appropriate instances are created in the provided collector to store the solution.

    • The collector in then saved back into the scenario data.

This principle allows to hide the scenario data mapping complexity and enables developing the processing features without concerns on the future context of execution:

  • Unit tests rely on collectors manually created from snapshot files.

  • Deployed code rely on collectors automatically created by DOC in the backend/worker task contexts.

1. Generating DOM Classes

The DOM is generated during the build phase of the gene-model-dom and gene-model-dom-python projects. Whether it is or not a Composite Data Model application, the generation is performed using the JDL files located in gene-model/spec. These file contains the entities and relationships that define the application data model. For more details, refer to Chapter Defining the Data Model and Section Capacity Planning JDL Samples.

The generated code can be found in the gene-model/gene-model-dom/build/generated/src and gene-model/gene-model-dom-python/python/src folders. As the naming implies, those files should NOT be modified since they will be overwritten any time a regeneration is performed.

The generated code includes:

  • A collector class (whose name is computed from the entities.jdl application java.collectorClass property).

  • A factory class used to create collectors.

  • A class for each of the entities defined in the entities.jdl file.

1.1. Understanding Generated Java DOM Classes

During the build phase, two files are generated for each entity: one interface file and one implementation file.

Here is an example of a set of generated interface files:

 1155 Nov  5 18:16 Activity.java
 3244 Nov  5 18:16 CapacityPlanning.java
  132 Nov  5 18:16 CapacityPlanningFactory.java
 1019 Nov  5 18:16 GeneIssue.java
  897 Nov  5 18:16 GeneParameter.java
  662 Nov  5 18:16 Precedence.java
  892 Nov  5 18:16 Requirement.java
  985 Nov  5 18:16 Resource.java
  779 Nov  5 18:16 ResourceCapacity.java
 1001 Nov  5 18:16 ResourceUsagePerDay.java
  878 Nov  5 18:16 Schedule.java
  868 Nov  5 18:16 SolutionSummary.java

1.2. Understanding Generated Python DOM Classes

During the build phase, a Python module that contains a class for each entity is generated.

Here is an example of a set of generated entities:

class Activity(DbDomObject, IndexKey):
  ...
class CapacityPlanning:
  ...
class CapacityPlanningFactory:
  ...
class GeneIssue:
  ...
class GeneParameter:
  ...
class Precedence:
  ...
class Requirement:
  ...
class Resource:
  ...
class ResourceCapacity:
  ...
class ResourceUsagePerDa:
  ...
class Schedule:
  ...
class SolutionSummary:
  ...

2. Creating a DOM Collector

Collectors are usually created and populated with data by DOC and thus do not need to be created explicitly except when working on unit tests.

In Java a collector is created using the generated factory class.

CapacityPlanningFactory factory = new CapacityPlanningFactoryImpl();
CapacityPlanning coll = factory.createCollector();

The collector class is an implementation of DbDomCollector.

In Python a collector is created directly using the constructor.

coll = CapacityPlanning()

3. Using the DOM Collector

3.1. Creating DOM Collector Instances

Instances should not be created by calling the constructor. The collector class includes a set of createXXX methods that should be used instead. Each creation method can be used to create an instance of the corresponding XXX entity.

3.1.1. Creating Instances in Java

The following code shows an example of createXXX methods generated to cover the set of entities defined in the tutorial:

public interface CapacityPlanning extends DbDomCollector {
  Activity createActivity();
  GeneIssue createGeneIssue();
  GeneParameter createGeneParameter();
  Precedence createPrecedence();
  Requirement createRequirement();
  Resource createResource();
  ResourceCapacity createResourceCapacity();
  ResourceUsagePerDay createResourceUsagePerDay();
  Schedule createSchedule();
  SolutionSummary createSolutionSummary();
  ...
}

To create a resource, one would declare the following:

Resource res1 = coll.createResource();

Each instance is linked with its collector which can be accessed using the getCollector() method.

res1.getCollector() == coll; // TRUE

3.1.2. Creating Instances in Python

The following code shows an example of createXXX methods generated to cover the set of entities defined in the tutorial:

class CapacityPlanning(DbDomCollector):
  def create_activity(self) -> Acticity:
    pass
  def create_gene_issue(self) -> GeneIssue:
    pass
  def create_gene_parameter(self) -> GeneParameter:
    pass
  def create_precedence(self) -> Precedence:
    pass
  def create_requirement(self) -> Requirement:
    pass
  def create_resource(self) -> Resource:
    pass
  def create_resource_capacity(self) -> ResourceCapacity:
    pass
  def create_resource_usage_per_day(self) -> ResourceUsagePerDay:
    pass
  def create_schedule(self) -> Schedule:
    pass
  def create_solution_summary(self) -> SolutionSummary:
    pass

To create a resource, one would declare the following:

Resource res1 = coll.createResource();

Each instance is linked with its collector which can be accessed using the getCollector() method.

res1.getCollector() == coll; // TRUE

3.2. Modifying DOM Collector Instances

The generated classes provide an API to get and set the different attributes and relationships of each instance.

We will use the following JDL entity declarations to illustrate the different ways to access and modify instances in Java and in Python:

entity Resource {
  // DOM [primary.keys] : [id]
  id String required,
  name String
}

entity ResourceCapacity {
  quantity Integer
}

relationship OneToOne {
  // DOM [affects.primary.key] : [true]
  ResourceCapacity{resource} to Resource{capacity}
}

3.2.1. Modifying Instances in Java

In Java, the JDL declaration translates in the following Resource and ResourceCapacity interfaces:

public interface Resource extends DbDomObject {
  ResourceCapacity getCapacity();
  CapacityPlanning getCollector();
  String getId();
  String getName();
  void setCapacity(ResourceCapacity obj);
  void setId(String value);
  void setName(String value);
}

public interface ResourceCapacity extends DbDomObject {
  CapacityPlanning getCollector();
  int getQuantity();
  Resource getResource();
  boolean isSetQuantity();
  void setQuantity(int value);
  void setResource(Resource obj);
  void unsetQuantity();
}

For example, to set the attributes of an instance, one would call

  Resource res1 = coll.createResource();
  res1.setId( "MyFirstResourceId" );
  res1.setName( "MyFirstResourceName" );

Relationships are managed in a similar way:

  ResourceCapacity capa1 = coll.createResourceCapacity();
  capa1.setQuantity(1);
  capa1.setResource(res1); // this automatically calls Resource.setResourceCapacity()
  res1.getResourceCapacity() == capa1; // true

  ResourceCapacity capa2 = coll.createResourceCapacity();
  capa2.setQuantity(2);
  res1.setResourceCapacity(capa2); // this automatically update the inverted relationships
  res1.getResourceCapacity() == capa2; // true
  capa1.getResource() == null; // true
  capa2.getResource() == res1; // true

3.2.2. Modifying Instances in Python

In Python, the JDL declaration translates in the following Resource and ResourceCapacity interfaces:

class Resource(DbDomObject):
    def get_capacity(self) -> ResourceCapacity:
        pass
    def get_id(self) -> str:
        pass
    def get_name(self) -> str:
        pass
    def set_capacity(self, capacity: ResourceCapacity | None):
        pass
    def set_id(self, id_: str):
        pass
    def set_name(self, name: str | None):
        pass

class ResourceCapacity(DbDomObject, IndexKey):
    def get_quantity(self) -> int:
        pass
    def get_resource(self) -> Resource:
        pass
    def set_quantity(self, quantity: int | None):
        pass
    def set_resource(self, resource: Resource | None):
        pass

For example, to set the attributes of an instance, one would call

  res1 = coll.create_resource()
  res1.set_id( "MyFirstResourceId" )
  res1.set_name( "MyFirstResourceName" )

Relationships are managed in a similar way:

  capa1 = coll.create_resource_capacity()
  capa1.set_quantity(1)
  capa1.set_resource(res1) # this automatically calls Resource.set_resource_capacity()
  res1.get_resource_capacity() == capa1 # true

  capa2 = coll.create_resource_capacity()
  capa2.set_quantity(2)
  res1.set_resource_capacity(capa2) # this automatically update the inverted relationships
  res1.get_resource_capacity() == capa2 # true
  capa1.get_resource() == None # true
  capa2.get_resource() == res1 # true

3.3. Accessing DOM Collector Instances

When primary key information is provided in the JDL declaration, it can be used to access individual instances from their collector using a generated API.

The following API is available in the Java collector class for example:

public interface CapacityPlanning extends DbDomCollector {
  Resource getFromResource(String id);
  ResourceCapacity getFromResourceCapacity(String resourceId);
  ...
}

In Python, the collector class provides the following API:

 class CapacityPlanning(DbDomCollector):
  def find_resource_by_business_key(self, id: str) -> Resource | None:
    pass
  def find_resource_capacity_by_business_key(self, id: str) -> ResourceCapacity | None:
    pass

3.3.1. Accessing Instances in Java

Here are some examples of use of the API in Java:

  Resource res1 = coll.createResource(); // res1 is added to the collector instances
  1 == coll.getResource().size(); // is true

  res1.setId( "MyFirstResourceId" );    // res1 primary key is complete so can be accessed with getFromResource
  res1 == coll.getFromResource( "MyFirstResourceId" ); // is true

Instances can also be accessed through the list of all known instances.

public interface CapacityPlanning extends DbDomCollector {
  List<Resource> getResource();
  List<ResourceCapacity> getResourceCapacity();
  ...
}

Entities that are declared as single.row are accessed as singletons: the following JDL declaration

entity SolutionSummary {
  // DOM [single.row] : [true]
  start Instant,
  end Instant,
  timespan Integer
}

Will cause the following API to be generated:

public interface CapacityPlanning extends DbDomCollector {
  SolutionSummary getSolutionSummary();
  void setSolutionSummary(SolutionSummary value);
  ...
}

3.3.2. Accessing Instances in Python

Here are some examples of use of the API in Python:

  res1 = coll.create_resource() # res1 is added to the collector instances
  1 == len(coll.get_all_resource()) # is True
  res1.set_id( "MyFirstResourceId" )    # res1 primary key is complete so can be accessed with find_resource_by_business_key
  res1 == coll.find_resource_by_business_key( "MyFirstResourceId" ) # is True

Instances can also be accessed through the list of all known instances.

class CapacityPlanning(DbDomCollector):
  def get_all_resource(self) -> FrozenSet[Resource]:
    pass
  def get_all_resource_capacity(self) -> FrozenSet[ResourceCapacity]:
    pass

Entities that are declared as single.row are accessed as singletons: the following JDL declaration

entity SolutionSummary {
  // DOM [single.row] : [true]
  start Instant,
  end Instant,
  timespan Integer
}

Will cause the following API to be generated:

class CapacityPlanning(DbDomCollector):
  def find_solution_summary(self) -> SolutionSummary | None:
    pass
  def set_solution_summary(self, solution_summary: SolutionSummary) -> SolutionSummary | None:
    pass

3.4. Deleting DOM Collector Instances

To remove an instance from a collector in Java, one should call:

  coll.getResource().remove(res1);
  0 == coll.getResource().size(); // is true
  null == coll.getFromResource( "MyFirstResourceId" ); // is true

  res1.getResourceCapacity() != null; // may be true. Removing an instance does not clear its relationships.

The following code in Python achieves the same goal:

coll.remove(res1)
0 == len(coll.get_all_resource()) # is True
None == coll.find_resource_by_business_key( "MyFirstResourceId" ) # is True
res1.get_capacity() == None # True: In python removing an entity from the collector clean its relationships
[Note]

Note that:

  • Removing an instance from the collector does not clean its relationships.

  • Bulk methods are also available to delete all the instances of a given type. Check the DbDomCollectorImpl class for details.

4. Saving Data Snapshots

A collector content can be saved in and loaded from multiple formats.

  • zipped csv, which is a set of CSV files put into a zip archive. This format is the default one.

  • dbrf, which is a CSV based format

  • xlsx, which is the Excel format.

Only the Excel format can be edited manually.

To save or load a collector, you should call the following functions:

  File snapshot = new File("xxx"); // File format will be deducted from the file extension (.xlsx, .zip, .dbrf, .dbrf.gz, .xcsv).
  coll.saveSnapshot(snapshot);
  coll.loadSnapshot(snapshot);

In Python the xcsv is the only format supported. One can call the following functions to save or load a collector:

snapshot_file = "xxx"
# When manipulating a collector structure
coll = load_collector(snapshot_file, entity_filter)
save_collector(coll, snapshot_file, entity_filter)
# When manipulating dataframe structures
dataframe_dict = load_data_frame_dict(snapshot_file, entity_filter)
save_data_frame_dict(dataframe_dict, snapshot_file, entity_filter)
[Note]

Note that snapshots are tightly linked with the business data model. They include references to class names and packages.

entity_filter is optional and indicates which entities to load. If not provided, all entities are loaded.

5. Customizing Generated Code

In some specific cases, the generated DOM Java classes require to be customized, for instance when migrating a legacy DOC 3.x application.

[Note]

Note that DOM Python classes are not customizable. This section only applies to Java.

5.1. Preparing the Generated Code

The first step is to modify the code generation configuration in the gene-model/gene-model-dom/build.gradle file:

modelGeneration {
    jdlFile = parent.file("spec/entities.jdl")
    javaPackage = "com.example.caplan"

    useCustomCode = true
}

sourceSets.main.java.srcDir "src/dom/java"

Above:

  • useCustomCode specifies that the generated classes may contain custom code. When set to true, the code generator will keep the custom code of the DOM classes when classes are re-generated.

  • sourceSets.main.java.srcDir must be modified to use the new generated code directory src/dom/java by default instead of ${buildDir}/generated/src/main/java.

The project is now configured to allow customization of the generated code.

5.2. Customizing the Generated Code

Each element of the generated code uses a specific annotation in order to distinguish generated code from custom code.

The custom code can be added at the bottom of the class, after the generated code.

For example, custom imports need to be added outside the block <generated-imports>.

// <generated-imports>
...
// </generated-imports>

import java.util.Map;

@Generated(value = "DomGenerator")
public interface Plant extends DbDomObject {

    @GeneratedMethod
    List<Activity> getActivities();

    ...

    /**
     * @return the sum of the activities duration that are using this plant
     */
    Double getDurationInHours();
}
// <generated-imports>
...
// </generated-imports>

import java.util.Map;

@Generated("DomGenerator")
public class PlantImpl extends DbDomObjectImpl implements Plant {

    @GeneratedField
    protected List<Activity> activities;

    ...

    @Override
    @GeneratedMethod
    public List<Activity> getActivities() {
        return this.activities;
    }

    ...

    /**
     * @return the sum of the activities duration that are using this plant
     */
    @Override
    public Double getDurationInHours() {
        return getActivities().stream().mapToDouble(Activity::getDurationInHours).sum();
    }
}