Module datarush.analytics
Package com.pervasive.datarush.analytics.stats

Class DataQualityAnalyzer

java.lang.Object
com.pervasive.datarush.operators.AbstractLogicalOperator
com.pervasive.datarush.operators.CompositeOperator
com.pervasive.datarush.analytics.stats.DataQualityAnalyzer
All Implemented Interfaces:
LogicalOperator
public final class DataQualityAnalyzer extends CompositeOperator
Evaluates a set of quality tests on an input dataset. Those rows for which all tests pass are considered "clean" and thus sent to the clean output. Those rows for which any tests fail are considered "dirty" and thus sent to the dirty output. In addition, this produces a summary model that includes the following statistics:
  1. totalFrequency: total number of rows
  2. invalidFrequency: total number of rows for which at least one test involving the given field failed
  3. testFailureCounts: per-test failure counts for each test involving the given field

  • Constructor Details

    • DataQualityAnalyzer

      public DataQualityAnalyzer()
      Evaluates a set of quality tests on an input dataset. By default the set of tests is empty; prior to graph compilation the following property must be set:

  • Method Details

    • getInput

      public RecordPort getInput()
      Returns a port for the input dataset to be tested.
      Returns:
      a port for the input dataset to be tested.

    • getTests

      public List<DataQualityAnalyzer.QualityTest> getTests()
      Returns the set of tests to apply to the input dataset
      Returns:
      the set of tests to apply to the input dataset

    • setTests

      public void setTests(List<DataQualityAnalyzer.QualityTest> tests)
      Sets the set of tests to apply to the input dataset
      Parameters:
      tests - the set of tests to apply to the input dataset

    • setTests

      public void setTests(String expression)
      Sets the set of tests to apply to the input dataset. The tests are expressed using the field derivation expression language. The general format of the expression language is: {expression1} as {metric1}[, {expression2} as {metric2}, ...]. The expression themselves are predicate functions that return a boolean value.
      Parameters:
      expression - an expression that evaluates to a set of quality tests

    • getClean

      public RecordPort getClean()
      Returns a port that will output the "clean" rows. A row is considered clean if all tests pass.
      Returns:
      a port that will output the "clean" rows

    • getDirty

      public RecordPort getDirty()
      Returns a port that will output the "dirty" rows. A row is considered dirty if any tests fail.
      Returns:
      a port that will output the "clean" rows

    • getModel

      public PMMLPort getModel()
      Returns a port that will output a PMMLSummaryStatisticsModel. The model will be populated with the following information:
      1. totalFrequency: total number of rows
      2. invalidFrequency: total number of rows for which at least one test involving the given field failed
      3. testFailureCounts: per-test failure counts for each test involving the given field
      Returns:
      a port that will output a PMMLSummaryStatisticsModel.

    • compose

      protected void compose(CompositionContext ctx)
      Description copied from class: CompositeOperator
      Compose the body of this operator. Implementations should do the following:
      1. Perform any validation of configuration, input types, etc
      2. Instantiate and configure sub-operators, adding them to the provided context via the method OperatorComposable.add(O)
      3. Create necessary connections via the method OperatorComposable.connect(P, P). This includes connections from the composite's input ports to sub-operators, connections between sub-operators, and connections from sub-operators output ports to the composite's output ports
      Specified by:
      compose in class CompositeOperator
      Parameters:
      ctx - the context