Cleaning results

Cleaning transforms extracted data into standardized formats for downstream systems.

Quick clean

Quick clean options include:

• Changing character casing to all uppercase, all lowercase, or sentence case.

• Removing characters by specifying characters to remove, with no comma separator.

• Reformatting date by selecting from available formatting options.

  In numeric-only dates like 06/01/2024, use the Input field as needed to specify whether the original value lists the month or the day first.

Quick clean options use Python functions to reformat data. There is no model processing and no unit cost.

Cleaning prompts refine field results using natural language instructions.

Follow these best practices to write effective cleaning prompts.

• Include before and after examples for varied input formats.

  Standardize address format to street, city, state ZIP. For example:
  123 Burnside St, Portland OR 97201 → 123 Burnside St, Portland, OR 97201
  456 Elm Avenue, Charlotte, North Carolina 28105 → 456 Elm Ave, Charlotte, NC 28105

• Handle missing or invalid data.

  If the field contains a valid dollar amount, return it in the format $X.XX. If no amount is found or the value is invalid, return “No amount specified.”

• Normalize reference fields as text.

  Convert each list item to the format: Item: [Item], Price: [Price], Qty: [Quantity] and separate each item with a semicolon.

For advanced cleaning requirements involving complex transformations, external data, or mission-critical results, write a custom cleaning function in Python.

For example, you might use a cleaning function to standardize phone numbers from various formats:

1
def clean_phone_number(previous_line, context):
2
    """
3
    Standardizes phone numbers to (XXX) XXX-XXXX format.
4
    A simple example of a custom cleaning function that handles
5
    formatting beyond what quick clean options provide.
6
    """
7
    # Skip processing if input is empty or None
8
    if not previous_line:
9
        return previous_line
10
11
    # Extract only digits from the input
12
    digits = ''.join(char for char in previous_line if char.isdigit())
13
14
    # Format 10-digit numbers as (XXX) XXX-XXXX
15
    if len(digits) == 10:
16
        return f"({digits[0:3]}) {digits[3:6]}-{digits[6:10]}"
17
18
    # Return original for non-standard numbers
19
    return previous_line

Cleaning functions accept these parameters:

Cleaning functions can return any value. The value is converted to a string when it’s passed to subsequent refinement lines or validation rules. If the cleaning function encounters issues, it must raise an exception.

For additional guidance about custom functions, see Writing custom functions.