Extracting data from documents

To process documents, you must specify which data points, or fields, you want to extract. If your project includes different document types, like a mix of passports and driver’s licenses, create a class for each document type and specify fields for each class.

You can create up to 250 classes per project, and up to 100 fields per class.

The classes and fields in your project form the project schema: the blueprint of information you want to extract from documents.

Projects in agent mode can use a model to autogenerate classes and fields based on uploaded documents.

Autogenerated schemas create up to 20 fields using basic field types like text extraction and list extraction. This approach works well for straightforward extraction needs or as a starting point for more complex schemas that you can refine.

1. In the editing panel, click Autogenerate schema.

   Commercial & Enterprise If your project includes multipage files, you’re prompted to optionally enable splitting files.

   Configure file splitting based on your production processing requirements. For example, if you intend to process compiled PDFs, enable file splitting even if your project includes only separate files. You can manually enable or disable file splitting in project settings.

   It might take several minutes for the schema to generate.

   You can use the schema as-is, modify it, or click the Undo schema icon to clear the autogenerated schema and manually generate your project schema.

Manually generating a project schema

To manually create your project schema, create classes for each document type in your project, if necessary, then create fields for the data points you want to extract. This approach gives you complete control over your schema.

Manual schema generation works across all processing modes and provides access to all field types.

Creating classes

If your project includes different document types, start by creating a class for each document type. You can then specify a different set of fields for each class.

Organization members can import prebuilt classes from a library of established schemas, such as paystubs, invoices, bank statements, and utility bills.

In projects with classification, a default class called other is assigned to documents that can’t be classified. You can’t delete or modify this class.

1. In the editing panel, click the Create classes icon , then select one of these options based on your AI Hub subscription and project requirements:

   • Create classes — Lets you create a custom class without any fields. If you select this option, enter a succinct name for your document type, then click ← to exit the class editing panel.

   • Browse prebuilt classes Commercial & Enterprise — Lets you add common document types and their associated fields based on a library of available schemas. If you select this option, choose the prebuilt classes that you want to add and click Add to project.

2. Use the Create classes icon to add more classes as needed.

3. When you’re done creating classes, click Classify documents.

   Commercial & Enterprise If your project includes multipage files, you’re prompted to optionally enable splitting files. You can split by document—which lets the model determine where document breaks occur—or split each page—which creates a new document at every page break.

   Configure file splitting based on your production processing requirements. For example, if you intend to process compiled PDFs, enable file splitting even if your project includes only separate files. You can manually enable or disable file splitting in project settings.

   Classes are assigned to your documents and documents are grouped by class in the document list. Any documents that can’t be classified are assigned the other class.

4. Verify classification. If documents weren’t classified as expected, edit classes to improve your results.

   1. In a class that wasn’t identified accurately, click the overflow icon , then select Edit class.

   2. Enter a description to help the model more accurately identify documents in the class, then click ← to exit the class editing panel.

      Effective descriptions include unique identifying details about a document class.

      In projects that don’t use file splitting, you can reference file extensions to help classify documents. For example, the description for an Images class might be Files with image file extensions, like JPEG, PNG, and TIF.

      As a best practice, limit class descriptions to 1,000 characters (4,000 maximum).

   3. Use the overflow icon to edit more classes as needed.

   4. When you’re done editing classes, click Classify documents.

Classification functions let you use a Python function to classify documents based on deterministic rules rather than relying on LLM classification.

When a classification function is enabled for a project, it replaces LLM classification entirely. All documents are classified using the custom function logic instead of AI inference.

Classification functions are useful when you need predictable, rule-based classification for documents that follow consistent patterns. Instead of relying on model inference, you can implement custom logic that identifies document types based on specific text markers.

For example, you might use a classification function to classify insurance forms based on specific identifiers:

1
def unnamed_custom_function(context, keys, class_names):
2
    """
3
    Classifies insurance documents based on form codes and titles.
4
    Returns the appropriate class, which must appear as a key in class_names.
5
    """
6
    document_text = context['document_text']
7
8
    # Check for form codes near the beginning of the document
9
    document_start = document_text[:2000].upper()
10
11
    if "RNST" in document_start or "REINSTATEMENT REQUEST" in document_start:
12
        return 'Reinstatement Form'
13
    elif "TXFT" in document_start or "TRANSFER OF POLICY" in document_start:
14
        return 'Transfer Form'
15
    else:
16
        return 'Other'

Classification functions accept these parameters:

Classification functions must return a value from the class_name_enum that corresponds to an existing class in your project. If the function returns a class name that doesn’t exist, the document is assigned to the default “other” class.

For additional guidance about custom functions, see Writing custom functions.

Splitting functions let you use a Python function to split multipage files into documents using rule-based logic rather than relying on LLM inference.

When a splitting function is enabled for a project, it replaces LLM-based splitting entirely. All files are split using the custom function logic instead of AI inference.

Splitting functions are useful when you need predictable, rule-based splitting for files that follow consistent patterns. Instead of relying on model inference, you can implement custom logic that identifies document boundaries based on specific text markers, structural elements, visual objects, or a combination of criteria.

Splitting functions process the entire file and return a list of dictionaries that specify page ranges and class labels for each document split. Each dictionary must include:

• class_label — The class name for the document split
• page_start — The starting page number (0-indexed)
• page_end — The ending page number (0-indexed, inclusive)

For example, you might use a splitting function to split files based on barcode separator pages:

1
def unnamed_custom_function(context, keys, class_names):
2
    """
3
    Splits documents based on barcode separator pages.
4
    Returns a JSON-serialized list of page range dictionaries.
5
    """
6
    import json
7
8
    page_by_page_text = context['document_page_by_page_text']
9
    splits = []
10
    current_start = 0
11
12
    for page_num, page_text in enumerate(page_by_page_text):
13
        # Check if this page is a barcode separator (mostly empty or contains barcode)
14
        is_separator = (
15
            len(page_text.strip()) < 50 or  # Mostly empty page
16
            'BARCODE' in page_text.upper()   # Contains barcode marker
17
        )
18
19
        if is_separator and current_start < page_num:
20
            # End previous document before separator
21
            splits.append({
22
                "class_label": "Document",
23
                "page_start": current_start,
24
                "page_end": page_num
25
            })
26
            current_start = page_num + 1  # Start next document after separator
27
28
    # Add final document if there are remaining pages
29
    if current_start < len(page_by_page_text):
30
        splits.append({
31
            "class_label": "Document",
32
            "page_start": current_start,
33
            "page_end": len(page_by_page_text) - 1
34
        })
35
    # If no splits were created (e.g., all pages are separators), create a single split for all pages
36
    elif not splits and page_by_page_text:
37
        splits.append({
38
            "class_label": "Document",
39
            "page_start": 0,
40
            "page_end": len(page_by_page_text) - 1
41
        })
42
43
    return splits

Splitting functions accept these parameters:

Splitting functions must return a JSON-serialized list of dictionaries. Each dictionary represents one document split and must include class_label, page_start, and page_end keys. All class_label values must match existing class names in your project. Page ranges must not overlap and must cover all pages in the file.

You can use splitting functions independently or together with classification functions. When used together, the splitting function determines page ranges and assigns initial class labels, then the classification function processes each split document to refine or override the class assignment.

For additional guidance about custom functions, see Writing custom functions.

Creating fields

Create fields for each of the data points you want to identify.

1. In the editing panel, click Add field.

2. Enter a field name or select a suggested field name, then press Enter.

   Data is extracted based on field name alone and the result is displayed.

3. Do one of the following, based on whether your result is accurate:

   • Accurate result — Click ← to exit the field editing panel and continue adding fields.

   • Inaccurate result — Edit the field. When you’re done editing, click ← to exit the field editing panel and continue adding fields.

Editing fields

If a field doesn’t return the results you expect, you have options to fine-tune the results.

Access the field editor for an existing field by hovering over the field and clicking the edit icon .

In the field editor, first choose the field type appropriate for the data you want to identify.

With a suitable field type selected, if necessary, provide a more detailed description or prompt describing the information you’re looking for. As a best practice, keep field and attribute names under 48 characters and use a description or prompt for longer content up to 1,000 characters (4,000 maximum). For best practices, see Writing effective prompts.

When you’re done editing a field, click Run to see results and further refine your edits if needed.

Field types

Choose the field type appropriate for the data you want to identify.

For more guidance, see Choosing field types.

The custom function field type lets you use a Python function to compute values or import data to your project schema.

For example, you might use a custom function to calculate total invoice amount using existing subtotal and tax rate fields:

1
subtotal = float(subtotal)
2
tax_rate = float(tax_rate) / 100
3
tax_amount = subtotal * tax_rate
4
total_amount = subtotal + tax_amount
5
6
return round(total_amount, 2)

Custom function fields accept these parameters:

For additional guidance about custom functions, see Writing custom functions.

Viewing results across documents

To quickly scan or compare results, click the Results table icon in the Documents header.

The results table corresponds to the current view in the editing panel, so the results you see change depending on your current task.

Reordering fields

To change the order of fields in the field editor, use the up and down arrows that display when you hover over a field.

Reordering fields can be necessary when creating derived fields, which can reference fields that precede it in the field editor. Additionally, reordering fields can be helpful to speed up reviews or support downstream integrations, because fields are displayed in processed results in the same order as in the field editor.

Hiding fields

Hiding intermediate or computational fields can help simplify human review and downstream integration output.

Consider hiding fields that are used exclusively as input for derived fields or custom functions. For example, you might extract individual date components in separate hidden fields, then combine them into a final formatted date field that reviewers and downstream systems actually need.

To mark a field as hidden, open the field editor and enable Hide field.

Hidden fields can’t have validation rules, because validations on hidden fields could create confusing review scenarios. If you hide a field with an active validation rule, the rule is removed. If you later unhide the same field, any previous validation rules are restored.

Hidden fields use processing resources and count toward field limits, but their visibility varies across different AI Hub interfaces: