Infer Data Type Transform

The Infer Data Type transform automatically detects and converts columns to more appropriate data types based on their content. This optimization can improve memory usage and enhance the accuracy of your data representation.

Basic Usage

To infer and adjust data types in your dataset:

1. Select the Infer Data Type transform from the transform menu.
2. Choose the columns you want to include in the inference process in the "Columns to Consider" dropdown.
3. Apply the transformation.

note

The transform will display the names of the selected columns for clarity. This provides transparency about which columns are being processed.

Configuration Options

Basic Options

• Columns to Consider: Select the columns you want to include in the data type inference process. The names of the selected columns will be displayed.

tip

If you don't select any columns, the transform will consider all columns in the dataset.

How It Works

The Infer Data Type transform performs the following operations:

1. Date/Time Detection: Attempts to convert string columns to datetime objects when possible.
2. Numeric Conversion: Converts numeric strings to appropriate numeric types (integer or float).
3. Categorical Identification: Identifies and converts columns that should be considered categorical.
4. Numeric Downcasting: Attempts to downcast numeric columns to smaller types where applicable to save memory.

Examples

Here's an example of how the Infer Data Type transform works:

Example: Inferring Types in a Sales Dataset

Input Dataset:

date_column	season_column	sales_column
October 10, 2023	Fall	123.45
10/31/2023	Fall	234.56
November 15, 2023	Fall	345.67
12/31/2023	Winter	678.90

Initial Data Types:

• date_column: object (string)
• season_column: object (string)
• sales_column: object (string)

Configuration:

• Columns to Consider: All columns

Result:

date_column	season_column	sales_column
2023-10-10	Fall	123.45
2023-10-31	Fall	234.56
2023-11-15	Fall	345.67
2023-12-31	Winter	678.90

Inferred Data Types:

• date_column: datetime64[ns]
• season_column: category
• sales_column: float32

The transform has converted the date strings to datetime objects, recognized the season column as categorical, and converted the sales values to an appropriate numeric type.

Best Practices

1. Review Results: After applying the transform, review the inferred types to ensure they align with your expectations and requirements.

2. Consider Domain Knowledge: While automatic inference is powerful, your domain knowledge about the data might sometimes suggest different type choices.

3. Handle Mixed Data Types: Be aware that columns with mixed data types might not be converted as expected. Clean your data before applying this transform if necessary.

4. Preserve Original Data: Consider keeping a copy of your original dataset before applying type changes, especially for critical data.

5. Check for Information Loss: Ensure that downcasting numeric types doesn't lead to loss of precision that's important for your analysis.

Troubleshooting

• If date/time columns aren't being recognized, check for inconsistent date formats within the column.
• For columns not being converted as expected, examine the data for any outliers or inconsistencies that might be preventing type inference.
• If categorical columns with many unique values aren't being converted, you might need to manually specify them as categorical.