Artie's Typing Library
Learn how Artie’s internal typing library ensures data integrity and schema evolution across your data sources.
Overview 🎯
Artie’s custom typing library is the backbone of our data transfer system. It ensures that your data maintains its original structure and types when moving between sources and destinations.
Key Features ✨
- Schema Evolution: Automatically detects and adds missing columns
- Type Inference: Intelligently determines data types from values
- Performance: 2x faster than Go’s Reflect library
- Data Integrity: Zero data transformation or loss
Data Type Handling 🔍
Artie’s typing library intelligently handles data type mapping between your source and destination systems. Here’s how it works:
- Initial Type Detection: When data first arrives, Artie analyzes the source schema and values to determine the most appropriate data type for your destination.
- Type Preservation: Once a data type is established in your destination, Artie maintains that type to ensure consistency.
- Automatic Conversion: If source and destination types differ, Artie automatically converts the data while preserving its integrity.
What happens if type conversion fails?
What happens if type conversion fails?
If Artie encounters a type conversion that could lead to data loss or errors, we’ll:
- Raise a clear error message
- Notify you immediately
- Work with you to resolve the issue
Numbers
We preserve the exact numeric type from your source (Postgres, MySQL, or MongoDB). Here’s how we handle different number formats:
5
→ Integer"5"
→ String5.0
→ Float
JSON Objects
We maintain JSON objects exactly as they appear in your source, with no flattening or transformation.
💡 Pro Tip: If you prefer to store JSON as a string, you can explicitly specify the column type as
STRING
in your schema.
Arrays
We support both simple and complex array structures:
- Simple arrays
- Nested arrays with objects
Timestamps, Dates, and Times ⏰
Our library supports 15+ different time formats with zero precision loss. We preserve your original time layout using our custom time.Time
object.
💡 Pro Tip: Like JSON objects, you can override time type inference by specifying your preferred data type in the schema.
Advanced
Postgres variable numeric
Postgres variable numeric
A in Postgres requires special handling when transferring to different destinations. By default, we will create this type as a string equivalent column so that we do not have to trade off data precision. You can override this behavior by modifying the destination column type.
Why do we treat a variable numeric column differently?
We treat this column type differently because Postgres will allow you to store arbitrary amount of digits in each cell up to the . This exceeds most destination’s numeric data type limits, as such we will create a string equivalent column to preserve data precision.
Need Help? 🤝
If you have questions not covered here, reach out to us at [email protected]!