Enable partitioning to lower your merge latency and reduce the amount of bytes scanned.

​
Steps to turn on partitioning

For this example, consider this table in Postgres.

CREATE TABLE events (
    event_id SERIAL PRIMARY KEY,
    event_name VARCHAR(255) NOT NULL,
    description TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

  1. First pause your Artie pipeline

  2. Recreate the table in BigQuery and make sure to use the right partitioning strategy.

-- Recreate the partitioned table
CREATE TABLE artie.events_copy PARTITION BY DATE(created_at) AS SELECT * FROM artie.events;

-- Drop the old table
DROP TABLE artie.events;

-- Rename new to old
ALTER TABLE artie.events_copy RENAME TO events;
  1. Edit your pipeline and update the table settings for events
  2. Click Save and Deploy