dataaaaa!

Ingestr is a CLI tool for data ingestion that was rebuilt in Go to overcome Python performance bottlenecks and dependency issues. It moves from a stateful to a stateless design to improve reliability. The goal was to create a CLI tool that would be the easiest way to ingest data.

Fivetran and dbt Labs are merging to build open data infrastructure focused on the agentic era. This integration aims to streamline data ingestion and transformation workflows.

dbt Wizard is an AI agent grounded in project lineage, metrics, and tests to automate refactoring and migrations. It operates via CLI or dbt Cloud to ensure context-aware code generation. dbt's native metadata engine retrieves exactly what the agent needs before it writes a line.

Agents Schema provides a standard metadata layer in data warehouses to help AI agents discover data lineage and metric definitions. By syncing tools like dbt and Fivetran into a central schema, it ensures agents discover what data exists, who owns it, and how to use it responsibly.

dbt State optimizes workflows by skipping or cloning nodes instead of rebuilding them. It compares logic and data freshness across environments to reduce compute costs. dbt State decides whether transformations need to run at all, or whether an existing table can simply be cloned.

Databow is an open-source Rust CLI that unifies database querying via Arrow Database Connectivity (ADBC). It eliminates the need for multiple vendor-specific tools like psql or snowsql.

Duckle is an open-source tool designed to streamline data engineering workflows by simplifying ETL processes. It focuses on enhancing the developer experience when working with DuckDB. The project aims to provide a more accessible interface for managing complex data transformations and pipelines.

Snowflake storage for Apache Iceberg tables is now generally available on AWS and Azure. This managed option eliminates the need for external cloud storage setup. Snowflake storage supports both permanent and transient Iceberg tables while enabling external engine access via Snowflake Horizon.

Omni AI Hub centralizes AI observability, allowing data teams to manage semantic models and validate prompt sets. It bridges the gap between LLM metrics and business logic. Data teams now own the model that powers every AI answer the business sees. This shift requires active governance.

Fivetran and dbt Labs have merged to build open data infrastructure for agentic AI. The union combines automated data movement with governed business logic to ensure autonomous agents access fresh, trusted context. Agents operate at machine-speed and require precise, versioned code to function.

Fivetran and dbt Labs have merged to create a unified foundation for trusted data agents. Key technical updates include dbt Core v2.0, which open-sources the Rust-based Fusion engine for 10x faster parsing. The two-engine era is coming to an end as the platform consolidates for scale.

Fivetran and dbt Labs are building open data infrastructure to provide AI agents with essential context like lineage and state. Key updates include dbt Core v2.0 with a 10x faster Rust-based engine and dbt State, which reduces compute by 30% by only rebuilding modified models.

dbt Core v2.0 unifies the engine with dbt Fusion, moving to a shared Rust-based foundation. It introduces significant parse time improvements and Parquet artifacts for high-performance metadata. dbt Core and Fusion will be built on a shared foundation to ensure a stable language spec.

Building a zero-cost data app involves a pipeline of Python scripts, DuckDB, and GitHub Actions. DuckDB serves as the analytical engine, processing CSVs into JSON without a warehouse.

Google uses zero-trust aggregation to collect population-level insights without exposing individual data. By combining one-shot cryptographic protocols with Trusted Execution Environments, they ensure individual raw data is never exposed or reconstructed in any server memory.

Generative data applications allow engineers to build interactive dashboards and analytics tools using simple prompts. This approach streamlines the creation of data apps, as seen with Mercor using AI analytics to unlock 100M in revenue. Gorgeous, interactive dashboards and apps you can build.

ClickHouse 26.5 introduces 38 new features, including a major optimization that pushes ORDER BY and LIMIT clauses through joins. This allows the engine to sort and limit data on the preserved side before the join occurs, resulting in up to 20x faster performance and 175x lower memory usage.

Streambed is a CDC engine that offloads analytical queries from Postgres to Apache Iceberg without Spark or ETL. It decodes WAL messages, writes Parquet to S3, and uses copy-on-write for updates.

Long-running agents are shifting from simple triggers to complex, multi-step goals like refactoring dbt projects or building libraries from scratch. By using a goal-oriented loop, agents can iterate until they verify success against an oracle.

DuckDB v1.5.3 expands Iceberg support with MERGE INTO for upserts and full ALTER TABLE capabilities for schema evolution. It now handles bucket and truncate partition transforms and Iceberg V3. MERGE INTO uses merge-on-read semantics and writes positional deletes to the Iceberg table.

ktx is a self-improving context layer that automates semantic mapping and wiki ingestion for data agents. It resolves fan traps and detects joinable columns to ensure LLMs use canonical SQL. ktx ingests those layers and combines them with raw-table introspection and wiki content.

dlt is an open-source Python library designed to simplify data pipeline creation. It focuses on loading data from REST APIs into destinations. The tool is described as the open-source Python library for data pipelines to assist with scaling and monitoring.

Boring UI is an open source framework for building agent-centric applications using a dual-pane interface. It integrates an agent chat with a workspace mini-IDE, utilizing the Pi harness to remain model-agnostic. The agent and the user should interact with the same core primitives.

Cost reduction in Snowflake relies on decoupling storage, compute, and services. By optimizing warehouse auto-suspend and refining RBAC, monthly bills can drop from 140k to 38k. Auto-suspend is your best friend for cost control. Avoid using ACCOUNTADMIN for daily tasks to limit risk.

The DiveMaxxing contest focuses on interactive data visualizations built using AI agents integrated with MotherDuck. Engineers use Claude or ChatGPT to perform live queries against datasets, moving beyond static charts. One example includes a star map built on ESA Gaia data.