dataaaaa!

Recap of the Data for AI community event highlighting data integration with Airbyte, vector search, and hands-on demos for data engineers.

Introduces dbt-ml-eval for integrating machine learning evaluation metrics into dbt, streamlining ML model analysis workflows.

Review of three technical data engineering failures in 2025, covering distribution shifts and pipeline contamination, with solutions using DuckDB and dbt for semantic monitoring.

Baboon is a minimal data modeling language and DAG-based compiler for declarative schemas and schema evolution, generating code for C# and Scala.

Describes how metadata enhances AI efficiency by structuring data, using tools like DataHub and Apache Atlas for management and lineage tracking.

The article outlines data engineering anti-patterns for LLMs, such as poor context management and misuse of coding abilities, highlighting open-source tools to optimize API calls and improve accuracy.

ClickHouse dataset with 28.74 million Hacker News posts and their vector embeddings, including steps to create a table, load data, build a vector similarity index, and perform ANN searches.

DuckDB introduces support for INSERT, UPDATE, and DELETE operations on Iceberg v2 tables through the DuckDB-Iceberg extension, including functions for table properties management and limitations on partitioned tables.

The article details how to identify and fix data bottlenecks using volume, variety, and velocity stress tests, instead of relying on observability tools that amplify noise.

Global Cloudflare outage caused by a ClickHouse database permission change, leading to 5xx errors and Bot Management module crash due to memory limit overflow.

Zalando implements Delta Sharing to provide analytical-ready datasets to partners, replacing fragmented methods like SFTP and APIs with an open, cloud-agnostic protocol for scalable data sharing.

Conceptual framework for anticipating future data platform needs through consumption patterns, emphasizing decoupling and data products as interfaces.

Platform offering SQL case studies to enhance data engineering skills, featuring practical examples and query challenges.

Headset cut Snowflake costs by 83% by optimizing ingestion with Estuary, implementing incremental dbt materializations, and offloading queries to Greybeam for embedded analytics.

Introduces conversational analytics as an emerging approach using AI for natural language querying of data, enhancing accessibility and efficiency in data analysis workflows.

Introduction to the dbc data format, a columnar format optimized for in-memory analytics, integrated with the Apache Arrow ecosystem and compatible with DuckDB.

We need a neutral standard for semantic layers. MetricFlow is not the answer, as it remains too dependent on the dbt ecosystem to be truly universal.

dbt Fusion engine employs state-aware orchestration to cut costs by only rebuilding changed models, optimizing data pipeline execution through Ahead-of-Time compilation.

dltHub introduces an LLM-native workflow using scaffolds to automate data ingestion, validate with the dashboard, and transform with Marimo and IBIS, eliminating boilerplate code.

Survey on data and AI stacks for data engineers, providing an enterprise AI playbook and exclusive insights in exchange for participation.

Explores code-first data infrastructure and a programmable lakehouse using Git semantics to safely isolate AI agents and prevent data corruption.

Comparison of Lance and Iceberg formats in lakehouse architectures, highlighting performance benefits for AI/ML workloads and technical differences in file and table formats.

Exploration of freelancing trends in data and AI, covering types of hiring companies, mission formats, and daily rates for data engineers.

Daggie is an Apache Airflow plugin that audits DAG quality with scoring based on best practices, featuring a FastAPI API and animated React UI.

Guide on building a scalable SQL query automation system, covering data engineering concepts and best practices to avoid SQL hacking.

Explores semantic layers for AI, moving beyond SQL generation to enhance data governance and interoperability.

Explores Git-inspired workflows for data to manage environments, test on real data, and deploy confidently, addressing versioning and rollback challenges in data pipelines.

Platform showcasing data engineering projects using Apache Airflow, dbt, and other tools for orchestration, ETL, and data analytics.

Multi-cloud cost analytics platform leveraging dlt for data ingestion, DuckDB for storage, and Rill for visualization, featuring incremental loading and integration of AWS, GCP, and Stripe data.

Technical roadmap for DuckLake outlining planned features including PostgreSQL integration, macros, and performance optimizations for data engineers.

Webinar on the dbt Fusion engine that boosts data pipeline efficiency by 29%, focusing on technical optimizations for data engineers.

Analysis of native LLM workflows emphasizing declarative convergence engineering to avoid fragile imperative approaches, using scaffolds to steer models towards correct outcomes.

Explores building a multimodal AI lakehouse with LanceDB, covering AI-native file formats and new architectures for AI agents.

Snowflake acquires Select Star technology to expand Horizon Catalog’s view of enterprise data for next-gen AI.

Plotly Studio uses AI to generate data analysis code in a deterministic way, with clear and modifiable specifications for transparent verification.

Explore an automated workflow for generating semantic layers with dbt, Cube.js, and Superset, syncing metrics without manual duplication.

B-Trees are essential data structures for databases, optimizing disk access with high fanout, reducing tree height, and minimizing search operations.

Data contracts establish formal agreements between data producers and consumers, protecting ML pipelines from breaking changes. They define the structure, quality, and semantics of data being exchanged.

Snowflake introduces its Well-Architected Framework to optimize data engineering solutions.

Microsoft Power BI integrates context-aware intelligence with Copilot and MCP protocol for faster and secure analytics.

The Power BI Modeling MCP Server enables AI agents to interact with Power BI semantic models through modeling operations and DAX queries.

BigQuery AI integrates advanced data and ML tools, converging data and AI.

This project uses dlt to extract cost data from AWS, GCP, and Stripe, models it with DuckDB, and visualizes it with Rill, providing a unified view of multi-cloud costs.

Guide on structuring dbt projects with recommendations on files, folders, and models for staging, intermediate, and marts layers.

DuckDB v1.4 introduces data-at-rest encryption with AES-GCM-256 and AES-CTR-256, detailing implementation, usage, and performance implications.

Google announces BigQuery Agent Analytics for the Google ADK, enabling insights from interactions.

BigQuery announces new connectors and features for its Data Transfer Service.

Presentation of 4 maturity levels for pipeline orchestration, from Cron to tools like Airflow or Prefect, with concrete examples using GitHub Actions, Lambda, and Docker.

Snowflake event in Paris about AI and data with hands-on workshops on Snowflake Intelligence and dynamic tables.

Keynotes, ateliers et autres vidéos du dbt Coalesce 2025