dataaaaa!

Introduces an AI agent starter kit with architectural patterns focusing on cost management, observability, state management, and defensive tooling to ensure production resilience.

The article argues that 90% accuracy in Text-to-SQL systems is insufficient, emphasizing the need for rigorous evaluation using metrics like Execution Accuracy (EX) and integration of tools like BigQuery for robust RAG pipelines.

Data contracts are crucial for governance and reliability in the AI age, ensuring data quality and management.

The article discusses automating data analytics workflows using AI, focusing on data quality pipelines, data engineering, and agentic AI systems to enhance operational efficiency.

Critique of a 15-page requirements document deemed ineffective, highlighting communication challenges and need definition issues in data engineering.

BigQuery introduces a feature for generating SQL queries from comments, streamlining query writing for data engineers.

Reviews alternatives to MinIO for local S3 emulation in single-node setups, focusing on S3 compatibility, Docker images, and open-source licensing, with examples including S3Proxy, RustFS, and SeaweedFS.

Benchmark of 14 analytics agents assessing reliability, UX, cost, and integration with data stacks like dbt and Snowflake, highlighting challenges in enterprise-wide deployment.

Introduces exa-d, a data framework for search engines, featuring typed columns, surgical updates, and parallel execution using Ray Data and Lance to handle petabytes of web data efficiently.

The article discusses trade-offs in using macros in dbt, highlighting that abstractions can increase complexity and friction in debugging and changes, despite their initial goal of reducing duplication.

Databricks open sources Dicer, an auto-sharder for building highly available and scalable sharded services, powering Unity Catalog and SQL orchestration to maintain high cache hit rates during failures.

Discusses the importance of community in data engineering, highlighting tools like dbt Community Slack and practices for building technical networks without burnout.

Fivetran init is a CLI command that accelerates custom connector development by scaffolding project structures and integrating AI assistants to streamline data engineering workflows.

The article traces the evolution of data engineering tools from the 1970s, highlighting persistent challenges in data ingestion, modeling, and transformation despite advancements like SQL, Hadoop, dbt, and AI.

Willis Nana outlines seven commandments for data engineering efficiency, focusing on fast delivery and clean scalability of data pipelines.

Examines emerging trends in observability stacks for data engineering, covering tools and architectures for monitoring data pipelines.

Snowflake enables custom instructions in semantic views using AI_SQL_GENERATION and AI_QUESTION_CATEGORIZATION clauses to direct Cortex Analyst in SQL generation and question classification.

Open-source weather API providing JSON data via HTTP, featuring high-resolution models (1-11 km), hourly updates, and 80 years of historical data for data engineering applications.

The article discusses reconfiguring AI as data discovery agents in B2B contexts, breaking down discovery into atomic problems like existence, access, quality, and intent alignment, requiring tight integration with data platforms to ensure trust and mitigate hallucinations.

Practical guide to enforce data quality across all pipeline stages using frameworks like Dagster, focusing on dimensions such as accuracy and completeness during ingestion and transformation to prevent downstream issues.

Victor Cumer, Data Engineering Team Lead at Qonto, outlines industrializing the data platform through four key projects: observability, data contracts, automation, and self-service, addressing adoption challenges and future steps in data governance.

The Apache Iceberg REST Catalog specification lacks operational SLAs for latency, concurrency, and cross-catalog sync, leading to unpredictable behavior at scale despite semantic interoperability.

Discusses an agentic analytics architecture using AI agents to automate data workflows and enhance decision-making, emphasizing data engineering patterns.

Vinted built a DAG generator using dbt manifests to standardize decentralized data pipelines, employing Airflow with task-per-model granularity to handle cross-domain dependencies and abstract orchestration complexity from teams.

Overview of an AI agentic platform for marketing, focusing on data engineering to process and analyze advertising data at scale.