- Fixed an error when trying to directly invoke a run status sensor when passing resources.
- [dagster-airbyte][dagster-fivetran] Fixed an issue where
EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.
- Substantially improved performance of the Dagster insights DBT/Snowflake usage job.
- The
OpExecutionContext and AssetExecutionContext now have a partition_keys property - [dagster-ui] The asset graph layout algorithm has been changed to a much faster one called “tight-tree”
- [dagster-ui] The Runs table filters has a top level filter for partitions
- [dagster-dbt]
dbt-core==1.7.* is now supported.
- Fixed an issue where some schedules skipped a tick on the day after a fall Daylight Savings Time transition.
- Fixed a bug that caused backfill policies that execute multiple partitions in a single run not to work with dynamic partitions.
- Fixed a bug that caused an error when
build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job. - We now raise an error early if the empty string is provided as an asset’s group name (Thanks Sierrra!)
- Fixed an issue where custom setup and teardown methods were not properly called on nested Pythonic resources.
- Added a warning message when op or asset config is passed as an argument not named
config. - [dagster-cloud] Fixed an issue where overriding the default I/O manager could break the Snowflake-dbt insights job.
- [auto-materialize] Fixed an issue where materializing an unpartitioned parent of a dynamic-partitioned asset would only result in the latest dynamic partition of that asset being requested. Now, all partitions will be requested.
- [dagster-embedded-elt] Fixed an issue in
dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect type - [dagster-ui] Fixed an issue in the launchpad where selecting a partition wouldn’t correctly overwrite fields using the partition’s specific configuration
- A docs fix to the testing concepts page, thanks @NicolaiLolansen!
- The schema can now be overridden per asset in DB IO managers, thanks @jrstats!
- Improved failure recovery and retry behavior when the daemon that launches auto-materialization runs fails or crashes in the middle of a tick.
- [asset-checks] UI performance for displaying check results is improved
- [asset-checks] Removed noisy experimental warning about
AssetCheckKey - [op-concurrency] Fixed a bug where concurrency slots were not getting assigned if a run that was assigned a slot was deleted before the slot had actually been claimed during execution.
- [dagster-pipes] The
PipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess. - [dagster-pipes] Exceptions are now reported in the event log for framework components and from the external process.
- [dagster-k8s] The
PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process. - [ui] The launch asset backfill modal now offers a preview dialog showing the targeted asset partitions and their backfill policies when partition mapping or varying backfill policies are present.
- [asset-checks] New
load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules. - Previously, the daemon process would terminate with an error if it believed that a thread might be hanging, which sometimes caused undesirable terminations when doing large backfills or auto-materializing many assets. Now, the daemon process will emit a warning instead of terminate.
- [dagster-dbt]
dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory. - [dagster-dbt]
@dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping. - [dagster-dbt] Self dependencies can now be enabled for dbt models that are represented by
@dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:
meta:
dagster:
has_self_dependency: True
- Fixed an issue where Dagster imports would throw errors when using
pydantic<2.0.0 but having pydantic-core installed. - Previously, asset backfills that targeted partitioned assets with a
LastPartitionMapping dependency would raise an error. This has been fixed. - Fixed a multipartitions partition mapping bug where a
instance is not available to load partitions error was raised. - [asset-checks] Fixed an issue with conflicting op names when using
build_asset_with_blocking_check - [ui] Viewing run logs containing very large messages no longer causes the UI to crash in Safari on macOS
- [ui] Drilling into the graph of a graph-backed asset with 4+ inputs or outputs no longer causes the asset graph to crash with a rendering error.
- [ui] On the backfill details page, clicking to a specific asset correctly shows the partitions that were materialized for that asset when partition mapping is in use.
- [ui] The Asset > Partition page now loads partition details more quickly in cases where calculating the staleness of the partition took a significant amount of time.
- Fixed a bug introduced in
1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion. - [auto-materialize] Previously, if an asset partition was skipped on a tick for one reason, and then processed and skipped on a later tick for an additional reason, only the most recent skip reason would be tracked. Now, all skip reasons are tracked.
- [dagster-dbt] Fixed an issue where if an
exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled. - [dagster-dbt] When invoking
DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.
- Global op concurrency slots are now released in between retries of op execution failures.
- Updated the tutorial to reflect best practices and APIs as of Dagster 1.5
- The
report_asset_observation REST endpoint for reporting runless events is now available.
- Dagster now supports using Pydantic 2 models for Config and Resources. Pydantic 1.10 continues to be supported.
- Added a
report_asset_observation REST API endpoint for runless external asset observation events - Dramatically improved the performance of partition-mapping, for basic hourly and daily partitions definitions
- [ui] When viewing a list of runs, you can quickly add the tag in the “Launched by” column as a filter on the list view. Hover over the tag to see the “Add to filter” button.
- [helm] The env vars
DAGSTER_K8S_PG_PASSWORD_SECRET and DAGSTER_K8S_INSTANCE_CONFIG_MAP will no longer be set in all pods. - [dagster-pyspark]
build_pyspark_zip now takes an exclude parameter that can be used to customize the set of excluded files. - [ui] Links beginning with http://, https:// in unstructured run logs (via context.log) are automatically converted to clickable links
- Fixed an asset backfill bug where certain asset partitions with no targeted parents would hang indefinitely.
- Fixed a bug where the
source_key_prefix argument to load_assets_from_current_module and load_assets_from_package_name was ignored - Fixed two bugs in
dagster_embedded_elt where the mode parameter was not being passed to Sling, and only one asset could be created at a time - Fixed a bug with handing default values for Pydantic validators on Windows
- [ui] Clicking an asset with checks shows them in the asset graph sidebar, even if live data on the page is still loading.
- [ui] Reported materialization events are presented more clearly in the asset graph sidebar and in other parts of the Dagster UI.
- [helm] The
pipelineRun configuration in the Helm chart is now deprecated. The same config can be set under dagster-user-deployments
- Added
setup_for_execution and teardown_after_execution calls to the inner IOManagers of the BranchingIOManager - thank you @danielgafni! - The
S3FakeResource.upload_fileobj() signature is now consistent with boto3 S3.Client.upload_fileobj() - thank you @jeanineharb! dbt_assets now have an optional name parameter - thank you @AlexanderVR!
- Added a
report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well. - The
config argument is now supported on @graph_multi_asset - [ui] Improved performance for global search UI, especially for deployments with very large numbers of jobs or assets.
- [dagster-pipes] Add S3 context injector/reader.
- [dagster-dbt] When an exception when running a dbt command, error messages from the underlying dbt invocation are now properly surfaced to the Dagster exception.
- [dagster-dbt] The path to the dbt executable is now configurable in
DbtCliResource.
- Fixed a bug introduced in 1.5.3 that caused errors when launching specific Ops in a Job.
- Fixed a bug introduced in 1.5.0 that prevented the
AssetExecutionContext type annotation for the context parameter in @asset_check functions. - Fixed an issue where the Dagster scheduler would sometimes fail to retry a tick if there was an error reloading a code location in the middle of the tick.
- [dagster-dbt] Fixed an issue where explicitly passing in
profiles_dir=None into DbtCliResource would cause incorrect validation. - [dagster-dbt] Fixed an issue where partial parsing was not working when reusing existing target paths in subsequent dbt invocations.
- [ui] Fixed an issue where the job partitions UI would show “0 total partitions” if the job consisted of more than 100 assets
- [dagster-duckdb] The
DuckDBResource and DuckDBIOManager accept a connection_config configuration that will be passed as config to the DuckDB connection. Thanks @xjhc!
- Added events in the run log when a step is blocked by a global op concurrency limit.
- Added a backoff for steps querying for open concurrency slots.
- Auto-materialize logic to skip materializing when (1) a backfill is in progress or (2) parent partitions are required but nonexistent are now refactored to be skip rules.
- [ui] Added 2 new asset graph layout algorithms under user settings that are significantly faster for large graphs (1000+ assets).
- Running multiple agents is no longer considered experimental.
- When the agent spins up a new code server while updating a code location, it will now wait until the new code location uploads any changes to Dagster Cloud before allowing the new server to serve requests.