AssetsDefinition.to_source_assets to method convert a set of assets to SourceAsset objects.MultiPartitionsDefinitions.mlflow pin from the dagster-mlflow package.When using build_asset_reconciliation_sensor, in some cases duplicate runs could be produced for the same partition of an asset. This has been fixed.
When using Pythonic configuration for resources, aliased field names would cause an error. This has been fixed.
Fixed an issue where context.asset_partitions_time_window_for_output threw an error when an asset was directly invoked with build_op_context.
[dagster-dbt] In some cases, use of ephemeral dbt models could cause the dagster representation of the dbt dependency graph to become incorrect. This has been fixed.
[celery-k8s] Fixed a bug that caused JSON deserialization errors when an Op or Asset emitted JSON that doesn't represent a DagsterEvent.
Fixed an issue where launching a large backfill while running dagster dev would sometimes fail with a connection error after running for a few minutes.
Fixed an issue where dagster dev would sometimes hang when running Dagster code that attempted to read in input via stdin.
Fixed an issue where runs that take a long time to import code would sometimes continue running even after they were stopped by run monitoring for taking too long to start.
Fixed an issue where AssetSelection.groups() would simultaneously select both source and regular assets and consequently raise an error.
Fixed an issue where BindResourcesToJobs would raise errors encapsulating jobs which had config specified at definition-time.
Fixed Pythonic config objects erroring when omitting optional values rather than specifying None.
Fixed Pythonic config and resources not supporting Enum values.
DagsterInstance.local_temp and DagsterInstance.ephemeral now use object instance scoped local artifact storage temporary directories instead of a shared process scoped one, removing a class of thread safety errors that could manifest on initialization.
Improved direct invocation behavior for ops and assets which specify resource dependencies as parameters, for instance:
class MyResource(ConfigurableResource): pass @op def my_op(x: int, y: int, my_resource: MyResource) -> int: return x + y my_op(4, 5, my_resource=MyResource())
[dagster-azure] Fixed an issue with an AttributeError being thrown when using the async DefaultAzureCredential (thanks @mpicard)
[ui] Fixed an issue introduced in 1.2.3 in which no log levels were selected by default when viewing Run logs, which made it appear as if there were no logs at all.
environment_vars argument to ScheduleDefinition is deprecated (the argument is currently non-functional; environment variables no longer need to be whitelisted for schedules)--path-prefix can now be supplied via Helm chart (thanks @mpicard)define_asset_job now auto-infer their partitions definitions if not explicitly defined.define_asset_job. This allows putting them on a schedule/sensor.instance property to the HookContext object that is passed into Op Hook functions, which can be used to access the current DagsterInstance object for the hook.create_table_schema_metadata_from_dataframe function to generate a TableSchemaMetadataValue from a Pandas DataFrame. Thanks @AndyBys!dag_run configuration on the integration’s database resources.dagster asset wipe-partitions-status-cache to see the partitions displayed.1.1.18, users with a gRPC server that could not access the Dagster instance on user code deployments would see an error when launching backfills as the instance could not instantiate. This has been fixed.build_asset_reconciliation_sensor from kicking off runs of downstream assets. This has been fixed.build_asset_reconciliation_sensor is now much more performant in cases where unpartitioned assets are upstream or downstream of static-partitioned assets with a large number of partitions.jobNamespace field in the Dagster Helm chart when using the CeleryK8sRunLauncher.en-US browser locale. This has been fixed.Dagster is now tested on Python 3.11.
Users can now opt in to have resources provided to Definitions bind to their jobs. Opt in by wrapping your job definitions in BindResourcesToJobs. This will become the default behavior in the future.
@op(required_resource_keys={"foo"}) def my_op(context) print(context.foo) @job def my_job(): my_op() defs = Definitions( jobs=BindResourcesToJobs([my_job]) resources={"foo": foo_resource}
Added dagster asset list and dagster asset materialize commands to Dagster’s command line interface, for listing and materializing software-defined assets.
build_schedule_from_partitioned_job now accepts jobs partitioned with a MultiPartitionsDefinition that have a time-partitioned dimension.
Added SpecificPartitionsPartitionMapping, which allows an asset, or all partitions of an asset, to depend on a specific subset of the partitions in an upstream asset.
load_asset_value now supports SourceAssets.
[ui] Ctrl+K has been added as a keyboard shortcut to open global search.
[ui] Most pages with search bars now sync the search filter to the URL, so it’s easier to bookmark views of interest.
[ui] In the run logs table, the timestamp column has been moved to the far left, which will hopefully allow for better visual alignment with op names and tags.
[dagster-dbt] A new node_info_to_definition_metadata_fn to load_assets_from_dbt_project and load_assets_from_dbt_manifest allows custom metadata to be attached to the asset definitions generated from these methods.
[dagster-celery-k8s] The Kubernetes namespace that runs using the CeleryK8sRunLauncher are launched in can now be configured by setting the jobNamespace field in the Dagster Helm chart under celeryK8sRunLauncherConfig.
[dagster-gcp] The BigQuery I/O manager now accepts timeout configuration. Currently, this configuration will only be applied when working with Pandas DataFrames, and will set the number of seconds to wait for a request before using a retry.
[dagster-gcp][dagster-snowflake] [dagster-duckdb] The BigQuery, Snowflake, and DuckDB I/O managers now support self-dependent assets. When a partitioned asset depends on a prior partition of itself, the I/O managers will now load that partition as a DataFrame. For the first partition in the dependency sequence, an empty DataFrame will be returned.
[dagster-k8s] k8s_job_op now supports running Kubernetes jobs with more than one pod (Thanks @Taadas).
AssetValueLoader.load_asset_value() didn’t load transitive resource dependencies correctly.RunConfig object with optional config arguments would lead to an error.ScheduleEvaluationContext.scheduled_execution_time to not be Optional.OpExecutionContext.partition_time_window ****(thanks @elben10).InputContext.upstream_output.log is no longer None when loading a source asset.async def ops are now invoked with asyncio.run.TimeWindowPartitionDefinition now throws an error at definition time when passed an invalid cron schedule instead of at runtime.cron_schedule_timezone inside of the config for a dbt model would not result in that property being set on the generated FreshnessPolicy. This has been fixed.GCSComputeLogManager when the session does not have permissions to generate signed urls.validate_run_config no longer accepts pipeline_def or mode arguments. These arguments refer to legacy concepts that were removed in Dagster 1.0, and since then there have been no valid values for them.Added experimental support for resource requirements in sensors and schedules. Resources can be specified using required_resource_keys and accessed through the context or specified as parameters:
@sensor(job=my_job, required_resource_keys={"my_resource"}) def my_sensor(context): files_to_process = context.my_resource.get_files() ... @sensor(job=my_job) def my_sensor(context, my_resource: MyResource): files_to_process = my_resource.get_files() ...
dagster instance migrate after upgrading to 1.2.0.dagster dev command that can be used to run both Dagit and the Dagster daemon in the same process during local development. [docs]DynamicPartitionsDefinition allows partitioning assets dynamically - you can add and remove partitions without reloading your definitions (experimental). [docs]TimeWindowPartitionMapping now accepts start_offset and end_offset arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets [docs].dagster-wandb with Weights & Biases allows you to orchestrate your MLOps pipelines and maintain ML assets with Dagster. [docs]dagster-snowflake-pyspark allows you to store and load PySpark DataFrames as Snowflake tables using the snowflake_pyspark_io_manager. [docs]dagster-gcp-pandas and dagster-gcp-pyspark allow you to store and load Pandas and PySpark DataFrames as BigQuery tables using the bigquery_pandas_io_manager and bigquery_pyspark_io_manager. [docs]dagster-airflow integration library was bumped to 1.x.x, with that major bump the library has been refocused on enabling migration from Airflow to Dagster. Refer to the docs for an in-depth migration guide.create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op).create_databricks_job_op op factory is now deprecated.Stay tuned, as this is only the first part of the overhaul. We’ll be adding more chapters - including automating materializations, using resources, using I/O managers, and more - in the next few weeks.
@graph_asset and @graph_multi_asset.project_fully_featured example now uses the built in DuckDB and Snowflake I/O managers.make_persistent_airflow_db_resource this enables support for Airflow features like pools and cross-dagrun state sharing. In particular retry-from-failure now works for jobs generated from Airflow DAGs.BigQueryPandasTypeHandler now uses google.bigquery.Client methods load_table_from_dataframe and query rather than the pandas_gbq library to store and fetch DataFrames.args instead of both command and args for user code deployments, allowing to include a custom ENTRYPOINT in your the Dockerfile that loads your code.protobuf<4 pin in Dagster has been removed. Installing either protobuf 3 or protobuf 4 will both work with Dagster.@graph_asset and @graph_multi_asset now support passing metadata (thanks @askvinni)!@graph_asset and @graph_multi_asset to be ignored.TableRecord.@multi_asset and other functions would register as type errors for mypy and other static analyzers.start_date would not be converted correctly causing runs to fail.dagster instance migrate:id column to the kvs, daemon_heartbeats and instance_info tables, enforcing that all tables have a primary key.The minimum grpcio version supported by Dagster has been increased to 1.44.0 so that Dagster can support both protobuf 3 and protobuf 4. Similarly, the minimum protobuf version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound grpcio pin in place in Dagster, and hope to be able to remove it very soon.
Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change.
[dagster-snowflake] The execute_queryand execute_queries methods of the SnowflakeResource now have consistent behavior based on the values of the fetch_results and use_pandas_result parameters. If fetch_results is True, the standard Snowflake result will be returned. If fetch_results and use_pandas_result are True, a pandas DataFrame will be returned. If fetch_results is False and use_pandas_result is True, an error will be raised. If both are False, no result will be returned.
[dagster-snowflake] The execute_queries command now returns a list of DataFrames when use_pandas_result is True, rather than appending the results of each query to a single DataFrame.
[dagster-shell] The default behavior of the execute and execute_shell_command functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in env={} to these functions.
[dagster-k8s] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are:
dagster/max_retries tag set. You can configure a global number of retries in the Helm chart by setting run_retries.max_retries to a value greater than the default of 0.Each of these features can be disabled in the Helm chart to restore the previous behavior.
[dagster-k8s] The experimental k8s_job_op op and execute_k8s_job functions no longer automatically include configuration from a dagster-k8s/config tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a k8s_job_op, you can set the container_config, pod_template_spec_metadata, pod_spec_config, or job_metadata config fields on the k8s_job_op (or arguments to the execute_k8s_job function).
[dagster-databricks] The integration has now been refactored to support the official Databricks API.
create_databricks_job_op is now deprecated. To submit one-off runs of Databricks tasks, you must now use the create_databricks_submit_run_op.databricks_client resource must now begin with https://.LogicalVersion has been renamed to DataVersion and LogicalVersionProvenance has been renamed to DataProvenance.DynamicPartitionsDefinition to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods.MetadataEntry have been removed.DagsterTypeMaterializer, DagsterTypeMaterializerContext, and @dagster_type_materializer have been removed.PartitionScheduleDefinition has been removed.RunRecord.pipeline_run has been removed (use RunRecord.dagster_run).DependencyDefinition.solid has been removed (use DependencyDefinition.node).pipeline_run argument to build_resources has been removed (use dagster_run)iteritems usage was removed and changed to the recommended items within dagster-snowflake-pandas (thanks @sethkimmel3)!@asset_graph decorator (thanks @simonvanderveldt)!DataVersions can now be returned on OutputOpExecutionContext.get_asset_provenancePicking a github issue was not correctly rendering, this has been fixed (thanks @Sedosa)!