Changelog#

0.13.4#

New#

• [dagster-k8s] The k8s_job_executor is no longer experimental, and is recommended for production workloads. This executor runs each op in a separate Kubernetes job. We recommend this executor for Dagster jobs that require greater isolation than the multiprocess executor can provide within a single Kubernetes pod. The celery_k8s_job_executor will still be supported, but is recommended only for use cases where Celery is required (The most common example is to offer step concurrency limits using multiple Celery queues). Otherwise, the k8s_job_executor is the best way to get Kubernetes job isolation.
• [dagster-airflow] Updated dagster-airflow to better support job/op/graph changes by adding a make_dagster_job_from_airflow_dag factory function. Deprecated pipeline_name argument in favor of job_name in all the APIs.
• Removed a version pin of the chardet library that was required due to an incompatibility with an old version of the aiohttp library, which has since been fixed.
• We now raise a more informative error if the wrong type is passed to the ins argument of the op decorator.
• In the Dagit Launchpad, the button for launching a run now says “Launch Run” instead of “Launch Execution”

Bugfixes#

• Fixed an issue where job entries from Dagit search navigation were not linking to the correct job pages.
• Fixed an issue where jobs / pipelines were showing up instead of the underlying graph in the list of repository graph definitions.
• Fixed a bug with using custom loggers with default config on a job.
• [dagster-slack] The slack_on_run_failure_sensor now says “Job” instead of “Pipeline” in its default message.

Community Contributions#

• Fixed a bug that was incorrectly causing a DagsterTypeCheckDidNotPass error when a Dagster Type contained a List inside a Tuple (thanks @jan-eat!)
• Added information for setting DAGSTER_HOME in Powershell and batch for windows users. (thanks @slamer59!)

Experimental#

• Changed the job explorer view in Dagit to show asset-based graphs when the experimental Asset API flag is turned on for any job that has at least one software-defined asset.

Documentation#

• Updated API docs and integration guides to reference job/op/graph for various libraries (dagstermill, dagster-pandas, dagster-airflow, etc)
• Improved documentation when attempting to retrieve output value from execute_in_process, when job does not have a top-level output.

0.13.3#

Bugfixes#

• [dagster-k8s] Fixed a bug that caused retries to occur twice with the k8s_job_executor

0.13.2#

New#

• Updated dagstermill to better support job/op/graph changes by adding a define_dagstermill_op factory function. Also updated documentation and examples to reflect these changes.
• Changed run history for jobs in Dagit to include legacy mode tags for runs that were created from pipelines that have since been converted to use jobs.
• The new get_dagster_logger() method is now importable from the top level dagster module (from dagster import get_dagster_logger)
• [dagster-dbt] All dagster-dbt resources (dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the dbt ls command: context.resources.dbt.ls().
• Added ins and outs properties to OpDefinition.
• Updated the run status favicon of the Run page in Dagit.
• There is now a resources_config argument on build_solid_context. The config argument has been renamed to solid_config.
• [helm] When deploying Redis using the Dagster helm chart, by default the new cluster will not require authentication to start a connection to it.
• [dagster-k8s] The component name on Kubernetes jobs for run and step workers is now run_worker and step_worker, respectively.
• Improved performance for rendering the Gantt chart on the Run page for runs with very long event logs.

Bugfixes#

• Fixed a bug where decorating a job with a hook would create a pipeline.
• Fixed a bug where providing default logger config to a job would break with a confusing error.
• Fixed a bug with retrieving output results from a mapped input on execute_in_process
• Fixed a bug where schedules referencing a job were not creating runs using that job’s default run config.
• [dagster-k8s] Fixed a bug where the retry mode was not being passed along through the k8s executor.

Breaking Changes#

• The first argument on Executor.execute(...) has changed from pipeline_context to plan_context

Community Contributions#

• When using multiple Celery workers in the Dagster helm chart, each worker can now be individually configured. See the helm chart for more information. Thanks @acrulopez!
• [dagster-k8s] Changed Kubernetes job containers to use the fixed name dagster, rather than repeating the job name. Thanks @skirino!

Experimental#

• [dagster-docker] Added a new docker_executor which executes steps in separate Docker containers.

• The dagster-daemon process can now detect hanging runs and restart crashed run workers. Currently only supported for jobs using the docker_executor and k8s_job_executor. Enable this feature in your dagster.yaml with:

  run_monitoring:
    enabled: true
  

  Documentation coming soon. Reach out in the #dagster-support Slack channel if you are interested in using this feature.

Documentation#

• Adding “Python Logging” back to the navigation pane.
• Updated documentation for dagster-aws, dagster-github, and dagster-slack to reference job/op/graph APIs.

0.13.1#

New#

• All dbt resources (dbt_cli_resource, dbt_rpc_resource, and dbt_rpc_sync_resource) now support the ls command.

Docs#

• Various fixes to broken links on pages in 0.13.0 docs release

Bug fixes#

• Previously, the Dagster CLI would use a completely ephemeral dagster instance if $DAGSTER_HOME was not set. Since the new job abstraction by default requires a non-ephemeral dagster instance, this has been changed to instead create a persistent instance that is cleaned up at the end of an execution.

Dagit#

• Run-status-colorized dagster logo is back on job execution page
• Improvements to Gantt chart color scheme

0.13.0 "Get the Party Started"#

Major Changes#

• The job, op, and graph APIs now represent the stable core of the system, and replace pipelines, solids, composite solids, modes, and presets as Dagster’s core abstractions. All of Dagster’s documentation - tutorials, examples, table of contents - is in terms of these new core APIs. Pipelines, modes, presets, solids, and composite solids are still supported, but are now considered “Legacy APIs”. We will maintain backcompatibility with the legacy APIs for some time, however, we believe the new APIs represent an elegant foundation for Dagster going forward. As time goes on, we will be adding new features that only apply to the new core. All in all, the new APIs provide increased clarity - they unify related concepts, make testing more lightweight, and simplify operational workflows in Dagit. For comprehensive instructions on how to transition to the new APIs, refer to the migration guide.
• Dagit has received a complete makeover. This includes a refresh to the color palette and general design patterns, as well as functional changes that make common Dagit workflows more elegant. These changes are designed to go hand in hand with the new set of core APIs to represent a stable core for the system going forward.
• You no longer have to pass a context object around to do basic logging. Many updates have been made to our logging system to make it more compatible with the python logging module. You can now capture logs produced by standard python loggers, set a global python log level, and set python log handlers that will be applied to every log message emitted from the Dagster framework. Check out the docs here!
• The Dagit “playground” has been re-named into the Dagit “launchpad”. This reflects a vision of the tool closer to how our users actually interact with it - not just a testing/development tool, but also as a first-class starting point for many one-off workflows.
• Introduced a new integration with Microsoft Teams, which includes a connection resource and support for sending messages to Microsoft Teams. See details in the API Docs (thanks @iswariyam!).
• Intermediate storages, which were deprecated in 0.10.0, have now been removed. Refer to the “Deprecation: Intermediate Storage” section of the 0.10.0 release notes for how to use IOManagers instead.
• The pipeline-level event types in the run log have been renamed so that the PIPELINE prefix has been replaced with RUN. For example, the PIPELINE_START event is now the RUN_START event.

New since 0.12.15#

• Addition of get_dagster_logger function, which creates a python loggers whose output messages will be captured and converted into Dagster log messages.

Community Contributions#

• The run_config attribute is now available on ops/solids built using the build_op_context or build_solid_context functions. Thanks @jiafi!
• Limit configuration of applyLimitPerUniqueValue in k8s environments. Thanks @cvb!
• Fix for a solid’s return statement in the intro tutorial. Thanks @dbready!
• Fix for a bug with output keys in the s3_pickle_io_manager. Thanks @jiafi!

Breaking Changes#

• We have renamed a lot of our GraphQL Types to reflect our emphasis on the new job/op/graph APIs. We have made the existing types backwards compatible so that GraphQL fragments should still work. However, if you are making custom GraphQL requests to your Dagit webserver, you may need to change your code to handle the new types.
• We have paired our GraphQL changes with changes to our Python GraphQL client. If you have upgraded the version of your Dagit instance, you will most likely also want to upgrade the version of your Python GraphQL client.

Improvements#

• Solid, op, pipeline, job, and graph descriptions that are inferred from docstrings now have leading whitespaces stripped out.
• Improvements to how we cache and store step keys should speed up dynamic workflows with many dynamic outputs significantly.

Bugfixes#

• Fixed a bug where kwargs could not be used to set the context when directly invoking a solid. IE my_solid(context=context_obj).
• Fixed a bug where celery-k8s config did not work in the None case:

execution:
  celery-k8s:

Experimental#

• Removed the lakehouse library, whose functionality is subsumed by @asset and build_assets_job in Dagster core.

Documentation#

• Removed the trigger_pipeline example, which was not referenced in docs.
• dagster-mlflow APIs have been added to API docs.