Workflow Commands

Manage Treasure Data workflows powered by Digdag.

Alias

tdx wf is an alias for tdx workflow

Commands

Command	Description
`use <project>`	Set default workflow project
`projects [pattern]`	List workflow projects
`workflows [project]`	List workflows in a project
`run [project.workflow]`	Run a workflow (interactive selector if omitted)
`sessions [project]`	List workflow sessions
`attempts [project]`	List workflow attempts

Project Sync

Command	Description
`pull <project>`	Pull project to local folder
`push [source-dir]`	Push local changes to TD
`clone [source-dir] --name <name>`	Clone project to a new name
`delete <project>`	Delete a project

Secrets

Command	Description
`secrets list [project]`	List secret keys
`secrets set [project] KEY=VALUE`	Set a secret
`secrets delete [project] <key>`	Delete a secret

Typical Usage

bash

# 0. List available workflow projects
tdx wf projects

# 1. Pull a workflow project to local folder
#    (this also sets project context)
tdx wf pull myproject
# Creates: workflows/myproject/

# 2. Edit workflow files locally
# Edit workflows/myproject/main.dig with your text editor

# 3. Push changes back to TD
tdx wf push

# 4. Run a workflow (interactive selector shows available workflows)
tdx wf run

# 5. Monitor sessions and attempts (context already set by pull)
tdx wf sessions
tdx wf attempts

The pull/push workflow enables version control and code review for workflow definitions:

bash

# Work from any subdirectory - push finds tdx.json automatically
cd workflows/myproject
git add . && git commit -m "Update ETL schedule"
tdx wf push

How It Works

The tdx wf commands help you manage workflow projects as local files. You can pull existing projects from Treasure Data, edit them locally, and push changes back.

Session Context

The default project is stored in your session context (same as tdx use). Running tdx wf pull also sets the default project automatically. Use tdx use to view all current session settings.

List Projects and Workflows

The list commands display results in a compact list format by default, with status icons and relevant metadata.

bash

# List all workflow projects (default: 100)
tdx wf projects

# Filter projects by pattern
tdx wf projects "cdp_segment_*"

# Fetch more projects with --limit
tdx wf projects --limit 200

# Output as table format
tdx wf projects --format table

# List all workflows
tdx wf workflows

# List workflows for specific project
tdx wf workflows myproject

# Filter workflows by project pattern
tdx wf workflows "lda*"

Output format:

Projects: 📁 project_name (with (system) label for system workflows)
Workflows: 📋 workflow_name (project_name)

When results may be truncated, a hint is shown: (use --limit to show more)

Sessions and Attempts

Sessions and attempts are displayed with status icons indicating their state.

bash

# List all workflow sessions
tdx wf sessions

# List sessions for specific project
tdx wf sessions myproject

# List sessions for specific workflow
tdx wf sessions myproject.daily_etl

# Filter sessions by status
tdx wf sessions --status error
tdx wf sessions --status running

# Filter sessions by time range
tdx wf sessions --from "2025-01-01" --to "2025-01-31"

# Fetch more sessions
tdx wf sessions --limit 200

# List all attempts
tdx wf attempts

# List attempts including retried
tdx wf attempts --include-retried

In interactive mode (terminal), tdx wf sessions displays a cursor-based selector allowing you to navigate to a session's timeline view:

bash

# View sessions interactively
tdx wf sessions myproject

# Use arrow keys to navigate, Enter to view timeline, Esc to exit

How it works:

Sessions are displayed in timeline format
An interactive selector appears with sessions listed
Use ↑↓ arrow keys to navigate
Press Enter to view the selected session's timeline
Press Esc to exit and view the static session list

Non-interactive mode:

When piping output: tdx wf sessions | grep "success"
When using --format: tdx wf sessions --format json
When writing to file: tdx wf sessions --output sessions.txt

Status icons:

✅ success
❌ error
🔄 running
🛑 canceling
⏳ pending

Output format:

Sessions: status_icon project.workflow session_time status
Attempts: status_icon project.workflow #attempt_id session_time status

Attempt Management

The attempt command provides subcommands for managing workflow attempts:

bash

tdx wf attempt <attempt-id> [action] [args...]

Show Attempt Details

bash

# Show specific attempt details
tdx wf attempt 67890

View Tasks

bash

# Show tasks for an attempt
tdx wf attempt 67890 tasks

# Show tasks including subtasks
tdx wf attempt 67890 tasks --include-subtasks

View Logs

bash

# View logs with interactive task selector
tdx wf attempt 67890 logs

# View logs for a specific task
tdx wf attempt 67890 logs +step1

# List available log files as JSON
tdx wf attempt 67890 logs --json

When no task name is provided, an interactive selector will display available tasks with their log file sizes.

Kill Attempt

bash

# Kill a running attempt (with confirmation)
tdx wf attempt 67890 kill

# Kill with reason and skip confirmation
tdx wf attempt 67890 kill --reason "manual stop" -y

Timeline Visualization

View task execution timeline as an ASCII Gantt chart, showing when each task started, its duration, and current status with colored bars.

bash

# Interactive session selector
tdx wf timeline

# Show latest session for specific workflow (uses session context for project)
tdx wf use myproject
tdx wf timeline daily_etl

# Filter by project and workflow
tdx wf timeline myproject.daily_etl

# Direct navigation using session or attempt ID
tdx wf timeline --session-id 12345
tdx wf timeline --attempt-id 67890

# Follow running attempt with live updates
tdx wf timeline daily_etl --follow

# JSON output for scripting
tdx wf timeline myproject.daily_etl --format json

You can jump directly to a specific session or attempt using ID options:

bash

# Show timeline for a specific session (uses last attempt)
tdx wf timeline --session-id 12345

# Show timeline for a specific attempt
tdx wf timeline --attempt-id 67890

# Combine with --follow for live updates
tdx wf timeline --attempt-id 67890 --follow

Option priority:

--attempt-id - Uses the attempt ID directly (highest priority)
--session-id - Fetches the session and uses its last attempt
[project.workflow] - Interactive selector or auto-select latest

If both --attempt-id and --session-id are provided, --attempt-id takes precedence.

Example Output:

Attempt #164700627 - success - 5m 23s
                                 2026-01-03   14:15:00    14:16:20    14:17:40    14:19:00    14:20:20
✅ +extract_users                  (1m 20s)│───────────────────────                                          │
✅ +transform_data                 (2m 45s)│            ─────────────────────────────────                    │
✅ +load_to_warehouse              (3m 10s)│                                    ───────────────────────────  │
❌ +send_notification                 (5s)│                                                            ×××   │
✅ +cleanup                          (45s)│                                                               ───│
Total: 5  ✅ 4  ❌ 1  🔄 0  ⏳ 0

Timeline Features:

Colored lines show task execution periods:
- ✅ Green lines (─) for successful tasks
- ❌ Red X marks (×) for failed tasks
- 🔄 Blue lines (─) for running tasks
- 🛑 Yellow lines (─) for blocked tasks
- ⏳ Dim dashes (─) for planned tasks
Time axis with auto-scaled labels (seconds to hours)
Task statistics showing success/error/running counts
Live updates with --follow flag (refreshes every 2 seconds)
Console URL link for detailed view in web UI

Timeline Options

Option	Description
`--session-id <id>`	Show timeline for specific session (uses last attempt)
`--attempt-id <id>`	Show timeline for specific attempt
`--follow`	Follow/watch timeline for running attempts (auto-refresh every 2s)
`--limit <n>`	Limit sessions shown in selector (default: 50)

Use Cases

Monitor running workflows:

bash

tdx wf timeline myproject --follow

Analyze completed workflow:

bash

tdx wf timeline myproject.nightly_batch

Export timeline data:

bash

tdx wf timeline myproject.daily_etl --format json > timeline.json

Retry Attempt

bash

# Retry attempt
tdx wf attempt 67890 retry

# Retry attempt from specific task
tdx wf attempt 67890 retry --resume-from +step2

# Retry with parameter override
tdx wf attempt 67890 retry --params '{"key":"value"}'

# Force retry without confirmation
tdx wf attempt 67890 retry --force -y

Run Workflow

Start a new workflow run (attempt) directly from the command line.

bash

# Start a workflow run
tdx wf run myproject.myworkflow

# Start with parameters
tdx wf run myproject.myworkflow --param key1=value1 --param key2=123

# Start with multiple parameters
tdx wf run myproject.myworkflow --param env=production --param date=2025-01-15 --param count=100

# Start with custom session time
tdx wf run myproject.myworkflow --session-time "2025-01-15T10:00:00Z"

# Combine parameters and session time
tdx wf run myproject.myworkflow --param env=staging --session-time "2025-01-15T00:00:00+09:00"

Run Options

Option	Description
`--param <key=value>`	Workflow parameter (repeatable)
`--session-time <time>`	Session time in ISO 8601 format (default: now)

Parameter Type Detection

Parameter values are automatically parsed:

Numbers: --param count=100 → {"count": 100}
Booleans: --param enabled=true → {"enabled": true}
Strings: --param name=hello → {"name": "hello"}
JSON: --param config={"a":1} → {"config": {"a": 1}}

Session Retry

To retry a session (as opposed to a specific attempt), use the retry command with the session: prefix:

bash

# Retry entire session
tdx wf retry session:12345

# Retry session from specific task
tdx wf retry session:12345 --from-task +step2

# Retry with parameter override
tdx wf retry session:12345 --params '{"key":"value"}'

# Retry with params from file
tdx wf retry session:12345 --params @params.json

Project Management

Pull/Push Workflow

The pull and push commands provide a sync-style workflow for managing workflow projects locally, similar to how tdx sg and tdx journey work for segments.

bash

# Pull project to local folder
tdx wf pull myproject                    # Creates workflows/myproject/

# Pull to custom directory
tdx wf pull myproject ./my-workflows

# Pull specific revision
tdx wf pull myproject --revision v1.0.0

# Preview changes without writing
tdx wf pull myproject --dry-run

After pulling, your local folder will contain:

tdx.json - Project metadata for sync tracking
*.dig - Workflow definition files
*.sql, *.py, etc. - Supporting files

bash

# Push local changes back to TD
cd workflows/myproject
tdx wf push

# Push from specific directory
tdx wf push ./workflows/myproject

# Push with custom revision
tdx wf push --revision v2.0.0

# Preview changes without pushing
tdx wf push --dry-run

The push command will:

Compare local files with the remote project
Show a summary of changes (new, modified, deleted files)
Display diffs for modified files
Ask for confirmation before pushing

Clone Project

The clone command creates a copy of a workflow project with a different name. This is useful for:

Deploying workflows to different environments (dev → staging → prod)
Creating variations of existing workflows
Pushing to different TD profiles (cross-profile deployment)

bash

# Clone from a local directory to a new project name
tdx wf clone workflows/myproject --name myproject-staging

# Clone using session context
tdx wf use myproject
tdx wf clone --name myproject-prod

# Clone with custom revision
tdx wf clone workflows/myproject --name myproject-v2 --revision v2.0.0

# Preview before creating
tdx wf clone workflows/myproject --name myproject-test --dry-run

# Cross-profile deployment
tdx wf clone workflows/myproject --name myproject-prod --profile production

The clone command will:

Check if the source directory contains valid workflow files
Check if a project with the target name already exists
Show a warning if the target project exists (will create new revision)
Ask for confirmation before creating/updating the project

Difference from Push

push updates the same project (based on tdx.json)
clone creates/updates a different project with a new name

Download Project (Legacy)

bash

# Download workflow project (without tdx.json tracking)
tdx wf download myproject

# Download to specific directory
tdx wf download myproject ./workflows

# Download specific revision
tdx wf download myproject --revision v1.0.0

Upload Project (Legacy)

The upload command provides digdag-style push for users who prefer the traditional workflow:

bash

# Upload workflow project (digdag-style)
cd myproject
tdx wf upload myproject

# Upload with custom revision
tdx wf upload myproject --revision v1.0.0

# Upload from parent directory with path
tdx wf upload ./myproject

# Upload with --name override
tdx wf upload myproject --name production_workflow

# Upload without validation
tdx wf upload myproject --skip-validation

Delete Project

bash

# Delete workflow project (with confirmation)
tdx wf delete myproject

# Delete by project ID
tdx wf delete 12345

# Delete without confirmation
tdx wf delete myproject -y

Command Options

List Commands Options

These options apply to projects, workflows, sessions, and attempts commands.

Option	Description
`--limit <n>`	Maximum results to return (default: 100, fetches multiple pages if needed)
`--format <format>`	Output format: `table`, `json`, `jsonl` (default: list style)

Sessions Options

Option	Description
`--status <status>`	Filter by status (running, success, error, blocked, all)
`--from <timestamp>`	Start time filter (ISO 8601)
`--to <timestamp>`	End time filter (ISO 8601)

Attempts Options

Option	Description
`--include-retried`	Include retried attempts

Attempt Options

Option	Description
`--include-subtasks`	Include subtasks in tasks output
`--reason <text>`	Reason for killing the attempt
`--resume-from <task>`	Resume from specific task (retry)
`--params <json>`	Override parameters for retry (JSON or @file.json)
`--force`	Force retry even if not failed
`-y, --yes`	Skip confirmation prompt

Session Retry Options

Option	Description
`--from-task <task>`	Resume from specific task
`--params <json>`	Override parameters (JSON or @file.json)

Pull Options

Option	Description
`--revision <revision>`	Pull specific revision (default: latest)
`--dry-run`	Preview changes without writing files

Push Options

Option	Description
`--revision <revision>`	Custom revision name (default: timestamp)
`--dry-run`	Preview changes without pushing
`--skip-validation`	Skip validation of .dig files

Clone Options

Option	Description
`--name <name>`	Name for the new project (required)
`--revision <revision>`	Custom revision name (default: timestamp)
`--dry-run`	Preview without creating the project
`--skip-validation`	Skip validation of .dig files

Upload Options (Legacy)

Option	Description
`--name <name>`	Project name (default: directory name)
`--revision <revision>`	Revision name (default: timestamp)
`--skip-validation`	Skip validation of .dig files

Secrets Management

Manage secrets for workflow projects. Secrets can be used in workflow configurations (e.g., ${secret:td.apikey}).

Default Project

Set a default project with tdx wf use <project> to omit the project argument from secrets commands.

List Secrets

bash

# List all secret keys for a project
tdx wf secrets list myproject

# With default project set (tdx wf use myproject)
tdx wf secrets list

# List secrets by project ID
tdx wf secrets list 12345

# Output as JSON
tdx wf secrets list myproject --json

Set Secret

bash

# Set a secret for a project
tdx wf secrets set myproject td.apikey=YOUR_API_KEY

# With default project set
tdx wf secrets set td.apikey=YOUR_API_KEY

# Set secret with value containing special characters
tdx wf secrets set myproject DB_URL='postgres://user:pass@host:5432/db'

# Set secret with spaces in value
tdx wf secrets set myproject MESSAGE='Hello World'

Shell Quoting

The shell handles quote removal, so use quotes to protect values with special characters or spaces:

KEY='my value' → value is my value
KEY="value" → value is value
"KEY='value'" → value is 'value' (quotes preserved)

Delete Secret

bash

# Delete a secret from a project (with confirmation)
tdx wf secrets delete myproject td.apikey

# With default project set
tdx wf secrets delete td.apikey

# Delete by project ID
tdx wf secrets delete 12345 db.password

# Delete without confirmation
tdx wf secrets delete myproject td.apikey -y

Secrets Command Reference

Command	Description
`tdx wf secrets list [project]`	List secret keys for a project
`tdx wf secrets set [project] KEY=VALUE`	Set a secret for a project
`tdx wf secrets delete [project] <key>`	Delete a secret from a project

Project Resolution

When project is omitted, it uses the default project set via tdx wf use.

Workflow Commands ​

Commands ​

Navigation & Discovery ​

Project Sync ​

Secrets ​

Typical Usage ​

How It Works ​

List Projects and Workflows ​

Sessions and Attempts ​

Interactive Session Navigation ​

Attempt Management ​

Show Attempt Details ​

View Tasks ​

View Logs ​

Kill Attempt ​

Timeline Visualization ​

Direct Timeline Navigation ​

Timeline Options ​

Use Cases ​

Retry Attempt ​

Run Workflow ​

Run Options ​

Parameter Type Detection ​

Session Retry ​

Project Management ​

Pull/Push Workflow ​

Clone Project ​

Download Project (Legacy) ​

Upload Project (Legacy) ​

Delete Project ​

Command Options ​

List Commands Options ​

Sessions Options ​

Attempts Options ​

Attempt Options ​

Session Retry Options ​

Pull Options ​

Push Options ​

Clone Options ​

Upload Options (Legacy) ​

Secrets Management ​

List Secrets ​

Set Secret ​

Delete Secret ​

Secrets Command Reference ​

Workflow Commands

Commands

Navigation & Discovery

Project Sync

Secrets

Typical Usage

How It Works

List Projects and Workflows

Sessions and Attempts

Interactive Session Navigation

Attempt Management

Show Attempt Details

View Tasks

View Logs

Kill Attempt

Timeline Visualization

Direct Timeline Navigation

Timeline Options

Use Cases

Retry Attempt

Run Workflow

Run Options

Parameter Type Detection

Session Retry

Project Management

Pull/Push Workflow

Clone Project

Download Project (Legacy)

Upload Project (Legacy)

Delete Project

Command Options

List Commands Options

Sessions Options

Attempts Options

Attempt Options

Session Retry Options

Pull Options

Push Options

Clone Options

Upload Options (Legacy)

Secrets Management

List Secrets

Set Secret

Delete Secret

Secrets Command Reference