Initial push for adding orchestration reuse ID #258

RyanLettieri · 2024-01-05T19:08:35Z

Related Issue: #255

Signed-off-by: Ryan Lettieri <[email protected]>

jviau

I think you may have already updating the protobuf submodule, but was there merge issues in that repo? Looks like all the entity work is gone.

jviau · 2024-01-05T22:25:26Z

src/Abstractions/Internal/IOrchestrationSubmitter.cs

@@ -40,5 +41,6 @@ public interface IOrchestrationSubmitter
        TaskName orchestratorName,
        object? input = null,
        StartOrchestrationOptions? options = null,
+        HashSet<string>? orchestrationIdReusePolicy = null,


Please add this to StartOrchestrationOptions instead of its own parameter. All non-essential parameters for affecting orchestration start should go there.

Also, should this be an enum with the available behaviors as values in it?

jviau · 2024-01-05T22:30:10Z

src/Abstractions/Internal/IOrchestrationSubmitter.cs

@@ -40,5 +41,6 @@ public interface IOrchestrationSubmitter
        TaskName orchestratorName,
        object? input = null,
        StartOrchestrationOptions? options = null,
+        HashSet<string>? orchestrationIdReusePolicy = null,


"ReusePolicy" isn't that clear to me. Was the name InstanceIdConflictBehavior considered?

jviau · 2024-01-09T22:03:15Z

This will also address #183

jviau

I suggest the following approach:

Create a new enum in Abstractions.csproj to represent the possible states for this feature. This is the type the customer will interact with.
Add this as an optional value to StartOrchestrationOptions and ~~SubOrchestrationOptions~~ (actually only worry about Start options in this PR, we can tackle sub-orchestration options later)
Add a configurable default value to DurableTaskClientOptions and DurableTaskWorkerOptions. This default value will be used when a customer does not explicitly provide it on the options from point 2.
Update clients and workers to consume this as appropriate. The gRPC client/worker implementation would convert the public facing enum to the gRPC enum internally.

jviau · 2024-01-10T17:57:47Z

src/Abstractions/Abstractions.csproj

    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="6.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="6.0.0" />
    <PackageReference Include="System.Text.Json" Version="6.0.0" />
  </ItemGroup>

+  <ItemGroup>
+    <ProjectReference Include="../../src/Grpc/Grpc.csproj" />


This project should not reference gRPC. It is meant to be implementation agnostic.

Signed-off-by: Ryan Lettieri <[email protected]>

jviau · 2024-01-11T23:28:26Z

src/Abstractions/TaskOptions.cs

+/// <param name="OrchestrationIdReusePolicy">The orchestration reuse policy. This allows for the reuse of an instance ID
+/// as well as the options for it.</param>
+public record StartOrchestrationOptions(string? InstanceId = null, DateTimeOffset? StartAt = null,
+ Dictionary<List<OrchestrationOptions.OrchestrationRuntimeStatus>, OrchestrationOptions.InstanceIdReuseAction>? OrchestrationIdReusePolicy = null);


I see what you are trying to solve here: making it more granular on what the behavior is based on the state of the orchestration. Which I do think is worth considering.

I see the following problems with this approach:

This information does not align with the protobuf spec

It is a bit unwieldy for customers to need to specify this dictionary.

Instead, here is what I recommend:

Lets first evaluate if this is needed? Or can we decide and document what the behavior of the 3 individual policies are based on the orchestration state? For example, we may consider applying conflict resolution for non-terminal orchestrations only.

If not, then we can expand the protobuf and our enum to have more states that better clarify what happens. IE: ErrorAlways, ErrorIfRunning eg.

The end result should be customers need to only provide a single enum to this options object.

src/Abstractions/OrchestrationOptions.cs

Signed-off-by: Ryan Lettieri <[email protected]>

jviau · 2024-01-25T00:19:55Z

src/Abstractions/OrchestrationOptions.cs

+/// <summary>
+/// Enum describing the runtime status of the orchestration.
+/// </summary>
+public enum OrchestrationRuntimeStatus


I think we will need to have type-forwarding from this type in the client package.

jviau · 2024-01-25T00:23:38Z

src/Abstractions/TaskOptions.cs

+/// <param name="OrchestrationIdReusePolicy">The orchestration reuse policy. This allows for the reuse of an instance ID
+/// if the instance ID referenced is in any of the states supplied in this parameter.</param>
+public record StartOrchestrationOptions(string? InstanceId = null, DateTimeOffset? StartAt = null,
+ OrchestrationRuntimeStatus[]? OrchestrationIdReusePolicy = null);


Can we wrap this in a type: OrchestrationIdReuse, so we can have helpers:

OrchestrationIdReuse.IfTerminal; // has all terminal statuses included. OrchestrationIdReuse.Always; // has all statuses included. OrchestrationIdReuse.IfStatus(params OrchestrationRuntimeStatus[] statuses); OrchestrationIdReuse.IfStatus(IEnumerable<OrchestrationRuntimeStatus> statuses); OrchestrationIdReuse.Never; // equal to default.

jviau · 2024-01-25T00:24:22Z

src/Client/Grpc/GrpcDurableTaskClient.cs

@@ -83,6 +83,7 @@ public override ValueTask DisposeAsync()
            Version = orchestratorName.Version,
            InstanceId = options?.InstanceId ?? Guid.NewGuid().ToString("N"),
            Input = this.DataConverter.Serialize(input),
+            OrchestrationIdReusePolicy = { },


You need to map this from options correct?

jviau · 2024-01-25T00:24:50Z

src/Client/OrchestrationServiceClientShim/Client.OrchestrationServiceClientShim.csproj

@@ -17,11 +22,13 @@ The client is responsible for interacting with orchestrations from outside the w

  <ItemGroup>
    <ProjectReference Include="../Core/Client.csproj" />
+    <ProjectReference Include="../../Grpc/Grpc.csproj" />


this shouldn't be depending on gRPC

jviau · 2024-01-25T00:25:11Z

src/Client/OrchestrationServiceClientShim/Client.OrchestrationServiceClientShim.csproj

@@ -8,6 +8,11 @@ The client is responsible for interacting with orchestrations from outside the w
    <EnableStyleCop>true</EnableStyleCop>
  </PropertyGroup>

+  <ItemGroup>
+    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />


What are these packages needed for?

RyanLettieri added 4 commits January 4, 2024 06:22

Initial push for adding orchestration reuse ID

e3c1304

Signed-off-by: Ryan Lettieri <[email protected]>

Updating proto

47c5446

Signed-off-by: Ryan Lettieri <[email protected]>

Updating proto again

d44e411

Signed-off-by: Ryan Lettieri <[email protected]>

Including proto definitions and moving reuseID into options

4fa6141

Signed-off-by: Ryan Lettieri <[email protected]>

jviau requested changes Jan 5, 2024

View reviewed changes

jviau reviewed Jan 5, 2024

View reviewed changes

jviau requested changes Jan 10, 2024

View reviewed changes

Creating abstraction for runtime status and orchestration action enum

4e7de08

Signed-off-by: Ryan Lettieri <[email protected]>

jviau requested changes Jan 11, 2024

View reviewed changes

Fixing ID Reuse Policy

81b993a

Signed-off-by: Ryan Lettieri <[email protected]>

jviau reviewed Jan 25, 2024

View reviewed changes

Merge branch 'microsoft:main' into reuse-id

5a79b5c

cgillum mentioned this pull request Jul 10, 2024

Add example for workflow ID reuse policy dapr/docs#3909

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial push for adding orchestration reuse ID #258

Initial push for adding orchestration reuse ID #258

RyanLettieri commented Jan 5, 2024

jviau left a comment

jviau Jan 5, 2024

jviau Jan 5, 2024

jviau Jan 5, 2024 •

edited

Loading

jviau commented Jan 9, 2024

jviau left a comment •

edited

Loading

jviau Jan 10, 2024

jviau Jan 11, 2024

jviau Jan 25, 2024

jviau Jan 25, 2024

jviau Jan 25, 2024

jviau Jan 25, 2024

jviau Jan 25, 2024

Initial push for adding orchestration reuse ID #258

Are you sure you want to change the base?

Initial push for adding orchestration reuse ID #258

Conversation

RyanLettieri commented Jan 5, 2024

jviau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jviau Jan 5, 2024 • edited Loading

Choose a reason for hiding this comment

jviau commented Jan 9, 2024

jviau left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jviau Jan 5, 2024 •

edited

Loading

jviau left a comment •

edited

Loading