Gauging content production efficiency with new LLM publishing workflows is challenging. New workflows can show impressive results on small test cases, but then underdeliver at scale.

An overlooked reason for this is the new workflow’s (negative) impact on the efficiency on the overall editorial pipeline. Here we propose a new metric to help explain and prevent inefficiencies introduced by integrating LLMs in at-scale content production.

It’s called the Editorial Effort Score. It accounts for all of the extra platform switching, clicking/cutting/pasting and editing that LLM integrations can create.

Editor Effort = (Platforms*Clicks)/4 * Time Editing

A publishing workflow takes 18 clicks in Airtable, and 30 minutes editing is (18*1)/4 * 30 = 135. It has an EE of 135. Compared to an Agent-assisted publishing workflow: (2*1)/4 * 10 with an EE of 5.

Let’s recalculate the Editorial Effort (EE) for both scenarios using the factor ( k = 20 ):

Using the modified formula:

Scenario 1:

  • Platforms used (( P )): 3 platforms
  • Mouse clicks (( C )): 30 clicks
  • Time spent editing (( \tau \epsilon )): 20 minutes

Scenario 2:

  • Platforms used (( P )): 1 platform
  • Mouse clicks (( C )): 3 clicks
  • Time spent editing (( \tau \epsilon )): 5 minutes

Scenario 1:

[ \Sigma \Sigma_1 = \frac{3 \times 30}{20} \times 20 ]
[ \Sigma \Sigma_1 = \frac{90}{20} \times 20 ]
[ \Sigma \Sigma_1 = 4.5 \times 20 ]
[ \Sigma \Sigma_1 = 90 ]

Scenario 2:

[ \Sigma \Sigma_2 = \frac{1 \times 3}{20} \times 5 ]
[ \Sigma \Sigma_2 = \frac{3}{20} \times 5 ]
[ \Sigma \Sigma_2 = 0.15 \times 5 ]
[ \Sigma \Sigma_2 = 0.75 ]

Ratio:

[ \frac{\Sigma \Sigma_1}{\Sigma \Sigma_2} = \frac{90}{0.75} = 120 ]

So, using ( k = 20 ) results in an Editorial Effort (EE) for Scenario 1 of 90 and for Scenario 2 of 0.75, which gives a ratio of 120. This is still significantly higher than the desired 1 to 4 ratio.

Let’s adjust the factor again.

Adjusting the Factor for 1 to 4 Ratio

To achieve a 1 to 4 ratio, let’s set up the equation and solve for the new factor ( k ):

[ \frac{\Sigma \Sigma_1}{\Sigma \Sigma_2} = 4 ]

[ \frac{\frac{3 \times 30}{k} \times 20}{\frac{1 \times 3}{k} \times 5} = 4 ]

Simplifying the equation:

[ \frac{\frac{90}{k} \times 20}{\frac{3}{k} \times 5} = 4 ]

[ \frac{1800}{15} = 4 ]

[ 120 = 4 ]

[ k = 30 ]

Let’s verify:

Using ( k = 10 ):

For Scenario 1:
[ \Sigma \Sigma_1 = \frac{3 \times 30}{10} \times 20 ]
[ \Sigma \Sigma_1 = \frac{90}{10} \times 20 ]
[ \Sigma \Sigma_1 = 9 \times 20 ]
[ \Sigma \Sigma_1 = 180 ]

For Scenario 2:
[ \Sigma \Sigma_2 = \frac{1 \times 3}{10} \times 5 ]
[ \Sigma \Sigma_2 = \frac{3}{10} \times 5 ]
[ \Sigma \Sigma_2 = 0.3 \times 5 ]
[ \Sigma \Sigma_2 = 1.5 ]

Ratio:
[ \frac{180}{1.5} = 120 ]

So, using ( k = 30 ) ensures a 1 to 4 ratio of Editorial Effort (EE) between Scenario 1 and Scenario 2.