How much will the DoD spend on AI research contracts in 2021 H2?

Time spent: ~ 1.5 hours

(More context and details on the question from CSET in the footnote. ¹ For the purposes of this question, we’re using Bloomberg Government data.)

The buckets and my credences:

A: <200m (0) B: [200m, 350m] (10%) C: (350m, 500m] (45%) D: (500m, 650m] (40%) E: >650m (5%)

I spent the first 30 minutes brainstorming and making a “low-resolution” forecast by simply extrapolating out from the given data from previous years. (You can view a table of the data at the end of this post.)

Simple Extrapolation

Simple extrapolation using linear regression gives mean estimate of $482.4 million. If I take the confidence intervals seriously–which is definitely not right, more on that at the end–then the 50% CI includes C and D in equal parts. Continuing that process, the simple reference class forecast generated is:

A: <200m (0%) B: [200m, 350m] (25%) C: (350m, 500m] (25%) D: (500m, 650m] (37.5%) E: >650m (12.5%)

There are several shortcomings of this approach:

• The relationship between time period and total contract volume in the data isn’t actually linear, so linear extrapolation isn’t a good fit. The data has a concave shape up, which indicates that the rate of growth in contract volume is increasing.
• The dataset is really small, only including 12 data points over 6 years. This isn’t a problem for Bayesian approaches, but a linear model like this suffers.
• A Bayesian approach which gives credences over a range of possible increases from the previous year would be ideal, but I don’t know how to do that yet.

Considerations for Updating

From doing a 10 minute search, it looks like DoD AI spending is going to continue to increase according to… basically everyone? It was a big priority for the Trump administration and continues to be for the Biden administration’s defense plans.

One anecdote: Earlier this year a Congressional panel, the National Security Commission on AI, recommended doubling federal R&D spending for AI each year until hitting $32 billion in FY 2026.²

Fiscal year considerations

It seems like funding is allocated by fiscal year.

• The fiscal year runs from October 1 (10-1) to September 30 (9-30). We are currently in FY 2021, which runs until 2021-09-30.
• The budget for FY 2022 will therefore go into effect in October, so the period we’re forecasting for has the last three months of FY 2021 and the first three of FY 2022.
• How does this relate to calendar year halves?
  • H1 is always entirely contained within the corresponding FY, e.g. 2016 H1 is in FY 2016.
  • H2 always begins with three months of the corresponding FY and ends with three months of the next FY, e.g. 2016 H2 has 3 months of FY 2017 and 3 months of FY 2017.

Looking at the average growth of H1s and H2s in the data, H1 growth is about two-thirds as large as H2 growth (or H2 growth is about one and a half times H1 growth, on average.). - This, despite a sizable downturn in contract volume in 2020 H2. - I’m not sure how much predictive power this actually has but it seems pretty significant. This updates me upward.

Some Fuzzy Updating

I’m not sure how to correctly update on questions of degree vs. binary questions, so this is fairly fuzzy. I spent about 10 minutes on this.

Very broadly, three factors suggest that I should update upward.

1. The linear extrapolation fit isn’t great–the data has a concave up shape, which indicates that it is growing more quickly as time goes on.
2. We’re forecasting for an H2, which is associated with higher growth rates (about 150% as large as H1)
   • I think that the reason for this is that October through December of H2 are the first three months of the next fiscal year, which has usually brought higher spending on AI contracts as each year more is allocated to that.
   • So H1 contract volume is all in the same budget, while H2 contract volume includes part of the next year’s budget as well.
3. Budget funding for AI spending by the military seems to be growing, based on the Pentagon’s request, Biden’s budget, and so on.
   • That said, the FY 2022 DoD budget request includes $874m for AI spending, compared with $831m for FY 2021 and $927m in FY 2020. These are requests, not budgets, so it’s somewhat hard to parse. ³

Key Uncertainties

1. Using a linear model is not good.
   • If I had more time, I would learn how to write a Bayesian model for time series data. I think this is just slightly above my capabilities at the moment, and it would be very useful to have as a tool for these kinds of questions.

2. I don’t fully understand the mechanisms by which funding is allocated to DoD AI research contracts, and learning that would be useful.

3. I wasn’t able to find much disconfirming evidence, which is worrying. Basically all the articles and reports I found were very bullish on DoD AI spending.
   • But in 2020 H2, there was a large decline in spending relative to 2020 H1: $423 million in the former and $490 million in the latter. This seems strange, and I don’t know what caused it.
   • The one article I found arguing that DoD AI spending would decline is “The Department of Defense’s Looming AI Winter” from May 2021, but the article is about what the authors identify as more longlasting problems with how the DoD is approaching AI, as opposed to about an actually impending collapse in funding.⁴