Filtered Indexes: Just Add Includes

I found a quirky thing recently

While playing with filtered indexes, I noticed something odd. By ‘playing with’ I mean ‘calling them horrible names’ and ‘admiring the way other platforms implemented them‘.

I sort of wrote about a similar topic in discussing indexing for windowing functions. It turns out that a recent annoyance could also be solved by putting the column my filter expression is predicated on in the included columns definition. That’s the fanciest sentence I’ve ever written, BTW. If you want more, don’t get your hopes up.

Ready for Horrible

Let’s create our initial index. As usual, we’re using the Stack Overflow database. We’ll look at a small group of users who have a Reputation over 400k. I dunno, it’s a nice number. There are like 8 of them.

CREATE UNIQUE NONCLUSTERED INDEX [Users_400k_Club] 
ON dbo.[Users] ([DisplayName], [Id]) 
WHERE [Reputation] > 400000;

With that in place, we’ll run some queries that should make excellent use of our thoughtful and considerate index.

--Will I Nill I?
SELECT [u].[Id], [u].[DisplayName]
FROM [dbo].[Users] AS [u]
WHERE [u].[Reputation] > 400000

SELECT [u].[Id], [u].[DisplayName]
FROM [dbo].[Users] AS [u]
WHERE [u].[Reputation] > 400000 AND [u].[Reputation] < 450000

SELECT [u].[Id], [u].[DisplayName]
FROM [dbo].[Users] AS [u]
WHERE [u].[Reputation] > 400001

SELECT [u].[Id], [u].[DisplayName]
FROM [dbo].[Users] AS [u]
WHERE [u].[Reputation] > 500000

If you were a betting organism, which ones would you say use our index? Money on the table, folks! Step right up!

Yes, Sorta, No, No.

That didn’t go well at all. Only the first query really used it. The second query needed a key lookup to figure out the less than filter, and the last two not only ignored it, but told me I need to create an index. The nerve!

Send me your money

Let’s make our index better:

CREATE UNIQUE NONCLUSTERED INDEX [Users_400k_Club] 
ON dbo.[Users] ([DisplayName], [Id])
INCLUDE([Reputation]) 
WHERE [Reputation] > 400000
WITH (DROP_EXISTING = ON)
;

Run those queries again. You don’t even have to recompile them.

Can’t you tell by the way I run every time you make eyes at me?

They all magically found a way to use our New and Improved index.

What was the point?

When I first started caring about indexes, and filtering them, I would get so mad when these precious little Bloody Mary recipes didn’t get used.

I followed all the rules!
There were no errors!

But why oh why didn’t SQL use my filtered indexes for even smaller subsets of the filter condition? It seemed insane to me that SQL would know the filter for the index is on (x > y), but wouldn’t use them even if (z > x).

The solution was to put the filtered column in the include list. This lets SQL generate statistics on the column, and much like getting rid of the predicate key lookup, allows you to search within the filtered index subset for even more specific information.