When I put together the invitation for T-SQL Tuesday #202, I wasn’t sure what kind of stories would come out of it.

The topic was simple:

That one SQL Server outage you’ll never forget.

I expected stories about bad queries, failed deployments, storage problems, or maybe a database that decided to have a very bad day. What I got was much more interesting.

There were stories about ransomware, corrupted databases, deleted storage, power and cooling failures, an identity column reaching its limit, and even a floppy disk. A server Meltdown too.

Some of these outages lasted hours. Others lasted days or even weeks.

And what I really enjoyed was that most of these stories weren’t just about what went wrong. They were about what people learned afterward.

Here are the stories I was able to collect.

Deborah Melkin: Sometimes a Floppy Disk Is All It Takes

Deborah Melkin shared a story from earlier in her career involving a server reboot that went very wrong. A bootable floppy disk had been left in the server. It contained an fdisk /mbr command. The server was rebooted, the command ran, and suddenly they had a much bigger problem than they expected.

The team had to find another server, reinstall SQL Server, and restore the databases from backup. The lesson seems obvious now, but that’s one thing I like about outage stories. Things that seem obvious after the fact aren’t always obvious when you’re standing in the middle of the problem.

Sometimes the lesson is simply to understand what is actually sitting in or connected to the server before you reboot it. And, of course, make sure your backups are somewhere other than the server you’re trying to recover.

Read Deborah’s full story

Andy Yun: The Outages That Live in Infamy

Andy Yun shares two non-SQL Server outages before getting to his SQL Server story. The first was when a backhoe literally dug up his company’s T1 line, leaving them without Internet for most of the day.

The second happened at a software company that supported market traders, where executives chose not to have a backup Internet connection after moving the office to VoIP. When the Internet went down, the entire office, call center, and data center were effectively offline, including their phones.

Both stories reinforced the same lesson: single points of failure hurt.

His SQL Server outage happened while Andy and another DBA were at PASS Summit, when a SAN administrator accidentally deleted the production transaction-log LUN for an instance with several hundred databases and several terabytes of data. Fortunately, their backups were good, and Andy used sp_restoregene to quickly generate the restore commands.

They initially tried four parallel restores, but the SAN couldn’t handle the load, so they stopped and worked with the business to prioritize the databases. The full recovery took three or four days.

What Andy realized afterward was that while they had tested restores for CHECKDB, they had never tested a full-instance restore at that scale. The experience also made him realize how important it was to understand the storage and infrastructure that your disaster recovery plan depends on.

Read Andy’s full story

Aaron Bertrand: When an INT Runs Out

Aaron Bertrand’s story is a good reminder that capacity problems aren’t always about disk space, memory, or CPU. In this case, an identity column reached the maximum value for an int. The result was an outage at Stack Overflow. The immediate solution wasn’t to change the column to bigint.

That would have been much more difficult to do during a live production outage. Instead, the identity was reseeded into the negative range, buying the team years of additional capacity. Problem solved. Until it happened again. A later operation involving IDENTITY_INSERT caused the identity value to move toward the positive limit again. So the team had another outage. And they reseeded it again.

I liked this story because it shows how an emergency fix can create another problem if the underlying issue isn’t eventually addressed. It also reminded me that we tend to think about capacity in terms of infrastructure. But data types have limits too. Sometimes the thing running out of room isn’t the disk.

Read Aaron’s full story

Rob Farley: When Corruption Leaves You With Very Few Options

Rob Farley wrote about an outage involving a bad disk controller that corrupted hundreds of database pages and even affected some backup files. This was one of those situations where the normal recovery path wasn’t enough. Rob and the customer’s CTO started looking at the database table by table, using clustered indexes, nonclustered indexes, DBCC PAGE, older backups, and other sources of information to reconstruct what they could.

One of the interesting parts of the story was discovering that a nonclustered index could still contain information that was missing from the corrupted clustered index. That meant even damaged parts of the database could potentially be useful during recovery.

Eventually, they were able to rebuild the tables and indexes and get the system back online. Reading this made me think about how different troubleshooting becomes when you’re no longer trying to find the best query plan or fix a blocking problem. When you’re dealing with serious corruption, you’re looking for anything that can help you recover the data.

Read Rob’s full story

Vlad Drumea: Two Weeks of Ransomware Recovery

Vlad Drumea shared probably one of the biggest incidents in this collection. His organization was hit by Ryuk ransomware in 2020. More than 60 SQL Server instances across more than 30 VMs were involved. Recovery took more than two weeks. This wasn’t simply a matter of restoring a database.

The environment itself had to be treated as compromised. Vlad described rebuilding VMs, recreating SQL Server directories, restoring system databases, restoring user databases, dealing with reinfection, fixing broken LSN chains, and rebuilding a VM from scratch. There was also a lot of automation involved using PowerShell, T-SQL, and dbatools. What really stuck with me was the human side of this story.

Recovery involved 16-hour workdays for more than two weeks, and Vlad talks about the burnout that followed. We spend a lot of time talking about backups, DR, security, and automation. Those things matter. But there are also people sitting in front of those computers at 2 AM trying to get a business back online. That’s part of the story too.

Read Vlad’s full story

Jeff Taylor: SQL Server on Fire, Literally!

Jeff Taylor shared two stories involving infrastructure problems. The first started with a power outage in an office that had effectively become a small data center. The servers had battery backup. The air conditioning didn’t. As the room heated up, the team started using fans and eventually began shutting servers down to keep the hardware from being damaged.

The temperature eventually approached 120°F. That incident resulted in a much larger infrastructure redesign, including better cooling, battery backup for the cooling, a generator, and fire suppression.

Then there was another incident involving a Dell server where Jeff was replacing memory and a drive. After powering it back on, he saw a flash. Then smoke. The problem apparently involved an iSCSI cable that had been damaged during the earlier heat incident and eventually shorted when the equipment was moved.

It’s a good reminder that SQL Server doesn’t operate in a vacuum. Power, cooling, storage, networking, and the physical environment are all part of keeping a database available.

Read Jeff’s full story

Thomas Rushton: When the Server Room Gets Too Hot

The Lone DBA shared a story about a server room in an old Victorian mill building that overheated after the air conditioning failed during a hot summer weekend.

The servers were shut down, and after things cooled down, most of them came back online without any obvious problems. One server, however, kept crashing intermittently. They patched it, updated drivers, replaced the memory, HBAs, CPUs, and even the storage, but nothing fixed the problem.

Eventually, while replacing the motherboard, an engineer discovered that a daughterboard had partially melted during the overheating event, causing an intermittent short circuit. It’s a good reminder that a server room getting too hot isn’t just a temporary availability problem. It can cause physical hardware damage that may not show up until much later.

Read The Lone DBA’s full story

M G: A Comment That Deserved to Be Part of the Roundup

A person who goes by their initial M G don’t have a blog, but left a detailed comment on my invitation. I thought the story was too interesting to leave out.

The incident involved SQL Server 2016 and a SharePoint database with around 15 million rows and more than 850 GB of binary data. Three databases were repaired, but the fourth became the real problem. There were hundreds of suspect pages, a corrupted clustered index that was also the primary key, and a DBCC CHECKTABLE ... REPAIR operation that had been running for weeks and failing.

What caught my attention was the eventual workaround. Changing the database’s PAGE_VERIFY setting from CHECKSUM to OFF allowed the primary key to be dropped. That exposed another problem: SharePoint had accumulated roughly 90,000 duplicate elements. This is exactly the kind of kind of troubleshooting story that is difficult to forget because there isn’t necessarily a clean checklist that tells you what to do next. You investigate. You try something. You learn something new. Then you try again. And sometimes the solution comes from a place you weren’t expecting.

What I Took Away From These Stories

After reading through all of these, I noticed something. The actual cause of the outage was often not SQL Server itself. It was the environment around SQL Server.

A floppy disk. A SAN administrator deleting a LUN. An identity value reaching its limit. A bad disk controller. Ransomware. A lack of cooling. Corruption inside a SharePoint database.

These are very different problems, but they have something in common.

You don’t always know what the outage is going to look like until you’re already in it.

That’s probably why these stories are useful. You can study SQL Server performance. You can learn backup and restore. You can learn Availability Groups. You can learn PowerShell and dbatools. You can learn monitoring. But eventually, something unexpected is going to happen.

The best thing we can do is learn from people who have already been there.

And that’s what I really liked about this month’s T-SQL Tuesday. These weren’t polished success stories. They were stories about things going wrong.

And those are often the stories I remember the longest.

Thank you to everyone who took the time to participate in T-SQL Tuesday #202, whether you wrote a full post or shared your experience in the comments.

And thank you to Steve Jones for giving me the opportunity to host this month’s T-SQL Tuesday.

Until the next outage…

No, God forbids. It ould be yours, and hopefully the stories above give you the resolution route.

Disclaimer: The following is built with Claude Code. Just need to justify my $100/mo subscription cost. This was meant for a private demo so I can learn more about parameter sniffing and memory grant, but I decided to open it to public. And you may say “Marlon, this isn’t you, this is too good.” Again, I will say it again, originally I was to keep this a private learning doc for me to understand Parameter sniffing. I asked Claude Code to build me a test for param sniffing and memory grant feedback. The following is the result of that after few prompts. You may use this for your own demo or POC. Run at your own risk. The Github Repo info is at the bottom of this post.

The stored procedure runs in milliseconds. It has been running in milliseconds for the last two years. Then one morning, it suddenly takes four minutes to complete. No code deployment. No configuration change. Nothing obvious changed. Restart the SQL Server instance, and it goes back to being fast — at least until later in the day.

You already know what the first response in the incident channel will be: “It’s parameter sniffing.”

And technically, that answer may be correct. But it does not tell you enough to fix the problem.

Parameter sniffing is not a single issue. There are different ways this can bite you, but two of the most common ones are easy to confuse.

The first is a bad plan choice. SQL Server compiles a plan based on one set of parameter values, but that same plan performs poorly when reused for a different set of values. For example, SQL Server may choose an index seek with hundreds of thousands of key lookups when a scan would have been the better option.

The second is a bad memory grant. SQL Server estimates it only needs enough memory for a small number of rows, but the actual query returns hundreds of thousands of rows. The result can be spills to tempdb, poor performance, and unnecessary memory pressure.

These two problems can look similar from the outside, but they require different troubleshooting approaches.

This is especially important with newer SQL Server features like Memory Grant Feedback. It can help correct inaccurate memory grants, but it does not change a fundamentally bad plan choice. If you do not identify which problem you actually have, you can apply a fix that was never designed to solve the issue.

The goal of this post is to separate these two behaviors, show how they are different, and walk through a demo you can reproduce yourself.

Sniffing is a feature

Before getting into the problem, it is important to set the right context. A lot of discussions around parameter sniffing make it sound like a SQL Server defect that Microsoft should have fixed. That is not really the case.

When SQL Server compiles a parameterized query, it uses the parameter values that caused the compilation to estimate the number of rows and build a plan. It looks at the statistics, checks the histogram, estimates the expected cardinality, and creates a plan based on that information. That plan is then stored in cache and reused for future executions, even if those executions use very different parameter values.

The first part of this process is actually what makes SQL Server perform well. Without parameter sniffing, SQL Server would have to create plans based on generic estimates instead of the actual values being searched. You would end up with a plan that is average for everyone instead of a plan that is optimized for the majority of cases.

The problem is not parameter sniffing itself. The problem is reusing a plan when the data distribution does not match the values that plan was originally optimized for.

This is where data skew comes into play. If values in a column are evenly distributed, most parameter values will produce similar row counts, and the cached plan will usually work well. Parameter sniffing only becomes a problem when some values return a small number of rows while others return a significantly different number of rows.

So the first question should not be, “How do I disable parameter sniffing?”

The better question is, “How is the data distributed, and how much skew exists in this column?”

SELECT   TOP (20) CustomerID, Rows = COUNT_BIG(*)
FROM     Sales.OrderLines_or_whatever
GROUP BY CustomerID
ORDER BY Rows DESC;

SELECT   TOP (20) CustomerID, Rows = COUNT_BIG(*)
FROM     Sales.OrderLines_or_whatever
GROUP BY CustomerID
ORDER BY Rows DESC;

If the difference between the highest and lowest values is only within an order of magnitude, then data skew is probably not your problem. There is likely something else causing the bad plan choice.

Building a demo; thanks, claude

I wanted to demonstrate both failure modes using actual query behavior, which means I needed data with a noticeable skew. The challenge is that standard sample databases are not always useful for demonstrating these types of problems.

For this test, I used Claude Code to help create the setup script and build the test scenario. WideWorldImporters, Microsoft’s sample database, was used as the starting point because it contains real order data. However, the data distribution is fairly uniform. Order lines are spread across customers, dates, and stock items. That makes sense for a sample database, but it does not create the conditions needed to demonstrate parameter sensitivity.

The skew in this demo is intentional and documented. The script creates Demo.OrderLinesSkewed using WideWorldImporters’ approximately 231,000 existing order lines, then adds another 500,000 rows tied to a single customer. The goal is to create a simple scenario where one customer has a large percentage of the data while most other customers have significantly fewer rows.

The customer values are not hardcoded. The script identifies the large customer and the smaller customers from the data and prints them out. This makes the demo more portable because WideWorldImporters installations may not all contain identical data.

I prefer demos that are transparent about how the test conditions were created. The important part is not the test data itself. The important part is understanding the SQL Server behavior we are trying to demonstrate.

There are three important details in the setup that make this demo work. Each one represents a common reason why performance demos like this can produce misleading results:

The index is narrow on purpose. A single non-clustered index on CustomerID, no INCLUDE columns:

CREATE NONCLUSTERED INDEX IX_OrderLinesSkewed_CustomerID
    ON Demo.OrderLinesSkewed (CustomerID);

CREATE NONCLUSTERED INDEX IX_OrderLinesSkewed_CustomerID
    ON Demo.OrderLinesSkewed (CustomerID);

TThe goal is to force SQL Server to make a real choice. It can either use the index and perform key lookups for each row, or decide that scanning the clustered index is the better option. Where SQL Server draws that line is the plan shape side of the problem.

If the index is covering, that decision goes away. You may still see a memory grant issue, but you will not see the plan change between executions. The result is a demo that only shows one side of the problem and misses how parameter sensitivity can affect plan selection.

The rows are intentionally wide. There is a char(200) filler column, and the stored procedure includes that column in the output. Memory grants are calculated using estimated rows multiplied by estimated row width. If the rows are too narrow, the memory grant behavior is not very interesting.

There is also no TOP and no ROW_NUMBER() in this demo. This is an important detail because many demos around this topic accidentally hide the memory grant problem.

For example, a query like SELECT TOP (50) ... ORDER BY UnitPrice DESC introduces a Top N Sort. The memory grant for a Top N Sort is based on the number of rows being returned, in this case 50, instead of the total number of rows flowing through the sort. Filtering a ROW_NUMBER() value against a constant can have a similar issue because the optimizer may rewrite it into a Top.

In both cases, the memory grant no longer scales with the actual workload. The demo may still run, but it is no longer showing the behavior we are trying to analyze.

If you build your own version of this test, check the execution plan and make sure the operator is a Sort and not a Top N Sort. The demo scripts capture the plan after each execution and flag Top N Sort because it changes the behavior being tested.

Here’s the procedure. It is deliberately boring:

CREATE OR ALTER PROCEDURE Demo.usp_CustomerLinesByPrice
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT  ol.OrderLineID, ol.OrderID, ol.CustomerID, ol.StockItemID,
            ol.Description, ol.Quantity, ol.UnitPrice, ol.OrderDate,
            ol.Filler
    FROM    Demo.OrderLinesSkewed AS ol
    WHERE   ol.CustomerID = @CustomerID
    ORDER BY ol.UnitPrice DESC, ol.Description;
END

CREATE OR ALTER PROCEDURE Demo.usp_CustomerLinesByPrice
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT  ol.OrderLineID, ol.OrderID, ol.CustomerID, ol.StockItemID,
            ol.Description, ol.Quantity, ol.UnitPrice, ol.OrderDate,
            ol.Filler
    FROM    Demo.OrderLinesSkewed AS ol
    WHERE   ol.CustomerID = @CustomerID
    ORDER BY ol.UnitPrice DESC, ol.Description;
END

The test is simple by design. One equality predicate against a skewed column. One sort operation where no index can fully support it. Those two things are enough to reproduce the behavior we want to analyze.

Failure mode one: sniff small, run big

Compile the procedure for the minnow. Then call it for the whale.

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- suffers here

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- suffers here

The plan compiled for the minnow is a good plan for the minnow: seek the non-clustered index, look up the handful of matching rows in the clustered index, sort them in a memory grant barely above the minimum. For a few hundred rows that’s exactly right.

Then the whale arrives, and the same plan does it 500000 times.

Two separate things have now gone wrong, and from here on I’m going to insist on naming them separately.

The plan shape is wrong. Key lookups are fine in the hundreds and catastrophic in the hundreds of thousands. The logical read count tells the story: 1519167 reads to return 500000 rows. A clustered index scan would have read the table roughly once.

The memory grant is wrong, and this is the part people find surprising. The grant is not recalculated per execution. It is baked into the cached plan at compile time, computed from the estimated row count and the estimated row width. Runtime reality does not get a vote. So the sort gets a grant sized for the minnow — 1 MB — while the engine’s own after-the-fact assessment of what it should have had is 0.53 MB.

When a sort doesn’t have enough memory, it spills to tempdb. Not a warning, not a retry — it writes sort runs to disk and merges them, and your query goes from memory-speed to disk-speed while holding its locks the whole time. In the actual execution plan you’ll see a warning triangle on the Sort operator. In the Extended Events output you’ll see sort_warning fire.

The gap between GrantMB and IdealMB is the fingerprint. Learn to read it.

Failure mode two: sniff big, run small

Now the mirror image, which most write-ups skip, and which is the more interesting half.

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- wastes memory here

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- wastes memory here

The plan compiled for the whale is a clustered index scan with a memory grant sized for half a million wide rows. Reused for the minnow, it returns a few hundred rows and finishes quickly.

Nothing spills. Nothing is slow. This query will never appear in your “top ten by duration” report. It is not broken in any way a duration-based monitor can see.

It is, however, greedy. It asked for 127.31 MB of workspace memory and touched 0.22 MB of it.

Here’s why you should care about memory a query didn’t use:

• A memory grant is reserved for the lifetime of the query, used or not. It is not lazily allocated and it is not shared.
• Workspace memory is a finite, instance-wide pool. There is only so much of it.
• When the pool is exhausted, incoming queries queue on RESOURCE_SEMAPHORE waits — they sit there, having compiled successfully, waiting for permission to start.

So one procedure with a badly sniffed grant, called from enough sessions concurrently, will stall queries that have nothing to do with it. The victim is never the culprit. That’s what makes this one hard to trace back, and it’s why duration is a bad detector for half of all parameter sniffing problems.

The detector that does work is a comparison, not a threshold:

SELECT  qs.execution_count,
        GrantMB = qs.last_grant_kb      / 1024.0,
        UsedMB  = qs.last_used_grant_kb / 1024.0,
        IdealMB = qs.last_ideal_grant_kb/ 1024.0,
        st.text
FROM    sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
WHERE   qs.last_grant_kb > 1024
  AND   qs.last_grant_kb > qs.last_used_grant_kb * 2
ORDER BY qs.last_grant_kb DESC;

SELECT  qs.execution_count,
        GrantMB = qs.last_grant_kb      / 1024.0,
        UsedMB  = qs.last_used_grant_kb / 1024.0,
        IdealMB = qs.last_ideal_grant_kb/ 1024.0,
        st.text
FROM    sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
WHERE   qs.last_grant_kb > 1024
  AND   qs.last_grant_kb > qs.last_used_grant_kb * 2
ORDER BY qs.last_grant_kb DESC;

Granted far above used means memory reserved and wasted. Ideal far above granted means the query spilled. Same three columns, two different diseases.

Two problems, not one

This is the pivot of the whole post, so here it is in one table:

Everything below refers back to this.

OPTION (RECOMPILE)

The blunt instrument, and the one that always works.

    ...
    ORDER BY ol.UnitPrice DESC, ol.Description
    OPTION (RECOMPILE);

    ...
    ORDER BY ol.UnitPrice DESC, ol.Description
    OPTION (RECOMPILE);

Both columns of that table go green. The plan is built against the actual parameter value on every single execution, so the shape is right and the grant is right, every time, by construction. You get a bonus, too: because the value is a literal constant at compile time, the optimizer can do things it can’t do for a cached plan — fold constants, eliminate whole branches of a query, simplify predicates it would otherwise have to keep general.

In the demo, scenario D calls the recompiling version with the minnow, then the whale, then the minnow again, and the grant tracks the actual row count in both directions: 1, 127.35, 127.34 MB.

Now the cost, stated honestly, because “just add RECOMPILE” is bad advice delivered confidently.

You pay a compilation on every execution. Compilation is CPU-expensive. On a procedure called four thousand times a minute, you have traded an intermittent memory problem for a permanent CPU problem, and the second one is harder to notice because it doesn’t spike — it just raises your floor.

You lose the plan cache as a diagnostic surface. After a recompiling statement runs, there’s nothing in sys.dm_exec_cached_plans to look at. Your monitoring gets thinner exactly where you were having trouble.

Statement-level, not procedure-level. OPTION (RECOMPILE) on one statement recompiles that statement. CREATE PROCEDURE ... WITH RECOMPILE recompiles every statement in the procedure on every call, including the nine that were fine. It is almost never what you want. If you inherited a procedure with WITH RECOMPILE in the header, that’s usually someone’s decade-old shotgun fix for a problem in one statement.

The rule of thumb worth internalizing: RECOMPILE is priced per execution. A reporting procedure called forty times an hour should probably just use it and stop thinking about this. A hot OLTP path called forty times a second should not.

And note something for later: after scenario D there is no cached plan at all. Hold that thought.

Memory grant feedback

Now the feature everyone wants to talk about, and the one whose limits are routinely oversold.

Memory grant feedback is a learning loop. After a query executes, the engine compares the memory it granted against the memory the query actually used. If the query spilled, the grant was too small — write a bigger number onto the cached plan for next time. If the grant was more than twice what was used, it was too big — write a smaller one. Over a handful of executions the grant converges on reality without anybody recompiling anything.

What you need for it:

That compatibility level column is where most people get stuck. A database restored from an older instance keeps its old compatibility level forever; WideWorldImporters ships at 130. You can be running SQL Server 2022 and getting none of this.

Scenario C in the demo sets the loop up to succeed: compile for the minnow, then call the whale six times in a row with nothing recompiling in between. That last part matters. Feedback is written onto the cached plan, so anything that evicts the plan throws away everything the engine learned. This is also why the adjustment always lands on the following execution — execution n discovers the grant was wrong, execution n+1 benefits.

The trajectory:

That State column is IsMemoryGrantFeedbackAdjusted from the cached plan’s XML, and it’s the cleanest way to watch the loop work: it moves from NoFirstExecution through YesAdjusting to YesStable.

Now the three caveats, which are the actual reason this section exists.

It fixes the grant. It never fixes the plan shape. Look at the PlanShape column across all six of those executions in the demo output. It does not change. It cannot change — memory grant feedback adjusts a number attached to an existing plan; it does not trigger a recompilation and it has no opinion about operators. Those 1519167 logical reads from failure mode one are still there on execution six. The query stops spilling and gets faster. It does not get good. If you go into this expecting feedback to solve parameter sniffing, this is where you’ll be disappointed, and it won’t be the feature’s fault.

It’s a learning loop, so somebody has to do the learning. The first caller always eats the bad grant. On SQL Server 2019 and earlier, so does the first caller after any cache eviction — a plan flush, memory pressure, a stats update, a failover. SQL Server 2022’s Query Store persistence exists precisely to stop throwing that lesson away, and it’s a good reason to have Query Store on.

It gives up if you make it thrash. A workload that genuinely alternates between tiny and enormous will push the grant up, then down, then up again. Rather than oscillate forever, the engine notices the instability and switches feedback off for that query. There’s an Extended Event for it — memory_grant_feedback_loop_disabled. Percentile grant feedback in SQL Server 2022 is the answer to this case: instead of chasing the last execution, it sizes the grant from a percentile of recent executions, which is far more stable across a genuinely bimodal workload.

Why RECOMPILE and memory grant feedback don’t combine

This falls straight out of the two sections above, and it’s the question that sent me down this path in the first place.

Memory grant feedback writes its correction onto a cached plan. OPTION (RECOMPILE) doesn’t leave a cached plan. There is nothing for the feedback to attach to.

You can watch this in the demo: after scenario D runs three times, query the cached plan view and you get nothing back. Compare with scenario C, where the plan is sitting right there accumulating adjustments.

This is not a conflict you need to resolve, and it isn’t a bug. RECOMPILE already produces an accurate grant on every execution by construction — there’s nothing left for a feedback loop to improve. But it does mean the two are alternatives, not layers. Don’t reach for RECOMPILE while imagining that feedback is also working quietly underneath, and don’t diagnose the absence of feedback on a recompiling statement as something being broken.

Parameter Sensitive Plan optimization

Which leaves the gap in that table from earlier: memory grant feedback never fixes plan shape, and RECOMPILE fixes plan shape but charges you per execution. Is there anything that fixes the shape without the compile?

On SQL Server 2022, yes. Parameter Sensitive Plan optimization caches multiple plan variants for a single statement and dispatches between them based on the cardinality the incoming parameter implies. The minnow gets the seek-and-lookup plan, the whale gets the scan, neither one triggers a compilation, and both come out of cache.

It’s on by default at compatibility level 160. Its limits are worth knowing: equality predicates only, at most three of them, and the column has to be skewed enough for the engine to consider it worth the trouble — PSPO is not applied to every parameterized query, only to ones where the optimizer sees a genuine sensitivity.

The most convincing thing I can say about PSPO is not an argument, it’s a confession about the demo: the setup script has to turn PSPO off. On a 2022 instance at compatibility level 160, scenarios A and B don’t fail. The engine handles them. I had to explicitly disable the feature to show you the classic behaviour at all:

ALTER DATABASE SCOPED CONFIGURATION
    SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF;

ALTER DATABASE SCOPED CONFIGURATION
    SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF;

Scenario E turns it back on so you can see the contrast: same procedure, same two parameters, no recompilation between them, two different plan shapes.

Things that look like fixes

A tour of the mitigations you’ll find in older Stack Overflow answers, and what they actually cost.

OPTIMIZE FOR UNKNOWN. Compiles using the density vector — the average rows per distinct value — instead of the histogram. You’ve traded a plan that is excellent for most callers and terrible for a few, for a plan that is mediocre for everyone. Sometimes that really is the right trade, especially when the terrible case is bad enough to cause an outage. But it should be a decision, not a reflex.

OPTIMIZE FOR (@CustomerID = 12345). You’ve pinned the plan to a magic constant. It works until the data distribution moves, at which point it fails silently and nobody remembers that number is in there.

Assigning parameters to local variables. The folk-remedy version of OPTIMIZE FOR UNKNOWN — the optimizer can’t sniff a local variable, so you get the density estimate. Same trade-off, but now it’s invisible, and the next developer will “clean up” the pointless variable assignment and reintroduce the bug.

Updating statistics, or rebuilding the index. This evicts plans, so the symptom goes away, so it looks like a fix. It will look like a fix again next week, and the week after, forever. This is the single most common way a parameter sniffing problem survives for years: it is permanently one maintenance job away from being invisible.

Restarting SQL Server. Same mechanism, more downtime, and it usually happens at 3am while someone types “resolved” into a ticket.

Splitting the procedure into branches — an IF that routes big customers to one procedure and small ones to another, so each gets its own plan. This one genuinely works. It also hardcodes today’s understanding of your data into control flow, and it ages badly. Reach for it when PSPO isn’t available and RECOMPILE is too expensive, and leave a comment explaining why.

So what should you actually do?

1. Confirm the column is skewed. Group by the predicate column, compare the top and bottom. Within an order of magnitude? It isn’t parameter sniffing. Go look somewhere else.
2. Work out which problem you have. Compare last_grant_kb, last_used_grant_kb, and last_ideal_grant_kb against last_logical_reads. Wrong grant, wrong shape, or both.
3. Grant only, on 2019 or later, with steady traffic — check your compatibility level is 150+ and let memory grant feedback handle it. Turn on Query Store if you’re on 2022, so the lesson survives an eviction.
4. Shape wrong, on 2022 — check whether PSPO is on before you write any code. You may not have a problem.
5. Shape wrong, low call rate — OPTION (RECOMPILE) on the statement. Measure the compile cost afterwards rather than assuming it’s fine.
6. Shape wrong, high call rate, no PSPO available — branch the procedure, and write down why in a comment, because in three years the reason will not be obvious.

The thing I’d most like you to take away is the second step. Almost everything written about parameter sniffing collapses the two failure modes into one story, and once you’ve separated them the modern features stop looking mysterious. Memory grant feedback isn’t under-delivering — it’s doing exactly the one job it claims to do, and PSPO is the feature that does the other one.

Run it yourself

Download the sql files from my repo (see link below):

01-setup.sql     run once, builds the skewed table and the procedure
02-demo.sql      run the whole file, all five scenarios, records its own evidence
03-cleanup.sql   restores everything it changed

Two things before you do. Turn on Query Options → Results → Grid → Discard results after execution in SSMS, because the procedure returns half a million wide rows about ten times over and you want to be timing the server rather than the grid. And know that 01-setup.sql raises your database compatibility level and turns PSPO off — both are recorded before they’re changed and restored by the cleanup script, but point it at a scratch instance, not production.

The demo captures every measurement into Demo.DemoResults as it goes, so you don’t have to sit and read execution plans between executions. The summary at the bottom flags spills and wasted grants for you.

Dowload the demo scripts from my github repo.