Remove the list view threshold (5000 by default)
This limit has always been a bit laughable, and is even more so as we develop more client side applications. In SharePoint 2007 we didn't have this limit and were allowed to make our own mistakes. Now that hardware is so much more powerful, we need this limit removed so that we can build enterprise-class applications.
The 5,000 item threshold is likely here to stay. We are aware that it takes awareness on both sides to get past this limitation: list owners have to build the right fields, indices, and indexed views so that end user queries can run successfully in large lists. And then, end users have to run the correct queries – they can’t just open the root of the list, they have to use a filtered, indexed view.
We want to pursue work that helps on both these ends – for example, proactively indexing more lists, making suggestions to list owners for views to create that will help end users, and helping end users at query time pick filters that will unblock their queries. Additionally, we want to increase our support for running more types of queries in large lists when an index is present – for example, instead of requiring an indexed filter for every query, maybe an indexed could suffice.
No guarantees or timelines yet, but that’s the way we’re thinking about the problem. Does that sound reasonable?
Jim K commented
Microsoft seems to be positioning sharepoint online as a google drive, box, dropbox competitor - i.e. a file server replacement. None of these other services have this ridiculous restriction. It's a deal killer for using sharepoint online as a file server replacement. Which, in turn, makes us want to look at other solutions such as g suite.
The response by the SharePoint Experiences Team seems to reflect an upside down view of the world. Technology is subservient to people, not the opposite. Asking list owners and users to work around, or against, limits not justified by current technology is not reasonable, and neither is expecting users to think like developers.
Balu Dharmarajan commented
This limit is quite low, I already checked the dll codes and sql queries and it looks like not too complicated to fix, please let me know I will tell you how it can be fixed.
At the Virtual SharePoint Summit, on a slide titled 'Transform business process roadmap', MS announced 'By year end: intelligent indexing for large lists'. Please MS don't forget about any column type: https://ktnnsharepoint.wordpress.com/2016/10/22/column-types-list-view-threshold/
Pramod Nathan commented
Remove List view threshold limit of 5000 by taking advantage of paging
The current list view threshold limit of 5000 with it's workarounds are very frustrating. We can all appreciate that we do not want to fetch too many items from the table (the underlying cause of the restriction). BUT paging exists and Microsoft needs to take better advantage of this:
1. The SharePoint interface is page based. Instead of breaking (which is what the solution does with a warning), MS can change the underlying call so it breaks down the call to SQL to fetch data as needed per page/set of pages. When the user navigates to the next page/set of pages - call SQL for the next set of data (e.g. if I am on the current view with 5000+items, I really don't care about about item 5000+1 immediately. BUT I do care that I can manipulate the entire set of items and that I can get to item 5001). So just fetch the first few page worth of items.
2. For sorting and filtering, the call can go to SQL on click to fetch the top X number of items depending on the filter or sort. (e.g. if the page has 5000+items and we want to sort by a column, it can call a SQL statement that does this but sends back only the top 3-4 pages or how many ever are feasible)
3. It will also make the underlying APIs very robust in terms of being able to create multiple calls of data to generate a large data sets that are stored in SharePoint lists and libraries.
the 5,000 is pushing us to pursue other solutions outside SharePoint.
We created a list to hold user's stats; and for stats purposes it is worthless to retrieve chunks of data when you want to know how many selected a, b, or c, or group them by hour.
As a workaround, we connected the list to PowerBI. From there we created a dashboard which actually can work with all the data at the same time. Until now, this workaround worked just fine, but we just got a message saying that dashboard sharing will no longer be part of PowerBI free.
We just want SharePoint to work and not to start thinking on workarounds or 3rd party applications to make it effective and efficient.
Steven Derveaux commented
And another customer bites to dust...
Keep in mind that the search box in document libraries (in the modern experience) is also struggling with this limit. Working with folders can overrule the 5000 items, but the new-autocompleted-search cannot cross this limit. MS Support pointed us to use the Search Center or create our own webparts. Meanless to say: you cannot replace this autocompletion search webpart because you cannot customize the modern view! Aargh!
So what can you do? switch back to Classic Design but then you loose the new copy functionality!
I have no words for this...
This limit of 5000 thousands is preventing us of creating many solutions in Sharepoint.
SharePoint is heading backwards in so many ways. As another contributor said, it's driving business away from SPO - but then, maybe that's what Microsoft want. SharePoint will become a simple (and small) file store). Lots of publicity re huge amounts of storage space, millions of items in lists etc. - except it's effectively unusable.
What makes this limit really unusable is that you cannot create / amend indexed fields. How useless is that!
Or am I being cynical?
The 5000 item threshold is ridiculous when Microsoft's own guidelines say a library can hold 30 million items. How in the world is anyone supposed to come up with indexes and filters to split that into 5000 item chunks and have it be anything but worthless? It's already difficult enough when people have barely more than 5000 and a view using Groupings, and suddenly it stops working after a migration.
Juan Pineiro commented
This has been a major issue for us and driving business off of SPO. It's not just about awareness, indexes or filters. I understand nobody need 5k+ items sent to a page, but after 5k, many setting features no longer work. We have a list with 25k items and you can't add new indexes or change calculated columns on it. Is that a bug or a protective feature?
Arvinda Sudan commented
We are really disappointed with the 5000 limitation as we are online users. Hope Microsoft will find a good solution for this in the future. We also appreciate if someone can suggest us a third party solution to overcome this 5000 limitation.
Russell Bain commented
Admin - "The 5,000 item threshold is likely here to stay. We are aware that it takes awareness on both sides to get past this limitation".
Awareness is not the solution - it is unrealistic to expect the average SharePoint user to develop the technical skills required to operate under this sort of limitation. Even those with strong technical ability are often unable to mange such lists and libraries effectively because finding the right approach often depends on identifying usage patterns and business needs which may not yet be established, or may change significantly over time. Besides, in many cases there is no approach which meets all of the critical business needs effectively.
There are no end of other software products which can serve up views with more than 5,000 rows. If there is technical issue preventing this in SharePoint Online, then it needs to be addressed.
Andrew Armanini commented
This is a show stopper for even a small office (around 50 users). If you read the "how to deal with large lists" article, it's a catch-22. The only way to get large data in is via creating a list by uploading an Excel spreadsheet. That method only allows the creation of a new list with custom columns. You can't retroactively add indicies and searches to custom columns. So you have an unusable list. And this is without thinking about a document library which excel can't deal with.
This requires the use of third party tools with subscriptions in the thousands of dollars a month, because there's no inbuilt way to migrate list items.
SharePoint Online as it stands is fundamentally broken. You can have 30 million items, as long as there's no more than 5000. SharePoint can do amazing things, except if you actually try to use it.
We at the very least need to be able to retroactively add indexes and switch a custom column to an inbuilt indexable column. It's too easy for a client to work their way into the threshold 5000 trap - with no way out of it.
Martin Coupal commented
Why not enhance and leverage the search engin? You should provide a way to create list/library views based on search and scoped to the list where the results would be presented in a similar layout than a list view (ECB menu etc.).There should be a way to provide real-time search results by using a kind of event driven indexing. This would probably be easier than change how sharepoint stores list data in SQL.
Clients who have worked on prem in a certain way multiple sorts and filters no matter how many indexed columns you have this will not work in o365 with the limits in place , this in turn results in clients looking at other solutions
It's 2017. 5,000 item threshold is just insane. Right now, I can't even setup a workflow to do a lookup against a list with 5,000+ items.
Also, if this is a limit in SQL, maybe MS needs to think about using Oracle for storing their back-end Database.
As a consultant, I absolutely cannot recommend O365 as an Enterprise Application for Document Management due to the 5000 item limit. It is ridiculous. Try going to the Business and telling them that they have to come up with Metadata (which they do not have) just to make the tool work.
SharePoint stores it's content in a SQL database. The 5000 items limit in a document library makes no sense especially when there is an item limit setting in the view, which btw is ok if it's lower than 5000. Many of my customers have up to 20000 documents in a single document library and wants to find duplicates which requires the use of "Show all items without folders" in views. A view without folders but with a item limit of lets say 1000 would work just fine. But it doesn't.