Organizations using complex file structures for document management - common?

In My Humble Opinion
1

I have been working at my current employer for a year, and I continue to be baffled at how complicated and… unevolved our systems are for what feels like basic document management.

A large part of my job involves completing documents and processes for other employees and outside vendors - generating contracts, filing invoices, reconciling expenses, making purchases, that sort of thing.

Let’s say I have tasks A, B, and C. If I’m doing paperwork for task A, I must keep all versions and associated artifacts in a file location seven levels deep. The locations do an ok job of being self-explanatory, but often are not. At some of these levels there are 20-30 file folders to read and select from. Sometimes there are identical folders in different locations that different folks have used to file the same paperwork at different times.

Now I’m doing paperwork for task B. This is maybe 6 levels deep on a different shared drive.

Now I need to collect a signature from a supervisor. Maybe there’s a folder two levels up appended with characters to make it appear “at the top” alphabetically, like “AA Get Signatures Here”. I have to copy my document there. Maybe someone is ‘monitoring’ that location, or maybe not- I probably have to email the person to let them know “I put a document in folder Q>R>S>T>‘AA Get Signatures Here’ for you to sign” if I want to be sure it will be seen. Then they will sign it, change the file name and leave it there for me to discover. I’ll then delete it from that folder and copy it back into the original folder.

Document management is tough. But this just feels like chaos. Again, if I need signatures from a supervisor for documents supporting process A, B, and C (and there are more than three), I am supposed to copy those documents into unique folders for each process, when I am just looking for the same thing from the same person on each of them.

There are “process and training” documents scattered throughout this file tree. Some are in folders titled “training” or “AA guides”, but others just in the folder where the documents for that process go. Need to learn how to do something… take a guess as to where it might be, or whether or not the thing you happened to find is the most recent procedure, or just something out of date that someone wrote three years ago that no one has changed or deleted.

Want to know all of the work associated with a specific employee (Like, show me the paper trail for all processes on behalf of employee X)? The only way would be to know every possible process and its associated location, and go digging in each of those places (and go with the assumption that it was the only location where a document for that process might be filed), and compile everything you find. It will be incomplete.

Is this common? Do others have a similar experience? Are my expectations that it ought to happen a different way unreasonable? We (other employees doing the same job) are constantly in a state of overwhelm and confusion, and at least part of it I attribute to the chaotic and disparate document management processes.

2

Paging @msmith537 to aisle 3 for a clean-up. :wink:

3

I don’t see any method at all.

4

The best I ever did was to manage the documents that were in my immediate control. I had a team of engineers and project managers who were responsible for generating documents for construction contracts. Everyone had their own ideas about file names. I had to develop a file naming convention that was sensible and consistent, and then force everyone to use that convention. It took a while to get everyone up to speed, but it made my life so much easier when it came time to sent projects out to bid. One of the most helpful parts of the convention was including a date, since may documents were frequently revised.

Signatures were always a clusterfuck. Some people who needed to sign off on plans were not in our organization, so emailing the documents for signatures was a task for the project managers.

5

Have you looked into getting a document management system?

6

I think so. For administrative documents we have a very similar system here as to what you have. Every few years someone comes through and says, “this must be simplified, from now on all AA documents go here”. Now AA documents are in their original place, and in the new place. Repeat for BB-ZZ documents…

We did successfully manage to wrangle things on the research cluster. A few things made this possible. First, we just did it from on high, and people had to deal with our changes. Second, graduate students and postdocs come and go, so the “culture” of storage is easy to change. Third, in many cases we’re contractually obligated to handle data in certain ways, and we could use that as a bludgeon to force people into a sane system.

7

Heh. We have sharepoint, but that ends up being just one other repository that some teams use on top of / in addition to the file structure.

And really, a big issue is that there is no project manager that actually owns any processes. So someone might come up with an idea “hey, let’s use sharepoint for collecting these reporting documents, so we don’t have to email them when we need signatures, and they’ll always be right there.” However, folks are still expecting them in their appointed location in the file tree, so it means that there are now two sources of truth: a location in sharepoint that is not universally utilized but also some place we need to pay attention to, and the original location. My impression is that at some point (as has happened in the past), someone will swoop in with the authority (but minimal depth of knowledge) and come up with a “new system” that no one will be trained on, with processes that have not been thoroughly described or thought out, and most people will keep doing things mostly the old way, and new employees will be endlessly confused about how to do their jobs, and most folks in management will have any idea how anything actually gets done.

I typed all that before you posted, @echoreply , but yes, that sounds exactly what is happening.

It also sounds like you are in the higher ed space, as am I. I’m wondering if this is something specific to the organizational sprawl of universities. My previous experience in the corporate world was not always perfect, but this is a whole different class of chaos.

8

The trick I think is to set up your Sharepoint front end to point at the old location at first, and at some point, just cut off access via file explorer, and only allow access via Sharepoint. Once that’s done, the file structure can be simplified/cleaned up behind the scenes.

Ideally though, a document/content management system would be the right solution for managing a lot of documents in an enterprise of any reasonable size. It lets you search, lets you set up intake forms, etc… and isn’t reliant on a folder structure for any particular file’s metadata. This is because the metadata is actually explicit- no more of the version of the file being determined by where in the file structure the document resides; it’s an explicit metadata value for each document. Same for project, owner, etc…

9

I think the big danger there is that you end up with two versions of the documents, with some people making changes in one place and other people making changes in the second place. And then who can reconcile the two versions? Ideally one location is just a pointer to the true location.

10

I still don’t entirely understand the relationship between OneDrive and SharePoint. It’s been explained to me, and I’ve read things, but they all seem wrong or outdated. OneDrive is file storage. SharePoint is a way to see those files on the web, except a file can’t be simultaneously on both, except when it can. And you have to copy from SharePoint to OneDrive to make a change, except not for other changes.

That is all prologue to we’ve started doing things in SharePoint, because people want a way to share documents that is not all of the other ways that they can share documents. The really big win for using SharePoint is that we can tie access to the Active Directory, so that everyone in the department automatically has access to the SharePoint site with their existing ID.

Big universities are so much more like collections of small to medium sized businesses, than large enterprises. Except when they’re also like large enterprises.

11

Oh sure. I have a doc I had to move around today. There are now:

Location 1: documentname.xls, documentname.pdf, documentname_sig.pdf, documentname_sigfinal.pdf
Location 2: documentname_sigfinal.pdf, documentname_sigfinal_approved.pdf
Location 3: documentname_sigfinal_approved.pdf

And then the document is upload to the official ‘submission portal’. I am supposed to retain a local copy in location 3. But I am also not supposed to delete “any” of the versions prior to final in Location 1 (though there is no definition around what or how many of those early version there should be. Just don’t delete any. Unless you do), nor am I supposed to delete any of the composite elements of the final document that itself is a collection of smaller docs which live forever in their own location, unsigned and unorganized.

But also, sometimes, if I ask someone else, I’ll get different instructions on what to save where. Best policy is ask different people until you get an answer that is most like what you want to do anyway, and stick with that.

12

So much. And I had no idea until I started working here. Dozens of little silos all communicating with each other through multiple systems of numeric codes and money exchange, each interaction sitting in a queue waiting to be dealt with in the order in which it was received.

13

Sounds like a nightmare.

14

And all of those pools of money governed by different rules depending on where it originated. US federal grant, state grant, private grant, tuition money, foundation money, endowment money, or something else that doesn’t cleanly fit in any category?

Depending on who paid for them, certain computers aren’t allowed to access certain documents. This is strictly followed to the letter in reporting and purchasing, and then usually ignored in practice. Fortunately rules around data security are much more likely to be complied with, usually because the rules actually make sense:

“Computers bought on grant A cannot be used to write up results on grant B” versus “highly confidential subject data on project A must not be copied off the secure storage facility”

Bringing it back around to the thread, you end up with stuff like some files may not be stored on OneDrive, some files must be stored on OneDrive. Some files must be deleted after a certain date, other files must be saved forever.

15

There’s a 255-character limit on filenames. THIS INCLUDES THE BLOODY FILE PATH!

There is nothing, NOTHING, I can do to change this this! STOP NAMING YOUR BLOODY DIRECTORIES “jacobson contract that we revised mid march without executive oversight on the tertiary thrombosic and cromulent refractors” Because they break everything, you nit!

16

I’ve had to deal with this sort of ridiculous filenames. A real pain when moving to a new file system when the path exceeds 255 characters.

17

We’ve had the reverse situation with file names where I work…we have a surprising number of documents so old that their name is no longer than 8 characters. So you end up with a situation where the part number is “1234-5678#A,” the file name is “12345678” or “TP123456,” and the document name is “TP-12345678A.” Sometimes the extra characters might sneak into the document name, even though our convention has been to exclude them, so you might find some other document like “ST-1234-5678_A.”
We also pretty much gave up on electronic document review once we all returned to the office, for many of the reasons already noted in this thread. Also, Acrobat struggles with massive documents (200+ pages) that have multiple page sizes.

18

19

Re: the XKCD standards cartoon

Yes, that is exactly what I was thinking about.

With multiple file locations sometimes I can get rid of old ones. Other times[1] the ideas repeat, and I can just reuse an old location that has been dormant for a few years.

  1. why is “sometimes” one word, but “other times”, two? ↩︎

20

I usually encounter situations like this only when the company or department is staffed by old-school sysadmins (or people part-time roleplaying sysadmins) who are used to a certain way of doing things and don’t want to do anything differently. “If it ain’t broke” is a perfectly cromulent approach to IT systems, I guess, if stability is more important than efficiency.

In the teams of younger people I’ve been a part of, nobody uses Microsoft anymore, specifically their online storage stuff. It’s all GDrive (especially Shared Drives), which is much easier to use and reason about. Every file or folder has a unique and easily shareable & bookmarkable URL (for both internal users and external collaborators), there is only a single source of truth (the cloud), and versioning is handled internal to the document itself. And that also makes it easier to tie in additional integrations to that storage, whether it’s a Google Form that saves responses to a certain place, an external service, a CRM, etc.

I’ve been in orgs that used a bunch of bespoke CRM or document management systems and others that just use plain old on-disk filesystems. The bespoke solutions tend to be a lot more efficient… for a few years. Then they inevitably get replaced when that admin leaves and somebody else wants to replace it with their own favorite solution. The filesystem solutions are universally hated, but at least consistently so over the years, by everyone. Not sure which is worse… :sweat_smile: