Rage Against the File System Standard 612
pwagland submitted a rant by Mosfet on file system standards. I think he's sort of over simplified the whole issue, and definitely wrongly assigned blame, but it definitely warrants discussion. Why does my /usr/bin need 1500 files in it?
Is it the fault of lazy distribution package management? Or is it irrelevant?
Still new to GNU/Linux (Score:2, Interesting)
I would think that even without a package handler to do it for me, the program itself would allow me to say where it should be installed...or is that just the Windows user in me talking?
he's pretty far off base (Score:5, Interesting)
It's not about lazy distributors. It's about administrators who are used to doing things this way and distributors going along with tradition.
Linux From Scratch (Score:4, Interesting)
Re:The Alternative? (Score:2, Interesting)
Re:The Alternative? (Score:3, Interesting)
Or you could do what DJB does with
Re:The Alternative? No Alternative! (Score:2, Interesting)
Use /usr/local for add-ons, keep /usr clean (Score:4, Interesting)
403
~> ls
36
~> ls
91
~> ls
220
~> ls
796
This is FreeBSD, which installs a relatively clean OS under
I like that much more, it is the old UNIX way to separate the essential OS from optional stuff. It really is a pity that most Linux distro's dump everything directly in
As for my slackware, I installed only the minimum, and roll my own packages for everything I consider not to be 'core Linux'; all these packages go under
Re:Package Management (Score:3, Interesting)
Jay
Translucent file system (Score:5, Interesting)
Even better would be if Linux had a translucent file system. Simply mount all the path directories on top of each other and let the OS do the rest.
For the uninitiated, a translucent file system lets you mount one filesystem on top of another filesystem, the idea being that if you tried to open a file the OS would first search the top filesystem, then the bottom one. In conjunction with non-root mounting of filesystems (e.g. in the Hurd) it removes the need for $PATH because you can just mount all the relevant directories on top of each other.
I wish unix had this... (Score:3, Interesting)
Maintance for applications like these whould be a no-brainer. Just move the directory and all the associated preference files and whatnot travel with the app.
-Steve
Related to yesterday's story (Score:5, Interesting)
I think the fundamental problem here is related to yesterday's story about new user interfaces [slashdot.org]. It's a problem of how and where storing our files. Regarding applicationsn, there are two ways to do it: you can store all files (binaries, config files, man pages, etc.) of the same application in the same directory, or you can store all files of the same type from different applications in their respective directories (all config files in /etc, man pages in /usr/share/man (I think), etc.).
Both approaches have their advantages. The problem with hierarchical file systems is that we have to choose one of them. I would love to see a storage system where we can use both ways _at the same time_. A system that groups file depending on relationships they have. Such that 'ls /etc' gives me all config files for all apps, and 'ls /usr/local/mutt' shows me all mutt-related files, including it's config file(s).
I have no idea how to implement such a beast. I'm thinking about a RDBMS with indices on 'filetype' and 'application', but I would love to see something much more flexible. All pictures should be accessible under ~/pictures and subdirectories, all files relating to my vacation last year in ~/summer2000. Files relating to both should be in ~/pictures/summer2000 _and_ ~/summer2000/pictures.
To a certain extent, this can be done via symlinks, but it should be much easier to deal with. You shouldn't have much manual work
RiscOS... (Score:4, Interesting)
There's also a unique shared modules directory in the System folder.
This system is at least 10 to 15 years old (not sure Arthur was as modulable, though) and sure proved to be an excellent way to deal with this problem...
I read this last night... (Score:2, Interesting)
I came away thinking "this man is insane".
He claims DOS had a better way of organizing applications. This is a red herring. I don't want to organize my applications. Ever. I want to organize my data. I don't remember many applications in DOS that were compatible with the same type of data. If there had been, the limitations of the DOS structure would have been readily made apparent. First, CD into the directory where your audio recording utility is and make a .wav file. Then, move the .wav file into the directory where your audio editing utility is and edit it. It works, but why not keep the data in one place and run programs on it as you see fit without regard for their location on your hard drive, and without having a 10-second seek through your PATH variable?
Besides which, DOS had c:\msdos50 (or whichever version you used). That was DOS's variation on /bin. Ever look in that directory and attempt to hand-reduce the number of binaries in it to save disk space? I did. A package management system would have made that doable.
You can have all the localized application directories you want in /usr/local. The point of /usr/local is to hold larger packages which are local to the system. (hmm... /usr/local/games/UnrealTournament, /usr/local/games/Quake3, /usr/local/games/Terminus, /usr/local/games/RT2...) And as a bonus, thanks to the miracle of symbolic links you can have your cake and eat it too - as long as the application knows where the data files are installed you can make a symlink of the binary to /usr/local/bin and run it without editing your PATH variable too! Isn't UNIX grand?
FreeBSD (Score:4, Interesting)
/bin for binaries needed to boot a corrupted system.
/sbin for system binaries needed to boot a system.
/usr/bin for userland binaries installed with the base system.
/usr/sbin for system binaries installed with the base system. The are not programs required to boot the system.
/usr/local/bin for locally installed user binaries such as minicom, mutt, or bitchx.
/usr/local/sbin for locally installed system binaries such as apache.
Large locally installed programs such as Word Perfect get installed in a sub directory of
FreeBSD has only about 400 programs in a complete
-sirket
How to handle PATH... (Score:3, Interesting)
export PATH
reset_path() {
NPATH=''
}
set_path() {
if [ -d "$1" ]; then
if [ -n "$NPATH" ]; then
NPATH="$NPATH:$1"
else
NPATH="$1"
fi
fi
}
reset_path
set_path $HOME/bin
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
set_path
PATH="$NPATH:."
unset reset_path set_path
I don't mind (Score:2, Interesting)
Re:he's pretty far off base (Score:3, Interesting)
Agreed, but does that make it right?
For the last few years, this is the kind of thing that has really been nagging me. All OSes seem to suffer from the same problem. Why are we so stuck with the mindset that traditions of the past shouldn't be challenged? Can't we, as "brilliant" computer scientist, start solving these problems and move on?
I recently demo'ed a good Linux distro to a friend and it finally dawned on me. When you load KDE, you are literally overwhelmed with options. My friend asked, "What is the difference between tools and utilities?". I didn't know. I tried to show him StarOffice and it took me a few minuets of digging in different menus.
No, I don't use Linux on a daily basis, and no, I'm not the smartest person in the world. But I think I see the problem. Everything seems to be an imitation of something else (with more bells and whistles). Where is the true innovation? Our computers and software are not significantly different than they were 20 years ago.
Why are we still using $PATH?
Re:Response (Score:3, Interesting)
Unfortunately this practice is common thanks to InstallShield being used by so many programs, as InstallShield always asks before deleting a so-called "shared" DLL. Keep in mind, half of the time the DLL is program-specific (ie not shared), and other times it's something the program itself did not install in the first place (was already there). I don't think Windows itself is to blame here...
Win2k still suffers from this, but if you do delete a DLL it almost always magically reappears. It's part of some scheme to protect the system from its users I believe, but it is a real pain when you actually want to remove a DLL...
As for the Unix side, I've always wondered about the organisation (or lack thereof) of programs. Many tools do IMO belong in central locations (cat, grep, ls...) but anything larger should have its own directory. I long for the day when I can say:
export PATH=$PATH:/usr/programs/*/bin
or something to that effect...
Most of your larger packages do attempt to install into their own locations; Apache by default ends up with
I personally hate RPM, and I generally snag a tarball over an
Unfortunately, I have my complaints about filesystem standards, but I don't have any solutions either, really. Too much software exists that depends upon our current system, though a proposed future standard might be nice. Maybe a new POSIX recommendation is in order... and once some years go by, software vendors will slowly migrate to the new standard... of course I don't know what that standard might be...
Missing the point (Score:2, Interesting)
One the aims of all the package management tools is to make the management easier. In particular this means, that you don't have to care, where are the files of application XYZ. So, if you wish to delete this, you ask your manager instead of finding all the subdirectories created by the package. You want to save your time, so you use the tools available. Era of manually managing everything is long gone.
Please note, that under Unices most of the applications are not installed in single directory - one is for binaries, one for documents, etc.
Under DOS and Windows, even the apps that went into their subdirectory had an annoing habbit of creating miscellaneous temporary/configuration files all over the place. And lack of file attributes did a lot to help this.
In some respects, I agree. (Score:3, Interesting)
Well we know about which road going where based on good intentions.
At any rate, part of the "problem" is there is a certatin point a section of the file system gets unmanageable. Where that is, quite frankly, varies.
RedHat has impressed me with its compatability but it does so with static libs. There are times when god forbid you should wish to compile something and get gripe messages that you window manager was done under X set of libs, your theme manager under Y's libs and your shared libs are of version Z.
That is just trying to update the WM, god forbid you wish to compile a kernel.
And with the static libs, the performance hit is astounding.
The other side, as with Slackware, is shared libraries can be as unforgiving as well.
Heh, as a newbie I deleted a link to a ld.so.X.
Hint: never, ever do this! ls, ln, mv et al stop working...oops.
Stupidity on my part, but, hey, I was a newbie. (finger; fire; burn; learn. simple.)
Back on track. Slack is fast, configurable but through sheer will, accident, or stupidity can be broken a lot faster (and in some cases fixed a lot faster).
Windows...well the sword cuts both ways. It impresses and suffers *both* of the good and bad points of RH/SL (or static and dynamic libs).
And, if the above does not either blow your mind or make you nod off consider OS X.1.1 (.1.1.1....)
Under OS X's packages system a 'binary/folder/application' (oye) can and does contain static libs. Ok, that can be good/bad.
Here is the kicker (and cool part): if it finds *better* or more *up to date* libs it can use them and ignore what *it* has.
If the new libs break the app, or cause problems, the application can be "told" or "made" to use only its own libs, or update the newer libs.
Most will see where that is going. It will be good to keep "static" then use "dynamic" or update the "dynamic/shared" libs.
The down side is the potential to fix one application and break 10+ others.
This has not happened...yet. However, the *ability* to make or break is there, just no information is given until a spec/CVS set of rules is fleshed out.
I will be the first to admit that the "binary folder" or "fat binary" (arstechnica.com article) idea sounded "less than thrilling"...until you realize the headache's it cures with this kind of file system bloat.
Think about it: You have an app, that is really a folder, that you can't see inside/manipulate/fix/break unless you know how *and* have a reason to.
In all three cases there are limits to even the most intelligent of design. Knowing this truth is easy to accept. Finding where it lies and where it breaks down...that is another discussion.
FHS 2.x - Filesystem Hierarchy System (Score:3, Interesting)
While not perfect, it addressed the following issues:
1) separating O/S from "other" packages;
2) maintain a sane place to put different packages;
3) support the notion of linking to specific package directories from a common place to keep PATH small;
4) was compatible with a number of "traditional" conventions.
Of course, FHS 2.1 has this concept of the "operating system" files and "other files". Presumably the "operating system" is that which the distro bundler provides... so Red Hat would be free to put as much as it wants under /usr. But this causes a problem if you looks at a common standard base for several distros, like the LSB.
Do you have a "standard base" part, and a "distro part", and then a "local part"? Clearly what's needed is a hierarchical way of taking an existing "operating system" and customizing it to a "custom operating system". Right now, FHS allows this for distro bundler and end user, but there is no support for the process iterating.
Of course, my experience has been with FHS 2.1 and have since moved on to employment elsewhere, so perhaps the FHS addresses these issues.
trouble... (Score:2, Interesting)
He does make a good point but I think once the history of the file system evolution is taken into account, the layout makes sense. The problem is, not every distribution adheres to the fs layout unwritten rules for various reasons and the result is a mess.
Hopefully, the Linux Standards Base will help to address this.
Re:The Alternative? (Score:2, Interesting)
unfortunaltly, not all software is written to the same standards. some code comes with Makefiles that have hard-coded paths, different build systems, hard-coded -L paths, things that make installing software on a random system painfull.
Package management is a way to standardize the way software is installed, upgraded, and removed.
I work in a system where we maintain our own software packages. the system was originaly slackware, but is now a custom distribution for our 200 odd linux workstations. Some days I would like to burn it to the ground, and replace it with a sane package managed system (debian). We have no real automated way to build the system up from source. software that conflicts is just piled on top of each other durring our automated install process. upgrading packages is painfull.
basicaly what we do to update the software on workstations is to have the workstations format, and re-install with an automated build process once ever 2 months. that way over a period of 2 months, we get all of our software fixes out to the workstations. we also have scripts to automate emergency patches to all workstations.
sometimes this is nice, because as we have to make changes to the system (new version of fvwm2 breaks old user config files). we only get calls every few days, and not all at once in the morning. and it keeps users from depending on
but having two people reproduce the work of a linux development group seems to be a waste of time in the long run. I am currently trying to come up with a good, solid sales pitch for my boss and senior admin to move us to a debian base system. and for us as a department to use our system maintence time to help give back to the debian community.
Why package management sucks (Score:5, Interesting)
It sounds very appealing. The problem is that a lot of the software I need right now (openLDAP, openSSL, etc) has packages that are a full development generation old. There isn't a 2.x package yet for openLDAP on RH 6.2, for example, and I don't think anybody in particular is in charge of building it.
Building from source is the only way to be current, although it is often an immense pain in the ass.
The other gripe I have is about packages failing to recognize libraries that are installed just because they weren't installed by a package manager. Yes, you can force a --nodeps sometimes and cross your fingers, but you shouldn't have to lie to the software to get it to work. Package managers should be a little smarter and be able to look around a little to satisfy dependencies.
If the package system really worked cleanly, it would be great, but I'm still using Pine 4.20 on my box because of conflicting dependencies in the 4.3x packages. I'm about to nuke the whole thing and build Pine from source - which I'll do as soon as I can get those library dependencies solved.
Grr.
Re:Translucent file system (Score:5, Interesting)
DESCRIPTION
The mount_union command attaches directory above uniondir in such a way
that the contents of both directory trees remain visible. By default,
directory becomes the upper layer and uniondir becomes the lower layer.
Non-FreeBSD users can read an online version [freebsd.org].
Re:Translucent file system (Score:3, Interesting)
It's my impression that recent developments in the Linux kernel (along with bind mounts [ibm.com], etc.) are moving towards making this easy to implement.
Re:Translucent file system (Score:3, Interesting)
Re:Related to yesterday's story (Score:5, Interesting)
I think the fundamental problem here is related to yesterday's story about new user interfaces [slashdot.org] [slashdot.org]. It's a problem of how and where storing our files.
You could also trace it back to the hierarchical database article [slashdot.org], which is when I started making a lot of posts on the subject. It seems there is finally a lot of interest being generated about this sort of thing.
I have no idea how to implement such a beast. I'm thinking about a RDBMS with indices on 'filetype' and 'application', but I would love to see something much more flexible. All pictures should be accessible under ~/pictures and subdirectories, all files relating to my vacation last year in ~/summer2000. Files relating to both should be in ~/pictures/summer2000 _and_ ~/summer2000/pictures.
This is exactly the sort of thing I'm doing with my Meta Object Manager (MOM) software called Mary. Metadata in the form of attributes and values is associated with each file/object and you can do a query (both textually and graphically) on that metadata. For simple paths like you describe, it is a value query irrespective of a particular attribute, but there is support for a more structured "path" (I actually call it a "focus" as it restricts your focus to a subset of the objects on the system) like /type=picture/location=Hawaii/year=2000. Because the focus items are metadata attributes, order is not significant. With such a system, there are no directories or symbolic links; it's all dynamically structured based on what your metadata focus is at any particular time.
Mary is just in the alpha stages at this point, but it already works well on the command line for the type of things you describe and I'm using it myself to manage nearly 350,000 objects that have been flowing through my system. I'm not exactly sure when it'll be ready for public consumption, and it'll require a GNUstep [gnustep.org] port to get working on Linux (I'm doing development on Mac OS X) systems. I was hoping year end, but I don't think I'll have the time. Summer 2002 has a nice ring to it, though. :-)
QNX has it (Score:3, Interesting)
Re:Package Management (Score:2, Interesting)
I agree, though. We don't necessarily need to separate programs for the most part. Espeically since the average program in the /usr/bin directory is only one file. But in the case that such a program requires several files (IE: Mozilla), these programs are already separated for the most part. So, it tends to be a non-issue anyhow. Mind you, the only real issue at hand would be the fact that one would have to create a standard for what gets separated, and what doesn't. But again, that's essentially a non-issue as a symbolic link typically takes a programs place within the /usr/bin directory anyhow.
So nothing new needs to be done.
Who needs directories? (Score:2, Interesting)
You'd run into naming conflicts, but those can be resolved transparently by the filesystem. Think of it like a database with a multi-column key: the filename and it's path.
Re:The Alternative [How about /usr/share ??] (Score:1, Interesting)
However the files which default go into the /usr/share... directory give me problems, because the share hierarchy that comes with the application usually has its own appname/version subdirectories and linking from /usr/local/{bin|man|lib|include..} gets clumsy (I think at least) comments?
- what it is the share hierarchy for anyway?
Yup, HP-UX is much worse. (Score:3, Interesting)
I've said it before - and I'm not the first or last to notice - HP-UX is a *train wreck* of a unix. HP puts Fibre Channel controllers that are necessary for the system to BOOT in the
--Charlie
Re:Who needs directories? (Score:2, Interesting)
I suspect that an elegant solution to this problem could be based on the core concept proposed here by CynicTheHedgehog, viz.: Organize the complex attribute data associated with each file system object by using a multicolumn table, containing various object capabilities and properties. And stop trying to encode these properties in a single name string, plus a chunk in a path variable.
Of course, this would require a new file system paradigm, and might have a tiny impact on existing distros.
Re:Why? (Score:3, Interesting)
You could theoretically (and actually, too, since you've got the sources
If everything works, there's actually no problem in glomming everything together in
This, BTW, is why I am fundamentally morally opposed to binary storage of configuration data (a la the win32 registry) versus plain-text storage. Binary is easier for the computer to handle, which is great as long as things work. Plain text is easier for me to handle, which is useful only when things break. Since the computer can work with either, plain text is preferable.
When things break, I must have the ability to go from zero knowledge about a broken system's configuration to a fully functional system as quickly as possible. Well-organized files that take full advantage of a hierarchical file structure, and plain-text config files, are much helpful in this situation.
(It just occurred to me that referring to the root directory as "/" at the end of a sentence produces an ambigous symbol; suffice to say I don't mean slashdot.org by "/.")
IF everything always worked, there would be no advantage to . You could eliminate the path statem
Interesting solution (Score:2, Interesting)
On first inspection, it of course makes identifying and finding what put whom where ludicrously simple.
find / -group KDE
On second reflection, it adds a finer layer to group/user mangement, and administrative delegations.
MacOS X - (Score:2, Interesting)
Darwin localhost 5.1 Darwin Kernel Version 5.1: Tue Oct 30 00:06:34 PST 2001; root:xnu/xnu-201.5.obj~1/RELEASE_PPC Power Macintosh powerpc
[localhost:~] local% ls -l
32
[localhost:~] local% ls -l
450
[localhost:~] local% ls -l
113
[localhost:~] local% ls -l
58
Not too bad, eh?