Red Hat's Linux Changes Raise New Questions 433
itwbennett writes "Last month two Red Hat developers proposed to replace the 30-year-old syslog system with a new Journal daemon. Initial reaction was mostly negative and 'focused on the Journal's use of a binary key-value form of data to log system events,' says blogger Brian Proffit. But now, says Proffitt, it seems that the proposal to replace syslog has less to do with the fixing syslog's problems than with Red Hat's desire to go its own way with Linux infrastructure."
Re:Is he not aware? (Score:4, Informative)
syslog the application or syslog the protocol? syslog the application? Yes, its past due, and things like rsyslog are much better.
syslog the protocol is fine.
The problem with this proposed replacement is that it does not fix anything. The only advantage it gives is to be able to tell if the logs were altered. That's it. You're far better off with a secondary/centralized logging system. Store your logs in text, compressed, encrypted, in a database, it doesn't matter. Just get them to a different location and then not only can you tell that the originals were altered, you can tell what was removed. All while using existing tools.
Re:One of the advantages of Linux (Score:5, Informative)
Nobody Ever Got Fired For Buying IBM
- GameboyRMH (bloody post limiter!)
Re:One of the advantages of Linux (Score:5, Informative)
I agree in general with "if it's not broken, don't fix it". Witness /. opinion regarding Unity/Gnome changes.
About Upstart, my lowly sysadmin opinion is this: It seems different from the other stuff Ubuntu's been doing in that, AFAIK, it's not alone in this. I think Fedora's going that way too.
Also, with Upstart I know if the webserver crashes for some reason, it'll restart without intervention. Yeah, I know, you're not getting to the root of the problem, but it beats being stuck to a top display looking if something burned.
Re:First post (Score:3, Informative)
Mod parent troll, Slashdot doesnt have articles, only comment threads. At least _IVE_ never seen any articles.
Re:One of the advantages of Linux (Score:5, Informative)
Agreed. I submitted this post [gerhards.net] yesterday, by the lead developer for rsyslogd (the most common syslog daemon in linux these days). He makes the point that most of the complaints made are actually wrong if they'd bothered to look at the last 10 years of development and IETF work around syslog.
Re:One of the advantages of Linux (Score:5, Informative)
What I don't understand is why you can't achieve both log security and log usefulness with the existing tools.
In a previous job (seems like a different life) - I set up all of the servers to utilize remote syslog. The syslog server then offered the log directory as a read-only NFS exports to each of the servers.
It was quick, it was easy, and it was secure. You could view the local logs on individual servers, but you couldn't alter them in any way except by generating log output.
Re:Avoid binary please!! (Score:4, Informative)
syslog is one of those things that needs to work when things break, so one can figure out what to fix.
Making it more complicated with more things to go wrong goes against this purpose.
Example:
Hmm, database server is acting weird, wonder what's wrong? I'll check syslog. Hmm, syslog is toast. Ah.....
Re:Good (Score:4, Informative)
Re:That works both ways (Score:4, Informative)
Well yes you can.
If you read the post from the developer
"My application needs traditional text log files on disk, can I configure journald to generate those?
No, you can’t. If you need this, just run the journal side-by-side with a traditional syslog implementation like rsyslog which can generate this file for you."
Just run journal and rsyslog and you have both systems in place.
Untile journal does everything odds are that it will be run with rsyslog in parallel. A boots and suspender type of solution. if you want to have it also generate a traditional syslog I would think that adding that functionality wouldn't be all that hard. If needed someone will add that feature.
Re:One of the advantages of Linux (Score:5, Informative)
Agreed. I submitted this post [gerhards.net] yesterday, by the lead developer for rsyslogd (the most common syslog daemon in linux these days). He makes the point that most of the complaints made are actually wrong if they'd bothered to look at the last 10 years of development and IETF work around syslog.
But about this part of what he wrote:
"Ages ago (2006?) I implemented high-precision timestamps (including TZ info) in rsyslog, and RFC5424 has brought them to the on-the-wire protocol. As far as I know, syslog-ng supports them for quite a while as well (but I am not a syslog-ng expert ;)). However, all distributions turn high precision timestamps off and set the dumb old format as this is a requirement to keep old tools working."
I enabled high-precision timestamps on my Debian system to get a feel for them. But I had to turn them off again: not readable enough, and took too much screen space making more log lines wrap. The tools weren't the problem; I just couldn't eyeball the damned things!
Re:One of the advantages of Linux (Score:5, Informative)
Except that logs are rarely used by the system. And on the few times where performance matters, the system is just interested on the last few lines, in real time. So, you just use them before (or while) writting.
Re:One of the advantages of Linux (Score:5, Informative)
The more a system becomes complex, the more one needs to see events as part of a whole and do some kind of analysis and correlation. This type of work is done more easily with databases. I like grep like everyone, but if I want to have a nice rollup of events based on time and source, I will get the info much more easily with a SQL query than with a regex piped into a reporting utility piped into a paging utility.
Typing 'grep <whatever>' is much much faster than: connecting to DB, typing query and realigning rows/columns on screen for readability.
I have to dig quite often through audit-log-like tables in DB created by our software and let me tell you that SQL doesn't make any correlations easy. Especially if we are talking about some production system were you end up self-joining a table with few dozen million rows (what you need to display for example the trivial thing as the time to the next/prev interesting event).
Neither the usual SQL tools are any good at displaying the data - as compared to displaying the SQL itself e.g. syntax highlighting. On text side of things, it takes minutes to create custom syntax for VIM for the problem at hand.
Why would you want to "reconvert" the Windows event log to text?
How many 3rd party applications actually use the Windows Event Log? I have seen probably one or two.
You know why? Because using it is a PITA - I have tried that twice as SW devel already in times of NT4 and W2K. (I was hoping to simplify critical error reporting of the Windows applications (including one GUI-less) and thought myself "WEL is just like syslog!" Oh gosh, Windows API proved me wrong.)
On Windows there is a lot of built-in capabilities for log exploring in Powershell or even in VBS/WMI. A toolbox contains many tools, not just grep.
Oh, so you like all that stuff over something as fool-proof, robust and simple as the grep? OK.
Re:One of the advantages of Linux (Score:5, Informative)
MySQL requires the daemon to be running, or at least access to some utility with the MySQL library. If a system has crashed or has reduced functionality due to system problems, a text log that can be scanned with the basic *nix stdio tools is a helluva lot more useful than a binary log.
I hate the Windows eventlog and binary logs in general precisely because they become rapidly less accessible the more issues a system has, which is quite often why you need to delve into syslog anyways. What exactly is the point to reinventing the wheel?
Re:One of the advantages of Linux (Score:5, Informative)
yeah, me too. while it's probably better to have the high-precision timestamps, for me it's more useful to have them readable.
I have the same problem with squid logs - they use unix time_t with milliseconds for the timestamp. more precise but less readable. I filter the lines through a small perl script to reformat the dates when i need to tail or process them:
#! /usr/bin/perl -p ;
use Date::Format
s/^\d+\.\d+/time2str("%Y-%m-%d\/%H:%M:%S", $&)/e;
this is similar to what is mentioned in http://wiki.squid-cache.org/Features/LogFormat [squid-cache.org] but with the improvement (IMO) that the timestamp still only takes one column (compared to localtime() making it take 5 columns), so it doesn't mess up other processing scripts that depend on the detail being in specific columns)
from this;
1322779413.527
to this:
2011-12-02/09:43:33
It would be annoying to have to do that for syslog logs too. I don't really need millisecond precision for my system logs anyway, near enough is good enough. All i need is accuracy and consistency across multiple systems - and ntp gives me that.
Re:One of the advantages of Linux (Score:5, Informative)
tail -f /var/log/messages
In mysql? How?
You missed a requirement: in a form that's still usable when the machine keeps going down hard in the middle of a boot. 'tail messages' still works, nothing to get corrupted or worry about a write-ahead log that can't get consistent.
Not that I spent the day today troubleshooting one of those or anything...
Re:One of the advantages of Linux (Score:4, Informative)
The problem here though is that the whole reason for the logs being plain text is that the time you most need to be able to read the logs is exactly when things are broken, most services won't start because of the breakage, and your special tools may not be working because most of the system just isn't there. With plain text files, if you can boot into the single-user maintenance shell (not even single-user mode, literally running the shell as PID 1) and get the filesystem the logs are on mounted, you can read the logs and see what happened. With a more complicated system you end up in a catch-22 where you need to fix the breakage to get the tools working to find out what you need to fix to get the tools working.
This is, BTW, why /sbin exists separate from /bin. You couldn't always guarantee that libc was OK, so /sbin had statically-linked copies of critical tools that you could use to fix the system after something had trashed the critical system libraries.